Chaos Engineering

Back it up Oh man, did I get ahead of myself in my last post. I started chatting tools, and I realize now that I really should have been talking more about why I’m using Sensu and Gremlin. But it didn’t occur to me until this past week at Monitorama. John Allspaw gave the keynote talk (Taking Human Performance Seriously) this year. While you can watch the talk here, I’ll highlight a few points here:

One of my earliest jobs was as an admin for an MSP. We’d routinely generate alerts that weren’t actionable, lacked context, and for most of our customers, were considered noise. From a monitoring perspective, it was bad. Customers didn’t trust in the alerts they received and often resorted to having some additional monitoring product installed on their systems. It’s safe to say that our auto-generated tickets and emails were largely ignored.