The Psychology of Chaos Engineering
A presentation at Agile + DevOps East 2019 in in Orlando, FL, USA by Matt Stratton
The Psychology of Chaos Engineering Matty Stratton, PagerDuty @mattstratton
Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. https://principlesofchaos.org/ @mattstratton
“By running Chaos Monkey in the middle of a business day, in a carefully monitored environment with engineers standing by to address any problems, we can still learn the lessons about the weaknesses of our system, and build automatic recovery mechanisms to deal with them. So next time an instance fails at 3 am on a Sunday, we won’t even notice.” -Netflix Technology Blog, 2011 https://bit.ly/netflix-chaos @mattstratton
What chaos engineering is NOT @mattstratton
@mattstratton
It’s not about breaking things @mattstratton
@mattstratton
Experimenting in production is preferred @mattstratton
You can’t do this without good measurement @mattstratton
Minimize your blast radius @mattstratton
Something’s broken… …it’s your fault @mattstratton
Some helpful tools • Netflix Simian Army - https://github.com/Netflix/SimianArmy • Gremlin - https://www.gremlin.com/ • ChaosToolkit - https://chaostoolkit.org/ @mattstratton
But what about the people? @mattstratton
How does it make you feel to know Netflix practices chaos engineering? @mattstratton
What about your bank? @mattstratton
@mattstratton
Management can get… …nervous @mattstratton
Consider your words @mattstratton
It’s about the philosophy @mattstratton
@mattstratton
Safety first @mattstratton
• Know your conditions - when will you shut down the experiment? • This isn’t about causing stress on your people - be transparent • There are humans at the other end of those numbers @mattstratton
Further Reading • Chaos Engineering Traps - Nora Jones bit.ly/2Pr53ZH • ChaosCat: Automating Failure Injection at PagerDuty bit.ly/2UCbdXN • ChaoSlingr: Introducing Security into Chaos Testing bit.ly/2GDZN1V @mattstratton
https://speaking.mattstratton.com @mattstratton
Pagey Says…. @mattstratton
Session Evaluations in the App
Chaos Engineering, failure injection, and similar practices have verified benefits to the resilience of systems and infrastructure. But can they provide similar resilience to teams and people? What are the effects and impacts on the humans involved in the systems? This talk will delve into both positive and negative outcomes to all the groups of people involved - including users, engineers, product, and business owners.
Using case studies from organizations where chaos engineering has been implemented, we will explore the changes in attitude that these practices create. This talk will include a brief overview of chaos engineering practices for unfamiliar members of the audience, but the main focus will be on human elements. I will discuss successful implementations, as well as challenges faced in teams where chaos was a “success” from a technical perspective, but contained negative impact for the people involved.
After seeing this talk, attendees will have a better understanding of the human factors involved in chaos engineering, good practices to care for the people and teams working with chaos, and be even more excited about this practice.
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
