The Psychology of Chaos Engineering
A presentation at DevOpsDays Baltimore 2019 in in Baltimore, MD, USA by Matt Stratton
The Psychology of Chaos Engineering Matty Stratton, PagerDuty @mattstratton
Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. https://principlesofchaos.org/ @mattstratton
What chaos engineering is NOT @mattstratton
@mattstratton
It’s not about breaking things @mattstratton
@mattstratton
Experimenting in production is preferred @mattstratton
You can’t do this without good measurement @mattstratton
Minimize your blast radius @mattstratton
Some helpful tools • Netflix Simian Army - https://principlesofchaos.org/ • Gremlin - https://www.gremlin.com/ • ChaosToolkit - https://chaostoolkit.org/ @mattstratton
But what about the people? @mattstratton
How does it make you feel to know Netflix practices chaos engieering? @mattstratton
What about your bank? @mattstratton
@mattstratton
Management can get… …nervous @mattstratton
Consider your words @mattstratton
It’s about the philosophy @mattstratton
@mattstratton
Safety first @mattstratton
• Know your conditions - when will you shut down the experiment? • This isn’t about causing stress on your people - be transparent • There are humans at the other end of those numbers @mattstratton
Further Reading • Chaos Engineering Traps - Nora Jones bit.ly/2Pr53ZH • ChaosCat: Automating Failure Injection at PagerDuty bit.ly/2UCbdXN • ChaoSlingr: Introducing Security into Chaos Testing bit.ly/2GDZN1V @mattstratton
Chaos Engineering, failure injection, and similar practices have verified benefits to the resilience of systems and infrastructure. But can they provide similar resilience to teams and people? What are the effects and impacts on the humans involved in the systems? This talk will delve into both positive and negative outcomes to all the groups of people involved - including users, engineers, product, and business owners.
Using case studies from organizations where chaos engineering has been implemented, we will explore the changes in attitude that these practices create. This talk will include a brief overview of chaos engineering practices for unfamiliar members of the audience, but the main focus will be on human elements. I will discuss successful implementations, as well as challenges faced in teams where chaos was a “success” from a technical perspective, but contained negative impact for the people involved.
After seeing this talk, attendees will have a better understanding of the human factors involved in chaos engineering, good practices to care for the people and teams working with chaos, and be even more excited about this practice.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
