Avengers Assemble - The Thanos Incident
A presentation at DevOpsDays Amsterdam 2022 in in Amsterdam, Netherlands by Matt Stratton
AV ENGERS ASSEMBLE T HE T HANOS INCIDEN T
@mattstratton
SPOILER WARNINGS @mattstratton
@mattstratton
SO, WHAT HAPPENED? @mattstratton
SO, WHAT HAPPENED? ▸ We will start by creating a post-mortem on the incident of “The Snap” ▸ Our approach will be the address this in a blameless fashion ▸ We want to understand what happened, as well as the process the Avengers used ▸ For purposes of this discussion, “The Avengers” will include characters not generally considered Avengers, please bear with me @mattstratton
SO, WHAT HAPPENED? @mattstratton
CONSTRUCT OUR TIMELINE ▸ Stick to the facts ▸ Include key decisions and actions taken by responders ▸ Avoid evaluating what should or shouldn’t have been done ▸ Start the timeline at a point before the incident began @mattstratton
HIGH LEVEL TIMELINE ▸ Thanos obtains the Power Stone and the Space Stone ▸ Thor and the Guardians of the Galaxy decide to split up - Thor to head to Nidavellir with Rocket and Groot, the rest to head to Knowhere ▸ Thanos retrieves the Reality Stone from The Collector on Knowhere ▸ Dr. Strange uses the Time Stone to view millions of possible futures ▸ Thanos sacrifices Gamora on Vormir to obtain the Soul Stone ▸ Several team members attempt to recover the Infinity Gauntlet from Thanos on Titan @mattstratton
HIGH LEVEL TIMELINE ▸ Dr. Strange decides to exchange the Time Stone for Tony Stark’s life ▸ Shuri works to remove the Mind Stone from Vision ▸ Various team members attempt to defend Vision in Wakenda ▸ Thanos obtains the Mind Stone from Vision ▸ Thor attacks Thanos but is unable to defeat him ▸ Thanos snaps his fingers, wiping out half of all life @mattstratton
Groot Cause Analysis Credit to @djpiebob and @this_hits_home @mattstratton
SYSTEMS ARE COMPLEX @mattstratton
THERE IS NO SINGLE ROOT CAUSE OF MAJOR FAILURE IN COMPLEX SYSTEMS, BUT A COMBINATION OF CONTRIBUTING FACTORS THAT TOGETHER LEAD TO FAILURE @mattstratton
Blameless @mattstratton
WHY DOES BLAMELESSNESS MATTER? ▸ This impulse to blame and punish has the unintended effect of disincentivizing the knowledge sharing required to prevent future failure ▸ The goal of the postmortem is to understand what systemic factors led to the incident and identify actions that can help improve the resilience of the system ▸ Stay focused on how a mistake was made instead of who made it @mattstratton
WHY IS BLAMELESSNESS HARD? ▸ When processing information, the human mind unconsciously takes shortcuts ▸ We are hard-wired from millions of years of evolutionary neurobiology to tend to blame ▸ The human mind optimizes for timeliness over accuracy, which is reinforced by cognitive biases @mattstratton
@mattstratton
COGNITIVE BIASES ▸ Fundamental attribution error ▸ Confirmation bias ▸ Hindsight bias ▸ Negativity bias @mattstratton
FUNDAMENTAL ATTRIBUTION ERROR @mattstratton
CONFIRMATION BIAS @mattstratton
HINDSIGHT BIAS @mattstratton
NEGATIVITY BIAS @mattstratton
HOW TO AVOID BLAME
ASK “WHAT” AND “HOW” QUESTIONS RATHER THAN “WHO” OR “WHY” @mattstratton
CONSIDER MULTIPLE AND DIVERSE PERSPECTIVES @mattstratton
ASK YOURSELF WHY A REASONABLE, RATIONAL, AND DECENT PERSON MAY HAVE TAKEN A PARTICULAR ACTION @mattstratton
ABSTRACT TO AN INSPECIFIC RESPONDER @mattstratton
CONTRAST WHAT YOU DID NOT INTEND WITH WHAT YOU DO INTEND @mattstratton
ALL PRACTITIONER ACTIONS ARE ACTUALLY GAMBLES, THAT IS, ACTS THAT TAKE PLACE IN THE FACE OF UNCERTAIN OUTCOMES. Dr. Richard Cook @mattstratton
YOU NEVER KNOW. YOU HOPE FOR THE BEST, THEN MAKE DO WITH WHAT YOU’VE GOT Nick Fury @mattstratton
What can we learn? @mattstratton
@mattstratton
Have An Incident Commander @mattstratton
DELEGATE AND COORDINATE @mattstratton
DECISION MAKER @mattstratton
SINGLE SOURCE OF TRUTH @mattstratton
SHOULD NOT BE A RESPONDER @mattstratton
Who should be the Incident Commander for the Avengers? @mattstratton
Rotations Matter @mattstratton
Escalate and Bring in Help @mattstratton
Hero Culture @mattstratton
Teamwork Makes the Dream Work @mattstratton
SHARE ON-CALL ▸ Carol Danvers is the only one who carries a pager! ▸ The more folks on-call, the less the load for everyone ▸ Having a consistent mechanism for bringing in experts for incident response is key @mattstratton
@mattstratton
SO WHAT HAVE WE LEARNED? @mattstratton
@mattstratton
And perhaps most of all… @mattstratton
@mattstratton
https://speaking.mattstratton.com @mattstratton
ACKNOWLEDGEMENTS ▸ Jeremy Meiss @IAmJerdog ▸ Karissa Peth @karissapeth ▸ Nell Shamrell-Harrington @nellshamrell ▸ Sarai Rosenberg @saraislet ▸ Ryan Kitchens @this_hits_home @mattstratton
Over the course of the Avengers storyline, everything has been leading up to the ultimate outage—when Thanos snapped his fingers, eliminating half of all life in the universe.
Come along with me on a journey to perform a retrospective on this greatest of all incidents in the Marvel universe. What were the contributing factors? How could the Avengers have followed better incident response procedures? And can this be reviewed in a truly blameless fashion?
In this talk, I will revisit the storyline across the Marvel Cinematic Universe that led up to a critical event: the “Snap,” when Thanos removes half of all life in the universe at the end of Avengers: Infinity War, as well as the resolution displayed in Avengers: Endgame. We will explore the activities of the incident response teams (the Avengers, the Guardians of the Galaxy, and more), to discover what they did well, what they could have done better, and why S.H.I.E.L.D. needs to invest in better Incident Response training.
The audience will learn how to engage in productive Incident Response practices, conduct blameless postmortems, and even why a properly used pager (ala Captain Marvel) can be a key element in successfully navigating even the most dire of universal crises.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Avengers: Endgame Timeline Explained: No Plot-Holes
-
We made a timeline showing the entire history of the Marvel Cinematic Universe leading up to ‘Avengers: Endgame’
-
PagerDuty Postmortem Guide
-
PagerDuty Incident Response Documentation
-
PagerDuty Postmortem Documentation
-
“Blameless” postmortems don’t work. Here’s what does.
-
Blame. Language. Sharing.
-
The infinite hows
An argument against the Five Whys and an alternative approach you can apply.
-
Crucial Accountability training
-
How Complex Systems Fail
-
Postmortem Action Items: Plan the Work and Work the Plan
-
Three analytical traps in accident investigation
-
Using Postmortems to Understand Service Reliability
-
Inhumanity of Root Cause Analysis
-
The problem with ‘5 whys’
-
The problem with ‘5 whys’
Buzz and feedback
Here’s what was said about this presentation on social media.
