The Journey From DevOps to Cloud Engineering

All right, well good morning. My name is Maddie Stratton, I'm the director of developer relations at Ivan, and that's as much as I'm going to talk about that. But what I do want to talk about today is kind of an evolution, kind of a journey. DevOps is 12 some years old, we've been doing this for a while, and things have maybe changed. The world we're looking at, the work that we're trying to do — so we'll take a look at that. We're going to kind of look at how this journey goes from where we hit DevOps and then this idea of cloud engineering.

I like to start by looking back a little bit. History can teach us a lot of things, sometimes it can just teach us the things that are funny. I don't know, we'll see what happens.

So one thing — we talked about DevOps, a little bit over 10 years old. In 2009 was kind of this real inflection point year and a lot of this stuff that we're talking about. So one of the things that happened in 2009 is John Allspaw and Paul Hammond from Flickr gave this talk at the Velocity Conference called "10 Deploys Per Day: Dev and Ops Cooperation at Flickr." Sitting in the audience of this talk — first of all, by the way, this blew everybody's mind. 10 deploys a day, holy crap. Right now we're like you tend to place a nanosecond if you're Amazon or something, but it really blew everybody's mind. The fun fact is what most people got hung up on was the "how are you doing this with the technology?" And you're like, no, the cooperation part. But sitting in the audience was a fellow named Andrew Clay Shafer and he was tweeting about it, and best we can tell from Twitter archeology, is the first use of the word DevOps was in this tweet from Andrew. So it's all his fault.

Another thing that happened in 2009 is Jez Humble and Dave Farley published this book called Continuous Delivery, and this was based on a lot of work that they had done at Thoughtworks. There's this fella that I always considered as the unsung hero of continuous delivery, and it's a gentleman named Chris Reed. Does anybody know Chris Reed? Cool, Chris says hi, he actually asked me to do that. The fun thing about Chris is the genesis for this book was a paper called "The Deployment Production Line" that was written by just Chris and Dave North a couple years prior to that. But everybody knows the Continuous Delivery book. My introduction to this was Jez was a guest on a podcast called the DevOps Cafe, and this is when I was first starting to learn about DevOps. I remember viscerally I was driving down the Eisenhower Expressway in Chicago driving out to see my kids, and I'm listening to this podcast and Jez is talking about the idea of continuous delivery. I'm working at a technology company at the time, I'm working at apartments.com, and I'm yelling at the radio in my car, "Well that's fine, but that wouldn't work at apartments." And then Jez explains that they did this at HP with LaserJet firmware, and I sat there in my car and I said, "Oh."

That also leads to another thing that happened in 2009: Gary Gruver and his team at HP published a book called A Practical Approach to Large-Scale Agile Development. This is a book about DevOps, they just didn't call it DevOps. So you know we think about like "I can't deliver software using this or that," but they're doing this with hardware firmware. It was pretty impressive.

And this all sort of leads into kind of the story of how we got here. We started talking about Andrew had this tweet, and then there was an Agile event. At this event, Andrew Clay Shafer proposed a Birds of a Feather session — like the Open Spaces we do — about doing Agile system administration. And one person showed up to that Birds of a Feather session. It wasn't Andrew. It was a Belgian gent named Patrick Dubois. I don't think Patrick's actually here this morning, so it's good so he can't check my history. But then sure enough, a little bit later they did get together and decided to put on a conference to talk about these ideas. So if you ever want to know why we call it DevOps, it's because "Agile system administration" was too long of a name for a conference. So they put on this event in 2009 in Ghent called DevOps Days.

The thing I think is fascinating: if we look at this first DevOps Days, we look at the six talks that were there — you could actually probably see almost any of these talks at DevOps Days today. The other thing that I think is really interesting when we think about the evolution of DevOps Days is when we kind of moved on and we take a look at where DevOps Days came from. DevOps comes from this community, so much of the practice of this is embedded in this thing. We kind of look at the growth over the years. So 2009 there was the one event in Ghent, we had a couple in 2010, the first one in North America in Silicon Valley Mountain View. And we really see this big peak, and then — I don't know, we're still doing the analysis — we're not really sure what happened in 2020 why they dipped like that, but we're coming back.

One thing I think is very fascinating and really cool when you think about the global community that is DevOps coming from DevOps Days: in 2019, for the 10th anniversary of the first DevOps Days in Ghent, there was another DevOps get-together, and the day before we had an organizer Summit. People who organized DevOps Days from all over the world came together to spend a day in a DevOps Days kind of way to learn from each other. If you looked at the numbers, there were more people at that organizer Summit than were at the first DevOps Days in Ghent in 2009. So the growth is really something. We tend to look at how many DevOps Days there have been this year so far and we're back to 2017 numbers, and it's a mix of virtual and all that. That's sort of one of the places that's really important to think about — the community, where these come from.

But what the heck is DevOps? Let's sort of — I like definitions. Definitions help us speak a common vocabulary. We've firmly pointed the finger of blame at Andrew for that tweet, but what does DevOps mean? So Donovan Brown at Microsoft has one way of putting it that I really like, where he said it's the union of people, process, and products. I like it not just because of the alliteration, but it's about value. A couple years ago I pressured Andrew to give me a definition of DevOps — he didn't want to do it and I kept badgering him until he finally did. And he said, "Okay, it's optimizing the human experience and performance of operating software with software and humans." He gets really mad when I show this one because he obviously didn't copy-edit it, but the key is the word "human" comes up in there twice, and so that's cool. We heard from Donovan, we heard from Andrew, it's a bunch of dudes. And then Emily Freeman, who's the author of DevOps for Dummies, has another one I really like, which is about collaboration, ownership, and learning. These are all kind of this core idea — we're driving value, we're collaborating, it's about the people, it's about the socio-technical system.

And of course, while we can know what DevOps is, a few years ago I asked Twitter what DevOps is not. So if you think it's any of these things you're probably wrong. It's not your khakis, it's not just reading The Phoenix Project, et cetera.

There's a well-known way of describing DevOps using a common acronym called CALMS. If you've heard of CALMS I'm going to go through this a little bit — not going to go too in depth but I'm going to use this as kind of a reference point. CALMS stands for Culture, Automation, Lean, Measurement, and Sharing. The order matters only because it spells a word, none is more important than another. So you could also say it could be CLAMS if you like that better, or SMAC. This has stood up over a long time. It actually originally was called CAMS, and Damian Edwards and John Willis came up with that in 2010. We're still talking about it today, and we keep trying to come up with new ways to describe DevOps and they don't stick around, so I think this one's here to stay.

So when we think about CALMS, I'm going to review them just very quickly about where I think about them.

Culture is a big part of why we're here. I love this quote from Lloyd Taylor. He says you can't directly change culture. I'm sure we've seen leaders and executives that are like, "We're going to do a cultural transformation in our organization, we're going to change the culture — here's the new culture." It doesn't work that way. But what you can do is you can change behavior, and the behavior exemplifies and expresses that culture. And actually the way that you change behavior is by affecting incentives. A lot of times people say, "Hey Maddie, what's the most important DevOps book I can read?" And I will say that book is Freakonomics — learn about incentives, you'll understand DevOps. Now Freakonomics is problematic in its own ways, but you can kind of skip those parts. Everything doesn't stand the test of time.

Automation is what a lot of people tend to think of when they think of DevOps — mostly people not in this room necessarily — but they'll think it's all about that. And the thing about automation is it's super key. This is a quote from the Continuous Delivery book that I really love, which says asking experts to do boring and repetitive but technically challenging things is one of the surest ways to have them make mistakes, short of having them be overtired or drunk. That's sort of the world we're in, right? Because it's really boring stuff but you can't — you have to be highly skilled to do it. And automation is really key for the delivery of value that we're trying to do. We can't move at the scale if we have to do things by hand. This is also the part of DevOps that requires the least amount of convincing to anybody because this is the fun part, this is the toys, this is the tools. People say we over-rotate on culture and it's like — because I don't have to convince a bunch of engineers to play with tools, we're going to just do that already.

Lean came in — at first it was CAMS, Jez added Lean a few years later. This is thinking about lean manufacturing principles and kind of it's a lot about waste reduction. A couple of the key ideas in this are elimination of waste and this idea of value stream mapping. I'm not going to dig too much into this, but I always find this really interesting when I work with organizations: very few people look at the whole process from beginning to end and understand everything that happens. We know our part. And when you kind of get everybody together and say, when we start if we're going on that journey from idea to feedback to the customer — or sometimes we say "commit to cash," right, the time between a commit and when we're getting paid — then you start to look at it and you go, "Oh, nobody really thought about that, there's this chunk here." If you'd like to learn more about value stream mapping, Steve Pereira, who's one of the organizers of DevOps Days Toronto, is a great value stream person. But we're really trying to look at reduction of bottlenecks, reduction of waste, et cetera.

When we think about Measurement, it's not about justifying our existence with metrics, but we don't know that something we're doing is changing or affecting things in the right way unless we're measuring it. I think this was really in a lot of ways exemplified by the State of DevOps Report that first came out a few years into the DevOps movement, because before we had that, before we had DORA and the DORA metrics and all those things, we kind of knew DevOps was good but we didn't know why. We had to take a lot of things on faith. So measurement helps us be able to see that we are making changes for positive or negative effect.

And then Sharing is kind of what ties us all up together and it's oftentimes one of the hardest things to do in an organization. Knowledge is power — people feel, especially at an executive level, that things are on a need-to-know basis. The reality is the list of things that are need-to-know is smaller than we think.

So this is pretty cool. I think when I look at this I'm like, hopefully we're maybe a little bit preaching to the converted in some ways. But when I first started learning about DevOps I'm like, this sounds like an amazing way to do work. So this is going on — CALMS is 10 years old, DevOps Days was over 10 years ago — all these principles and ideas, we must be living in software Nirvana now. Things must be so good. What the hell happened?

How are we still having these same conversations?

So maybe one of the things that comes up is: is it all about automation? There's this idea that when we talk about DevOps practices and stuff they all go back to automation and tools. It's like, how are we automating this process? And I mentioned that — I said automation is where we gravitate to, but DevOps isn't just about automation, that's just one of the letters in CALMS. Is DevOps just Kubernetes? Can we just throw some container orchestration at this stuff and we'll get ourselves a bunch of value? And the sad reality is no. I mean I love Kubernetes — my car's license plate is "CUBE CUDDLE" — but really it used to be "DEVOPS." I have a keen sense of where the industry is going and I illustrate it in my vehicle registration.

It's not that. And then this is one of my favorite little things: a few years into the DevOps thing there was a thankfully short-lived movement called Enterprise DevOps. The idea of Enterprise DevOps was it was DevOps without the C — it was AMS, you didn't need culture. And one of the proponents of this was fond of saying "culture is for yogurt." The thing that saved us from all of that is a little thing called the DevOps Enterprise Summit, because they put on DevOps Enterprise Summit and all these enterprises came out and talked about what they were doing and it was all about culture, and that whole little movement went away.

But this is the thing: these are the hard things. And the reality of the problem we have is that DevOps is being sold to you. Someone once said "you can't buy DevOps, but I can definitely sell it to you." And that was me. You've reached a certain level of either notoriety or self-satisfaction when you quote yourself in your talks. But this is a reality — we got into this position where DevOps is something that has been marketed and sold. I worked for DevOps vendors, it's fine. But along the way, I said I wanted to blame Andrew for the term DevOps, and what happened is we got into this thing where we said, "We had DevOps — cool. But that doesn't seem very inclusive. There's more to delivering software than development and operations, so we probably need DevSecOps." Okay, but what about the biz? We have everyone in the business, we're doing business value. Do we need to get serverless in there? We need to do DevSecDBOps, we have the Ops for our DB Ops. And then somewhere along the line someone's done DevOps DevOps, and it kind of sucks. And the reality is the alternate for this slide would be that astronaut meme where it's like, "Oh, DevOps includes security? It's always has been." Right? But we don't understand nuance, and so we have to create all these other words to explain the thing that we've been trying to say for 12 years. And this is the problem — words are hard, communication is hard. All this comes down to communication. But don't worry, I have more words for you, so we will solve the problem with more words.

I think one of the problems that we're in is that we had this idea, we've been working on this idea, it's perhaps been misunderstood, it's been reappropriated, it's been redirected. So there are some more emergent practices that are going on now. And I think when we look at today, the other thing is we've been talking about this for 12 years — our landscape and our world looked different in 2009, in 2011, in 2015, even, than it does today. We're in a much more cloud world. A lot of things, when we think about this emerging practice of cloud engineering — one of the ways it's been described is: modern cloud engineering is applying standard software engineering practices to our infrastructure, our application development, and our compliance. That sounds real familiar to me. This sounds like a lot of what we've been trying to do with DevOps.

When we think about cloud engineering, it's the sort of idea where we have these cloud resources which we use to build platforms, we deploy our applications, and then we manage them with policies. This is sort of the core of cloud engineering. And what I want to do here is take these three tenants — build, deploy, and manage — and say how do we apply CALMS to that? How do these things go together? How do we operationalize DevOps? How do we apply these ideas to things that maybe fit a little closer to our life cycle?

One of the strengths and also weaknesses of the DevOps movement was to not be prescriptive, and here's the thing — it was kind of a good idea. We're like, "We're not going to tell you how to do these things, it's the ideas, it's the outcomes." Awesome. Well, you know what happens when we don't provide ways to do it? People will try to sell you a tool to do it. So we're going to be a little more prescriptive.

When I think about the build part — this sounds like this is just about writing code, this is about your software engineers, this is about building your applications. But when we think about the build area of cloud engineering, it's about creating the services and the infrastructure that provide what our customers need. In today's world we're using cloud resources generally to build these applications, services, and infrastructure. So it's all the things that we build. When we kind of think about that, you know, maybe they make up a shared services platform. We've had a lot of conversations about platform engineering — this is the new hotness. Again, new words, new words. But at a certain scale, shared services platforms really make a lot of sense. James Governor had a comment — I wish I had the tweet in front of me — but it was something to the effect of everybody's in a race to the bottom to recreate Heroku. Right, that's what we're all trying to do, and so we want to do our own Heroku inside.

There's a reason for that, because it's really advantageous to create reusable infrastructure components. The things we're building — a lot of this stuff comes as common second nature from a software engineering perspective, and then we're still learning this stuff in infrastructure. The reason of being able to create these reusable components — what this helps us do is it's letting us focus on our differentiators rather than reinventing the wheel. Reusable components throughout how we're doing infrastructure in our organization gives us a consistent and standard implementation using good practices and existing practices of our teams inside our organization, and expressing our own — I just said "best practices," I never say that, but I do say "practices," there's more than one — and common configurations and approaches when we build our infrastructure. This lets us apply existing frameworks and tools.

Here's the thing that's interesting — I've seen this happen in all sorts of cases in lots of organizations where we want to create things because it's fun. I spent a little bit of time helping different government agencies in the United States with their cultural transformation. Yeah, that was fun, that was in 2020 too. I used to say, "Hey, are you the US Department of Continuous Delivery?" Because focus on that. We laugh at it — you're supposed to laugh, that was a joke.

But we do that, and there are enterprises that do this. There's one particular one in the United States that I'm not going to call out. It's interesting because they're generally considered a DevOps darling — they're one of the ones we look at and say, "Oh, X company does DevOps super well." And they have been continually building new tools that already exist. The idea behind that I think is, "Well, this will make engineers want to work here because we work on this cool stuff." But that's not what you do. You are a retail company, you sell stuff. You don't build deployment platforms, you don't build PagerDuty clones. And the thing is you're focusing your stuff on that and it's never your most important thing. So we want to focus on differentiators, number one, because that is providing the most value to your organization. But also your side project doesn't get the funding and attention that it does when it's your main thing anyway. That's a whole other talk.

So I want to kind of step through the different ideas of CALMS in each of these and say how does it approach somebody thinking about how we're doing the build stuff. The cultural ramification of that is, again, we talked about the focus on differentiators — we're creating a common development experience, whether you're developing in different software engineering teams across different feature teams. But even your infrastructure engineers are having a similar experience in how they build things as your software engineers, as your security engineers, and that drives empathy. The lack of empathy comes from lack of understanding. The easiest way to do empathy is to actually have had the same experience — being empathetic to someone who has a completely different lived experience is way harder. So sometimes we can cheat.

When we think about the automation part, reusable components lets us move more quickly, leveraging those ecosystems as I talked about, and avoiding those bespoke implementations — the artisanally crafted pipelines, if you will.

From a Lean perspective, this is helping us focus on value. What is the most important value that you're providing? Again, this doesn't mean that everything's about the almighty dollar or pound or euro or whatever currency — it's not just about capitalism. But the value: is it providing value to your internal or external customers appropriately? And this also lets us review for improvement. The more that we're having this shared thing — consistency breeds visibility. The more similarly we're doing things over and over again, it surfaces our ability to see. And from a Sharing perspective, by leveraging an existing ecosystem, that promotes outside sharing. One of the big values of open source in general is that we can learn from each other. The stuff that is not your secret sauce — it does not help you to not share it. You do not disrupt Netflix by knowing how they ship software. Netflix is not going to open-source the recommendation algorithm, but their deployment thing — you can't go disrupt them because you know how they do that. We can all get better together, and this helps us learn from existing practices.

So now we kind of go to the next tenant, which is deploy. It doesn't count until it's in production. I mean, I've written a lot of code that no one's ever seen, and that's a good thing that a lot of people haven't seen it. Code and infrastructure don't give any value until it's in front of our users or customers or colleagues. But doing it in a manner that's highly efficient and quality consistent is what's really key. Deployment processes that take too long or require too many manual steps can absolutely block us from getting new features to our customers or resolving service issues when they have incidents. We have to be able to do this stuff relatively quickly.

When we apply software engineering practices to our deployment process, this ensures that we ship the same way every time. And I mean every time. A lot of times when we talk about having a process for deploying software, one of the first things that comes up is, "Okay, that's great, but what about when there's an outage? I need a way to bypass the process because this important thing happened." I spent a fair amount of time working in ITIL shops and ITSM shops, and the one thing I've learned is that when you have an exception process, what that means is everything becomes an exception. If you're like, "Well, if you follow the standard process it'll take two weeks, but if it's an exception it can go in three hours" — guess what happens? Everything becomes an exception.

It's become common practice for us to apply the principles of continuous integration and continuous delivery to how we build application software, but when you apply these same principles with our infrastructure, that means that our new and updated infrastructure resources meet our quality controls and help us understand what changed. And security is just another aspect of quality in my mind. This is letting us put quality and security checks along every change that we're making.

Checklists are great, but we need to automate them. They're built by people, but we let automated systems apply those checklists, which helps us with the consistency of doing things the same way each time.

When we treat our infrastructure as part of the application, it's not just sort of a side thing. The application is what creates value in many cases; the infrastructure is there to support that.

How do we apply these things to CALMS? The culture again — it doesn't count until it's in prod. We have to get it there, and this is that fast feedback, these short feedback loops of iterative development, which enables our continuous improvement. When you think about our pipelines, pipelines are great, and they're especially great when they're relatively consistent inside your organization. Every squad doesn't need a completely different way to deploy software. And again, I love checklists.

The wonderful and frustrating thing about computers is they're jerks. When we have a process that a person has to approve, people do favors for each other sometimes. We've all done it, it's fine, it's not even a judgment. You might be like, "Hey, I gotta get this thing out, the CMO's breathing on my neck, I gotta get this thing. Hey, you know what Maddie, can you just push it? The test is red." It's like, "I just — push it?" "All right, you know what, for you this one time." Computers: "Screw you buddy, I don't care." So that's the wonderful and terrible thing about computers — they are jerks. And checklists are great, let's automate them.

This again enables fast feedback and gives us visibility to the supply chain. This is a pretty hot topic of the last year or two — software bills of materials, right, understanding all the things that bring that together. And we have a consistent approach to our deployment that gives us visibility to also identify those bottlenecks as well as those threats.

This is giving us our visibility. We're able to measure that cycle time. Cycle time — again, that "commit to cash," how long does it take to get from an idea to value. It's not just about speed though. We alluded to it — it was Flickr, it was "10 deploys a day, holy cow!" And you're like, "All right, Amazon deploys a thousand times a minute, we should do that too." No. Should you? Are you Amazon? But it's the ability to move at the speed that's appropriate to what your business needs to be able to provide that value.

And it gets into this: everyone knows what changed. When something goes squirrely, what's the first thing we ask? "What changed?" Because stuff doesn't just break magically — something changed. Maybe we didn't change it, maybe our customers changed behavior. We want to be able to isolate those pieces.

And then finally, when we think about the manage piece — getting our services into production is a key step, but it's not the end. Just getting into production — I mean, this is not your first DevOps talk you've seen, right? You've all seen the Wall of Confusion slide, I'm sure — the old, terrible, olden days when those jerk developers used to throw stuff over the wall to the poor downtrodden Ops people. Theoretically we don't think that way anymore, but we still do a little bit. We're like, "Okay, well at least we shipped it." But we have to continue to run our services, they don't just stop. We're not literally shipping software on a CD-ROM to people anymore. Maybe your use case — anybody actually shipping on physical media? I know someone's got to be doing it. Maybe I'll have an Open Space about that. But the thing is, our customers are constantly using our services and applications. So it's not just about continuing to run or to add features — our customers' behaviors change, their uses change, everything's a moving target. We have to be able to manage all those resources.

You might have heard the adage that security is everyone's job. I have a lot of thoughts on "shift left." Again, it's a short term that has nuance, and we suck at nuance. We want to shift the attention to the left — not just dump all the work to the left. But again, what do you think most people are doing?

The idea — the real truth is we consider security and compliance, and I talk about uppercase and lowercase compliance, right — whether it's regulatory policies or just our organizational policies — to be closely integrated into our work. So if we're treating our policy as code, just like infrastructure is code, this is really powerful. Because when we express those policies as code rather than prose in a document written up in some Google Doc or Word document, we can apply those checks just like any other test. This also gives us a common vocabulary across our compliance, our security, our operational teams, regardless of where they sit in the org chart, and that gives us visibility again.

This is a nuanced statement: "security is everyone's job" does not mean you need to get rid of people whose jobs are security. It means it's just the thing we think about. I always used to say that security and quality are closely linked. If I were to say, "Okay, in our world we do not do any quality checks until the very last week before we deploy something," you would think I was nuts — or you would have worked at some of the places I've worked. But we do that with security all the time. We do hardening sprints and we only pay attention at the end, and then we of course have to ship it, which means we get an exception from the security team. And I've got news: the bad guys on the internet don't care that you have a note from your mom that says it's okay. They're still going to hack you.

So again, this common vocabulary enables our empathy. The other thing is that controls and processes enable and enhance. Guardrails are actually really empowering. If I put things in place so that I can then feel like it's very hard for me to accidentally drop the production database, then I'm going to be a little more confident in making other changes. So guardrails enable, they help with our culture. Our collaborations are enabled through a common code experience, and this increases common understanding across disciplines, which is kind of the core of DevOps.

The automation part again — computers, in addition to being jerks, also can't lie. There's a whole other conversation about audit theater I've been involved in. So many audits that came down to, "Maddie, do you always do this?" "Oh yes, I do, of course, yes of course I do that every time." The computers will actually give us the actual answer. It gives us trust in our process and our checks.

When you think about the Lean approach to that, this is letting us determine the improvements for safety and express those value stream changes in a codified way. This gives us visibility into our current policy, because when our policy is expressed in a Word document sitting in a SharePoint folder somewhere, we don't necessarily know what's true. I used to think that infrastructure as code is executable documentation — the same thing is true for executable policy. It gives us our view of our current state of compliance, and most importantly to me, it helps us understand when policy and value collide. It's like, "Hey, we have this policy but it's giving us a challenge in expressing value." We want to see where those happen so we can then have a conversation and figure out what we should be doing.

And from the Sharing perspective, again we have a shared vocabulary, helping us utilize success patterns in different parts of our roles and teams and squads inside our organization, and helps us share the learning when we do have incidents and things. One fun fact real quick: when J. Paul Reed did his Master's thesis on post-incident reviews, he found that the larger the organization, the less likely teams were to share their postmortems with each other — whereas ironically those are the organizations that need to do it the most.

So what I'm suggesting here is basically: let's take DevOps back. What was DevOps really about? It's about those principles, about those ideas. Let's work together, let's bring it back to what it's been.

Thank you very much. You can find me on Twitter. My slides and a bunch of other resources are at speaking.mattstratton.com. We occasionally still do my podcast called The Rest of DevOps, and I look forward to chatting with all of you in the Open Spaces.