Full-Service Ownership With PagerDuty

Transcript

Scott McAllister: Welcome to Page It To The Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve both system reliability and the lives of the people supporting those systems. I’m your host, Scott McAllister @STMcAllister on Twitter. We’re going to try a different type of show for this one. In the studio today is just me and George. And we’re going to talk about a bit of the work that we’re both particularly excited about. That work is version two of our full-service ownership ops guide. We haven’t really talked about ops guides before. So George, do you want to tell our audience what ops guides are?

George Miranda: Sure. Ops guides are something I think are fairly unique for our industry. There are sort of like a white paper, except they’re actually useful. They’re a step-by-step process guide. And these process guides are vendor agnostic. They’re free and they’re open source frameworks that are designed to walk you through how to accomplish newer and more modern types of operational tasks. And so I kind of joke about white papers, right? Some white papers actually are kind of useful. But when I say that ops guides are like a white paper, what I mean is that they have a tremendous amount of depth and detail. And what’s unusual though, right? Is that white papers are typically about a technical product, right? They’re deep dives in a product specifics. And we start with some generic use case. And the idea is that by following this guide, it illustrates how you adapt this tool to accomplish the things that you need for that use case. And so ops guides are a little different because they focus on process, right? And I think that’s something that’s super needed and super unique for the industry. And so I’ll give you an example, right? Two of our most popular ops guides are incident response and postmortems. And those guides are designed to create a bridge between theory and practice, right? We talk about effective incident response, but how do you actually do that in real practiceable ways, right? Blameless postmortems, like what even does that mean? And what can you actually do in a real way to help you practice that? And so, it’s not like one process works for everyone. And so these process guides are written a little bit like a white paper, right? Like we have this generic type of use case that we’re going to accomplish. So this guide shows you how to do it in a flexible way that illustrates how these things actually work when you put these theories in a practice, right? So ops guides layout, a step-by-step way of doing things that you only typically hear about in conferences or in podcasts like this one. So what we’re going to do is today, we’re going to talk about a new ops guide that we just recently released. I’m pretty excited about this one. I did not realize I was as excited about this ops guide when I first heard about it, as I was towards the end when I saw how this thing was unfolding. And so the ops guide that we’re going to talk about today is a step-by-step way to transition into a full service ownership model. Sometimes even when I say that, I get a little eye-rolly because it sounds a little different like full-service ownership. What does that even mean? So Scott, what is full service ownership? Can you tell us what that is?

Scott McAllister: I see what you’re doing there, George. You’re seeing if I actually read this ops guide, aren’t you? See? So, actually, yeah, so full-service ownership is essentially saying that those who build something are responsible for supporting it all through every stage of its life cycle. It brings developers and product people closer to the customers and leads you to honestly write better software. Because now you’re not just taking code and checking it over the wall and saying here, someone else needs to release that. And so taking full ownership of your service that you have is key I think to making better software.

George Miranda: You did read the guide. That’s awesome.

Scott McAllister: Yeah.

George Miranda: Yeah. And I think what I want to put out there about that term at least is I think the term service ownership means a very particular thing, especially with older, more established IT shops. And we are talking about the ways that people own the services that they run in production. But I think the differentiation here is when we call it full service ownership, right? It’s like taking that full-service approach, right? Soup to nuts. We’re going to do the whole thing. And in a way it’s talking about DevOps, right? How do we take software development and make sure that we own every component of it, including running it in production? That’s something a little bit different about this approach.

Scott McAllister: I see. So when we typically do these shows with guests we have that are outside our team. We asked them this question. But since it’s just two of us and we’re both hosts, we’re going to talk to each other today about the common myths or misconceptions about something. And today we’re obviously we’re talking about full service ownership. So George, what are some of the common myths or misconceptions about full-service ownership that you want to debunk?

George Miranda: Sure. I think one of these in our introductory episode, I started to debunk, which is the idea that you don’t need to run software in production to be a really good software developer. So I’ve harped on that one before. So I’m going to choose something a little bit different today, which is the myth that this type of approach won’t work in my org, right? When I was saying that there are older shops, for example, that have a very defined set of things they mean when they say service ownership. I think these are the types of orgs that I’m talking about, right? We’ve been developing software this way for 20, 30 years, right? And we have tried and true processes and this new fangled DevOps approach, right? That’s never going to work here, right? So just take this dev and shove it up your ops, and I don’t want to hear about it. And I think that-

Scott McAllister: I’ve never heard that before.

George Miranda: You like that? Well, I mean, it’s kind of true. I hear that a lot, right? Which is, these new-fangled ways are great, but they’re just not for us. And I think we see a lot of that with DevOps transformation in general, and I’m not going to open this bag here. But what I am going to say is that this guide specifically is set up for organizations that sort of think about restructuring big monoliths, right? We’ve got a million lines of code that have been developed over the last 20 years. How do we even start to introduce these new-fangled operational models to them? This is one of the things that happened at PagerDuty and I’ve seen this happen in a number of other orgs. But at PagerDuty, we had a very big monolithic web application that needed to be restructured, that needed to have different teams responsible for it, right? And this is a model of that approach that we took. And so for those orgs that think this is impossible. It’ll never work here for things like that. I always use marathon analogies because I don’t want to say accomplished. I’ve run a number of marathons. I’ve run like double-digit numbers of marathons. I was never a runner ever in my life. And when somebody first told me, “You’re going to run 26 miles,” I thought this was impossible, right? But the way that I did it, or the way that I was able to do it was I followed somebody advice and just learned how to run a mile. And then it was running two miles and then three and then five and then 10. And this thing that I thought was impossible, it just really needed to be broken down into small bite sized chunks. And I just needed to follow a way of thinking about that and to have a kind of a plan for doing that. And I think that’s what this guide is all about, right? How do we take that big audacious thing and break it down into manageable ways that you can think about that, right? So if you think this isn’t going to work in your org, I definitely challenge you to check out this guide and tell us what you think.

Scott McAllister: George, I didn’t know you’re a runner.

George Miranda: I know. It’s one of those things… I’m less of a runner these days. But I can still hang in like the five to 10 mile arena. By the way, if you’ve never done a marathon, I don’t recommend doing a marathon. Like half marathon, that’s the right length. But after that, your body just rebels and hits you. So, come talk to me later. I’ll give you the rundown.

Scott McAllister: I think you just helped me scratch something off of my wishlist of things I wanted to do, my bucket list.

George Miranda: Go cheer on marathon runners. That’ll work. That’s a much better way to go about it.

Scott McAllister: There you go. The method I want to debunk, because it was along the lines of that I heard a lot of software engineers say, at previous jobs that I worked at when I was an engineer, and different engineers in our org said, “We work here because we’re not on call. We don’t want to be on call.” And so I think there’s a misconception that just having full-service ownership means that it’s the engineers that are the ones who are on the hook for a service and that it runs well. But it’s not just engineers that are involved with building a service or building a product or a feature. You have product people who are designing it. You have people writing the documentation, you have support people. So you still have a whole team of folks that are going to be owners of a service. I think that’s something that people should keep in mind, is it’s not just engineers that are on the hook. They’re definitely a part of it. But we’re all part of a team that’s getting something out the door.

George Miranda: When it comes to being on call Scott, I got to say one of the things that I have tended to harp on is sort of an industry safety view of supporting the things that you put out there, right? And so there’s this great XKCD comic about voting machines, right? And the panels in the comic kind of started with, they ask an elevator engineer about elevator safety. And it’s like, yeah, well, all these things could go wrong, but we’ve built in these enormous resilience safety systems. So generally, you’re going to be okay. But you ask software engineers about their code, and oh God, no, right? We would never take a mission critical application and rely on buggy software to do that. And I think putting folks on call starts to get to the root of that, right? We’ve been okay with writing buggy code that might work well in development, but never scales it production because we haven’t had to support it, right? And so maybe in the beginning, especially if you’ve never done that, there’s a little bit of a learning curve and a little bit of pain you have to go through. But ultimately, putting folks on call is what gets you to building more resilient systems, right? It gets you closer to the problem. It gets you closer to understanding that. And so, yeah, you know what? I’m going to go back to the marathon analogy. Those first few miles, right? Learning to run three miles was the hardest part. But then after that, between mile three and mile 15, your body just cruises, you’ve done the hard part. And then you kind of get to the top end of the scale where other problems started happening, right? But we can talk about those later.

Scott McAllister: Yeah. So you mentioned before, as you were teeing up the conversation that this particular ops guide was taking a different approach to the topic. So, you participated a lot in the writing. You wrote most of the guide. What are some things that you think people should know about the approach that you took?

George Miranda: Oh, well, one, I’ll say I did a lot of final editing of this guide. I think we had a number of contributors, a lot of folks internal and external to PagerDuty. So there’s a lot of credit to go around. But what I will say is I spent a lot of time with sort of the narrative and trying to understand how do we go from this end state of things that we have, which is a little bit different, right? We know that for example, PagerDuty went through this and we know what that end state looked like. But we kind of had to go back in time and figure out, okay, well, how did we start? And what were the cohesive ways to do that? And like, how do you present that? How do you take being able to run this marathon and break it down into steps? And I think that is what I got pretty familiar with. And so what I would say is that when it comes to using this guide, one of the things that you should be aware of with any of the ops guides, is that when I say they’re a framework, the idea is it’s not always like a flow chart of, do you think A and then B and then C? I think this ops guide in particular is a little bit of a primer around the things that you should be considering. And some of those, some of the things in the guide are very much like a step one, think about the shape of your service, right? And step two, think about which operational teams are going to become responsible for that. And some things are kind of mechanical thing that you can follow on. And some things in this ops guide are a little bit more of a, well, this is kind of a general set of considerations that are a great starter way to think about it. And so if you start answering some of those questions, it’ll uncover more. And so it’s not necessarily just follow these steps. It’s think about your organization, think about your technical stack, in these particular ways, in this order. And that’s going to start really helping you put together a plan that’s right for your org, right? Versus this kind of generic thing that we’ve written. So, Scott, I got to say one of the main reasons that I wanted to do this podcast with you is because you are a very solid developer. And one of the things that we’ve talked about on a number of occasions, not in this podcast. Is that you’ve had a lot of interest in learning a little bit more about DevOps and running software in production and sort of I think what really, a lot of the practices in this guide. So obviously, you’ve read the guide, we’ve put that to the test earlier. So what’s your take man? What did you get out of it? What’d you think about this thing?

Scott McAllister: What I enjoyed about it and was a pleasant surprise for me is, because I, when we all think of new topics, they’re kind of a black box, right? And we’re kind of nervous about, okay, I hear this thing. This concept of like DevOps topics and DevOps thinking and things. And I’m like, “Well, I’ve never really done it. Because I’ve only been the engineer and I’ve kind of in my past life, never was really part of a DevOps process.” And so as I read this guide, I realized, wait, this is putting into place into words a lot of things that make sense. And then allows it to, like you said, be a framework. I like that term as a framework because there’s not a lot of frameworks in software that help you, guide you in a direction, but yet don’t prescribe exactly how to do something. But they tell you how, if we’re going to do these common practices, we’re probably going to do them in these ways and we’re going to define them this way. And so with this guide, it helped define roles of people who deal with the different services, who are on the teams that deal with the services, what they need to do throughout the life cycle, things to keep in mind, how to name things. And that’s all those guides, those things that people don’t really talk about until you get onto a team. And they’re like, “Well, this is the kind of the way we do things.” And I think it’s nice to provide a guide that way you have a framework that possibly, if you go from one shop to another shop, it has a similar way of thinking, a similar way of doing things. Like, “Oh yeah, this is how we do full-service ownership because we follow the same guidelines, the same concepts.” It was good for me to see.

George Miranda: That’s a really interesting observation because one of the things that I think changed as this guide got closer to being published was how roles were defined. I think it’s one of those Moore’s law things, right? Like you will create mechanisms that kind of mimic the shape of your own organization. And similarly, I think this ops guide had a lot of functional roles defined, right? Like what sort of functional roles do we need in service ownership? And we started to group them into things that mirror what the SRE team does at PagerDuty, right? And what development teams do at PagerDuty and what project management teams do. And it’s not the same at every organization, right? Every organization has slightly different bounds around who’s responsible for what, when they have those titles. And so if you use titles like that, and especially in a document like this, they have so many presumptions baked into them that sometimes you don’t entirely step back to re-examine well, where does this function really live? And that’s actually one of the things that I really liked that might be a little counterintuitive when you first read this ops guide, the section on service ownership functions does things like breakdown. Well, who’s responsible for documenting what your service does, right? And who’s responsible for running that in production. And we don’t put job titles next to that. But we tell you, this is what the function should be, right? These are the kinds of things that you need to put into documentation that’s useful by your customers, right? Consuming an API, right? Here are the kinds of things that you should be thinking about when you manage this software in production, right? And so maybe that’s an SRE team in your org or a sustainability team or whatever you want to call it, right? Maybe that’s software developers that are doing that as well. But you know what? We’re just going to focus on the types of functional support that you need, including things like thinking about your needs, right? Things that are typically owned by product. Thinking about ways that you interface with the rest of the business, right? Things that are typically not being done by leadership and managers, right? And we’re just going to describe all of those things and then you can figure out how you start assembling those cross-functional teams inside of your own org.

Scott McAllister: Nice. Yeah. Like I said, it’s nice to be able to relate it to things that I already know. And so to have those guidelines really helps out. So talk about ops guides in general, how do they get written? You say they’re open source. So, can people contribute? How would they do that? Kind of talk about the process that goes on there.

George Miranda: Yeah. Here’s what I’ll say about our ops guides. I think if you look at some of the more mature ops guides like incident response, we’ve gotten a lot of contributions from other folks that have taken this ops guide and forked it or made it their own. We talked to a lot of our customers. We want these communal best practices to be a reflection of what is actually being practiced in the industry. And so, here’s what I would say. The PagerDuty GitHub repo has docs, not only for this ops guide, but all of our ops guides. They’re formatted currently through MkDocs. And so it’s mostly marked down. There are some strange conventions going on. And so not everybody is very comfortable just figuring out where to plug in content or what they might want to adapt or change. So there are a couple of ways that you can go about it. One, I would highly suggest if you read one of these guides and think that it might be useful for you, well, you can do is, again, you can fork it, you can make it your own, use those docs in your own organization to start talking about how you do these things. Or if you have contributions to give back, open up a GitHub issue, right? Talk to us about what you think should be different or the things that you are seeing in your org. And we can figure out how that actually plugs into a larger framework. It’s really weird, man, because we’re a bunch of engineers writing these things. And so I think we have a very engineering mind frame around this. But at the end of the day, this is a narrative document, right? Telling you how to do a particular thing. So, just creating a pull request, doesn’t always merge in super cleanly. But what I would say is talk to us. And in fact, I particularly, I know like the entire team really wants to hear what you have to say. And so I think GitHub issues are the best way to do that. Probably the easiest way to do that. If you find anything, let us know what you think, and we can figure out how to plug in your thoughts.

Scott McAllister: So, you’re saying the poll request that I submitted last night was not the way I should have done it.

George Miranda: Talk to me, Scott. You know where to find me. Yeah. And it’s totally fine, too. Like if you want to write a PR and you’re like, “You know what? This section is full of crap. And here’s the view of the world that I think is right.” Submit it, right? And we can talk about it and we can figure out how to plug it in. It’s all grammar and wordsmithing sometimes. But we can treat it like code, right? We’ll massage it into the right thing that fits.

Scott McAllister: Nice. Yeah. That’s I found. It wasn’t anything major. I found an erroneous A was sitting there that didn’t make sense. So I thought I could make my own contribution.

George Miranda: I love it.

Scott McAllister: So are there any common pitfalls or assumptions that are baked into the guide?

George Miranda: Yeah. You know what? I think this one in particular when I was saying that it was pretty stoked about the way that it turned out. I once upon a time spent quite a bit of my career in large enterprise settings. And I think this guide in particular is sort of aimed at folks. When we were talking earlier about the people that would say, “I can’t do this in my organization, that’s never going to fly here.” I think in a way we’ve tried to specifically target those teams, right? Which is you have this legacy way of operating that you are interested in adapting to a new model. And I think that’s one of the assumptions that’s really baked in, right? That you have a big monolithic code base, and not necessarily just the code base, but like a way of operating, right? That’s monolithic in nature. And you want to start breaking that down into smaller components, right? And if so, this guide is really useful. But as a greenfield team approaching a new project, I think this is very useful as well. There’s a lot of stuff, especially around service lifecycle, around those service ownership functions that you can use to think about how you should structure any new project. But I think a lot of the stuff in the guide, tips for success. A lot of the additional stuff that’s baked in, really I think is looking to speak to those large scale teams that are starting with a lot of traditional ways of operating and are looking to change those up.

Scott McAllister: Nice. Well, thanks, George. Where can people find this a full-service ownership ops guide?

George Miranda: Well, you can find this particular ops guide at ownership.pagerduty.com, right? That’s the full-service ownership guide. And if you want to find all of the ops guides that we write, you can find those at pagerduty.com/ops-guides. And there you’ll see a landing page that has the various bits of training that we offer that are open source. It’s more than just the operational guides at present, but you can see all of the types of things of this nature that we create, that kind of focus on that process oriented view of the world.

Scott McAllister: I have actually been reading the other guides as well to see if I can get as smart as the rest of you all on these topics. So they’ve been really beneficial for me. And I highly recommend listeners checking them out.

George Miranda: That’s really good to hear, man, because I think again, right? The thing to keep in mind here is that unlike pretty much any of the material that I’ve gotten a chance to write in my career, one of the things that I really appreciate about what these ops guides are doing is, again, they’re they’re vendor agnostic. They’re not about a particular tool. They’re about how your teams operate. And I don’t think we see enough folks creating that, right? So if you find this content useful, reach out to us, let us know that you like it. Let us know some of the other things you think might be useful to tackle, because I think you’ll be seeing hopefully a lot more of these in the coming months.

Scott McAllister: For sure. So that brings us toward the end of our show. We normally ask us some regular questions, some repeating questions to our guests, but you and I, George we’ve already answered those questions. So we have some different ones today. Instead, for this one, we’re going to do what’s called page it picks. Where we pick a few hot things that you should check out. These picks are about everything, from tech or not tech, books you’re reading, films you watch. So George, tell us about things that you’re just super excited about right now.

George Miranda: You know what? I’m going to tell our listeners a little bit of how things work around here. So Scott, you and I tried a sample episode when we were thinking about this podcast. We were trying different formats to see what would work and you and I did a sample episode like this to see how it would go. And I had some picks for that episode that I particularly liked. And I actually looked back to see what those were and I now have those same picks, but for different reasons. So I’m going to start with my first pick, which is The Good Place. So The Good Place on NBC just wrapped up. And this show, if you haven’t seen it, it’s great. It’s wonderful. It’s one of the most witty heartfelt shows. I think that have aired on recent television. And one of the things that I hear folks have a little bit of trepidation around with a name like The Good Place. The show is basically about the concept of heaven and hell, right? And I think some folks that I’ve talked to have been a little put off because it seems like it might be very religious and overtone. And you know what? It’s actually this show that’s just sort of grappling with what it means to be a decent person in today’s complex world. But it does it in just these really philosophical, funny, resounding ways. There’s a lot of sharp one-liners, they’ve gotten a lot of mileage out of recurring gags. And I got to tell you it’s pretty forking funny. So, shirtballs. Right? If you’ve never gotten a chance to watch it, season four, just wrapped, it was incredibly touching. I’m not going to say a bunch about it, but it’s a great time to binge this show if you haven’t started it. So, I’m going to highly recommend The Good Place. And then the second thing I’m going to recommend is a drink called the verbena. I’m a little bit of a cocktail connoisseur. And you can find this drink if you are in Las Vegas, by going to the Chandelier bar at the Cosmopolitan. It’s not on the menu. But if you ask someone for a verbena, what they’ll do is bring you out this cocktail. That’s a little bit like a margarita, but it’s served with a Sichuan flour. I never knew about a Sichuan flour, but they basically call this a buzz button. You take a little bit of a bite of this thing. And what it does is it makes your tongue tingle and it changes your flavor perceptions as you drink this cocktail. So it starts out tasting a little bit like a margarita, and then it’s super sweet and a little bit creamy and it’ll adjust the flavor profiles of this drink. Yeah, you should try it first before you take Sichuan flour, but it will blow your mind in terms of how your taste buds and your perceptions of taste can change. So, I’m going to highly recommend that if you find yourself in Vegas, like I’m about to again, go check out this drink. You will not be sorry. So Scott, what are your picks?

Scott McAllister: Let’s see, I didn’t actually go back to my old recording and talk about the thing or think about the things that I talked about on that one. I’m so happy you brought that other recording up. But anyway, so recently, I just finished an audio book I was really, I really enjoyed. It’s called The Nickel Boys by Colson Whitehead. A fantastic read. It’s a good reminder of how you can persevere through hard things, but also how terrible humans can be. And then there’s a interesting twist at the end. So I highly recommend that book. And then recently I’ve gotten into a new hobby. Where I just discovered that the small town I live in, in Washington actually has two disc golf courses. I mean, we don’t have to have anything, let alone two disc golf courses. It’s fascinating. I love it. I found it very, very therapeutic yesterday. It was actually sunny here for like the first time in months. And I decided I have 45 minutes. I’m going to get out to the disc golf course, excuse me. And do a quick nine. And it was just so nice. So anyway, disc golf, super cool, really low barrier to entry. If a course is near you, the discs are not quite as expensive as real golf clubs. It’s nice way to get out. And then finally this other one’s kind of off the wall, interesting. Last night as I was actually reading my full-service ownership guide, I opened up my record cabinet to see if I could find something I could listen to, to kind of put some background on. And I found this old record that I haven’t listened to in years. Years is been like, what? Five, six years ago. By a band called inventions. So if you’re into kind of atmospheric sort of post rock stuff, inventions, check them out. There was a guy who’s called Alluvium and there was a, he mixes with a dude from Explosions in the Sky, who a lot of people might’ve heard of. And they kind of mix their sounds together. And it’s quite nice. I listened to it several times last night while reading this guide. So, I recommend that completely.

George Miranda: Scott, that is quite the pick. I’m really into Explosions in the Sky and I had not heard of this record. So I’m all over it. And also for our listeners, I’m going to say, I’ve talked to you at a little bit of length around disc golf, and it has been absolutely mind blowing to me just how deep into disc golf you can go and how developed a sport this is. And as somebody that is terrible at throwing a frisbee, you convinced me. I also could do it. So, there’s definitely something to this.

Scott McAllister: Awesome. No, it’s super fun. I’m terrible at it. Just like anybody who goes out and hits a golf ball and just is terrible at it. I throw the disc and it’ll just go way off in the wrong direction. And I’m like, “Oh, that was bad.” Yeah. I should also throw in a plug for The Good Place. I’ve watched the first three seasons because those were on Netflix and love that show. Super witty, Kristen Bell is a genius. Just a really, really good show right on. Well, all right. Thanks a lot, George, for talking to us about the ownership guide that is now out on the ops guides section of the PagerDuty site. This is Scott McAllister @stmcallister on Twitter.

George Miranda: And this is George Miranda @gmiranda23 on Twitter.

Scott McAllister: And we’re both wishing you an uneventful day. That does it for another installment of Page It To The Limit. We like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you liked what you’ve heard. You can find our show notes on pageittothelimit.com and you can reach us on Twitter @pageit2thelimit. Using the number two. That’s @pageit2thelimit. Let us know what you think of the show. Thank you so much for joining us. And remember uneventful days are beautiful days.