Planning for Service End of Life With Sean Steacy

Transcript

Mandi Walls: Welcome to Page it to the Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve both system reliability, and the lives of the people supporting those systems. I’m your host Mandi Walls. Find me at LNXCHK on Twitter. Welcome back folks to Page it to the Limit. This week with me I have Sean Steacy, who is senior manager for product operations here at PagerDuty. Sean, welcome to the show.

Sean Steacy: Thanks for having me. I’m happy to be here.

Mandi Walls: So today we’re going to talk about something I think is super cool, which is a new workflow for sunsetting stuff here at PagerDuty. Tell us about what you’ve been working on with this.

Sean Steacy: Absolutely. Yeah, it’s pretty cool actually. A little bit of a backstory here. So I’ve been with PagerDuty now for seven years, actually, this July. So in PagerDuty years, I’m a dinosaur. That’s a long time, the longest I’ve ever been with a single company in my career. Anyways, I started way back in 2015 as a scrum master, under engineering. So very interested in team dynamics and process and operations. I worked out of the Toronto office. And back then the Toronto office was quite literally 100% engineering. No other business functions. We had one IT guy and a jack of all trades HR person, and then everyone else was in engineering and that’s it. And so the point is that I’ve gotten to see and experience this company’s incredible growth and maturation. And a big part of that has been critical aspects of the software development life cycle that you don’t really think that much about when you’re a startup, right? Like, so EOL, end of life, sunset deprecation, end of service. All of those terms, we can get into a little bit more detail, maybe a little bit later in the show. But for me, the overall kind of EOL process really does fall into that category. I mean, when you’re at a startup, it’s like, are you going to be around long enough to actually end of life anything or sunset anything? So it’s not typically something that is front of mind from an operational perspective. But now, end of life is incredibly important. We have 30 plus development teams. It can’t be hitting home runs all the time. Some of our bats are going to whiff. And also we at PagerDuty pride ourselves in being able to innovate quickly, which means we have to experiment. You can’t leave a bunch of half baked experiments hanging around it. It’s bad for your customers and it’s bad for your development teams. So you need to establish some sort of repeatable process around how to deprecate and sunset your features and continuously improve. And that’s what the EOL program and project is all about. Yeah, I mean, that’s what I would say. Just kind of like grand scales. It’s been interesting just to see that you don’t really think about it, and then you get to some sort of inflection point and you’re like, “Oh, wait a second. This is really important.” And that’s really kind of how it started is product managers were saying, “Hey, we’re getting to a point where, we need to sunset some stuff and we are not sure what steps should be taken. And we were kind of like reinventing the wheel every single time.” And it’s at that point, product operations got involved and said, “Okay, well, let’s put some structure behind this and make it repeatable and allow other product managers to stand on the shoulders of those who have come before and leverage that knowledge and experience.” And so we’ve still got a long way to go. It’s really, really important that we’ve started that process.

Mandi Walls: Absolutely. Because you can’t really have a yard sale. We got to have a plan for what we’re going to do with this stuff. So set that baseline for folks. Like EOL, sunsetting, deprecation. Having the shared understanding of what we actually mean when we talk about that across the entire organization, what are our definitions for those terms?

Sean Steacy: Right. So that’s actually been a bit of a learning curve for the entire organization as well until we started to learn a little bit more, we actually just called it deprecation. Because that’s what we thought it was. And as we started going through that process and learning more about it, and this was really kind of like learning about it as you’re doing it, there wasn’t really time to separate that. But deprecation, so really there’s two main parts to this end state of the software development life cycle. Which is end of lifeing the features. And the sunsetting and deprecation are kind of colloquial terms for end of life and end of support, or end of service. I think end of support. I mean, whatever, I mean, it fits the same acronym, so it’s fine. So really the way that it works is end of life is sunsetting. It means that you’re turning the thing off, it’s that’s it, that’s all. And that could be as a result of a failed experiment. That could be the result of going off in a different direction. It could be the result of many other things. But the point is that sun setting means end of life, which means literally turning the thing off. It’s not going to be supported, it’s not going to be developed. You can’t use it anymore. Which is different from end of support, which is a deprecation. Which is we are no longer going to support. We’re going to leave that feature on. We have customers who use that feature, find value in that feature. And so we’re not going to yank the bandaid off and say, sorry, that’s it. And so that’s the difference between the two. And another way of looking at it is deprecation or end of service is a step on the path to end of life. Is the progression that you would follow. And that doesn’t mean that you have to go through that progression every single time, you can go straight to an end of life. And then that the flip side of that is very interesting. And I think we’ll get into this a little bit as well. Which is when you’re talking about things that are in beta at PagerDuty, we all know that we call beta early access or EA. So we have a lot of things that are in EA that should be sunset. That should be end of life. They shouldn’t just be hanging around there in EA. So again, the choice comes down to the product manager and their team. Obviously against the backdrop of PagerDuty’s larger strategic in initiatives for product development. But the choices of when to do that, really do rest with the team. But it’s something that I think is really important that, if you’re a team that like has a bunch of EAs that are kind of hanging around that you haven’t even put into end of support, that creates a significant amount of drag on your team. There’s a cognitive burden to be had there. So it’s just something that you want to be aware of.

Mandi Walls: Absolutely. Like you say, it’s a drag. You’re pulling these scenes along. They might get into a state where there’s a security or a CVE on them or something else that has to be remediated. You’re like, what’s actually the value of continuing to maintain this thing if it’s never going to get to live in the real world or whatever?

Sean Steacy: Exactly right.

Mandi Walls: Yeah. Do you think there’s anything particular about doing this kind of process with PagerDuty being a SaaS versus on prem? So you don’t really have the, we’re not sitting around waiting for people to update anything necessarily on their own side. But at the same time we can be cognizant of who’s actually using the thing.

Sean Steacy: That’s a really great question. Yeah, I do think that there’s things that you need to consider you’re running a SaaS and a continuous delivery situation, which we certainly are. And that is that I think you just, you need to be much more deliberate and careful about your communications, both internally and externally. These things are happen really quickly, we’re growing and scaling. Our customers are hopefully growing and scaling with us. And so I think you need to be, in our situation, extra diligent about really, I mean, it comes down to GTM, your go to market approach to end of lifeing features. So you just need to be more careful. Because it’s almost like there’s a forcing function in an on-prem. Where it’s like, there’s a forcing function by virtue of the fact that it’s on-prem. You probably has ProServ in there. A lot of those kind of boxes are already ticked just by virtue of a lot of that control is in the hands of the customer.

Mandi Walls: You never get to a point where you have to make a conscious decision to renew a lease.

Sean Steacy: Exactly.

Mandi Walls: Or to talk about an operating system change or any of those.

Sean Steacy: That’s exactly it.

Mandi Walls: None of those inflection points for this.

Sean Steacy: Exactly right. So, I mean, that’s what I would say, is I think you need to be more careful. And that’s interesting because it’s one of the main reasons that product operations even became a thing at PagerDuty, is again, we’re scaling very quickly, we’re going very quickly. And when you work in continuous delivery, as we do, it’s very easy to get the horse blinders on and follow agile principles and say get stuff to market and in the hands of your customers as quickly. But you still need to balance that with the awareness piece, internally and externally. So you don’t want to catch either your internal stakeholders and go to market partners off guard, and you certainly don’t want to catch your customers off guard. And the other thing as well is that PagerDuty was really, in the beginning at least, was really aimed at the responder, the DevOps person. People that are pretty cool with, “Hey, continuous delivery, no problem, give me the new stuff all the time, we’ll figure it out. That’s great.” That’s different for the enterprise. And to your point, where they had all kinds of internal documentation and processes that they need to follow, because you’re talking about, teams on massive scales and not some individual or like a handful of individuals. So I mean, the complexity there of that change management is massive. And so you need to be careful under that circumstance as well.

Mandi Walls: Yeah, definitely. And one of the thing you were talking about there, is talking to your other partners within the organization. Talk us through the way the plan came together. Because as I’m looking through the documentation, I noticed we’ve got like product marketing, which is a team that I sit under is involved plus legal and like all these other folks and your customer service representatives and all these other people who are then involved in the messaging and working with the customers and how all this stuff sort of hangs together. Like this a complex sort of plan.

Sean Steacy: There’s a lot of hands in the cookie jar for sure. The way that we did it is, the first thing is understanding the workflow. I hesitate to use that term. But there is some sort of workflow that exists prior to the structure that you bring in. And what I mean by that is product operations, didn’t come in and say, “Here’s the playbook for how you do this thing.” Product operations came in and said, let’s talk to all the people that we think are involved. We’ll cast as wide a net as we can. And we’ll have those conversations with those individuals, what are their needs? How do they contribute to this? What are their inputs and outputs? And then you can start to draw a picture of what the actual kind of more ideal workflow should be. So it’s like, there is some sort of, even if it’s not documented, and fairly chaotic and only in a certain group of people or a certain person’s head. There is some sort of unstructured kind of workflow or process that already exists. I think that the approach needs to be okay, let’s take an inventory of what that is. Let’s literally draw it out. I find it really helpful to make visualizations of that workflow. Who’s involved, what do they produce, what do they expect to receive? How does an end of life, for example, kind of move through that chain? So that’s really, really what it’s about is the first step is really like observation. It’s like, who are all the players? What are their needs and how can we get them working together more effectively? So it’s really just about having those conversations, understanding what exists, making some small suggestions for improvements. And then you’ll see you’ll start to pretty quickly… and so you have to be careful because there’s an inflection point here as well, where there’s a tipping point where it becomes, okay, now we’re starting to get like too many people involved in this. So when you hit that kind of inflection point where you’re like, we need to actually figure out critical path so we can solve this problem of being more effective. Working cross functionally across all of these business units, that’s really how it all comes together.

Mandi Walls: Yeah. Was there anything in particular that sort of kicked off codifying all of this?

Sean Steacy: It was really, as I mentioned a little bit earlier, it was really the friction and the pain point that product managers were feeling. Where it was like, “Hey, we’re being told that we need to do these things, but we are not sure…” of course all the product managers understood what end of life meant. And some of the baseline things that you do around it, but they didn’t know what the expectations were in terms of who they needed to talk to and partner with internally. And that’s really what kind of kicked this all off. Is being like, we don’t have a good idea of what that needs to be. And as I said, who all the players are, what their needs are, what their timelines are. And so it was really the product manager is being like, “Hey, we get it. We need to do these things where absolutely, we’re behind doing these things. We’re just not sure what are all the things that need to be done? Who are all the people that need to be involved.” And that’s what really was the catalyst for doing this.

Mandi Walls: Yeah. This is definitely the first time I have seen something as comprehensive as this plan is. And I find it super exciting. I have been on the opposite end where I’ve been dragging around legacy systems because no one wants to make a decision or have a plan for turning them off, and what that actually means. From this perspective, I think this is absolutely fabulous.

Sean Steacy: Well, thank you that, I appreciate that.

Mandi Walls: One of the things that we like to ask folks on the show are what’s a myth or a misconception that folks have about this process that you can debunk for us? I’m sure there’s a bunch of good ones.

Sean Steacy: There are. There’s a lot. Wow, there’s a few to choose from. I think the one that’s that’s top of mind for me is we were touching on it a little bit earlier, we alluded to it, is features that stay around in early access for data for too long with no clear or intentional path forward. And in this case, no intention to sunset or to deprecate. And the myth is that it doesn’t have a negative impact on the development team. That it’s like, “Oh it’s not a big deal.” And I mean, if the team is still actively supporting the EA feature, then the impact I think is clear. But that’s not the situation that I’m talking about. The situation that I’m talking about is EA features that are, are no longer supported whatsoever. And the myth is, is that these kinds of features don’t have a negative impact on a team. They do. And in a lot of cases, I like to think about engineering software development as a craft. It really is a craft. And it’s something that these individuals, once upon a time, many moons ago, I was a developer as well. It’s been a decade or two since I was, so a lot has changed. But you develop an emotional attachment to your work, to your craft, and to what you produce. And so when something gets stuck in this never ending limbo, I think it’s natural to ask yourself why and what ifs and it’s nuanced. But that’s part of the drag that I was talking about earlier, like there’s other drags, and he brought up some really good stuff that is more around technical debt and some of the other things that can occur. But there’s also this like strange kind of nuanced drag that happens on an emotional cognitive level, because it stays within the consciousness of the individuals that make up that team. So, I mean, it also clearly has a negative impact on our customers. The customers will sit there, I mean, Sean Scott will laugh about this because he makes a comment about it all the time. But pretty much every single product, all hands the specter of, well, what are we doing with, I know the name of the service is Mortimer, I can’t believe that’s how I’m going to recall it. But yes, our post mortem feature.

Mandi Walls: Oh yes. That’s a…

Sean Steacy: And so that gets asked all the time. And that was something that came out of a hack day. We put it out in production. Some of our customers really liked it and were like, “Hey, we really see some value here, but we would like to see some iteration, we’d like to see this evolved,” and PagerDuty for very good reasons is not ready to make that commitment right now, that investment right now. So that’s a good example of customers that want to see something involved. So there’s something your listener may, or may have not heard in the past, but I think it really fits here. It’s a cliche that I really like. The best answer that you can give to your customer is yes. And the second best answer you can give them is no. Not, maybe. Not we’ll see. And that’s really my thinking around, I think it’s a good fit for EOL. Which is it’s much better to intentionally and transparently tell your customer that a feature they find value has got to go the way the dodo bird. I think the worst that you think that you can do is string them along. Even unintentionally. In fact, most cases it’s going to be unintentionally. And the same is absolutely true of your development team. The impact of the customer, I think, is very easy to understand. I think what we miss sometimes in the myth that I’m trying to break here, is that you’re also paying a price with the development team as well. For reasons that I mentioned, that they naturally form attachments to the work that they do. And so if you just leave something completely in limbo, it actually does have an impact on that team’s morale.

Mandi Walls: Yeah. And I’ve seen that happen in other places where I’ve worked, where we had an architectural decision and kind of split in two different directions and never committed to either one of them. And then you never re-coalesce afterwards. You have one thing that’s kind of hanging out over here, it’s still sort of in beta, but the cool users are using it. And then you have the people who are paying you money, who are using the old thing. And then what do you do? And you never get to a point where anybody can make the intellectual decision to cut bait and go one direction or the other. And then you can’t make architectural decisions going forward because you can’t make any assumption

Sean Steacy: You’ll hamstring yourself.

Mandi Walls: Right, you hamstring yourself completely. Because I can’t build on this thing because it’s still in beta. And do I build on this thing because it’s technically legacy, what do I do? Oh.

Sean Steacy: Yeah. Yeah no, you’re 100% on the money. And that’s another kind of anti pattern that I’ve seen at PagerDuty and other companies as well. And you just brought it up, where there’s this, it seems insane when you talk about it, because it is insane. Where a specific team will develop a beta or an EA. And there’s always going to be reorganizations and instructional changes. And then that EA somehow gets handed off to another team. Which is a terrible situation. But compounding that is maybe you think you’ve handed it off to a new team, they need to learn how it’s built, what it does, how to run it in production. And then another team decides that it’s going to build another beta on that beta. And it’s complete insanity. It’s like [inaudible 00:19:55]

Mandi Walls: All the way down.

Sean Steacy: Yes. And that should just be like, as a matter of just a strict policy, in a product development organization that you do not build betas on other people’s betas. Just don’t… anyways. So a little bit of a tangent there, but…

Mandi Walls: A good lesson to learn though, because those side quests.

Sean Steacy: Oh yeah.

Mandi Walls: Yeah. That ends in tears more often than not.

Sean Steacy: Yes it does. It does.

Mandi Walls: So what’s one thing you wish you had known sooner when you went off on this adventure?

Sean Steacy: This isn’t specific to end of life operations. This is more kind of generic question around what’s something you wish you’d known sooner when it comes to running software and production. And there’s this book that I love, I think everybody in our business should read. I don’t know if you’ve ever heard of it Mandi, it’s called The Tau of Programming by Geoffrey James.

Mandi Walls: I think I have a copy of it around here somewhere actually.

Sean Steacy: Nice. It was written in 1987. It’s a tongue in cheek literary exercise where he really kind of tries to speak in this kind of spiritual way of about programming, obviously from a Taoist perspective. And so anyways, the way that I would answer this one, what do I wish I’d known sooner when it comes to running software and production, is book five in The Tau of Programming, which is maintenance. The quote is this, “Thus spake,” S-P-A-K-E, “The master programmer. Though a program be but three lines long, someday it will have to be maintained.” And then he goes on to say in 5.1, and this is where that spiritual slant is really cool. He says, “A well used door needs no oil on its hinges. A swift flowing stream does not grow stagnant. A deer blends perfectly into the forest colors. Software rots if not used. These are great mysteries.” So that’s what I would say in answer to that question. Is it doesn’t matter how small something is that you’re running in production. At the end of the day, it’s this really kind of crazy thing. If it’s not maintained, it doesn’t matter how, there will be prices to pay for that. It’s just a quote that I thought was really cool and lined up really well with that question.

Mandi Walls: It absolutely does. 100%. On my side like I feel like there’s always an XKCD for it. Because there’s one where it’s like all blocks. And like there’s one little thin little leg holding up this entire super structure. And you’re like, this is a bad plan. But it always happens. There’s always something back there that you’re like, here’s the magic and don’t talk about it.

Sean Steacy: Yeah, exactly. I love that you bring up XKCD that’s that’s such a great comic, so good.

Mandi Walls: I figure I failed if I don’t have at least one XKCD in like every talk because there’s always, always something that is pertinent to whatever it is we’re talking about for technology topics. It’s crazy.

Sean Steacy: 100%.

Mandi Walls: Yeah. Awesome. Well, we’re almost out of time. Is there anything else that you found was interesting that you want to share with people or surprising about this whole process?

Sean Steacy: I think I would just kind of two things. One, I would reiterate a various astute kind of observation that you made. That the overall structure of the workflow and then the process behind this is pretty straightforward. It’s not rocket science. But you mentioned a very high degree of complexity and you’re absolutely right. And if that doesn’t come from the process itself, whatever it comes from, at the end of the day, it’s human beings that are trying to go through this process. And we’re talking about coordinating and orchestrating a lot of human beings. And that’s where the complexity comes from. So I would just stress that it’s really important to keep those conversations going, to do this kind of thing with empathy, to always have your eye towards the continuous improvement. Because I can’t make this thing better in a vacuum. I need to partner and collaborate with a lot of individuals to do this. And so I think I would just kind of underline that and also say it’s kind of a call to action that at the end of every quarter, I run a retrospective with all of the teams that have gone through an EOL. And the purpose there is to try and improve. So I would just say, please, if you have gone through an EOL or if you’re interested in improving this process, you have your opinions or feedback, please don’t hesitate to reach out to me. I’m always happy to talk shop. And even more important if you actually have gone through an EOL, please do try to make that retrospective, because you have boots on the ground experience with how it exists today. And you’re going to have the best insight on how to improve it moving forward.

Mandi Walls: Yeah, absolutely. Because you will find dark corners in unexpected places.

Sean Steacy: Absolutely.

Mandi Walls: That’s what it is. I think this is great. I hope folks out there who are struggling with, or considering what to do with things that are no longer fit for their purpose can take a programmatic and thoughtful approach to just getting ready to turn them off.

Sean Steacy: Exactly. Just turn them off.

Mandi Walls: It’s going to be okay.

Sean Steacy: It is going to be okay.

Mandi Walls: [inaudible 00:25:11] Make more space for new stuff to try and go from there.

Sean Steacy: Exactly.

Mandi Walls: So Sean, this has been great. Thank you so much for-

Sean Steacy: Thank you. No, this was a lot of fun.

Mandi Walls: It’s awesome. Yeah. All so we finish up this episode of Page it to the Limit for this week, and we will wish you an uneventful day. That does it for another installment of Page it to the Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes pageittothelimit.com and you can reach us on Twitter @Pageit2thelimit using the number 2. Thank you so much for joining us, and remember, uneventful days, are beautiful days.