March Book Club - Kill It With Fire by Marianne Bellotti

Transcript

Mandi Walls: Welcome to Page It to The Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve the system reliability and the lives of the people supporting those systems. I’m your host, Mandy Walls. Find me at LNXCHK on Twitter.

Mandi Walls: Alright, thanks for coming back. We’re back with another book club episode as we collect up all those fun books that folks recommend to you, maybe at conferences, maybe from your coworkers. Back with me again is Hannele. Hello, hello, hello.

Hannele Kormano: Hi there.

Mandi Walls: This month we are talking about Kill It With Fire and we are so excited to talk to you about this book and this is by Marianne Bellotti. It was published in 2021 and this is one from No Star Press, so it’s available in all your favorite formats and it’s also on O’Reilly’s Safari platform so you can find it all over the place. Marianne’s busy lady, she’s talking to SRECon, we’re recording this before SRECon, but she’s speaking there next week. So by the time this goes out maybe you’ll find a recording and you can find more about her at her website bellotti.tech, which I’ll put in the show notes. So you recommended this and I’m so glad you did. There’s so much in here, but kick us off, what was your first impression when you read this? What resonated with you?

Hannele Kormano: Oh, for sure. I’ve reread most of it as preparation and this is something that I’ve struggled with at almost every software job that I’ve ever had, but it’s wonderful to see how universal they are, it’s also frustrating at the same time, but it was almost like therapy. Everybody has to deal with these problems and the reasons that they happen are very human.

Mandi Walls: Absolutely. So the whole premise of this particular book, and she has some others, but Marianne spent it sounds like years doing system modernization at places that are traditionally super conservative. So she doesn’t get into a whole lot of detail. She did work at the UN for a while. It sounds like she worked at various US federal government kind of places. I didn’t get all of that in there exactly where she was. But yeah, so dealing with modernizing legacy stuff that may have been around for decades, some of it.

Hannele Kormano: There was a bit where she was talking about modernizing a system that had only been around for three months even and it still felt like the exact same problem.

Mandi Walls: Absolutely. Because some of these places they kind of go in even to a new system with maybe outdated assumptions or obsolete planning or things that are, it’s one thing to not be at the cutting edge state of the art all the time, but yeah, bringing forward old things just into your new systems, it’s very sad.

Hannele Kormano: Or even just using the wrong tool for the job sometimes I feel like that’s where a lot of things come about. I know we struggled with that at PagerDuty for a bit. We were using Cassandra as our database of choice and that definitely has its places where it makes a lot of sense and it has its places where it really doesn’t. And so that’s where we’ve largely migrated away from it at this point. But for a while it was a struggle.

Mandi Walls: Absolutely. And databases I think are one of those things. We had a similar situation when I was at Chef, the initial database that we selected for the initial parts of the system. We outgrew them so fast and then you have to have an exercise, what’s going to fit best and how far into the future should you look as you’re looking at the next piece. And one of the good quotes in here, and there’s so many, there’s, I struggled to not highlight something on every single page in this book.

Hannele Kormano: Very true

Mandi Walls: One that was early on that really stuck out for me. She said, old computer programs are artifacts of human thought and I’m just like, yes, hear hear, mic drop right there super early in the book. And I’m like, oh yeah, we’re going places with this. It was super good. But yeah, you’re putting into an artifact all the things that you were thinking, all the assumptions you were making at the time, all the expectations and hopes you had for that system. Maybe when you wrote it in 1978, it’s still there.

Hannele Kormano: Yeah, I mean one of the big things that she highlights, I think she does a really good job of structuring the chapters so that you can kind of jump into any one without having to have reread the previous ones. And sometimes that makes things a little bit repetitive, but again, like therapy, it’s like these are lessons you kind of have to learn over and over again. And one of them is that just spending time recovering context on these kinds of legacy systems is so, so important because one of the reasons they’re hard to change is because that context has been lost. Nobody was there when they were built or nobody remembers why we made those decisions. That was one of the key things that stood out to me is it’s one of those things is just spending time recovering that context.

Mandi Walls: Absolutely.

Hannele Kormano: Pretend you were designing it today, what choices we would make, pretend you were designing it, then what limitations would you have been forced to make? And she had so many good exercises of different ways of figuring these kinds of things out.

Mandi Walls: Absolutely. One of the things I really liked about how this is kind of set up is that each chapter has a bit of prose about the thing that she’s tackling, that she’s talking about, and then there’s a bunch of examples and exercises that she kind of goes into. And before we started recording, I was like, if this were a business book, there would be a companion workbook with it. This is material that would work really well with a workbook and maybe she does this in her consulting practice, but there’s a lot of really good stuff here that you could take directly into a project like this directly out of this book and right into real life. There’s a lot of really good stuff in here.

Hannele Kormano: Definitely, definitely. Yeah. One of the other ones that I really liked was that an old system is not a specification. Make it the same as the old one is not a valid plan. And I’ve definitely encountered that numerous times over the course of my career.

Mandi Walls: Absolutely. And that one too made me think about Park Pathways and Desire paths. You put the sidewalks in a certain place, but people find other places and when you’re with legacy systems, you really have to sit down and talk to people because they’ve probably come up with weird little hacks to get around the old assumptions of the system and deal with the problems and all that stuff. And you can’t just do that by looking at the spec of the system. You have to look at what people are actually doing with it.

Hannele Kormano: Nevermind just looking at the code and trying to sort out a specification from that. Especially when we’re talking about stuff from the 1970s, the documentation does not exist. The only thing you have is the code and how people are using it, and so you have to engage with both to figure out how you’d want to replace

Mandi Walls: It. Definitely.

Hannele Kormano: I also appreciated that one of the things that she recommends as a last resort is rewriting the whole thing. It’s always tempting. It’s always tempting to just throw out the whole thing and start over, but I think a lot of folks have talked about why that’s not a good idea most of the time.

Mandi Walls: But it’s still a lot of folks sort of first assumption that you’re going to go in and rewrite the thing and yeah, we know that’s not going to be what you want, but it’s the first impulse is to just throw it out and start over.

Hannele Kormano: Yeah, yeah. No, I liked at the beginning of one of the chapters she was talking about how everybody knows how to do agile software development when it comes to modernizing a legacy system. Suddenly everybody wants to do waterfall again.

Mandi Walls: Yes.

Hannele Kormano: There are so many just interesting observations as you go through the book.

Mandi Walls: Yeah, she must have seen some real crazy business.

Hannele Kormano: Oh, I’ll bet.

Mandi Walls: Oh my gosh. Just to have collected up all this and things, one of the things that she sets out at the very beginning that I found kind of interesting, I don’t think I had sort of deeply considered legacy versus technical debt. I think if you’ve got it deployed in production, it’s probably technical debt already, but in my mind, legacy was more like stuff that just runs and you don’t develop against it anymore. And she put together a pretty specific set of definitions, which is kind of interesting. And she said Legacy is an old system. Its design patterns are consistent, but they’re out of date. You can upgrade the capacity and that gives you some performance increases, but it’s hard to onboard because the technology is so old from the system was built and then technical debt, it happens at any age. Yeah, absolutely. It’s a product of subpar tradeoffs and I don’t know that I make that assumption about technical debt. It goes into partial migrations, quick patches or out of date, unnecessary dependencies.

Hannele Kormano: I mean feel like everybody disagrees about what the definition of technical debt is.

Mandi Walls: It’s hard to put your head around. Absolutely.

Hannele Kormano: It’s not just technical debt. I think there’s a lot of what you might call product debt involved also, where it’s not so much that the technology is the problem, it’s that there’s overlapping terminology that’s confusing to sort out. There’s features that look a little bit different from different parts of the product or there’s features that kind of have overlapping functionality. This is a bit off topic for this book. That is something I think about a lot. One might say partly because I work at PagerDuty but I don’t want to too much shade at the company at work at

Mandi Walls: And I mean there too. Yeah, I mean PagerDuty’s 13 years old, so there’s some stuff going on and earlier in my career I worked at AOL and was, I mean that was a very definition of legacy. People were still dialing up to the internet at that time and it’s still producing a whole lot of money, but you don’t want to touch any of this stuff that’s underneath there. The whole thing might fall apart.

Hannele Kormano: A hundred percent.

Mandi Walls: So yeah, it’s kind of all over the place. So yeah, I definitely thought this. There was a whole lot of really, really good stuff in here and like you say, her approaches to modernization of yes, reminding everybody, Hey, we can totally do some agile on this and put things in place in iterative processes. And one thing that kind of stuck out, she doesn’t really come out to say it explicitly, but as she’s working through all the different parts of the chapters, this stuff takes a lot of time and I feel like one of the things that drives people to rewrite is that they feel like that’s going to be quicker than doing a real scientific deep dive on this old system and really getting in there and being smart about replacing it in pieces.

Hannele Kormano: I was reviewing the chapter a little bit on different tools that you can use like transpilers, these do have their place, but it’s important that you understand the trade-offs of those kinds of tools and that one of the big things that she brings up a few times that there are no silver bullets, there’s no one approach that’s going to work for every modernization effort. That’s why there are so many different strategies that can come in handy. Another one that I liked was just finding one measurable problem to solve and focusing on that because that can help cut out a lot of discussion about which technology is more appropriate. Well, you pick the technology that moves the needle the forward the most, that sort of deal. So picking one measurable problem to solve was a good thing to think about.

Mandi Walls: Yeah, absolutely. And she goes into detail there where she’s talking about incremental improvement, which I thought was really insightful. She’s breaking it down, okay, if everyone, she goes through an exercise example where you have the team and can everybody on the team come up with a three to 5% improvement and if you’ve got five, six people on the team, you could potentially get 25% improvement of whatever your measure is on the system if everybody has a small suggestion that is an incremental improvement. And I thought that was super interesting. I don’t know that I’ve ever been in a practice where we’ve sat down to really think about it that way to get ideas from everybody and put together the sum of all the parts of all the things we could be doing because with a lot of these projects, people start to look at them and she goes into this a little bit and they’re overwhelming and how do you take the first bite off the elephant? How do you take the first sip of the ocean when you’re modernizing these systems? That seemed like a really interesting way to get started for teams that are looking at some of these things.

Hannele Kormano: Oh yeah. And she does dive into also just the morale issue. It’s hard to keep people invested in these projects both in terms of the engineers working on it, it feels like thankless work and in terms of the executives who would obviously very much rather have new features than spend time re-implementing the things we already have.

Mandi Walls: Absolutely.

Hannele Kormano: That’s where the measurable problems come into play I think is when you have something that you can visibly improve and if you can communicate effectively what will improve that is one of the ways that you can keep that sort of momentum going.

Mandi Walls: Yeah, definitely. I really appreciated the, there’s a couple places in a couple of different chapters when she talks a bit about the organizational culture ramifications of some of these projects and like you said, some of our larger companies in the industry kind notorious for this, everybody wants to work on the shiny new thing. Nobody wants to do the maintenance on the things are just kind of hanging out and working and she does go into that a bit and there’s a bit about what kind of rewards people like and having a place in your meetings for talking about incremental improvements and not waiting until you have a big release to get praise for the work that you’re doing and some of those things that seems it should feel intuitive, but you’ve probably not really sat down and fought heavily about it before unless you’ve been through one of these sort of programs.

Hannele Kormano: And I know I like some of the bigger companies, everybody knows who I’m talking about. The companies that regularly and unceremoniously get rid of old products all the time, even if people love them and things like this, I kind of understand. They just operate at a scale where they can just do that to some extent, they don’t have to care probably only to a certain extent. At a certain point, everybody stops trusting you. I can make a lot of people look a little bit sideways at things like Google Cloud platform for that reason nowadays because it’s just so hard to build on it when something that you depend on might just up and vanish tomorrow.

Mandi Walls: No, I totally get it. Right. A pour one out for Google reader every day, but that has organizational impacts. What you incentivize is what you get out of your teams. And you’re not incentivizing being an active participant in maintenance. You don’t get maintenance. It’s just how that goes. She pulled in one other thing from Google that I thought was interesting and I had never heard of before, and that was the code yellow chapter six and when she’s talking about coming in midstream and how to do the archeology of some of these projects if you are not there from when they get started, and the code yellow was pretty interesting and I’m kind of surprised that Google has never kind of put this in any of their many, many books on their best practices and the things that are there, but she’s basically talking about being on the lookout for bad practices and having a clearing house or a place to go where here’s what the bad code is that we’re attacking right now and here’s what to do if you find it.

And I think especially for large code bases, this makes so much sense. You kind of think about a corkboard or a slack channel that has the list of things you want to avoid or a place where everybody is looking at that and getting the idea of, oh yeah, if I do find this, I’m already working in this piece of code. I should refactor out of this kind of thing. It was super interesting kind of methodology that I’m like, I’m surprised they have not published this before. It’s super interesting. She does talk about the code yellow team and a couple of other places where pulling people into, she calls them working groups versus committees I think was one of the other places, and that kind of comes back to the you get what you incentivize because you’re putting folks into a place where they are responsible, incentivized, I say responsible for the actual change that needs to get done.

Hannele Kormano: One of the key things is just the people are most affected by the changes should be the ones, that should have some ability to make decisions about the direction that it takes and that’s definitely one of the key things for me. Also, it’s tough because a lot of the time she brings up things like cognitive load and you can never understand the system all the way down to the bottom. There’s a reason people have to specialize nowadays and to some extent you have to depend on the expertise of other teams and things like this. Even if you do have a perfect DevOps culture at your work, you should probably still have a team who’s running your kubernetes cluster in general and is managing the overall performance of that.

Mandi Walls: Yeah, absolutely. I like how she mentions one sort of cautionary tale where they had implemented one of her practices but had used instead of the working group being made up of individual contributors and technical staff, it was made up of executives and you’re like, oh, you’ve got a bunch of your director and aboves hanging out in the conference room as your working group and you’re surprised that the stuff doesn’t get done. Yeah. That’s not what you want to be doing there. You want the people who are moving things forward.

Hannele Kormano: Nobody is very invested in the approach that you’ve handed down to them from on high. Right?

Mandi Walls: Yes, I’vw bequeathed to you these commandments for your new system

Hannele Kormano: I bequeath these tools that you must follow and that will solve all your problems so you can stop telling me about them now. Thank you.

Mandi Walls: No more budget for any more tools.

Hannele Kormano: Exactly. You don’t have to do this a legacy modernization project anymore because we give you this tool.

Mandi Walls: Exactly. It’ll make all the problems go away. The vendor totally told us that.

Hannele Kormano: I’m throwing a little bit of shade. I don’t think I’ve ever been in a place where it’s been quite that bad, to be honest. That’s never quite come across my way, but yeah, I’m sure other folks have.

Mandi Walls: Absolutely. And yeah, a million years ago we were moving things, one of my jobs off of Solaris and IRIX onto Linux, so I’ll date that right there. And like I mentioned transpilers and that stuff earlier and those dudes were selling some snake oil. They’re like, man, we’ve got stuff we don’t even have the source code for. If you think you can get that off of IRIX and on Linux, I’d love to see it, but yeah, it didn’t happen that way.

Hannele Kormano: Yeah, no, I’ve definitely had my share of, have completed legacy modernization projects also that helped me to this day if you like assignments from school that you never got to finish. You’d almost go back and almost, I’m not saying I would, but you almost want to go back and finish it for free just because it weighs on your soul.

Mandi Walls: Yeah. It’s one of those things, you get a recurring dream where I’m always thinking I’m missing a history essay or something like that, and you’re like, you know, have unfinished business somewhere and it just haunts you forever. Yeah, absolutely.

Hannele Kormano: It’s tricky. It’s tough though. I can’t remember if it was brought up in the book, but one of the things that’s usually derailed it for me is just a reorg happens and suddenly you don’t have a team that is dealing with this particular thing anymore they have to work on all those have sort of blown apart and partly in terms of an explosion and partly just of the wind flowing things apart and there’s nobody who can focus on it anymore. So that’s something I still struggle with is how sure are you that you can finish what you’ve started. It’s something that I still struggle with sometimes.

Mandi Walls: No, and I totally get that. I, and she does kind of mention it and I forget which chapter, but a little bit about reorgs in particular, and I get salty about them too. People will be like, well, we’re going to reorg to do X, Y, and Z. I’m not only going to be helpful, you’re completely tear everything apart and try and start over. That seems crazy.

Hannele Kormano: Sometimes it feels like a reorg gives the illusion that somehow there will be less work to do, but I think you will still have the same amount of work to do as just some of it might fall on the floor. Which maybe people want, maybe executives don’t want people to be focusing on those problems and they want them to fall on the floor. I feel like they do have a way of coming up again, and I’ve definitely experienced a couple of occasions where they broke apart this team and six months later or a year later or two years later, all of the things that that team had been solving are now even worse and suddenly we have a team dedicated to that again. That seems like one of these recurring sort of things.

Mandi Walls: Absolutely. It feels like they go in cycles of like, oh, we’re going to break up and do our teams this way and then that works for two years and then we’re going to go back to the way we used to and we’re going to try that again for a while.

Hannele Kormano: Or you pulled together a tiger team to solve this particular issue. I have mixed feelings with tiger teams. I think they can work in certain situations, but I don’t think it works for, again, with the silver bullet sort of approach, it’s very tempting to say, let’s spend six months focusing on this particular area of the products and we’ll make that good. And then it feels like there’s an unset assumption that we will never have to deal with that problem again. But it, it’s really rarely true. If the code exists, it must be maintained at some point and that’s a tricky bit.

Mandi Walls: Yeah, absolutely. And I think part of the things she didn’t really go into too much is putting together the culture of maybe this second to last chapter maybe a little bit, but changing the culture so that the new system doesn’t decay as much as the old systems might have. And if you’re dealing with a system that’s decades old, you get a bit of a pass, things happen, people retire, their grandkids are maintaining the system now or whatever, but when that accelerates, when that kind of decay happens almost out of the gate where you’re like, oh, we’re going to push off that security update or we don’t really need to update that library, deferring that basic maintenance. Yeah, it’s hard to come back from that. And she does go into a little bit about if you don’t fix something and it doesn’t break, you lose kind of the risk assessment of that thing breaking ever happening. And that was pretty interesting too, to think about like, oh yeah, we do have a lot of these companies that don’t want to patch everything that comes down the line because there’s just so much of it and eventually they’re going to get bit.

Hannele Kormano: But then you get a security vulnerability out of nowhere and suddenly you have to do three years of security updates in a week or something. And sometimes it involves refactoring things because the library does not come up through the generations with you, or maybe there’s a language feature that you should use now instead. It’s all these kinds of tricky businesses where it’s trying to keep your room clean or trying to keep your kitchen clean. Maybe if you never scrape off your baking pans at a certain point, they’re not baking pans anymore. They’re just a crust of various burnt on bits of food and it does not work very well as a baking pan anymore.

Mandi Walls: You can tell yourself it’s adding flavor, but you can

Hannele Kormano: Tell yourself it’s adding flavor. Exactly.

Mandi Walls: Really not if you’re using that kitchen. You have to maintain it so that everything works.

Hannele Kormano: It’s hard.

Mandi Walls: Yeah, it is. All of this is going to be challenging. Lots of different organizations that try and go down that path,

Hannele Kormano: And obviously I am informed by previous experience at PagerDuty and other places, but it is a thing that literally everyone struggles with finding that appropriate ratio of maintenance work to new future work, and obviously there’s an existential thing where if you don’t make enough money, you no longer have a company at a certain point, so it’s a tough balance.

Mandi Walls: Yeah, it definitely is. It definitely is. Yeah, and maybe your new features are going to bring in new customers, but if you’ve got things in there that are degrading and impacting your reliability, they’re not going to pay you for that anyhow, so you have to find all those balances that work for your customer base and all that good stuff. Yeah, so this was really good. I am so glad you recommended this. I had not seen this on the radar. It has the best cover on it with a flaming dumpster, which is amazing. Yeah, this was really good read for folks. Like I said, it is super dense, so it’s good to go through probably a couple of times if you’re going to go into one of these projects or if you’re in the middle of one right now, dip into the places where you feel like you’re struggling and get some insight. I do

Hannele Kormano: Want the workbook version that you brought it up

Mandi Walls: Totally, totally want one workbook for this if I was headed this direction. So it would be super helpful. And there’s a lot of interesting stuff in the footnotes too of different things that, a bunch of stuff that was new to me about just different kinds of ways of thinking about projects and organizing work and stuff like that, so there’s a lot of really good stuff in here.

Hannele Kormano: Yeah, yeah, no, I even liked the last line of the book. “This is not the work of technical janitors, but battlefield surgeons”. Sometimes the war metaphors, it feels like a little bit too much. It’s not literally life and limb, but it has been the greatest honor to serve among them. It is like a common arm situation as much as it is like a business sometimes.

Mandi Walls: Yeah, you’ve had some maybe traumatic but definitely stressful experiences with a group of people, so yeah, no, this was really good, so thank you for recommending this one and participating in another book club with us.

Hannele Kormano: I can’t overstate the therapeutic aspect.

Mandi Walls: It is. There’s a lot of therapy in here and a lot of like, oh, I’m glad I’m not alone when I feel this way kind of stuff.

Hannele Kormano: Hundred percent.

Mandi Walls: Absolutely. Well, this has been great. Thank you so much for joining me again for the book club. For folks out there, if you’ve read this and you want to chat about it, you can join us on our forums at community.pager.com. Like I mentioned at the top, you can pick it up, Kill it with Fire by Marianne Bellotti in all your favorite places as well as on O’Reilly’s Learning platform. Thank you so much for joining us this month.

Hannele Kormano: Perfect. Until next time,

Mandi Walls: We’ll wish everybody out there an uneventful day and we’ll be back with you next month.

Mandi Walls: That does it for another installment of Pager to the Limit. We’d like to thank our sponsor, PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pager to the limit.com and you can reach us on Twitter at page it to the limit using the number two. Thank you so much for joining us, and remember, uneventful days are beautiful days.