Shifting to Full-Service Ownership With Leandro Pinto

Transcript

George Miranda: Welcome to Page It To The Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve system reliability and the lives of the people supporting those systems. I’m your host, George Miranda @GMiranda23 on Twitter. Today, we’re going to dive a little bit deeper into what it takes to entirely own the code that you write from design, to development, to production, and then eventually retiring that software. Rather than discussing this from a theoretical perspective, we’re going to dive a little bit deeper by talking about practical experience and dealing with real challenges that teams experience when shifting to this kind of Full-Service Ownership model. To do that, we’re joined in studio today by Leandro Pinto, VP of Engineering at MessageBird. Leandro has been deep in the midst of working with teams who’ve made the shift to owning their services in production. That model of Full-Service Ownership is one of manifestations of DevOps. It’s something that we started talking about a couple of episodes ago. Today, we’re going to explore what happens once teams actually make that shift. Thanks for joining us today, Leandro. Do you want to tell the audience a little bit about who you are?

Leandro Pinto: Yeah. Thanks, George, for the quick introduction. I always have trouble to be honest to myself but I think the best way is I am a software engineer. That’s where I started my career and that’s also where my passion is. Throughout my career, I have grew into leadership of course to manage engineering teams and help them through different phases of the life cycle of development. [inaudible 00:02:06] a lot of people go, “I started from the production environment and end up become more like engineers and software engineers from one side.” But my career went the other way around, started from an engineer, but also growing my way to production and helping the teams to actually build in scaling systems, bringing this idea of how you bring the business mindset to how you manage software in production.

George Miranda: That’s great. I’m actually really excited about this because I think one of the things that we share is that same background. We were both engineers for a long time, and I think we’ve grown into roles that, one, connect us a little bit more to the overall business value of what we’re doing. Two, I think there’s a lot of coaching and development involved, which when you’re making a transition like this is definitely needed. Thank you for that intro. I thought that was actually really good. I don’t know that you have a problem introing yourself at all. That was great.

Leandro Pinto: Okay.

George Miranda: Tell us a little bit MessageBird. Tell our audience who MessageBird is and a little bit about what the company does.

Leandro Pinto: Yeah. Okay. MessageBird is in the essence a communication platform. What we have been trying to do is just reduce the friction in between communication from between customers and their business. In our API we provide different channels. When you think about [inaudible 00:03:23] as like SMS and voice, for example. But if we look into our propositions mostly to indeed reduce this friction when you were actually trying to solve a problem or talking to a business, that’s where we enter. That’s also where we connect to why availability and performance, for example, is a key element for what we try to deliver to our customer. Because always when you have friction is where all those things become also very, very important for you.

George Miranda: Great. That leads to the topic of today’s show, which is setting up a model of Full-Service Ownership for your teams and I think you’ve touched on a lot of reasons that’s important. Why don’t we start first with hearing how you describe that model of Full-Service Ownership that you’ve been trying to implement at MessageBird. What is that all about?

Leandro Pinto: Okay. The way I see it and it’s very close to be honest to what you and the PagerDuty team has put into the Full-Service Ownership Ops Guide. It’s basically given the full ownership of a system to a single team. Which not only include feature related ownership, but also how you bring your features to production and how you run your systems. Another way I also like to put it is giving the ownership of both functional and non-functional aspects open application to a single team. If you think about production features from delivering feature to the customer, but also nonfunctional like, again, availability, performance, and also security. All of these goes into the ownership of a single team.

George Miranda: Yeah, 100 percent. I think there are a number of, as you put it, non-functional responsibilities that absolutely belong with folks that are thinking about that everything that it takes to run software and production. With that, what’s one common myth or misconception about that model that you’d like to debunk for [inaudible 00:05:34?

Leandro Pinto: I have to be honest, I have already discussed it a lot, but I just want to give a little bit of a different twist to it, which have been on-call. I know in previous episodes we talked a lot about being on-call and developers being on-call. I want to take from two sides. One, you hear a lot of people saying, “Oh, but developers don’t want to be on-call.” And that’s the first side that I wanted to debunk. You would be surprised of how much people actually will step up and take the ownership when you give them the responsibility of such call like Full-Service Ownership. People will take on that responsibility and they will actually go the extra mile to actually deliver this ownership, which include being a bit on-call for the service that they build. These of course, inclusive developers and engineers in the team. The other side that I honestly also want to bring to the discussion is that, to be honest, nobody likes to be on-call. I am on-call, I don’t like to be on-call but I do it because it’s part of our … It’s an expectation that we need to create. Also for engineers its like there is a reason why when you look back to more traditional operations those people, it’s not that they like it to be on-call but we created the expectation that they would need to be on-call. That’s also where I see when we start introducing things like Full-Service Ownership, we also started to drive the industry in a way that this starts to become something that is also expected from engineers, instead of always falling back to the fact that people don’t like to be on-call. That’s how I like to see it.

George Miranda: That’s 100 percent true. I got to tell you I spent about 15 years of my career as an infrastructure engineer on-call that entire time and I think back then, I used to see it as just really disruptive. For a number of reasons on-call was managed horribly, horribly in my organization many, many years ago. I think we have better, more humane ways of doing that now. Now actually when I do find myself on-call, the thing that I see myself doing is creating a little bit of space and time, like this is the time when I’m just going to stick around and not really make plans in case something occurs. The beauty of that is that when my on-call shift is over, okay, great, I’m done and I can move on and it’s somebody else’s responsibility and it’s very clearly defined. Unlike earlier times in my career when I felt I was always on-call. Now just having this brief time when this is my specific defend duty during these hours and when it’s not, great, somebody else can take it makes it way more manageable. Then I’ll also add, I believe that when you are on-call you’re right, it does alert you to things that you might not have seen in your day-to-day responsibility if you’re not on the front lines of doing the things that go wrong. There’s a big educational component that I think … I think people don’t like on-call because it’s managed poorly and it can be inhumane, but if you do it right, it leads to all the things exactly that you’re talking about.

Leandro Pinto: It did. And talking about being more human I think that’s also what we have been trying to do a lot. I have been one of like really pushing for this internally like how can we make this thing like actually be manageable to everyone? Everybody has their private life that you need to be able to have even if when you are on-call. Let’s respect each other, let’s help each other. When you are on-call you need to have someone to override your shift for some reason, let’s help each other, let’s bring this and work as a team.

George Miranda: Yeah. I know when trying to work as a team in those ways is super important. Part of the reason that I wanted to have this discussion with you is I think we’re both at a very interesting intersection. I think for most of our careers, we’ve both been responsible for operating software and production in different ways. You’re dealing with making the shift to this Full-Service Ownership model at MessageBird. I wanted to really get your take from that pragmatic practical viewpoint, that change of operating model, what are some of the things that that change really introduces that teams tend to struggle with? Like what’s been hardest for you?

Leandro Pinto: Yeah. I think it’s interesting because I have seen, as you said, with the experience you sit both sides. I have seen traditional Ops and more like people who are responsible for the production environment where developers are already dealing with building software. One of the struggles that I see and you get from this and compared to like this model of Full-Service Ownership is more like it’s unavoidable that things will fall through the cracks. Especially if you look into the full journey of a specific feature of your application. There is not really clear ownership who owns like performance and how some of the aspects of the things that you do can actually be improved. If [inaudible 00:11:02] you’re only look into your software, you’re also looking to your production environment. That’s one of the big changes that I see. If you actually have your developers who also understand your production environment, they will also come up with different solutions for problems that if you don’t have it owned by the same team, they won’t be able to come up with some of the same solutions. That’s one of the biggest difference that I see from more traditional models to this one that we are discussing today.

George Miranda: I have a specific question for you about this actually. Because you’re right, you’re shifting a number of responsibilities there’s only so much that people can do. There’s only so much capacity. And I hate talking about people in terms of capacity because people aren’t resources, but there’s only so much that you can handle as a person. When you have this team that traditionally was responsible for more parts of software development, and you start changing these responsibilities and roles and duties. When you’re talking about things slipping through the cracks, what does that mean? Does that mean that you have to hire a whole new set of people or like double the size of your team or what happens to some of those things that get moved around? Do they just go away? Do people do more than they were doing before? How does that pan out?

Leandro Pinto: Yeah, I think that’s actually an interesting question because I’m not sure if I’m going to answer your questions rightly, but it’s actually something that is actually quite important to bring into the mix of our conversation. It’s like one of the things that I had seen moving in the industry and that today it makes much easier. Or actually the entry level for a Full-Service Ownership is much lower is because of the technology that we have available for us today. It’s very challenging for an organization who are still like for example, running all their stack in experimental or directly on VMs. Actually, if you compare them to organizations like us they’re running their infrastructure in more and modern environments like Kubernete and in the cloud. You get this Q Gap much like it’s smaller. Then the things that would potentially fall through the cracks or not be properly managed by any of the sides to date it gets more manageable. Because now the level of abstraction also for the teams is much higher. That’s where I see that, yes, I agree with you from the side of there is just how much a team can handle. But there is also a lot of progress that we made in terms of technologies that we actually instruct a lot of the complexity, for example, of running software in production. Basically the way we do it at MessageBird, we still do have what we call a platform team that is more like knowledgeable on the internals of the Kubernete or more like network related stuff and you need some people which also bring this level of skills or level of expertise. But one, we also invest a lot on trainings for our product development teams. Also to help skill them in a production environment, as I said like Kubernete and other more production oriented technologies. Instead of having a gap you have a little bit of intersection in their knowledge that will also bridge a little bit of the gap in between what we now today call the platform teams and also the product development teams. I don’t know if I answered your question directly, but just twisted a little bit.

George Miranda: I think you did. What it sounds like, and this is something that in a former lifetime, I guess still in this current lifetime, I was dealing a lot with CICD pipelines and introducing a lot of automation. The idea was that by leveraging tools that we had to do a lot of the tasks that would normally consume our time, we used to frame that as it frees teams up to be able to innovate. Like that’s what you do with that extra time. It sounds like in this, what you’re doing is you’re innovating the team structure and how it works and trying to figure out how we shift that around. Yeah, that’s absolutely important I think you touched on a key thing there, which is that change in tooling. It’s not always about tools, but depending on the abstraction layers that you’re using, the ways that you open up time and responsibilities are absolutely where things go so it’s not really falling through the cracks. Let me ask you though, so when you make the shift, whether it’s tools and whether it’s our platform attraction or learning Kubernetes, what are some of the more challenging things or what are some of the easier things, both of those, that you see your teams struggling with internally, or just like absolutely loving and doing very easily?

Leandro Pinto: Yeah. I guess like going back to what we’re discussing, I guess definitely one of the challenges indeed most of the engineers, they will have more of development background. Even though a lot of them actually enjoy a lot and will actually take the challenge with very quickly to really embrace all the production environment, there is a little bit of an initial gap, especially when you get people that don’t have this experience in the past. As I mentioned, we have been investing a lot on up skilling the team in training, especially for the production environment but also talking a little bit more openly about being on-call. Letting them know why this is important for them. We have our own modern teams at MessageBird, for example, basically you are on-call from day one. Just so you start to feel a little bit what does this mean. How does this change and also embracing the idea of the responsibility and accountability from day one. But I’ll say if you look into what makes things easier is indeed the closeness that you have from the business in your development teams and how to actually get to the bottom of different issues or solve different problems that you wouldn’t be able to if you didn’t have the full scope of ownership of your service. I guess it brings also people closer to the punchline in terms of solving problems that you have in your day-to-day in a way that by suffering from the problem you also be motivated to solve the issues yourself.

George Miranda: Let me ask you, do you think that suffering at the end of the day, does that make life better for your engineering teams or does it make it worse?

Leandro Pinto: No, in the beginning it might feel that it’s worse, but it will make it better in the long-term. The reason I say that, one of the problems that we had in the past, and there’s also a very common you can see a lot [inaudible 00:18:25] in different blog posts or books is amount of alerts or waking up in the middle of the night by a critical alert that was not supposed to be a critical alert. You see these discussions arising in the teams. “Hey, why is this alert firing?” And people challenging each other because they are, again, as I said in the front line actually receiving that alert and actually being a little bit penalized by then facing that issue then they will be motivated to solve the problem themselves. The short term, yes, you suffer a little bit from it, but in the long run I actually believe that this actually helped us to build more reliable software that we can trust.

George Miranda: Yeah, now I completely get that. For our listeners, I want to say full disclosure. Leandro you reached out to me about recording an episode of Page It To The Limit because you’d heard the podcast and the things that you were discussing, but you also reached out to me as a PagerDuty customer. You’re doing very interesting things in terms of how you’re restructuring your teams but I think how you manage alerting is part of that. Let’s talk about that for a minute. Can you tell us a little bit about what you’re doing in terms of alerting to help drive some of these practices?

Leandro Pinto: Yeah. Basically one of the changes that we are doing is how we actually structure our on-call for example. We have been structuring in the best alert based on escalation policies and in the teams structure. But then this creates a lot of overlap of, for example, which services are actually alerting more or which services are actually generating more of the false positives. Then we have a restriction, in this actually put the services at the center. And actually have our own call rotations and escalations based on the services itself and who owns the service other than actually reflecting our organization. Then of course we have a connection in between who owns the specific service together with all the other descriptions for the services. But we want to make sure that we are able to actually get the insights for alerts and how we manage them. How are we getting this at which time, who gets alerted. It also gives us a good understanding of the help of our assistants and why we should put effort on to solve a potential issues that we might have both for the people but also for the systems.

George Miranda: That’s a really interesting view of using service definition rather than the shape of your organization to set up structures for who’s going to be notified when. I think that’s a really important concept that makes, one, just your general overall management structure much more modular and easy to manage. Two, it makes them much more likely that you’re going to find the right person to respond or the right team to respond when you need them to respond. On that note when we were talking about this and what you were doing at MessageBird, you mentioned a specific tool that you were using in order to bring people closer to an incident. Can you tell us about what you were using?

Leandro Pinto: Yeah. That’s a little bit of a baby for us because we love this internal tool that we have. And it’s basically for someone to be able to start an incident and actually trigger our incident response process very easy. We have an integration with one of the internal communication tool that we use that it’s makes you very easily, again, to just trigger an instant and engage the right people to help you with a specific incident. This is not one person in specific that do this is actually the teams who can start instance and that actually connects very closely to what we have been discussing today. That is people have the ownership not only for developing but also dealing with issues in production when they arise. When we start an incident it actually automate or it engage in instant manager but also already create a ticket in our issue tracking tool that actually will help us to follow-up in the incident after. It creates a document that we will help you to take the notes during the incident. But also what it does actually also is also put people in a mindset that actually make them feel that we are creating an environment that they can also learn from the failures that we have. They actually brace their whole idea of like blameless culture and dealing with incidents, which also it’s important for the teams to actually have this in their day-to-day and how we actually operate the systems.

George Miranda: Yeah. That’s 100 percent spot on and that’s one of the things that we talk about a lot at PagerDuty, having anybody able to trigger the matrons and response process is very important for just making it accessible for saving time, for really ensuring resiliency of your services. You touched on a number of great things there, but the one thing I’m going to point out is that the particular tool that you’re talking about is a Slack integration. We use Slack at PagerDuty quite a bit and we have a Slack Bot that we use internally. That’s very customized to our needs. I’d love to put a link to the repo for the tool that you’re talking about in our show notes, so that folks can check it out and see how you’re doing it as another example of ways that you can build that yourself.

Leandro Pinto: Yeah 100 percent and that you also see going from myself there that also my hobby.

George Miranda: Nice. Let’s switch gears a little bit. You’ve been at this for awhile at MessageBird. How did this start? Is it something that organically certain teams decided we’re going to restructure this way? Have you been getting support from the business? How’s that journey going or where are you?

Leandro Pinto: Well, honestly I have to say this definitely grew organically. It’s more the way we see that for us to be successful also has it like a scale up, we would need to adopt this kind of approach for how we deal with our services. One of the things that I also have been doing a lot is more bringing their awareness that we are actually doing this and we are doing this with intention and actually adopt a lot of the ideas around this and bring this awareness to the teams itself. Also, that in the future they don’t expect that this won’t be part of their responsibility anymore but as we take most of the benefits of having this model. And one of the parts that is definitely a lot of contributions from the business is definitely on my principal SLO, defining expectations. What is expected from the systems and from the teams and how we want to, again, from availability and also all the non-functional aspects of an application, where how can we bring this transparency to the teams that is expected also from them on that side. And that’s why we have been trying also to bring the business there. But also with the product managers right in the teams also made sure that we are prioritizing properly work that needs to be done from a reliability or performance perspective because this is also something that we could also make it a little bit of a myth. Is that product features, will always be prioritized on top of reliability and availability. That’s something that you always need to make very clear to the business and the product teams. We have been able to find a very good balance there also because our product teams also embrace this idea to their heart as well. This has been very important for us.

George Miranda: Yeah. I think there’s something to having your product teams or your product owners embedded with yourself for development teams and your service owners running this in production. It brings them a little bit closer to understanding pin points for the team and just general availability. As you described non-functional features of your software and its availability, as well as having that balance with the business in terms of balancing innovation and new features. And just that it’s a very difficult dance and I don’t think anyone ever gets it completely right. But just having that closeness on both sides is absolutely essential.

Leandro Pinto: Yeah definitely.

George Miranda: This has been a really great chat, but as we’re getting close to the end of our time, Leandro, are there any other bits of advice you would have for teams that are thinking about restructuring in these ways?

Leandro Pinto: I think it’s a little bit more general. I think initially this idea might sound scary, especially when we talk from a skills gaps perspective, or a question that I always hear is like, “Oh, but do you really think that developers can run and own services in production?” And what I always say is like honestly, software engineers they have the foundation that is needed for this to actually be successful. The foundation is that is just like more on top of that they need to learn and the skills that they need to acquire, but most of them are very eager to actually do this step and take the actual level of ownership and responsibility. I understand how this feels scary at the beginning, but in the long run, it pays off both from a business perspective but also from a career and motivation for your people. That’s a little bit also a light that I want to just put in to the mix.

George Miranda: Yeah. That absolutely makes sense and it’s interesting to hear you say that now, because I feel like there’s been a couple of waves of DevOps. And I think in the early days, what we saw a lot of was Ops that were learning development and I feel like that is where this movement got started. I think within the last couple of years, we’ve really seen the other side of that divide which is now really focusing on software developers and teaching them those operational skills. But it’s really weird that it went in that order. I 100 percent echo your advice, I think it’s interesting that, that’s where we are now.

Leandro Pinto: I agree.

George Miranda: All right. If you’ve found this interesting Leandro, you mentioned the PagerDuty All Service Ownership Ops Guide that’s available at ownership.pagerduty.com. If you’re interested, there’s a Page It To The Limit episode eight where we discussed what that guide is. If you’re interested in checking that out a little bit more, I highly suggest listening to that episode. Leandro, there are two questions that we ask guests on this show. The first is what’s one thing that you wish you would’ve known sooner when it comes to running software in production.

Leandro Pinto: It’s definitely that it’s okay to make mistakes. I think that it’s even something that has been very stressed in the late last years about like blameless postmortems and dealing with incidents. I would benefit of knowing this sooner. And also that you don’t have 100 percent uptime. It’s impossible to deliver that. Again, I would definitely benefit of having this kind of insight sooner in my career to be honest.

George Miranda: Those are two things we could absolutely talk about more and we should on this show. One you’re right, there’s no such thing as 100 percent uptime services are always failing even more so when they’re distributed. I think the new downtime is actually slow performance. That’s another can of worms that I’m not even going to open. But the other thing you touched on is psychological safety and I think that is a vital component a thing that operating [inaudible 00:00:30:22]. The other thing you touched on is psychological safety, which I think is a vital component of being able to operate in these ways and you’re right your teams have to be able to make mistakes and learn without fear of retribution. There is a learning curve, we’re changing the way that we work, we’re changing levels of experience and exposure. When those things occur, I think we need to see them as an opportunity to learn rather than just thinking it’s a mistake that’s unacceptable so 100 percent I completely agree. Speaking of mistakes and fear of retribution, is there anything about running software in production that you’re glad that we did not ask you about on this episode?

Leandro Pinto: Yeah, to be honest, I guess is that what was the biggest mistake that I did?

George Miranda: So. Since you brought it up what is the biggest mistake that you’ve made?

Leandro Pinto: Now this is a joke, but of course I think especially if you step into leadership position, sometimes you need to take a different approach in some of the decisions that you make. I did make decisions in the past where I decided to, for example, de-invest in some of the critical systems that we have, like moved people from a team to [inaudible 00:31:35] one. Just thinking like, okay, the system is fine for now, but then of course, two years later it blew up in my face. This is also interesting not only from a production and running systems in production but more how you lead teams that actually run systems in production. And how you actually make sure that they do have the right level of bandwidth as you mentioned at the beginning, like capacity, do they actually have the right level of manpower? Let’s say to actually deal with the systems that they’re giving us criticality that they actually have for your business. I did make this mistake in the past, but I did learn from them. I’m very critical these days on how we actually, even those systems are actually perform in a production, that we move people to different teams or move the focus off the teams, because if something is actually running okay today doesn’t mean that it will from one month from now. Especially if you’re dealing with high growth of traffic and all of those things. That’s definitely something that I’m glad you didn’t ask.

George Miranda: I think those lessons in figuring out where to make investments that comes with experience and figuring out how we manage these things over time. I’ll tell you, since you’ve shared early in my career, I made a really bad decision around how we invested in our backup strategy. And we relied a lot on tapes that were very faulty. This is how long ago that was. We used to use tape for backup. I did not know that we have two bad tapes in a rotation of six that needed to be used for backup of a data warehouse. We experienced a disk failure on that data warehouse, and we lost about four weeks of data that took us about 12 weeks to reprocess. It happens, we all make those mistakes. We learned the hard way and we’re just lucky when we use those as experiences that we can learn from rather than events where somebody can get fired. So 100 percent hear that. Leandro, thank you very much. This has been a great discussion. I feel like we could keep this discussion going and there’s a lot of practical experience that you have when it comes to restructuring these teams. So for our listeners, if folks want to reach out or they have questions, how can we get ahold of you?

Leandro Pinto: Yeah. You can also reach out to me via Twitter. My account is Leandroshp. I’ll be glad to also answer and keep up the discussions ongoing from there.

George Miranda: Excellent. And we’ll go ahead and put that in the show notes. Leandro, thank you again for being here. This has been a wonderful discussion and I’d love to continue this again at some point.

Leandro Pinto: Yeah. Thank you very much for also having me as a guest.

George Miranda: Well, thanks for being here. And again, this is George Miranda, wishing you an uneventful day. That does it for another installment of Page It To The Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast, if you like what you’ve heard. You can find our show notes at pageittothelimit.com and you can reach us on Twitter at Pager To The Limit using the number two that’s @pageit2thelimit. Let us know what you think of the show. Thank you so much for joining us and remember uneventful days are beautiful days.