Best Practices With Ivan Merrill

Transcript

Mandi Walls: Welcome to Page it to the Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve both system reliability and the lives of the people supporting their system. I’m your host, Mandi Walls. Find me @LNXCHK on Twitter.

Hi, folks. Welcome back to Page it to the Limit. This week, I have with me Ivan Merrill. Hi, and welcome to the show.

Ivan Merrill: Hi. Thank you, Mandi. Thank you for having me.

Mandi Walls: Tell us about yourself. Tell us about what you do, where you’ve been.

Ivan Merrill: So, hi, I’m Ivan Merrill. I am currently a Head of Solutions Engineering at a small little company called Fiberplane. We are making collaborative tools for SREs and things, but I’ve got a big history in observability and monitoring. So, I previously ran the UK monitoring team at Capital One UK and spent a few years there, and then previously spent 10 years mostly in APM kind of things. So, previous configurations [phonetic 00:01:11] of monitoring and stuff like that. HSBC, the technical lead for some of their APM roll-outs. So, a long history in monitoring and reached through to observability and everything else like that. And a lot in financial services, actually, as well. It’s own kind of beast, really.

Mandi Walls: It definitely is. We’ll have to circle back to that in a little bit. So, you’re going to tell us a bit today about best practices and encouraging that across organizations. Why are best practices important? Why are… Why do you feel they’re important?

Ivan Merrill: Yeah, that’s a good place to start. So, one of the things that we’re all… Not necessarily waking up to, but really aware of at the moment is that observability, monitoring, they’re super important. Right? They are a big driver of reliability, which, as we all know, is a product’s number one feature. So we all want it, right?

And there’s a lot of pressure on teams and people to deliver things, to deliver new features. There’s many types of development that exist but, most often, we find ourselves using date-driven development, where something has to be out the door by a certain time. And, as a result, that means that with all these pressures there are things that easily get dropped. They don’t happen when you… You know you want them to, but they’re just not going to happen within the time. So, one of the things that we kind of frequently see is that monitoring and observability and building services that are observable is very easily lost in the wayside at times, particularly for the first release and everything. And that’s a problem, right? We want our reliability, but we are almost willfully saying, well, actually, that’s not an MVP. We’ll put that in later. Right? And that’s a factor of life and there’s ways that I think and I feel that we can do to help people and teams bring that back in and make it more important and just encourage people and ease people into a way of actually building it into their processes, into their delivery model, into their way of working. And, if you can do that, if you can help people rely on these best practices, you can give them a little leg up on their way to get to there, then, actually, they’re more likely to do it. Then, once they do it, it’s a bit of a snowball effect. They see the benefits and everything else like that. So, having these best practices, having some kind of guidance for people so that they know what they’re doing is great. Because there’s often quite a lot of obstacles to these things and one of them is of the obvious technical thing of we might need to deploy an agent or we instrument our application stuff. But the other thing is the human side of it. And I think that’s an incredibly important thing and there’s knowledge of how to instrument their services, but then also how to use the data that they get out of it. Right? What should they be doing and how do they do that? And that can be, not necessarily scary for people, but it’s an obstacle. Right?

Mandi Walls: Yes.

Ivan Merrill: It’s an obstacle to overcome, and while I’m trying to remove that and make it as easy for people to get involved in these things, and that’s a way that we can kind of encourage them to do what we know to be the right thing or the good thing, but maybe that isn’t necessarily always happening.

Mandi Walls: Yeah. Well, as you’re encouraging folks, is it a carrot and stick process? Or do you feel like once folks see it succeed, they’re more likely to be on board with it and, as you mentioned, like snowball effect? We see that with other practices. Hey, team B over there is doing really well and they’re doing this thing. I want to do that thing too.

Ivan Merrill: Yeah. That’s a really good question. My personal preference is always going to be the carrot, right?

Mandi Walls: Uh-huh (affirmative).

Ivan Merrill: I once had a really great conversation with a developer many years ago and he said Ivan, all you do is you come to us and you’d help us. You’d tell us you’re going to help us know when our stuff is broken or it’s not working. He says then our stuff works. Our stuff works all the time, right? Why are you telling us this? Why don’t you tell us you’re going to help us understand that it works 99.99% of the time as of our SLO? That’s what we want to hear. That’s what we want to know. That’s a really good point, actually. So, I think it’s one of those things where when you are having these conversations with the engineers, when you’re having conversations, particularly with the product owners, when about the prioritization of this work, it’s like I want to show what a great job you’re doing.

Mandi Walls: Yeah.

Ivan Merrill: Right? I want to give you the backing, the evidence, the proof of what you’ve done is great and it works and it’s reliable and it’s fast. Coming back to the human side of it, I think that’s a much more pleasant conversation to have with people, right? Because saying to people you need this because when your stuff breaks we need to be told. It’s like okay, I’m sorry. It’s a very… It’s a letter of the law kind of thing, thou shall make your service observable. That’s not really… Whilst it might be true, it’s just not a very nice way to approach things, so…

Mandi Walls: Yeah, absolutely.

Ivan Merrill: Definitely go with the carrot. Yeah.

Mandi Walls: That’s interesting. And I’m glad you brought up the product folks too. Because how does that conversation go? I feel like it’s evolved a bit in the last five to 10 years. As we’re seeing, yes, reliability means more customers and happier users and all of these things. But I remember the dark days before, when fighting for good operational features was a lot more drama, I guess you could say, than it is now. How have you seen things evolve as far as the product owners and prioritization of non-feature components of a service and how that relates into reliability?

Ivan Merrill: I don’t think it’s ever going to be a perfect conversation. I mean, maybe it is somewhere, but I’ve not reached that kind of nirvana myself. But I think, again, it’s about, as said, the human aspect of things. Fundamentally, I think that no one is there to say, yes, that’s a good idea, but we absolutely don’t want it.

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: Right? No one’s there saying, no, we want a really unreliable product or we don’t care about reliability. Right? It’s a good practice. We’re wanting to make observable services because it’s the right thing to do. So, I think that one of the first things to do is assume best intentions might be a good mantra to go by there. But I think, fundamentally, as I said, it’s about hey, I want to help you and this is something that is in your best interests. Hopefully, you can explain to someone what you want to do and why it’s important and how, actually, if they are going to release that new feature, they want to know it works, they want to know it’s secure, it’s reliable, all these things. And, in order to do that, they need to have the data and actually they want the data. Right? And it’s like hey, I’m your ally in showing you all the great stuff that you can do and that you have done. That’s obviously a thing, but sometimes it isn’t always that easy. And there are times when you do… I don’t know, it might sound a big extreme, but almost shove value in front of people’s face. Right? Just like hey, this is the stuff that we are doing with another team. Look at how they’ve got this really nice conversation between yourselves in product and the engineers because they’ve got their service-level objectives, they’ve got the data to back it up. They can say that they can push that release in really rapidly despite the risk, because they’re smashing it on their error budget. Right?

Mandi Walls: Yeah. Yes.

Ivan Merrill: You know? That’s a thing. So, hopefully, it’s a really nice one and they’re all definitely signed up for it, but that’s not always the case. And sometimes, let’s say, you do have to be a little bit still trying to be carrot, still trying to say hey, look at these great things, here’s what you could have won. But at the same time, yeah, just trying to get it out to them and trying to show them there is a way that’s helpful to them.

Mandi Walls: Yeah. Yeah, the prize is behind door number two, should you choose.

Ivan Merrill: Exactly.

Mandi Walls: Has the conversation changed or gotten easier as the tooling in this space has changed? It’s the hot sort of developer tools right now to have more robust observability, to have more of those features even into your legacy monitoring platforms as they release new components into those. Does that make all of this easier for teams, or maybe not?

Ivan Merrill: That’s a very good question. People are always attracted to…

Mandi Walls: We love the shiny things.

Ivan Merrill: Yeah. Exactly, right? And observability is, as you were saying, a very hot topic. Right?

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: It’s great to see that things like SREcon and all these things are really bringing observability to the forefront. And, in these conferences, a lot of them have their own tracks and things like that, and that’s amazing. And people do go oh, that’s shiny. And look at what they’re doing, I want some of that. And that’s great. And the tools are trying their best to keep progressing, keep doing great things and everything else. But for me, I mean, fundamentally, again it’s a human thing. Right?

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: I always find if you really want to get some budget for a new monitoring tool or every other [phonetic 00:09:56] thing, you approach the people with the money straight after a big incident. Right? It’s a much easier conversation.

Mandi Walls: Okay.

Ivan Merrill: Much easier sell then, because people are seeing okay, that was bad, we don’t want that. Essentially we can spend this amount of money to get this tooling, which will hopefully help us. Okay. That’s a much easier business case, right, when you’re saying well, actually, we could have known about this if we had these tooling. But I think, as well as that, there is also the fact that tooling isn’t everything. Right? And we just need to be careful that… I always kind of say that you can have the best implementation in the world, the most perfectly instrumented services. But if you don’t have the people engaged, if they don’t really know what they’re doing with it, or anything else like that then, essentially, you’ve spent a lot of money and gained no benefit. Whereas it’s entirely possible on the flip side to have a really basic, rudimentary tool…

Mandi Walls: Yeah.

Ivan Merrill: And some really basic things. But, as long as you have the people that know how to use them, you have the good processes and everything else of that, you’re going to be in a better place than the person that has the best tool. And that’s just a thing because, as said, it’s a very human thing.

Mandi Walls: Awesome. So, in your role, what kind of things are you doing with teams when you’re working on, say, encouraging best practices? What does that look like on the practical side?

Ivan Merrill: Yeah, that’s a good question. And I think it’s running down to one of those things where you say it depends. Right?

Mandi Walls: Uh-huh (affirmative).

Ivan Merrill: And that’s a bit of a cop-out, but reason is because there is no two services that are the same and it’s going to be different for everyone. But I think there are a few fundamental things that you can do. So, one of the things that I’ve recently written a blog about is things like hypothesis-driven investigation and actually being able to say this is how you can do it. And then certainly, with Fiberplane, what we do is we provide templates to allow people to lead them through that process. Right? To help them say okay, right, I know now that if I do this and the next step is this and the next step is this… Just to help people and guide people through that process and give them an element of safety in their…

Mandi Walls: Okay.

Ivan Merrill: Decision-making and everything else. So, there is an element of consistency…

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: Across it all, right? And that’s in its broadest sense, because I’m not a believer in everyone must do this thing exactly the same every single time because that’s probably not going to work. And then, the one time that it doesn’t will be the biggest incident and…

Mandi Walls: Yeah.

Ivan Merrill: That’s the way the world works, right? But it’s a case of, as said, just encouraging good practices that… Things that you know work, so vaguely standardized investigation process. Making sure that you are getting the very basic level training for people, right? Good training, good documentation. In my… I joked with my team in my previous role that we were going to have documentation week, where we were just going to spend a week writing documentation on everything that we did, and they weren’t too enthusiastic. I’m not [inaudible 00:13:02].

Mandi Walls: Of all the investments you could make in most environments, a documentation week would be amazing for so many people.

Ivan Merrill: Yeah. I think it… Yeah, absolutely. It would be. It wouldn’t be the most exciting week and…

Mandi Walls: No.

Ivan Merrill: I think they were slightly miffed in that I’m probably the worst person at getting around to writing documentation myself, so that was a little bit hypocritical. But the point was we wanted to enable people to be self-serve as well.

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: That’s a really big thing. We want to lower the barrier of entry for observability and monitory. We want to make sure that both the technical implementation… Can we automate it? What can we do to make it easier for people to get on board so that when we’re going to the product manager or product owner or the engineering team, it’s only this teeny amount of effort?

Mandi Walls: Yes.

Ivan Merrill: Half a sprint’s worth of thing to get a fully-instrumented process that goes through everything. So, we’re lowering the barrier of entry, we’re making it easier technically, but then we’re also making it easier… I’d say knowledge-wise, there is an obstacle there. So, what can we do to make… We used to run, in my old team, every tool. We had a 101 session for it.

Mandi Walls: Okay.

Ivan Merrill: We ran them on monthly, monthly 101 sessions, and we encouraged all new engineers. We put it out on our Slack channel. Hey, everyone, we are running the basic intro to this particular tool so sign up, come join our call and we’ll do an hour or two hour session and we’ll give you an intro to the basics of this tool and what it does and what it can do and everything else like that. And that is something that is great because not only do people then start to understand okay, not only do I know how to log onto this tool but I know what to do with it, but then there’s a certain amount of enthusiasm about it…

Mandi Walls: Yes.

Ivan Merrill: That it encourages. People can go oh, it’s really cool, you can do this and enact all and oh if I click this button then whoa, look at my fancy graph or… Do you know what I mean? Those things are… That helps. And, again, maybe it’s a… You might be bored of me saying this by the end but, again, it’s a very human thing. Right?

Mandi Walls: Yeah.

Ivan Merrill: So, we’ve just got to make sure that we are encouraging people. We’re removing the obstacles, we’re just trying to make it more accessible, easy for people to do.

Mandi Walls: Yes.

Ivan Merrill: And just, again, coming back to the documentation, more self-serve. I want people to be able to say hey, I’ve got this problem, it’s a challenge for my team. We should be able to get people more self-serve for… If they post a question in our Slack channel hey, here’s our documentation here… Next question, hey, here’s our documentation, here’s the page that shows it. Right? So, trying to be helpful to people, obviously, but also trying to encourage that practice of you know what, we’ve done our due diligence, essentially…

Mandi Walls: Yeah.

Ivan Merrill: To make it as easy as possible for you, so here you go, we are not a blocker for you. There is no obstacle…

Mandi Walls: Right.

Ivan Merrill: Go forth and make your service observable.

Mandi Walls: Absolutely. You don’t want to be the bottleneck to everyone else’s feature shipping or success or holding anyone back because they’re waiting to get a time on your calendar.

Ivan Merrill: Yeah.

Mandi Walls: To get the help that they want. We had a past episode with Rich Lafferty, who’s the principal SRE here at PagerDuty talking about similar things that we do here with the different components of the infrastructure and putting together the golden path or the paved road for the various pieces that the service teams then need to put all their stuff together so that… You want to accelerate people. You want them to be able to get their stuff out.

Ivan Merrill: Yeah.

Mandi Walls: And in front of users and creating value and bringing that delightful product experience out as fast as possible. It’s great to apply that then to observability, which I feel, like you say, for some folks, it still feels a little bit like magic and maybe a little bit more convoluted or a little bit more complex than some of the other things that just say oh, here, build these AMIs and you’re going to be fine.

Ivan Merrill: Yeah. Yeah. I think the golden path is exactly… I mean, I wrote some notes for this and I’ve got golden path mentioned several…

Mandi Walls: Excellent.

Ivan Merrill: Times, because…

Mandi Walls: We’re on the same page with the vocab, at least.

Ivan Merrill: You don’t want to prescribe to people, again, thou shall do this.

Mandi Walls: Yes.

Ivan Merrill: You want to say hey, here is maybe the path of least resistance, the golden path that… The way that, if you follow this thing, loosely follow it, it’s not an exact thing, but it will give you a good outcome. Right?

Mandi Walls: Yes.

Ivan Merrill: And you might want to choose to change it, edit it, make it work a little bit for you. Hey, that’s cool, go wild. But here is a simple way that we, as a team, have taken on our responsibility to make as easy as possible for you. When I’m running a monitoring and observability team, I always said there’s two parts to our job. There is the technical implementation.

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: That’s the easy bit, right? I mean, we’ll install some stuff, we’ll run some stuff, we’ll build some things and it’ll work. Great. But we need people to use it. So there is almost this sales job. And then the post-sales job, once they’re sold on it, to say we want you to use it. So, how can we make it easier for you? How can we make it better for you? How can we make that golden path that means that you can resolve incidents quickly, right?

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: I mean, because, fundamentally, we all do this for a reason. We don’t, most of us anyway, don’t just do it for fun. Maybe a little bit, but…

Mandi Walls: Maybe a little. Yeah.

Ivan Merrill: Yeah. But there’s a reason that companies are investing in observability tooling, people are excited about the subject. We want reliability. That is our end goal. Right? So, what can we do as a team to make it easier for everyone else so that they can reach that themselves? With, firstly, as little effort as for us, because that’s not a model that scales. I can’t have one engineer for every team that we work with.

Mandi Walls: Yeah.

Ivan Merrill: Right? But also to empower those people to be then thinking okay, well, we’ve got this and that’s really good, but actually we had that incident so we’ve been thinking about, well, we’ve got this new feature. What can we do to make it interesting? What data, what information is valuable to us? And once you’ve got people thinking about that throughout the creation phase, when they’re building things, then you most have the weight there, because people are thinking about observability as they’re building things. And that’s a big win.

Mandi Walls: Yeah, absolutely. Because you can think about them. Once they have the tooling and are familiar with what it’s going to be doing for them, they can make better decisions, better architectural decisions about here’s our dependency map, here’s what it looks like, here’s how we can do more defensive coding based on what we saw in this problem we had before with this thing, or better defaults for simple things like time-outs or whatever when they’re making requests. Just to play into having all this tooling that’s going to give them more telemetry around their systems. It sounds fantastic. Right?

Ivan Merrill: Yeah. It does sound fantastic. We’ve… No, that’s the probability. There we go. We’ve cracked it. Right?

Mandi Walls: We’ve solved it.

Ivan Merrill: Yeah.

Mandi Walls: Yep.

Ivan Merrill: That it…

Mandi Walls: Solved all the problems.

Ivan Merrill: Yeah. It can be as simple as that. Right? So, when I was with a team and we saw that Lambda was clearly going to happen, it was going to be this big technology, server-less in general and everything else like that, we looked at it and we’re like well, we need agents on things. What are we going to do? This is an issue for us in terms of instrumentation. So, what we did is we built a central solution whereby okay, here are some libraries that you can add into your code and what they will do is they will allow you to generate some really nicely structured logs that will send over there. Don’t worry about any of the other things you send it for, here’s our really nice documentation. Explains everything you need to do. Yeah. You just include this library, add some things in and boom, there are your beautiful logs that are really nicely structured that you can search over it. You can do this, you can do that. And it’s amazing how you build these things, you give people these great things, you do the sales job, because, again, they’re not going to just know it’s there or anything else of that.

Mandi Walls: Yeah.

Ivan Merrill: You go out to teams, you say hey, we really think that this a… And if it isn’t working for you, why isn’t it working for you? What can we do to make that better? And then, all of a sudden, it was a success. We had lots of people. As soon as they build a Lambda function, they’re thinking what can I put in, what information would be useful? Maybe we do want that bit of data. I’ll add it in, it’s as simple as just a line of code and I’ve got all this stuff and great.

Mandi Walls: Yeah. Try it out. See what works. If it doesn’t, it’s code. You can always delete it. It’s fine.

Ivan Merrill: Yeah.

Mandi Walls: So, of the things that you’ve worked on, what hasn’t worked with folks in your experience on things like this?

Ivan Merrill: Yeah. Well, I think I’ve touched on it a couple of times, right? And I think it’s mandating things.

Mandi Walls: Yeah.

Ivan Merrill: It’s saying to people you have to do this, you have to have this minimum amount of metrics, you have to have done this thing. If you don’t, we will not let you go live or whatever, because what that does is it becomes… You’re making observability and building services observable and adding monitoring instrumentation, you’re making it an obstacle. You’re making it a thing that needs to be overcome on the way for people to get live and everything else like that. And, fundamentally, if you start putting obstacles in front of people, they’re not going to go oh great, I look forward to…

Mandi Walls: Oh, something to jump over? Oh, thanks. That’s great.

Ivan Merrill: Yeah, or I know need to do this or that or the other, right? This is not the… That’s very much the stick approach coming back to the earlier question. In general, I find that you are quite likely to… Firstly, people aren’t happy about that. But then, secondly, you’re likely to get disengaged people, demotivated people. They’re not going to come back to you and say hey, thank you for putting that obstacle in my way. What else have you got? They’re going to look at it and go oh, we have to do this because that… They’re making us do it or something.

Mandi Walls: Yeah.

Ivan Merrill: And that’s a really, really tough thing. Right? It’s not where you want to be. So, if I could say one thing to not do, don’t make it an obstacle. Think about a nice, happy golden path as opposed to a requirement like that.

Mandi Walls: Absolutely. Awesome. So, on the show, we have a recurring question that we ask everyone. Is there a myth that you would like to debunk about either establishing best practices for teams or about observability or any of the other things that we’ve touched on so far?

Ivan Merrill: Yeah. So, I think the most pervasive one that I see that I think is maybe the most harmful is touching again on the previous subject which says that monitoring and observability isn’t just something for running things in production.

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: It shouldn’t be thought about as a last-minute thing. Oh, we’re going to go live, so we need some monitoring. Quick, whack it in. Yeah.

Mandi Walls: Yeah.

Ivan Merrill: I’ve got two days to get this in before we hit the… Firstly, that’s the most expensive way to do it. Right? You’re trying to retrofit something.

Mandi Walls: Mm-hmm (affirmative).

Ivan Merrill: You have no idea what information you got back, you’re going back to bits of your code you might not have touched in a while and stuff like that. But, secondly, by the time your service reaches production, you want to know what it looks like.

Mandi Walls: Yeah.

Ivan Merrill: Right?

Mandi Walls: You had lots of opportunity to meet it and talk to it in other environments. What’s it telling you…

Ivan Merrill: Yeah.

Mandi Walls: Before it even gets to the customers?

Ivan Merrill: It’s unfortunate, I would say, to be polite. The amount of times I have come across where people have incidents in production and you say okay, well, what happened when you released it in your pre-production environment? And they’re like I don’t know, we don’t monitor that. Or they’ve got their beautiful production monitoring equipment…

Mandi Walls: Yes.

Ivan Merrill: But they’ve got nothing in…

Mandi Walls: Nothing.

Ivan Merrill: Repo, just maybe an agent installed or some keep alive checks. Oh, look, it broke there too. Oh, yeah.

Mandi Walls: Oh yeah. It did, four weeks ago. Yeah. Right.

Ivan Merrill: Could have saved yourself an incident there.

Mandi Walls: Yeah.

Ivan Merrill: Nevermind. So, if I would say one thing not to do is to say that it’s something to be tacked on at the end.

Mandi Walls: Yes.

Ivan Merrill: You want to know what’s going on with your service, what’s going on with your new feature before you release it to people.

Mandi Walls: Definitely. Awesome. Was there anything else you’d like to leave folks with that we haven’t touched on?

Ivan Merrill: I think that it’s been a really great conversation. I’ve certainly really enjoyed it. And if you would like to know more for anyone, I do blog reasonably regularly on our Fiberplane blog that you can find at Fiberplane dot dev. Or, if you’re interested in the tool, we’re looking at really making incident resolution and everything a collaborative experience. So, you can sign up for a trial there. So, also, yeah, just come and have a look and see what you think.

Mandi Walls: Yeah. Awesome. We’ll put some links in the show notes for folks who are interested. You can check that out. Or hoping to see or hear more from Fiberplane as this expands in our ecosystem at PagerDuty for instant response. So, should be really good. Ivan, thank you so much for joining us. This has been super fun. So, who would’ve thought a best practices discussion would be so much fun? It definitely is.

Ivan Merrill: I’m glad you think so. Yeah. Thank you very much for having me.

Mandi Walls: It’s all good. Yeah. I hope folks have picked up some tips here. And, in the meantime, we’ll see you on our next episode and we’ll wish you all an uneventful day. That does it for another installment of Page it to the Limit. We’d like to thank our sponsor, PagerDuty, for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at Page to the Limit dot com and you can reach us on Twitter at Page to the Limit using the number two. Thank you so much for joining us. And remember uneventful days are beautiful days.