Tools and Usability With James Governor

Transcript

Episode Transcript

Quintessence Anx: Welcome to Page It to the Limit, a podcast where we explore what it takes to run software in a production successfully. We cover leading practices used in the software industry to improve system reliability and the lives of people supporting the systems. I’m your host, Quintessence, or QuintessenceAnx on Twitter.

Quintessence Anx: Topics in cultural change. How do our tools change what we do, and how do we change our tools by what we do? We are joined today by our guest, James Governor, who founded RedMonk in 2002 with Stephen O’Grady. RedMonk focuses on developers as key influencers in technology. Thank you for joining us today, and welcome to the show.

James Governor: Thank you so much. Really happy to be here.

Quintessence Anx: So to get started, can you tell us a little bit about what we’re going to be talking today with regards to tools and cultural change?

James Governor: Yeah. I think, from my perspective, we’re all on this journey all the time, trying to become more effective, trying to move towards production excellence, trying to become more effective in terms of the software we build and how we manage that software. And one of the things that I think is an interesting tension in the industry as a whole is that we’re building these platforms, we’re building these tools, and then people will say things like, “Oh, no DevOps. That’s a culture change, or Agile is a culture change, or SRE is a culture change. It’s not about tools.” And I think that that kind of misses something about what we’re doing here, which is that there is an interplay between the tools we use and the methods we choose and how we work. And I think it’s really important to get beneath that idea that tools won’t bring you change. Because actually I think if we look at some of the fundamental waves in technology and how we build it and manage it over the last few years, tools have been a huge part of that.

Quintessence Anx: That’s amazing. So it sounds like you touched on this a little bit, but would you say that’s a myth or a misconception that you’ve encountered, is about…?

James Governor: Definitely it’s a myth. It’s a myth. And I’ve seen multiple people stand up at tech conferences. They get up on stage and they’re like, “Oh, anybody selling you tools and saying it’ll drive DevOps transformation, that’s snake oil. The thing you have to understand is, it’s a culture change.” That’s a very common tech topic. And I have some sympathy for that. Although sometimes they’re like, “Oh, there’s a person trying to sell you tools. No wonder they say that,” and quite often it’s coming from a DevOps consultant. And I’m like, “Interesting. The consultant is saying that people selling you tools are bad.” So I think from my perspective, that trope that it’s either/or is not necessarily helpful, and it’s not really helpful for organizations, because they want advice. They want opinions. They want platforms. They want roads that they can walk on. They want golden pathways that are going to get them to being in a better place. And so just saying, “Oh, it’s a culture change…” Like, well, okay. What do I do now?

Quintessence Anx: Right. And kind of to that, we talk about people who are selling the tools and so forth. So let’s talk a little bit about how the tools can actually help drive that cultural change, where that dependency is.

James Governor: Yeah. So I mean, I think some of it is just about, frankly, working practices. And 2020 taught us a lot about… So for example, let’s consider remote work. Some orgs are like, “Oh no, we don’t do remote work.” And suddenly it’s like everybody does remote work. Has using Zoom changed how we work? I think it has, because we haven’t had an alternative. Certainly the world of going to conferences or going to meet-ups, that’s gone. There are no in-person meetings. That has been a change. We’ve had to work out new things like, oh, some people say you’ve got to have video on, but then some people are like, “Actually, this is my home, and why should I always have to have video on?” So there are cultural norms, but we’ve learned to use these tools.

James Governor: And so I think that the tools like Slack… Slack is another one I think is interesting in this regard, because they actively went on a sort of PR campaign about, “Oh, Slack is great for remote work. We’ve always been about that.” When in fact, they had always been working on campus. So they’ve been on somewhat of a journey as well. But yeah, if we think about… Perhaps a better example would be GitHub. Now, obviously there are all sorts of things you need to learn in using the tool set, but anyone that says using GitHub and/or Git and associated platforms doesn’t change how you build software is… I don’t know if I can say this, but is smoking something. Clearly there’s a culture change associated with the adoption of the tools.

James Governor: And finally, I would just go back to… Here we are. We’ve got this notion of DevOps. And one of the core technologies of DevOps would be in and around infrastructures code, previously the wave of configures code tools. And the idea that organizations would have been able to do DevOps in the early days without a Chef or a Puppet or a SaltStack to me doesn’t really stack up. So I think it’s that these tools can indeed engender a change in our working practices.

Quintessence Anx: And that makes sense. And to your point about GitHub, when you’re working with version control back when that was… I’ll say newer, because you go a while back for when it was legitimately new. But you make the choice to do version control. You make another choice to choose Git or Subversion or whatever. And then that second choice feeds back into the first choice of, “Okay, well now how will we implement version control?” And going to the push backwards, because it is a bi-directional, when you make a tool choice, there’s your first choice, like, “I want to do Dev Ops,” or et cetera. And then when you start choosing the tools to support that top-level choice, what are some of the other decisions that the tool choices kind of push down and say, “Okay, well now you’re going to be doing it this way, because you’re working with these tools and their designs”?

James Governor: Yeah. That’s a great question, because there are always first-order changes and then second-order changes. So if we look at GitHub, you start asking yourself questions about, “Do our developers understand how to use distributed version control? Do they understand Git primitives?” And we need to get them to understand them. We need to think about what it means to do trunks and branches and to ship trunk. We need to start thinking about how that affects our other tool choices around testing, CI/CD. And so the GitHub choice then begins to instantiate, I think, a bunch of other changes in and around tool choices and process choices.

James Governor: And we have these periodic, “Oh, everything should…” People will be like, “Oh, everything should be GitFlow.” And then a bunch of other people will say, “No, that’s nonsense. That makes no sense. Let’s go with TBD.” And that gets to the idea that there should, ideal world, be some kind of flywheel where you want to achieve an end, you invest in some technology and some skills and people, quite often bring new people in that know some of those things, they begin to do things, and then that going forward makes you start saying, “Okay, now we need to embrace a set of other choices and decisions.” A good example… Everyone wants to do DevOps. Everybody wants to do modern development. They use GitHub, and then they’re like, “Huh, there is this new approach emerging for automation and managing centrally the sort of scripting that we’ve previously done and having some control and some desired state and some who did what and when, and we’re going to do GitOps.” So yeah, I think it’s that interplay that I find sort of fascinating, and it is associated with tools.

Quintessence Anx: Yeah. And pivoting a little bit to the people, because that’s what you mentioned… So the developer and operator’s experience in these tools and being able to properly train people and bring on the new people… How can we talk a little bit about that experience gap and how to bridge it?

James Governor: Okay. Well, I think there’s a couple of things there. One, obviously, onboarding experience, and that’s something that we definitely have not given enough attention to as an industry. We talk about developer experience, and we’ll get to that in a second, but I just think onboarding experience… So what does it mean to join a team that has a set of norms, that has a set of considerations and processes that they’re used to? Because for us it’s like, “Oh, onboarding. Here’s your Mac. Let’s get going. Here’s a couple of repos.” So really thinking about a much richer, I think, experience in terms of onboarding with some training, with some mentoring, with a broader set of education around the platforms that will be used, I think, is super important.

James Governor: Because one of the things that we’re doing in the industry at the moment… On the one hand, we’re like amazing. We’re going to invest in practitioners, and we’re going to give them choice, and it’s going to be great. And it’s Nirvana. It’s a golden era for the software developer. And there’s all of these tools they can choose. But the problem is, they’ll choose them, then. And once they’ve chosen them all, then the organization’s like, “Well, great. Okay, you chose these tools. You can glue it all together.” And the job of gluing the tools together becomes pushed out onto the developer or practitioner in a way that isn’t always helpful. And so I think the organization should always have an opinion about these things. Developer choice is great, but it has a cognitive overhead for the individual, and it has an organizational technical debt overhead for the organization. So I think that’s where really thinking about onboarding and having a set of choices upfront is tremendously valuable.

James Governor: At the risk of getting a bit philosophical, I think we get a little bit confused in modern life about the nature of freedom. And so it’ll be like, “Oh, what is freedom?” To me, I think freedom’s a bit about responsibility and working within constraints. I love freedom, but I also know that if I don’t, say, ensure that I have food on the table for my kids, then that’s not good. I mean, it would be… Well, I was going to say it would be great if I could go to the pub, but obviously with lockdown I can’t do that. But there are certain responsibilities that you have. I think my other one is, “Oh, great. I’m completely free. I don’t have to wear a mask. I don’t have to get a vaccine.” I mean, that’s a spurious notion of freedom to me. Freedom is about working within constraints.

James Governor: And also just freedom… I mean, I hate supermarkets. It’s sort of my cross to bear. I mean, at the moment obviously I don’t go to them. But you go in and there is this massive choice. And I think humans are more effective when there’s a choice architecture and when they’re like, “Oh, here’s the way we do things, and I have responsibilities, and if I meet these responsibilities, then I and the organizational community that I live in will become more effective.”

Quintessence Anx: Yeah. Kind of going a little bit deeper with choice architecture, something that comes up a lot with the tooling choices is, you choose tools, or at least I would choose tools, based on a combination of needs and familiarity, because the more familiar I and my team are with the tool, the less to onboard with the tool specifically. So if I already know one pipelining tool, and most of my team knows the same pipelining tool, why would we choose a different one unless there’s a reason, unless there’s another reason to do so? And when that happens within teams… Let’s say there’s more than one dev team, or they’re separate with role specialization, and now you have the tool explosion. When I’m kind of interpreting with choice architecture is to kind of put some guardrails on that so that you don’t have five teams choosing five different tools to accomplish the same task when they could be trying to work more in harmony, would you say?

James Governor: Yeah, I think that’s really fair. I mean, I’ve, kind of been thinking about this in terms of the developer experience gap, which is, “Here are all these choices you can make. Now glue them together, either as an individual or a small team,” that there’s this sort of a glue code or a new code. So one is, “Oh, we need glue code to glom all of this stuff together,” and that’s the world that we live in of baling wire and chewing gum and duct tape and bash scripts and YAML. And it’s like, “SSH into this so that we can integrate with that.” And it ends up being a mixture of like glue code and humans bashing away at things. Or there’s kind of, I think, new code, is “Here is a slightly more opinionated platform. We’re going to get more productivity out of this. Can we as an organization, agree to use this new thing? Because it’s going to give us more productivity and perhaps we’ll have a bit less technical debt.”

James Governor: And so I think there’s a bit of a transformation there that happens that we’re seeing now actually in the industry from tools and glue code… And that will always be a thing. Integration will always be a thing, no matter how much automation you have. Integration will always be a thing. But the new code world of, “Here’s a platform…” And this gets back to our discussion of tools driving culture change. I should have probably said platforms driving culture change. So, “Here is a platform that’s going to make us more effective, that has some guard rails,” as you said, “that is going to enable us to be more productive.” And I think that’s one of the reasons we’re seeing some MNA. And from a PagerDuty perspective, you’re kind of doing both. Like, “Use our tools,” but of course you’re using all of these other tools. You have to do a mix of glue code and new code.

James Governor: But certainly I think from an organizational perspective rather than that of an individual, it’s quite nice to think, “Let’s buy the platform that will enable us to scale the change beyond one team, two teams, three teams, five teams, and into the 20, 30, 100, whatever number of teams we have.”

Quintessence Anx: Yeah. And when you were talking about glue code a new code, it kind of reminded me of another conversation we have a lot, which is build or buy, which is kind of a similar conversation, almost the same but not quite the same.

James Governor: Yeah, related.

Quintessence Anx: Especially in our industry, she says for no reason whatsoever, people like to build things on their own. So you might have someone who just as an exercise… When you’re more junior, you might build things like chat apps or things to understand how to build the server and the messaging and all the underlying components. But then you get that habit where you’re like, “Let me build my own monitoring system and such.” Some teams… I’m not going to disparage anyone from doing stuff. They have the engineering head count to do it. And some teams it’s a pressure, because once you build out version one, you’re now maintaining it forever. And I’m kind of curious to hear your thoughts on this kind of, I’ll say, the internal tug of war, build or buy. Should I build an in-house solution? Should I have somebody else maintaining this? And my trade-offs when I make these decisions.

James Governor: Yeah. I think that’s such a great question, and it sort of works in lots of different ways. So for example, certainly in an age of open source, where there are all of these… Like I said, it’s a golden age. There are things that I can go and I can bring them together, that I can glue them together from a composite perspective to create something great. And it might be because I’m interested in Kubernetes, and then I’ve got these other things, and I want to do Prometheus and Grafana and bring those together, and some Elastic for the logging side. And you end up basically creating, in many cases, technical debt, which…

James Governor: I’m not saying it is always bad to have that kind of freedom. But I think open source has shown us repeatedly that people may begin with the idea that they want a vanilla open source stack that they can adopt, but very quickly they realize in fact that they want it to be managed. So that’s one of the reasons why the cloud has been… And we have this interesting interplay at the moment between open source technology, which was the previously greatest software distribution mechanism ever known, into the cloud now, where everyone has an Amazon account and they can choose services that run there. But basically, the industry as a whole is repeatedly choosing managed services.

James Governor: And in any of these areas, yeah, you go in to customers and they built their own incident management systems, or they built their own feature flagging systems, or whatever it is. It’s a bit like offices. As 2020 showed us, does the office work for us, or do we work for the office? And I think that happens with technology. Do we work for this piece of DevOps infrastructure, or does it work for us? And it should be working for us. So I think that the organizations… Of course it’s great to build, and that is always going to happen. But arguably, identifying what it is that you’re providing the value on, you kind of outsource that by buying the SaaS solution, or the managed service, or indeed… I mentioned open source. It might just be a distro or…

James Governor: If you look at what we’re saying in on-prem Kubernetes at the moment, you’ve got in Google with Anthos. You’ve got Red Hat and OpenShift. You’ve got VMware with Tanzu. They’ve taken these open source components and packaged them in a way that makes running and managing these systems more effective, because it doesn’t… It’s very few organizations to be managing a Kubernetes platform. I mean, I joke about this. In the cloud data world, at the moment we’ve, over-rotated a bit, so that we’ve now got a chief Kubernetes officer. And it’s like, “What’s your job?” “Oh, it’s managing Kubernetes infrastructure.” And it’s like, “Well, why are you doing that?” “Oh, it’s because microservices.” And you end up in a circle, kind of cat chasing its own tail, where you’re like, “Well, why are you doing that again?”

James Governor: And yeah, we build and rebuild, and some of it makes sense, and some of it doesn’t. Developers love to… They’re always like, “Oh, I’m going to build my own blogging engine.” And you’re like, “Why don’t you just write the blog?” And they’re like, “Oh, I need to learn all of this new stuff about static site generation and…” But yeah, whether the org wants to maintain that, I don’t know. We build a lot of stuff. It’s always interesting. I tweeted recently. I said, “The modern tech industry is basically folks just endlessly remaking remakes of Heroku.” It really struck a chord. So I got like a thousand likes on this throw-away comment. But a guy called Dan Young came back, and he said, “This observation seems like a joke, but the question it prompts is deadly serious. If building Herokus meets more needs than actually using Herokus, what does that tell us about this industry?” And I think that’s really interesting, that we sometimes focus too much on building and maintaining things that actually we should be adopting something else for.

Quintessence Anx: And kind of relevantly to that, because sometimes you’re adopting the best tool for the job, and sometimes it’s a cost conversation. So you have a budget. You may not be able to afford. Or you might find yourself in a situation where we need five tools, we can afford four tools, but now management wants us to homebrew a fifth tool. I guess, how would you empower developers to better navigate these conversations with the people who actually have the yay/nay purse strings power, so that they can get the tools that make the sense for them to have and minimize how much legwork they have to do for homegrown solutions?

James Governor: Yeah, I think that’s a great question. I think Spotify provides some really interesting answers for this. So at Spotify, everything in their head is in terms of the full-time employee, which at Spotify is a full-time engineer, like an engineering resource. So any decisions they make about technology adoption, tech debt, anything else, it’s always in terms of, “Well, the organization could have five more software engineers if we did this thing.”

James Governor: They’ve built a platform called Backstage, which is like a developer portal for building developer portals. It’s basically, at Spotify, when a developer, an engineer starts their daily work, it’s an interface where they go and they see what they’re doing, and so it’s also where they will kick off requests for VMs or containers or whatever it is, or do pipeline management, get monitoring. It’s all in there. But they’ve also included now a functionality called cost insights, where the developer themselves sees the implications of their decisions in terms of cost.

James Governor: And it’s all in terms of engineers. So if you go, “Hey, we should totally automate this thing. We’ve been doing it by hand. This makes no sense. Let’s automate this.” And then you say to yourself, “Well, it’s going to take us five engineers to do this, and it’s going to take us two weeks,” Okay, then you begin to get a sense of headcount. “Okay, what’s that going to earn us?” Whereas if it’s like, “It’s going to take us 20 engineers and six months in order that we can save 15 engineers' time…” So they are really focused on this, developers thinking in the context of people and humans.

James Governor: Now Spotify, it’s also really interesting because it’s not even within individual orgs. For them, it’s just about supporting Spotify itself to hire more engineers to work on more problems. It’s not like an OKR where if we make the right decision, we get this other benefit. But I think that framework, really thinking about our people and the work they’re doing, if we can automate away something that is a manual task or something that is just quite time-consuming, what’s the potential value of that?

James Governor: And what I like about that, the discussion at Spotify, is it becomes their bean counters… I don’t think they officially call them bean counters at Spotify, but there are financial teams, cost optimization teams and engineers, they want the engineers to take responsibility for an FTE conversation. And I think that is a framework for almost any IT org where they’re really thinking about, “Are we freeing up IT resource by investing in this infrastructure?” And for them it might be build it or it might be buy it. But if we can outsource this thing, then let’s get more productive. I think it’s just a nice framework, thinking… And literally the interface shows you number of engineers rather than number of dollars. And I think that’s a profoundly interesting way of thinking about it, to be honest.

Quintessence Anx: That sounds like it makes a lot of sense. Thank you so much for all of that. And to our listeners, thank you for listening to part one of our interview with James Governor. Tune in for our next episode, in which we will cover part two, which we will be talking more on this same topic.

Quintessence Anx: That does it for another installment of Page It to the Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pageittothelimit.com, and you can reach us on Twitter at pageit2thelimit using the number 2. That’s pageit2thelimit with the number 2. Let us know what you think of the show. Thank you so much for joining us. Remember, uneventful days are beautiful days.