DevSecOps for Development and Operations With Patrick Debois

Transcript

Quintessence Anx: Welcome to Page It to the Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve both system and liability and the lives of the people supporting those systems.

Quintessence Anx: I’m your host, Quintessence, or QuintessenceAnx on Twitter. Today, we’ll be talking about DevSecOps from the perspective of element and operations. We are joined by our guest, Patrick Debois, currently the director of market strategy at Snyk and author of the DevOps handbook. Thank you for joining us today and welcome to the show.

Patrick Debois: I love it. It’s a pleasure to be here. Long time PagerDuty fan, so glad to be here.

Quintessence Anx: Awesome. And before we get started, can you tell us a little bit about your experience with DevSecOps and security?

Patrick Debois: So my experience with DevOps is a little bit different from most of the people because I was 10 years ago already researching and exploring it with other people. My journey to DevSecOps, I came to it more because I saw it having attention and having a similar problem set that I saw 10 years ago with DevOps. It made me an interesting thing to see, is there a pattern repeating, but just with different players in the organization that want to come along? And that’s how I got interested. And then also joined Snyk, and then from a security perspective, I could learn how security now has an issue where in the past the ops had the issue, which we still have.

Patrick Debois: But we tend to believe we fixed it, and now security comes along. So that’s a little bit my journey, although I have followed a few of the security courses and certificates in the past, like the SANS and the Firewall, I used to run all that stuff too. But I haven’t been in a security silo in my life where the whole purpose of the team was security. I haven’t done that. So it was always there as a team within the operational sides and on the coding side to help the operational side, but for me personally, not on the security side only.

Quintessence Anx: Gotcha. And while you’ve been going through all this with the new perspectives you’ve been getting, what are some myths and misconceptions you’ve encountered?

Patrick Debois: It’s interesting that the narrative gets repeated. We want to believe that it’s culture, but then we talk all about, “Here’s tools to get you bootstrapped.” So it’s not a myth, it’s just a repeating pattern. And then the other one is, well, DevSecOps doesn’t work for us, it’s the same rejection that DevOps has had over the past. Like, “It’s not for us,” and then all of a sudden after 10 years, everybody from the big bang to the startup is doing it. DevSecOps will eventually get a similar thing. The resistance and the pain, yeah, we were blockers, security, yes, ops was blocker. We can’t stop the developer. And this whole narrative.

Patrick Debois: It is interesting in a way that I see a difference between companies having done a DevOps journey and then going on a DevSecOps journey. And there’s others who use the DevSecOps term as an excuse to actually start their DevOps journey. So it’s not always that you already have the DevOps practices in place when you think about DevSecOps. If security is your main concern, it might have been that your DevOps automation and collaboration hasn’t been there yet, but the pressure point has now come on the security side in those companies. So it’s a little bit of a variation of the team.

Quintessence Anx: So out of curiosity, in that case, it sounds like these are companies that didn’t yet have a DevOps transformation that now are going to do immediately a DevSecOps transformation. Is that what you’re saying?

Patrick Debois: Correct, yeah. And so they were discovering both the SDLC automation metrics and the security at the same time, which is a real fire hose for them at that time.

Quintessence Anx: I was going to ask if there’s any difference between, I’m sure there is, but if there’s any difference between those who staged it and did Dev then Ops then DevOps then DevSecOps versus people who jumped immediately into DevSecOps, if it’s more painful or less.

Patrick Debois: Well, they seem to be going similar parts in a way that I always talk about this, that DevOps was a bottleneck, but it’s not the only bottleneck that can exist in a company. It could be maybe not in your pipeline or delivery, it could be in your hiring process, it could be anywhere else. And maybe they didn’t feel it in the delivery side, but the companies felt it really hard in the security side. But they will probably benefit from all the automation and the metrics and the monitoring replaced to do a better job at DevSecOps. But then again, I sometimes make the joke, you don’t need all the automation to do DevOps.

Patrick Debois: In a way that if the collaboration is good and you automate it just enough, because that’s the danger of DevOps that we’ve been over automating, and it’s become a thing on itself to keep automating and keep automating and losing sight of the costs and the benefits trade off, then security can be done without certain tooling as well. And John Allspaw puts it best, in an example, if you ask an engineer, “If you’re away for 50 minutes will your system still be working?” It was like, “Yeah, of course.” “If you go away a day, will it still be working?” “Probably, let’s hope so.” If you go away for a month, “Oh yeah, I don’t know if it can survive a month without me.” And the point is that it shows you that the humans are still an important factor, whatever we’re doing at a tooling. And I also see that happening, if the automation fails, sometimes people don’t know what to do.

Quintessence Anx: It’s true.

Patrick Debois: And then that means that the person who has built the automation, they understood what it was doing, much like programming, you write the code and it makes sense while you’re doing this, but then somebody else looks at this and is like, “Why were we doing this? Remind me again?” And it’s like going back and forward. But my main point is that there’s a lot you can do without putting all the tools and all the tool spree, is just thinking about these things in a collaborative way. And that’s still, for me, the most important part of DevOps. If you look at the CAMS acronym, culture, automation, measurement, and sharing, I sometimes say, C, the culture is collaboration. And I think that part is still for me the most valid. It’s not the easiest though, so that’s why you might go to automation because we don’t have to talk to other people.

Quintessence Anx: True, true. And tying in those concepts, so we have a secure delivery pipeline, let’s say. What we have, probably the security folks or Sec ops that know why things need to be implemented, somebody else who might be actually building it. So you have a division of skill and labor. What are some common gaps and what are some good ways to bridge them that you see when people are trying to build a secure pipeline?

Patrick Debois: Coming back to the parallel of DevOps, in the early days, a lot was about making the pain visible. And so we started getting a little bit better dashboard to management tools of showing when systems went down and what happened at a certain point. So that happened. And make it a little bit more tangible. In security, usually what I see is that the security folks, they hint to the developers, for example, or the ops people of what might be an issue. Or they say, “What is an issue?” So the first thing is broadcast what is the pain, we’re broadcasting to the other person. And then maybe it’s the nagging, maybe it’s the interests that the other group says like, “Hey, maybe we should do something about this. There’s something going on, we don’t understand that. And what’s going here and what we can do to make this better.” They got annoyed about the frustrations, about the bills being blocked, about the delivery being blocked, and they get this conversational thing going.

Patrick Debois: And then the next phase is usually security zooms in maybe on the biggest pain, similar to ops, like, “Where’s your pain? Is it version control? Is it the script? Is it the cloud?” We try to find our bottleneck that brings us the most value and is an easy win. And sometimes that’s library scanning, sometimes that’s configuration, sometimes that’s the top CVE list or OSP training that the developers need. And you start picking the easy ones and feed that into potentially a team that becomes aware of this, that might act on this. And then you hope they pick it up. There’s certainly another threat, which was very similar again to ops, which is the expedite threat.

Patrick Debois: “We have something happening in production, we need this fixed now, please do this, drop everything else.” Well, we didn’t have very good experiences with that model being forced on us. But still, it’s not a way of, at least if you have too many incidents, if you have too many problems, it’s an indicator that something is going on. The force model is not the preferred one, but if your company is not yet savvy enough or it’s just scratching the surface of security off of ops, sometimes it makes sense for somebody more knowledgeable to step in and just say, “Go left, go right, do this. I don’t expect you to understand everything, but this bootstrapping is where you need to go.” That’s the start of the journey, there’s probably a lot more off there, but this initial ping pong you know SYN-ACK SYN-ACK going on between the two groups that makes sense there.

Quintessence Anx: And for the developers, there’s a lot that would need to be done because you’re going to be asking them to do at least certain things that are going to be a little bit outside of their areas of expertise, different parts in the pipeline or things that they just need to do and submit and whatever. So what are the best ways to get them the most knowledgeable or in the good enough place to be able to do these things in an efficient way and from a place of understanding?

Patrick Debois: From what I’ve seen, people have tried different ways. Much like Agile, security teams have tried to find a champion in a team that has just an interest to that, and pride proxy hopes that the team picks this up and spreads the love because of the closer connection with that person. Another trend that I see, if you show developers how easy things get hacked, and that’s more again of an awareness thing, then it’s like, “Oh, I didn’t know that. Oh yeah, okay, I’m sorry my code did this.” At least if you’re proud of your code, they will do better. If not, you’ll have a different problem. So that’s opening the eyes of what can be done, how easy it can be done to do that. And then there’s another trend, which is when your team reaches a certain maturity level of also operating the software itself. So they truly go in this hybrid mode of dev and ops, and the same team is, you build it, you run it. They start to feel the pain when things go south in production.

Patrick Debois: So that gets them like, “Oh, we need to do this. We’re proud that we have the uptime, we want this to be running.” And those are kind of like three trends that I see influencing the motivation. Is it enough? I can’t tell, but it’s definitely three trends that I see happening over and over again in teams looking at security. And there’s obviously always somebody who’s like, “I want to go completely immersed in security, give everything to me.” But what’s interesting is that the developer teams are getting a lot of tasks. They have to operate things now, they have to secure things, then they have to think about the finance, and then you have to talk to the business. And that puts a lot of pressure on the team. If you look at it from the perspective that a developer should do everything, if you just look at if your team is balanced between multiple more focused people working together, somebody is focusing on the ops, somebody on the security, somebody on the developer, then it’s more like collaboration within the team.

Patrick Debois: And then that burden of cognitive load gets spread, but it isn’t overspread. And each of them learns, the security people often have to learn what it is to deploy an application, to make it, to do the coding, to do the feature, to be in the rat race of delivering features and not just saying no. So everybody has to learn this little bit of, we often say it, empathy, between the different pain points we have. But it doesn’t mean we all have to do it at the same time and we all have to have the knowledge. But it helps, the more affinity you have, the more you understand, obviously the more you can prevent. But I sometimes pity the role of the developers these days on the pressure the companies are putting on them.

Quintessence Anx: Yeah. It does seem like in that case, they’re caught in the middle. And going around to the empathy, I know that one of the things that we like to say is it can help to have security own a product or a service that’s in production. That makes sense, so vault or equivalent of something like that, that is security relevant that they can run so that they can say, “Oh, this is what the development ops teams are roughly doing, or this is how I talk to them.” And with that mindset, what are some things to increase awareness, I know we talked about training a little bit, to really increase awareness in a way that doesn’t make them feel like they’re going to have to become security experts? Just again, increasing, what do I know, and how can I apply it?

Patrick Debois: I know this is the lamest ever, but it all boils down to trust.

Quintessence Anx: Oh, okay.

Patrick Debois: The book called Team Topologies, there’s a website called the DevOps Topologies that talk about different ways to organize your security or operational side, whether it’s embedded in the team, whether it’s outside the team, whether it’s a hybrid mode, sometimes in, sometimes out. I started taking the belief that that pattern, it could show you that you have a problem, but the pattern itself isn’t the problem. If I trust my security team completely, they could be in a different group, I don’t care.

Quintessence Anx: Yeah.

Patrick Debois: As long as they help me all the time, they give me feedback that way, it doesn’t matter too much, but that trust must be earned by them. And I often say it’s quite similar about DevOps. Operations was having a bad rep at always saying no, but in essence, nothing was stopping them from collaborating. So either it wasn’t that they were in a different silo that was actually holding them back, it was more of a mentality shift on being more customer-centric, being more focused on what others want and what you want yourself. I know people have been deploying the concept of platform teams where it’s an abstraction layer in a way that you help the other teams, but people often put it hierarchical. They think about the technical stack, building on top of that.

Patrick Debois: But I think the more interesting model, if you just start seeing them as other teams that are doing the security feature or are doing the build feature, and then it becomes more not being hierarchical, it’s just a collaboration between different teams. And whether you’re writing the authentication code for your software or something else, you’re going to focus. Obviously there’re pros and cons on microservices and teams focusing because you can’t escape complexity, it comes back because then nobody understands the whole system anymore. And then there’s a team caring about the whole system and trying to make everybody collaborate across the system. But that’s the way of how I see this collaboration improve.

Quintessence Anx: And that got me thinking a little bit about wrapping into… I know we’re focusing on understanding, but compliance as well. So when you’re implementing things, sometimes it’s, “Oh, this is quote unquote the ‘best way’ to solve this problem. But sometimes you need to make other decisions because you have certain compliance requirements, big names that people recognize, GDPR, HIPAA, et cetera.” So when we’re talking about compliance requirements, how aware of compliance requirements do Devs need to be versus how much is going to remain in the wheelhouse of security in terms of awareness and specialization?

Patrick Debois: Yeah. So I don’t think this will land completely in the developer, or let’s say the team realm. But similar to, we talked about across skills, across operational security and so on. If they understand enough why they do something, then that’s fine. They don’t need to be doing the whole audit and understand all the benchmarks in the industry and compare them. But if they trust the other group and they find like, “Oh, you need this, how can I help you?” And maybe the relationship is 5%, 95% security still, but the interface on the code is maybe where the translation is happening between what the world of compliance needs and what the developer needs. Shifting the whole thing in is again, we sometimes think about the full stack or the 100 percent stack, then we were getting the 400 stack developer. I don’t know where this is going to end. It won’t end well, so I can tell you. What’s interesting, and it was Mark Burgisette who made me aware of this, a lot of it in DevSecOps, we hope that the collaboration becomes voluntary between the groups.

Patrick Debois: It’s not always the case, sometimes we want to avoid forcing things on other groups. And in promise theory, it talks about agents. And agents are making promises. But the idea of a promise is that a promise is allowed to fail. And it’s similar to the thinking about uptime, we have to build systems that stay up forever. No, no, we assume failure, it can fail. And it’s the same thing with this collaboration thing. We want to get everybody to do better and so on, but we already have to assume that it will fail and maybe put some mitigations in place, and that’s probably the first layer of defense that we can put there. And maybe the lost resort of defense is just throw money at it, in case we can’t do it, or we buy it or something. But when you mentioned audit saying, “This is the best way you do things,” there’s one part that I think you left out. “This is the best way we think you should do things, the way we understand your context.” And that is a difference because a lot of these can’t see your context in your company.

Patrick Debois: And I don’t want to argue that every company is special snowflake and do this, but there’s always history, there’s different people, there’re different things happening. There’re reasons why some things get more attention than the others. I remember my first firewall audit and I was so annoyed with this, like you said, there’s too many ports open. It was like, “That is no argument. These are open for the business. And if you don’t understand why we’re opening these up, then don’t audit me as a checkbox with the number of open ports.” And the same thing is with the audit. I’m not saying about public companies that have to do their best and so on. And it is not best practice, good practices, but good practices have context where they are good, and maybe some contexts where they aren’t good. And that negotiation is also a big part of the security job is understanding, translating why we’re not doing this, putting the exceptions, showing what was good. We learned that we can free up some of that time through automation if part of the discovery, what’s out there, how it’s being done, we can just present that.

Patrick Debois: So we don’t have to collect those things in spreadsheets anymore, which get outdated all the time. But the conversation is still the most important one. It is similar to, if you look at these audits like an SLA, you need to comply by X time or something. I think when I was young, I saw SLAs as there’s no way around it or we’re going to get hit with a penalty, a big size or anything like that. I think what I learned from the field is that these are good aspirations, and I don’t mean that you shouldn’t go there, but quite often the thing is that if you’re showing that you’re improving, it solves a lot of the discussion points. And you just have to show that you’re doing well and acknowledge where you are not good and put mitigations in place. And that is quite often how I’ve seen SLA disputes being solved. It’s just, can you show me you’re doing better? And the last resort is we’re going to send you a big invoice for a penalty lost, but usually there’s humans that make it a little bit human.

Quintessence Anx: That’s awesome. Thank you so much for all of that. And for everyone else, we’re going to have links to some of the things that we’ve been discussing today in the show notes, so make sure you check those out. But before we head out, there are two things we like to ask every guest, are you ready?

Patrick Debois: Go for it.

Quintessence Anx: What’s one thing you wish you had known sooner about DevSecOps or security?

Patrick Debois: Okay. How much time do I have? It’s probably also the answer to that question, but no, I wish I would have experienced it myself. I don’t know if that’s the answer, but to understand somebody’s pain, it’s best to have lived through it.

Quintessence Anx: And that makes absolute sense. And what is one thing that you’re glad we did not ask you about?

Patrick Debois: The spelling of DevOps.

Quintessence Anx: That’s an easy way to get into a fight, awesome. Thank you so much for joining us.

Patrick Debois: You’re welcome.

Quintessence Anx: And this is Quintessence wishing you an uneventful day.

Quintessence Anx: That does it for another installment of Page It to the Limit. We’d like to thank our sponsor Pager Duty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pagelimit.com and you can reach us on Twitter at pageit2thelimit using the number two. That’s pageit2thelimit with the number two. Let us know what you think of the show. Thank you so much for joining us, and remember, uneventful days are beautiful days.