How to Start a DevSecOps Program With Franklin Mosley

Transcript

Quintessence Anx: Welcome to Page It To the Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve both system reliability, and the lives of the people supporting those systems. I’m your host Quintessence, or @QuintessenceAnx on Twitter. Today we will be talking about how to start a DevSecOps program and measure its success. We are joined today by our guest, Franklin Mosley, engineering manager of product security here at Pager Duty. Franklin has worked in InfoSec and software development for 20 years, and brings a wealth of experience toward organization. He is an advocate for security and has appeared as a keynote speaker, sharing his insight into the world of security and dev ops. Thank you for joining us today and welcome to the show.

Franklin Mosley: Thank you for having me. Pleasure to be here to speak with you today. You and I go back a few years. It was our first time actually doing something together and I’m happy to be on the podcast.

Quintessence Anx: Awesome. And I look forward to working with you too. This is great. I’ve been looking forward to this all month. Not going to lie. What can you tell us about DevSecOps program?

Franklin Mosley: DevSecOps programs, it’s kind of all the rage now. I’ve been out here speaking about DevSecOps for a few years, and I get a lot of questions about how to start a program, or what is DevSecOps? I kind of have this love/hate relationship with the term DevSecOps, because it kind of implies that security is not… shouldn’t be in DevOps. And it should, and it should inherently have been in there from the beginning, but we know it wasn’t. It was something that was kind of an afterthought that was left out, and now people are starting to think more about how do we make sure that these builds that we’re putting out there at this rapid pace, have security built in. The term DevSecOps is kind of more this explicit, kind of, let’s make sure that security is included.

Franklin Mosley: To me, DevSecOps is really just automated security decisions at speed and at scale. How do we make sure that security is not that roadblock that traditionally was. As you said in the introduction, I’ve been in security for 20 years. And I can say that, we wore a roadblock, we wore “the organization of no”, and we did stop down a lot of things. And I could see why in early days of DevOps, we were circumvented and not included because we were seen as something that would slow down the pace of pushing software into production. Starting up a program, security needs to recognize that you need to work together with your developers, with your Ops team, and figure out what it is that they’re doing. And some of the tools and processes they have place, and figure out how security can best integrate into that, and not take our old methods and ways of doing things, and try to fit them into the DevOps world.

Franklin Mosley: I think one of the things that we can do as security people, when we think about how we can do that, is to think like engineers, and not how we used to think as just security people and taking our checklist, and saying, “Oh, this is what we’d normally do, it’s what we did in the past.” We will now take this same process and fit it in place. But think about new ways of how we can still make sure that we’re managing our risks, and doing that in a way that’s frictionless or make it easy as possible to integrate security in.

Quintessence Anx: That’s awesome. Speaking of these disconnects, on the… I guess I’m going to say on either side, I was going to say, on one side, but you know what, on either side, what are some myths or misconceptions that you’ve seen either from non-security engineering or security to engineering, if that made sense?

Franklin Mosley: Some of the myths and misconceptions. I think it goes back to who was responsible for security. It’s not solely on your security organization or your security team, but security should be a shared responsibility. And I think we’ve been hearing a lot of that over the last couple of years or so. And I think more and more teams are starting to recognize that, and believe in it actually, because they’re starting to take on that full service ownership model. We’ve seen the shift where these development teams, have gone from just owning the development itself, but also owning the deployment pipeline, the performance of their application, the reliability of it. It’s no longer just coded, handed off to someone else, and get it deployed. But, basically taking ownership from the idea portion, all the way through creation, and then through the deployment and maintaining the services while they’re in production.

Franklin Mosley: Security is just another piece of that, that also gets thrown into that mix of ownership. And then that helps to kind of build upon that idea that security is just solely the responsibility of security organizations, security team, but it’s a shared responsibility, and teams are starting to pick up on that. I think another misconception, another myth, and this is from the security side, looking back at the developers, is that the developers don’t care about security. And that’s a misconception. There was a study that was done, and I forgot where this report came from, but I’ve seen it in several different places. I’ve actually quoted it in some of my talks, that it shows that approximately 50% of developers recognize the importance of security. It’s just they don’t have time to work on it. And it’s because they’re being tasked with so many different things. And that security is not one of the higher priorities that are on their list, and they just don’t have time to work on it, but they recognize it is an important thing.

Franklin Mosley: And one of the things that we can do to help it be recognized as a higher priority, is that security should be seen as another feature, and it’s not just some afterthought, but if you prioritize it as you’re planning out your work, as a feature, then it gets kind of the recognition, and it gets your development teams working on it, that you actually seek that would make an overall better quality product. There’s just a couple of things that I think are myths and misconceptions that we’re starting to address now, when it comes to DevSecOps.

Quintessence Anx: Something you said about, the myth that Devs don’t care. I worked in Ops primarily before I switched into advocacy work. But, even from the Ops side, I remember really appreciating what security was doing and also not entirely understanding what was happening over there. I understood in the way that they were able to make the information succinct, like, “We need to do X, Y, Z because of the…” But I didn’t actually understand mechanics of what was happening or any of the more granular. I don’t know why the decisions were being made the way that they were. Which I thought was super interesting. And I know that we’re going to talk a little bit today about starting a Dev SecOps program. And if this is related to anything that we talked about previously, about some of the things that can be tied in, there are ways that can help those of us sitting outside security really understand, and be involved with that.

Franklin Mosley: I guess your original question was, how do we start a program?

Quintessence Anx: Yeah.

Franklin Mosley: Starting a program, like I mentioned originally in my statement, was security recognizing what the DevOps teams are doing, and figuring out the best way to work with them, and to get a better understanding. I think like with any relationship, you need to understand your partner, and what their needs are, what their wants are. And it happens from both sides. Not just security, understanding the Dev scenes, Ops scenes, but DevOps understanding the needs of security. Essentially the role of security, when you really break it down, is about managing risk. And our goal is to not just stop builds just because we can, but we’re just trying to figure out the best way to manage risk within the workflows that are established by these DevOps teams.

Franklin Mosley: And then for us as security, we need to recognize that the DevOps teams are trying to develop, quickly and rapidly develop software, and release it as soon as possible to get these features into the hands of customers to create value for the customers. How can we make sure that we manage risk and recognize and identify any vulnerabilities as early as possible, as quickly as possible, while still being able to get the software developed and deployed into the hands of customers, so the customers can get the value from the product that we’re offering? Now that we recognized what the needs are from both sides, we can work together. And I also think when it comes to starting a DevSecOps program, it’s helpful to get the buy-in from leaders of the various organizations. Because I also think that a lot of that comes, and it starts from the top.

Franklin Mosley: And I really feel like a top down approach actually can help to get that buy-in from the different teams as they come together to try to start working on DevSecOps program. I actually worked at a company that that buy-in came from actually the CEO. As a CEO starts to talk about how this security was a shared responsibility, because he was talking about the strategic vision for the company for that year. And he was talking about the four different pillars that would make the company successful. And one of the pillars that he mentioned was security. At that point, he level set and let everyone know that security is a shared responsibility. At that point it was this trickle down effect.

Franklin Mosley: Now when I’m coming to one of the development teams, and I’m asking them to work with me on helping to identify how we can make the deployments, the development process more secure, I already have the CEO telling them that security is a shared responsibility. Something they need to recognize as a feature that we’re offering to our customers. Getting that buy-in from the top is one way to help start that program.

Quintessence Anx: And actually, related to that, I know from speaking with others, sometimes getting that buy-in, isn’t easy for whatever has happened, or whatever organizational or personal reasons that the people at the upper echelons of management have. When people… When security specialists, when Dev teams that want to support their security specialists, find themselves in a situation where they need to get a buy-in, and it’s not a straightforward process, what advice do you have for things that they can say that can really help to start that conversation successfully?

Franklin Mosley: When it comes to how to get the leaders to buy in, how to make the business case for security, for starting a DevSecOps program? I think one of the easiest ways is to point to the data breaches that are occurring out there on a regular basis, with a bunch of different companies. And by being able to point to that, and how businesses are affected from the outcome of a data breach, where you’ve have a loss of customer confidence in the company, or in the product itself, or there’s lost revenue that occurs, then that kind of strikes at the heart of what business leaders are looking at, what the stakeholders are looking at. And that’s the dollars and cents, or the bottom of it, of the business.

Franklin Mosley: With these publicly disclosed data breaches, now these things are bringing visibility into the boardrooms, and the members of the board are now asking the CEO, the C-level executives, how are we positioned to handle this situation? How are we positioned to ensure that we’re reducing our risks to make sure that we don’t end up on the front page itself for a data breach. That’s bringing security to the front of mind, of the executive team. And then it makes it easier for the development teams, the security teams, to be able to go to their leadership, and say, “Hey, we want to start this DevSecOps program.” because security is now something that’s become more top of mind for your executives.

Quintessence Anx: Yeah, I can see that. One thing I’ve been thinking about while you were speaking, is something that would come out sometimes on the Ops side, not about security, but in a parallel concept, where sometimes it’s hard to upsell what could potentially not be working, when things are working. Data breaches that never happened because of security posture, or other types of outages in the gap side, where we’re like, “Please give us these very expensive tools for monitoring and logging and whatever else we’re asking for.” And it’s one of those things, where at least for the Ops side, sometimes you would have a hard time, because you care most about those things, aside from the past reporting, when there’s a problem. And then the rest of the time, they’re just kind of chugging along. And I’m curious if you get the same kind of mindset with some of the security topics too, where it’s working when it isn’t, but there’s panic when there’s not. And if there’s a way to bridge where it’s still… Where the value isn’t always just about avoiding panic, I guess.

Franklin Mosley: Yeah. I get what you’re saying. No, it’s the same thing on the security side. And I think the best way to make the case is to, if you’re able to actually quantify the risk, and translate that risk into dollars and cents, then I think that really speaks volumes. If you could say, “Okay, we recognize that there’s a risk here. We don’t check for secrets being put into our application.” And if someone is able to discover one of these secrets, and the secret then leads to access to our cloud account. And then they have access to the cloud account. And we haven’t put in proper access controls. That particular secret leads to access to our database. And all of our customer data is in there. And we have a million records of customer data. And each record we value at, I don’t know, X amount of dollars per record. Then you can say that, “Okay, this is the bottom line of what that data breach would cost us if we don’t implement a program to recognize as risk and to help mitigate it or reduce it to an acceptable level.”

Quintessence Anx: Okay. That actually makes sense because then you’re translating what the potential, what you’re trying to protect, to something that’s very tangible. And speaking of measurement, when we’re talking about the types of measurements that are going into the… Not just the risk assessments for the conversation we just had, but just more broadly about the programs and things, how are we measuring what success means, or what things we should even be looking at?

Franklin Mosley: Yeah, there’s a couple of different things there. One of the things that I like to tell others about is, measuring the maturity of your program. Of course in the very beginning, you’re just starting out, there are some ways that you can assess your program to get a baseline of the maturity of it. There may be some practices you’re currently doing, but not under the guise of a DevSecOps program, but they are activities that can help lead to the successful building of a DevSecOps program. There’s several different maturity models out there that could assist you in actually getting a baseline measurement of the program. And then over time, as you start to implement more processes or tools, measure how your program is effectively growing and maturing. That’s one part of measuring your program, measuring how it’s maturing over time.

Franklin Mosley: I’ll tell you, there’s a few different maturity models that exist out there. There’s BCM building security in maturity model, which is more descriptive in nature. And it’s descriptive, meaning that it’s telling you the activities that participating companies are engaged in that have been successful in application security, in somewhat, in DevSecOps as well. But then there’s also a maturity model, like the one provided by OWASP. OWASP is the open web application security project, and it’s a non-profit organization that focuses on security best practices for web applications, and also helping identify common weaknesses that exist within web applications.

Franklin Mosley: And there’s a bunch of different projects, but the OWASP Sam project is a flagship project that focuses on the software assurance maturity model. And it gives a bunch of prescriptive activities that you should be engaging in to help build your application security program, and DevSecOps program. And even more specifically, there’s a DevOps software assurance maturity model as well. But these models give you a way to kind of take a baseline measurement of your program as you’re starting it. And then as you start to do more and more activities, and integrate them into your DevSecOps program, then you can see how your program was maturing. And then specifically, how you measure its effectiveness, is identifying what are some key performance indicators that you can obtain from your activities.

Quintessence Anx: That makes sense. What you mentioned before about the maturity models, and the frameworks and such, how would someone, when they’re taking a look at these, because it seems like from what I’ve seen of them, they’re very comprehensive. You’re probably not going to roll all that out at once if you’re just getting started. How do you start to identify what the minimally viable is, versus next steps, and grow outward in a way that is going to be comprehensive, so you don’t have too many gaps early on?

Franklin Mosley: Within these maturity models, there are a lot of activities that they prescribe or will describe depending on which model you’re using. And all these activities aren’t necessarily something you should aim to achieve. You shouldn’t aim to achieve the highest maturity rating across all these activities, but your program should be catered to what makes sense for your particular environment. And it may be that a maturity level to across these various activities is what makes most sense to you. But again, you have to identify your particular environment and activities actually fit. One time we did an assessment… This was at a previous company I was at. And during that assessment we recognized that certain activities didn’t really fit our cloud transformation project and the way we were rolling out DevOps. And we dropped those activities from being part of our maturity assessment and things that we achieved to get to… What we strive to get to because it just didn’t really fit our environment.

Quintessence Anx: And when you were making those decisions, it also… not the ones that entirely didn’t fit, but is there a way to do kind of a… I’m trying to find another way to say cost benefits, because it’s not exactly… I guess it’s more risk versus benefit, where you’re trying to assess, “Okay, I know I’m making this decision today for these reasons, but I know, or I need to know how to evaluate that it’s opening me up to these other things.” And I was wondering if you could talk a little bit about how, how do you start to explore that?

Franklin Mosley: I’m glad you said that, because that’s the point of the maturity assessment, is to identify the gaps in your DevSecOps program. In identifying those gaps, you want to recognize, what are the risks that exist if we don’t address this gap. And am I able to identify the risk. And if you’re able to come up with a measurement of that risk, hopefully in a quantifiable way, then you can start to prioritize the things that really matter. And they need to be addressed before other activities, or other gaps that you identified that maybe impose less of a risk, and then you can leave to later on down the road. But then also when you’re addressing those risks and those gaps, you can also look at what is the cost of implementing those. Cost of time, of labor, of tools that you may acquire to help address those gaps. And then making that trade off will help you make some decisions on what is needed now, and what’s needed in the future, or what’s not needed at all.

Quintessence Anx: And that makes sense. I guess I feel like some might have some anxiety about making sure that they’re doing so correctly. Is there any thing, or any advice that you can give for people for how they can course correct, as they’re moving along and they’re like, “Oh, this was more risky than we initially assessed or believed,” for whatever reason, and how they can kind of patch over that, and move forward.

Franklin Mosley: Just like we talked about DevOps, and how DevOps is constantly getting feedback and iterating, you have that whole feedback loop. It should be the same thing with your DevSecOps program. As you’re making changes, constantly measuring, getting that feedback, and then feeding that back into the program, as you start to grow it. Identifying those key performance indicators that will make sense. That provide the measurements, or the metrics that will show you positive or negatively, whether there’s an impact to your DevOps teams, as far as their workflows. And also if you’re actually affecting the risk that exists in various areas of your DevOps program as you’re integrating security in. When it comes to what are those key metrics, it kind of varies based on your program, because I believe in this bottoms up approach, where you take the data elements that you have, and then figure out the relationship between those data elements. And then by figuring out the relationship between the data elements that you have, you’re able to identify what are some key performance indicators that make sense, and actually tell the story of your DevSecOps program.

Franklin Mosley: An example could be, if you’re integrating some checks in your pull requests, for example, or some data elements that you have that may exist, or that you have, would be a number of repos that you’re checking. A number of pull requests that have been submitted and that you’re running your security checks. And all those pull requests, number of critical, high, medium, and low vulnerabilities that you’ve identified. And then if you’re giving feedback to your developers through those pull requests, “Hey, we’ve recognized that there’s this high vulnerability that exists from this particular dependency that you’ve introduced within this pull request, did that vulnerability get fixed in that pull requests?”

Franklin Mosley: And then you start to put all those data elements together and recognize, “Okay, this is a key performance indicator that we can then use to measure, and then report out.” You may come up with an average mediation time, per the development team, as a key performance indicator. And then you can use that to measure how effective is my program. We recognize the vulnerability, and got it fixed within X number of days. And that’s an improvement over what we used to have, where it was maybe twice as long. We’ve reduced average time remediation by 50%.

Quintessence Anx: Wow. Okay. And that makes sense. It makes sense that you kind of need to track, there’s going to be the so-called best practices, and give you a starting point. Be like, measure these things, but you’re going to need to start tracking and base-lining, and then looking around those things to see if there’s anything that suits the way that things are actually working at your organization that give you more or less visibility data, et cetera, to see exactly what you’re describing, and whether or not things are being resolved in a way, or could be improved. That’s awesome.

Quintessence Anx: And thank you so much for all of that. And before we go to our next part, just going to say, thank you so much for your time. And we’re going to be linking to the frameworks that Franklin’s been talking about in our materials, as well as the DevSecOps guide, which I actually interviewed him for when I was writing. There’s tons of good thoughts and information about all of that. But before we head out, we have two questions that we like to ask every guest, are you ready?

Franklin Mosley: I’m ready.

Quintessence Anx: All right. What is one thing you wish you could have known sooner about security or DevSecOps?

Franklin Mosley: I’ll say… I’ll give you something about my entry into DevSecOps. I was doing AppSec, and I got asked to join a team. And it was really… I got embedded with this DevOps core team as they were starting to build out their DevOps program, and they were migrating all their development teams into DevOps. And they were doing this whole cloud transformation. I’m thinking, “Okay, cool. I’ll just take all the stuff I’m doing from an AppSec perspective. And I’ll throw it into DevOps.” And that didn’t work.

Quintessence Anx: Oh no.

Franklin Mosley: That did not work. One of the initial mistakes I made was I tried to take this DAS tool. A DAS tool is a dynamic application security testing tool. And essentially it treats the web application as like this black box. And it throws all these tests at it. These requests, which are just various security tests. And it looks at the responses that come back from the web application, and determine whether or not that test was successful. Looking for things like cross-site scripting, potential SQL injection, et cetera, et cetera. A DAS scan takes… Can take a long time, because what it’s doing is spidering or basically checking every page of the web application, and every input at fines, and then submits a request of that input field. And like I said, it checks the response.

Franklin Mosley: A DAS scan can take anywhere from 30 minutes to several hours. You think back to how DevOps, we were trying to make these changes quickly, and get them pushed out into production. A scan taking several hours, and holding up a deployment, does not work. I didn’t realize that in my initial foray into DevSecOps, that I can’t take all my old practices, and just neatly fit them in, into DevOps. I need to be more careful and more thoughtful about the things I was doing. That goes back to what I was saying earlier, about taking on an engineer’s mindset, and kind of shifting from just being a security person, to a security engineer, and thinking about tools and practices of DevOps, and how to best work within those.

Quintessence Anx: That’s awesome. I mean not awesome about the eight hour scan. That sounds like someone, a whole bunch of someone’s had a bad day. Speaking of, is there anything about DevSecOps, or security, you’re glad we did not ask you about.

Franklin Mosley: What tools we’re using.

Quintessence Anx: Very relevantly, what tools. Thank you so much for joining us Franklin.

Franklin Mosley: Thank you for having me.

Quintessence Anx: Absolutely. This is Quintessence wishing you an uneventful day.

Quintessence Anx: That does it for another installment of Paid it to the Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast, if you like what you’ve heard. You can find our show notes at paidittothelimit.com, and you can reach us on Twitter at pageit2thelimit using the number two. That’s pageit2thelimit, with the number two. Let us know what you think of the show. Thank you so much for joining us. Remember, uneventful days are beautiful days.