Threat Modeling With Gene Gotimer

Transcript

Mandi Walls: Welcome to Page It to The Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve the system reliability and the lives of the people supporting those systems. I’m your host, Mandi Walls. Find me at LNXCHK on Twitter. Welcome back folks. This week I have with me a special guest. He’s the other DevOps Gene, Gene Gotimer is with us this week, and we’re going to talk about threat modeling. Gene, welcome to the show.

Gene Gotimer: Thank you for having me, Mandi.

Mandi Walls: Tell us about yourself. Tell us about what you do and whatever else you want us to know.

Gene Gotimer: So I am generally a federal contractor doing development, and lately DevOps, DevSecOps more and more over the past couple years. But almost always working for either really big organizations or I’m outside of DC so government work is plentiful, let’s say.

Mandi Walls: Absolutely.

Gene Gotimer: Yeah. Most of the time it’s all about bringing DevSecOps or more security or more quality into their environment. They already have some plans, they already have some needs, but they’re just not getting the results they want or anything like that. So, I try to help them out, build pipelines and add tools, add security, add automation, anything to help them move smoother and get to where they want to go quicker.

Mandi Walls: Awesome. So you had given a talk at DevOpsDays Baltimore that was amusing but also very informative, it was about threat modeling. And the best part of the talk doesn’t translate to podcasts because your slides were all castles, and then you kind of went through how to think about threat modeling with castles, and I thought it was great. It was a hilarious little way to put the things together and it was very engaging for the rest of us in the audience. So let’s start there. We’ll start with threat modeling as a practice. Why is it something you thought about giving a talk at a DevOps conference about? What’s your approach to threat modeling and that kind of stuff?

Gene Gotimer: Yeah. Well, threat modeling is one of those things that we always say we should be doing, and then we never quite do it. Because if you do a formal threat model to sit down and get everybody involved and really map out all your pieces, it just seems daunting. It’s a lot. But the reality is that it is something we always do without thinking about it. I gave the example during the talk of crossing the street. You do all sorts of threat modeling in there, it’s a natural instinct. And I think that if you remind people that, hey, you can do some lightweight threat modeling, you can do back in the napkin type threat modeling, just let’s start talking about the pieces we need to, let’s start talking about the things that could go wrong and how we’ll fix it or remediate it or mitigate it. It’s really not that big a deal to get the basics down. And one of the things with security in our industry is that the bar is just so low. Most people do so little, at least as developers. I mean, teams do good jobs, companies do good jobs, organizations do good jobs, but individuals or development teams, most of the time I don’t think developers take it seriously. It seems daunting, it seems hard because they don’t feel they have the background, they don’t have the pieces they need. Once you feel like, hey, I’m not in control, you stop taking responsibility for it, it’s someone else’s problem, I kick it over to the security group, they’ll magically fix everything for us. Why do I need to do anything for it? So if we can do something really straightforward, a little bit of threat modeling, just the basics, just, “Hey, what could go wrong? And if it goes wrong, what are we going to do about it?” Or, “How are we going to make sure that we don’t go out of business because we did that one wrong thing?” Then yeah, we’re already jumping ahead of a lot of other people.

Mandi Walls: How do you approach it with folks that you’re working with? Like you say, it’s daunting and thinking about the potentially infinite number of things that can go wrong, it doesn’t mean they will go wrong or even that there’s a probability of T-Rexs appearing into your data center or anything completely insane. But I have worked places where we had barriers in front of the building so no one could drive a truck through them. So there’s a limit. How do you find that place where you’re comfortable talking about that stuff?

Gene Gotimer: The way I usually approach it is just with the basics of security, start getting some of the pieces that the organization already requires us to do and start getting the teams to start paying attention to that. And then once we get a few of those pieces underway, then we start saying, okay, “Hey, let’s talk about this next piece coming up. How could we be a little bit more proactive?” So dev teams already run static analysis tools, they get back feedback. But when they get that feedback, that same type of feedback from a SaaS tool, a static analysis security test tool, they just say, okay, check. I’ve done it. I’ve run the tool. So you start getting them to pay attention to it and say, “Hey, this is an opportunity feedback. This is a way for us to get better. For us to catch problems now and not worry about them later.” And I think right there, that gives people a feel like, “Okay, well now I am actually involved. Really wasn’t that big a deal. I was treating it like a checkbox, now I’m treating it like a feedback loop like so many other things we do in DevOps.” Once you start getting into that habit of applying some security, I guess the mystique of it kind of wears off and you start saying, “Hey, you know what? At least this part of the security, that’s not all that tough. I mean that’s fairly easy.” And I really do think it comes down to there’s just so much and there’s so many other things we have to deal with, and you just get scared by how much there is. But once you start doing it, you realize, “Hey, I can do actually quite a few things. Maybe not even a majority of the things, but I can do quite a few things that’ll get us moving in the right direction.” And I think once DevOps practitioners get in that mindset, or even just agile practitioners get in that mindset of, “Hey, I’m moving in the right direction, doing continually getting a little bit better, making things a little bit better each time we go through, that’s kind of become natural to us.” So I think that that helps us get on the path. And once we have the tools and the scans and we’re starting to implement some of those mitigations, remediations and fixing problems and vulnerabilities, I think then start talking about, “Okay, what we really did is we looked at this problem and said it could have a disastrous effect, critical it’s high or whatever the rating is. So we fix those first. These are minor, so we’ll only fix them if we get around to it.” And what do you know, we’re doing threat modeling.

Mandi Walls: Imagine that. Oh my gosh, just appears, right?

Gene Gotimer: Yeah.

Mandi Walls: So over the past five or six years, I guess, we’ve been talking more about moving more things like shifting left, putting more work earlier in the work stream for software projects, and a lot of the security stuff has kind of been a victim of that. Whether folks like it or not, it needs to be done earlier and earlier. What have you seen in your practice with that sort of stuff?

Gene Gotimer: Again, the security piece is a tough one to move forward. And I was a huge proponent of DevSecOps is a terrible term. It suggests that there’s a version of DevOps without security, I’ve given up the fight. The term has stuck, start realizing that that security needs to be part of it. But I think they still kind of treat it as an outsider. And we start getting people to move that stuff left and to start seeing it earlier. But again, they have to take some action on it. Some places it’s terrible. You get the scans done, it’s like, “Ah, it’s just slowing down my builds. I’m not getting any benefit out of it.” And you’re dragging people kicking and screaming to try to get them to do anything with those other places have been really surprised. A group I was in gave their developers, I think it was just a pilot even of a SaaS scanner that ran right in there, showed the results in their IDs. The idea was to let’s take a look at what we can find, and over the next couple of sprints we’ll start talking about how we’re going to address it. And within a sprint or two, every easy to fix bug had been cleaned up because it showed up in their IDs and they were used to, “Hey, I want to keep my ID clean. Those red squiggly lines, I want to get rid of them.”

Mandi Walls: Oh yeah, of course.

Gene Gotimer: So they started fixing them, and it turns out that in any situation, most of your problems in the code are always going to be just a handful of different types of problems just repeated over and over, and so they started fixing stuff. And I don’t know what the magic sauce is to get teams to behave like that, it’s happened before for me. Other places not. But I think the idea is just to keep telling people, “Hey, some of this stuff is easy. We’ve got 1,000 vulnerabilities according to the scanner. That’s too much to handle. But we’ve got this one problem 400 times and we can literally fix it with cut and paste.”

Mandi Walls: Yeah, absolutely. We see that stuff in other places too. Like, oh, it’s always going to be this package and yeah, there’s a lot of scans. It’s all the same package across the entire fleet, so fix this one thing, please. So yeah, definitely. How do you recommend folks get started if they don’t really know what they’re after? If they’re just kind of like, “Oh, we want to be more secure.” By I’m just like, “What’s that mean, man? What do you really want to do?”

Gene Gotimer: Well, and that’s kind of the point of threat modeling, it’s to figure out what is the most important thing for you guys to do right now, for your team to look at or for you individually to look at. But I would say that for most teams you can just start with, let’s just take the tools we’re already forced to use and let’s start using a feedback loop. Not treating it as this esoteric security tool that we don’t have to pay attention to because the security team’s going to look at the PDF we attach to our paperwork to deploy. That’s not the right way. We need to actually get it as feedback loop and just start looking at what the suggestions are that it makes. Most of the tools now are getting really good about, here’s what I found. Here’s the way to fix it. Even to make things easier most of the problems people find, and most of the problems that companies that have been hacked have found that the mistake they made was not updating software. Keep your stuff current and the tools that you’re using, make sure they do some sort of SCA, software composition analysis. Say you’re using older versions of this software, there’s known vulnerabilities in it, just upgrade place. Them at now has a tool that actually goes through our repos, looks at our package builds, and says, “Hey, these are the things that can be updated. Do you want me to just go ahead and do them?” There’s all sorts of ways to get that automated or to get it close to automated to help you to say, “Hey, just keep current.” Just keep all your libraries, all your tools, just keep updating. Make that a natural part of your process. It’s not a, I need to be reminded once every couple of sprints to do it. Just, “Hey, I’m touching this code. What do I have to update on it or what can be updated on it? Let me get to the latest and greatest.” It has a bunch of benefits aside from the security where you’re fixing bugs and known vulnerabilities, eventually just about every piece of software you use, every library, every component is going to force you at some point to update. There’s a huge problem with it. It won’t work with the new framework you’re picking it. It has a known vulnerability, whatever the case is. If you just get in the habit of, “Hey, I’m going to upgrade from version 1.01 to version 1.02 to version 1.03 every time those things come out. Yeah, I’ve updated 20 times over the last six months.” But, that time when they say, “Okay, now you have to upgrade to 1.1, and by the way, it’s a much bigger update than it was from where you started.” Or I have to go to version two and it’s a huge jump, except that if you’ve been doing all the updates along the way, it’s not near as big an update as it may sound and those things, you just pay it incrementally. So many of the things we talk about trying to do is do it incrementally. Yeah, maybe over the course of a year, it’s actually a little bit more time spent, but you stayed up to date the whole time. You never had the panic of, “Hey, someone says, I need to get these updated and they need to be updated this week. Drop everything. We’re scrapping this sprint. All we’re going to do is fix this one security hole that security just reported to us.” So you get proactive. It makes it a lot easier. A lot less stressful.

Mandi Walls: Definitely. We talk a lot about that kind of hygiene, especially around automation and those kinds of tools just to make sure that stuff is up-to-date all the times. Even with our projects at PagerDuty, we have a lot of upstream dependencies that we’re consuming all kinds of libraries and runtimes and other things to build our own stuff. Keeping all that stuff up to date is, it’s a job on its own, but it has to be done because like you say, you don’t want to get to a point where it’s a Thursday afternoon and there’s a zero day or some crazy business comes out and you’re like, “Oh crap. It’s in all of these packages and now we have to scramble because we haven’t been updating stuff.”

Gene Gotimer: Yeah, that’s exactly it. You become proactive and you get to do it on your terms, on your schedule. If you put it off for a day or two, nobody even notices. Put it off for a month or two, you notice at absolutely the wrong time.

Mandi Walls: Absolutely. I tell people it’s like brushing your teeth. You could get away with not brushing your teeth a day or two, and maybe only the people that live with you are going to kind of notice. But if you haven’t brushed your teeth in a week, and it’s like everybody who sees you is anywhere near you is going to be like, “You haven’t brushed your teeth in a while.” Everybody’s going to notice. Really starts to become part of what you’ve got going on there.

Gene Gotimer: Yes, that’s definitely a good way of thinking of it, putting in perspective.

Mandi Walls: So some of the other tools that have come out in the past couple of years are sort of methodologies and things like that, things like SBOMs, right, that are sort of meant to do what those things. Do you see those as just one more box to check, or are they sort of something that’s actually useful for teams to be working on?

Gene Gotimer: I think if you start building a practice that’s actually paying attention to this stuff, it’s just more information for you, and you get the right tools to help you manage that. Most of the big security tools will handle that automatically for you. But if you don’t have a big budget, if you’re not spending the money on a huge package to do that type of security management and vulnerability management for you, [inaudible] got some stuff out there that will take just, you build an SBOM, you feed it into an open source tool, and it just keeps track, and then you start feeding that from every one of your software builds and one of your deploys. All those SBOMs they just go into what seems like this pit. You’re just sending them in. But then one day that PIT speaks up and says, “Hey, I noticed you have this version of Log4j and there’s a brand new vulnerability out for it, and these are the packages that are using it.” And all of a sudden doing that seemingly pointless exercise of throwing this stuff in, all of a sudden pays off in a big way. I mean, Equifax, that was one of the big failures, was not only they upgrade Struts and got hacked through a vulnerability in an old version of Struts that they had probably as a transit of dependency. Probably wasn’t something they were actively using, it was something else was requiring it. Even once they realized that they didn’t have any idea what systems depended on it was all manually search through source code, see if you can figure out who uses it. I think if you start doing something like SBOM and actually come up with a practice, like I said, it doesn’t take a lot, generate the SBOM, feed it into the server, and maybe it seems like pointless right up until the time when it’s not.

Mandi Walls: Right, and it feels that way. So are your backups, any kind of preventative measure definitely feels like that.

Gene Gotimer: Exactly. It’s definitely a maintenance type task that will pay off at some point or save you from getting into big trouble. The fact of the matter is now if you do any work with the federal government, you’re supposed to be generating SBOMs. Executive Order 14028 says you got to have SBOMs and you should ask for SBOMs for anything else you get. I don’t think we’re at the point where it becomes a natural thing that companies, they’re dealing with each other, just, “You show me your SBOMs I’ll show you mine.” But hopefully we can get to that point where it’s just a normal part of the delivery. You’re delivering the software, here’s the SBOMs. They ingest into their black hole of SBOM database, but then they get told, “Hey, what do you know? They have a vulnerability. Contact them and see if there’s an update for it.”

Mandi Walls: Yeah, definitely. You just don’t know. And some languages are greedier than others about the things that they pull in, and you really just aren’t even sure what’s going to chain into your software unless you’re really, really paying attention.

Gene Gotimer: I’ve been a Java developer for a long time, and I’ve worked with Maven. First time you build something with Maven on a computer it has to download the internet, turns out that there are a lot of other languages out there that are even worse.

Mandi Walls: Yes. Some of them won’t download them all at once. They’ll just decide to pull something down somewhere randomly in the future because the runtime, and you’re like, “Knock it off. What are you even looking at that for? Why do you need that particular weird thing?” So yeah, a lot of strange stuff out there.

Gene Gotimer: I mean, so much of our software is not stuff we write now.

Mandi Walls: No, absolutely.

Gene Gotimer: 70% is from a framework rate, 70% of our code, something like that. It’s probably even north of that now. We had that problem a couple years ago where the left-pad.

Mandi Walls: Oh, left-pad. Oh, left pad. Yes.

Gene Gotimer: One-liner JavaScript code trivial for anybody to write on their own, but tens of thousands of developers decided not to even bother, they just included it.

Mandi Walls: So much chaos [inaudible]-

Gene Gotimer: … Decided to pull it off and all of a sudden half the Internet’s down.

Mandi Walls: That’s right. Oh, left-pad was amazing. Oh my gosh. Absolutely crazy. And that’s one of those weird things that for folks who don’t work in the space, there’s no way to explain that. There’s no real world equivalent of like, “Well, someone just randomly pulled up this road that everybody was using. It just disappeared.” But yeah, it doesn’t happen other places. It’s crazy business. So we have a couple of recurring questions we like to ask folks on the show. Our first one is, what’s a myth or a misconception that you’d like to debunk that folks kind of get wrong about maybe threat modeling, maybe security in general that you come across?

Gene Gotimer: For security in general, that it’s not my problem.

Mandi Walls: There’s a good one.

Gene Gotimer: Security team will take care of it. At best all they’re going to do is yell at you and tell you you did it wrong. I mean, you’re still on your own to fix it. But fact of the matter is security is not that hard to do the very basics. Yes, pen testing, very tough, need to know how to do it. Locking down, building secure enterprises, setting up secure networks, all of those are big tasks. But for a developer getting started with security, again, just pay attention to the feedback. You’re already running a bunch of tools. They’ll tell you some. If you’re not getting enough usable feedback, change the tool, get more tools, ask the internet. Somebody will tell here are some tools that can give you good feedback. I mean, the commercial security tools that are going after developers and DevOps teams, they’re focusing on making the feedback more usable to developers. So they know that that’s the holdup. But I think most people think, again, it’s too daunting, there’s too much, it’s not my problem. I’m not the one responsible for it because I’m not part of the security team. Reality is you are responsible for it because it’s your code. It’s not that hard to get started. Just go ahead and start working on it at any other type of feedback. Ignore the fact that you get told if you don’t indent the right way or you’re not naming your variables using the right format or whatever. Security shouldn’t be treated as something separate, you get a lot of that SaaS feedback, a lot of that static analysis feedback says, “Hey, this could be a SQL injection. Fix it. You should update these packages, fix them.” Do them proactively. Just make it part it. It’s not as daunting as people think. I think that’s the biggest myth.

Mandi Walls: Absolutely. Very cool. And then what’s something that you wish you would’ve known sooner? Is there something you feel like you had to learn the hard way? Maybe it just wasn’t apparent when you first got started on stuff or something that’s emerged in the meantime that folks need to know?

Gene Gotimer: Yeah. Hackers don’t care that it’s just your dev system in the cloud.

Mandi Walls: No, they don’t. Absolutely. Oh my goodness.

Gene Gotimer: They’re just looking for systems. They don’t care if it’s your prod. They don’t care if you’re storing your grandma’s recipes or your country’s nuclear secrets or whatever. To them, it’s just a system. They’re probably going to use it for Bitcoin mining-

Mandi Walls: Exactly. I don’t care about your problem.

Gene Gotimer: … or emailing or something else. But they also don’t care whether you said, “Hey, I’m not going to secure this. It’s just dev.” [inaudible] just dev in the cloud. It’s in the cloud, secure it. And doing those types of things, the types of things you would do in prod to lock down your dev system. Not only is it great practice now that we have infrastructures code it, you get to practice it and develop it and fine tune it there. But you also find a lot of problems that, “Hey, this worked in dev, but now it doesn’t work in prod. Oh, that’s right because we locked down prod, so we don’t allow those types of messages to go through.” Or, “We have it set up differently.” Or, “There’s an extra jump through a firewall.” Or whatever the case is. I wish I’d paid more attention to the make dev as much like prod as you can. I think I’ve learned that now, but I learned it the hard way.

Mandi Walls: That’s a good one to learn. When I talk to people that are really reluctant, I totally understand it, especially if you’ve got a big complex environment. It’s super expensive. You need to put a lot of resources on it. But it’s going to bite you in the butt every time you release something. It’s like, “Oh, now we have to punch a hole you didn’t tell me I needed, and that’s a five day SLA.” So unless you’re going to escalate this to your vice president and they’re going to go talk to the networking vice president, this is not going anywhere. Yeah, definitely.

Gene Gotimer: Yeah. The flip side is those things, they get escalated. The pushing it out is always going to win.

Mandi Walls: Oh, of course. Yeah.

Gene Gotimer: So you’re going to get forced to do this. It doesn’t matter if it’s Friday night when they find this and now you’re up all weekend trying to fix it. You would’ve been better off doing it ahead of time. Again, proactive on your schedule, not on someone else’s schedule. But I think everyone has to learn some of those lessons the hard way.

Mandi Walls: Yeah. Yeah. I kind of think so. Absolutely. So what’s your favorite part of all this stuff? What’s your favorite part of your job?

Gene Gotimer: I really like getting everything put together with automation. You go from a bunch of manual-ish handoffs, developer develops here and then runs over to someone’s desk, says, “Hey, the pull request is in, go ahead and start looking at it.” And then this guy’s going to merge it, and then we’re going to send it to this team and they’re going to do the build for us and all that. Take it from a system like that to someone sits at their desk, they commit it, and everything just starts flowing. The tests happen. Everything’s clean. Even if it doesn’t eventually get pushed out into production right then, it gets to stop there and someone says, “Hey, this is waiting for you now. It’s all set.” As soon as the business says I want to push, we push and it’s a done deal, and it becomes a complete non-issue. I think the most excited I ever was for DevOps deployment. We weren’t even really doing DevOps, it was just CD. We had stuff working, pushing it out. We were going from every six months, one deploy that took us months to get through, and now we were doing every two weeks, and we had the client sitting with us one Monday afternoon. He’d been around all day, we’re talking after lunch, and he said, “Hey, just occurred to me. We had a deployment on Friday. How’d that go? I didn’t hear anything about it.” And we said, “You didn’t hear anything about it. It was perfect.”

Mandi Walls: There you go. That’s what you want.

Gene Gotimer: It was a non-issue. Everything was clean, and that was all the process we’d put in place, it wasn’t even all automated at the time for us, but everything just went so smoothly at that point that it wasn’t even a noticeable thing anymore. And that to me was, hey, we got past a hurdle there that seemed impossible to get to in the past. These deploys would go so smoothly that nobody would even notice they happened.

Mandi Walls: You feel like you’ve made it at that point.

Gene Gotimer: Yes, yes, yes, yes, yes.

Mandi Walls: Awesome. Excellent. As a parting thought for folks, is there any advice you have for folks who want to get more involved in security, security engineering and those kinds of things where they can start to learn about more stuff or pick up a few tips?

Gene Gotimer: Well, the easiest way is just start doing it. For a long time I didn’t do anything with security because it wasn’t my job and somebody else’s job, and I didn’t want to step on toes. There’s almost nothing in our development processes now that people don’t want taken off their plate, you’ll have plenty. So if someone starts doing a little bit of your job for you, take it as a win, not a [inaudible] they’re not stepping on your toes, they’re taking stuff off your plate. Just start looking into it. So pick some small piece. If you have a tool, an SCA tool that’s telling you to do the update, start doing them and then go to security and say, “Hey, look, we’ve started doing this proactively. Is this helping? What’s the next step?” Or, “We have this SaaS tool. It’s doing our static analysis and giving us some security feedback. As a team, we’re picking these top two findings and we’re just going to fix them throughout our code, each sprint. Friday afternoon after we’ve done the sprint review and you’re looking for other stuff to clean up, that’s the technical debt we’re going to focus on.” And just start doing little bits, the easy parts, the parts that you’re not worried about, the parts that you understand already, and then you’ll start finding, hey, you’re getting better and better at it, and you’re learning more and more. And just always be learning. Just always be reading. If it’s a tool that sounds cool, sounds like would help you, do some reading on it, at some point there’ll be a need for it. There’ll be a reason to start putting it in. Or someone will say, “Hey, I wish we had this problem solved.” And you can say, “There’s this open source tool or there’s this commercial tool that does just that. That’s why they have it. Let’s do it.” And all of a sudden you’re the security expert.

Mandi Walls: There you go.

Gene Gotimer: Who actually started doing it.

Mandi Walls: That’s right. And then you can wear your roller blades to work and put your leather and your neon colors on and you’re ready to go. That’s right. That’s how that works. Gene, thank you so much for being on the show. This has been great. This has been super fun. Where can folks find you if they’d like to hear more from you.

Gene Gotimer: On Twitter, I’m OtherDevOpsGene. That’s probably best bet. I’m OtherDevOpsGene on YouTube, on GitHub. That’s the easiest way to find me.

Mandi Walls: Awesome. Fantastic. Well, thank you so much for being on the show. This has been great. And for everyone out there, we’ll wish you an uneventful day. Thanks very much. That does it for another installment of Page It to The Limit. We’d like to thank our sponsor, PagerDuty, for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pageittothelimit.com, and you can reach us on Twitter at Page It to The Limit using the number two. Thank you so much for joining us, and remember uneventful days are beautiful days.