DevSecOps and Security With Brad Lhotsky

Transcript

Quintessence Anx: Welcome to Page it to the Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve both system reliability and the lives of the people supporting those systems. I’m your host Quintessence or Quintessence Anx on Twitter. Today, we’re going to be discussing the DevSecOps cultural transformation, both where we’re at and where we want to be. We’re joined today by our guest, Brad Lhotsky, a system and security administrator at Craigslist. Brad has been working in security since 1999 and has worked with PCI, DSS, FISMA, HIPAA, SOX and GDPR compliance programs, as well as being active in the observability in pro communities. Thank you for joining us and welcome to the show.

Brad Lhotsky: Thank you for having me. It’s an honor to be here.

Quintessence Anx: Awesome. So to get started, can you tell us a little bit about your perspective on the DevSecOps cultural transformation,

Brad Lhotsky: Doing security for a long time. Back in 1999, I had one of my personal servers routed by some script kiddies three times, and I kind of wound up accidentally being involved in security in every organization since then. What I’ve seen is security going from not existing in organizations at all to just being a guy who happens to know things about security, who does the whole security thing, to have being seen as maybe like a hall monitor where we just tell people what they can’t do. There’s organizations also tend to view security as a cost center and where we’re just they’re paying people to sit around and tell their other employees not to do things. Things progressed in security became almost invisible unless something went wrong and then an attack happens or there’s some ransomware or something like that. And then everyone’s panicking and suddenly security becomes visible, but the rest of the year security doesn’t exist.

Brad Lhotsky: And that was really the best we could hope for quite some time. Recently, we started to see security or teams get seats at the table during design discussions around products, services, processes, even policies, and things like that in organizations. And it’s exciting to see the DevSecOps stuff where security professionals are becoming more involved in the actual development of some of these products and processes. They’re not just there for the design and then disappear. My hope is we’ll get to a point where security teams can actually be seen as a value add to an organization rather than being seen as a cost center.

Quintessence Anx: I mean, I hope for that too, because I think especially with the increase in incidents and things that we see at least on the non-security size of things that are happening, it’s becoming more and more obvious that proactive is the best way to move forward. And relevant to how we’re kind of integrating and moving forward, what are some myths or misconceptions you’ve encountered as people try and integrate security into their team flows and practices.

Brad Lhotsky: One of the first things that strikes me as kind of a myth or misconception is this kind of approach when organizations get serious about security and then all of a sudden they start looking at all these best practices in the industry, right. And there’s a ton of best practice lists you can pull up. And they go, “You know what, we’re going to do security better than anyone. So let’s do all of the best practices.” And there’s just a lack of connection between what those best practices are designed for and the organization and the business itself.

Brad Lhotsky: And so you wind up in a situation where… The most common example I give people is password policies where you have short lived passwords with high complexity rules that store a number of passwords for a long period of time. And you can’t reuse passwords in a long period of time. And you have password lockouts, so you can only get three tries and then you’re locked out of your account and you’re locked out for 30 minutes. I actually saw a researcher writing his password in Sharpie on his laptop because he could never remember it because it changed so frequently. So he just had passwords and he would cross them out and rewrite them. So when you get into those kinds of situations where you’re applying all of the security best practices without understanding the impact that it’s having on the organization and on the people. That could be counterproductive to say the least

Quintessence Anx: Those who can’t see me, I’m visibly cringing at the thought of someone writing their password with Sharpie on their laptop.

Brad Lhotsky: Yeah. That was special. Not just the post-it note. Because he said he tried post-it notes and they fell off. So he started writing in Sharpie on the laptop. The other misconception that I’ve seen a lot is that technical solutions are always the answer. And I’ve seen this on pretty much every technical team that I’ve been on. Is like we have a human, we have a problem with process. Let’s solve it with more technology. A lot of times the first step to solving those problems is having conversations with people that are involved with stakeholders, with users in your organization, with users outside of your organization that are using your products, just having some sort of good communication channels rather than just going, okay, well they’re doing this. So let’s add in a new rule to our firewall or add a new rule to our intrusion prevention system to prevent this from happening. Usually that conversation will spark something that adds value to the company.

Quintessence Anx: Gotcha. And that makes sense. So kind of tying it into how things are working at these companies, especially since most are not yet successfully culturally transformed, we’ll say, what is your experience for how these teams and by these teams, I mean, security, InfoSec teams, as well as development operations, et cetera, how they’re working together now, how the gears are entirely meshing and how that’s moving to kind of explain why we’re looking at this transformation in the first place.

Brad Lhotsky: Sure. I think one of the bigger failings that we have is, and I think DevSecOps is addressing this is that security has always been a separate piece of the organization. I mentioned during the transformation that security eventually gets a seat at the table and they’re involved in design discussions, but then they disappear and they only get called in again when something breaks. And a lot of times their input is they have no way to guarantee that their input is being actually placed into the products and the processes and the services.

Brad Lhotsky: And that can be dangerous because if you end up inviting security professionals into a design discussion and you leave wit agreeing that this is a good design, you both walk away to your separate areas. And meanwhile development decides, well, we can’t do part B of this because we don’t have support for it in the organization. Or we don’t have the technology that fits in that hole. And then it gets to operations and operations decides, well that doesn’t really fit the way we work. So we’re going to re-engineer some of this stuff. Meanwhile, security is designing their defensive strategy against that design. So by the time you end up shipping something, the reality doesn’t match the expectations that the security team had.

Brad Lhotsky: So the security team has built potentially defenses and insight based on those designs. And now you don’t have that in reality. So that assurance of that safety, that the security team was able to give to the development and operations team is no longer there. And the insurance of the design is no longer present for the security team. So you wind up with potentially a false sense of security where you think everything is going according to plan and you’re monitoring the right things. But it turns out that there’s nothing going across that particular event stream because somebody ditched that idea of sending that particular event that would notify the security team of a potential let’s say password stuffing attack or something.

Quintessence Anx: Yeah. And I did wanna, just because I think, especially those of our listeners who aren’t in InfoSec, maybe aren’t aware of what goes into a defensive strategy. And you mentioned that security really relies on their expectation to be roughly met, right, so that they can build this defensive strategy. Can you just talk a little bit about what that actually means. Building that defensive strategy, what does that mean to kind of context that?

Brad Lhotsky: An example might be, let’s say you’re designing something underneath GDPR and you need to protect personal information. And so security might sit down with you and say, “Okay, let’s build this personal information API where we’ll send you back a token so you can store it anywhere in the organization. And that token will give you the key to be able to unlock the personal data. But we’re going to keep all the personal data over here in this protected database.” And then what might happen is for, let’s say convenience sake, maybe that API isn’t moving as fast as development would like. So they are fearing a performance impact to the users.

Brad Lhotsky: So they implement a caching layer for that personal information inside of the, let’s say untrusted zone where they’re cashing that personal data and they don’t send any TTLs or they set long TTLs that aren’t in compliance with the actual GDPR policy that the organization has agreed on. You get into that situation where you have multiple sets of the data. And one side is very protected and is following the rules. But if that company were to be audited in that second caching layer were to be discovered, it could potentially open up the organization to a legal issue. That’s just one example of where if design doesn’t match with reality, and there’s not that communication between the two pieces of the organization, you can actually be subjecting the business to more risk.

Quintessence Anx: Thank you. And that makes a lot of sense. And we kind of talked about, or you just talked about it a bit when you were talking about the design being left out of the conversation. But let’s talk about a little bit about the points of friction that can happen. So you mentioned one where, when the design falls out of sync, but what are some other points of friction that happens between security and the development operations teams, when things fall out of sync like this?

Brad Lhotsky: From the security perspective, as an InfoSec professional inside of an organization, some things that have been issues are primarily communication-based. When security has an issue that needs to be addressed quickly, it’s very difficult to get the right people to take that thing seriously. There tends to be a little bit of knowledge about security in the general population. Let’s just say technical people, sleeve management out of it, right. So there’s this generally… Most web developers are aware of cross-site scripting and cross-site request forgery and SQL injections and things like that. But security professionals actually that’s their currency in which they operate. And so I’ve been in conversations with people where I’ve said, “Hey, look, there is a cross-site request forgery attack on this login page. We need to address it.” And the technical person goes that’s just an academic thing. It’s all theoretical. You can do that, but it’s got no actual security value to fix this because it’s just a theoretical attack. And then it becomes a security professionals responsibility to demonstrate to that individual or to that organization that this is something that needs to be taken seriously.

Brad Lhotsky: Again, that can be very difficult in an organization where security is seen as a cost center, because in those types of situations, security doesn’t have the ability to say, “Look, we have years of experience, and we’re saying this is a big deal and it needs to be fixed. You should listen to us.” Because on the other side you have a developer and now the developer is seen as an asset to the company, they’re adding value, they’re creating product. And the developer is saying, “No, it’s just theoretical thing. It’s one of those play attacks. These guys they’re losing their mind.” So then would become the security practitioners jobs to have to actually take their time out of their day to build out the proof of concept exploit and demonstrate that exploit successfully to the organization in some way. And that takes away time from actually solving the problem.

Brad Lhotsky: And that can be tricky, especially if you have complex apps, complex services where a security professional can really point out, “Hey, this is a problem. I know this is a problem.” But patching that problem, patching that bug, patching the cross-site request forgery, they may not know how to do that inside of your code base with the assumptions that you have baked into your front end and where you’re learning JavaScript from and where you expect how the front end actually works.

Brad Lhotsky: I know I was certainly overwhelmed with that in previous positions, like being able to demonstrate, “Hey, this is not good.” And at the same time, not knowing how to fix it because of a complex code base that had a lot of assumptions baked in. That can be incredibly frustrating when you’re attempting as a security professional to do what you’re being paid for and say, “Hey, this is a problem here.” And people are like, “Man, I don’t listen to him. It’s not a problem.” Or the organization just turns a blind eye to it.

Brad Lhotsky: So that’s probably the biggest source of friction I think as a security professional I faced in my career is just being able to say hey, this is a problem without sounding like I’m crying wolf. And then understanding the sense of responsibility going on my side of table for raising that red flag and saying, “Hey, something needs to change.” Because there is a cost to stopping development and addressing a cross-site request forgery. You’re pulling how many devs off of their projects to fix this problem. So you can’t just wave that flag any time you run a scan and you find a bug because not all those bugs are actually actionable.

Brad Lhotsky: So that friction between like how does security demonstrate that they’re not going to cry wolf and at the same time when there is a problem, and when you first start down this road, you may find a lot of problems. So it makes sound like the security organization is crying Wolf, but really there’s a lot of technical debt that hasn’t been addressed. And it’ll take some time and there will be a lot of findings until you can get to kind of a more sane pattern with your security problems, hopefully.

Quintessence Anx: Right. And you did touch on two things that I wanted to ask about. And one was when you were talking about cross-site scripting being more theoretical than applied, at least in perspective, right? Someone’s perception rather. Are there ideas that you have for people outside of security to improve their security posture? I know when I did some reading, I came across like capture the flag games and such, but the idea that I’m kind of asking from is for people who are not InfoSec to become not knowledgeable enough to do damage, but knowledgeable enough to have a conversation with you or with their security teams.

Brad Lhotsky: Yeah. I mean, there’s a lot of really great InfoSec resources out there. If you search for InfoSec, you’ll find a bunch of stuff. I’ve been doing this for a long time. And so I’m a fan of following Bruce Schneier, Richard Bejtlich. Are two of the guys that I followed pretty closely early on in my career. And they deal with thinking more in a security mindset. And that’s one of the things that when I ran the security team and we had devs and CIS admins come in with no experience. So one of the things I wanted to get them into was looking at the security mindset.

Brad Lhotsky: So not necessarily learning how to weaponize an XSS or cross-site request forgery, but more understanding that the way that you approach your product as a developer is very specific and very efficient. You’re looking at your project and your product is you have a series of tasks that you need to accomplish for the customer to achieve a goal. And you’re inside of that box the whole time. And the world is much larger than that box. And that’s good box that the security professional has to look at. Which is not only how did the developer decide to utilize the series of web forms to get to the form submission process, but what didn’t they think of? And that’s the unknown unknowns to most developers.

Brad Lhotsky: So I would look at developing that security mindset. Bruce Schneier has a book. He’s got a few books on this topic, and there’s a lot of really interesting coursework and talks in this area. In there, there was a talk that was done at Black Hat some years ago about an InfoSec program in the US Navy, where the final exam was to write out PI to a certain number of digits, something that would be completely for most people, unable to be done from memory. And that was the final exam. And that the way that the whole course was designed was to get people into that mindset of defeating all these basic assumptions in your environment so that you can sneak in that cheat sheet that has PI to a hundred digits so that you can ace the final exam. And that’s the kind of thing that you need to kind of exercise and develop as more of that state of mind, more so than necessarily a practical attack or something like that.

Quintessence Anx: And related a little bit to the other thing I wanted to ask you about where you talked about security being perceived as a cost center, instead of a value or a value add or something that improves the overall, I’ll even say wealth of the company, right, because people have users will have more trust, right? So I’m on to know your thoughts on how to start to shift that perspective of security so that they’re viewed as doing the value that they are, right, instead of as a cost.

Brad Lhotsky: So this may be different for other people, but I can always speak from my experience. And unfortunately I’ve found that that onus falls entirely on the security professionals to prove their value. And so the best way to do that is to become more active in the design process in these new products, new services, if you can get involved in sprints and sit down and had a user story meeting or something like that. And in the agile process to hear what people are asking about, just to kind of find where the problems are. And identify sources of friction for your developers, for your engineers, for your management, for your customers, and try to solve them. And solve them in a secure way so that when somebody has a problem and is stuck on something, instead of just going off on their own, they have this thing in the back of their head going well, Bob, from InfoSec, the last time we came across a problem we didn’t want to solve, he came up with a solution and we really liked it. Let’s reach out to him.

Brad Lhotsky: And as the organization, if you’re looking to kind of change this behavior, the concept of a security sentinel or a security advocate where someone from the security team will go and spend time on the dev and the ops teams. And I think it’s also as important to have some of those ops and dev people rotate into the security team and spend some time there learning kind of security mindset, because it’s not something that you can learn inside of a sprint or inside a scrum. You can’t really get into that out of the box thinking when you’re looking to produce a minimum viable product or to ship your bug fixes or ship new features, you don’t always have time to kind of step back and think about those larger things.

Brad Lhotsky: So rotating into a security team can be a really good way for a developer to go, “Okay, let me spend a little bit of time here. Maybe spend a week or two in the security team just learning about security in general, and then take a week of just like taking what I’ve learned and applying that to the last feature that I shipped and seeing where I can improve it.” And then when that person goes back into their team they’ll, they’ll have a, hopefully a merge request or pull requests for that last product they ship that shows this new concept that they’ve come up with this new secure coding pattern that they’ve come up with they can share with their team members.

Brad Lhotsky: And there might be really interesting conversation if they have completely separate from the security organization about the way that that developer attacked it, or learned how to use cross-site scripting to inject code from an outside network and then modify cookies or something. So it’s kind of fun. I think we all have that heist. Is the heist and the big theft kind of things are kind of fun. So when you these types of concepts, they tend to be infectious. So having the rotation not just out of security, but into security, I think is a really good way for organizations breed a better relationship between security and the rest of the organization.

Note: This is where we begin the discussion on mental health in industry.

Quintessence Anx: That’s awesome. We’re going to do a quick pivot here because we want to make some time and space to have a conversation about burnout in the InfoSec community. So I will actually give you the space to level-set on your concerns that you wanted to address.

Brad Lhotsky: That was one of the things that I wanted to make sure I use my voice to talk about was the fact that if you look at burnout rates and mental health disorders up to, and including suicide, they’re abnormally higher in the InfoSec community than in most of the rest of IT. And a lot of that has to do with that old mindset of security as a cost center, of security professionals just being viewed as the people that always say no, of us being invisible until there’s a big problem. Because during that invisible time, security professionals are probably freaking out because they know what’s coming. They’ve identified problems, but because they’re invisible to the organization and they’ve been on the soapbox talking about how they need to solve these problems. I think the organization’s generally tend to almost grow immune to security descent to things that they’re doing or when security does say something, it’s again, going back to crying wolf. And so it falls on deaf ears.

Brad Lhotsky: And so the security professionals are aware of some of the attack services that the businesses choosing to actively ignore. So they’re already sweating and then something does happen. And a lot of times, at least in my experience, the conversation turns back to, I’ve had C-level executives say to me, “We paid you to protect us from this, why did this happen? We’re paying you to do this, who should we fire?” And you look back at the emails you have in your sent folder of all of these problems just screaming into the void asking for someone to help and no one did. That can take a toll. It can manifest as in burnout, if you don’t have good support structure around you and you spiral in those cases. There have been a lot of cases of suicide or self harm in the InfoSec community.

Brad Lhotsky: And so what I want to say is if you find yourself as an InfoSec professional in a position where you feel anxious all the time, that’s not normal. And you should not get used to that. And you should not wear that as your red badge of courage that you deal well with stress, and you’re always anxious. And that’s just your state where you’re highly focused. It manifested for me in literal tunnel vision. Even when I wasn’t working, when I was driving my car, I noticed something was off and that’s something was, I remember driving when I was younger and being able to see everything on the road. During that stress, I couldn’t see anything except for what was directly in front of me. So there were literal physical manifestations of flight mentality that we’re permeating through my whole life.

Brad Lhotsky: And it made me not a nice person to talk to. I wasn’t good company. And so I think it’s important to identify that in yourself. And if you’re a team lead or a manager to identify that in your colleagues and be there for them. Just say, “Hey, what do you need?” “Take time off if you need to take time off, step away from things. You are not your job. You are not the product of your work. You are yourself.” And if you do feel that constant feeling of anxiety, reach out to a mental health professional. I did, and it changed my life for the better. And I wish I would’ve done that sooner. It probably would’ve saved me a lot of pain and suffering.

Brad Lhotsky: I wanted to make sure we acknowledge that in this conversation because a lot of organizations are starting to get into the place where they want to incorporate security more in a positive way into the organization. But I do know that there’s a lot of organizations where that’s not the case. And there are people that are probably suffering right now in InfoSec that are just feel like that’s part of what their job description is. And they’re shouldering that all by themselves. And I thought it was my job responsibility. I thought I needed to do that. And I was wrong. It was very expensive mistake for me not to reach out to someone sooner.

Quintessence Anx: Thank you for sharing that. And just to echo a little bit, if you find yourself in a team or a company that you’re starting to manifest some of the anxieties, symptoms of self harm and suicide, and the things that he’s discussing right now, and please make sure that you do reach out to someone, reach out to your support system. Sometimes reaching directly out to therapy as a first jump is really big leap to make. So reach out to your support system and your friends and let people know that you really need a little bit of extra help. Ans to those support systems, make sure you’re looking out for your friends, especially if you know, they’re in high stress positions, even if they’re not voicing that they’re struggling right now because they could later, and it will open the door for them to reach out to you when things may be start to go awry.

Quintessence Anx: And I just wanted to thank you again, Brad, for sharing that with us. And just to close out, if you wanted to look at Brad’s personal website that has his book and his materials and his blog with his thoughts, it’s edgeofsanity.net. We also have in our resources the DevSecOps guide that he contributed to with us, and thank you so much for that. And that will be in the notes as well. We’re also going to list the information for some of the authors and things that he mentioned so that you know where to look for them. And this will be a huge tone shift, and I acknowledge that, but there are two questions that we always ask at the end of every episode. And are you ready for that?

Brad Lhotsky: Sure thing.

Quintessence Anx: Okay. What is one thing you wish you had known sooner about DevSecOps or security?

Brad Lhotsky: I think the one thing that I wish I would known sooner is that it’s really simple to have a good security posture in your organization. And probably the one simplest thing you do is keep your software up to date.

Quintessence Anx: Awesome. And is there anything about DevSecOps or security you’re glad that we did not ask you about today?

Brad Lhotsky: I really can’t think of anything. No.

Quintessence Anx: Yes. Awesome. All right, well, thank you again for joining us, Brad.

Brad Lhotsky: Thank you for having me. It was a lot of fun.

Quintessence Anx: Awesome. And this is Quintessence wishing you an eventful day. That does it for another installment of Page it to the Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pageittothelimit.com and you can reach us on Twitter at pageit2thelimit using the number two. That’s pageit2thelimit with the number two. Let us know what you think of the show. Thank you so much for joining us. Remember uneventful days are beautiful days.