Building Trust in Security Reporting With Breanne Boland

Transcript

Kat Gaines: Welcome to Page It to the Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve both system reliability and the lives of the people supporting those systems. I’m your host Kat Gaines and you can find me on Twitter at strawberryf1eld using the number one for the letter “I.”

Kat Gaines: Hi there folks. Welcome back to Page It To The Limit. This is your host Kat Gaines and today we’re gonna chat a little bit about security in a broad sense. More specifically we’re gonna talk a little bit about alerting. We’re going to talk about some frequency of alerts, folks in businesses who might be uniquely targeted for the information they have. And we have a guest today with us to chat about these things. We have Breanne Boland from Gusto. I’m gonna go ahead and give Breanne just a moment to introduce herself.

Breanne Boland: Yeah, hi all. I’m Breanne. I work at Gusto in product security and a lot of what I do is helping people to build features that are safer for our customers. It lets our engineers focus a little more on being ambitious and doing big cool things and a little less on obsessing on all of the security details because that’s my job.

Kat Gaines: That is an important job. I think all of us would love to obsess less and just get stuff done more at the end of the day and it doesn’t always shake out that way, but that is why we have security teams. So Breanne, just to get us started, we usually talk about just kind of describing our topic for anyone who is maybe new to or unfamiliar with the skill of managing these types of things. So when we had discussed what we wanna talk about for this episode, we had talked a little bit about alerts and how it can actually be a little bit helpful to have maybe sometimes high frequency of alerts and understanding what it looks like to have helpful frequency versus filtering out noise. So do you wanna start just kind of explaining a little bit for folks who might not have had a chance to experience that or experie haven’t had that particular viewpoint come up in how they manage incidents of any kind?

Breanne Boland: I think that it’s actually better for some periods of time to have more alerts. You know, generally it’s really easy to have a sense that if we have fewer alerts, things are doing better. And that’s true. Like if things are doing better, we want there to be fewer alerts. We want people to enjoy the beautiful quiet that happens when you get into that nice meadow between bad things happening. But really I would rather have an honest sense of, you know, I feel like I should look for wood to knock on for this a dozen alerts going on if it accurately reflects what’s going on. There can be an inclination sometimes to have fewer alerts because that makes people who like numbers happy. You know, a lower number on that means like, but we’re doing a good security. We’ve had only one alert in the last month.

And if it means that fraudsters and attackers and you know, just general mistakes are at a lower number, that’s great. I love one alert in the last month, that’s wonderful. But I would rather have a culture where people feel comfortable surfacing things and knowing that they won’t have, you know, a punishment for being honest about things. That to me is the most important part of security culture is teaching people that security won’t be mad at them. I put a lot of work into that, that if someone reports something to me, I say thank you and I keep my voice super even and modulated and I check in with them after going, did that feel okay? Do you feel okay doing this again later? Which partially is just because my goal in this world at this point is I just wanna make life softer for everyone.

‘cause Things are hard enough as they are, but it’s also really security serving in that I want people to have zero hesitation coming to me and going, Hey, I think there might be a problem because that’s what I wanna hear about is like I have a funny feeling not I saw, you know, 1500 logs that just told me something terrible happened. I want them to come to me when they’re like, I just kind of feel a little uneasy. I think something went wrong. That’s when I wanna talk to them. And that does result in more alerts. But if we have more lower sev alerts, amazing outcome, that’s great. Like let’s catch it when it’s a sev four, then everyone has a nice weekend.

Kat Gaines: Ideally that’s the goal. I really like what you touched on there about the human side of it because when you think about it, it is a space where you have potentially a higher saturation of human generated alerts or issues rather than, for example, incidents where you may just have monitoring, picking things up or for a lot of us, our incidents come from automated tooling and security is a space where you do want to empower people to like exactly like you’re saying, feel like something is off, something feels wrong, whether it’s overly seeing an issue outta hand or just having, you know, bad vibes. Right. and I think that’s interesting too because there’s a lot of safety that has to just be kind of explored there. And so I’m curious how, you know, not that there is a playbook for this necessarily, but how you kind of cultivate that. You mentioned one or two things that you do, but can you go into that a little bit deeper?

Breanne Boland: Sure. One of the ways that we do it actually is playbooks, you know, we can’t do it for everything, but there are some things that are just going to happen. Like we want them to happen less. We wanna give education to what happens less, but every, so for instance, someone will commit something that they shouldn’t have, you know, hopefully like not a really important credential, but it’s just normal. Say you’re making a PR and you have attach like 12 files every so, and it happens. And so you have to kind of come at it through two different ways. Like we want to empower you to not do this thing because we don’t want that to happen. But also a very gentle kind of when it happens, this is what to expect. And I find that setting expectations is really important because most developers, you know, I would say you could round up really easily to all of them.

They don’t want to cause an alert. No one likes to cause trouble, no one likes to feel like they did the wrong thing. Like no one’s ever come to me going like, guess what I did? They’re always like, you know, the shoulders are kind of up. They’re like, I think this happened. And yeah, by telling them, you know, for the more predictable ones what to expect, it lets people feel calmer. So I consider it my job to keep people aware of what’s going to happen and to follow through on that and if something changes, I thank them for teaching us an important lesson, you know, like our goal and our job is to not have to learn the expense of lesson twice.

Kat Gaines: Yeah.

Breanne Boland: So devs do key way for security by accident sometimes when we say thank you

Kat Gaines: Yeah. When you talk about that knowing what to expect piece, does that look different for different teams? So for example, if you’re working with development teams, if you have, maybe your sales team might spot security issues or have them reported in from their customers and they might not know where to go. What does that look like in just setting expectations across the org?

Breanne Boland: Yeah, it definitely varies depending on what people know and what they’re used to. You know, it’s one thing to talk to a team that you know knows how to do a really difficult, like pulling something out of the history of Git and it’s something else when it is someone whose job is not to write code or make prs and yet we want them to feel, you know, again super empowered to come talk to us and know that we’re going to ask questions and teach them to let them know that they are the experts of their own domain. They don’t have to know everything. Like I’ve weirdly, I’ve worked in marketing in the past, but I was a really bad marketer so I never presume I know what’s going on on their end either. I ask a lot of questions and part of that is just figuring out exactly what happened because you know, this is a classic help desk thing that people don’t ask the thing they really need to know about.

They ask the question they’ve kind of gotten to and trying to figure out how to get their job done. But the real question that needs answering is so often here it’s the same thing. You know, they’ll come and be like, I saw this and I think it’s bad. Like cool, what were you doing when you saw this? What were you doing before that? How did you get to this? And it’s a mix of, please tell me what’s actually going on. And also reminding them that they have knowledge and power in the situation and I depend on that too.

Kat Gaines: Yeah, that makes perfect sense. And I think the repro step is one of those things that people will sometimes forget about in these moments. We do tend to focus on the symptoms or the report and really understanding like you’re saying, how you got there can uncover what’s actually going on rather than the, I think this is what’s happening, it’s really easy to hypothesize, right? And I think we all do a really good job of telling ourselves stories around what’s going on, especially when we’re elbow deep in it and we’ve been just kind of spinning gears on it for some amount of time. And so having those fresh eyes and that fresh perspective to say, okay, but let’s just back up a minute. How did you even get here? I think that can be really calming and helpful for the person experiencing the thing to realize that no, you don’t have to get caught up in the chaos of what’s going on right now and what you’re freaked out about. We actually wanna just take a step back and kind of slow down a little bit. It doesn’t always have to be a complete panic and that’s a nice reframing too because when people think about these things, they tend to think about security issues with a lot of urgency and it probably brings up panic very easily as an emotion. Right. And I like that you’re kind of diffusing that a bit with this approach.

Breanne Boland: Yeah. I also think a lot of people, if they’ve been in tech for, you know, more than a few years, they’ve probably had a brush with security that they don’t think of super warmly. It’s something we used to ask as part of our onboarding, just, hey, new batch of new colleagues. Tell us about a time that you interacted with security in the past. Like what happened, what came up, what do you think went well and what didn’t, you know, in part to figure out if anyone’s kind of scared of us because it gives us a good chance to just try to put a little energy into that and calm it down. But also just to know like who we’re, you know, who are we dealing with and what are they bringing into this new situation that we should be aware of?

Kat Gaines: Yeah, absolutely. Let’s go back a little bit to our topic of kind of frequent alerts and expected noise. We’re talking a lot about enabling people, empowering people to feel good about talking to security and that does bring up a little bit more noise when you start to notice that things are getting really noisy, you’re having either frequent security incidents or you’re just having people bring up concerns a lot and it’s getting a little unbalanced in terms of how often you’re dealing with that work versus the preventative work you want to be doing. What do you do to manage that? What do you start to shift focus on and how do you work with other people around that?

Breanne Boland: I think it’s a mix of things. One is often if I find things are kind of popping up that are causing issues, you know, especially if it’s coming with a lot of big feelings, I think that it’s a really good idea to get a person in the mix as soon as you can. You know, whether that means just security popping in on different engineering team meetings just to show up and be like, I’m not mad, we’re here to help. You know, because it’s one of those things like when you start being in a really heated DM with someone and you realize, oh no, we need to talk with voices because this is not helping. Yeah. I find it it very like that just security stuff can seem kind of scary, kind of abstract and kind of sudden, but if you put it with a person who wants to help or even even in you know, a page like command, that’s the thing we did recently.

We just realized, oh it’s not written in enough places that when this even faintly seems like it’s going to happen, you know, one of several kinds of possibilities. Page us, we will not be mad about hearing from you. It is a pleasure to pop into a situation, look over it and go, oh, it’s fine actually. Like that’s great. Let me have the option of now and then to go, oh no, no, you’re okay. I understand why you worried but this is not a concern for us. You know, page first, ask questions later in those times when we can tell people are a little bit freaked out because it’s better for people to have the chance to be equipped with the information they need then for them to, you know, try to be a hero and everything’s worse than it had to be.

Kat Gaines: Yeah. That is exactly what you don’t want. I like the accessibility of having a page command. I think just as you know, PagerDuty being a business that deals in incident management and incidents, that’s something we’ve seen a lot of customers do for their internal security reporting processes too. Just making sure that hey, that’s accessible to everyone and like you’re saying page first ask questions later, you’re not gonna be punished if it turns out to not be a massive issue. Right. You’re going to be, if anything just thanked for bringing up something that seemed a little scary, even if it turns out to be a false alarm. And I was at an event recently and it came up in the conversation, this event that I was speaking at, I ended up telling a story of, it wasn’t a security issue but there was a time when I was running a support team at the time and one of my teammates who was in customer success called me after hours late and basically had a customer who had noticed some kind of issue that was clearly going to be an incident and he was just like, I don’t know what to do.

I, it’s after hours. I literally have no idea how to get hold of the right people to deal with this. And so I was like, okay cool, open up your laptop, I’m gonna go do the same thing and I’m gonna teach you how to trigger a PagerDuty incident from Slack. I’m gonna teach you how to do that and show you our internal incident process so that you know that this is something you can do. And that was the thing, the feedback he gave me at the time. It was, oh, I didn’t realize that I was allowed to do that. I thought I had to get another team involved or something. And I was like, no, you can use that slack command. That’s why it’s a slack command and not like a super secret lockdown process because you get to be involved in this and you have the context so you should be the one passing it on. Right.

Breanne Boland: Yeah. Very much like I want someone to look at the PagerDuty roster and go, these are the people that can help me, not, these are the people I desperately don’t wanna bother. Like I want it to be a button that someone feels hap not happy, no one’s psych to do this empowered and embraced to push like push this button for help rather than, you know, this is a scary red axe behind glass and if I reach for this, it’s gonna be a whole thing. I mean the, some of the best moments at work are when I get, you know, called into an incident zoom and I listen for a few minutes, I ask a few questions and then I get to go, no, you’re fine in this respect. Cool. Have one less thing to worry about, have a good day. Tag me in if anything else comes up that’s different later and I just get to close it. Like, who doesn’t love that?

Kat Gaines: Exactly. being called in and realizing there isn’t really an issue is better than not knowing about a potential issue. Absolutely. What about kind of just internal hygiene? So we’re telling stories about people needing to be in power not knowing that they can do things. How do you kind of scale out these processes in the organization? So I’m thinking of again, back from my running a support team days when we had folks on the team who were really excited about making sure that we had good security training across the team and they took it upon themselves to partner the security team, build out a support specific training and just say, cool, we have the book now we’re gonna put it all in our internal wiki. And the reality is you’d always have that champion on every team who’s willing to kind of take on that work. So what’s, what’s something that you’ve seen work as just sort of a methodology for scaling out the comfort and the accessibility piece?

Breanne Boland: A lot of it is, you know, looking at looking at situations and looking for patterns and finding places that can be well addressed by a playbook or wiki. You know, not everything can, but it’s something I do regularly. I have a project for it right now actually where I’m just looking at, you know, recent events and going, is there a, are there two or three common things behind this that we could address that we’ve been missing? You know, that’s one thing that’s always really important. It’s one of my favorite things to do. Like I love finding patterns and behavior. I love finding common elements in things that are reasonably considered to be chaotic. And sometimes, yeah, it’s truly chaos, especially if you deal with attackers. Like there is a randomization in things that we cannot avoid. You know, the fact that there’s always someone on another continent across the world who’s generously doing QA on our public facing assets, , if you wanna look at it very optimistically, you know, there are things you just won’t ever be able to, you know, totally prepare for.

And from there I think it’s really important to get into working with individuals. You know, one thing I like to do is watch for people who are really enthusiastic insecurity activities. We do security awareness month in October, like lots of companies do. And along with just being fun because what’s more fun than to talk about something you love and see other people light up going like, oh wait, this is really interesting. But we also look for people who really light up and really step forward and clearly love this stuff and we make them our friends , we reward them as much as we can with knowledge with, you know, the, the chance to like lead a secure code training session. That’s a really fun one to give when you see someone who loves this stuff enough that you know, they’ll bring something really new and interesting and they’ll get to be in the spotlight for a bit.

You know, along with just wanting everyone to feel empowered and full of information. My other secret goal all the time is if I work with someone on something for a period of time, if they could get a new skill to put on their resume, that’s super exciting. So those are the kinds of things I look for. It’s a mix of just like, hey, security sees you and appreciates you and wants to give you things that you enjoy that will make more of your working life better. But also just, and by the way, on a bad day you can go and look at these resources and we’ll work through together and make everything okay. But yeah, lots of empowerment and stickers. I like handing out stickers too.

Kat Gaines: Yeah, , that’s always fun. I like that point about someone who’s really excited and getting a new skill on their resume too, because that’s something that I’ve seen happen in real life where it’s just a career pipeline for that person where you have someone who’s really enthusiastic and they start working on the security work within their team team and then before along the security team says, Hey, we have an open headcount. Do you wanna, you wanna come over here and hang out with us ? And I think that’s really fascinating to see when that happens because it just means that that person kind of found their thing that they’re excited about for now and it’s a retention strategy, it’s a little bit of burnout, prevention, being able to give someone something new and exciting to work on. And honestly, it’s what I hope we should all be investing in, which is the growth of our teammates and making sure that if we want good people to stick around, they have those opportunities, right?

Breanne Boland: Yeah. Gusto does team rotations sometimes which work for all of those things. Just, hey, take three months, you know, work it out with your team so that it’s not too disruptive, but take three months, hop on over and learn something new. Just do something different for three months. And yeah, along with learning as you say, it’s such a great piece of burnout prevention just because you can do something else. You know, unlike regular work, it has an endpoint, like there’s a really sharp, clear definition of done which I think is one of the less discussed causes of burnout. Just that there’s, it’s never done, it just keeps going and you can feel like you never really make a change, but if you get given a, you know, three month assignment to go over to a team and just, you know, get one really specific, you know, medium sized thing done, it’s just a nice reminder that you can finish stuff, which I think is really critical.

Kat Gaines: Yeah, I agree. You do sometimes just get stuck spinning your wheels on ongoing projects in your own team so much that it can just feel like, ah man, none of this stuff is ever gonna reach a fruition point that I can be satisfied with. And having that extra energy to kind of inject it into what you’re doing can be really invigorating.

Breanne Boland: Yeah,

Kat Gaines: Yeah, definitely. We’ve been talking a little bit about just folks in other places in the org and how they can partner with security. Right. And I think something that we haven’t touched on too is the fact that those people can sometimes be targets for folks externally looking to exploit a business. And I feel like where I’ve seen this happen the most is in customer facing teams, right? Where it’s someone who writes in and they either want some information that isn’t necessarily something they should have from a support team or someone like that or who might get in touch with the sales team. We have internal trainings around, you know, if you see a message from our CEO that doesn’t really look to you like it’s from our CEO, maybe question that once or twice, right? And so let’s talk about that a little bit just in terms of prevention for those folks and especially for people who have a lot of external noise coming into their day-to-day, what do you focus on when talking to them about what to look for, what to flag, those types of things. Big

Breanne Boland: Thing is setting, setting those norms. A phrase I heard recently before that is authenticated communication channels, which is a nice, I like that version. Yeah. It’s basically saying like, the CEO’s never gonna text you, he is never gonna text you,

Kat Gaines: You don’t text normally, you don’t have that rapport like don’t expect it. Yeah.

Breanne Boland: Yeah. And just to set the norm, like within the company, that is not how any of us communicate with each other and to stick to it really strongly, you know, that’s a big part is just setting norms. You know, this is what you can expect if you see something that doesn’t fit within this, come talk to us. Worst case scenario, it just turns out someone was, you know, a little sleepy today and didn’t write to you in the right place and they’ll be reminded not to do that. Yeah. Just giving people the tools to recognize weird behavior with, you know, ulterior motives when they see it. Just to take a lot of the guesswork out of it. And in that respect, I have so much sympathy for salespeople and marketing folks and everyone you just described who has to interact with the wider world that much more, you know, not that I don’t get like weird LinkedIn messages occasionally. Everyone does. We

Kat Gaines: All do.

Breanne Boland: Yeah. But like I’m never going to have, you know, my professional contact information on a webpage. I am safe and snug behind an alias like so many people are, but you know, not sales and marketing. They have to go be people and hand out their contact information and the threat model is so much different. I don’t envy it. And you know, again, the, the thing to do is just to have an ongoing relationship so that ideally when someone experiences something strange, they don’t have to initiate their first contact with security the day that something hinky happened, the ideal is they’re like, you know what, I’m gonna go talk to that. That nice pink haired lady from orientation, she seemed to wanna hear about this. You know, rather than having to start, you know, the things that feel really high effort when you’re already kind of freaked out. It’s not just like I have observed behavior that’s a problem, but also I have to go, oh what was there a slack handle? Like what do I, is there a channel

Kat Gaines: Right. Hunting for the process?

Breanne Boland: Yeah. Like, you know, what I wanna do right now is argue with confluence, search that’ll help. You know, and instead like just to have people know exactly what to do and to have a face within, you know, a team whenever they can because that does seem to help. It’s totally okay just to go like, hey security, there’s a thing. That’s a great response. We like that too. But especially I think maybe especially with sales folks because it is so personal, it’s always like people talking to people and establishing relationships to have that face and that name and you know, that contact in, it’s not gonna be in Salesforce. My brain was going to Rolodex, but whatever it is right now, just like I know what button to push to do this. Yeah. So much of it is establishing relationships between teams that otherwise don’t necessarily talk a ton and that allows room for weird stuff to happen.

Kat Gaines: Yeah. And I think too that’s been exacerbated maybe a little bit the last few years, right? Where more people are working remote than they ever did. And personally I love that. I hate commuting, I love being at home, I love being with my pets and my partner and getting to make my own lunch. But the only thing about that is that you don’t see faces very often. So you don’t get to build up what you’re talking about where you’re like, I know that person and I know that I can go talk to them. So I think there’s a lot of importance to what you’re saying around building that rapport in every single way that you can and putting a lot of intentional work into it being highly visible where someone knows they can reach out to you even though they can’t walk across an office and find that person that can’t and shouldn’t be a barrier. Right.

Breanne Boland: Or say something four desks away from me and I, you know, am quietly eavesdropping as I always yeah, . And was when I wasn’t in office because that was really useful for that. The last time I was in in office on the regular, I was working on an ops team and just being able to sit and occasionally hear someone go like, I don’t understand this AWS thing and just levitate over going like, hello, how can I help you with this? You sound confused, I would like to fix it instead. There’s just things like, I’ve called it Slack surveillance and conference talks, which is a, a sharp term for what it is, but it’s what it is. I read a ton of slack channels. I’m that I don’t have regular business in just different team channels just because that’s the best way I’ve got in this era. Sometimes to see something come up before it’s a real problem.

Kat Gaines: Yeah. I do a lot of that too still. I think especially these days where I even go as far as the keywords I have set up in Slack. So if I’m in like an external community, slack of some kind where I know people might mention PagerDuty and I can help, I just set up that keyword and someone mentions PagerDuty. I’m just like, Hey, what do you need? I’m here by the way, hi, hello . , can I help? Yeah, because you don’t wanna see people flailing if you know you can help jump in and especially if you hear that they might not know the right person to go to. And so you do the job of both seeing the thing that’s happening in the moment and then building that connection for the future so it scales out to more people.

Breanne Boland: Absolutely.

Kat Gaines: Okay, so I think we’ve got a couple of questions that we like to ask everyone who shows up on the podcast. The first one I wanna ask you is, is there anything you wish you would’ve known sooner around these topics around working with others in security or empowering others to feel okay to work with your team?

Breanne Boland: I mean really just even going further back from that, I think the thing I wish I would’ve known sooner is how hard it is to keep staging and prod identical because it touches everything but it’s just so much ongoing work and even if there’s so much deliberate work put into it, there’s just drift. And I wouldn’t have guessed that when I made my switch to tech eight years ago. It’s a specific thing that is important and strange and needs to be dealt with. But I think it relates to everything else too, just to remember that there are expectations you might reasonably have that will not be true unless you put work into them frequently and regularly that you might think like, oh you know, we did an event, we introduced ourselves, we have onboarding, amazing onboarding. But that doesn’t mean things will turn out the way that you want unless you put a lot of maintenance into them and really put the work in to make sure things look the way you want them to. There’s just drift happens and drift is not as bad as say like an attacker, but it is a regular quiet presence that needs to be pushed back against all the time or else things are not gonna work the way you want them to.

Kat Gaines: Yeah. Checking those assumptions is always solid advice in anything in life really. But definitely here. And then our second one that we always ask folks, is there anything that you’re glad we did not ask you about today?

Breanne Boland: Well probably the times that I’ve tanked prod, like those are stories I really like telling, but they’re great stories to tell under friend DNA over a beverage of your choice. Yeah, I love trading stories like that. But yeah, that these were good questions about how to make things better and not, you know, tell me about your worst day , which I’ll, but that’s my first conversation.

Kat Gaines: Yeah. My goal isn’t to bring up the trauma in these conversations necessarily. .

Breanne Boland: I appreciate that.

Kat Gaines: Okay, well thank you Brian, so much for joining us. It was great having you on the podcast today.

Breanne Boland: Oh, thank you so much. This was great.

Kat Gaines: All right, and folks, again, this is Kat Gaines and thank you as well for joining us. Go ahead and check out our show notes where we might drop in a resource or two as well as just kind of where to connect with us. And please have an uneventful day

Kat Gaines27:54): That does it for another installment of Page It To The Limit. We’d like to thank our sponsor PagerDuty for making the podcast possible. Remember to subscribe in your favorite podcaster. If you like what you’ve heard, you can find our show notes at page it to the limit.com and you can reach us on Twitter at page it to the limit using the number two. Thank you so much for joining us and remember, uneventful days are beautiful days.