Security Careers With Megg and Patrick

Transcript

Mandi Walls: Welcome to Page It to The Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve the system and liability, and the lives of the people supporting those systems. I’m your host, Mandy Walls. Find me at L-N-X-C-H-K on Twitter. All right, welcome back to Page It To The limit. October is cyber security awareness month, so we have some folks with us from PagerDuty Security team. I’m going to let them introduce themselves, tell us a bit about what they do and we’ll go from there. So let’s start with you, Meg.

Megg Sage: Hi. So I’m Megg, I am a senior security engineer for PagerDuty. I’ve been here for about four months now, so I deal with a whole bunch of different things. Specifically I am in product security, so it’s more focused on securing the product. And prior to that I was actually a software developer for about seven years, so really familiar with more of the dev side of things, dev workflows, that sort of thing. And a lot of the struggles that developers have, and clashes they’ll have with security.

Mandi Walls: Awesome. And Patrick?

Patrick Roserie: Yes, Hello, I’m Patrick Roserie. I am a cyber security engineer as well on PagerDuty, same team as Megg and started at the same time. So it’s kind of a, hey, you got two new people that’s been here for about four months. But my background’s a little bit more into security. I’ve been an engineer for about seven years now and most of my duties involve just solving complex problems and integrating systems that natively don’t integrate with each other. We see all kinds of stuff and you would think, hey, let me buy this expensive tool, and this expensive tool, and it’ll work out the gate. That doesn’t happen. Not at all in any way, shape, or form. So it’s more making things that don’t happen, happen.

Mandi Walls: Oh, interesting. That’s a great way to put it. All right, well, so my prep for this episode was to rewatch Hackers, so that’s my baseline for security. Get us started. Patrick, you said you’ve been at this sort of a while, so how’d you get into security? What drove you in this sort of direction?

Patrick Roserie: This goes back eight, nine years. I was in HelpDesk, this was when we were upgrading Windows XP to Windows 7. So I was part of that whole transition where companies were like, “No, we’re not going to upgrade XP.” And then Microsoft was like, “Yes, you are.” So we had four months to upgrade a massive amount of computers and of course we didn’t meet the deadline, but hey, whoever meets the deadline. But just doing it and just learning that I don’t like manual processes in any way, shape or form. When you’re migrating a computer and then you have to migrate the data, it is horrible. It’s horrible experience. So I took that and I started automating a little bit of a few pieces because there weren’t that many automation tools at the time. So then I took that and kept it in the back of my mind. That was actually pretty cool. I had the idea of getting into networking before getting into security, so I went deep into it. I got my Network Plus I got my CCNA, and I’m like, “Awesome, someone’s going to hire me now.” Absolutely not. They did not hire me. No one wanted me. To be a network engineer you needed to have X amount of experience, and I had zero experience. So it was a little wake up call for me. Okay, well yeah, what the hell? So, at the time I was in New York and I had the opportunity to go through a training program that sends you on an internship at the end. And that’s how I got my foot into security and I’ve just been growing ever since.

Mandi Walls: Oh, that’s super interesting. So not your original plan by far, right?

Patrick Roserie: No, it wasn’t my original plan, but it worked out for me.

Mandi Walls: Awesome. And Megg, so as a software engineer, what tipped you in the security direction?

Megg Sage: It was something I always thought was really cool and interesting. Obviously I’d seen things like Hackers and stuff like that, but I also had friends that had actually shown me some of the cool hacking tools and stuff like that. I’m like, that’s so cool, but I don’t think I could ever actually get into security. And then after working as a software dev for a few years, I started just like, well there’s a cyber security digital forensics program part-time at the local tech school. I’m going to least do it for fun, and we’ll see what happens from there. And ended up just falling in love with it, this is amazing. All right, how can I get into this field? Previous place I worked had a Security Champions Program. So I got involved in that as my team’s Security Champions. I did a lot of liaisoning with the security team, working with getting vulnerabilities resolved, dealing with looking at the results from our pen test, making sure everything was remediated, doing lots of knowledge sharing and upscaling sessions on security topics and things like that. And sort just kept taking every opportunity to try to be as much of a security cheerleader as I could, and then eventually I decided, everything’s saying that there’s tons of cyber security jobs right now so maybe it’s time to see if I can make the switch, see if I can find somewhere that will value the dev experience. And so that’s how I ended up here.

Mandi Walls: That’s great. So for PagerDuty, since we’re a SaaS platform and you’re working with the engineers, what kinds of day to day do you have? You mentioned pen testing and some other things that would be maybe activities that you’d run on some of our stuff, but what do you do day to day? Is it sitting in a close beating on keyboards, or what do you get to do now?

Megg Sage: Day to day is all over the place. Sometimes it’s just spending half the day chatting to 10 different people on Slack because you’re trying to track down who knows about this random thing, and who’s the knowledge holder on this, who knows about this, how does it run, that sort of thing. Other times I might spend all day writing a script that’s scanning our GitHub for something, monitor it, or it might be looking at logs to look into something else. Might be reviewing documentation on something, it’s pretty much all over the place. There’s rarely a day that’s the same as the rest.

Mandi Walls: Patrick, anything else that you do during your day?

Patrick Roserie: Just to add on a little bit to what Megg said, it’s not just PagerDuty. We have PagerDuty, the Rundeck and Catalytic, and it’s doing the security for all of these teams. I have a project coming up to integrate one of them a little bit more into our processes and stuff, and finally finish that merger and bring them on. I know they have their at PagerDuty emails, but there’s still a little bit more work to get done in the back end before it’s fully considered. All right, these are all our assets, this is what we own, and this is all 100% PagerDuty right now. Another thing that we do, technology is ever changing. It changes so quick and so fast, it’s ridiculous. We’re getting into the Kubernetes, so we have a migration going on, we’re going to that platform. And it’s just again, figuring out those complex problems. Okay, cool. The app work here isn’t going to work here. It should work here. How’s it going to work there? And having these discussions with teams and try to make the least amount of mistakes.

Mandi Walls: It sounds like everything’s going to be a trade off. How do you keep up with everything? We have lots of upstream dependencies and I only passively watch the things that come through on some of the GitHub repositories. There’s a lot of stuff that goes into our products sometimes there’s CBEs that pop up in different components. How do you stay on top of everything that’s going on that might impact us?

Megg Sage: Some of the stuff for CBEs specifically, we do just have tools that scan for them. So sometimes when pops up right away, a tool will flag it and we might have to add a temporary exception or something for a team because they’re like, this was just posted yesterday. I don’t have a fix for it.

Mandi Walls: Give us a minute, give us a minute.

Megg Sage: Have a temporary exception so I can keep working. So those, we do have a lot of automated tools that pop up and give us detections. Some stuff we just read about or people share it with us, especially some things like there was Zoom X point for Macs recently and that came out of DEVCON and people were sharing articles about that. So some of it we do find out about that way. Either we’ll read it on mailing list or someone shares it with us, so there’s lots of different ways to keep on top of that sort of thing.

Patrick Roserie: Yes. These CBEs and these vulnerabilities can come in many different places and many different areas. It could be that bugging your code, it could be a package on your operating system, it could be the actual operating system, the kernel. It’s a lot to keep up. And yes, we do have tools for that and I’m actually working on something to integrate those tools and consolidate and bring it in a dashboard so we don’t have to sit here and click and go, “All right, this tool doesn’t show me what I need to do, what I need. Let me go to another tool and let me go see.” So this way we can kind of consolidate and just reduce that operational overhead.

Mandi Walls: How have things changed? Since I’ve started using GitHub, we now have things like Dependabot and other little tools and bots and things that are meant to help folks. But as that’s happening, we’re also getting messaging from the industry that we’re shifting security left, and we’re going to do more earlier, and all these amazing things are going to happen. So how have things changed, and how do you see things going for the engineers that you work with to think more about security or incorporate it more into their sort of daily lives?

Patrick Roserie: So the idea of shifting left is giving more feedback earlier in the process to developers because the process is like this, you write your code, you compile your code, you touch your code in the computer, it works. All right, cool. Let me ship this off the GitHub and GitHub once it’s there, you create your PR and it’s going to run through all these series of checks. And then you get your feedback there on, okay, this worked or it didn’t work. And that feedback can be anywhere from five minutes to maybe an hour, who knows? And just going through that process to then find out an hour later that your stuff didn’t work? Man, that’s a killer. So the idea of shifting left is to give that feedback much earlier in the process and that’s using some of our tools, having developers integrate with those tools in their IDE, their integrated developing environment and just giving them that feedback earlier so it’s less strained later down the line. Because I’ve been through that, I’ve written stuff and I’m just like, okay, cool, it’s going to run through my tools, what’s going to happen? I probably should practice what I preach.

Megg Sage: Yeah, we’re really working towards trying to give developers the feedback as soon as possible. Basically as soon as reasonably feasible so it’s not blocking them, because the last thing you want to do is get something like add in the library that does something and your code written up and you’re like, “Oh, this version has vulnerability. I can’t use this version.” And then it turns out you need to switch to a different version that has changes from the one you wrote. You have to redo everything, so we don’t want things like that happening.

Mandi Walls: Yeah, cheaper to fix all that stuff earlier in the dev cycle than to wait until it sneaks its way into production and then you have a problem. So yeah, definitely. So one of the things that we like to ask folks about on the show is debunk a myth, I feel like there’s probably so many myths about security, application security, cyber security, whatever pieces of the whole pie that are out there. So what’s a favorite myth? Patrick, you put one in our notes to talk about.

Patrick Roserie: Yes, my myth that I’m on social media all the time, I’m looking and people are like, “Hey, what certification should I get? What should I do? How to get into cybersecurity,” and all this other stuff. I wanted to make sure early in the process I let you know, cool, I wanted to get new networking. Let me get these two certifications by CCNA Network Plus. Didn’t get anywhere, so the certifications are just I guess the backbone for learning. But it was a lot easier to get certifications back in the day because you had brain dumps, you had all kinds of other stuff and no one took it seriously. So it’s just like, I still get people now that look at my CSSP and they laugh at me. Oh, okay. Yeah, that just proves how to take a test. And I’m sitting there like, that actually hurt. I worked hard for that one.

Mandi Walls: Cheers, man. Yeah.

Patrick Roserie: Yeah, I couldn’t cheat for that one. I worked for it. It’s a thing. You can get into cyber security without any. I believe you don’t have any, right Meg? Or you have one?

Megg Sage: I have a couple little ones now. Granted, one of them I didn’t get till after I started at PagerDuty, but yeah, I didn’t really have that many. I don’t even have background in Comp Sci. I’m actually mostly a mix of self-taught and web app development, so it’s definitely a myth that you need certificates for sure. You need the knowledge from certificates, absolutely, because that’s really what a lot of it comes down to. For me personally, I have a couple of certificates from SANS, which is one of the big prestigious cybersecurity schools and they’re great. But in all honesty, the certificate I have from the local tech school is far more useful for the knowledge I gained because it was a much more in depth certificate. It was a lot longer, took me three years to do. So that was so much more valuable in actually getting into security because I gained so much more knowledge from it, compared to a certificate that people, say, might recognize more. But you don’t actually gain as much knowledge cramming yourself through a three month course, versus doing a multi-year thing. Because in an interview they’re going to ask you a whole bunch of questions and your piece of paper isn’t going to save you if you can’t answer them. It’s about the knowledge you have.

Patrick Roserie: I’d say the ability to find answers will take you a whole lot further than a certificate would. Because for example, that thing you posted in Slack the other day, Megg, or was it today or yesterday? I don’t even remember. I didn’t know the answer, but I know damn well how to trouble shoot. I’ll sit here and I’m like, okay, I’m going to find this out and I’m going to figure it out. So I went and I was like, oh okay. I think this is what the problem is. So just that ability to want to learn will take you a lot further than the certificate would.

Mandi Walls: So what’s your favorite thing that you have learned? What’s something that was super cool or super fun that you learned?

Patrick Roserie: Python.

Mandi Walls: Really?

Patrick Roserie: Yes.

Mandi Walls: I’m not sure I believe you.

Patrick Roserie: I used to be heavy when it came to vulnerability management, pulling results from Qualis. I’ll open up Excel spreadsheet and I’m working with a hundred thousand lines in Excel spreadsheet. I’m like, this is awesome. Okay, let me create my filters and do this. And then one day I was like, let me try to do this in pipeline. So open up the file, write up what you want, and I had the answer so much quicker than using Excel. I was like, okay, let me see what else I could do. And then it just went from there.

Mandi Walls: Maybe you’ve missed your chance to participate in Excel e-sports. That may have been a missed opportunity for you.

Patrick Roserie: I have seen some pretty badass stuff on Excel.

Mandi Walls: Right? Yeah.

Patrick Roserie: I was like, I didn’t know you could create an email in Excel. Who the hell would’ve thought?

Mandi Walls: Yeah. Meg, what’s one of your favorite things?

Megg Sage: Python’s probably pretty high up there too. I didn’t actually learn Python until I started doing cybersecurity courses, so it’s pretty high up there. I think for me though, it was actually getting to see and play with some of the tools for doing exploits and just how easy it is to exploit a known vulnerability. It’s a couple commands, it can take a couple minutes against the system with the right vulnerability. So that’s really cool, and also demoing that to other people and just watching their eyes go wide when you’re like, “Ah, hey look, that took five, 10 minutes. I have read access on this system,” and they’re just stunned. I didn’t have to write any code, I didn’t have to do anything like that. I just had to run some scripts and it’s like suddenly they understand how important patching things are. That’s probably my favorite thing.

Mandi Walls: It’s so cool, right? I feel like at one point doing some kind of kernel memory overflow, buffer magic, whatever, and that was the hard stuff. And now you can download things off of this crazy thing called the internet, and somebody has already done all the hard work for you and magic happens. With derogatory. We used to call them script kiddies, but it’s still interesting tools. So really take a look at where your soft underbelly is in a lot of your applications, because this stuff is already out there and if you know about it, you’re probably too late. So turn around and somebody’s behind you in your system. What’s something that you’re looking forward to learning? It feels like security is probably a part of our industry that you have to be constantly learning stuff and looking at things. We talked about how you keep up with what’s going on, and the CBEs and other things that are happening, but how do you choose what else do you want to learn about and what do you looking forward to learning over the next say year maybe?

Patrick Roserie: I’ll give you one quick example of, so I have my CSSP. I have to maintain my CSSP, which goes into a lot of the stuff that I do learn, but I’m actually ahead of schedule because you’re supposed to have 120 educational units in three years. I just renewed this year and I’m already at 60.

Mandi Walls: Oh, wow.

Patrick Roserie: So I’m kind of like, okay, let me take a little break from learning and just focus on other stuff. So non-technical stuff, how does the business run? How does the business work? Just little stuff like this, you can actually tie into security in some way, shape, or form. So I do want to get a little bit more into the business side of things.

Mandi Walls: How do you think about how security plays into the business and how you contribute to the overall success of an organization from a security standpoint?

Patrick Roserie: Good question.

Mandi Walls: You got to find out for us.

Patrick Roserie: Yeah, I had to find out. Some stuff I learned very early in my career, when I was talking to one of the CISOs I used to work with, he told me about capital expenditures, operational expenditures. I’m like, what do you mean? And he’s like, “Yeah, we got to spend this much money by this time.” And I’m like, “Oh, we have to spend it? You can’t just pay me that money.” And he’s like, “No.” So I’m just trying to understand a little bit more about budgeting, all that other good stuff.

Mandi Walls: Awesome. Megg, you got anything on your plans that you [inaudible 00:19:35]?

Megg Sage: Always in the background I like learning more about pen testing and ethical hacking. But as far as PagerDuty specific goes, I’m really looking forward to getting to work with some of the teams more, learning more about what they do and their workflows and how we can get security more integrated into their processes. Again, I’m still fairly new, so learning more about how everything works. So I think that’s sort where I’m looking to gain most of the knowledge. Because for me, still a lot of the exact dev workflow is still a bit of a mystery to me because I haven’t really seen it, because I came from someone somewhere very, very large and very bureaucratic previously. So it was a very rigid process there. It’ll be interesting to see what all the processes are, how they even vary team to team if they do, and how security can get more integrated into there. And also just what developers want to learn about security, because that’s always fun teaching people about what they are excited about, or what they wish they knew more about.

Mandi Walls: Yeah, I was going to say, do you have a feel yet for what folks are looking for, or what they feel like their places are that they need to learn more, or any of that stuff yet? It is kind of interesting to see what folks already have a feel for in the security realm. Oh, we’ve got some [inaudible 00:21:03] or whatever sort of basics, versus what else they could be looking at in their processes.

Megg Sage: Not yet, but that is something that we are wanting to learn more about. Yeah, that’s definitely one of the initiatives that is coming up. Finding out more about what developers wish they knew more about or had more training on, and whether it be actually directly work relevant topics like cross-site scripting, or if it’s just like this is a security topic that they just wish they knew more about but might not be directly related to work. That sort of thing. And we will probably hopefully get to have more just training and knowledge sharing sessions come out of that, that are more directed topics that people are really interested in.

Mandi Walls: Yeah. Do you have a list of basic ones that you like to go over with people, or do you feel like there’s maybe a top three or top five of things that for your average engineer out there, they should have some knowledge of a handful of security bits and pieces?

Megg Sage: I think everyone’s had a SQL injection drilled into their heads by now.

Mandi Walls: I would hope so, the best.

Megg Sage: But I think what I at least found as a developer is I didn’t really find I fully understood what you could do with cross-site scripting until I actually had it demoed how you can use it to actually obtain session cookie information to actually access an account in something. And then it’s like, oh, all right. Now this all makes sense. And I remember finding a lot of the training that we did, I did as a developer didn’t really go into enough detail that you can release it. You’re like, okay, I get this. This makes sense, I understand what this is, I understand how to prevent it. But I didn’t fully understand how bad it can be. Yeah, I think that’s probably one. But yeah, it probably depends on developers and what they do, what’s the most-

Patrick Roserie: To add onto that, we might be shooting ourselves in the foot. One, because PagerDuty has some of the smartest people I’ve ever worked with to date and I’m just like, my God, I don’t even know what’s going on here. But another one to add, I think, I can’t remember what the OWASP top 10, I think broken access control is pretty high up there as far as identity access manage. All other stuff right there, that is very hard and that’s one of the main places, a point of entry.

Mandi Walls: You mentioned the OWASP list, I’ll put that in the show notes for folks if you’re not familiar with it. Is there anything else that you use as guidance for folks that way, that if we have any engineers listening, that there might be some things they could read up on?

Megg Sage: If you’re not familiar with that, that’s a great thing to start. They also have lists that are more mobile specific, too. If someone’s a mobile app developer, that might be more relevant. There’s some other lists that you can look at, it sort of depends on what your interests and your goals are. If you’re more concerned about what’s relevant to you as an individual person, or are you more concerned about what’s relevant to the work you do.

Mandi Walls: Cool. All right. Is there anything else that you would like folks to know about security in general, or anything else that we should cover before we say goodbye this week? What’s your favorite security pet peeve?

Megg Sage: People calling hashing encryption.

Mandi Walls: Oh, that’s a good one. Yes.

Patrick Roserie: I’m going to say yes. I don’t, when they say that people are the problem. They’re like, “Yeah, don’t click that link email, don’t open it,” and they’re going to shame you for it. And here at PagerDuty we don’t shame people because you’re going to open up the email. It’s okay. We’ll investigate and we’ll help you minimize the effects after the fact, but don’t be afraid to be human. And that’s what I feel like a lot of security professionals do, shame people on being human.

Mandi Walls: I feel like since I’ve joined PagerDuty too, our security training is very collegial. It’s very much they feel more like the security team is your peers and your friends and they’re going to help you with stuff. And the fishing stuff is so tricky now, some of it’s really, really clever and unless you’re being very conscious all the time and that’s hard to do, it’s exhausting. We get a lot of garbage emails, and a lot that look legitimate that could be nasty. And our security training, which is online, we have a version of it that’s open source and online, so I’ll put that in the show notes for folks too, is super supportive for actual human beings that are doing a thing and living on the internet where bad things sometimes happen. So yeah, it’s super great from that perspective for all of us non-security people.

Patrick Roserie: And I think I can remember a few weeks ago when there was a PayPal one going out, where people were abusing PayPal services and emails were coming legitimately from PayPal about a company saying, “Hey, you owe us money.” So it’s like how do you train and prepare people for that? You tell them don’t click a link, it’s not HCPS. They’re not going to do it. And this is a legitimate email coming from a legitimate company for illegitimate purposes, but that’s where it does get tricky. These criminals are getting too smart.

Mandi Walls: Yeah, definitely. And there’s way too many of them for sure. All right, well I think we’ve covered some good stuff here. Is there any other parting thoughts you’d like to leave folks with this week? Change your passwords.

Patrick Roserie: Go to pageduty.com/careers.

Mandi Walls: There you go. Yeah, folks can always check out our careers page. Does your team ever take an intern? Our intern program is amazing. We had an intern on our team for two years ago, it was incredible to have her working on actual projects for us. But yeah, folks are looking for intern projects, intern program, our internship program is amazing.

Megg Sage: Yeah. Yeah, there’s supposed to be one coming up. Yeah, we’ve definitely had interns in the past, I believe actually one of the people on our team used to be an intern. I think Nan used to be an intern originally.

Mandi Walls: Yes.

Megg Sage: Before either of our time, obviously. But yeah, I think she actually started out as an intern.

Mandi Walls: Awesome. All right. Well, this has been super fun. You guys are great and I’m glad you’re with us at PagerDuty helping us out and keeping everybody safe since we have so much important information from all of our customers and all the good stuff. So thank you for joining us this week.

Patrick Roserie: Thank you for the invite. It was fun.

Megg Sage: It was great to be on here.

Mandi Walls: Awesome. All right, so thank you everybody out there. We will see you again in two weeks. In the meantime, we’ll wish you an uneventful day. That does it for another installment of Page It To The Limit. We’d like to thank our sponsor, PagerDuty, for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes at pageittothelimit.com and you can reach us on Twitter at @pageit2thelimit using the number two. Thank you so much for joining us and remember, uneventful days are beautiful days.