Automation and Operations With Brittany Woods

Transcript

Mandi Walls: Welcome to Page It To the Limit, a podcast where we explore what it takes to run software and production successfully. We cover leading practices used in the software industry to improve the system reliability and the lives of the people supporting their system. I’m your host Mandi Walls. Find me at L-N-X-C-H-K on Twitter. All right, welcome to the show. This week, we are going to talk about automation and operations with Brittany Woods of H&R Block. Brittany, welcome to the show.

Brittany Woods: Thank you, Mandi. Happy to be here.

Mandi Walls: Yay. Tell us about yourself and what you’re doing at H&R Block these days.

Brittany Woods: Yeah. I am, obviously, Brittany Woods. I am a manager of the server automation team now at H&R Block. That’s a new thing. Before that I was a lead cloud automation engineer. My specific focus is in infrastructure automation, both on-prem and in the cloud. So things like Terraform and configuration management and all of those goodies. That’s still a focus now that I’m managing the server automation team. Essentially what we’re doing is we’re trying to take historical operations tasks, things like push button deployments, et cetera. We’re trying to get all of that running, get that going and automate the things.

Mandi Walls: Yeah. So, what processes do you guys follow for this? Are you using… Are you doing all the… Checking into git and all that infrastructure as code stuff? How do you go about taking the old things and turning them into new things?

Brittany Woods: Yeah, so really the biggest thing that we’re doing is we’re going through a process of analysis. So we’re trying to determine where we stand in terms of automation. What is there that needs to be automated? And then as we automate it, we’re following all those best practices. We’re shifting into basically GitOps, even though that’s an odd word, that’s new and fun. We’re shifting into that GitOps mentality where we’re taking everything, we’re turning it into code, we’re checking it into GIT. GIT takes it into pipelines and delivers that code to whatever in-point it needs to go to. Does sanity testing along the way to make sure everything looks good and that code isn’t accidentally released to production before it’s ready to be released. And so we’re trying to check all of the right boxes as we go through in terms of DevOps and automation and building those skillsets internally, giving all of this separate teams the ownership to own their destiny, if you will.

Mandi Walls: That’s awesome. So, as folks are learning how to do this, what kinds of things are you finding that is new for them? You’re going to build on the skills they already had, but what are some of the things that might be completely new to somebody coming in to maybe an automation project?

Brittany Woods: Yeah. So I mean, the biggest thing that you have to get used to is in the areas where you’re going from maybe scripted automation into continuous configuration and continuous automation, you have to get over that understanding that you are running the automation. Basically my philosophy is if you have to click a button to run the automation, it’s not automated. So really getting people from saying, oh, that’s automated because they have a script or something that they have to trigger and more into thinking that’s automated because I don’t have to touch it anymore. So, that’s the biggest learning curve, I think, is teams understanding what it means to run this continuously. What it means to keep things running without a human intervening and how their code should be crafted in a way that that is okay.

Mandi Walls: Yeah. No, that’s awesome. So as we think about automation sort of internally as for some of our products, we have it on a continuum for you start out with not a whole lot, and then you try it and then it’s a trust building exercise. So that the folks who are putting this automation stuff together they’ll run it a few times by hand, and then they’ll run the script with their own requests or whatever. And then eventually get to the point where you feel you can trust it and it’s not going to go bonkers on you and turn into some Mickey Mouse with all of the brooms in Fantasia that’s just out of control.

Brittany Woods: Right.

Mandi Walls: So, yeah. That’s fantastic. So what benefits then do you see downstream of all this stuff? I feel like everybody’s super busy and there’s a million things that are going on. Are you reaping time back? Are you working on better projects? What kinds of things happen then?

Brittany Woods: So realistically the big thing that we are seeing now, and what we want to see really in doing all of this, is the age old buzzword of shifting all of this left because ultimately we have some teams that are very far in this process. They’re managing their own stuff and that’s great. And then we have other teams that are pretty reliant on operations groups and things like that. What we really want to do is want to shift all of this, so they can control their configuration. They’re the ones creating the application. They understand what’s going on. Their code and their configuration and their systems, they should all live within the same realm of ownership to help with resolving incidents and things that. So we want to get that back. But the other thing we’re seeing is honestly, just increased ownership. So increased ownership of the process and understanding of what’s going on. Long-term we want that to equate to faster resolution, things like that. But the short term, where we’re at in the short vision, is just trying to get that ownership in place and get them able to manage their systems from end-to-end. Able to build their systems from end-to-end quickly. And then just start scaling that across.

Mandi Walls: Yeah. So as part of that process, are you building samples for them? Are you building good, blessed examples or processes that they can adopt, or you walking them through building all of their own stuff there?

Brittany Woods: So it’s a combination of both because what we wanted was a stable environment with the guard rails that you need in place. So, my team, for example, we’re really deep into the configuration management portion of this. And so my team is creating all the pipelines and all of the governance information for the environment, and then acting more as consultants or liaisons with the various teams to understand what their needs are. Help them understand what they need to do in terms of automation and then get that in place. But in saying that though, as acting as those consultants, we’re doing whatever it is the app team needs of us. So if the app team, in order to get to this good state, needs us to create the things and then enable them to manage those things, that’s what we’re doing. If they tell us to kick rocks and that they can do it themselves, then we kick rocks, they can do it themselves. And we just reserve some of our capacity to help them if they need our help. We’re coming to these teams with varying levels of experience in terms of automation, varying automation in place. So we’re doing both honestly, but the end goal is to teach them to fish, so to speak. We don’t want them to have to come to us if they don’t want to or need to. That’s what we’re trying to do is we’re trying to take ourselves out of the equation as much as possible.

Mandi Walls: Awesome. No, that sounds great. As you’re working with these folks, do you see different kinds of questions or problems with different kinds of teams? Is there a difference between working with folks who are primarily still on premises versus in the cloud or any other weird, hidden things that happen in the infrastructure?

Brittany Woods: I wouldn’t say that there’s necessarily weird, hidden things, but not necessarily at H&R Block, but in my previous experience, people get really set on the way things are done. I haven’t seen that yet, but I’m anticipating that there might be some fear or uncertainty surrounding, what are we trying to do? And that goes into myths that we can talk about, but basically worrying that things are changing. Things might be changing too rapidly and that they might miss that train. And so the goal is to get them on the train and take them with us. So we’re not picking them up at stop number three, we’re starting at stop number one, right?

Mandi Walls: Yeah, definitely. Are you offering… Do you offer training and stuff like that to bring them along?

Brittany Woods: Yeah, so we have an internal training program for all of the initiatives that we have going on.

Mandi Walls: That’s awesome.

Brittany Woods: We make sure that everything’s heavily documented relevant to what they’re going to see and the steps they’re going to need to follow, because we’re meeting them very much where they’re at. Even if where they’re at is not the desired emd states for automation, we’re meeting them where they’re at. So they can start to understand the platform and then be like, wait a minute. I can do all of these great things. Why am I doing it this way? Because we want them to have that aha moment themselves.

Mandi Walls: Yeah. You want the success that generates more interest than further success that… Oh, those people on that team over there and they’ve got all this awesome stuff that they’re doing, can we do that too? And then everybody’s beating down your door.

Brittany Woods: Exactly.

Mandi Walls: Fantastic. Well, you mentioned a myth and there are a couple of good ones about automation. And we do sometimes have featured on our show here about debunking a myth. So one of the myths as I’ve come from automation as well that I hear or had heard at a lot of places is people are afraid they’re going to automate themselves out of a job. And you just want to say, no, man, you can automate yourself into a better job. But do you still see that in your teams there?

Brittany Woods: Yeah. I mean, that’s that fear and uncertainty that I alluded to is, people whether verbally they express it or not, people are worried that when they automate things, then there’s not going to be a role for them because they’re not going to be doing whatever it was they were doing before. But first of all, I’ve never seen that happen. If anything, I think automation causes more work than less. I mean, really the answer that I always try to give people is, somebody has to be the one that’s maintaining that automation and keeping that running. And do you want to be the person that doesn’t know how to run that automation? Or do you want to drive that automation and become intimately familiar with it? Because then you haven’t automated yourself out of a job, you’ve automated yourself into a new one. The other thing that I know people in companies often talk about is, they want you to be able to focus on more beneficial things. But I think that’s really true though, because when you automate these menial tasks, these things that are easily automateable, you’ve enabled yourself now to jump into new areas. So maybe you want to dive into Kubernetes instead of clicking a PowerShell script to build a server. And you can do that whenever you jump into that world of automation, because you’re starting to understand things like orchestration and CI/CD, and all of those core foundational things that you have to understand when you start digging into things Kubernetes, cloud automation, and cloud resources and services like Lambda and stuff like that. So if you don’t understand those core concepts, you’re not really giving yourself what you need. So if anything, I think, to go back to what you said, if anything, it’s not possible to automate yourself out of a job in tech, because tech is boundless and endless. So you just enable yourself…

Mandi Walls: It just keeps moving. [Crosstalk 00:12:15].

Brittany Woods: …to learn. Right.

Mandi Walls: Absolutely. You just keep moving to something else and something else and something else. And it’s you pick something apart and there’s some other rabbit hole to go down and shave a Yak sometime later.

Brittany Woods: Exactly.

Mandi Walls: Yeah, absolutely. So what’s something that… You’ve been at this a while I’ve known you for years now. And I know you’ve been at this for some time. What’s something that you’ve learned in the past few years that you wish you maybe known earlier? Something that you’d give to the youngins out there, sort of just starting on maybe looking at doing some automation?

Brittany Woods: Something that I’ve learned across the years is very much that, and I don’t know how relevant this is to the question, but it’s very much that it doesn’t matter how much you think you know because things are going to constantly change. And when it changes, you don’t have to be worried. Maybe new products come and they’re doing something similar to the old product you have. There’s no reason you have to be afraid of that new thing, because ultimately as long as you understand the principles of what you’re doing, and as long as you’re passionate about that, you’re going to be trying. Configuration management, for example, you can take the principles of configuration management that you’ve been doing for years, and you can apply that to any other thing. And that’s fine. So maybe just don’t get too attached is what I would say.

Mandi Walls: It’s you can’t step into the same river twice. Every company has, you’re going to be doing the same tasks and the same sort of stuff to get things going. But there’s an infinite number of choices for the tools that you pick along the way. And sometimes you find a tool that you like and then all of a sudden they get bought or they decide not to build the tool anymore or something else happens. And you’re like, oh, well this other thing over here does the same principles and it’s time to move. So yeah, there’s definitely…

Brittany Woods: I think that the other side of that too is, it’s okay if you’re using whatever you’re using, it’s okay to use that. Even if it’s not in fashion at the moment it’s okay to still use that. And I think telling younger me that instead of being like, oh, well, no, that’s not cool anymore. I need to do this or that sounds cool. I need to do this other thing, telling younger me that I think would just cause a lot less stress because at the end of the day, a tool is a tool is a tool. Whether it’s the same one that you’re used to using, or whether it’s a new one, whenever you go to a different company, it’s all fine. It’s all good.

Mandi Walls: Yeah, absolutely. Yeah. That gate-keeping really is exhausting. You see it for even the most ridiculous things. My preference for this parser library is XYZ and that’s the only possible solution to this problem. That is such some deep minutiae that no one should really care about, but Twitter exists.

Brittany Woods: Right? Exactly. Tech Twitter is a thing. Things are going to pop up.

Mandi Walls: Yeah, absolutely. Oh my gosh. Just what else can you share with us? I saw you on, was it Sky Crafters, the cloud infrastructure automation forum, earlier this year? What things are you looking at now? What are you learning from next? What else is coming up?

Brittany Woods: Yeah. So, I mean, Sky Crafters is actually really exciting. I longed for an area where you could communicate with like-minded cloud enthusiasts, if you will, because Sky Crafters, it doesn’t matter if you’re new to the cloud or if you’re a seasoned veteran or whatever. It’s just a bunch of people talking about cloud engineering and cloud automation. How they can get better, how they can level up. How they can level up others. And so I really longed for that because on Tech Twitter, you say the wrong thing and immediately you’re either viral on Tech Twitter, or some bro post something somewhere and it becomes a thing. So, the things they’re doing over there are really exciting because it’s really just a bunch of forums and we can all talk about maybe somebody wants to know what certifications they should get and veterans of the industry can come in and say these are the ones that amplified my career. So maybe go down that path or whatever. Or we can talk about things like multi-cloud and whether we think that’s a smart thing to do or not a smart thing to do. So I’m really excited about Sky Crafters. I should get on it more often. I check it probably once or twice a week, but that’s one thing that I’m involved with. Just being a member of that community and helping drive the conversation. Let’s see, what else do I have going on in my super exciting pandemic life? I definitely miss conferences. I’ll be speaking in my normal speaker spots on SHPE conference coming up in September. So I’ll be speaking there, just talking about it doesn’t even have to be SHPE, but just talking about how you take an organization that doesn’t have configuration management, how you do that. How you implement that, because it’s just not something that people talk about a lot. They’re… Yeah, you should manage your config. And then config management goes away at some point, but the truth is, is it doesn’t. You’re just doing it in different ways. So really taking it from point A to point B and how you implement that in an organization cause I’ve done it three times now. So that’s coming up pretty soon. And then I’m also working on turning that into, I hesitate to use the word book, cause that’s not true, but into a guide, if you will.

Mandi Walls: Oh excellent.

Brittany Woods: So translating that knowledge into a guide and then I’ll put that out so people can enjoy it.

Mandi Walls: Oh, fantastic. Oh, that sounds great. Well, we’ll be waiting for that with much anticipation. For folks listening, I will put the Sky Crafters link on the show notes so that you can find that because it is like you say, definitely an interesting forum for folks that are hip deep in the cloud. I was surprised when I saw Sky Crafters, I was like, oh do I really need this. And as I was… I’m digging through it and I’m yeah, this doesn’t actually exist somewhere else. It was an interesting thing that was sort of missing. So yeah, it was super neat. What do you think of multi-cloud?

Brittany Woods: Originally I was like, maybe you shouldn’t put all your eggs in a single basket. But then looking at it objectively, the cost that would be associated with being in multiple clouds and then all of the duplicated automation and code, because everybody has their own resources. And Azure uses resource groups where AWS is like, who is she? I don’t know her. Feasibly it doesn’t make sense. I think it would be great to be able to have that cohesion and that kind of safety net. But at the same time from a cost perspective, it just doesn’t make sense.

Mandi Walls: No, I feel if you’re going to keep it to the point where it’s going to be easy for you to spread things out across multiple vendors, you’re not going to get enough of the additional benefits of actually going to the cloud in the first place. All of that networking gobbledygook and all that other weird stuff that goes into your capacity planning and your auto scaling and all those things are so significantly different. And require so much additional engineering on your part that you’re just if you’re going to, if you’re not going to do those things why are you going to the cloud in the first place? Use somebody else’s computer and just whatever.

Brittany Woods: Right, exactly.

Mandi Walls: Oh yes. It was a dream. It was a dream at some point, but…

Brittany Woods: I know. I remember the young naive Brittany being oh yeah, we’ll put this in Terraform and we’ll make sure that it’s not vendor specific and then we can move it.

Mandi Walls: Right! Yes!

Brittany Woods: And that lasts for two days. And then you find something and you’re oh, well this I have to, I can’t not. And then it’s blown.

Mandi Walls: It’s just gone and it we’ll mourn it. But yeah, I totally understand the draw and yeah, still have that discussion with folks. I’m just maybe you’re barking up the wrong tree, but yeah. It’s a bummer but at the same time it’s nice to see that the cloud vendors have gotten so sophisticated. It does feel like the stuff that you would have built if you had the budget to do it internally anyway, versus just some white boxes in a closet somewhere else that somebody was renting to you. It really does feel like good solutions now versus what it was at the beginning.

Brittany Woods: Yeah. There’s more accountability, I think, than there was in the beginning. In the beginning, it was like I’m going to run this, but they’re giving me no guarantees on whether it’s going to be available or any of that. So I feel it’s a little more of a competitive market now. Now that you have GCP, which is the thing, and Azure is pretty huge now. And AWS has been doing its thing with its five million services. Everything’s to a competitive place now where they don’t want people to jump ship. So, they’re taking some accountability.

Mandi Walls: Yeah, absolutely. We’re could be sitting here talking another five minutes and AWS will put a new service out and be you’d never even noticed it in the dashboard. It just appears there. It’s like, what is that thing? And yeah, definitely getting to know all those things and becoming an actual cloud engineer and having that expertise. That pathway seems like a real thing now too. Just getting into to all of those components and all the pieces and how they all get configured. And it’s crazy, crazy business now.

Brittany Woods: It’s crazy how quickly that became an actual career path instead of just I have this skill. Now it’s an actual thing because you can’t really, I mean you can, but that’s a whole job. Configuring cloud and cloud services, and understanding how they work and that’s a whole job now because it’s so vast. And it’s just weird how quickly that happened.

Mandi Walls: Yes, yes. When you’re building for scale on premises you have the dudes that put the parkas on and go into the data center and do your rack and stack. And then you’ve got the other little dudes who sit in the closet and plug things into patch panels and do all your networking for you. And those are two very different teams. But you put that stuff in the cloud and then it’s well, Brian and Barbara are going to do all of that stuff because it’s cloud. And you’re like, hang on, man. That’s crazy talk. We’re translating what was 12 or 14 different tasks and job types into cloud engineer. And you’re like, oh boy, here we go.

Brittany Woods: Right. Right. So, I mean, it’s a whole thing now and looking at it now, cause I fell into that. I started, I cut my teeth as a Windows system administrator, which then turned into a configuration management guru, which then turned into cloud automation. And so I fell into it now I’m looking at it from a few feet back and I’m like, that’s really daunting to look at the requirements to have cloud experience with no specific service. To look at that, and somebody wants you to have that experience, look at all the services there. How do you get that experience without falling into it like I did? Where you’re just like oh, I have to mess with these things every now and then, and then you get used to it and put more things in there. How does somebody break into that industry? And going back to Sky Crafters, I think that’s why something like that is important because it’s so daunting.

Mandi Walls: Oh yeah. Absolutely. Again like you say, I wouldn’t know where to tell folks to get started to figure out what they would be interested in, then going into dig down deeper into all those things. It’s just, there’s so, so much there now.

Brittany Woods: So, so much.

Mandi Walls: It’s good for us.

Brittany Woods: Right.

Mandi Walls: We’re going to be around a while. I got years. I got decades before retirement.

Brittany Woods: I got student loans to pay still.

Mandi Walls: Right? Oh my gosh. It just never ends.

Brittany Woods: Oh yeah.

Mandi Walls: Just craziness. Yeah. So is there anything else you’d like to share with us here before we wrap up? Is there any more piece of advice for folks or anything else that you’d like to share?

Brittany Woods: I think I can probably wrap everything up by just saying on the topic of automation and operations and all of that, don’t think that you’re going to automate yourself out of a job. I know we’ve already talked about that, but don’t think it. Don’t even entertain it. Think more about how you can be an asset to your own career, instead of thinking about how you can be an asset to your current organization, which is of course important. But in being an asset to your own career, you’re essentially being an asset to your organization. So grow those skills. It doesn’t matter if it’s something that you’re used to doing. Just keep growing them and automation won’t matter.

Mandi Walls: Awesome. That sounds fantastic to me.

Brittany Woods: Yeah.

Mandi Walls: 100%. Well, Brittany, thank you so much for coming on. It’s so nice always to talk to you. We’ll put your ChefConf link in the show notes as well. Hopefully we’ll see some folks there. I’m also speaking and this will come out before ChefConf. So we’ll see everybody there and thank you again. And so folks thank you for joining us. And this is Mandy Walls signing off and we’re wishing you an uneventful day. That does it for another installment of Page It To the Limit. We’d like to thank our sponsor PagerDuty for making this podcast possible. Remember to subscribe to this podcast, if you like what you’ve heard. You can find our show notes at pageittothelimit.com and you can reach us on Twitter at PageIt2theLimit, using the number two. Thank you so much for joining us and remember: eventful days are beautiful days.