API Security With Rob Dickinson

Transcript

Scott McAllister: Welcome to Page it to the Limit, a podcast where we explore what it takes to run software in production successfully. We cover leading practices used in the software industry to improve both system reliability and the lives of the people supporting those systems. I’m your host, Scott McAllister, @stmcallister on Twitter. Welcome folks. We’re joined today by Rob Dickinson, co-founder and CTO at resurface.io. Rob, why don’t you tell us, what’s Resurface?

Rob Dickinson: Resurface is a first party data link that monitors all of your API call traffic, and helps you scan for quality and security problems.

Scott McAllister: Nice, all right. Well, that’s good because Rob’s here today to talk to us a little bit about API security. Just to get us kicked off, what gets you excited about API security?

Rob Dickinson: Gosh, it’s so present. It’s so now. This is in the news on almost a daily basis. I was in Australia last week in the wake of the Optus hack there, that released data on 10 million Australians. It’s the largest hack in Australian history, and everybody’s wondering, “Are we going to be next? Are we vulnerable to the same kinds of things?” And the answers there are a little scary. It looks like the cause of that hack was simple user error. Not complete negligence, not complete inattentiveness, but just basically a deployment error that went undetected.

The folks that work on these systems that see this stuff up close all the time, we know that these systems are incredibly complex, and that makes them fragile. And we’re changing them constantly. It seems like it’s almost every day that something related to cybersecurity is in the press in some form or fashion. We very quickly moved beyond anything that Hollywood could have dreamed up around this. Show me a bank heist movie that’s as exciting as what’s going on in API security right now.

Scott McAllister: For real. And it’s a constant thing, isn’t it? It never sleeps, it’s always something to keep top of mind. What are some common misconceptions or myths about API security that you would want to debunk?

Rob Dickinson: There’s a bunch. I think one of the things that’s really fun about API security is you can use a lot of the classical things that you know about security, and certainly carry that over. But then, the fact that these are API-based, the fact that this is all programmatic actions, the fact that this tends to focus on public cloud deployments, these are all new vectors. One of those things to keep aware of is, if you deploy your application on the cloud, you’re deploying your application into a known IP address range. There is nothing private about that, inherently. And it really makes me cringe when I hear people say, “Well, we have these APIs, but they’re only for use by our mobile apps.” So, who cares? Your attackers don’t care what the intended use of that thing is.

Part of what that means then is that a lot of the folks that we talk to who are running in these public environments, they see significant number of attackers. The way it used to be back in the old days, you’d put up a website or a web application, and maybe eventually you’d have some kind of security problem. And it was almost like a little bit of a badge of honor, like, “Look, somebody’s really paying attention to what we’re doing.” We’ve run our own experiments where we’ve put applications up into the cloud, and then used Resurface to monitor all of that. And we’re not promoting the app, we don’t link to it from our website or anything like that, these are dark apps. And the longest one took an hour to attack, the average was under 30 minutes. And we see hundreds of attacks per endpoint, per day, by doing nothing more than just deploying an application. It’s crazy.

Scott McAllister: And is that because of the cloud provider that you’re on? Is it because you’re in that known range of IPs like you were saying? Or is it just because attackers are constantly coming everywhere? Would you be more or less secure if you were on a lesser known provider, or your own host?

Rob Dickinson: We certainly don’t have any data to support that. Basically, everywhere that we’ve run a honeypot, a hundred percent of the time folks show up to attack it. It doesn’t really vary that long. And that was one of the questions that we asked ourselves, is this something where you can safely say that one of the cloud providers is far ahead of the rest? I don’t really think you can say that, at least not with the testing that we’ve done.

Scott McAllister: And just in case folks are not security experts, describe what a honeypot is.

Rob Dickinson: A honeypot is basically a fake application you use for the purpose of attracting attention from hackers. It’s the digital equivalent of super-gluing a hundred dollar bill to the sidewalk to see who stops and tries to pick it up, and how desperate they are. And security companies have been using that concept of honeypots for a long time. Let’s figure out ways to get these attackers to identify themselves, and we can profile them. And combined with the recording technology that we have, that’s a really, really powerful way to understand how these folks show up.

Scott McAllister: Are there other ways, because the honeypots sound like a great way to see voluntary activity, you’re putting the honey out there to attract the bees, that’s where my head was going. But the a hundred dollars bill on the sidewalk, that’s brilliant too, because that’ll absolutely solidify in your mind what you’re doing in a physical way. But as far as detecting and understanding the behavior of attackers, what are some other ways you all use to anticipate what attackers are going to do?

Rob Dickinson: The really classical ways of looking at this, which are still valid are, you need a good firewall, you need a firewall that understands your kinds of applications, whether that’s a WAF, or whether that’s API-centric or whatever. But you’re going to need some kind of firewall that’s going to sift out some of the most egregious things. We have folks that we’ve talked to where their firewall blocks 50% of their traffic, which is a huge number. Again, it goes back to that idea that we’re really living in an age of constant attacks now. So, you’ve got to have some kind of firewall.

The biggest downside to that though, is that you immediately get into this trade off, where I can ask my firewall to do more, but it slows down more, and it generates more false positives. And so, you have to find the right balance point with how much latency you’re willing to put on it, and how many legitimate users you’re willing to stomach blocking in order to get that level set. The other thing that we see in here quite a bit is folks doing pen testing. And again, that’s a very classic way of doing this, we’ll sit outside the app, we’ll pretend to be a hacker, and we’ll see what we can come up with. Those are good things to be doing, and I would certainly never argue against anything that has the potential of making your security better. But unfortunately, it’s not just as easy as doing those two things.

And where we really see the greatest need, and that’s why we focused on this with Resurface, is we think the largest unsolved problem today is around observability at runtime. If you think about this in the physical world, I might have a security guard at the front door of my bank, that’s kind of like a firewall, and I may pay someone to see if they can break into the bank, that’s kind of like a pen tester. But I know, in the physical world, I need surveillance cameras, and an audit trail of what’s going on inside the bank. The security guard shouldn’t just wave me through, and then it’s on the honor system from there.

But unfortunately, that’s where we see a lot of inadequate protection today. And if you can’t observe what’s going on, if you can’t go to the game tape like you do in football, you’re really missing one of the best sources of intelligence that you have. So, it’s great to have a firewall, but a lot of the folks that we talk to, they’re not even super clear on how well that firewall is working. It’s great to be doing pen testing, but are you really pen testing each and every release that you do when your development team wants to push that thing as fast as it can? And especially if you’re in a DevOps environment where you fetishize doing 50 releases a day, or whatever huge number you’re trying to brag about, that makes your pen tester and your security auditors' heads explode. How are we supposed to test and re-audit the entire application 50 times a day when you’re changing things that radically?

So, that’s where we really feel like having that surveillance camera and having that audit trail of what’s going on… As a developer myself, I really want to understand what were all the inputs to my APIs, and did the system do the right thing in all those cases? Whether that was a malicious case, or that was a legitimate case, I think in the modern context, that’s the most important thing to understand.

Scott McAllister: That analogy is so vivid, because it instantly makes you think of a bank heist movie. Every bank heist movie, getting past the security guard was pretty simple, figuring out how to get in the building, they got that. But it’s always like, “Oh, how do we get around the camera?” Or, “How do we get around the lasers?” or whatever inside. But think about trying to protect a bank that’s constantly changing. Could you imagine the physical security, because our applications are constantly changing. So, with security, and cybersecurity, API security is a subset of cybersecurity? Or is this just a naming change? Tell me about API security, how have we as an industry gone from cybersecurity and now focusing on API security?

Rob Dickinson: Yeah, I think this really just reflects the overall shift in traffic patterns that we’ve seen on the web. And it’s been such a stark shift. We started noticing this going back more than 10 years now. Akamai for example, one of the estimates that I saw from them was that 87% of traffic flowing across their network was API traffic. You also hear from folks like Expedia, for example, what was historically a web-based travel portal, 90% of their business now is through APIs. A lot of these folks, they still have websites, but those websites are no longer the focus of their business.

Then you see things like the directives from the Biden administration, for example. We should have API- level access to all our healthcare data, specifically calling out that it should not be on a website, that should not be a pdf, that’s not the right model. It actually has to be available in a programmatic way. That’s incredibly powerful. But I think one thing that I wish people more understood is that that is not a futuristic view, that transition has already happened. It is here right now today, and we’re still coming to terms a lot with what that really means.

And I think one of the biggest challenges about this, you’re absolutely right to say API security is a refinement of ideas that have long been in the wind around cybersecurity, but it’s how do we apply that more specifically to what’s going on right now? But I think part of what makes that really hard, that we all need to think about from a technical perspective, you’re pretty technical, I’m pretty technical, for us to sit down and look at some API docs, and maybe pull out Insomnia and make a couple calls to the API as a way to understand how it works, that’s pretty normal to us. But think about when we were based on websites, and web forms, and email-based flows, almost anybody in the company could go to the website, figure out if the website was working right, did the transaction get processed right, did I get the right kind of confirmation email?

Now all this has gone to APIs. And again, this has already happened, but as we’ve done that, one of the problems and challenges around security is that those properties now are dark. There’s a very small percentage of the organization actually even knows how to access them anymore. And of course, you get some gatekeeping from the development organization around that. It’s amazing, when we started Resurface in 2019, we were talking to so many people, and they would be like, “Well, I’m not sure I have any APIs.” And you’d be like, “Dude, I’m on your mobile app right now. I’m pretty sure you have APIs.” And even today, you ask people, “How many APIs do you have?” And they’re like, “Well, I think it’s like a hundred, but it could be 200, or it could be 300.” They don’t know what they have, these APIs are just sprouting up everywhere like mushrooms. They don’t really know what they have, much less who’s using them and how they’re being used.

Scott McAllister: Do they not know because of the way that the application is structured, they’re kind of going with the whole microservices sort of architecture? And so, they’re just like, “There’s APIs everywhere, and they’re doing all sorts of things, and so we don’t know everything that’s in there.” Is that what’s going on? Where’s that disconnect?

Rob Dickinson: I think there’s two things. I think the first thing is that a lot of folks are still mid-flight in their journey towards really being API-driven. So many of these digital transformations are not finished yet. And so, a lot of these folks, they’re still focused more about the construction phase. And most security people, if not all the good ones, would say that you really should be considering security at that stage from the beginning. But let’s face it, it’s hard. When you’re building something new and you don’t even know if anybody’s going to want the thing, and if anybody’s going to use the thing, you don’t have any evidence of product-market fit yet, how much of an investment do you really want to make in hardening and securing all of that? You’re probably going to fake it till you make it a little bit. And again, the biggest risk is just that those attackers are going to show up much earlier than you thought.

Scott McAllister: Right, before your paint even dries, they’re going to be there. If something’s online and they can find it… interesting. On the topic of APIs, and in development today REST APIs are pretty common, but also GraphQL is making a charge. Does security change when you start dealing with GraphQL as opposed to REST?

Rob Dickinson: I think it does, yeah. It’s funny, because if you think about it, GraphQL is like SQL injection done on purpose all the time. So, from a security perspective, it’s kind of a nightmare scenario. If you ever have the chance to describe to a security person who doesn’t know what GraphQL is, and you give it that introduction, “Well, it’s kind of like a framework built around command injection,” they react with horror, like, “Why would you possibly want to do that?” And I’m bullish on GraphQL, I like GraphQL, I’m not a hater. It has some really wonderful, amazing qualities to it, and that’s driving its adoption.

But from a security perspective, yeah, it’s a bit nuts to be doing that. And I think that’s why you still don’t see a clear consensus from the community about where GraphQL should be used. From what I’ve seen, still about half the community says GraphQL is too dangerous to expose outside, so what you’re doing outside should all be REST. So, all your north-south traffic should be REST, but then once you’re inside your enclave, then you can do GraphQL within that layer all day long and you’re safe. But there’s still a bunch of folks that are really exposing those GraphQL APIs in a public way, and really stumping for that.

But there’s some big challenges. How do you properly protect that so that I don’t send you a batch with a million GraphQL operations in there, to where it takes minutes just to parse the batch? And I can construct a denial of service attack just around garbage like that. It’s a whole new way to get infinite loops, and all kinds of resolver issues that you might not have thought of. The thing that’s I think scariest about security is that so many of these security things, they end up being failures of imagination. It just didn’t really occur to you that someone would do that, or that that condition could exist. And that’s partly why these things can go undetected for so long. So, when you have a very powerful framework like GraphQL that’s very flexible, and really hangs its hat around those qualities, there’s always going to be a downside. And I think unfortunately, security is a little bit of a fly in the ointment there.

Scott McAllister: Are there things to think about, if they’re going to adopt GraphQL and it’s going to be public facing, it’s not going to be the internal stuff, you were saying that it could definitely open opportunities for infinite loops and things, and having DDoS attacks. Are there other things that people should keep in mind, that they should say, “Okay, we’re going to adopt GraphQL. From a security perspective, let’s do this and make sure that we’re staying safe”?

Rob Dickinson: Yeah, I have a non-sexy answer to this one, which is watch the research, watch the literature that’s coming on. There are a lot of people writing their masters and PhD dissertations on GraphQL. It is the new kid on the block, it is getting a lot of attention from that perspective. And there are new ways of exploiting GraphQL that people are finding all the time. And partly just because that’s new. And I would expect, over time, more of those protections will be baked in automatically to those platforms. But it’s a little bit of a wild, wild west right now with GraphQL. And that’s fine, I think the folks who are there, they know it, they’re proud of it. So, go to the conference, I’m here at API World this week, and there’s been a couple talks specifically about security and GraphQL. You want to make sure that you’re tracking to what those latest attacks look like.

Scott McAllister: Which sounds like the pattern that we take in our industry. As things progress, something new comes on the scene, we keep up with the research, we keep up with the changes, and then we make adjustments, and then we research some more, and then we make some more adjustments. That sounds like a standard way of attacking things or approaching things. Which leads me to my next thought and question, what’s the deal with bot detection and API security?

Rob Dickinson: Oh, that’s one of my hot buttons. It’s funny, you still see security vendors that talk about bot detection, and they’re trying to apply that thinking to APIs. And I would argue from my comfortable, comfortable soapbox, that’s really misplaced. With APIs, everything is a bot. Where that thinking comes from is from web-based operations. We used to partition our web traffic into real users using real browsers that supported JavaScript and the libraries that we needed and all that stuff. Those were the valid users, the trustable users. And then, anybody who wasn’t using a real browser who was a bot, was either a robot, or a scraper, or a spider, or maybe an attacker. But it was pretty easy, based on the type of client you could make that judgment call.

And of course, that line was never perfectly delineated, of course you have hackers that drive headless Chrome as the way that they do their attacks, and that level of impersonation. But at that time, there was so much investment and thought put into, how do we separate those real users on the real browsers from the bots? They’re likely to either be malicious, or at least something that we don’t want to be optimizing the experience to.

And I think this is one of the biggest things to really get your head around. If you want to be Twilio, or SendGrid, if you want to do banking as a service, if you truly want to be open in that regard, you will not have any concept of a trusted client anymore. You may have your own client that you produce that you’re very proud of, but it’s not like mobile apps, where that’s the only way to consume the service. If you’re trying to do banking as a service, for example, you want lenders, you want integrators, you want other kinds of software to be able to use those services, and you’re not going to underwrite the development of all that stuff, you really do lose that concept of trusted client. With it, you lose a lot of those security strategies that you could apply if you control the client, and then you could control the interaction with the server. It’s a fundamental rethink in terms of your separation of concerns and where you have to be enforcing those policies.

Scott McAllister: Yeah, APIs can’t trust anyone. Every request that comes in, you need some sort of authentication and authorization to identify that the request is coming from a trusted source or a trusted client, as you mentioned.

Rob Dickinson: And even that, honestly, is under threat. One of the biggest vectors that we’re seeing is that attackers sign up as paying customers. Because they know when they do that they’re going to breeze right through the firewall. Even if you are in a country that has Know Your Customer laws, let’s say you’re a bank and you have Know Your Customer laws… I made that statement to a CISO at a major bank in Australia, and he said, “Well, I’m not sure I believe that, because we have strong Know Your Customer laws and an attacker can’t just sign up on the web, they can’t just create an account and be in.”

And so, we walked through that scenario. It was like, “Okay, well I’m a criminal gang, so to get around Know Your Customer laws, all I need to do is I need to recruit young people with clean criminal records. They go, and they show their proper ID and everything, they create bank accounts. And then I use those bank accounts to do privilege escalation attacks against other bank accounts.” Try to find those people. From a statistical perspective, from a numbers perspective, those people are there. And that’s not a difficult kind of attack to carry out.

Regardless of what you think about, we can identify our customers, we know our customers, they had to show… regardless of how high you try to set the bar there, your attackers are engineers too. They know that those protections are there, they know about those policies. And again, think about the heist movie. What’s the first thing they do in a heist movie? They go and they open an account at the bank, and they get a safety deposit in the bank so they can get in the vault and they can look around, they can see what the security looks like. You can’t tell me that you wouldn’t run a bank completely differently if half the people that showed up were carrying AKs and there to rob the place. You would have to do things dramatically different. That’s where we are with the APIs, we’ve gotten a little bit ahead of ourselves in this journey, but the attackers are really coming on strong. And that’s why you see this in the news on just about a daily basis.

Scott McAllister: For real. That image of thinking about all the customers being potential threats every time, that’s a memorable one. So, tell me about Resurface. What do they offer as far as a security protection on APIs? And how is it different than others?

Rob Dickinson: So, we’re not a firewall. Most of the folks that we work with have firewalls already, and we don’t want to have to rip and replace that. What Resurface is, is a surveillance camera that sits behind your firewall. And so, we watch all the traffic that actually makes its way to your API endpoints, and then we see how those API endpoints actually respond. And that’s actually the most interesting part of this. If you see an attack, and the attack generates a 404 because it basically bounced off, well, who cares? Ignore that, put that in a separate bucket. But if you see an attack and the system responds with customer data, you know there’s a serious problem. Being able to put together both the request and the response, what we’re doing is we’re creating a data lake out of all of that API traffic that includes all those details about the requests and response, so you can look at does the request represent a kind of attack or a kind of threat? What did the system do in response to that?

And the best thing about what we’re doing is that all of our competitors have a model where they’re a multi-tenant SaaS, and so the first thing you have to do is you have to send all your data over to their SaaS. And the little security person in me freaks out, like, “Wait a minute, if I literally have to clone all of my traffic to some other group, and their employees, and their insiders, and their supply chain, didn’t I just double my problems?” And I don’t mean to impune anyone. If you were starting a company today and you weren’t doing a SaaS, you probably should have your head examined. We actually started as a SaaS, and we just ran into too many people, especially in the verticals where we’re operating, in healthcare, in financial services, and in government, where that was one of the first questions they asked, was, “Where’s my data going?”

So, it’s harder and harder every year to ship all this data around all these third parties. Meanwhile, it’s cheaper than ever before to run your own docker containers wherever you want to run them. So, we basically took the Docker and Kubernetes stuff that we had been building to develop our SaaS, and we just released them as containers and Helm charts. Then our customers, they’re basically able to deploy, this is a true monitoring sidecar alongside the APIs that they’re already running, and all the data’s captured on that cloud, in that legal jurisdiction, under your first party control. There’s no data caps, because we’re not having to provide all that over the wire, so you can select star out of the entire database if you want to, you can chunk through all the data if you want to, it’s your data. You don’t pay any data transfer fees from having to mirror all that traffic somewhere else, and you don’t have to sit down with your security team and your legal team and fill out a whole new set of security disclosures around the third party data processing aspect of that.

It’s really been wonderful. It’s been a very atypical way. Like I said, anybody who isn’t building a multi-tenant SaaS right now, you really have to answer the question of why not? But how we got there was just by following the breadcrumbs. And in the end of the day, our software is going to run on AWS just like anything else, but not having that data handoff. And it’s funny, when I tell people about that, I’m always cognizant of, “Did you just do the engineer thing? Do people really care how many cylinders in the engine, or they just want a car that can go fast? Do they really care what’s under the hood?”

And it took a long time for us to really prove this out in the conversations that we’ve had, and the pipeline that we have built. But people are more sensitive than ever before about where their data is actually going, how long it’s going to be kept there, who’s going to own those datasets in the future. That really has become one of the great philosophical challenges of our time. And I think we’re on the right side of history, and I’m okay being the only API security vendor that postures like that right now. A lot of our feature sets are very similar to what you get with the cloud-based guys, but we’re not making you pay with your data. For everyone that we have in our pipeline that we’re working with, that is number one or number two issue on their list.

Scott McAllister: Nice. Yeah, that makes a lot of sense, as far as keeping data where people want to know where it is, just like you said. So, what’s one thing you wish that you would’ve known sooner about API security?

Rob Dickinson: I wish I had recognized sooner the new threats to be worried about are not the script kiddies. This is not attackers a generation ago, where you had these hackers that would run scripts that they couldn’t write themselves. It’s not the old days of, “We’re just going to port scan the Pentagon looking for unsecured SQL server instances that have a known vulnerability.” You still hear folks hanging on to that thinking, and I think it holds you back.

The modern hacker is not that. The modern hacker has a background in software engineering, they work as part of a team, they’re well compensated, they’re highly organized, it tends to be collaborative. This is big business. The reason why this is showing up in the news every day is that there are real dollars being attached to this. This is the dark side of crypto. I don’t think cyber extortion as we know it would be possible without something like crypto. And unfortunately, that’s one of the things that’s driven the growth of this. But attacking these APIs, that is a growth industry. They are well funded, and they take what they’re doing just as seriously as the folks on the other side that are trying to build and maintain those systems. So, do not underestimate your attackers, they are ferocious.

Scott McAllister: Indeed. And another recurring question we like to always ask is, what are you glad that we did not ask you about API security?

Rob Dickinson: One of the questions that I get all the time is put like this, “Now, Rob, you’re a developer. Tell me, why don’t developers care about security?” And it just kills me. I don’t know, because we’re not incentivized to? We’re not trained on it, and we don’t see those systems in production. The security team does that, or the NOC team does that. The network ops team, they’re running the thing in production, and I don’t have access to that thing in production because we have proper compartmental boundaries in our organization. I should be able to create the app, but I shouldn’t be able to get to it in production. You have that kind of proper thinking about these things. So, as a developer, what do I see? Well, I see how the code runs on my laptop, I see how the code runs in my staging environment, which is relatively clean, and then it actually goes out in production, and I don’t know what happens.

And then if you tell me that half the users that show up in production are not there to use the app, they’re there to violate the app and use it in unexpected ways, it’s even a more dangerous footing, because I don’t see how those folks normally show up, because I don’t normally see those environments. So, I always cringe a little bit when I get that question, “Why do you guys just want to build all this stuff and you don’t care about securing it?” Where’s the incentives? Where’s the feedback loops that make me do that?

I have a QA team because we need somebody in the organization to call bullshit when what development is putting out has problems. And you have to intentionally build that feedback loop. If you want to get developers to care about security, you have to intentionally build that feedback loop and find a way to do it. And again, not to toot our own horn, but I will anyway, that’s certainly a focus that we’ve had, taking that data from production, and bringing it back into pre-production in safe ways where user identities are protected, and production-level data is protected. But trying to help answer this question, how do you design all these systems to be resilient when you don’t even see what those attacks look like on a daily basis? That’s holding those designers, I think, to an unacceptably high bar.

Scott McAllister: And so, it’s got to be a teamwork thing, right? You have the people who build this stuff, but then you also have your security experts involved early, so that way you make it usable, fun, but also secure.

Rob Dickinson: Yeah, absolutely right. The most advanced organizations, I think, they’ll level-set around security, and then they’ll try to make sure that things never get worse than that. The same way that you try to level set around quality, and then you say, “Yeah, we’re still going to have problems, we’re still going to have issues, but we’re never going to fall short of the standard of care. And then we’re going to try to ratchet that standard of care up over time.” But you’re right, and that’s one of the big challenges of this, is that it’s cross-departmental, it’s not just the security team, it’s the dev team, it’s the QA team, it’s technical support, it’s anybody that handles money. That turns out to be an awful lot of people at a tech company that’s API-first.

Scott McAllister: Absolutely. Rob, this has been an awesome conversation. Thanks for coming on the show today, and we appreciate you taking some time, especially while you’re out at API World.

Rob Dickinson: Thanks for having me. It’s my favorite topic to talk about, so always down for that.

Scott McAllister: Awesome. We appreciate it, a lot of good knowledge shared today. Thank you all for joining us, and this is Scott McAllister, and I’m wishing all of you an uneventful day. That does it for another installment of Page it to the Limit. We’d like to thank our sponsor, PagerDuty, for making this podcast possible. Remember to subscribe to this podcast if you like what you’ve heard. You can find our show notes on pageittothelimit.com, and you can reach us on Twitter at pageit2thelimit, using the number two. Let us know what you think of the show. Thank you so much for joining us, and remember, uneventful days are beautiful days.