Rob Dickinson, Resurface Labs

All right, here we are, another episode of Breaking Changes, where we dive into the latest API issues that business leadership should be thinking about. I’m Kin Lane, your host of Breaking Changes. I’m also the Chief Evangelist at Postman, and I’m working my way through many different companies, brands, and business sectors to find the latest folks who have the knowledge we’re looking for when it comes to the business, the technology, and the politics of APIs. Today I have Rob Dickinson with me from Resurface Labs. Rob lives and breathes APIs. He’s worked at Intel, Dell, and Quest Software, and he’s really tuned into the layer of APIs that I find fascinating, helping us see what’s happening in this very abstract layer, bringing more observability to what’s happening. Thanks for joining us today, Rob.

Thanks so much for having me. Pleasure to be here.

This is great. I really like what Resurface is doing, because for me APIs are everywhere. I see them everywhere, but for many of us, and especially the more APIs we have, with hundreds or thousands of APIs now, it’s really difficult to see what’s going on. So to get going here, let’s start with the basics. What is Resurface Labs and what do you all do?

That’s a great place to start. What we’re really solving for is the need for observability around APIs, especially for companies that are going through digital transformations where they’re moving assets that used to be website-based to now being API-based. That’s something we’re hearing over and over again. A lot of that is taking APIs that used to be internal and also exposing them to outside parties, so that could be customers, partners, or integrators. In some sense, it’s really the erasure of the traditional boundaries and perimeters that we think about. With a truly API-first architecture, there isn’t really a concept of inside and outside anymore, or at least that concept is significantly degraded and fuzzy compared to how it used to be. That’s really what we’re doing with Resurface. There are a lot of monitoring tools that are well geared toward system monitoring or website monitoring, but not as many yet that really have that API-first perspective, and that’s where we’re really trying to lead with Resurface.

A much needed view on the space. So to kind of set the evolution here, websites have been evolving for the last 20, 25 years. We’ve got analytics and all of that, and then around 2010 we saw a lot of API management providers step up with reporting at the management layer: who has access to your APIs, what are your error rates. How does this observability layer evolve those concepts?

As somebody who’s been in APM and observability, web monitoring, for a long time, it’s really about the data. It’s really about the databases that are behind these things. When you roll back the clock and look at APM 10, 15 years ago, the way that we did APM is we would record these traces and fit them to a statistical model, and you would throw away a lot of the fidelity. Big data hadn’t happened yet, so the idea of being able to keep every trace, to keep a database of everything that had happened, was really not practical, certainly not affordable in the way that it is today. That’s really the shift. You look at APM vendors that really focus on observability, like Honeycomb for example. What that means is they’re storing all of their traces and keeping them in a database where you can go back and slice and dice that data after the fact. We’re doing the same thing with Resurface. We’re sitting up one level from the system. We’re not really looking at system health, we’re looking at the actual API traffic: here’s the input to your API, here’s how your API responded, and creating a database out of those conversations. For me, observability is the process of doing that, but doing it while keeping all of the fidelity as much as possible, the fidelity in all the original traces. So you’re not down-sampling to a statistical model, you’re not down-sampling to a host-based model, you’re not down-sampling to a data-flow-driven model of who’s talking to what. You’re actually trying to record all the conversations. One of the analogs in the physical world that we’re used to seeing all the time: if I call my stock broker and I buy stock over the phone, you hear that robot voice that says your call may be recorded for quality assurance. That’s an observability solution. It’s the need to keep a record. If I’m putting down my credit card, if I’m doing an important business transaction, I expect there to be a receipt for that somewhere. Somebody needs to have a receipt, just like it would in the physical world. We’re just helping API-driven companies catch up to that message.

So as we learn more at this layer, we can always go back and check and build upon the knowledge we have. We have all of the receipts to be able to do the accounting that’s needed to move forward in an intelligent way as we learn and evolve and as we see this layer.

Absolutely. One of the things that’s so fun to work at Resurface and help people through this process is when you see somebody for the first time who now has this level of understanding about what’s going on. Almost immediately people start saying, well, why is that happening? What’s going on with that? Why is that returning that? What’s the case where this is happening? It’s funny, it’s something that we hear over and over again: I didn’t really know what I didn’t know about what was really going on. Even though we know that, it’s just like if you’re a property owner and someone comes onto your property and breaks their leg or gets hurt, you’re kind of responsible even if you weren’t. It’s the same thing being an API provider. How could you not be, especially if these are business-facing transactions or revenue-bearing transactions? It’s sad but true that a lot of the folks we’re talking to will say something like, we’ve got great coverage for our web properties, but for our APIs we’re really guessing. We’re relying on our customers to tell us when things are broken. That’s just not a great place to be.

You guys use a phrase, “API system of record.” It caught my attention as I was looking through your materials. What does that mean in this context and how is it helpful?

What it means is that we’re really creating a database, a data platform that keeps those records and keeps them intact the same way that a court reporter would try to keep a record intact of what’s being said, your stenographer. The other part about that term is we’re trying to position this in a way that people kind of have a sense of what it is, and also the sense that they don’t already have one. When you talk about, oh, we’re a logging solution or a monitoring solution, it’s really easy to say, oh, I’ve already got that, I’ve already got New Relic, I’ve already got Prometheus, I’m already monitoring the hardware, I’m all good. To take that up a level is to say that there are lots of cases where the systems may be working just fine but they’re doing the wrong thing. The outcome is wrong. They’re available, they’re performant, they’re responding, but there’s a percentage of cases where they’re not doing the right thing. They may be losing the company money, they may be really irritating people that are trying to use them, but you look at your traditional system monitoring tools and everything looks green. That’s kind of how we got on this track. My first company was more of a traditional system monitoring company, and we started seeing all of these cases. Another way of saying “system of record” is that there are all kinds of databases in the world: OLAP databases, OLTP, graph databases. Salesforce is the database for your sales process. But there isn’t really a highly used database today that’s purpose-built for API traffic, that really natively understands those concepts of user privacy, user consent, security, that understands things like JSON payloads and GraphQL payloads. So what we’re doing as an industry is we’re taking big data tools and trying to overlay all those concerns on top, and of course there’s a lot of reinvention and cost that comes with that. We believe it’s inevitable that there will be purpose-built databases for this observability problem specifically, and we think Resurface is really the first one that has a chance to make a lot of noise in that regard.

It’s interesting because observability has been talked about pretty loudly for the last couple of years. It’s a pretty well-known concept. I don’t think it’s anything new, it’s just the new way we’re talking about it. Look at traceability, it’s definitely along those lines. But your approach to the database piece, I would say, adds a kind of provenance to the whole reality of this that I hadn’t really seen before. Now not only are things observable and traceable, but you have that backwards in time, so you’re able to connect the dots and make sense of things.

Yeah, and that’s something that we’ve really focused on as a key requirement that’s substantially different from really any other kind of logging system that I’ve ever built before, including the things we did in our previous company, which we used at some giant accounts. With a lot of logging and monitoring systems today, you’re making a lot of decisions up front about what is the data you’re capturing and what are the specific signals you’re looking for. You’re saying, I’m going to do things based on response code, I’m going to look for these specific data elements. If I’m doing security, for example, I’m going to plug in my web application firewall so it’s protecting me against certain data patterns that are malicious. At the end of the day, though, you don’t know what you don’t know, and we know that zero-day things are happening all over, zero-day failures as well as zero-day attacks. That’s really what got us onto this idea that you need to be able to bring all the data into the system of record and then be able to apply analysis rules that retroactively apply to all of the data that you have. So when a customer reports a problem and you’re hearing it for the first time, or when there’s a new attack vector you’re considering, you can very easily apply that retroactively to all the data you have. The next question you’re going to have as soon as you find something new is going to be: how often has this been happening, who’s it been happening to, how severe is it? With a real system of record you can actually connect all those dots. You can say, these are the people who were actually impacted by this failure, this is what they were trying to do, these are the payloads they were trying to send, these are the transactions they were trying to complete, and we can now go back in time and complete those transactions because we have the record. That’s really our origin story around this whole thing. If you’re an APM product or a system monitoring product, you can detect a problem but you can’t detect who was affected. That’s really what got us going with Resurface: the idea that I can’t just find a problem, but then I can do a query out of the database and it’ll tell me who was actually affected by that problem, and then I can go back to those customers and actually make it right.

That’s pretty rich. I could see this really having a business impact. You mean security, and brought this into the security side of it. When I talk to a lot of folks about their APIs and security, they’re like, well, we have MuleSoft, or you have to have a key to get access to our API, so thus it’s secure, people can’t access it. And then the second part of that evolution, that was 2010 through 2016, 2016 through 2018-2019 they’re like, okay, we’re scanning for OWASP Top 10 vulnerabilities, we’re making sure there’s no holes. So you have to have a key and there’s no holes getting in. But what you’re telling me is there’s a lot more awareness that can be harvested at that layer over time, and as we progress, retroactively we can look back and see how secure we’ve been or not, and actually understand how we got here.

Yeah, and really what you’re getting to there is it’s a fundamental shift in what we consider private versus public, what we consider to be gated access versus ungated, or even what that gating really means. If you’re a bank, for example, and we’re talking to a couple of banks, they’re moving from website-based properties to more of APIs, and in the process they’re expanding their surface area. They’re expanding how much of their systems are actually exposed to the outside world. It’s not just the customers now, it’s the partners, the lenders, all the other vendors, everyone trying to resell loans. There’s tremendous value from the business side. There’s tremendous value with the more that you can open up, the more opportunities there are to monetize those systems. That’s incredibly attractive, to think about taking your existing systems and getting more money out of them. But what I think is fundamentally different in what I’ve seen over the time frame you talked about is how prevalent these attacks are and how that really changes the nature of how we have to think about our systems. When I first started doing websites and web systems 20 years ago, attacks were relatively rare. They were kind of ornate when they happened. It was interesting when they happened, it was novel. Now it’s a constant state of life. Half of your traffic or more is going to be bots or malicious actors in 2021, and that’s just the state of things. It’s not going back to where it was. The hackers are engineers too. Some of them are state funded, they’re highly organized, and they’re serious about advancing their craft as much as the white hats are about securing theirs.

So think about it this way: how would we map that to something in the physical world? Let’s say you’re a bank running a physical branch office. How would you think about your bank differently if you were thinking about getting robbed once in a while versus thinking that half of the people who walk into the bank are there to rob you? You’d have to conduct business very differently in that scenario. Unfortunately, what we see too much today is people running their banks without the surveillance cameras inside, the observability solution. They’re missing that part. They’re really just sticking to perimeter security, and the perimeter security is, you’ve got a bouncer at the door trying to look at people and see if they’re there to rob the place or not. Is it really any surprise that cybersecurity is kind of in the state that it is? I don’t mean to impugn anyone in saying that, but I think we’ve taken that idea of perimeter security and inside versus outside, and that being a binary flag, can I trust you or not, and we really have to move beyond that to a much more contextual notion, the same way that we have to talk about privacy in a very contextual way. If I’m just walking around a store, maybe I feel weird about that being recorded and I might have an issue with that, but as soon as I walk to the register and put down my credit card, I want a record of that transaction. I don’t want that to be anonymous. I want there to be a permanent record of that. So all these notions of privacy and security have become a lot more muddy, and the one-size-fits-all solution of perimeter security will just keep all the bad folks out. You can see the appeal. It sounds easy, it sounds very reasonable. But think about that bank analogy: if half the people are there to rob you, you’re going to have to shift your thinking about your operations. Well, how does that apply to all your bank employees and your bank partners and your bouncer at the front door who says, hey Bob, how you doing, walk in, I’m not going to rob the place?

Zero trust, back to what you were saying earlier. I think this is one of the biggest lessons out of the web, the website world. There was still kind of, and even mobile, this thin veneer of WAF, firewall, all right, we’re protected, our data and resources are behind this website, we’re behind a mobile app. The number of people I talk to who are like, do you have any public APIs, and they’re like, oh no, no. And I reverse engineer, I download their mobile application, run it through a proxy, print out their entire API surface area, and go, well, you got a couple hundred API endpoints here, how are you securing those? And they’re like, where’d you get that, did you hack our app? No, it’s public APIs, you’re using public DNS. They’re like, well, we secure our mobile app. I’m like, great, that’s good, I support that, but how are you securing your APIs? Or if you’re not even seeing these APIs, I came in from outside and showed you these APIs, what sort of observability or traceability or system of record do you have for activity at this layer?

Absolutely. That’s the Peloton attack, right? This is happening quite a bit. It’s funny, I’ve had the same experience. You’ll meet someone and they’ll say, what are you doing? Well, we’re an API monitoring company. Oh, that’s interesting, maybe we’ll have some APIs someday, that’s really forward-looking. And then you say, well, do you have any third parties that do any integrations, or do you have a website, or do you have a mobile app that calls back in? And they’re like, oh yeah, of course we’ve got a backend that our mobile apps use. It’s like, oh, well, there you go. People just don’t see.

And “see” being the critical word here. I think this is the part of observability: if you don’t see it, it doesn’t really exist. There’s so much in the digital realm that we don’t see on a daily basis, so thus it doesn’t exist. So the more observability the better. This was the balance that I saw with APIs. I’m an old database guy, going back to the 80s, and when APIs, service-oriented architecture, but then once web APIs in 2004, 2005, I saw the externalization of what was very much a power center in a traditional org. We were poking holes in that and opening it up to partners and even public, and that changed the business game. It gave us a little bit more agility and flexibility to do things, but it was striking a balance between access and control over our digital assets. We were making some trade-offs, opening up the access so we could move faster and do things. So how does what Resurface does contribute to this agility and this ability for a business to move faster?

That’s really what it’s all about in the end. What we’re really trying to move the needle on is the idea that when there’s an attack or when there’s something wrong, especially around security, the outcome should be to reinforce the perimeter. It doesn’t work. The better outcome to get to is to improve the system to better resist, or improve the system for better quality. All these are ultimately design problems. If technologists created all these problems, it’s up to technology to solve them. And again, that lack of observability even hinders perimeter security if you’re trying to do it. One of the things we hear a lot is not a lot of confidence that the web application firewalls are actually configured right. You’ve got all the rules that you feed in. Who’s keeping those in sync with the applications as the applications change? That’s a whole job unto itself. The WAFs historically don’t do a great job of logging everything with enough context to really understand: here is the thing that got blocked, here’s all the things you know about it, in case you want to revisit what that is. I’m shocked by the number of folks I talk to who are actually running their WAFs in non-blocking mode, just reporting mode instead of blocking mode, because they don’t feel like they have a great continuous improvement process around it. And they’re also terrified of the false positives. The risk or the damage of a false positive is almost seen as more immediate a threat than a bad actor getting in and somehow figuring something out. So what we’re really trying to do with Resurface is provide the data and insight to really drive that continuous improvement engine. With something like Resurface, we’re going to tell you on a daily basis these are the top things you should be looking at to improve your quality and your security. There’s probably all kinds of other things going on, but it’s about being able to whittle that down to these are the most severe kinds of failures and attacks that you really should be responding to. The response to that could be to improve your WAF, maybe you do just want to block that, or block the source, or block it at the perimeter. Okay, well, now you actually know what the cases are that you want to block against, and you can detect a recurrence of that. If you think you’ve fixed your firewall and there’s still cases where that’s coming through, let’s now have the records that we need to go back and harden that. And it’s exactly the same workflow if you have a failure or you want to change the behavior of the system. One of the analogies we use: let’s say your system is seeing a lot of SQL injection attacks, classic OWASP Top 10 kind of signature. What makes more sense, do you want to block the actor, or do you want to harden your system so that your system just ignores that bad stuff and doesn’t leak any data? Sometimes it’s an account management response that comes out of it. It’s, hey, did you know this customer is still hanging onto that deprecated API? Can we get them off of that somehow? So it’s not just about being a better bouncer and managing the perimeter better, although we can contribute to that. The narrative we’re following is, once you can see every input and every output to the system, and once you can start to separate signal from noise, these are the attackers, these are the proper users, these are the failures, these are the known signatures, these are zero-day, you can really turn that into more of an organizational mission than just trying to play whack-a-mole at the security level.

We’ve been heavily focused on the security aspect of this, the dangers, and we can easily get caught in this trap where we see things as dangers and, oh, block that person, block that. And you’re saying we need to be more aware, make the system more resilient. There’s also a lot to learn at this level. One of the early API conversations that got me really excited about the potential of APIs, I can’t remember their names, but there were two Swedish brothers who started what became Google Maps. The company that did Google Maps got acquired by Google, and they had that JavaScript API that you could embed the little maps. They were designing the API for Google Maps at this point, and classic technology, they designed the API, but then they started tuning into how people were hacking the JavaScript embeddable, tuning into the parameters they were using and the way they were doing it. They noticed there was a bunch of really crafty people that could have been seen as a threat, but they were actually doing very interesting things, and if they just had a more robust API, these could be partners, these could be people who could do interesting business outcomes with the API. So they just ended up watching this activity for like 30, 60 days, and then they designed the Google Maps API. So it’s not just about threats or what we perceive as threats. You could develop all kinds of other awareness at this layer as well.

Absolutely. So many of these problems really start at the design phase. When you think about the whole software development life cycle, you’ve got developers that are trying to do the right thing, but what they actually see is the code running on their laptop, especially pre-DevOps. As a developer, I never got access to production systems. That started to shift with DevOps, but traditionally development’s kind of in the dark. You’re doing your best, but you have very limited raw information about what’s going on. You’re relying on your product management team, you’re relying on your executive team to fill in all that. QA is doing the same thing. They’re trying to do the best thing they can in isolation. They’re working off the requirements, off the latest builds, but so far everyone’s just guessing. Nobody actually has any raw intel. How often is it that developers actually sit down and do an end-user survey or study on their own and see the first-party data? Do your QA people actually get to interview your customers and find out what they’re doing and what they aren’t doing? No, they’re doing their best, and they’re trying to gatekeep on that information as much as they can and be responsible for representing that. But what’s the raw intel that they have? It means an over-reliance on customer support, account management, and product management to represent the customer, and you’re playing telephone.

Exactly. To your point, it’s entirely different when you’ve got an open way to gather and inspect and discover what those behaviors actually are. It’s such a key part of the general design phase. You’re supposed to design something, put it in the hands of someone, and watch what they do, and you’ll learn a lot out of that that maybe didn’t occur to you. We don’t have enough of that kind of feedback in the software development life cycle in general, I think.

And not to make this all about security, but security is the one that is the most obvious. You’ve got your security people running around saying we need to have our developers care more about security. What does that really mean if you aren’t giving them anything? If all that I see as a developer is what works on my machine and I never see what an attack actually looks like, I never see how the system actually responds to that kind of attack, can I really design with that in mind? I can do my best. Like I was saying earlier, that’s been the most fun part about Resurface, literally watching that moment where people grasp, oh, I kind of had no idea this was going on, and going on in volume. And now that I’ve got this flight recorder, this system of record, suddenly I can see it. And how often those things are relatively trivial, like, oh, I could just go file a ticket for that right now and get it fixed. And you come to find it’s been broken for months.

So there’s a whole DevOps shift-left aspect to this observability layer that you’re introducing. It’s not just me, the product lead or command-and-control. This is something my entire team can have access to and learn from and grow and evolve based upon.

Yeah, and it’s role-based. So it’s not like you’re racing to erase all the boundaries within the organization. There still is certain kinds of information that needs to be partitioned off. Certain kinds of data needs to be protected more than others. There’s no real one-size-fits-all solution to this. Really it’s things like Hyrum’s Law that kick in. You do your best to design something, just like you were saying with the Google Maps analogy, you design something, put it out in the world, and a lot of times you find that the way it’s actually used is slightly or substantially different than what you expected. The users don’t have exactly that priority in mind, they see a different potential, and they end up shifting their usage patterns to that, whether that’s good or bad. Sometimes it’s bad. But what’s really bad is if you really don’t know. It’s about gaining a deeper level of understanding, and doing that in a way that’s safe and responsible and still fits within those traditional roles. We still expect that a security analyst is mostly interested in security and developers are mostly interested in building new things and making those things better. The trick is really to map to the existing processes that are already there within the organization. I’ll pick a security example just because it’s an easy one. One of the examples is, instead of just asking your developers to care about security, let’s put a code scanning tool in as part of your CI/CD pipeline. Now it’s not an extra thing that I have to care about, it’s a quality measure that’s going to be applied as part of my continuous integration process. We see Resurface very much in the same vein. So Resurface is kind of like a linter that’s running all the time in production. We’ve got hundreds of inspections that we’ll run against your APIs all the time as new data arrives. Some are around security, some are around quality, some are around usage and profiling. And then if you’re a developer, you’re going to get notifications in Slack or Microsoft Teams, where you already live: here’s some top-of-mind things you should be paying attention to. If it’s security, we’re going to fit that to your existing SIEM or stack. The way to really get this on a good footing is you want to find a way to introduce these things as very incremental changes on top of the processes we already have. It’s not that I want you to care about security, it’s that I want to be sure you’re spending time on security every day. I want it to be something you’re thinking about. I don’t want that field to go untended. Once we get into that continuous improvement mindset and can drive that kind of continuous improvement workflow, things just get better and better, and if you can tie very clear ROI to that, then you can really get somewhere pretty quickly.

It sounds like there’s a nice healthy dose of drip, drip, drip literacy and education and awareness, whether it’s security or other aspects of our API operations, that we can expose the whole team to, or a specific set, and notify them and alert them based upon their existing world. They’re not just immediately bombarded with security flashes that they know nothing about. They’re eased into it, and with that awareness comes the ability to actually do something and have it impact how they actually do business. I like that. That education and awareness, because you just can’t turn on everyone caring about security overnight. You have to bring it home in some way. This is, I would say for APIs in my realm, how do I get people to care about and see APIs? Because I’m constantly saying APIs are beneath everything, our cars, our television, and then I look at people the way they look at me. I’m at the barbecue at someone’s house and they’re like, this guy’s crazy, he’s just everywhere, what’s going on with him? I’m like, no, no, they really are. No one’s going to care about APIs until there’s something there that is meaningful to them. So from a DevOps operational standpoint, something that impacts their job and makes them better, saves them work, stops them from getting yelled at, or a freakout session because of some security thing that they did or did not do. I think that steady drip of knowledge and information is super important, so I like the approach. I think it’s healthy.

So moving out of DevOps, there’s regulatory and compliance benefits here, I can only assume. I’m sure you’re in that business of helping alleviate the auditors and the government from being up in our business as well, right?

Yeah, and again it’s the easy way to get compliance, to find a way to do it that fits with your existing processes, your existing workflows, your existing tech. GDPR compliance is one of those good examples where it’s easy to say, whether it’s GDPR or CCPA or the new Chinese version, oh, that’s a great idea. But I can’t just write a check and be GDPR compliant. I can’t just buy a thing and be GDPR compliant the way I could buy a radon detector for my basement and be assured I’m in compliance with my environmental hazards. It’s not that easy. When you look at the landscape today, if I were to go to you as a technologist and say we’ve got this massive API at scale, now we need to record all this stuff for governance purposes, that sounds like a multi-million dollar project. That sounds like big data, Hadoop is probably in there, Kafka’s probably in there, Spark’s probably in there, you’re going to have to hire people. It’s going to be a total sidecar to whatever it is that you’re already building, unless your core business is that kind of compliance, and then it’s all going to be extra. Then you look around and say, well, can I just buy Splunk, can I just buy Elastic? You can, and you can spend a lot of money, but there’s also a lot of assembly required, and there’s just not enough out-of-the-box intelligence. It’s not like you can just buy one of those tools and stuff just starts popping up for you to start hammering down, or giving you that long-term data storage. The other thing, and this is specific to Resurface, but I’ll shill for it here, we’re also a first-party solution. We’re actually providing you the software to run this kind of a system, versus taking in all of your data as a SaaS, and that on its own is very appealing from that regulatory perspective. Whoever you share data with ends up being another kind of regulatory set of problems you have to deal with. I’ve talked to CTOs who said they’ve done little else over the last year or so other than running down GDPR compliance waivers from everybody and getting all the paperwork in place. Never improved the system at all, just to get compliant. So that’s why with Resurface we’ve geared our solution to say, you’ve got your existing APIs, your existing API gateways, whatever that architecture looks like, we just want to be able to attach to that with very minimal changes, be able to build the system of record, and you just keep doing what you’re doing. The worst thing is if caring about security or compliance means I have to throw the brakes on everything else I’m doing and now become a security expert or a compliance expert overnight. We’ve all seen that from the development side, where your executives are telling you you need to be doing this and that, and what’s your response? Oh, well, should I just not do all the other stuff you wanted me to do over the next year that we were already late on doing, and you’re going to ask me for more of that, or do you want me to do the security or compliance thing? One of the biggest challenges is that none of this is static, all of this is being changed all the time. We want to be able to increase the rate of change. So how do you do all of that kind of at the same time? You have to have better tools and techniques. We just won’t have more hours in the day to be able to take that on. But I do think GDPR, CCPA, some of the new regs that are coming out, they absolutely have their heart in the right place. I think they also make some really good distinctions between what’s necessary for first-party data processing versus what’s really nice to have from a third-party perspective, trying to tap more into the real-world context and set some better standards of care along a spectrum. I think that’s really a step in the right direction. The response from the technical community, though, needs to be: here are more turnkey solutions you can just apply and retrofit onto the systems you already have, versus having to start over or assemble a big data team at very high expense to solve that problem. If that’s the answer, we’re going to have problems for a long time. We know that security people are in short demand, just as much as big data people are in very short demand, and people who really understand GDPR are in short demand. So if that’s your strategy, that everyone on your team has to become experts in all those different areas, good luck with that. Much better to have a solution that takes more of a layered, measured approach. We’re not going to turn you into a security expert overnight, but every day we’ll give you an opportunity to be better.

When you’re not outsourcing your knowledge capacity, you’re keeping it internal rather than just outsourcing or opening up a Trojan horse and giving all your data away, you’re finding that best balance. I would say APIs, and this is a common theme with everyone I’ve interviewed for this show, APIs are about wrapping some capability, the technical but also the business, the knowledge, the expertise, the teams behind it, wrapping it as an API, building SDKs and then other tooling and integrations, as you all have done with existing gateways and existing APM solutions, so people can just keep doing what they’re doing. But APIs allow us, with Twilio I don’t have to become an expert on the whole telecom system, with Stripe I don’t have to become an expert on the payment system, and when it comes to security but then also regulatory and compliance, we don’t want to be experts in those areas. We want to stick to it as business owners doing what we do best, being able to outsource those to professional teams, but still develop internal knowledge capacity as far as how it impacts our business and what’s going on. So again, it’s about that balance, trying to allow us to move forward without becoming experts in all these areas, but not just giving away the farm either.

Yeah, one of the best engineers I ever worked with just always seemed to turn out code that was really awesome and really fast. At one point I had a conversation with him about, how do you feel about premature optimization, are you spending a lot of time optimizing this code, because whatever you come up with is really fast? And he was like, no, I just make a habit not to write stuff that’s slow. It’s not like I’m going out of my way to do extra benchmarking or getting hung up on that. I’m not that much of a purist, but I just don’t, as a habit, do things that I know are crappy, whether that’s performance or quality and behavior or security. That’s ultimately, I think, what we’re really trying to get to. Does every developer really want to become a security expert? Probably not. Do a lot of developers want to be part of the solution and want to grow and be better engineers and more valuable engineers in the future? Absolutely. And you’re a more valuable engineer if your habits aren’t constantly introducing security problems or quality problems. But if all I see as a human is what works on my machine, that’s where the joke comes from. It’s really hard, as humans, to care about stuff you can’t see. You might have every good intention, but as far as really making that actionable in a way that affects your daily life, you need that feedback loop. I think that’s why customer care and growth is slowly merging with DevOps, security around DevSecOps, especially operational security is really merging together. I think that’s ultimately why, when I started going through that process 10 years ago doing DevOps at places like Dell, it was amazing. As a developer, I’d never had access to those operational systems before, and as soon as we started to see what is actually running in production, it’s like, oh, I could optimize this, I could do that better, I could put this in. I just never had seen it with my own eyes. I know we’ve talked a lot about security on this session today, but that’s really what’s been exciting for me in the last year, to be exposed more to what these attacks look like, how prevalent they are, how successful they are. When you see it with your own eyes, it’s like, wow, I really didn’t know that half the people showing up here were here to rob me. That’s going to be a permanent, fundamental shift in how I’m thinking about designing and rolling out my systems, and now I’ve got an appetite for more of that. I want to see how else these things can be abused.

So we talked about DevOps and the lower ends of the spectrum on our operations. When it comes to business leadership, the folks we’re targeting with this show, do you think they need to care about APIs and think about observability at the API layer, or is that something they shouldn’t be thinking about?

Again, to put it in the easiest real-world terms, could you imagine running a bank with no ledger, no record keeping? You’re trusting all your employees to do the right thing, you’ve got a bouncer at the front door that you’re hoping is going to keep out all the bad people, and if you’ve got blocking turned off on your web application firewall, you don’t even really have a bouncer. You’re just propping the door open and saying whoever wants to can come in. You would never imagine running a physical business that way. You would be laughed out of the room. The fact that we can’t see API traffic the way we can see foot traffic in a business, I think that’s the thing that has to be bridged. When I’m talking to other CEOs or CTOs, that’s the message I’m leveling up. Just like you were saying earlier, you want more foot traffic, you’re looking to increase that foot traffic, you’re looking to increase the revenue that comes with that, you want more qualified customers showing up buying stuff on their own. There are all those benefits to the C-suite. There are new opportunities for partnerships, new opportunities around acquisitions, opportunities to embed, opportunities to extend. But the fact that you can’t see them, that they’re going over the wire, means we just have to turn it into something visible. The moment you can turn it into something tangible, at least in our experience, there’s an immediate appetite for that. It means any kind of operational reporting that I’m doing now can be couched in terms of what’s actually going on. There are really great notable examples of that, like you mentioned, the Twilios and the SendGrids of the world, the folks who are really leading API-first kinds of companies. Most of the company speaks in terms of API calls. It’s this kind of call, it’s that kind of call, it’s this interface, it’s that interface. That is the actual shape of the business. It’s a little harder to get to that mindset if you’re from a traditional background, a traditional company, and you’re exposing some of those APIs out for the first time, but you have that same opportunity to get to that. And these things flip really fast. I remember when we first started working with folks like Expedia and Travelocity back in our web days, 90 percent of the traffic would come in on the website and 10 percent would be on the API, and now it’s flipped. Less than 10 percent of the traffic is through the website, all the traffic is through the APIs. Salesforce is another classic historical example of that. All their traffic is completely flipped away from their websites. That was customer demand that drove that as much as them deciding to make that change.

These are the stories I’ve been telling for years, and we’ve been talking about them, but they’re really starting to make an impact when it comes to leadership. They’re hearing it, they’re seeing it. Those are the solid examples that I’m using to help convince people and change attitudes. So we’re coming up on the hour here. I kind of want to wind things down from the deep technical and the APIs. When it comes to just staying aware of what’s going on in the world, what do you do to stay informed, where do you get your information?

Gosh, it’s such a hard problem, so many sources of information. Something I’m personally doing a lot more than I’ve ever done before is really trying to go direct to people working in the field who are experts, trying to get as much first-party information as possible, trying to do as much of my own research as possible, using good trusted sources to help guide on that, but ultimately really trying to get to experts, whether that’s following the right podcast or different ways of doing that. One thing, for anybody like us who’s doing an early-stage company or venture, it’s amazing how many doors will open, how people actually want to help. When you reach out founder to founder and you have a question or you’re trying to learn something, teach me about this, it’s really amazing. Even though everybody’s busy, everybody’s got too much to do, when you really reach out founder to founder or expert to expert and say, hey, I’m really trying to wrap my head around X, could you take just 10 minutes and let me pick your brain about this, my experience is about 80 percent of the time people will say yes, even if they don’t know you in advance, which is kind of amazing. Especially in the entrepreneurial community, anybody who’s been a founder, even if they’ve been a founder in a previous life or hope to be one in the future, a lot of people will share their knowledge, share their expertise. So it’s harder than ever with the amount of information out there to really find out what’s the best, but to hear it from the people who are really trying to solve the problems and hearing the full context around that, what is driving you to do that, what’s the context of what you’re trying to solve for, tell me about the world that you live in and why you’re reacting that way, it’s really amazing what people will open up. But you’ve got to ask, you’ve got to reach out. I know a lot of developers are mildly to severely introverted, so thinking about calling for help or getting a lifeline is not necessarily top of mind. It’s a lot easier just to do a Google search and see what’s on Stack Overflow. But if you’re really trying to understand what’s going on right now and what’s happening in the next few months, there’s really no better substitute than just trying to find friendly folks who will give you the answers.

I think you summed it up well for me. In building this show I’m doing lots of research and reaching out to CTOs and CEOs of startups, and then product managers and engineering managers at larger enterprise organizations, to understand how they’re seeing things on the ground within their operations. I’m stalking and trolling people on LinkedIn pretty heavily, just reaching out, and I’m amazed, the same, sometimes it takes a while for someone to respond because we’re all very busy, but I would say about 70, 75 percent of people I reach out to are like, sure, I’d love to talk event-driven architecture with you. A lot of the conversations I’m having are just brainstorm sessions, and I’m not even getting them on an episode, I’m just learning more about how they do things. And if they’re interested, if they’re a little more extroverted and their company allows, then I work to get them on the show, and ultimately that’s how I meet folks like you. So I think it’s pretty good advice.

It’s the timeliness that’s the most important thing. That’s why I really enjoy programs like this, because you can look at what’s happened historically, you can read the books, read the accounts, do all that stuff, but the fact of the matter is, especially when it comes to what’s happening online, the future is not going to look like the past. What’s happening right now doesn’t even look like the past. So really getting plugged into what’s going on right now is super critical.

Great, well, I think that’s a perfect note to end this on. Great advice. Thanks for all your insight when it comes to observability at the API layer. I think the system of record is pretty critical for us, trying to understand how we’re getting to where we’re getting with each iteration and evolution of our operations. So thanks for taking the time today, I really appreciate it.

Thanks for having me. And lots more information at resurface.io, just give that little plug.

Yeah, no problem, I’m happy to plug it. So thanks. And maybe, as we’re going to go into season two sometime this winter, in the future I’m looking to pull people back in for little interviews and little pieces, little nuggets that we can add to keep the conversation going. So I’ll be reaching out.

Love to.

Alrighty, enjoy the rest of your day.

All right, thanks so much.