Matt Machuga, Auth0

All right, welcome to another episode of Breaking Changes. Today I have with me Matt Machuga, engineering manager at Auth0, as well as the host of the Bits and Trees podcast. Welcome, Matt.

Hello, thank you for having me.

Yeah, thanks for joining me. I appreciate you jumping on here to talk some APIs with me. So let’s start with the basics. For folks who aren’t familiar, what is Auth0, and what do you all do there?

Auth0 is an identity management platform as a service. We try to alleviate the need for a developer to worry about their own identification, both authorization and authentication. There are a lot of ways to screw it up, and these days if you don’t have multi-factor and a number of other things protecting your website, you are at a greater risk of getting exploited. We see this all the time, you see “this site’s been hacked,” if you’re on Have I Been Pwned you get a list of all the sites that have been attacked. Our biggest goal is to prevent you from needing to worry about that, and putting your own users at ease that they are protected by Auth0. So that’s us in a nutshell.

Yeah, it’s the most critical layer of the API space, I would say. It’s the one that causes the most problems, but when done well and done right, it can make our lives infinitely easier, so I appreciate what you all do. It’s made my life easier on many fronts dealing with many APIs. So tell me a little bit about yourself. What is your role, what do you do at Auth0?

Yeah, so I will have been at Auth0 four years tomorrow, so I’ve been here for quite a while. I’m an engineering manager, I’ve largely always been an engineering manager of some sort here. When I started we were about 230 employees, and then we were at about 900 before the Okta acquisition, and now I think the number is closer to 4,500. So I went through a few different job titles. The gist of it is I was managing people and trying to grow the team, hire new engineers, help make the product as resilient as it can be. I’ve hired a number of people here so far, and it’s been a great experience. At Auth0 we have a strong set of values, and they’re all positive. Is cursing allowed on this show?

Yeah, I don’t mind it at all.

Okay. One of our company values is, “we give a fuck.” People here are very passionate, and you can really feel it in every Zoom call or Recital conversation. We really care about making the best experience for our users, we care about making the best experience for our employees, it’s a really good vibe overall. We also have “one team, one score,” everybody’s always helping each other. My team is a big believer in this, and we are always striving to help others. And we have the “n+1 is greater than n” value, which is, we make iterative approaches to continuously improving ourselves, our product, and our teams. So those are kind of what attracted me to being here. Those values had different names when I joined, but it was the same core premise, and it has carried through the entire time, so I feel a lot of integrity with the company. It’s just been a very fun and attractive position for me.

Yeah, I like the values and the culture aspect, because I think that’s pretty critical to doing this well, in any sort of startup as well as within the enterprise. So we’ll dive a little bit more into your team and what it is you do, but what’s the Bits and Trees podcast about?

Yeah, so Bits and Trees is a largely engineering-focused podcast, but we also weave in some managerial things or ops-related things. It started off as, hey, this show, I can interview industry experts, or people who’ve been through anecdotal experiences or things that just not everybody has had the chance to do, like some sort of novelty. What it wound up being is, I have a lot of conversations with people offline or in Slack and they’re just very interesting, so I asked them, hey, can we go talk about this on the air, let’s pause here, let’s record something, because I think there’s a lot of value here. And then I get together with them, we record, sometimes there’s two of us, sometimes there’s three of us, and we just get together and discuss and try to bring the little nuggets that we’ve all learned over time onto the air. I’m not regular about the scheduling. Ever since the pandemic happened I’ve taken a pretty decent step back from it, so now it is very much, when my interest is piqued by a topic then we just happen to schedule a call, push it. I added intro music, that’s like the most editing that goes on anymore. So it’s a very naturally flowing podcast, and I just kind of start and stop when the conversation makes sense, I think. So I’ve had a lot of fun doing it. I plan on doing more, it’s just not very regularly scheduled.

I like it, I like your style. And flowing into this, I try to have structure to what we’re doing, and we’re continually trying to improve Breaking Changes, but I find the conversations that get the most attention and have the most impact are the ones where we just dive into it and follow where the conversation is going rather than sticking to a structured script. So I like the style. And for me personally, the storytelling is the most important part of this. My wife has her master’s in folklore and storytelling, and the storytelling piece of the tech sector is what matters to me. It’s what makes the change, internally within our organizations but externally within the industries. So I like just following when you’re feeling the story and something interests you and piques it. I like that a lot. I’ll be tuning in. That’s one of the things that caught my attention when I was looking for new folks to invite, you seem to be focusing on some pretty interesting things, so I appreciate you sharing that here.

So diving into what you do on the ground floor at Auth0, tell me about how your team is structured. You talked about you’ve been growing your team and evolving. How are you organized and how do you all work together?

Yeah, so my team is called the Insights team. We largely are responsible for providing Auth0 customers more insight into how their subscription is doing. Some customers, like in retail, they know certain patterns that occur through the year, what days are going to be spiky, and we can help present the data to them. We can say, you have this many active users compared to this year, you have more attempts at exploitation, like maybe a credential stuffing attack. We can present this information to them in a nicely formatted way so that they can gain more information on what is happening, and this will lead into, we can actually give them some help, we can nudge them in the right direction, or we can point out issues to them.

We also handle the raw stream of this data. So we have the concept of tenant logs in the dashboard, I think they’re just called logs now, but we take this stream of business events that come from Auth0. Anytime you do anything, if you sign in, fail to sign in, we exchange a token for you, we detect a brute force attack, an API limit is reached, all of these generate events across our bus, and my team is responsible for ingesting them, formatting them, indexing them, and then presenting them back to the user in a usable format. So they can get this fire hose of data, or they can get our aggregated form of the data, and for different purposes these are both excellent. Some customers like to trigger automation, maybe a new tenant administrator was added to their group, maybe they want that to go to a Slack notification or a PagerDuty notification, maybe they just want all of their things streamed to Datadog or to Splunk so they can look at it later and try to match up correlations between their system and our system, or maybe they just want to dump it to an S3 bucket and be done with it just for their auditing purposes. We kind of facilitate all these needs.

And the team makeup, we have five engineers right now, another one coming in next month, another two coming in next month, and we’re still hiring for two more positions. There’s a good balance of seniority. We have two high-level seniors, two mid-level seniors, and two mid-level engineers right now. We also have a junior that is coming in next month, so that’s pretty exciting. We love the mentorship aspect of growing engineers on our team and showing them what we consider to be best practices now, or the things that have bit us along the way, so that we can help them grow in that aspect. I think that’s the gist of a team makeup. I’m sure there was more to your question, but I rambled for so long I kind of forgot what it was.

No, yeah, I’m taking notes, so I can riff off and play with several things in there. Let’s start with, what are you looking for in new team members you’re hiring? What sort of skills, capabilities, personality fit, culture fit, are you looking for?

The most basic answer is, we really look for human beings who know how to interact with other humans. As I mentioned, we really value the “one team, one score.” We have a very strong “no ego” policy at the company, so if your ego is going to get in the way, we can’t reasonably work with you well. Everybody is very humble on the team. If we make a mistake, we’re the first to say, hey, this is my fault, I did something, I want to fix it as fast as possible, can you help me? I am the first to say if I did anything. I’ve shipped broken code to production, I’ve taken out production servers because I thought they weren’t used, and I have a very supportive team, and we have a very supportive engineering department. So if I can’t solve something myself, another team was able to solve it for me, and with our team, if they can’t solve it themselves, we are there to help them. So humility and coming in egoless is very important. Technical knowledge is very important for whatever degree you’re coming in at, but that’s really only part of it. We need the professionalism, we need the ability to work with others and have that functioning very well. As I mentioned, my team is very big on the “one team, one score,” we’re a very empathetic team, so we are always looking to learn, always looking to help, and always looking out for our customers’ needs. Our team handles a lot of sensitive customer information, we’re kind of the pass-through to get that information back to them when they need it, so we take our security very seriously and we take professionalism very seriously here. That’s not to say we don’t take ourselves too seriously, we’re always laughing, always having fun, but when you’re dealing with customer data, you need to take that in a very responsible manner.

Are all team members engaged with the customers? Does everyone have to be front-facing like that?

No, they’re not front-facing necessarily. We allow the option to go to customer interviews, so when our product manager, or a designer, or I are speaking with a customer, we like to invite the developers along, but it’s not a mandate on the team. It’s just, we handle their data as it’s passing through the bus, so if an authentication event comes through, we are responsible for shepherding that data in a safe way back to them, so there’s nobody jumping in the middle and intercepting information.

So drilling in on that data that gets passed back, that you said you format, is it pretty structured to begin with? How is it validated? Is it JSON Schema validated, and then you format it, and what does that look like generally?

Yeah, in general, behind the scenes we control the format. So if you send a request to the authorized endpoint, we know what shape that data is going to come in as, and we format it and make sure everything’s set up through an internal SDK that we have. So regardless of what you send to it, we know we’re going to accept certain fields out of it and then use that data to transform it and send it back in an ingestible way. JSON Schema is currently being used in the newer format. It’s kind of ad hoc in the last one, so there’s a variety of different ways. But it’s effectively not just, the customer sends us data and then we return it back. We kind of return it, we redact things, we format things, we move things around, and then we send it back to them. It’s a multi-step process in a pipeline.

Yeah, so you know what’s going to be most meaningful to them, most usable across what comes in, so that you’re only giving them the signal within the noise, apparently, right?

We try to listen to the customers. They tell us, we don’t find this key to be very reliable, or we don’t find this key to be as informative on what we think it is. And sometimes we look at that and we think, this is a very informative key, but then if you step back and you interview a few other customers, you realize the name of the key doesn’t really make sense to the customer, it makes sense if you have Auth0 internal knowledge. So then we realize, okay, we need to change this. How do you change it without breaking somebody? And there’s iterations made that way. It’s a very interesting balance. It’s fun from an engineering perspective and it’s fun from a product development perspective, but you really want to make sure the customer is not experiencing breakages in their workflows as you try to improve it for them.

So how much of these events are kind of known knowns versus unknown unknowns? Is it pretty standard, run-of-the-mill events most days, and then some days things go weird, or is it pretty much a mix of weird and normal?

There’s a good mix of weird and normal. The weird parts come in more from the inputs rather than what we send back with the outputs. There are weird things that come in sometimes where, if we’re in a situation where we’re trying to analyze what happened and we can see a strange payload being sent across, sometimes people are sending us the wrong website’s information, they’re sending like an entire web page to the Auth0 authorized endpoint. We’re not really sure what happened there, but generally that still will pass an event through, we just try to eliminate the noise in it so it’s not going to break someone else’s automation. So those are kind of the weird ones that come in, just accidental, I’m sending you something that’s not meant for this endpoint, what is it, what happened? Because it’s still important to pass through certain things to the customer for their auditing purposes. If they’re getting weird submissions, it’s their right to know about it, as long as it makes sense and it’s not going to put them at risk. So there’s some decisions that need made there in terms of what “weird” is defined as. It’s a hard balance.

Yeah, and it feels like you really get to know the messages and events, really get to know what’s going to come through for each customer, and then get to know what your customers actually want and need, and find striking that balance across all of that.

Yeah, it’s not so much us knowing what comes in for an individual customer, it’s like we know the shape of what the aggregates look like, and we know the shape of what messages should be formatted as, and if we see breakages, that’s when the team gets an alert that something has failed or something is just odd and we should take a look at it. Because there’s ways to abuse, there’s ways people think there’s ways to abuse the system, and they’ll send garbage data, or it’s an accident, 99% of the time it’s an accident, but it’s enough to trigger automation to try to figure out what happened.

So the value to the customer is definitely, you have a view across a lot of customers and you know generally what’s going to be something malicious and what’s something probably harmless and accidental.

Exactly. And then it is the customer’s choice how far they want to protect themselves with this information. So we have a number of anomaly detection services, and if you want to prevent certain types of requests from being made to your system, we can shut those off more or less at the load balancer, so they don’t make it to your system. And this is important for a customer who’s very high attention and maybe they get credential stuffing attacks frequently. We’re there to try to mitigate those for you. We’re also there to try to provide you advice. If our aggregation services are detecting a trend, we can pitch ideas to you on how to prevent malicious behavior or how to take advantage of a situation.

Yeah, I could see that being pretty valuable advice, especially in folks’ busy day when they’re just trying to do business and get things done, and we’re not experts at what’s flowing through this dimension, but it’s a super critical dimension of our business operations. So speak to me about the notifications a little bit more. Let’s dive in there. You mentioned I can route things to Slack, to email, to Datadog, my APM layer, or just dump it to a bucket. Is that a pretty core group of services, and pretty common what people are looking for?

There are a wide variety of integrations that Auth0 offers. We offer extensible solutions, I think that’s the best way to put it. Historically, we’ve offered more than 13 different external integrations, but we’re pivoting models right now. The new one is our log streaming system, and this is one that we’re very happy about. We can pick strategic partners, and we can also develop ad hoc different solutions that people are really looking for. And we’ve picked the ones that have been asked for the most and the ones that we have data on people using from our old extension system, which ones they use the most, and those were kind of the first ones that were developed. So if you look in our log streaming system right now, you can see we have Amazon EventBridge and we have Azure Event Grid. Now, both of these can route into those respective cloud platforms, and you can more or less do anything with them. And they both have some pretty interesting first-class support for how to handle events inside of those systems. So with EventBridge, if you want to route it to CloudWatch you could do that directly, or you could do it through a Lambda and do your own processing. If you want to skip Amazon or Azure altogether, you could use our custom webhook, and this you can point at any URL, you can attach an authorization header to it, and we can send all of your events to that location. So if you want to route up something manual you can do that, if you want to route it to a Lambda that sends it to your own BI layer you can do that as well. And then we have the common ones like Splunk, Datadog, and Sumo Logic. Those have been the ones that have been asked for the most. And we’re currently in the process of making it easier for us to add integrations and for others to add integrations, so that expansion and extensibility story should get a little bit better here with the marketplace in the near future.

Yeah, I would say that real-time platformification of this layer is driven by, I would say, volume of data, correct me if I’m wrong, and then also just the need for that customization and transformation, like you said, going to a Lambda layer. And so I find, just as the volume of data grows and the maturity of our customers grows, the need for this type of routing is just getting even greater. So talk to me about that custom integration. What’s it going to look like if I’m a pretty mature, sophisticated customer? How can I evolve and customize those integrations?

Yeah, so let’s take a look at this from two different perspectives. There was actually an interesting article that came out a few days ago on the differences here. We have two primary places where you can route into the events and then do something with them. One is what I just mentioned, the log streaming. If you set up the custom webhook, you have the ability to route it to wherever you want. We do some DNS validation to make sure you’re not trying to attack a customer with data, we are being cautious about that sort of thing, we have some additional validation in there for other purposes, but in general you can send it to any one of your endpoints that you want to do some sort of transformation or event handling. Now, volume is an interesting concept here. Some customers are way busier than others. We have free tenants on there that are fairly inactive, and we have very large enterprise customers that are extremely active, and we have to facilitate both, and we have to do it in a reasonable manner. So if you generate a lot of events and you only find value in like five or six specific event types, you can just exclude all the other ones that you do not care about and say, I want to listen to these only, and that cuts down the throughput and the volume significantly. Some customers care about everything but two or three, you can also do that. And then some customers, as they’re migrating from one version to the next, so if they are reading from our API, which we’ll talk about next, and they want to move to log streaming, they can start their cursor at a specific point in time and move forward from there. So there’s a few different ways of doing the migration.

Now, the other one is the API events, so it’s the same data, they all just come from our public management API. So if you go to /api/v2/logs, you can search two different ways. One is an ad hoc query, you’re just trying to get some information, you’re looking for a certain type of event, you can use a Lucene-like query and it will provide you those results. Now, we put a cap on the pagination there, so you’re not going too far in time, you’re using a pagination method that we don’t really want you using for infinite indexing, so we have a different query style, which is just a cursor in time, and you can keep going forward on that cursor, back and back and back, and try to get what you’re looking for that way. This is how we have encouraged customers to export their data over time, and it’s how the old system, the extensions, it’s how they work. The extensions are just custom code using the management API running on our system. Log streaming is a little bit more open-ended, and we can give it a little bit more ability than we could with the management API, and we do not consume your management API rate limit, so there’s a few benefits there. The one article says that providing an API endpoint is more important, more resilient, and in some ways it definitely is. So we give customers a choice: either let us handle the resiliency, or you can handle the resiliency in a way that you are more comfortable with.

Interesting. So do you see an evolution in your customers as far as their awareness and handling of this, like coming in, oh, we can just handle all this, we want all the data, and does that shift over time based upon how you guys introduce them to new concepts and help them evolve and learn, so that they become more effective in how, and then maybe rely and depend on you more and stop processing on their end?

Yeah, I think the last part there, the rely on us more, is important, because sometimes a customer’s integration, they really don’t want to write a ton of custom code, so we are moving towards more of a no-code setup, or a low-code if that’s not suitable for you. And the log streaming is very much like, you just click a couple drop-downs and we take care of all of that for you. You don’t have to worry about retries, we’ll handle that, you don’t have to worry about which events to exclude, you don’t have to worry about how to pass a token around to get your cursor back on track, we handle all of that back-end information. Now, some customers want to do certain things, so they will write their own automation with our public API, they’re comfortable writing that code and that’s something that they prefer to do, or they will take our information from the custom webhooks and just send it to their own endpoint and do their extra transformations there. So we see, it’s not really a black-and-white thing. Different customers in different shapes prefer different behaviors, and we just try to cater to all of them and make sure that they have the choices they need to get the business operations they need done.

Yeah, it really feels like, what I’m seeing from the Postman ecosystem, as we grow our user bases, is we have more developers who don’t have a lot of time and don’t have time to write code, so the low-code, no-code options, webhooks, and plug-and-play integrations, but they’re relying more on us as they get busier and busier, and rightfully so, they pay good money for our service and they should lean on us to do what we do well so that they can carve out their world. But we’re also seeing business users who aren’t developers get more savvy in this and starting with basic APM, like, hey, just pipe it all into my dashboard, but then six months later they’re a little bit more aware of what they want, they’re a little more in tune and a little bit more integration-aware, and if we can offer them low-code, no-code options for orchestrating and automating their world, and the awareness and education that goes with it, that’s important.

So I’m going to use your word, “insights.” What sort of insights come with this journey? You talked about the manual kind of, your team and your awareness, and then there’s, whether I go with the full fire hose or the API and get it into my APM, but is there any machine learning in there, any other insights that Auth0 offers me as part of the package?

There are. I don’t want to speak too much on the machine learning portion, because it’s on a different team and I’m not really sure what they do for the training. On our side, we are training our systems to look through it, it’s not a machine learning system, it’s just more or less how we represent your information internally and how we detect certain things. So we are constantly trying to evolve different patterns. We have certain things that trigger the anomaly detection as well, along with the neural anomaly detection ones that have the machine learning. I think most credit card companies look for certain patterns, I think they have an overlapping set of, I’ve seen this type of transaction, now I’ve seen this one, this is suspicious, I’m activating fraud prevention, your card is deactivated. We have similar things where we know certain patterns. One example that is probably a given is if you see like password attempt failed, failed, failed, failed, succeeded, email changed, password changed, that’s really suspicious-looking, so that will activate some sort of internal detection. But we’re not just constantly training a model on our team, we’re just trying to learn from the data, and we’re trying to learn more from what customers are asking us for. And I think the biggest learning we found, and the part that I love the most, is that different users use it completely differently, and they ask for a conflicting set of information, and you have to try to balance, you have to walk the line to make sure you’re giving each side what they need. You can’t always cater to the wants necessarily, but the point that you made, business-focused people who are not tech-focused, they’re coming up with very interesting dashboards that they will show us, and they’re saying, look, this is why I can’t see what I want to see on our dashboard, and the dashboards that they’ve assembled outdo what I can do on most given days.

Yeah, I’ve seen Datadog dashboards that are so heavily customized I couldn’t recognize what they were doing anymore, and it was not an engineer who made it, it was a business analyst, and that kind of blew me away. Marketing departments are getting highly sophisticated. At Auth0 we have marketing engineers, and I think that’s helped paint a picture of how they’re getting so evolved, but they come up with great things. Data engineers come up with completely different requests than somebody who’s trying to write the automation. Security teams and ops teams, sometimes they overlap, sometimes they don’t. PII is always a concern and wants to be handled differently depending on which department inside an organization you’re talking to. So most of the learnings that have really driven what my team has been working on for the past couple years have been that difference in personas and difference in what the different teams are looking for, what kind of information they want to learn over time, and how they want us to present it to them. So there’s a lot of reading between the lines, a lot of digging in on what the job to be done is for those specific roles, and then trying to come up with that middle ground so that everybody can get the information they need, and maybe we just have to have them work a little bit harder in one area to get what they want, if it’s a bonus. So that’s what the extensibility portion is focused on.

Yeah, a couple things to unpack there. So one, your honest answer regarding the machine learning is something I definitely look for, because as I have conversations with security folks, testing folks, especially in the security space, there tend to be two types of security providers. There’s the well, we have these models, they’ll figure it out for you, just give us your data and we’ll figure it out, and then there are the people who are like, well, no, let us humans process the data and understand the pattern, understand what’s going on, and then yes, there’s areas of known things that we can train models on and automate away certain aspects, but we’re always going to need humans who have a lot of expertise paying attention to this on both ends, in this case the Auth0 side as well as the customer, to find that optimal balance.

And then that goes into the other area of the business users. I think it’s the XKCD cartoon that’s like, the most sophisticated algorithm is the church spreadsheet in Nebraska or Iowa or whatever. That’s the sophisticated user who’s been running the church for the last 20, 30 years, just super advanced, and that’s what we’re really seeing when it comes to business users. If you give them not just dashboards and access to the data but you bring them literacy and awareness around, hey, you can tailor what comes into this, and then augment them with folks like you with the skills, that’s just a win-win for both sides. And you can really go to entirely new levels that you could get to with developers and tech folks. It’s not that we’re not capable, it’s just I think these folks are closer to the business problems and what actually needs to happen. So I commend your approach on that. I appreciate that honest approach to my ML question, not that it was a gotcha.

No, I’m very blunt. I am so bad at ML concepts that even if I knew what was happening I couldn’t properly tell you about it. I’m much more focused on the human side of it and the partnership side of it. As I mentioned, we’re always talking to different companies, different customers, different partners, and we try to be your full integration partner. We try to understand what you’re trying to do with it so that we can suggest the right features to you. Sometimes you might be in a business that would benefit from location awareness. If you operate on a system where you are positive that you will never have a situation where somebody activates a VPN, we can protect you one way. If somebody has different purposes that might be a non-starter for them, so we can’t activate that feature for them. We just have to listen to our customers’ needs, suggest what is correct for them, and learn from that going forward. And sometimes that might just be, you tweak the messaging on the features so they understand the risks, like you say, add the literacy, make sure the documentation is clear on what kind of things they are opting into or out of, so that they can make the right choice, and our professional services teams can recommend that to them. It’s all super important. It is manual to some degree, it is just part of your Auth0 configuration. There are things that we automate, there’s things that we don’t automate, and I like the blend.

And so there’s a self-service layer there that your customers can depend on, if they don’t want to engage with your team and they just want to figure this out on their own, there’s a healthy self-service layer around the API and integrations and all of this to make sense of.

Yes, yes. So we have a few different levels of documentation. We have our public-facing auth0.com, we also have the management API documentation, so if you want to go explore that there’s an API explorer to let you walk through it, there’s a Postman collection that helps you walk through and understand these things. And then there is our community support, which there are Auth0 employees who help monitor it and help support on it, and then there’s also the wider Auth0 ecosystem where people like to chime in, they like to help, our ambassadors are on there. Sometimes a customer will say, you know, I’m really trying to do this, I can’t find the right solution, and two or three people might just chime in, oh, I’ve solved this in the past by doing X, I don’t know if it’ll work for you, and the knowledge sharing builds up from there. So you do have an Auth0 ecosystem support center, and you also have our self-service, hey, I just want to go read the docs, I want to try something, I want to use the API explorer to try and fail a few times and see if it does what I want. So there’s a few different levels there.

Yeah, I’m always trying to help leadership understand the importance of investing in those layers, and you’ve touched on the people connection at almost every dimension of your operations, from your team to working with your partners, your customers, and the community and ecosystem. And that for me is when it comes to APIs and why APIs were different than service-oriented architecture or what came before, because APIs aren’t anything new, but the current breed of APIs that started very publicly had the feedback loop mechanism in there. This is how you iterate and how you evolve. There’s a forum, self-service materials, there’s people there, DevRel to help. And so that people connection piece alongside the technical piece, and that rapid iteration that comes with it, is super critical, and trying to help leadership understand the importance of investing in those teams so that the documentation is good, the community resources are valuable and rich, there’s lots of tutorials, and then there’s people there who are like, hey, do you need help? If you do want to talk to someone, we can actually share this knowledge. So I think the people thing really shines through in what you bring to the table as far as Auth0.

Yeah, I’m really happy with the work we’ve done. And the one part I forgot to highlight is we have a team that handles all of the quick starts, so you jump in and we give you a repository that you can just start running, you add your client secret and client ID to it and it will be a working Auth0 installation. And the SDKs, so if you want to build something in Node, browser JavaScript, Ruby, I think there’s a Rust one, PHP, we have all of these libraries to support you. So if you don’t want to work directly with the web API, you can use a wrapper around it so it looks a little bit more native to your language, and you do not have to worry about refresh tokens or things like that, we just try to abstract it and make it a little bit easier. So that was an additional, I’m a big fan of the work that that team does, so I wanted to celebrate them a little bit. In terms of the people-oriented thing, they’re always talking to customers, they always want to understand what works better, so this is another area where we can make it a smoother interface for them.

Well, I would say that gets into a different dimension of SDKs and code libraries people offer. I encounter a lot of companies that are like, well, we have an OpenAPI, we just want to generate it in as many programming languages as possible, and I’m like, well, why, wait, why do you want to do this? Because what you just touched on, and what your team delivered, is we want to abstract away the complexities and speak into their world, into the language they’re familiar with. So you again took it to another level from the technical, which a lot of people are just like, oh, we want 15 programming languages, just get it out there. No, you’re bridging two people. And so doing it well and having a team like that is pretty critical to doing SDKs right. The most sophisticated teams that I work with, as part of Breaking Changes but as part of Postman, Stripe, Twilio, others, that’s kind of a hallmark of those, they have teams, again to leadership because this is what this show is focusing on, invest in these teams, make sure they’re human beings who are going to build SDKs that actually are good and perform and abstract away what is needed. So you touched on that, you brought a very human piece to that, so thank you for that.

What else from what you do really helps people make, because I imagine this layer of our world is just getting more complex. So let’s start there: is it getting more scary, is this front line, I think it’s one of those things we hear in the news as far as outages and breakages and brute force attacks, is it getting scarier at this front line, or are the tools keeping up with the threats, would you say?

I would say the tools mostly keep up with the threats. I don’t think we’ve hit something where we weren’t able to overcome it quickly with a combination of tools and knowledge. As I mentioned, our engineering team is really brilliant throughout the entire company, and when something comes up there are numerous people who have an answer to use, and then the whole team as a collective can come up and sort through which one of these makes most sense under certain criteria. Is this a timely fix, is it a resiliency fix, is it something that we need to roll out over six months to make sure that it’s a continuous improvement but it’s not breaking customers in the process? The business unit itself and the engineering team and the product team, with the help of the data team to properly show that we’re analyzing everything, everybody just comes together to solve the problem in a mindful way, an engineering-mindset way. So we’re not just throwing things at the wall, we’re taking the time to analyze it, see what happens, see what could happen, and move forward from there. We largely have enough tooling around everything for introspection. I think there’s a space that’s yet to be filled where the observability space versus just the standard structured logging, I think there could be some reconciliation there where the tooling gets a little bit more full or a little bit richer, especially as we see more companies moving towards microservices, or services in general, or message bus related things. There is observability there, it’s just not quite as refined, or the setup process or the tracing process is still not as holistic as I would like to see. It’s not saying anything’s wrong with any of the tools, it’s just I think we could level up to get to where we’re at now.

Do you think that observability needs more user context, do you think, or is it just more technical, there’s just too many microservices out there and we need to be able to trace through all of them?

Contextual information I think is the more valuable. There’s the fact that service A might talk to service B, sort of C, which one of these communicates with the gateway, do they all communicate with the gateway, why is that? That is all helpful, but understanding a flow might be more interesting here.

Yeah, like if this is part of my core authentication flow and I can see there’s a disconnect between A and B, why is that disconnect there, what does that signal to the rest of the system?

Usually those signals are generated at the application layer, not the observability layer, so just finding something that you can weave in some workflow-type information could be a little bit better. And I think, you know, systems like Sentry and whatnot over the years, they’ve always added in a little bit so you can add user context or event context, but it’s not the holistic approach that I would like to see flowing through the whole thing.

Yeah, and this feels like a side effect of microservices. We’ve gotten, we’re segmenting everything out into individual services that do one thing and do it well and they don’t know about each other, but we’re humans and we’re operating in a global, human-driven economy, we need to understand what those workflows are. So I think, like you’ve done with almost every conversation today, the observability space, techies, we’re really good at this, I think observability and traceability are getting invested in and getting involved. I see lots of startups doing interesting and innovative things, I talk to enterprise organizations who are finally getting a handle on their dependencies through their traceability practices in some healthy ways. But I agree, the human piece, like, why did this 15-step human flow, and going beyond just the basic auth flows, the critical business flows, the searching for a product all the way to purchasing it with multiple products, how, what’s the human context of that, and how do you see that or observe that in the system?

Absolutely. And when you have people on different paradigms, I think it makes it a little bit harder, and microservices means something different to every company. For me, I still haven’t adopted the term, I don’t think Auth0 has microservices, I think we just have services that happen to isolate most contexts well, but it’s not like we have 100 microservices each one doing something independent, each one needing to talk to each other. We separate the concepts out, and I would really like to see context from A in B when it makes sense, and I would like those workflows to be visualized a little bit better. I would like to understand, how did I jump from A to B back to A, why did that happen, and where did the customer lose context here? There’s room for improvement, and I think a lot of the tools that we use are fantastic, but if I could wish for something else, that would probably be good.

Now you hear that, everyone? Okay, we’ve got to invest in observability and traceability, but we’ve got to invest in visualization, and being able to really understand and see what’s happening, and tie and link it back to actual business value. But things that occurred in that, I think that would be the important aspect of the context, all right, here’s a visualization that has all the technical details, you can drill in, but a business user could sit down and go, oh yeah, that’s the shipping and logistics flow for this type of partner or reseller customer, and we do $3 million in business using that flow every year. I think that’s the part that’s got to happen, it’s got to be visualized and it’s got to make sense to business users. So whoever’s watching, get to work on that.

And we’ll probably pass-driven development.

Exactly. Well, we’re getting close to an hour here, I would say 10 minutes away or so. I tend to wind down the technical piece, but you’ve done a really good job, I have to say, of the human layer, weaving that in here, and I think that’s the critical piece that I’m looking to convey as part of all these tech discussions. But what about you, what do you do outside of Auth0? What’s your hobbies, interests, passions, what keeps you coming back? Your team and your culture sounds pretty invigorating, but what do you do to renew yourself and get back to work?

More people things, but a smaller subset of people. I spend a lot of time with my family, I really enjoy that. I have two young daughters, so playing with them is my biggest, I’m done with work, I’m going to go hang out with them, nobody to serve me during this time please. So I do a lot of that. I try to do woodworking when wood prices aren’t extreme, I try to make some home improvement things around here. I built them a swing set before the prices went up, so thankful I did that. And then I still play around on a BMX bike, I’m in my 30s but it’s still a fun hobby, just crashing hurts a lot more now than it did a decade or two ago.

Nice. Well, I like the family thing. I’m about to send my daughter off for a year, and she’s 21, so you’re, in a university in Korea, in South Korea, she’s going to do Yonsei University. So I don’t, unfortunately, get to play and build swing sets for anymore, but I get to fund her going to university and learn Korean so I can go over there and get the grand tour.

I’m envious of where you’re at. Enjoy it, those are the beautiful years, I really miss those sometimes.

Yeah, I try to make the most of it, I know they’re short, they fly by so fast already.

But, man, isn’t my favorite, it’s a good time.

Yeah. And the woodworking, I like that. I live in Oakland, California, so we’re looking at buying another house once this kind of crazy whatever we’re in right now, I’m not sure if it’s going up or down, I can’t tell with everyone migrating out, it’s kind of hard to understand. But I hope to get back to the home improvement stuff. The last house I had, we brought in a mill and I had a deck of logs and I was able to do my own hardwood floor, chose the logs, had it milled up, had it kiln dried, and then did my own hardwood floor, and then I did my own cabinets, I got some cedar and pine tree and did some siding. So I miss that. That’s a realm that is a whole different headspace to be in than the tech sector, and I miss being, it’s very hands-on, tactile, you and your brain just work differently.

So I hope the lumber prices come down for you so you can get some more projects.

Me too. Work in that direction. So information-wise, any tips for our readers as far as where do you get your information, where do you stay in tune with what’s going on, do you read books, blogs, feeds?

Yes to all the above. I actually tend to prefer audiobooks. It’s off to the side here, but I do have a bookshelf set up, so there are certain books I have the physical copy of, but mostly I like to do audiobooks, so if I’m walking the dog or I’m in transit somewhere I can just kind of listen to it and try to pick it up quickly, and do the same with podcasts in that regard. I’ve always been a big fan of conference talks, but since everything went digital, it’s a little bit overwhelming for me to try to keep up anymore, so I’m kind of off conference talks at the moment, just reading more, listening to audiobooks more, things like that. I do enjoy the website Lobsters, so it’s kind of like the technical topics that I like to follow, on there I get more or less what I’m looking for, and then I just add the keepers to my RSS feed, because that never went away for me, I still like RSS, that might be dating myself a little bit. And then I still really enjoy talking to people. I learn the most when I’m having conversations. I think I learned most of my developer tricks early on because I would screen share with people, or I would sit next to them and learn their tips and tricks. I remember watching RailsCasts back in the day, and I would pick up a lot of what I knew just by watching the nuances that were in those screencasts, or Destroy All Software, I would pick things up from there. And now that I’m a manager and I tend to not do as many IC things, I have to pick up most of my learnings from books, or from watching other managers or directors interact with their teams and trying to understand the whys of, why do they explain it this way, how can we improve the way people understand it, or at least relate to the decision a little bit more? Those tend to be the problems that I solve these days, so the input is a little bit different.

Yeah, I appreciate the manager focus and learning from others, that’s one of the things I’m trying to replicate here as part of Breaking Changes. I started the show with my Rolodex of analysts and API pundits, there’s quite a few of them out there, and I’m finding where I get the most nourishment, but then also folks who listen, is talking to people like you, hearing what they’re doing day to day, how they’re trying to get things done within their teams and orgs and then make their customers and partners happy. So I think we touched on all of that, that hopefully is going to help other folks think through. And so I’m trying to weave my way through different industries. I just did Ford, I just did eBay, and I’m trying to go very international too, so it’s not just the North American, USA-centric thing. I just did Belvo, which is a Spanish company but selling to Brazil and Colombia, financial data. So I’m really trying to stick with those practitioners who are doing things, their stories, but make it international and as diverse as I possibly can.

Very nice.

Yeah, we’ll see what I can pull together. But I’m going to try to not bombard you, but ping you, if I can come up with really interesting ideas to encourage you to invite me to your podcast sometime. See if I can find an idea that will get your imagination going, something off the beaten path that’s different, and you’re welcome to just say no, that didn’t work, but to really push me. So that’s my new challenge, I’m going to try to find something.

I like it. That’s the side of the questions I prefer to be on. I like to be the one asking, so to be more comfortable for me anyway.

Yeah, well, I appreciate you being on this side of it for me, and I learned a lot. The people aspect is really, I think, the most important takeaway from today’s show, I hope that leadership really thinks about. It’s not just the right people, it’s the culture, the connections, the relationships internally but also externally. I think you represent that well, so thank you very much.

Thank you, I appreciate it, and I think it’s a good direction for companies to go.

Absolutely. All right, well, I appreciate your time today, I’m so stoked we could make this happen. Enjoy the rest of your week and stay in touch, feel free to share ideas with me and I’ll do the same your way.

Very good, sounds good. Thanks for having me, I appreciate it.

All right, thanks, Matt.