Melissa Benua, mParticle

All right, here we are again, another episode of Breaking Changes. I’m pretty excited today to have Melissa Benua from mParticle with me. She’s the Director of Engineering at mParticle, focused on platform engineering and dev test SecOps, which I’m really keen on diving into. Thanks for being with me today. Yeah, thanks for having me today.

So let’s, I want to get to know you a little bit better, but let’s start with the basics. When it comes to mParticle, how does mParticle simplify data infrastructure for organizations?

Yeah, so data infrastructure is really our strong point. We’re a CDP, a customer data platform, but our emphasis is on letting our customers manage their data infrastructure so that their data makes sense across all of their applications and touchpoints, not just from a marketing standpoint but from a customer service, customer support, customer experience standpoint. Our customers have something like 100 different places where their data might be living across all their various appliances and touchpoints, and so our goal is to help them unify into a single point of truth, so that it’s easy to then take advantage of the data that they already have and do whatever it is they need to do to make their business successful.

And those data points are only growing, right? Only growing, this is getting worse. Only growing. Remember a decade ago we used to talk about API or SDK fatigue. You’d have, oh no, six or seven SDKs in your app. So now you’ve got dozens to hundreds. I mean, it’s more than the average person can keep up with. And so, beyond just aggregation of this data, what are the immediate benefits to business users when it comes to this?

Yeah, so it’s absolutely way too much for any one person. So by being able to have all your data in one place, you’re able to be orders of magnitude more efficient with everything that you do, from the decisions you make about what do customers want, to what’s not working well for them, to what’s the next big thing we should be investigating in. Once you have a single point of truth, you can very easily start making these really intelligent insights, as opposed to having to cobble together something across six or eight different dashboards, and then you’ve got data lost in transition, and oh, it’s duplicated here, and oh, it’s fragmented over here, we’re missing this chunk here. Being able to simplify, make it simple, can be just a huge enormous win, whether you’re a big company or a small company.

Yeah, I as an individual have challenges managing my data, just my personal data, but as a professional all the way up to what we do at Postman as a company, it’s a big challenge. So I’m thankful for folks like you, and I’m thankful that there’s APIs to help us do all of this. So other than just core tech companies, what are some of the top industries that mParticle services?

Yeah, so we’re really prominent in a lot of media and entertainment. If you watch streaming video, mParticle is probably involved in some way. Retail. We actually had a lot of our customers in retail who were really sensitive to efficiency, a lot of QSR restaurants, quick service restaurants. A lot of our customers and our uptake during the COVID time was in retail stores, who, it was really important that they made their spend as efficient as possible. And when every penny counts in your margin, you need something like mParticle to help make sure that you’re spending every penny really intelligently.

Well, and I would say, just bringing it back home to my own selfish needs, is this show. I’m syndicating it across so many different platforms, and trying to pull together the numbers on what is success is a lot of work. And that’s only going to get worse, the more we syndicate it, the more shows we do. So as we said, that is a growing problem. It’s only going to get worse. So what type of folks within an organization tend to put your platform to work? Is it primarily developers and more technical folks, or are there also business folks involved in this?

There are, and so once upon a time it used to be primarily marketers and growth people, ages ago. And now we’re seeing developers, and of course the product org along with them, are our primary decision makers. They want something that’s going to make their lives easier, going to bring something to the table so they don’t have to bring in all of the expertise you need to build out massively skilled distributed systems. That’s a really specialized skill set, it’s kind of hard to hire for, but it’s definitely hard to build from the ground up, especially if that’s not your core business. And so we’re finding that developers are becoming a really important part of our customer base.

Yeah, and for us, that line between developer and business user, the gap is closing, and we’re giving more tools and capabilities to business users, but also helping developers be more efficient. So I think our goal is to grow the number of developers out there and expand the definition of what a developer is. So yeah, again, thankful for what you all are doing.

So before we dive into the technical details, I’ve got some thoughts I would love to dive into, like dev test SecOps, but I’d like to get to know you a little bit better, so people can base what they hear on getting to know you a little bit. So how long have you been working at mParticle?

Yeah, so I’ve been here about four and a half years. It doesn’t feel like that long, but it has been. And before I was here, I was at a really teeny tiny startup, and before that I spent about six years at Microsoft on a couple of really big systems that you probably have heard of, being Xbox. Oh yeah, definitely.

So what put you on that journey? How did you end up at mParticle? Just looking for more challenges?

Yeah, so when I was at Microsoft, I got exposed to hugely massively scaled distributed systems. We were talking a million requests per second in systems I was designing, that’s a lot of API calls. And so I discovered I really loved huge scale distributed systems and the problems that came with them. And then when I was at my little tiny startup, I discovered that I also really enjoyed the systems engineering work that went into helping a small set of people build these massively scaled systems, but from a teeny tiny set of resources, taking advantage of what the cloud has to offer and best practices in DevOps. DevOps was a movement that was just coming around as I was leaving Microsoft and joining my tiny startup. So I got really passionate about figuring out how do we enable little teeny tiny teams to keep up with the titans at Microsoft and Google and Facebook, because the tools are out there, and little tiny teams have their advantages. They have the agility and the hunger, and resource constraints can sometimes be a really good thing in order to slingshot way ahead. And so that’s basically what I came to mParticle to do. I think I was the 50th employee or something like that, it was like the 20th engineer. I came to join this system that was already pretty large scale, but to figure out how do we do 10x and 10x again without having to 10x and 10x again the engineering team, in order to build a really cool, awesome scale distributed service.

I love it, the web scale of APIs. And you nailed it as far as the purpose of APIs is to allow us to do what we do best but then have access to all these other resources so that we can compete with the giants like that. I just got off another call with a healthcare regulatory, so Center for Medicaid and Medicare, and how do we empower the small developers to compete with the healthcare giants? And APIs is the answer. So I love hearing that, that’s music to my ears.

So how’s your role changed in the four years you’ve been at mParticle?

Yeah, so I wasn’t the very first engineer hired in Seattle, but I was pretty close to it, and I was hired to basically spin up the office, to bring in a bunch of new engineers. I was also hired to work on two things: to work on our connectivity layer, especially our inbound and outbound integrations, and build up a whole team around there, and also to work on our CI system and the build process and all of those cross-cutting initiatives. And anybody who’s been in a small startup knows sometimes you wear a lot of hats, especially when you’re engineer 20 or thereabouts. So it was fine to wear a bunch of hats. And so I started that, it took me about a year to hire my first engineer, and then I hired the first five, and then it hit an inflection point about a year ago where we ramped like crazy and hired about 20. And since then I have started focusing my work down a little bit, because once you get to a certain point you can’t wear so many hats anymore, it doesn’t scale. And so I’m focusing on a lot of our cross-cutting API infrastructure and build process, all the good cross-cutting pieces that make up a solid platform engineering effort.

Yeah, and you put forth the phrase, the word or phrase, I don’t know what we call it, so DevOps, which I’ve seen grow over the years to DevSecOps, but you put it as dev test SecOps. So what does that mean?

It’s a little bit of a tongue twister. We’d like to joke, you can’t hire you if you can’t pronounce dev test SecOps with a straight face, without making any weird accidental words out of there. But basically what it means is, as a part of any continuous integration, continuous delivery effort, both testing and security are first-party partners. You cannot have a continuous anything if you don’t have at least a little security integrated in, and your testing efforts integrated in, otherwise you’re just continuously shipping garbage into production, and that’s not actually what anybody wants.

Oh man, you’re speaking my language. So how do you ingrain this in your developers, in your team members? How does this become part of their DNA?

Yeah, so I used to joke, I like to make the right thing really easy and the wrong thing really hard. And by making the right thing easy, I mean you shouldn’t have to worry if you’ve broken the build or not, because you should just know, it should just happen for you, and if you’ve broken it, it should just tell you. But also be blocking in a way that it’s really obnoxious to work around. Not that you couldn’t override it in theory, because you always need the emergency escape hatch, but then it’s really obnoxious to do. In our case you have to go justify to our chief architect why in this case no, it’s fine that this unit test failed. So good luck with that. And so basically I stitch a lot of things at the place where they’ll be most impactful, which in our case is at pull request time. So when somebody is ready to merge a change, they have a series of gates they have to get through with a bunch of automated checks. They happen in parallel, they happen automatically. Some happen for every PR anybody ever makes, some happen just for certain subsets of the code that are more sensitive, or that are under ownership of a particular team, or that have special testing requirements. So we just use some basic heuristics to tie a set of changes to whatever it is that needs to be done to ensure those changes are safe, whether that’s unit tests, integration tests, standing up environments, whatever it looks like.

So is it safe for developers to learn in that environment, or do they have to go into it with a wealth of knowledge and understand how things are going to work, or are they able to fail forward and understand and learn in the moment?

Yeah, it’s actually really safe for developers, because they cannot land a change that’s not going to meet the bar. They just can’t do it. And so part of our job is to make sure that when you have a failure of something, it’s obvious and it’s actionable. And that includes things like making sure the tests are reliable to at least a minimum of 99, preferably 99.9 percent. I track my reliability in my tests like I track my reliability in my APIs, because they’re basically very similar. And so making sure that what you’re supposed to do is discoverable and automatic in most cases, and if you have actions to take to remediate issues, that they’re very clear: here’s your five unit tests that failed, here’s the error messages, go fix it.

Yeah, so that monitoring of the test, is that for all tests, like security, everything that you’re doing, you monitor the reliability of all of your tests to make sure that over time they’re doing what they should do?

I do, yeah. We monitor reliability and runtime as well, because if most of our unit tests execute in 100 microseconds and one percent of our unit tests execute in five seconds, we know where our long pole is, we know it’s disproportionately impacting run times. Just like you can have long API calls that are ruining your 90th percentile, I track my 90th percentile of my test execution times to look for outliers and potential problems. It’s honestly very similar.

Yeah, but performance, I mean, you just nailed performance at the testing layer. So I’m guessing this helps you move forward at a faster rate, this helps with overall team velocity and release velocity, to be able to optimize at that layer.

Absolutely. I mean, when I very first joined, eons ago, the only automatic build was the one that happened after merge. And so you would get build breaks two or three times a week, not because people didn’t test locally, but you know, sometimes you miss something, you push a change. So we’d get build breaks two to three times a week, and I don’t mean unit test failures, I mean build breaks. And so we started just building, and all of a sudden we didn’t have build breaks anymore, so the releases weren’t hung up waiting for somebody to investigate the build break. And then we started with unit test failures, and at first there were a lot of unreliable unit tests, and so there was a lot of velocity lost in reviewing this list of 15 failures and are they good failures or false failures, we’d lose a lot of time there. And you lose the whole engineering team’s time, because if you’re blocking a release you’re blocking everybody. And so once we started doing unit tests in the PRs, all of a sudden the unit test failures went to zero once you were in a deployment candidate. So just by making sure the automated pieces were running consistently and at a time when they were really actionable and relatively low impact, it’s much cheaper to your velocity to block one engineer who probably made the problem themselves than it is to block 30 or 40 engineers on a problem they didn’t make, that they have no idea what’s happening. It’s much more efficient to, I don’t know what they call it, shift left, but to push it upstream to the person who can take action on the failure and the person to whom it is the most relevant, and that frees up the velocity for literally everybody else.

Yeah, it’s not just shift left for testing and security, it’s shift left for responsibility and accountability. Exactly, that’s pretty key. Exactly. Nobody cares that somebody else broke the build, you just want to know when you broke it. Yeah, so you can get to work on fixing it and saving face and not have it be an issue. Exactly, so you can fix your mistake.

So what does automation look like? Is it all pipeline driven, or are there other forms of automation across your approach?

Yeah, so automation looks a little bit like a traditional test pyramid, which some people really hate, some people really like. I think it’s a useful visual diagram. We have tens of thousands to hundreds of thousands of unit tests, we have hundreds to thousands of integration tests that require something deployed, that require some kind of running external resource, and then we have maybe tens to a hundred high-level manual tests, or semi-auto tests that require a human. And so we have a much smaller set of those, and not that they’re not important, but they require a human, so they’re intrinsically slow. So what, oh sorry, of course the dog is now begging to be let out. Get out of here, shoo. Ah, I can’t win, my dog did that as well, but my wife just happened to be out in the hallway listening, and she quietly opened the door.

So what are some of the challenges you face while automating this, and why do you keep some things manual, and what were the challenges in automating the rest?

Yeah, so challenges in automating generally come down to investment and return on investment. Some things are very easy to automate, and to be honest, most things are very easy to automate. Depending on your code architecture, you’ll have a better time automating at the unit test level or an integration test level, but in general most things can be automated. But there’s a difference between most and all. Some things aren’t worth, for example, the return on investment. Some things are very difficult to automate, some things you just have no idea how to do. And so I find some people are able to automate everything, especially when they have infinite engineering time, but in practice there’s usually one or two corner cases that you’re like, you can just do a quick manual check of this, and it’s going to be more efficient. And so that includes, sometimes you want to go take a look at your metrics, take a look at a set of counters, make sure they’re hooked all the way up to your counter system before you go live. You could write an automated test for it, but it’s probably not worth your time. Some security tests you want to have manual. Some static analysis is very easy to do automatically, but plenty of things are not. And every code base is a little different, the things that you can’t quite automate or aren’t worth automating are going to be a little different, but I think it’s important to accept the reality that probably you’re not going to automate literally everything, because most people don’t have the engineering budgets for it. It’s like a 90/10 rule.

Yeah, priorities, other priorities get in the way. We’re all short-handed, we never have enough resources no matter how large we grow. So we just build, oh, go ahead.

So I just built into the process the assumption that hey, we might need to stand up some code and have a human look at it. In a lot of cases you don’t, but because it’s built into the capabilities, built into the process, it just makes it really low friction.

Yeah, so how do you organize your teams to optimize this reality?

Yeah, so I’m not a huge fan of running a traditional style QA org, which I know is sometimes a controversial statement. I’m a big fan of building empowered teams that have really one primary mission and goal they’re trying to accomplish, and then letting those teams accomplish, setting them free to accomplish their mission. So that means supporting them, making sure that they have usable pieces in place, building blocks they can use, whether that’s the unit test framework, existing CI pipelines. They shouldn’t have to invent from scratch, but they should be free to innovate within their space as aligns with their mission. So I try to build pretty balanced teams that are very clear and aligned on where they’re going and what they’re trying to do.

Sounds like they have a lot of agency too, but the overall CI/CD provides them, well, I wouldn’t say just CI/CD, the dev test SecOps kind of scaffolding helps them optimize within that agency. They have a lot of freedom to do what they need to do, but the system helps them out with the rest.

Exactly. It’s sort of like guardrails. In the end, the system that we have is guardrails around where they really can’t go, but within those guardrails they have a lot of opportunity to set up narrower guardrails of, hey, my team actually needs to be here, we’re going to have some additional special checks to ensure that we’re here in this narrow guardrail instead of along the broader highway.

I like that, because sometimes I have bad days, I have times where I’m overloaded, and my team needs to have a special set of guardrails for this process. So giving us additional frameworks and scaffolding I think is pretty key.

Exactly. I want my free teams to be free to test, but if they want to just use the existing pieces, okay, you can just use the existing pieces, you don’t have to invent that. It’s important agency. And I think that’s, for me, what goes with the DevOps, that’s what it should be. I think a lot of folks look at the technical side of what DevOps can be, having access to systems across what’s happening, but it’s really that, it’s the freedom, the agency, as well as the right access to tools and services and infrastructure.

Exactly. The actual benefit of DevOps is not, yeah, you’re on Kubernetes or whatever, it’s in the agency and the freedom and the creativity that you give your teams to accomplish a goal.

Yeah, and with the learning that comes with the way that you’ve set up your testing in the pipelines, that people can fail, and you can learn, and you can fail and learn, and you can win and succeed. And that’s what education and learning, that’s what our schools should be like, that kind of environment.

Exactly. I don’t care how many times you fail. Your PR build failed 30 times. Write different unit tests, watch them fail horrifically, iterate and iterate and iterate, it’s fine. Nobody tracks that, it doesn’t matter, you’re free to experiment and make as many mistakes as you like, because you’re not going to affect anybody else, just yourself.

Yeah, amen. That’s how we learn. That’s how I’ve learned to be a decent and good engineer over the years, that kind of environment that gives me the room to do me and operate like I do each day, but face some pretty big challenges over and over and grow and evolve with those challenges.

Exactly. Go ahead. I’d like to tell my soon-to-be senior engineers that they can’t actually become real senior engineers until they’ve made one horrific mistake in production, until they’ve taken a site down or done something bad.

Yeah, no, we’ve all had them. Mine, my story in this, is I used to run SAP events for SAP, and I ran Sapphire, which is their 50,000-person event. I ran registration and all of that, and I set up the database, it was a Microsoft SQL Server, and had it all ready to go night before, stayed up all night, was groggy, this is back when I still stayed up all night. And four hours before reg open, and it was the first time I was using Amazon Web Services in production and trying to prove it would be worthwhile, I deleted the AMI, or the instance. And luckily I had backups, but still, to get it all set back up was like another two hours. I didn’t have as much automation. So what was yours, do you have a similar one?

I do, I have a million-dollar outage. Yeah, nice. I was the on-call responsible for our system, so our servers were the front of basically the entire serving stack. You couldn’t get anywhere in any part of our system without our servers. We were the thin edge. And I was on call that week, and I had actually found a crashing bug if we pushed a certain type of config change. And I got together with my manager, it’s super late, and I said, hey, so we’ve got this crashing bug, we can’t do this type of config change, do we want to try to fix it tonight? It was like seven or eight o’clock, and he’s like, we can wait till the morning, it’s just you and me, we’re the people who would do this, we just won’t do it, we just won’t push this terrible shiny red button, and we’ll just fix it in the morning. So that was fine, and we made this decision, but we didn’t tell anybody this decision. And meanwhile in the night I’d had a bunch of pages overnight from our China DC and our European DCs, and so I was tired, I had a crappy night, five alarms or something. And we had this big marketing push in the morning, which our team knew about. And come morning, we had received, like 7 or 7:30 a.m. from the East Coast people, a request to enable something as a part of the marketing push, and oh by the way, that involved pushing this config change. And I wasn’t awake yet, because I had been up four times the previous night, and one of my teammates saw the request and took pity on me and said, oh, I’m an early morning person, I’ll help Melissa out, I’ll push this config change for her. And then crashed every service we had worldwide instantaneously. And it took hours for it to come back up, because we would cascade traffic around the world. This kind of predates a bunch of the edge services that would be put in place now. We just sent traffic cascading around the world, killing data centers until we could cut everything off and bring up enough to support the traffic load in the middle of our marketing push. So I took the services down worldwide for, I don’t know, an hour and a half, two hours. It’s a million dollars. Nice. Yeah.

Wow. I mean, these are the things that make us who we are, right? They make you very cautious. You don’t know why you’re cautious until you’ve done something like that, then you understand why you should be terrified of everything you do in production.

Yeah, I had an interview with a gentleman who works on the HTTP standard, he’s the chair of the HTTP working group, and talked about his reputation for saying no, and why, given he’s got a lot of experience, he should say no to things and have caution and think twice about doing things. So I think a lot comes with wisdom and experience.

Yes, yes. There’s no substitute for some of this hard-won experience. Hopefully you don’t need a lot of it, but a little bit. Yeah, and hopefully we don’t get too jaded along the way.

Yeah. Well, one of the things that I found interesting about what you’re doing and kind of your shift, what I like to focus the show on is, I really like companies who can tell the internal engineering manager stories about operating APIs, but you also have a lot of experience using other people’s APIs and depending on those. And that’s, I find this makes the best API providers, as people who felt the pain of using other people’s APIs. Talk to me about that. Those types of integrations, are they just self-service and you integrate and things work, or are they high touch, do you have relationships with these folks, what do they look like?

Yeah, so we have pretty much a pretty broad spectrum depending on who the partner is and who our customer is who wants the connection. In some cases we write it because we have to, because the partner won’t or the customer won’t. We also publish an SDK, so in some cases the partner will write against us, and in some smaller cases the customer will pay a third party to write between the two of us. So we have pretty equal representation across the three. My team that I was running did a lot of writing their own integrations against partners, and then were also the publishers of the SDK that people would use to write against us.

What are the biggest areas of friction when it comes to partner integration?

Yeah, so the biggest area of friction is that no two APIs look alike. And they don’t even look a little bit alike. We have seen literally every API design pattern and anti-pattern that you could possibly ever imagine, from APIs that are 90 percent JSON and 10 percent XML, but the 10 percent XML is totally undocumented and you only find it in a bunch of weird corner error cases, to APIs that have interesting interpretations of the HTTP status codes and what they should mean, to APIs with really aggressive rate limiting, to APIs with no rate limiting. Some of them of course have sane rate limiting, a lot of them are very sane, but we’ve seen a lot of really interesting choices. Many, many cases where documentation doesn’t line up with real life.

Yeah, I would say documentation and rate limiting are the two top that I hear from folks who are in similar situations. Are you able to get them to raise rate limits at any point, sometimes in some situations, or is that harder to do?

We can, sometimes. It depends if the rate limit is a soft thing or a hard thing. So if the rate limit is, hey, it’s a billing tier or it’s just in case, usually we can negotiate a change. But sometimes the rate limit is because there’s a technical block. We had an issue where we were getting throttled really heavily by a partner because it turns out the shape of the traffic coming from our customer through us was causing like a 4x performance degradation over what it should have been. Sometimes the shape of the traffic is the problem, not even the quantity. And so they had to put really aggressive rate limits in place because the shape was causing a problem, until we could negotiate a solution that worked for both of us, that didn’t also involve the customer having to totally reshape the traffic, because that’s usually not the answer.

Yeah, so if any of these partners are watching, what can they do to make your life easier?

The best thing that we can always have is really good, accurate documentation, where the documentation matches the API. If they have an OpenAPI doc, all the better. But the easier it is for engineers to figure out, okay, here’s the API I need to call to solve problem X, here’s the parameters, here’s the performance characteristics, here’s what I can expect. Are the API calls blocking or non-blocking? Should I be waiting for an instantaneous response but the response is meaningless, or should I be waiting a hundred milliseconds for a response but the response is really useful? It doesn’t necessarily, one isn’t necessarily better than the other, so much as we know which, and we can then behave appropriately.

Yeah, up-to-date documentation, the number one pain point I hear across the board, and then that rate of change between versions and breaking changes.

Yeah, that’s a big one, definitely what we hear. Some of our larger company partners tend to do breaking changes on us more frequently than we would perhaps prefer. But that’s an advantage to our customers, right? As we’re at least shielding them from that pain. It’s one thing for us to update our server and then we’ve matched a new API version, it’s another thing for our partners to have to update their mobile app, go through cert again. And you can’t always force your customers, once they have an app on their phone, to update it. So it’s much more painful to have to increment an API version on an app on a device than it is server to server.

Yeah, that’s a really strong, important argument right there, I would say. So I work on a lot of policy and standards, so healthcare, banking, just did a session with the Government Accountability Office, which is an authority out of Congress and regulates how the federal government puts data out by APIs. And then some of these other conversations are in regard to the FTC and regulating Facebook and others, specifically the case around Facebook’s anti-competitive practices, and then the healthcare interoperability in there. And I’m impressed that government agencies have come to me and said, hey, we notice that people are introducing breaking changes and moving fast to open up a competitive, to make it harder to stay up with the API, so that the smaller individual companies have a harder time keeping up. And they said they notice big players are doing this on purpose. So that’s really a vote of confidence for mParticles of the world, to be able to be the aggregators and be the relief valve on this behavior.

Absolutely. I mean, we’re extremely cautious when it comes to breaking changes in our APIs. I can’t think of the last time we did one. I know we have, but we take a long, careful, very deliberate choice. And usually we just choose to stand up a new version and then deprecate the old version three years later, however long it takes, making sure we’re monitoring that we’re not going to accidentally kill some major version that’s deployed on somebody’s eight-year-old Roku system or what have you. Yeah, we’re operating, because it’s not our data, it’s our customers’ data, and so they trust us with the integrity of that data, and we take that very seriously.

Yeah, you’re a shock absorber in between that, I would say. Not only are you aggregating, you’re acting as that absorber for the change in the velocity, when you’re driving down the country road. Exactly, and all the potholes. That’s important. Exactly.

Talk to me about security. How do you view security across your operations?

Yeah, so, all right, we have a security department who we have, so I work in conjunction to enable them. So I’m definitely not going to be able to give you the holistic and perfect answer. But my view, my job as a part of the dev test SecOps strategy is to help security get the pieces in place that they need to minimize their manual workload. So that includes from really simple silly things like making sure that secret scanning is on so nobody checks in secret keys, it’s a very simple automated check, the simplest possible security static analysis you could do but really impactful, to more advanced static analysis, vulnerability checks. Nothing is perfect of course, but anything we can do to help is a plus.

Yeah, this is not a security, API security conversation, so I just always like to hear how folks view security, and try to, I think shifting left, as you said, and making it, so the dev test SecOps is a key part. It’s something we all have to do, but none of us are, well, some of us are experts, but most of us are not going to be experts, so I think that’s important.

Exactly, the same thing with tests. A lot of us aren’t experts, but we can build on the work of experts, and we can make the experts’ work more impactful. And that’s why we’ve got test and security in the DevOps thing, because it’s about taking those experts and applying their expertise as broadly as possible and as efficiently as possible.

Yeah, and that’s where your reliability at the test layer, I’m fascinated by that, comes in, is, you know what’s working, and we can optimize that. So I can count on that as a tester, I don’t have to be the best, but I know we can keep improving and understanding the performance and reliability of our tests.

Exactly. It doesn’t have to be perfect, right? Just like somebody claiming 100 percent uptime, 100 percent reliability, is just not measuring. We all know there’s no APIs at literally a hundred percent. No test is a hundred percent reliable, but they can get close.

Yeah, agreed. So you talked about not every API is the same. You could use 10 different image APIs and they can be wildly different. And so I spoke a little bit about the standards work that I do, but it’s hard to convince API providers to use common patterns. And I don’t want there to just be regulatory, hey, you have to use this within an industry. I would love for folks to adopt common patterns. Do you have any advice, or is mParticle going to become the de facto standard in a lot of these domains?

I wish we could. We had previously actually done some work around this with GDPR, and the name of the specific initiative is escaping me, but we had spun up an initiative to put a standard together, us and a couple of our other partners, of, hey, here’s what these privacy API requests should look like, here’s the standard, here’s the frameworks that will help you implement against them, so it’s not such a burden on you that you have to reinvent the wheel of what a data deletion request looks like. I wish I could remember where that was. But yeah, I would love to be able to publish, hey, here is a really good standard for what an event looks like, a media event, a commerce event, and to just have agreement. It doesn’t have to be our standard. I tell this to my teams and my manager all the time, I don’t care if my standard or my idea wins, so long as we have a standard, an idea. Everybody’s got slightly different views of what the perfect API is, and I really don’t want to micromanage for perfect, but good enough will get us good enough.

Yeah, so what’s the scope of effect of this time-wise? Are you guys going to save money, time, resources if things were standardized, if every image API was similar? Where’s the biggest return on investment there?

I mean, if every image API was similar, it would be very simple for us to write integrations with our partners in the ecosystem. It would really greatly expand the ability to stand up a new startup who’s doing whatever on image processing, and to have them integrate really easily into an existing ecosystem, as opposed to having to start from scratch and then beg other people to write against them and their custom API, which, god help us, maybe it’s on a WebSocket API and it’s live streaming or something. Because you can’t even assume people are going to be over HTTP/2, they may be over a WebSocket, they may be over a custom protocol. We definitely have a couple of partners that are SFTP uploads. Oh yeah. So you really, it’s an API, you just can’t make any assumptions about the basics. I would be delighted if we could settle on, yes, it’s HTTP 1.1 and it’s JSON and it has these few fields and the rest is just a blob. That would be miles and miles above the current world of infrastructure that we have right now across all the breadth of this.

I feel your pain. One of the successful startups I did back in the day was called, is called, IDX Broker, aggregating real estate. And you call up a multiple listing service, oh, do you have an API? Oh yeah, we have an API. And they send you their FTP location and the credentials, here you go, here’s your API access. And then they send you the legal terms of service of requirements of how you can use that data, what you can access. And it’s not managed through an API management layer or anything, because it’s FTP. So it’s a manual regulation of how you use this data, how many times you can pull. And you feel like a lot of these servers are just in uncle Bob’s basement on some cinder blocks. Probably, no, that kind of thing. Yeah, but there’s still an API, and that’s what, yeah, and this is continuing to evolve.

So you said WebSockets. So what was our release for Postman a couple weeks ago, we released a WebSockets client, and then we released a WSDL editor, so you can edit SOAP all at the same time. Ooh, I think I have an authentic, that I hate, the SOAP services. Oh, trust me, the issue asking for WebSockets has been there the entire time Postman’s been there, and we’re just now tackling it.

So how do you prioritize what integrations you offer next? Is it customers screaming for it, is it some grand strategy, how do you decide from an integration standpoint what comes up?

Yeah, so we have a prioritization matrix of course, based on how many customers want something, how difficult or easy it is to implement, and then strategically what customers would we want to pick up who might be interested if we support integration X. And then based on that matrix of inputs we end up with a prioritized list. Some things we can do ourselves, sometimes we can convince a partner to do it, sometimes we can get a third party to do it. And so based on who wants it and how much.

Yeah, it’s interesting to watch as the landscape keeps expanding, our needs keep, the sprawl continues, like what are customers going to demand, want, need, and how that algorithm of yours will handle. I like it.

Yeah, we don’t want to be caught chasing. We don’t want to always be behind the curve on integration. So in some cases, of course, how many customers are asking for it is important, but sometimes you need to be a little more strategic around, we want to go after these new customers who would only be interested in us if we have this new integration.

So working at mParticle, how have you managed to keep challenging yourself while working there? You’ve changed roles, but how else are you challenging yourself?

Yeah, so changing roles has been a big part of it. The work that I’m doing has changed really significantly from day one to today. My first year I was writing a bunch of code, really elbows-deep in the code base, writing integrations, working on integrations, and that switched over to building and running a really small team, and that was a totally different set of challenges while still keeping myself pretty technical, maybe not elbows-deep, maybe only wrist-deep in the code. Into what I’m doing now, which is a lot of strategic planning, not even so much day-to-day management, which I do, but I have managers who are doing the day-to-day management, and now I’m focused very much on strategy and buy-in and making sure that we are anticipating the needs of our customers and the needs of our fellow engineers, because my group, we’re the servants of the other engineers, not just our customers’ engineers but our engineers as well. So making sure that we are serving the needs of everybody else in a way that is efficient, sometimes solving them before they know that they’ve got a problem. So it’s such a different set of skills that I’ve been building all this time, that it’s kept me really engaged and really interested, growing a lot.

I like that. That’s why I’m in the API space, I love the diversity of challenges and problems, and it keeps me interested. Otherwise I don’t think I would stay in the technology sector for much longer, I would go get a gardening job or farm, something agriculture now. Because, yeah, that’s my hobby, flowers keep me happy. Nice, nice. That was going to be one of my next questions, what else do you use to take your mind off work? You got your flowers, what else?

I do, so for those of us who are watching the video, you can see the latest cut out of my garden. I do a lot of gardening, dahlias are beautiful this time of year. A lot of time relaxing, I mean it’s not exactly relaxing, it’s still a lot of work, but physical work. I actually take a lot of one-on-ones from the garden, because I find them more effective if I’m not subject to Slack distraction. So I’ve really enjoyed work from home, because I can take one-on-ones in the car. We used to do walk-around one-on-ones, get away from the distraction of computers and really focus on the other person. So I find that garden one-on-ones also fill that same niche.

I can see that being huge. So one of my first jobs as a teen, I was probably 13, 14, a lady down the street was weeding and working in her dahlia garden, it was like a quarter acre, her prized reality, and she would sit at the window of her house and watch and yell at me if I did anything wrong. But I really liked it because it was beautiful, it was an amazing, just colorful world. So they’re one of my favorite flowers.

Yeah, I only have maybe 10 square feet of dahlia, but I love them very much. It’s been a shockingly successful year this year for them. Crazy. It’s crazy, the colors, the shapes, the size, the dimensions, they’re pretty amazing. I mean, flowers in general are, but they’re specifically, yeah, the diversity is crazy.

So how has the pandemic impacted your work world and your team’s work world? I hope at some point I can stop asking this question, but I’m really fascinated by the different answers I get.

Yeah, so I have two little kids, they’re seven and four, and so that was a huge change. They were generally not in the house, I’m obviously not a stay-at-home parent, neither is my husband. And so figuring out, okay, now the kids are home all the time, how do we continue to work, but also to be effective parents of happy children? That was a huge transition to make. I’m glad that it was my transition to make, because I think it helps to build awareness and empathy for everybody else who had to go through that as well, my directs who had to go through it, my other teammates, my peers. The fact that I went through it but was able to be in a prominent enough position where I could be pretty frank about, hey, I can’t be in these meetings this afternoon, my kid’s having a meltdown, or, I’m going to be in this meeting today, and by the way, there’s a toddler on my lap. I’m able to kind of normalize some of that for everybody else, to reflect the reality of a lot of people’s lives on the ground.

So true, so true. I’m sending my, so I have one daughter, she’s 20, she’s in the room next to me here, but she’s going to Seoul, South Korea on Sunday for a year, going to Yonsei University, and my anxiety levels are through the roof this week because of that. I’m happy for and proud of her, but I’m also sending my baby to another country for an entire year where I don’t have a lot of control. And in a pandemic. So I think most of my anxiety comes from there.

So what would you say, when it comes to Melissa, what’s the personality trait that has benefited you most in your career?

I think the most beneficial thing has been curiosity. And I talk about it with my daughter, with my seven-year-old, I talk about it as, what’s your superpower? What’s the thing that you both love and are really good at, that’s a part of you? And for me that’s learning, figuring out how things work, how to take them apart, put them back together again, how to manipulate it, what the boundaries of the system are, what it can do. I really enjoy learning. And like, that’s all we do all day, is everything that I interact with is a system, it’s a system with harder parts or squishier parts, but it’s still a system. And so figuring out how all of these systems work and how we can make them more efficient, how we can make them better, whether that’s a team, a team is a system, a set of teams is still a system, the build system is a system, the workflow, everything that we do is a system. And so being able to apply this love of learning and this passion I have for figuring out how these systems work, once I figured out how to harness that, has been really useful for me. I like technology a lot, but I’m not the person who’s going to write code all day and all night because that’s my passion. I like it, it’s not my passion. My passion is systems and figuring out how systems work and how to make them work better.

And through talking with you, I would say an important characteristic of that, and you mentioned the technology, is the human piece of it. You talked about where the manual process comes in, where the agency of your team members comes in. So the role of the human being in these systems.

Yes, the humans are an intrinsic and important part of the system. The systems are for the benefit of people, people are a part of the systems. And I think it’s folly to try to divorce all the things that make us human from trying to run these systems. Instead, I prefer to try to harness those strengths, empathy, creativity, even something as simple as rubber ducking, that’s a fundamentally human thing that results in better output.

I like it. Well, I think for dev test SecOps, almost gotta say it carefully, dev test SecOps. I don’t know if I’d pass your test to be able to say it with a straight face, because I’m always going to snicker a little bit when I say it. It really shows, and that’s the DevOps part that I think is super critical, is the human piece of that. And for me, APIs, that’s why, what attracted me to the API space. So I think your description of your whole build system and the dev test SecOps embodies that. I think it’s something that there’s a lot folks can learn from, so I appreciate you sharing it all with us today.

Yeah, thank you so much for having me here today. I love talking about it and evangelizing for making better systems for people. Yeah, well, keep doing what you’re doing. You’re bridging a lot of important things, and I think reducing friction for folks, making people’s life easier. You’re making the machine work more efficiently, you’re connecting the important data points, so keep up the good work, and I appreciate you being with me today. Yeah, thank you, we’ll have to talk again soon. All right, well, enjoy the rest of your week, and we’ll talk soon.