Sam Ramji, DataStax

All right, here we are for another episode of Breaking Changes. I am your host, Kin Lane, also the chief evangelist for Postman, and today I’m pretty excited to have my friend Sam Ramji from DataStax over here with me. Thanks for coming by.

Fantastic to be here, Kin, thank you so much for having me on.

Well, I want to dive into a little bit about who you are and where you come from, but let’s start with the basics of DataStax. What is DataStax, what do you do over there?

DataStax is a hybrid database as a service built on Apache Cassandra. So it’s probably the best known commercializer of the largest, most scalable open source database on the planet. The core Cassandra database is named because it’s an inversion of Oracle, right? They said, who was an insane oracle that nobody believed? Oh, Cassandra. And it was invented as a wide columnar database. So think about key value pairs, but instead imagine that you can have as many values as you want per key, that’s the idea behind a wide column database. Advance to the future, right, to where we are now, from 2008 when the project was first invented to 2010 when the company was founded and the project became a top-level Apache project, Apache Cassandra. Apple has gone on record saying that they are running iCloud storage, iMessage, all those iServices on top of 200,000 nodes of Cassandra. So probably the most proven, longest running extreme scale, globally partitionable and globally federated data infrastructure. It also works really good in clusters of three, right, because it turns out people really want availability. It’s 2 a.m., do you know where your data is? And a basic three node cluster of Cassandra will stay up for years.

Wow, so it’s really doing it at the scale we all imagine it takes to run the web and real infrastructure that everyone’s going to need.

Yeah, and of course then the flip side is, can you scale down? And that’s one of the really cool things that’s happened lately, is we’ve been marrying Cassandra to Kubernetes so that you can have a minimum footprint, self-operating, self-healing distribution, which we cleverly called K8ssandra, K-8-S-S-A-N-D-R-A. I know, it’s ridiculous, never let me name anything, my names are the worst. But we have a lot of fun with it. So it kind of combines some history that I had and my passion for Kubernetes, which drew me to Google a few years ago, and then Cassandra, which I encountered in practice earlier at Apigee when we were building a large-scale infrastructure underneath API management services and products, to most recently at Autodesk where we had a five petabyte Cassandra infrastructure that was streaming in all of our manufacturing, production, design, engineering information from Autodesk Fusion and from other products like that. So I kind of got to see the power of Kubernetes and Cassandra separately, and coming to DataStax the opportunity to put them together was just too attractive. So now we’re marrying those two technologies and trying to bring those two communities together, which is super interesting, because site reliability engineers, which are kind of springing up all over around Kubernetes as a dedicated role, it’s kind of where DBAs who used to manage Cassandra and Oracle want to go and where they’re growing towards. So it’s really interesting to see how the technologies emerge, how they affect each other, and then how the communities start to change and adapt based on what the technology capabilities are. It’s a lot of fun.

Well, it sounds like scaling down in that is just almost just as important as scaling up, right?

Absolutely, because when you think about the biggest objection to some high scale infrastructure, it’s YAGNI, right? You ain’t gonna need it, hence Y-A-G-N-I. And that’s a very reasonable complaint, like, why do I want something that’s gonna run to 100 nodes, I don’t even know if anyone’s going to use my app? So if you can be there in the beginning, then you have a chance to be there in the end. Otherwise people come to you because something that they did fell over, and that’s Cassandra’s historical reputation, is it’s the database of last resort, like, when everything has let you down for scale, then come to Cassandra and she’ll take care of you, right? Macy’s I think bet on Cassandra about seven years ago, and believe it or not, they have had 100% uptime on their data tier. So some fairly amazing things. But you only do difficult things when all the easy things have been used up. So what we’re trying to do is scale down and even make the minimum three node version of Cassandra easy.

Well, in a DevOps world, and as that shift that you talked about that’s going on in how we run infrastructure, it really sounds like you have the flexibility, the scale you need, you have the reliability and performance. But is it something that you do across the board for all of your data needs, or do you silo it and do it for just specific aspects of your operation?

Yeah, I think Cassandra is part of the open data stack, right? So if you look at what companies are doing with their operational and their analytical data infrastructure, you’ll see that there’s kind of a common set of things, there’s no one thing. There’s Kafka and Cassandra and Hadoop, you’ll see Spark, you may see Samza, you may see Pulsar. There’s a number of different technologies, there’s an open data stack which is sort of an ecosystem of these open data technologies, and some of them are really predominant on the operational side. So when you think about the application plane, that’s often talking to Cassandra’s, talking to Kafka, and then you get into the analytical side and it’s talking to Spark, it’s talking to Samza, it’s talking to Delta Lake. So you have kind of still two modes of operation, of your operational or transactional data infrastructure for your app experiences, and then you’ve got sort of a warehouse or lake or lake house architecture that’s serving your analytics and your reports. So all that stuff has gotten eaten up I think by open source in the last decade, and that makes this whole field of play pretty interesting and still wide open.

So what’s the role of APIs in all this?

Well, so what we found, and as some folks might not know, you and I met over a decade ago, as we were probably late 2009 or early 2010, I think, 2010 I believe, yeah, I met you and John Musser of course at ProgrammableWeb. We were all trying to figure out, how do we communicate the power of APIs in the world that we see ahead? And so we all worked on that, you evangelized, John measured, I tried to help offer business strategy around APIs, there’s a whole bunch of, obviously there are hundreds of people involved, thousands of people involved over time. And now you see companies like Kong or Solo, and API management’s what’s old is new again, right? VMware is coming out with a whole new fleet of API management capabilities, so that’s awesome. The gap that we always found at Apigee about APIs was that you didn’t have enough good quality APIs to offer, right? As an enterprise, let’s pick on Sears or Kmart, you have a bunch of folks internally building APIs using HTTP and returning XML payloads, eventually JSON payloads, but they’re all kind of friends and family for how you run the enterprise. Often those aren’t really worth putting out there to customers, to developers, to partners. So there’s work required. So we thought we’d start there, take the APIs people cared about, get them in the gateway, and help grow them up and move them on. But the really tricky part, as we found, and I think as MuleSoft did a brilliant job, and I profited from Ross Mason and the team there adding API capabilities on top of their integration middleware, is getting access to the data, getting access to endpoints, and bringing lots more things into the world of APIs was always the hardest part. So where DataStax fits into that world, and where the open data stack fits into APIs, is we have an opportunity now to get at the data directly. When we were building microservices 1.0, the data was almost a side effect of the behavior you wanted from your app, and you built your API to power one app, and then you built it to power two apps. Now that API might be powering dozens or hundreds of apps, and there’s an entirely new set of folks, data scientists, machine learning specialists, business managers, product managers, who are trying to figure out, what’s the user journey across everything we provide, what are the different products that we could provide out of data, let me get at that data, never mind the microservice. So this opportunity to say we can actually plumb you directly into the data layer, we can start to give you well structured APIs automatically as a side effect of building software, there’s a lot of power there. Everybody’s figured out the data is really valuable, but how do you get human beings to change what they’re doing and go and make that available? That’s really hard, because all of us are working hard already, the easy jobs are already taken, and the product manager is sitting in our head telling us that we need a new feature in the app, and somebody else says, you know, it’d be really good if you did some domain driven design and thought about the structure of the data in this way and allowed access to all these other ML ops and data ops folks. Yeah, that’s a job for Tuesday, like, I gotta fix Monday. So our opportunity collectively is to make data way more accessible, and what we’ve learned I think over the last decade is RESTful APIs, and now I believe GraphQL based APIs, plugged directly into the data, give us an enormously greater agency to get our data into places that it will be useful. And so we’re going to see an enormous growth I think in the population of APIs available in the next five to 10 years.

Yeah, I think we underestimated, I think we thought that we could just build CRUD APIs and keep up with, you know, tap into the core database, but I don’t think we grasped the scale, the scope, but also the velocity forward that’s going to be required to do all of this. And I think we had a lot of the parts and pieces in place, but I think there was definitely some gaps in our vision, and now we’re getting there. I would say, like you said, microservices to, you know, moving on, that’s the first version of microservices. I feel like, oh, we figured it out, we’re like, oh, we need to do public APIs, products, and then we’re like, oh wait, no, actually internally we got to get our stuff together and figure out, and so we started getting a handle on that and stabilizing, we figured out the velocity, I would say, where, when you touched on this, we started widening the pool of people who can be involved in this conversation and identifying who the roles are, and then now we’re ready for this kind of next wave of growth. And I would say the database moving from a power center to the gateway moving to a power center is an interesting kind of plane, as you mentioned, VMware getting there. So I think that reflects, everybody’s got their own gateway now, and I think that reflects that evolution. So how else is that layer, what else is going on in this, as far as people who are the people that you see at this layer of new types of API developers and people being able to deliver value at this layer? Is it the data scientists, or is there other roles?

I think it’s interesting that we all saw microservices 1.0, but we didn’t iterate over time to see where those teams were going to end up, and I think if you follow those teams you’ll see who the new players are. So when we got our microservices teams together in the 1.0 era, we’re like, look, full speed ahead, feel free to make any choice that gets you to the future faster, just build, build, build, build, build. And that’s awesome for getting your first progress, but then of course human beings have behaviors, they leave teams, they get bored, they get promoted. So you have churn on that microservices 1.0 team, and then part of it stabilizes, and people want some other capability, and so you end up putting the first version of the microservice into maintenance mode, but then people no longer remember how that data actually got wired, the cluster’s growing, you need to pay for more servers, you need to pay more licenses, but now you’re almost in a sort of Cobol and Fortran in Y2K problem, right? Literally it’s only been two years since you built the microservice, but nobody’s left in the team or the company or the organization who remembers how that thing was built. So as you survive, you usually get promoted into management, and so once you’ve seen that you go, let’s do that differently this time. As the old joke goes, you go to the doctor and you say, doctor, doctor, it hurts when I do this, doctor says, don’t do that. So the first step is, don’t repeat the past. So how do we start reconciling these things? Well, we start to see data architects really showing up for the first time in software teams. I’ve seen a lot of data architect titles in data and analytics reporting to the CDO, and usually what they’re doing is their Informatica architecture, right? They’re trying to figure out what’s the ETL jungle of how all this information gets in and ultimately turns into governance or reports or audits or whatever. But data architects showing up on the application side, you know, directors and VPs of data services for applications, that’s a big shift, starting to create a coherent platform engineering team, something we see a ton of. You might see data engineering, platform engineering, but these things tend to start to stick together in a centralized organization, and that’s where people have the wherewithal, the training, and the critical mass to ask bigger, longer term questions. Because again, microservice 1.0 works perfectly, you can do it every time, but eventually you’re going to end up being drowned in things that you can no longer maintain. So once you start to add, what will we do with this microservice in 10 years, you have a different kind of person, you have a different conversation, and you constitute a different team, which looks like a platform engineering team or infrastructure engineering operations. That ends up being really helpful for the evolution of, how do I get some of that data to the data scientist? The data scientists tend to live in a very different group and almost never talk to the platform engineering team unless they get convinced to. One of the reasons for that is infrastructure and platform engineering tends to report to the CTO, the chief technology officer, or the chief information officer, whereas the data scientists usually work for the CDO, or the chief data officer, and these are two very different tribes, right? The data folks, in some ways culturally they were set in the 70s as the people who could only get overnight access to the mainframes when they weren’t doing the actual valuable application processing. Oh, I want to be able to pull this information out, oh, well, it’s not valuable enough, it’s just reports, it gets in the way of us making money, so we’ll give you access to the mainframe from midnight to 2 a.m., right? Hence the introduction of batch jobs. And so you had this kind of class-based society of the upper class, right, the application engineers who were building the things that got you the value and got you the transactions, and the underclass which were the data folks. Now that’s starting to change. Now that everybody says that data is the new oil, says the other movement, that we see is software engineers moving into data operations and being called data engineers. And there’s some amazing, there’s a data engineering podcast, there’s a few I can think of, a couple that are really powerful to talk about this move to bringing more and more people with very strong computer science backgrounds into making sure that the data is fed properly and handed in to data scientists. And of course in corporations, hierarchy, hierarchy, hierarchy, power and reputation often come from who’s getting paid the most, and as data scientists have become obviously valuable and very high paid and hard to find, the priority of data scientists in the organization, and therefore the role of the chief data officer, those have all gotten promoted, they’ve gone upwards. So now there’s more of a peerness between the CIO/CTO group and the CDO group, so we’re starting to see a little bit more balance.

Interesting, it’s that expansion of the universe, and we have to really prioritize and make sure data scientists are being able to continue asking and answering the questions they’re doing, but continuing to ask the hard questions over a long period of time. How does, you mentioned model ops and ML ops earlier, how does machine learning fit into this expansion of the universe that’s happening right there?

So there’s older machine learning, which is kind of pointed at understanding fine-grained patterns in our customer behavior, so we can come up maybe with better segments, or we can do some mathematical analysis of our supply chain, we can do some linear programming. I think of that as slow and smart, right, it’s a methodical application to a set of data that you’ve isolated and that you’ve stabilized, right, you’re looking at historical data, it’s not going to change, done. Now we can ask some big questions about the business or some small questions about the business, like, how might we tune a pricing guide? You might have an algorithm conceptually, but you’re not running your pricing algorithmically, dynamically, right, it’s not happening in real time, it’s happening offline. So that’s kind of the historical application of models and ML and data science. What’s happening now is everybody wants smarter apps, and the reason the stakes are really high for having smarter apps is that if the applications have enough traffic, right, if your website has enough traffic, if your app has enough traffic, if it’s intercepting enough customer and partner behavior, then your ability to dynamically nudge their behavior is the kingmaker. So what is required is you have to take these streams of data that are operational in nature, they’re coming out of the app, for the app, by the app, into the microservices or whatever their backing infrastructure is, and somehow you have to interject an opinion that is different from the opinion you had yesterday, right, the system itself has to learn and make slightly different decisions, make different recommendations, understand you better, make particular nudges, suggest different actions. And that’s this other domain of data science that moves slowly. So now this interesting gap is, in DevOps we have a really well realized culture and practice of applying lean manufacturing to app development. It’s really an app development practice, right, how do I get the app up, how do I get the app infrastructure up, and how do I keep that moving forward, how do I minimize my cycle time or my takt time, how do I maintain availability? That’s actually really well structured now, you can find a lot of different job roles and a lot of competencies in DevOps. Then we get to data engineering, which is the act of taking data out of those operational databases or data infrastructures and moving them somewhere else, that historically is very kind of clunky, batch oriented. Now that’s becoming data ops, they’re asking the same questions that we asked in DevOps, how can I make this lean? And the core of lean of course is flow, you want a pull-based economy where the thing is ready when you pull it, it’s not prepared in advance and required to push, that way you avoid batching. Now data ops then feeds into ML ops, because once you’ve got the fresh data, it’s now hopefully in a state where it’s close to what the data scientists want to be able to consume, so they’re relying on fairly fresh data of a particular format that they can find new features in, right. If Spotify could find a feature that improves your likelihood to stay on Spotify and listen to the next song by one percent, you know, times hundreds of millions of customers, that’s a super valuable thing to find. So ML ops comes into play where you say, how can we make sure all of our machine learning specialists are working on roughly the same copy of the data, that is, they’re iterating their features, that we can train other models in fact on the parts of the data that they’ve identified, we can reproduce their findings and their features, start to make that a little bit less of a wild west, which is kind of what machine learning workflow has been, kind of a bespoke nightmare, into something that is more structured, more consistent, more replayable. I think of the core of DevOps really about replayability, when you say this thing is broken, let’s fall back to what we had before, we’ll just rewind and hit play on the piece of infrastructure that worked before. Data ops, same thing, ML ops, same thing. The final novel piece that we’re seeing showing up is model ops, which is not about improvement in the speed of developing an ML model, that’s ML ops. Model ops is recognizing that applications and models are fundamentally different creatures. An application is like a recipe, you type out the recipe and you can trust the application’s going to do that again and again. A model is like a puppy, once you put that puppy in production, it’s going to learn things, and we have terms like model skew and model drift, and these are all kind of changing behavior of the model as it finds the real world, and sometimes those behaviors aren’t what we want. So model ops is about creating the kind of observability and traceability that we get out of DevOps for applications, but applying those principles of observability and governance to our models when they’re in production. There’s of course a huge gap in the return path to get models from model ops serving at the scale that they’re needed to in applications that are hosted in DevOps, and that’s really kind of the near future. I think over the next two to three years we’re going to see that gap start to close and a lot more consistency around how those teams work, what are the tools and systems of tools, and how do those things integrate. Above all, how does all that sit on top of data whose primary job is to make sure that you can build new features in your apps and engage your customers and partners, how does that emerge as a first class citizen so that all the different parts of the community in an enterprise that want to get at that data can get at that data, hopefully as soon as it’s written and not wait until the next day.

Yeah, and much more tailored and precise than it was before. It’s not just quick access to raw data, it’s much more dialed in and exactly what’s needed for the data scientists, for the models, there’s much more velocity and then alignment at all those layers. And that model ops to be able to really dial that in sounds like you could really fail fast in that and find what intelligence works or doesn’t work and really dial that in, and then keep that feedback loop kind of feeding it and evolving it. But now, does, in a DevOps world, does everyone got to be an ML expert in that reality, or are your ML people just specialized on creating the models and then optimizing operations and all of that?

That is a fantastic question. I tend to think that it’s going to show up in two different modes, right. So the first mode is the one that we know already, which is you’ve got folks who are coming out with, you know, maybe a PhD in biophysics or other scientific background, increasingly we’re seeing people with a masters of science in data science and ML, they’re going into the machine learning specialty and they’re building out these capabilities to make better, faster decisions, including decisions that can be made dynamically. But that’s a small number of people, right. What I’ve learned is the data scientists take a long, long, long time to make. You have to emphasize, in data science it’s about a scientist who can data, it’s not about a data person who learns science, right. Science is like a way of being that has to start when you’re about three or four, right. So we’re not going to see a huge expansion of data scientists compared to how many jobs there are to be done, right. Last I saw, we have on the order of like one and a half, two million data scientists actually practicing in the world, and demand for something like twice that. You can find lots of numbers, but I think whatever numbers you find will show you that there’s a lot more demand than there is supply, and it’s going to take years to build more data scientists. So where does that leave us? That leaves us with the other mode, which is developers. So developers probably aren’t going to become data scientists and machine learning experts overnight, or maybe even ever, but what they could do is they could become better drivers, right. You don’t have to know all of the models and techniques in your car to drive it properly, and as cars become self-driving, you’ll only have to know your destination and kind of your policy. So this emergence of AutoML, the ability to take the application data that you already have and have a machine, through, you know, could be generative adversarial networks, it could be a range of techniques, where you can spin up a set of models dynamically and see if something in there is valuable to the developer. A classic thing that you would want to do as a developer is count, right, it’s not just about transacting, it’s about telling me how many transactions I did and how many in unit time. That’s something that a database or a data service should probably provide out of the box. Another thing that they may want to know is, what are my outliers, what are my best, fastest responding customers, what are my least engaged customers, that kind of outlier statistics on the data. So the birth of statistics that you can get about the data that you’re already writing as an app developer starts to make you look a little bit like you’ve got some ML magic right there in your data fabric that you are using as an app developer, without having to learn how the model’s made, you can learn how to use the model. So I think data services will supplant databases, and what I mean by that is a data service is something that is not a monolith, it’s something that allows you to plug in a bunch of extra pre and post smarts, particular algorithmic capabilities, different classes of query, different ways to interpret your data, maybe through a graph, in order to give the developer a gift with purchase. You used us for getting and putting your data, let us give you a whole bunch of bonus extras. So I think that’s the second big mode, right, so that’s how we’ll kind of take care of 25 million JavaScript developers that GitHub says we’re going to have in the industry as professionals by 2026.

So you just merged kind of, you went with the car analogy and then applying it to DevOps and developers, and it’s reality. So if I take that even further, Uber, DoorDash, delivery, so your developers can take these models out, they can drive them, they can apply them, use them, evolve them, break them, figure out what works, what doesn’t, give you feedback out in the field. And so it’s not even just a personal, it’s a commercial thing. And to kind of bring this back to John Musser, you mentioned John Musser earlier, we did an episode with him about Ford, and he talked about what you have when you have an entire Ford fleet in Europe with internet connected cars that are producing all this data, these data points. So what does that translate to, using your analogy, to DevOps and model ops, like you’ve got a couple thousand developers of varying skill levels out driving your models, pushing them, applying them, making sense of them, with a relevant feedback loop to have control and have governance over that? That’s huge.

I think it’s increasingly within our grasp, because I studied artificial intelligence and neuroscience a long time ago at UC San Diego, I graduated in ‘94, right, and we talked about AI, and AI was always magical, always scary, always over promised and always under delivered. So what I think we’re seeing now in the field of applied AI is actually a lot of automation, right, a lot of personal automation, personal augmentation. And my favorite example that’s relevant to this conversation, of augmented intelligence as opposed to artificial intelligence, is of course GitHub Copilot, and overnight that combination of words has sparked the creative imagination of millions of people in the technology industry. When I had the privilege of working at Google with Melody Meckfessel and with Eric Brewer, back in 2016, 2017, 2018, we looked at how could we make Google internal developers dramatically more productive, and what we started to realize is we have all these insights about the Google fleets, and the two billion lines of code in the monorepo and how that changes, we could actually in the process of checking in your code into Piper, right, into our internal infrastructure for managing code, while accepting your equivalent of a PR, right, for normal GitHub inspired folks, we could actually predict if some aspects of your code might fall over in production in three months. So you can imagine like a super linter, right, that could see in the future through ML and understand some of the things that we’ve learned about production, and so we could put that recommendation right in line as you checked your code in, that’s the perfect place to catch it. So that concept, we thought, oh, that could really be pretty powerful if we get that to enterprise customers, so we started this group called Google Cloud DevOps, a lot of trying to figure out how to bring a lot of Google internal development infrastructure to the rest of the world. Then we worked on acquiring GitHub, of course, as you all know, Google did not succeed in acquiring GitHub, Microsoft did, it was very close. But I think you can see that the net of all those concepts has been made real in GitHub Copilot. Now why is that relevant for self-driving models for developers? Because again, it’s a developer who’s getting some augmentation. So imagine if you will, you’re sitting at your IDE, you’re writing some code, you’re accessing some data, the copilot can see that you’re accessing the data, and the copilot also knows that eight or so models have been spontaneously generated around that data that you don’t know about, but if you want to click here, you can see what that model is generating, if you click here, we’ll inject the code directly into your TypeScript file and then run it and see if that’s useful, oh, and we can link it out to some information on what kinds of use cases might be applicable for a model like this, what other people have done. And now we can start to build out these cognitive affordances, right, almost this cognitive exoskeleton that lets the developer know just enough about how to drive those models so they can drop in something that makes their customer experience just a little more delightful, makes them just a little less likely to put in wrong input, makes them just a little more likely to make a better choice faster. So this whole idea of ubiquitous weak AI that they talk about in Competing in the Age of AI, a recent book by Marco Iansiti and Karim Lakhani, that ubiquitous weak AI is going to get generated off of these kinds of tools across millions of developers, it’s not going to be a narrow breakthrough of strong AI by a very small number of data scientists working at a tiny number of companies, it’s going to be everywhere.

And people are really captivated by the autocomplete in an IDE, but you’re saying it’s actually, you have to zoom out, you can’t focus on just that little piece, it’s everything around that, it’s the ops, it’s that safety net but also armor or augmented capabilities for that developer on top of that, and then in aggregate and in scale, all with really smart people optimized behind the scenes making those smarter, creating better models, feeding that and gathering from that feedback loop, and it’s about that full motion, not just the individual motion, right.

Absolutely. And what are the side effects of large scale automation, as you can see in any industry, whether it’s manufacturing or military or software, is you go from doing to monitoring, because your work is more on the automation than the doing, right. You’re not doing the CLI, right, you’re not sitting there at the command line interface pounding keys to solve every operational ticket, because you’ve gone from being an operator to an SRE, so you’re engineering automation. Now once the automation’s engineered, what do you do? You sit back and you’re monitoring, you’re looking for, building good graphs, what are the outliers, what are the signals or the alerts that this thing should send me when some part of it is not working. And that lets us build systems of much higher scale and much greater complexity. When you think about the need to build dynamic decision-making models that can inject their ideas into the application as it’s going, you would imagine that if the application gets updated and there’s new data, there’s a new data schema, there’s some shift in the normal ranges of that data, an alert ought to be generated in the ML team to say, your model hasn’t failed yet, but boy, it’s super likely that your model’s in for a world of hurt, so you might want to stop whatever you had planned to do tomorrow and just work on understanding this application change, get that model trained, use all the functionality that you have in your ML ops infrastructure, get it out in your model ops environment, make sure it’s working properly, and get it back in. So what we will ultimately have to do is have this lean approach that goes from DevOps to data ops to ML ops to model ops, and then all the way back to DevOps, and that’s going to require automation, observability, and a sense that you are building a system that you are going to have to monitor, and the more intelligent that it can be, the more that can be an alerting task where it tells you what’s up with it, rather than it being a monitoring task where you have to figure out what’s going on.

Wow, yeah, and it’s like what we’ve been iterating and evolving the awareness with CI/CD and the code layer over the last decade, but this is that next layer. We’re getting our data house in order, we’re now, this is that next layer of pipelines and linting. I really like, you kind of captured me with that linting piece, like, how are we going to lint and make this safe and fail, and really truly enable our developers to fail fast and do things and create the data and test the boundaries and have that feedback loop that allows us to do this sensibly.

Yeah, we want you to fail in novel ways, we don’t want you to fail in ordinary, boring ways, that’s where the linting comes in, right. So your hyperlint should predict what’s going to go wrong with the code you’re writing because it saw somebody else do that, and like they hit their head on the doorway coming through, so you know, you should duck, don’t write that code that way. But your ability to replay, right, this whole x-ops, right, I really think is about replayability, can you go back one step when the step fails, right. It’s a bit like mountain climbers going up a mountain, right, you carry all that rope and the hammer and the pitons so that when you inevitably fall, hopefully you only fall about 10 feet and not a thousand, right. So hyperlinting to predict the boring failures and prevent you from doing that hopefully, and then replayability so that when you go out and make novel failures, they don’t kill you and you can learn and iterate from there.

Wow, this is pretty powerful.

It’s going to be an amazing decade for software, my gosh.

Yeah, I kind of feel like, I felt like the last API wave for me has just been training and getting my PhD, I feel like the next is going to be actually doing what I was trained for, and I’m not entirely sure what that’s gonna all entail, I’m kind of winging it, but I’m pretty excited about it.

I feel the same, I think a lot of us do. So many of us have cut our teeth on compute, right, and what we’re learning about compute is replayability comes from containerization, and I’m seeing this huge wave, right, there’s obviously been data pioneers working on, you know, Hadoop, Spark, Cassandra, Kafka, you name it, who came a long time ago, like a decade and more, but there’s a herd almost of us moving from compute and apps into data, and so I think we’re bringing a sensibility of what makes sense in a fairly well understood flow environment in DevOps, and then saying, what are the technical affordances, what are the tools, and what are the systems of tools that we’re going to need to make the rest of this efficient. And frankly I think we’re like on the five-yard line, I think, you know, data is such a wild west, there’s so little prior art about things like containerization of data. When I say, hey, should we containerize data, almost invariably somebody says, that’s a really good idea, we should do that, but where’s the prior art? It kind of doesn’t exist. When we containerized compute, the prior art was in Unix processes, but the containerization of data is like, I don’t know, it’s a little bit of math, a little bit of storage, it’s some tricky stuff, but we have to nail that so we can get this replayability, and when we can, then we can link DevOps and data ops and ML ops and model ops, and the whole system can work real well. But we’ll get rewarded I think for every small win, so it’s not like we have to get it perfect first time, thank goodness.

Yeah, well, lots of good stuff to think about, I’m gonna have to wrap it up for today. Moving forward, you’re going to be one of the last episodes in season one of Breaking Changes, and we’re figuring out what season two is gonna look like, and it’s gonna be very topical based, so one, I want to get you back, I want to do another session on model ops and ML ops, and then we didn’t even touch on observability and traceability. So if you’re game, coming into the fall and into next year, I would love to do two more sessions on just those areas, and then give us some more time to keep expanding this conversation. Does that sound fair?

Absolutely, it’d be a privilege, there’s so much to learn, and frankly it’s just so good to hang out with you again after so long, so thank you so much for that.

Agreed, agreed. Well, thank you, Sam, I really appreciate your time today, insightful, I learned a lot, I didn’t even stick to the script, it was just natural as always, and I appreciate your time.

Likewise, thank you, Kin.

All right.