Shai Almog, Lightrun

Thank you for tuning in to today’s TLDR episode of the Breaking Changes podcast. I’m your host and Chief Evangelist for Postman, Kin Lane. With Breaking Changes we explore specific topics from the world of APIs, but look at it through the lens of business and engineering leadership. Joining me today we have Shai Almog, developer advocate at Lightrun and co-founder of Codename One. Shai took me down the rabbit hole of debugging in a production environment, sharing how in some situations how essential it is to be able to debug complex systems in a live environment, finding it in their natural state. Let’s start simple, let’s start with the basics. Who are you, what do you do?

I’m Shai Almog, and I’m the developer advocate for Lightrun. Before that, and during that actually, I co-founded Codename One. And before that, I’ve been a programmer professionally for roughly 30 years, working throughout the industry in companies as varied as Electronic Arts to big major banks and security companies, and you name it, practically everything that’s available. And during a long period of time, I worked as a consultant. In that period of time, I worked for, I think, over 100 companies doing various types of consulting. And a very long period of time was spent at Sun Microsystems, where I worked a lot on low-level virtual machine work and things of that type. I also worked a lot in enterprise back end, mobile devices, and things like that, and got all the way low into the low-level assembler code of the VMs, and literally wrote a VM from scratch as part of my work at Codename One, all the way including the GC, including everything, and it’s used in production quite a lot, actually. And on the other hand, I did big enterprise stuff for everything from major companies in practically every industry. I worked in telcos and banks, insurance, you name it. And right now what I do is I work at Lightrun. I was actually the first developer hired by Lightrun that isn’t a founder, and I literally wrote a large part of the code here, mostly the server stuff, also the plugin and the interconnect stuff, mostly architected the basic concepts underlying it. And about earlier last, mid last year, I started transitioning to a role of developer advocate, which is something that in retrospect I’ve been doing since, well, over 20 years to some degree, because when I founded my consulting company, I started working the community. And one of the things I did a lot was lots of open source. The work I did at Codename One is pretty much entirely open source, and also a lot of the work I did at Sun Microsystems and later on at Oracle, it was also open source. And I’ve been very active in the community in general, speaking and just supporting people in forums and doing all that sort of stuff, lots of Stack Overflow and all of that.

So I’m kind of, as a consultant, I think it’s inherent that we try to solve problems. It’s a very deep thing in me, and that’s one of the things that attracted me to the work at Lightrun. We do this thing called, we actually until recently called it continuous observability, and it’s something that just changed over the past week. One of our other developer relations guys came up with developer observability, which is a much better name, I think. And that’s what Lightrun does, and it’s a very unique approach to observability. Because the field of observability is a very, it’s a great field, but it’s very much targeted at DevOps. And DevOps are great, and they’re developers to some degree, but they have very different needs from the needs that our R&D developers have. They need different things, they need different information, and it’s important information to have, but it’s very different from the type of information that we as developers need. A DevOps would want to see graphs and statistics about various things, performance, errors highlighted, and also track the big picture sort of thing. And a developer needs to dig deep into a specific flow and get information there. Our home turf is the debugger, the IDE, maybe the logs to some degree. I think that’s where DevOps and developers somewhat converge, in the logs. But our home turf is the debugger and the IDE. And DevOps, on the other hand, their home turf is mostly the monitoring tools and things that they need to do there, and prompts into the Kubernetes and things of that nature, where they work at a bit of a different scale, more in the production and less in the staging or the development.

And that’s part of what we’re trying to fix, the shift-left movement. Where we take the information that we somewhat lost, you know, we’re both old enough to remember that age where we used to go to the server and kick it to hear the hard drive spin. Oh yeah, it’s still working. That period, and that had an advantage, because you knew it worked. You literally saw the machine, you could touch it, you could log into it, you saw what’s going on. And with Kubernetes, I mean, it’s amazing, it solved so many problems and it made scaling amazingly easy, but it kind of also threw us a bit backwards in some regards, because now every small problem scales. The problems become huge at a huge scale, and it’s much harder to solve these problems at that scale than it was back in the day. So we have better tools, we have CI/CD at a level that just didn’t exist 10 years ago, we have logging and ingestion which didn’t exist and didn’t cost nearly as much then as it did 10 years ago. But still, even with all of these advances, we’re still in a position that’s a bit of a regression when it comes to development bugs in the code, because now at this scale things like race conditions, where in the past you might not have noticed a race condition in production because you had five servers and VPSs or whatever, and one of them was active all the time and the other ones were sort of okay if it fails, and the traffic wasn’t distributed properly. So now we have these things coming up and down and you have a thousand agents running concurrently in different containers and something somewhere failed in container X over there. Well, you obviously can’t connect a debugger to debug the production there. It’s not realistic and very insecure. Saying insecure is kind of like, so I’m just saying it. It’s a huge problem, and solving those sorts of problems in production, I mean, that’s the stuff of nightmares. Literally, you wake up for an on-call and you’re sweating, because you literally have to start looking and digging. Our symbol here, in recent conferences, is Sherlock Holmes. We literally come with caps and a pipe, because that’s literally how you feel when you’re trying to investigate something like that, and you’re looking through clues maybe hoping something made it into the log. And if it’s not there, then you’re stuck. I’m trying not to use a dirty language here, sorry. But that’s the feeling, when, I don’t want to break the age rating here, but luckily, it’s so frustrating. I usually say that’s why I’m so bald at this age. I’m actually old, so it makes sense that I’m bald, but I think I lost it faster than my dad because of, you know. Anyway, that was my introduction. I need to heart that. Oops, sorry.

It’s all good, it’s all good. This is why we’re here, to just talk through all this, understand your view of the landscape. So if I was to summarize what it is that you’re trying to solve, with DevOps and Ops there’s what you see when it comes to observability. Observability is, in the simplest terms, the ability to monitor the state of a system using its existing outputs. So in what most people are doing, they’re monitoring the outputs of production to understand the state and the health of the system. But developers are working in the IDE locally. Source control kind of connects that, CI/CD connects that to different states, different environments, whether that be development or staging or production. But Ops has the shadows and the blind spots to developers’ local and other development environments, while developers have blind spots when it comes to production. They can’t see everything that’s happening, they need help understanding. And so, I mean, I wouldn’t say I’m a fan of working in production, because I’ve had to do it similar to you, I’ve had to do it because, A, it needed to be fixed now, that’s the thing, and B, I can’t recreate that situation of all the dependencies in my development environment, so I just have to go in there, it’s the only way I’m going to figure this out and be able to fix it. So that’s the different incentive that developers have over Ops: I need to fix this, not just identify and see it, I need to actually do something about it.

You hit the nail on the head there. And there’s obviously, as you were aware, a huge movement of debugging production and literally deploying directly, especially in the serverless community, where it’s impossible to reproduce this locally. And we’re seeing it with lots of customers who have an environment that they can’t debug any other way. Because locally they have something that’s completely disconnected, just running unit tests, nothing else. And that isn’t very valuable in a microservice environment and a serverless environment. So debugging in production is hugely important in those environments, because you can sort of guess the fix from logs, but logging is coded into the inside. And if you have a process of release which takes, you know, two hours is a good release process, a fast release process, sometimes it takes longer, and you need to add a log because there’s a problem in production, you’re not sure what it is and there’s a log missing here, then this sort of churn is terrible. And you start that process, and in the middle of the process, wait, I want this additional information, and suddenly you put in too much information and the system in production becomes slow because of all the logs you’ve added, and it’s too noisy and it screws up your ingestion, making the cost skyrocket. So there is so much that you need to balance here to get the production into a state that you can understand what’s going on there when there is a failure that’s a failure in the code, a bug from R&D. And with observability tools, one of the things that’s really important there is that you can observe changes, observe state within the production, or the staging of course, but you can observe state without changing the code, without deploying new code. And that’s key. Because when you can observe a system without doing anything, literally see it instantly, it opens up this huge window of possibility.

So a bug and a problem, that’s the drama, that’s the big thing. There’s a bug in production. And this is what I talk about most times, because when you want to sell this to C-level executives or even to developers, that’s the drama, and that attracts us. But the value is so much deeper. So what I see with lots of customers is that they use this as a tutorial, or as an inspection tool to understand their own code. So for instance, a good example is, they hire a new developer and they need to learn the system. And one of the things that I do when I try to learn a new system is just run it in a debugger and start verifying assumptions. Oh, I think this code happens with these variables such and such, and it’s called through there in most cases, just place a breakpoint, sometimes put a conditional breakpoint on a piece of code and see what happens, and understand and verify that my assumptions actually happen. But the thing is, with a production system it’s really hard to understand what’s going on, because you’re seeing this, okay, I’ll see it locally, but reproducing the things that happen in production, it’s a very different environment. And here you can see the same sort of things. You can put a snapshot, which is similar to a breakpoint, loggers, things like that, and sort of learn the system and how it actually works in reality in production, and understand that your assumptions regarding the code that you’re trying to study are correct.

This is also very valuable when you’re trying to understand how the code works, because in a huge project with lots of changes going in and out all the time, with big things happening, it’s really hard to know if a specific piece of code is actually still active, or are there still users using legacy API 1.0. So you can task a developer with a task to start building a special bit of code to see who uses API 1.0 and see if we need to start sending out automated emails to those users, and all sorts of, that’s like months of work for a junior engineer to collect all of that sort of information and log it or save it to DB. With us, we want to see if someone uses an API that we want to deprecate, we just place a snapshot or a logger there and we see the user that’s using it. We can do that dynamically. We can put a counter to see how many people are using this specific API, or how many invocations are for a specific API, and test if something is actually used. When we see a performance problem, we can measure the performance of a specific method, or a specific line even, not the entire web service or something like that, because that’s often too coarse for what we need as developers. We need to know if, oh, this specific query, is that the thing that’s causing a problem in performance, how much is it taking, how many times are people using it in production? Because, you know, I can profile all forever, but I’m profiling my local machine. Profiling the production, even if it’s a bad profiling, even if it’s a level of profiling where you just do a log with a small timer, that’s a huge advantage, because you see how it actually works in the production environment.

Well, I appreciate your time today, it’s been an interesting journey. I don’t know much about debugging at this level, so it was very informative. I appreciate your time and being here and sharing it with everyone, and we’ll look at having you back to talk about some other specific areas when it comes to debugging and working in production. But I like your view of things, thanks for sharing.

Thank you so much.

Thanks again to Shai for stopping by. You can find more about Lightrun at lightrun.com, Codename One at codename1.com, and Shai is on LinkedIn. You can subscribe to the Breaking Changes podcast at postman.com/events/breaking-changes. I’m your host Kin Lane, and until next time, cheers.