Gregor Hohpe, AWS

Thank you for tuning in to today’s episode of the Breaking Changes podcast. I’m your host and chief evangelist for Postman, Kin Lane. With Breaking Changes, we explore the world of APIs through the lens of business and engineering leadership. Joining me today we have Gregor Hohpe, enterprise architect at Amazon Web Services. Gregor’s work with enterprise infrastructure has been the foundational reading for the last 20 years, and I find his pragmatic view of architecture and how we deal with change very refreshing. Let’s start with the basics. Who are you, what do you do?

Oh, what do I do? During the day I’m an enterprise strategist, that means that I meet with AWS customers, discussing cloud migration, but I do a lot more things. So I write books around cloud strategy, and my favorite topic is the role of architects and architecture and all the stuff that we do.

I’ve been following your work for a long time. I feel like your books are required reading if you’re coming from the SOA world, but even if you’re brand new to the API world, I feel like they’re timeless, and I definitely recommend the Enterprise Integration Patterns. It’s pretty fundamental, bedrock stuff that I think folks should tune into. So what do you spend your days working on?

So interestingly, yeah, the Enterprise Integration Patterns, that sort of classic, we wrote this in 2002, so that is 20 years, and then it was published in 2003. So I actually spent a reasonable amount of my time recently re-implementing some of the examples in the book on top of serverless platforms, so basically the same design ideas. What Enterprise Integration Patterns did, if we teleport ourselves 20 years back, it was EAI, maybe some early SOA days, so people started building distributed integrated solutions, but it was all very fragmented by different vendors. So what we did is define a common vocabulary that helps people design better messaging, asynchronous systems. Now, what I did starting last year is pick that back up, so I have 20 years of technology evolution to take advantage of, but I found that we’re still making similar design trade-offs, and using those design patterns is still really a good way of going about getting serverless systems.

I like it. For me there’s a lot of fundamentals in there, and it helps me to reread something, because I read it back in the day and then re-read it in today’s times, because it kind of triggers a bunch of emotions and thoughts about how we got here, what’s different from then to now. How would you see things? I know you’d like to see things in terms of constraints, as far as what limits us and what defines how and what we can build. What’s different from then versus now when it comes to API infrastructure and architecture?

Very good question. And as I said, I’d like to think about the role of architects and architecture. So when I consider what we’re doing, I always see a sort of coming from two directions, from two angles. The top angle is, really you have a certain intent, you’re trying to do something and accomplish something, and that’s where the design patterns really give you a better language to express your intent. Like people might say, oh, I put a certain component here, name your favorite product, that doesn’t really reveal what you were trying to do very much, that is your choice of service, of product. So what the patterns from the top down trying to do is they give you language to say, oh, I was building a message filter, meaning a series of messages goes in and then based on a condition some messages are passed through and others are taken up. Now you could say, aha, the reason I use this thing, if it’s Step Functions or EventBridge or any other of your favorite, this can be any ESB kind of product, now you can say why you’re using this and what you’re trying to accomplish, and I think that has remained as relevant as ever 20 years later. The bottom half, that is what products do I have at my hands, and you said the keyword, so now what constraints am I operating under? We often think we have a blank slate when we do software design, but the slate isn’t quite as blank, because we don’t want to build everything from scratch, we want to use products or services that are already there or open source projects, so we have some building blocks that you’re starting with, and those blocks come with some constraints. And that’s where in the last 20 years things have changed enormously. So when we wrote Enterprise Integration Patterns, in order to build any kind of sample application we had to install software, you know, run things locally, and one of the most tedious parts was, in order to make a little code snippet to say, this is what a message filter should look like, is ideally like five lines of code: get a message, we apply the predicate, if it’s true send the message, if not don’t send, if you want to be very simple about it. So those little five lines of code needed like 100 lines of scaffolding. This was all the stuff like getting set up, message receivers and senders, and Java JVM settings and all the other things around it. So the signal-to-noise ratio was relatively poor at the time, and that has changed a lot. So the constraint of having to install software and being dependent on the operating system that you’re running on your laptop, that has pretty much gone away with the serverless solutions. It all runs in the cloud, you can go to any kind of cloud shell that you have and basically do everything without ever installing a single thing. And the signal-to-noise ratio has improved enormously. You can use your serverless event routers and you can literally put your event filter in there, and you end up with maybe not exactly five lines, but maybe you end up with nine or ten or eleven lines. So the way we build these kinds of solutions has become a lot more expressive, because it shows the intent and it gets rid of all the noise.

Well, these constraints have gone away, like me having to set up some of the physical ones. I used to have to actually rack-mount servers and do a lot of that, those things have gone away, I don’t have to install as much software. But as these things are fading away there’s new ones emerging, new types of constraints. Serverless isn’t free of constraints, it introduces some new concepts that we have to wrestle with, some new trade-offs.

Yeah, and I think we’ve both been in this industry long enough to sort of see a little bit what the game is they’re really playing. We always have to work within the constraints we have. If you can’t get hardware, if you’re racking and stacking and it takes you a month to get any hardware, you will make certain design decisions, you will put as much as you can on the one server that you did get, you will maintain that thing forever, you will make a lot of design decisions based on that. So removing some of these constraints, like, yeah, I can get your compute nodes or the actual functional element, I’m no longer talking about VMs or containers, I’m now talking about an event bus, an orchestrator, a queue, a publish-subscribe channel, I can actually get functional elements, that constraint goes away. What we do, though, the game that we tend to play is, we have a new playground, we can do these things of, hey, I can have functional components at any point in time in any form or shape and as many as I need, and I only pay for when I actually have traffic. What we do is we push that new approach to the limits, because, oh, this is fantastic, so I want more of it, and then we find new constraints. And sometimes we find constraints on the technical side, that we build systems that ultimately don’t work the way we would want to, or we had mental constraints, so that we build something that we can no longer understand. And I see this cycle all the time. We’re in a certain pocket because of technical constraints, we make this go away, then we shift over here, everything seems so much better, but then we hit new kinds of constraints, until we remove those, and I think that cycle we’ve played quite a few times, at least since I’ve been in the field.

Yeah, I mean, I’ve been doing this since the 80s and then the 90s and the 2000s, I’ve seen several cycles come and go, and I feel like that’s helped me see through the change and address change. Do you have any other recommendations for teams trying to understand and wrestle with change and deal with these cycles? Is it just experience, you just have to be in the industry long enough, or are there other ways we can help our teams deal with this?

This is a very architect question, sort of the slightly tongue-in-cheek definition of the architect being the wise person in the corner, like, you tell us from the many attempts that you had, did you learn anything? So I think in this case it’s a mix. It’s good to be excited about the new technology. As I said, like the 20 years, it’s just day and night from implementing the same patterns. So we should be very excited and thankful about the modern platforms we have, like serverless, cloud, microservices architectures, event-driven things, streaming, AI, I mean all this stuff is just amazing. At the same time, you need to have an alter ego if you wish, which takes a very rational view to it. And the way I always think about it, when something new comes along, I’d like to ask the most simple but most profound questions. The one is, which problem does this really make go away or reduce, which problems stay, and then what new problems does it perhaps introduce. And I think it’s very good to have a balanced view. What falls into this category? Let’s take microservices, a very common thing. Well, it makes some problems go away, it makes the problem of software inventory go away, like, I have nine new modules that work and one that doesn’t, but I need to ship a single binary, so I can’t do that. So it makes that problem go away. It also makes the problem of independent scaling largely go away. I have a big application, some stuff is pretty benign, some stuff is really heavy, before that led to very poor hardware utilization, so I can separate those pieces, scale out the piece that’s very heavy, and leave the other stuff alone. What it doesn’t make go away is us having to deal with modularity, for example. I need to structure my thinking and my solution. A lot of the microservices discussion has now shifted to domain-driven design, because having a good design of modularity for your application, that is as much needed as before. And then if you look at new problems that it introduces, well, I have more moving parts, I have more complex runtime behavior, I have then things like circuit breakers where partial failures propagate through my system. So there are more things at the runtime that I need to deal with. And I think going through this exercise in your head leads to two really important results. The one is it gives you a much more balanced, a much more, let’s say, architecture kind of mindset, so I guess sometimes I could, and here’s some things I need to watch out for. But it also lets you think about, can I achieve similar results with other approaches? It detaches you a little bit from the buzzword. So let’s take the software inventory. Well, if I have a really good CI/CD system and I have really modeled a software, so it may still be a monolith but I have really good control over my builds and tests, well, I can probably reduce my software inventory even with a monolith, because with feature flags and other things, the one that isn’t ready just doesn’t go in, the other ones I build, I deploy, and off I go. So it really helps us to not latch on to these buzzwords and say, oh, everything must be microservices for some reason or another, it allows us to see the trade-offs that we’re making, and I think that is really key to the architect mindset. Shifting some problems, we just shift from left to right, we make different trade-offs, but we should see those clearly.

So help me, I like that, so help me apply that to serverless, because that’s one area, that’s in the same realm for me, I’m trying to understand and explain what the business value is, why my leadership should be investing more in serverless. So walk me through how serverless applies to all this.

So serverless is an interesting one, because, similar to the microservices, it reveals itself at different layers. Microservices started as, hey, I make my application into smaller pieces, but then you realize that in the end it leads to a different way of actually building software. And I think serverless takes that to a whole other level. So the first important comment probably is, yeah, I just compared serverless to microservices, I come from a very application-topology and application-structure perspective, because I think that helps us really understand. Too many people see serverless as, oh, it’s VMs, it’s containers, and serverless, and I think that’s too narrow a view, that’s just the compute runtime for one component, when the whole point of serverless is you can have many small components. So the interesting thing about serverless is that you have many interconnected small components, and that’s where it really moves the needle. Now that’s the architecture point of view. How do you translate that into why should my business care? I call this the architect elevator, my other book I wrote, Riding the Architect Elevator. You have exciting stuff in the engine room, how do you translate that into something that your boardroom, your penthouse, should be interested in? And that’s why I think the economic aspects of serverless play an enormous role, and they come in a couple of flavors. One flavor is, we already mentioned, essentially no setup cost. The thing is just, you need one, you have one. You have full automation, so no developers are spending days and weeks patching and configuring and installing stuff, that’s obviously really good economics, and those economics play out especially well if you run experiments. If you want to build something small and quick, not having the setup cost can bring your overall effort down maybe by two-thirds or even 90 percent, depending on what you’re trying to do. The next part that’s exciting to the penthouse is, you have very different economics. If we are honest about normal application delivery, the run cost of an application is kind of a black box. You have like 10 VMs and the thing runs on there, and the 10 VMs cost you pretty much the same amount of money every month, so you have no insight into your marginal cost. So let’s say you have an app that serves like 100,000 users, well, how much does one user cost you, how much does signing up one more user cost you? The real answer is you have absolutely no idea. I was talking to an airline, and we know the last two years have been super tough business for airlines, so they have a management system for all the aircraft, where’s the aircraft, maintenance, all that kind of stuff, as you would imagine. And basically, because due to the pandemic, they went from 100 aircraft down to three aircraft. Well, now you can guess what happened to the cost per aircraft. The cost per aircraft increased exactly 33 times, because the cost was all fixed. So that means they have absolutely no way of managing their IT infrastructure by unit cost. They have no idea how much does one extra aircraft cost me. It seems to cost nothing until maybe you have 150 aircraft and then a huge system upgrade comes and somehow strangely the 50th extra aircraft now costs a million dollars, because system and hardware upgrades, you cannot really manage IT like that. So serverless makes all that go away, and you can now start to discuss your IT cost by function. So how much does it cost me to manage one aircraft, how much does it cost me to sign up a new customer, is that profitable, is it meaningful for me to reduce the cost of signing up a customer, or should I rather reduce the cost of doing the shopping cart or closing a transaction, or whatever functional business you might be in. And that completely changes the way you look at your software.

And the cost-benefit ratio between those, is this an effective way for me to look at my legacy infrastructure? Does it give me the technical and the business toolkit I need to tackle my legacy systems?

So legacy is an interesting word. It’s anything that was done before yesterday, so it tends to be legacy. So I worked in Silicon Valley for a long time, and people always said, oh, you can do everything you want because you have no legacy, and I said, no, no, this works differently, we’re moving faster, so our legacy also arrives more quickly. We had legacy problems all the time, but we did something about it. So legacy, there isn’t a magic button you can push, because the very definition of legacy is generally something that’s hard to change. If you have an old system that’s easy to maintain and easy to change, most people wouldn’t call that legacy, they’re like, oh, it’s just the system that’s been really good, it’s been around for a long time. So by definition, changing something, getting something into a modern setup that is currently legacy, is going to be difficult. I think what we’re learning there is a couple of things. The lack of transparency is probably the worst part of legacy. You don’t know what you’re dealing with and you have no safety net, it’s like walking the high wire in the dark, you can’t see what you’re doing and when you fall it’s going to be very painful. So what a lot of people do is a couple of things. A lot of folks first just re-platform, like the lift and shift, that generally isn’t our favorite word, because you sort of get the same old thing in new packaging, but it allows you to get a little bit more transparency, how does this thing behave at runtime, I can get some better monitoring, observability around it, so I can turn one light bulb on at least by shifting this thing to a different platform. And sometimes these are emulators, simulators that run old COBOL things on a Linux box, and they seem like a crutch and they are, but at least they get you, like I said, they turn one small light bulb on, so you have some idea. Then on top of that people use incremental approaches, so boiling the ocean doesn’t tend to work very well with legacy systems. So people take one module, maybe even a read-only module, something that calculates something. When I was in insurance, calculating quotes was always a good example, it’s used a lot because people want to know how much is it going to cost me to buy my car insurance, so it’s used a lot, and it’s kind of stateless, mostly stateless, it’s a read-only calculation. So people take one small piece out of that and then you can run the systems in parallel, you can see if they behave similarly, and then you can repeat this. And serverless is a great target for these things, because, let’s take the quotation, well, now, a, I know how much a quotation costs me, so I can decide, let’s say an aggregator, an insurance aggregator, a comparison portal, I want to get insurance, please everybody tell me how much it costs, I can decide whether it’s worth responding or not, because if hardly anybody ever signs up, maybe the calls just cost me more money than it’s worth. So I can make an intelligent decision. And in many countries car insurance is a very cyclical business, many countries have like annual cycles for it, so you have a traffic spike and it’s very quiet the rest of the year, so I don’t pay anything if it’s the rest of the year. So we have some very nice stories where it can be done. It does require some grip, though, I always tell people, it’s not going to happen overnight.

Yeah, serverless is interesting and how it’s helping us address, I would say, in a little more structured way it feels like, than microservices. Microservices definitely felt like it was much more Wild West, but microservices have a little bit more of a set of constraints or logical constraints to it that help us think through this. But you mentioned monitoring and observability there. Would you say that is the differentiation, an important one, of legacy? So anything created yesterday is legacy now, but if we have that observability and monitoring and that understanding of how it works, is that going to shift what legacy means in the future? Like if everything moving forward from here has monitoring and observability into it and traceability, is that going to change how legacy impacts us in the future?

I think so, and for a couple of reasons. One thing that can really help us make legacy less bad is having things run on a common platform. What you really don’t want is the legacy that runs on some piece of hardware that nobody makes anymore, there are no more spare parts, and if that thing ever breaks there’s just like nothing you can do. So lifting things on, whether it’s the cloud or on premises, whatever you choose, lifting it on a common baseline underneath that evolves and is maintained, yeah, it takes a lot of the pain out. So even though one system might be old and built very differently from the other one, there is some common layer underneath. And in the past, if you were lucky, that common layer was IP networking, but now I think we can have a lot more things in common. Is it on a standard VM that is patched, does it have common monitoring, observability, can I see what the respective systems are doing? I think those items can really help us a lot. Where I warn folks, though, as I said in the beginning, the problems that go away, the problems that stay, and the new problems that come, is with the new systems, in order for them to not become tomorrow’s legacy, you need to realize that as you’re making smaller runtime pieces, we’re basically decomposing our application, that started with microservices and then we took it to the next level with serverless or functions as a service, I can now deploy individual functions, so we make finer-grained solutions, and not just at design time, we’ve been doing this a long time, at design time we make classes of methods very fine-grained things, now we have fine-grained things at runtime. And as I said, it makes many great things, scalability, elasticity of the solutions, cost effectiveness, etc., but I have more moving parts, so my runtime now gets its own complexity, and this is now the territory of queuing theory and back pressure and loops, where things sort of basically escalate, a small problem escalates itself into a large problem. And this is exactly where monitoring and observability are the best compensating element, because we shift some of the complexity into the runtime for good reasons, because it gives us capabilities that we like, stuff scales, it’s efficient and fail-safe, etc., but we need to deal with the new problems that it introduces, and that is, we have much more complex runtime behavior. So in that sense I agree that if you build a solution like this and you don’t consider monitoring and overall systems management, you probably build tomorrow’s legacy very much so. So if you have the transparency and the manageability of your solution, I believe you’re going to be much better off.

Yeah, I feel like this new world, the microservices and then the push and the serverless, when you’re focused on an individual function or a service, I feel like I have more control over what’s happening, and when I have my blinders on and I’m working at that kind of atomic modular level, the cognitive load is much lighter here than what it was with the monolith. But as soon as I pick my head up and look across everything and think of it, and go up the elevator as you would say a little bit, I’m not sure that I have as much control than I used to. Maybe DevOps is getting me a certain level of control, but how would you say that control or lack of control shapes our world when it comes to how we build systems?

So I think you can have the same level of control, but you need to make a conscious effort. Since we talked about Enterprise Integration Patterns and what has happened in 20 years, one thing that hasn’t changed as much in 20 years is that when people build integrated solutions, they always think about the pieces. Well, I have these two pieces and I just need to connect them, that’s how it always starts. So you very much focus on the pieces, and the connection is sort of, yeah, well, obviously I need a connection to make this thing work, but, ah, it’s just a line, I have two big boxes and this little line, and I think that’s good. Well, we learned 20 years ago, and we learned this time and again, as your system becomes more decomposed, the lines are becoming much more important. So I tend to give a talk at conferences where I show two very simple diagrams. It’s like the boxes A, B, C, D, four components, and there’s two systems that have exactly the same components but they’re wired together very differently. One is sort of a stack, like a layered, A talks to B talks to C talks to D, and the other one is basically fully connected. And the two systems have completely the opposite properties. On the left side, very clean architecture, I can replace one module very easily, but long latencies and maybe a single point of failure, if A can only talk to B to C to D, if C goes out nobody can talk to D anymore. The other system, which is fully interconnected, has almost exactly the opposite properties, it’s very resilient, if one path goes down everything else still works, it has shorter latencies, but maintainability is poorer, because if I want to take C out, now I need to redo all the connections to A, B, and D. And this is a great example, I think the lesson we need to learn time and again is the lines are as important as the boxes. The system topology is of equal concern as the boxes are. And it’s so much easier, like you said, to put the blinders on a little bit where I say, oh, I just want to work on my one component, it’s such a tendency, we need to actively counter-steer that. The overall system behavior is equally important to the behavior of the small component, and they should get the same attention and tooling. Like testing, I test my components, traditionally, so I test each component, I wire them together, you know what happens? Nothing works. That’s the classic we always have. It’s the same thing I say, like, you have a complex system, every server the light is green, all servers are running, but the system is not working. That’s how it always ends up, the problems end up between the pieces. So what I’m a big fan of is getting the tooling for the overall system to the same level of tooling and discipline that you would use for a single component. Coming back to the observability and monitoring, that is one key part of it. And I think the second key part of it is automation. Cloud isn’t just another server, the cloud is, it’s programmable, I can get my resource now via API. And that’s where I spend most of my time when I’m working on a serverless solution, is, can I make it so that I code the components in my favorite programming language for coding the components, but can I code the application topology, how things are wired together, can I also code them? And the answer is yes, like with CDK, which we have from AWS, and there’s a Terraform version of CDK, I can do that, I can code my application topology, and now both layers we talked about, the boxes and the lines, they both have a programming representation. I can start doing unit testing, I can do refactoring, I can do a lot of cool things, and that’s to me most exciting, because now finally the two layers get the equal attention and the equal tooling that they deserve. That, I think, is easily underestimated, how big that shift is.

Yeah, that’s going to give us a lot more control over the future and the direction that we go, because we’re not just at, I don’t know, it’s a much higher-level thinking, and we’re actually able to see the landscape rather than, we seem to get our blinders on and get stuck at the lower level. So how do you ride the elevator, how do you maintain your architecture level, your high-level view, and ride the elevator to the top floors, but then also stay relevant by going down to the lower floors? Because I know that’s a common challenge for architects, you like staying up here up high, how do you understand what’s going on on the ground floor and keep your skills relevant?

Yeah, the view is always better in the penthouse, so some people don’t feel the need to go back down, but the elevator goes up and down. I always say riding it once up, that does not count. So for me personally I think a couple of things are required. The one is, you’ve got to have a natural curiosity, you just got to be interested in what’s going on in the engine room. So when I started building the loan broker that was the example from Enterprise Integration Patterns, I’m just like, hey, I should just try this in serverless, we have all these cool tools, I’ve never used them as much, so just curious, like, what’s going to happen. I think that’s number one, you got to have the urge to be connected to reality. The good news is, and that comes back to our serverless topic, in the past trying something out in the engine room was super difficult. I remember when I worked as chief architect before, if ever I wanted to do anything hands-on, I would pair with somebody else, and that had some good reasons, and the good reason was, I like pair programming, it was a great way of knowledge transfer, but it also had a pragmatic reason, getting a development environment set up for any of our systems would have taken more time than I even had. It would take three days to even write one line of code for me, because all the setup. Now that serverless made go away, you can download some samples, you have some sample CDK, run that, well, here’s your system, I’ll make a change, deploy that. So basically the time to learning, and what you want to do is make a change, see what happens, learn from that, make another change, so the time it takes until you get into that mode has gone from days and days to maybe minutes, or worst case hours. So that’s the really good news. The third part is, there is so much debate about should architects code, and I generally say yes, but for a different reason than a coder codes, and I think that’s important. So the architect doesn’t code to become the best coder, then you should be a developer, I think that’s fairly obvious. The architect should code so they can understand the trade-offs and constraints better, all the stuff we just talked about. So when you do your coding exercise, your goal isn’t to write better code than the person at the keyboard next to you who does that for a living, but to come back with new insight, like things we just said, like, ah, the serverless thing is great but too many moving parts, it’s easy to forget the system image, the topology, so basically when I do anything serverless I want CDK, for example, because I got to be able to program my components, to program my topology, and that’s the way it should be. So architects have a different objective when they code, it’s not about learning the latest library that you might have or squeezing the last bit of CPU cycle out of something, for us it’s taking insight back that we can then use on other projects and really learn more about the system structure and how we build good systems.

What’s the best way for a developer, say with varying levels of experience, to start understanding how to program at the topology level and poking around and getting curious, when they may not have a lot of experience? What’s the best way for them to learn at that level?

So people learn different ways. I like learning by really building something from scratch. I remember a couple years ago, Kelsey Hightower wrote a book, it was like Kubernetes the Hard Way, and a lot of people say, like, why do you take me through all the details that Kubernetes essentially makes go away, why do I need to know all this stuff? Well, his answer was very easy, he’s like, well, so you actually understand what the thing does. Don’t just take the magic for granted, start with the pieces, like, here’s how you set up a cluster from scratch, and once you understand this you can get to use all the magic. I like that approach very much. It works for me at least. I didn’t use, when I started with serverless, I didn’t use any of the magic templates as much, I’m like, let me just build this from scratch, can I get a Step Functions to talk to an SQS queue and can I have Lambda consume that, and little struggles here and there, see how this goes, and then learn from that. So that worked well for me, really make baby steps, and give yourself almost like a controlled experiment environment, build something relatively small and play around, change some dials, put a queue, then take the queue out, put something else, see how that behaves. What that does, it overcomes the natural tendency. Experienced engineers that I know, I had a really great time a long time ago, I got to pair program with Ward Cunningham, and he was working on a PIC chip, like a tiny eight-pin little microcontroller, and he intentionally had bought like the cheapest, least capable version that he could find, like this thing, I think like 32 bytes of RAM, something ridiculous, he couldn’t do anything. But what you find is the really experienced architects, they find a lot of richness in the simple things. He was finding all these nuances, and he was trying CI/CD, he was debugging with a TV, because the thing had no useful output, but he could output patterns, so he could show the patterns on a CRT screen because he could emulate the TV signal. And you might say, oh, isn’t that just geekery, is this actually useful? And the answer is, it is. The experienced architects and engineers, they find so much richness in a relatively simple setup that they can learn a lot from that, that they can use later. The novice engineers, they always chase bigger and bigger and more complicated things, like, oh, we need microservices, we need this and we need that, and in the end they don’t really understand what they built. So my advice there is, build something small, build something that looks simple, but understand the ways in which it’s actually not simple, find the odd behaviors, find the dynamics, see how this thing behaves, and that is your real learning, because then as you move up to more complex systems, you will understand how it works, or sometimes it doesn’t work, and then you actually have a chance to go fix it, versus you use the magic, and if the magic show sort of ends, there’s nothing you can do.

That’s back to the constraints, where we’re using constraints to really understand the space that we’re operating in, existing in, and pushing the boundaries as you said earlier, so that we truly understand what’s possible, what’s not possible, and that really helps us truly, I think, get to the details, the technical details of what’s happening, and it teaches you what the boundaries are.

Sort of the image that comes in my mind, sometimes we see very simple animals and they just run and then they hit the glass pane, because they can’t see the glass, and they keep doing this, and you’re like, oh, that thing isn’t very smart. Well, if I look around the development area, I see a lot of the same thing, people like, oh, I have this new toy, it’s amazing, I’m going to run with it, and it looks really good until you hit the glass pane, and you’re like, oh, ouch, that didn’t work, oh, what was that, let me try again, oh, ouch. And that just doesn’t go anywhere. So what the style of development and architecting I described, that you understand the mechanics, the nuances, it keeps you from running into the glass. You’re like, ah, here’s the sweet spot of this thing, this works well, but it works a little bit less well, you start to understand what the terrain looks like, versus the novice that’s like, oh, this is like paradise, everything is great, until it, boom, it hits you. Versus me, like, ah, it’s a little steep over here, maybe I stay off, maybe over there is okay.

And being able, like you said, the constraints, the trade-offs, basically knowing your terrain, understanding your terrain, and being able to maneuver on the terrain, I think that totally changes the way you build systems.

Yeah, that sounds like a really healthy state of existence, or relationship with the world around you, and especially in technology that can be pretty expensive. The digital landscape can be pretty vast and endless, and so really working hard to make sense of it is important. So how do you stay so curious and interested in what you do all the time? You’ve been doing this a while now, what are you working on that keeps you so interested and so curious?

So there’s two dangers in the architect elevator. The one danger is people who never go down, and they just become sort of buzzword folks, I wouldn’t call them architects, so that is not useful. But then the other danger is exactly the opposite, once you go down into the engine room you will find so many exciting things, like you never want to leave, because, like you said, the tech universe is, a, very exciting, and, b, it’s very big, and, c, it’s probably also reasonably complex. So all the good ingredients for a person to get lost. So what I tend to do is, as I said, I pick what looks like a simple problem and build it from scratch. So I try to stay close to the metal, really build things up. So when, for example, when I built the loan broker, at the very beginning it didn’t have automation, I did anything and everything on the command line, just very simple, and then I ratcheted my way up, saying, okay, well, that obviously isn’t a good way of doing it, so I’d write CloudFormation, I tried SAM, and then I tried CDK, like, oh, that’s the thing I want. So I made small baby steps. The other thing that probably helped me, I had a specific goal in mind. It wasn’t like, well, I need to learn serverless stuff, that would have led to two results, either it’s really boring and you don’t learn anything, or it’s really exciting and you never come back. So my goal wasn’t to learn serverless, my goal was, well, implement this loan broker example from Enterprise Integration Patterns, build that in a modern architecture. So I had my goal, my scope, and what I wanted out from my engine room, what I wanted to bring back, I had that defined. Now, did it still get like five times bigger than I thought when I went in? Of course it did. I blogged on my Enterprise Integration Patterns side, and it’s like five massive blog posts, and I didn’t even write down all the pieces, and I think in the end it’s like 20,000 words, it could have been like a small book almost. So you got to understand there’s a multiplier factor, but you have a certain scope and have something you want to bring back, like a learning for you as an architect, and I think that puts a helpful boundary box around your engine room adventures.

I like that. So are we going to get an updated version with all your recent experiments at some point in the future?

So you mean the updated version for the book. So there’s a couple of things there. The website, this enterpriseintegrationpatterns.com, actually what I did is I published modern examples. I started this a couple of years ago, and whether this is Kafka, whether this is the different cloud providers, I just said, hey, there’s new implementation examples. So what we found is that the patterns themselves and the pattern language hold very well. I always said, like, some of the design guidance and overall considerations is expressing intent, like a message filter, a content filter, an aggregator, all these things still exist and they’re still useful. So the only thing we would need to swap out is the mapping to the implementations. So what we’re doing there right now is, rather than update the book, just have this online, it’s so much easier, so people can buy the book and get the design intent, the trade-offs, and then if you say, hey, here’s what this looks like in serverless, you just get that from the website.

What do you do when you’re not at the computer and you’re not being an architect? What keeps you interested and curious?

So I think you’ve obviously got to balance it, the left and the right brain. So when I’m not architecting, I’m usually out and about. I used to do a lot of winter sports, but I live in Asia, so that required international travel, so that’s taken a little bit of a hit. But yeah, I like the outdoors, it gets fresh oxygen in your brain, and I think when you do that, the left side of the brain doesn’t have to shut off completely, it can run in the background. And this is where I come up with ideas, like the famous article I once wrote is, like, Why Starbucks Doesn’t Use Two-Phase Commit. Those are the things, if you detach from your tech stuff and you go off in the real world, but the tech stuff runs a tiny bit in the background thread, you end up with stuff like thinking about two-phase commits when you’re at Starbucks. But most of my hobbies are actually to move the body, get outdoors, cycle, snowboard, I think that gives you a good balance.

Yeah, agreed, and it’s essential in these COVID times, I think, to have a nice balance. And like you said, if you have lots of things to think about when you’re out there, you come back with a fresh view and hopefully an elevated perspective of what’s happening.

Correct, that’s what keeps me going as well. So I said in the beginning, I have a little bit of a double life of multiple jobs, and that really relates to seeing things from different levels. So there’s the Enterprise Integration Patterns of building serverless stuff, very engine room, but then I run the other side, architectelevator.com, and this is more about seeing things from a higher level, this is more like the penthouse, the boardroom kind of perspective and the architecture. So what also helps me is having more than one tech persona, if you wish. So sometimes I’m deep in the engine room, sometimes I’m still left-brain tech stuff, but the architecture thinking is already, it’s not so totally left-brain actually, it’s like thinking about how to explain things better, like you said, how do you explain this to your company leadership, so there’s a lot of right-brain actually that comes in. So I think having a tech persona at different floors of the elevator can also help you to not get stuck in this one little thing.

So when you’re at the higher levels, would you say more of your work is having to make it speak to business, make the technical speak to business, or is it more simplifying and distilling down the tech into simpler terms that leadership’s going to understand?

So I think it’s really the combination of both. So the biggest problem I see is that when we speak to the business users or higher-level executives, we believe we’re speaking to children, we want to make everything super simple. I’m like, hold on, you’re speaking to the most successful and accomplished people in your organization, so why do you assume they can’t understand anything? And then people say, oh, but they’re not technical, and I’m like, well, define technical. And then, oh, they don’t know what a Helm chart is, well, that’s not technical, that’s an arcane piece of technology stuff, they don’t need to know. But they are very capable of understanding things that I call technical, like trade-offs, all the stuff we talked about, architectures, topologies, trade-offs, constraints, they understand really well, because that’s how they run a business, it’s the same thing. So the key here is to build on the kind of thought patterns that executives tend to do very well. They know how to make decisions, they know trade-offs, they know risk, they know implications, they know boundaries, they know constraints, otherwise you wouldn’t have a company, and then translate your engine room things into those mental models. So it’s simplifying. Simplifying is a tricky word, because in this case, yes, it is simpler, but it’s not dummified. It’s not, oh, we should all be agile, oh, we should all be on microservices, that’s what I call dummifying, that’s not simplifying. Simplifying is finding the right level of abstraction. Like the trade-offs we talked about, microservices and serverless is fantastic but you’re shifting more things to the runtime, so you need better runtime tooling. Those, I guarantee you, every business executive, every high-level executive, will be able to understand, maybe with two or three more sentences than I just did now, but these are trade-offs and balances you can convey. So the key is finding higher levels of abstraction, and they look simple at the surface but they’re not simplistic, they still allow you to reason about what you’re dealing with. And I think that’s the magic of the elevator. It’s not about telling the people here one thing and telling the people here another thing, that doesn’t work, that’s what I would call lying if you want to be blunt, that’s not riding the elevator. Riding the elevator is, you’re taking what is here and forming useful abstractions that allow people to understand the essence of the decisions we’re making here without having to know all the product details, the language details underneath. And that’s what the elevator is about. And every time I do this people are super happy, because they go, finally I can understand this stuff, no jargon but actual thinking. You don’t want to be hit with jargon, like random stuff, you want to be hit with a mental model, and then we can together think in this mental model, they’re like, ah, so should we move to this thing gradually, is the serverless or microservices better for some part of our estate, should we use it everywhere, how do we get this into the organization, is this good for certain markets, maybe we have volatile markets, is that good there? Suddenly you make them part of the thought process, and that’s when you won. When you have your executives thinking through the problem with you, that means the elevator has really worked.

Yeah, I think I need to, I’ve been spending a lot of time at enterpriseintegrationpatterns.com, and I feel like I need to spend equal time or more at this point right now at architectelevator.com, because, I would say this is where I’m at, at least for me, I’m really trying to learn to speak that language and translate it, but I want to be riding the elevator back up and down on a regular basis, but maybe for a different set of reasons than an architect would. But I think the architect elevator is a good place for me to start, so thank you for that, I appreciate it.

Yeah, I hope you enjoyed it. It’s a strong metaphor, and people really latch onto it, especially, as I said, it’s about making the connection and going back up and down on occasion. So I think the metaphor is becoming quite sticky with software engineers, who tend to be frustrated, like, oh, the managers don’t understand what I do, or they don’t want to support my project, or whatever complaint they might have, people really start to see, okay, at least half of the equation is on me, and I need to understand how to bring things to the other levels, and they find the architect elevator extremely helpful.

Yes, definitely. Well, I appreciate your time today, Gregor, this has been really enlightening. I definitely want to find some time, I’d like to have you back and talk about a whole session on just the architect elevator, because I think there’s a whole world there that developers and folks need to tune into and at least think about. So I appreciate your time today, thanks for joining me.

Yeah, thank you for the great chat.

Thanks again to Gregor for stopping by. For more on Gregor Hohpe you can visit enterpriseintegrationpatterns.com or architectelevator.com. You can subscribe to the Breaking Changes podcast at postman.com/events/breaking-changes. I’m your host Kin Lane, and until next time, cheers.