Vijay Challa, Boys Scouts of America

Thank you for tuning in to today’s episode of the Breaking Changes podcast. I’m your host and Chief Evangelist for Postman, Kin Lane. With Breaking Changes we explore topics from the world of APIs, but through the lens of business and engineering leadership. Joining me today we have Vijay Challa, the CTO and CIO of the Boy Scouts of America. I was pleasantly surprised to learn that the Boy Scouts of America were API-first, and I’m really happy to have them in my toolbox to be able to showcase when I’m telling stories about how you can make a social impact with APIs and really talk about how they are doing more with less by being API-first. Well, let’s do this, let’s start with the basics. Who are you and what do you do?

I’m Vijay Challa, I’m the CIO of Boy Scouts. I call myself the CIO, CTO, Chief Problem Solver, the person who makes things work here in technology, including repairing Wi-Fi connections or whatever you put in. So we do it all here within the IT department in the Boy Scouts. I manage a very capable team of people that build great digital experiences for our employees, for our volunteers, for our Scouts, and high adventure bases. It’s a complex organization, but we think we have it.

So I was pleasantly surprised when I got introduced to you and started the conversation. I did not expect, but would you consider yourself to be API-first in what you do when it comes to technology?

Anything and everything. That’s one of our missions here within the IT department. I came in about five years back, and that’s when we started our digital transformation journey, and one of the core tenets that we laid out was, being a non-profit and always being short on resources, we were not going to build something that was going to consume the organization a whole lot of money. And at the same time, being right there in front of kids and volunteers and parents whose demands with technology are growing on a daily basis, including my own, we had to be there, we had to be there at a low cost, we had to make sure as an organization we drove a lot of reuse in whatever we built out. So APIs were right there for us. The API-centric strategy was all about reuse, all about simplification, all about standardization. So we set those as our core goals within the blueprint that we laid out, which is, hey, simple interfaces, standard interfaces, a whole lot of reuse. So that’s why our APIs sort of fitted, and then we went forward with it. So it’s central to our journey.

I’m impressed, I’m impressed. So walk me through the types of resources that you maintain as part of this whole stack.

Okay, so we are a non-profit, we support about a couple of million kids and a million volunteers, so it’s a complex non-profit, and the organization itself is a company of companies. So we are split out into 250 councils that are spread out across the country, and each council manages their group of kids and volunteers, Scouts and volunteers. So as a part of this, every year we manage and maintain memberships for these Scouts, the units, the charters that they’re a part of. We have every unit charter pretty much every year. This is sort of an agreement between the organization that’s hosting the unit and the leaders within the unit, that basically lets them meet at that location, ensures the right leadership is in place, and ensures the leaders have the youth protection credentials that would take them forward, those sorts of things. We do that on a yearly basis, so registration, rechartering, or renewals. Fundraising is a big thing. We do a whole lot of events, like camping events, those are there. And then within the Boy Scouts, we’ve sort of gamified the journey of a Scout. This whole gamification process started even before gamification was a term, I think. So we have this whole notion of the kid advancing between ranks, a Cub advancing between, you know, a Lion, a Tiger, a Wolf, we’ve got this whole ranking system through which the kids advance and eventually get to something called an Eagle Scout, which is about one in every 100 kids. So we have that journey for them laid out, all of that is gamified, and all of that is again a chunk of APIs that we handle on our domain, we call that the advancement. So membership, the recharter renewal, the fundraising, event management, the advancement, and as a part of all this then you get into these horizontal layers like payments, which again are pretty central to pretty much everything you do. And then people, because again they’re central to pretty much everything, and then we have organizations that we manage. So there’s quite a few domains that we’ve carved out, but they’re laid out across four main business functions, which is registrations, renewals, advancements, fundraising, and events.

Impressive. The domain-driven design that’s evident there as far as how you organize things into domains and your resources like that. So how do you maintain quality across this, so that all of these APIs are as reliable as they need to be across all your consumers?

So we do a few different things. Again, we are a cloud-first organization, so we have continuous integration and continuous deployment all through the process. We have our API test suite, some that run in Postman, some that run in Newman, we have automation that’s triggered and linked back into our continuous deployment and integration mechanisms, wherein the APIs themselves, we have two tiers of APIs, we have core APIs which are data-centric and domain-centric, and then we have our experience APIs which are product-centric. So whenever we make these API changes, we have these test suites that kick off as part of our build cycle that give us information on where we stand with the quality of the APIs themselves, that’s the API tiers. But then we have our products themselves, and we have Selenium record-replay suites that we’ve written, some where we couldn’t write the record-replay, we had Node.js as a technology stack that went and scraped content, did a few things, and tested this stuff. So testing is pretty central. We also have bots that run in our production systems that go on a five-minute cycle and kick off these scripts that test whether our APIs are functioning good, our systems are functioning okay. We use Uptime monitoring that monitors our websites constantly on workflows we’ve recorded, just to ensure things are good and working. And when things don’t work well, we have our PagerDuty notifications that come in through Slack. So we have Slack integration tied back into PagerDuty, and depending on where the cycle is, we have our DevOps team looking at it and escalating further as it goes. So we try to make sure it’s tied end to end together. APIs are again central, we truly believe, we have these APIs, that’s where most of our change is occurring, so we just make sure those layers are pretty robustly tested. On the front ends we also use Sentry, Sentry.io, and Sentry gives us, I mean, we’ve talked to the team, they’ll tell you they cannot live without a tool like Sentry. And that gives us information beyond the APIs themselves, it gives us information on how our users are seeing it, and if they’ve suddenly experienced errors and problems, and whenever we see our Sentry spikes we go right there and then do problem analysis. Sometimes, irrespective of how many strategies you have in place, it’s really difficult to have like a hundred percent coverage on anything and everything you change, so you’ve got to have your compensatory mechanisms that go look at your different viewpoints. One is a customer-centric viewpoint, another is a product-centric viewpoint that we drive through Uptime monitoring, the third viewpoint is the API-centric viewpoint, which is us formaling our API stacks with test requests to see how and where things stand. So we try to cover it in a few different ways.

Yeah, that’s pretty robust, I would say, a look at how to do quality and make sure that the overall experience is desirable. So I’m guessing security is probably top of mind here too as well. What’s your view of security look like?

Again, security is pretty top of mind. As an organization we’ve had to go through, sometimes security is in your control, sometimes it may not be, I mean, if you use third-party products and they have breaches, you’re still responsible for it. So for us, all our API products are front-ended with a WAF. We use Alert Logic as our web access firewall, and for us, the way we’ve designed things, everything is a denial until it’s whitelisted. So anytime we build an API, it has to be whitelisted before it gets accessed, that’s right there in the center protecting us from SQL injections, from DDoS attacks, and a whole lot of different things. The other kinds of security we do, because we support, 98 percent of our APIs are all secure APIs. So unless you have a session token and a JWT token that establishes your roles, you’re not able to really access it. And the tokens themselves have an expiry, the sessions expire for 30 minutes of inactivity, the JWT token expires every eight hours. So we protect our APIs, and we have role-based protection, organization role-based protection, which is again on top of whatever we do. So we have things like role types that define what roles a person has within an organization, and depending on how and who’s accessing the API, we give fine-grain control to that entity’s profile. For us within the Boy Scouts, security and youth protection are pretty central, so we don’t let leaders who do not currently have access to youth from other rosters get connected. So we have all notions where we protect PII, that is very core and central, and like most organizations we also go through our PCI audits, we have our SSAE 16 audits. So quite a few different things we do here.

Okay, yeah, so I’m assuming because you have a handle on your domain landscape, the design of your APIs, that really helps when it comes to PII and other audits, that you know where your PII is, you don’t have any doubts about that.

Yep, we know where it is, we know who’s accessing it, and whether they should access it or not, you’re right.

Yeah, that’s so important. That’s the biggest challenge that a lot of large enterprises face, is they just don’t know where their APIs are, they don’t know where that PII is going to exist. And so that kind of discovery and known landscape, but also having observability and reporting, so I’m guessing you’ve got a pretty robust observability over all of this operation as well.

Pretty good. Like I said, Sentry.io is kind of our observability platform for what’s happening out there and what our users are seeing. The end-to-end observability for us, we integrate into New Relic today. From a telemetry standpoint, I think we’ve got a whole lot of metrics that are going into New Relic, and whenever we get into problem diagnosis or management, it’s right there. And what we’ve done about a year and a half to two years back was, once we were mature enough on our transformation journey, which we are right now, we got into trying, our motive as an organization was to detect problems before they were reported. So that’s what we wanted to get into, and Sentry was a step in that same direction. What we also did with observability, it’s a great platform, we’ve used it over the last year and a half to see how our error trends are, across the core API platform, which is our data platform, as well as our experience API platform. So now we have weekly review sessions where we look at the top five errors that are affecting either of the platforms. We also have review sessions where we look at how we did in comparison to last week, so if our error rate went up, went down. I think as groups mature, as technology matures, and as we make these biggest strides around maturing as a platform, the focus needs to be on observability and how the end user experiences. So we’ve tuned our processes now, and I’m very, very happy to see the progress the teams have made, and we get to see everything from increased errors, the top five errors, to increased latency, and we qualify errors into 400s and 500s, and we talk through it as a group. So it’s paid dividends to us.

Yeah, that reflects what I’m seeing across other enterprise organizations who are further along in their API journey. The way you use collections, you can have collections and your tests running in the pipelines, but many are also running those on a schedule and then piping into Datadog or New Relic, those are the two top areas. And so that observability over your contract testing or integration or performance testing, we’re seeing more security testing, but then now I’m also seeing people, because a collection you can run can connect to any API, so I’m seeing people connect to their gateway and pull data and then pipe that into New Relic to augment it, so more operational-level observability of the infrastructure behind the APIs. Having that observability view within your Datadog or New Relic or other APM solution is just super critical right now, because our infrastructure is just way too large, we can’t see it all, you have to have those kinds of approaches.

Impressive, yeah, totally agree. I think especially when you get to a distributed sort of a world, which we are all in right now, observability becomes key in maintaining stability, even more so now than in the past.

Yeah, and that relationship quality. A lot of folks focus on testing quality, but that observability so that you can actually see the health and the state of the overall system, not just individual or even by domain. So who are your consumers? I’m assuming you’ve got mostly internal for your own applications and needs, but I’m guessing you’ve got external consumers as well as probably third-party APIs you depend on as well.

Yes. So when we talk about internal, we have employees that are our consumers, but then we have our Scouts, parents, volunteers, that they’re like internal-external for us, but then we have our leaders and councils also. So where we’ve had the external parties come in and chip in, sometimes there are capabilities a volunteer wants to add, and once we vet the volunteer and know them from a security profile standpoint to have confidence in their abilities, now we’ve had instances where somebody was building a website for managing Merit Badge camps. So we work with the volunteers, and then the important APIs around validating our youth protection status, or validating somebody’s membership status, things like that, are APIs that we open out to the volunteers, where they’re able to link into important aspects that’ll help me and the organization. If they’re going to build a product and they could link into these important aspects, at least it will give us, the organization, confidence that they’re bringing in the right people within their products. So those sorts of integrations we’ve done. We’ve linked back into other programs within the Boy Scouts, like the Order of the Arrow program, where we have again membership verification APIs and youth protection training status APIs that we’ve linked back and forth. What we also do is sometimes pick up, where we are not very good at, or where we think as an organization we would have to spend a lot of money, we look at the commercial off-the-shelf products, COTS products, where in some cases we’ve had to pick up, let’s say a product like NewBook that does event integration, where they’re very good at shopping experiences for, let’s say, our campsites. But for us to build a shopping experience that would be world class, I think it’d take a lot of resources, and that’s not a best use of our resources here. So we let the COTS product do that shopping experience, but where it links back into the BSA, that could be the volunteer check, or the training check, or the membership check, we link back in. So sometimes the integrations are bi-directional. We have the event management registration system on our side while we have the shopping experience outside, so where it gets into bi-directional integration, which is, hey, the person shopped for this and needs to pick an event, and they’ve purchased this, if we need a notification mechanism back into us, we have mechanisms where those systems are able to tap into. In some other cases we’ve integrated back into payment platforms, like Square or PayPal, again there are API integrations back into those systems. Internally between applications also we have pretty robust API integrations, going back to the payment platform that supports Square, PayPal, VPay, Chase, and anything else that might come in the future, we have a webhook mechanism through which all applications consuming payments know about settlements as and when they happen. So a whole lot of integration back and forth. We have internal customers, internal-external, we have a product from the outside, commercial off the shelf, we have integration from the inside, app to app. So yeah, that’s what we consider.

I mean, you guys have an API platform, it’s not just about building APIs, you’re consumers of other external APIs in that relationship and back and forth, and that really is, for us, the forward motion of the API life cycle, and your operations are enabled by that. You’re able to build the services you need, purchase the services you need, and make them all work together in a seamless way that’s easy for everyone involved, and then you’re able to monitor, secure it, and have that control. But the other thing I heard there also, which I’m hearing a lot, here in Europe you have regulation that’s called know your customer, you have to in the payments industry, but in API circles, in developer circles, I’m starting to hear “know your developer,” and it sounds like you’re already ahead of that game, because you have those certification systems, you basically have a know-your-developer program in place.

Yep, some need to be a little more formal than the others. I think we just started with the know-your-developer program, but unless you can trust, so what we do here at the Scouts is, we hedge our bets. If you know your developer, but you always need to start off with something simple and easy and something that you can control fine-grain. So we give them, when we set somebody up, we give them their client ID, client secret separate, so we can cut them off at any point in time if they become an abuser of API. So know your developer, but also control what you don’t know.

Yeah, I spend a lot of time convincing very old enterprise organizations who are earlier on in their digital transformation, and doing API scares the hell out of them because they feel like they’re giving up control. So do you feel like, though, that API-first and the whole combination of your approach to testing, security, all of it, you have even more confidence with doing this publicly and finding this balance, because it is a balance between access and control and change?

I think I’m more comfortable with access and control, because we can cut the access at any point in time, and if you give it the right way, I think you can control it. The change part is what gets to me sometimes, which is, we’ve seen, when we had one external customer versus when you have three or four, then you start seeing how this could impact the people. So we’ve had a recent Mule 3 to Mule 4, I mean, we use MuleSoft for our experience APIs, so we recently had the 3 to 4 conversion. As part of the conversion we probably updated a few APIs because they didn’t have the right interface, or we had to morph and get to the next version. So when we sunset some of our older stacks, when you have these external parties consuming it, that’s where you sort of lose them out. I’ve seen, we’ve had issues where we suddenly moved the organization API to the next version and then the consumer from the outside lost connectivity. So we, as an organization, I think we need to get a little more mature with that practice, but the change part gets to you more than anything else, I feel. Everything else you control, but you can’t get them to change at the right time.

Yeah, that change, that forward motion, it’s not just you as a producer, it’s you with all of these consumers, everybody has to move forward, and like you said, you’re a consumer of other third-party public APIs, so it’s the same relationship back, all of that has to move forward in concert, and that’s not always easy, to find that consensus and getting everyone doing what they need. So yeah, change is important, it’s a top priority. So when can I get an API badge? Will I be able to get an actual badge someday that shows me as a Scout I’m able to hack in and work with APIs?

Maybe one day soon. It’s not, we have a digital badge on the Scouting side, so why not make an API badge? One small story is, when I started off here five years back, the organization was not talking about APIs. Now you come in and API is a common term out here, at least within the IT organization. So we’ve matured a whole lot, and I think the organization also has grown exponentially when it comes to digital and consuming digital. So I think there is a day where API sort of enters the Scouting handbook.

Yeah, no, so talk to me about what that journey looked like. How did you convince everyone? Was it just meeting after meeting and demonstrating what the potential was? How did you move and get everyone aware of why APIs mattered?

Again, it all depends on how you connect to your people and then break it down for them. If you break down things, the whole domain model when you talk through a blueprint and talk to people about it, I don’t think a whole lot of people will have questions around it. When you talk to them about what they handle on a day-to-day basis and then just break it to an entity-slash-domain model, it is pretty obvious what you’re trying to do. Then you get to the conversations around reuse and integration, and showcasing the problems that they deal with today versus what they may not have to deal with tomorrow, which is a monolith, a fully integrated product that you needed 10 people to manage, versus a supremely decoupled, decentralized sort of a product which can be managed by, let’s say, five people. So those sorts of levers are what get the buy-in from people. When we started this journey, we laid out a blueprint, we put these things together, APIs, again, when you talk to people about APIs in the beginning, just talk about reuse, reuse, and reuse, and talk about business interfaces and reuse, and then I think as you work through the journey, API just becomes a common term. So I don’t need to convince people on APIs, I think you just need to convince people on reuse, and we get to APIs pretty easy.

Pretty cool. Yeah, this is consistent with what I’m seeing in our customer base. We have 20 million users, but the growing number of people who are involved in using and working with APIs has gone beyond developers, QA folks, security folks, we have product managers, we have analysts, so this growing sphere of people that are involved in the API lifecycle. And one of the top questions I’m getting from large enterprises who are further along in their journey like you, is they’ve been doing the domain-driven design, they’re contracting OpenAPI Swagger, that kind of thing, but they’re looking for more resources to help train average business people about, what are APIs, why do they matter, get them involved in the domain-driven design process, modeling the objects, the events, the things that are happening, and that really helps lower the vocabulary we use to make it really make sense to the business stakeholders. And then, as you said, you speak in terms of solutions and problems that they face and pain they suffer, then they’re going to be on board, they understand it, they get it, and it’s both sides are better for tech and business.

Yep, totally agree.

Yeah, it’s impressive. I’m really impressed with the journey you’ve been on to make this happen. So what’s big on your roadmap? What’s next, what’s your top concerns or things you want to invest in when it comes to your API journey?

So this year was a pivotal year for us within our digital transformation journey. This year we had a legacy platform that was 22 years old, and the last leg of the legacy platform, which was the back office, was what we sunset in January. So now we are all on the cloud, on a fully API-enabled platform, so we’re all good. But what the legacy left behind for us is this notion of user duplication. We have duplicated users, I mean we have the same user with five different profiles within the data stack right now. For us, our biggest piece for the journey this year is to figure out ways to normalize this to one user having one identity. So centralizing identity. We’ve done single sign-on and all that, but we still struggle with the problem of centralized identity on a person. And I think that’s a problem a lot of organizations are going through, because of how people get into different systems from different parts of business. That’s a problem we’ve had also, because we had this 250 different councils that act as our extra entities that bring in people, and we’ve had systems that were built to keep user identities independent, but now it’s become a big problem. So this year’s biggest mission is to centralize that identity, and it’ll probably be a very exciting one for the group here.

Yeah, that’s another tough layer, like you said, with the balancing between access and control. The identity layer, it’s a pretty critical one in privacy and security, it’s a balance there. Federation, because you all have a bunch of different organizations and groups, but how do you standardize that, and identity and authentication I would say are the front lines, but then that access control is that next tier, and everyone’s struggling with this, and this is why you see more regulation, GDPR, CCPA, kind of speaking to this. So you guys are at this modern place with your infrastructure, you have to figure this out just like everyone else. So what keeps you in this role? What do you enjoy most about doing APIs at this level?

What kept me in this role and what keeps me in this role is the amount of change we’ve been able to bring into the organization, where we had probably been delivering, let’s say, X amount of work, now with how we are stacked, we are almost able to deliver 10 times as much. So we’ve come up with experiences that we built for our leader volunteers in about three to four months. We built something called a Den Leader experience that was done from envisioning to delivery into production, we called it Project Moonshot, in less than four months. Our organization never saw that in the past, and we’ve been able to do that, and that keeps me excited in this journey, the amount of impact we are able to bring back to our volunteers and Scouts, that keeps us engaged.

Yeah, that’s got to feel good. Like you said, you’re a non-profit, you have a mission, you’re focused on social good, to be able to do more with less. Any enterprise organization wants to do more with less, but that’s huge, that’s significant, that’s a real-world impact, congrats.

Thank you.

Well, I’ve really enjoyed this, this has been great. It’s been an interesting journey, like I said, I was just totally not, I’m totally surprised. I don’t know why, I see interesting people, really smart people doing amazing things with APIs in all corners of every business sector in the globe, but this makes me happy because it is you guys, you have an important mission. For me, I didn’t quite make it to Eagle Scout, but I’ve been to a lot of jamborees and I had a lot of badges, and it was a pretty formative part of my youth, getting me outdoors and helping me find balance when I was growing up. And it helped me get into computers too, because some of the people that were part of my Scout troop also were computer nerds, and we kind of met in school and started working together. So I think you guys play an important role, and it’s really impressive to see how robust your platform is, and I appreciate you coming by to share it with us.

Thank you, thank you so much, Kin, and I’m so happy that Scouting had an impact in your life. It’s had a real big impact in my own. For the last five years plus I’ve learned a lot from this organization, as much or even more than I have given to it, so thank you for that.

Yeah, thank you.

Thanks again to Vijay for stopping by. You can find more about Vijay on LinkedIn and Boy Scouts of America at scouting.org. You can subscribe to the Breaking Changes podcast on postman.com/events/breaking-changes. I’m your host Kin Lane, and until next time, cheers.