Maxime Beugnet, MongoDB

Thank you for tuning in to today’s TL;DR episode of the Breaking Changes podcast. I’m your host and chief evangelist for Postman, Kin Lane. With Breaking Changes we explore specific topics from the world of APIs, but looking at it through the lens of business and engineering leadership. Joining me today we have Maxime Beugnet, senior developer advocate at MongoDB. Maxime shared his journey with me becoming a developer advocate with MongoDB and what the modern database stack looks like behind the web and mobile applications that we all depend on. Let’s start with the basics: who are you and what do you do?

Hey, so first of all, thanks for having me on this podcast, I’m really honored to be here and to be noticed, and to be on the Postman podcast. So I’m Maxime Beugnet, I’m a developer advocate based in Paris at MongoDB. So I work directly for MongoDB, the database company, and I’m a DevRel, so I build a lot of content for the community, I take care of the community, I try to solve their problems, you know, the community forum for example. I publish blog posts, I occasionally participate in podcasts, or I do YouTube content or live content as well on Twitch. And also my main role is doing conferences, so I send a lot of CFPs to a bunch of conferences, and then of course I go, they are usually in person if I can, or with COVID sometimes it’s online, and I try to deliver those afterwards.

So what got you into DevRel?

That’s a good question. It was kind of an accident, I guess. Initially I was a consultant, I was working for Zenika, which is a consulting company based in Paris, so working for clients, etc., the usual consulting job. And this company also happens to be a training center, so that’s where I actually started to become a trainer, and upgraded my skills to talk in public, present things, explain, build up some pedagogy skills, etc. And I learned about Java, about MongoDB, I became an official trainer, and I got interested as well in the meetup in Paris, and I took the lead actually of the MongoDB meetup in Paris. I felt like there just wasn’t enough, so I became a leader, and then I was running the meetup for a couple of years, and that’s how I met my boss, basically, Drew DiPalma. And he just came to me, he was like, well, you do that for free, and the MongoDB meetup in Paris is like one of the most successful ones in the world right now, and you’re not even an employee of us, you just do that for free, that’s amazing. So that’s how we started to talk, and it was like, you know, we could pay for that. I was like, really? And that’s how I came in.

I always want to try to share with other folks how we got into the industry, because developer relations is a really growing area, and I always try to encourage people to join and participate. So I think taking leadership in your local meetup is a good way to do it, because meetups are kind of, well, before COVID and hopefully again after COVID, are kind of central to how we do what we do, and connect and networking and help each other, as well as all the other things you talked about, podcasts and blog posts and other things. So when you do your work, we as developer relations tend to focus around a handful of technologies, tools, platforms, what do you focus on when you’re doing developer relations? MongoDB, but is there a special part or database aspect that you focus on?

So it depends, because the team expanded. If you ask me the same question four years ago, it would be a different answer, because it was just me and my boss basically covering all Europe at the time. Now in Europe we are a team of 10 people, and probably double of that in the US, so about 30 total, I guess, in the team. So we developed some specific areas. For example, one of my colleagues, Mike Lynn, is working exclusively, almost, on the MongoDB podcast. For me, I’m more doing conferences, I’m doing content on the blog, I’m doing Twitch live for example, which is not everybody doing that in the team. So we develop specific areas where we want to focus more, and it goes by affinity, it’s not imposed on anybody, but Michael just likes doing podcasts, he’s like a sound geek, I would say, in the team, so he’s all about this. So yeah, I would say that’s where naturally he wanted to work more.

And for the MongoDB side, what we talk about also depends on what we like. For example, I’m more of a backend person, so usually I’m more working around the different languages like Java and Python, for my part. Some other people like focusing on C# or Node.js and other languages that can connect easily with MongoDB. And recently I was also on the team called the Realm team, which is a MongoDB backend service, the serverless backend service that MongoDB offers on top of MongoDB Atlas, and so I was focusing mostly on that area. But now I just changed team, actually, so I’m covering more the whole MongoDB platform, so I can blog or talk or present anything that’s related to MongoDB.

So I’m an old database person, I’ve been doing databases since the 1980s, for a long time, and I’m big on relational databases, and then I learned a lot about the NoSQL world. What’s the current state of databases, and what does it power? I know web and mobile, but what are the specific features or capabilities that support single page applications, mobile devices, what are the benefits beyond just data storage these days?

Well, the main benefit of MongoDB is the fact that we don’t store data like SQL, in a tabular way, and the schema design is very different. Usually when you move from SQL to MongoDB, you have to rethink the whole schema design, and you have to redesign completely the way you store your data. Usually when you do a single page application, it means that behind that you have one query at least that’s going to run to the database, fetch some data, and then build the page. And usually in SQL it means that it’s not one query, but it’s actually a bunch of queries that are going to do some selecting on a few tables, or it’s a giant query that’s going to rely on select and then join, join, join, join, because you have to actually travel through multiple tables to fetch all the data you need.

But in MongoDB we try to change the perspective and store all the data that lives together, that needs to be presented together, in just one single collection. So we try to store basically objects that make sense for you as a developer or for the user as a person. For example, if you manipulate persons and contracts and these kinds of things, you’re going to find exactly the same object, mapped, I’m going to talk in Java for you, from your POJOs, or the equivalent mapping layer on your application. So those objects that you manipulate in your code are going to be mapped one to one almost in the MongoDB database in the end. In SQL you would have usually to chop down all those tables or pieces of objects into tables, because you can’t store arrays for example, and you can’t store multiple addresses, so you need to do a join table, etc. So all that layer of complexity, usually you can remove that in MongoDB and store everything together, and basically the data that needs to be accessed together needs to be in one single collection, one single document. That’s usually how we try to build that, and because you do that, then it’s more efficient query-wise, or for the indexes, etc., so it makes the whole thing more performant and more efficient.

So it’s really tailored for that, the application, how I’m trying to deliver this information and use the information in the application, rather than just a massive database for database’s sake.

Yes, and it’s not only that. If we go back to the roots of why NoSQL exists, why they started to exist initially, they tried to offer a different solution that SQL could not fulfill. So basically the main problem of SQL databases in general is that they don’t scale well. They were designed in the 70s or 80s, now like 40 years ago, and at the time internet was barely a thing, so they had no idea that they would serve those massive systems that we have today, like Netflix, LinkedIn, Facebook, whatever. This was not even a dream at the time, so they were not designed with that perspective in mind. While MongoDB is now like 12 years old, I think, maybe 13 now, so this was created with that in perspective, the web 2.0 or 3.0, whatever you want to call it, this was designed with this in mind. So for example, when they created NoSQL, they dropped everything that was preventing SQL databases from scaling, like joins or ACID transactions, all those things were dropped in favor of scaling.

So MongoDB was built first with scaling in mind, with the replicas, the high availability, the sharding, so you can distribute your workload on multiple shards and multiple servers. So that’s called horizontal scaling, versus vertical scaling that you usually do in the SQL world. And once the technology was proven to work, and once we had the scaling basically in place and working correctly, after a few years we started to redevelop those features like joins, which in MongoDB is dollar lookup now, or we started to develop the ACID transactions, but now they exist in a distributed system, which is very different. So those features were added again, but later in time, they came as a secondhand feature, if you see what I mean, and it was only possible, I think, in this direction. That’s why today databases like Oracle, MySQL, etc., they have issues to scale because they were not designed that way in the beginning, they still have to develop scaling now, but with those pre-existing problems of ACID transactions and joins and all those pre-existing features that are blocking them from really scaling correctly.

Yeah, that’s interesting. The web scale is definitely a different beast, and I can support the fact that vertical scaling of a massive database is not easy, and it’s really difficult to do and support a wide variety of use cases that the web introduced, and that’s one of the reasons I switched to doing APIs, is because I got tired of doing that database work and I enjoy APIs a lot more. So what else, you have this progression of capabilities that are designed to support not just web scale but the needs of web and mobile applications, where you have latency over networks, you have different levels of quality of networks, you have multiple devices. What else does MongoDB enable when it comes to that type of world that we operate in now?

Yeah, so basically what you get for free, once you enable the scaling and the sharding, you get the possibility to distribute your database across the world. So we work with AWS, GCP, and Azure on MongoDB Atlas, so you can create those clusters with a few clicks, you can just select the regions where you want to deploy your database, and so you can distribute freely your database across the globe to reduce the latency or solve the problem of data governance, for example, with GDPR. And the, I don’t remember it, the Patriot Act or something, the CCPA in California is what’s driving it. So okay, I completely forgot the name, but GDPR-equivalent things, so you need to make sure that the data of some particular countries stays in certain countries, so you have to deploy a particular cluster in that place, or shard at least, and make sure that that data stays in that country. So that’s basically what MongoDB offers. I forgot the backbone of the question, but you get a lot of features like this from MongoDB.

And that’s just for the scaling part and the distribution part, but MongoDB is a fully featured database, so anything you can do with SQL today you can do with MongoDB as well. Like we have the ACID transactions, like I mentioned, recently we added some special time series collections, for example, to have a better answer for people with time series workloads, we have a lot of features, I don’t know where to start, but we have the aggregation pipeline, for example, that makes it easy to transform and reshape the data and serve it correctly to your users.

And that’s just talking about MongoDB here, which is the core database system that we have. But MongoDB Atlas is a fully fledged data platform which enhances the whole experience for the user. So in MongoDB Atlas you have the monitoring, you have all the automation to upgrade or update or create new MongoDB clusters, you have all the monitoring in place, and you have all the automated backup, for example, and you have also some other tools on the side that are actually very practical, that make your life a lot easier. Like for example you have MongoDB Charts, which permits you to build those nice dashboards, nice charts. You have also MongoDB Atlas Data Lake, which is a way to retire your data that is getting old in your main hot MongoDB cluster that is of course more expensive, you can now retire and archive data to other S3 buckets, for example. So it’s a way to save money actually, because if you are working every day with like three terabytes of data and one is completely cold data, you can actually reduce the size of your hot main cluster and put some data in the cold S3 storage, so you can save money, because of course it’s a lot cheaper there.

And on top of that, we have a feature called federated queries, which is built in MongoDB Atlas Data Lake, which actually allows you to send a query to the two data sources that you have now, because you have the hot and the cold data, and you can actually send a federated query that’s going to be resolved on both sides and then merged back at the end. So it can be completely transparent for you whether or not you are querying cold or hot data. And that’s super cool, of course it’s going to be slower if you hit the cold data, but it’s a way to make sure that all the data that you archive you can actually still query and use and find value in that old data that can still be useful for reporting, for building some pages on the side for marketing maybe purposes or whatever.

So on the API front, REST APIs are the most common approach to APIs, but really nowadays on top of databases we’re needing a whole spectrum of, we need GraphQL, we need, you mentioned Kafka, we need streaming. What sort of APIs does this enable these days?

Well, you can build anything on top of MongoDB, but we built something that’s called MongoDB Realm, which was the consequence of the acquisition of the Realm company that we bought like two, maybe three years ago now, and we fully integrated the Realm tech in MongoDB Realm now. So in MongoDB Realm, it has a serverless backend system that’s living on top of MongoDB Atlas. MongoDB Atlas is where you deploy your clusters and where you store your data, so that’s kind of the data layer of your application. MongoDB Realm basically sits on top of MongoDB Atlas, and that’s where you can build your backend system in that serverless environment. So for example, you can host a GraphQL API, which, if you click, you basically just enable it and you get an endpoint, you can also enable the authentication system, and basically out of the box, in like five minutes, you get a GraphQL endpoint that you can use to build and connect your frontend system. And that frontend system you can also host directly in MongoDB, because there is a hosting part, so you can also host the frontend, for example built in React or Vue.js or whatever, you can host it completely in Realm. So you can really host the whole thing in MongoDB if you like. And you can build the REST API as well, we have something called HTTPS endpoints, and you can build REST APIs in there.

That’s actually what I used in one of my recent projects, which we call the Open Data COVID-19 project, which we developed about two years ago, in January 2020 really, when the COVID outbreak started. So we started to develop this project where we wanted to make the data about COVID available to everybody, and so we are actually serving freely a REST and a GraphQL API that anybody can use, and they are actually using it, it’s very successful at the moment. I don’t have the numbers in mind, but I have a few thousand queries per hour running on the REST API for example, which is hosted by Realm, and we are just serving this for free for the community, and they build whatever charts, whatever data they want to provide.

Can you share a little bit more about how, because I know there’s quite a few data sources available for COVID-19, but there’s only a certain amount of good quality data sources, there’s a handful of spots. So can you talk to me about how you aggregated that data and made it available by the database?

Yeah, sure. So the only one almost that existed at the time, in January 2020, was the Johns Hopkins University data set. Initially it was provided in a very unpractical Google spreadsheet, which was not really easy to work with and we could not download it. So initially I developed an HTTP parser to just curl the whole page and then extract the data from the HTML page, it was kind of ridiculous. But then soon enough they developed CSV files which they put into a GitHub repository, which was way easier to work with. And so what I’m doing actually now, I developed this like at least 10 times now, because I change the way I interact with the data they provide, etc. So the way I’m working today with it is, I work directly with the CSV files they provide in GitHub. I download everything every four hours, I have a Python script that runs on the data, they transform all the CSV files into proper MongoDB documents with a proper schema design. Like I said earlier, the schema is very important, so if you want to have efficient queries or a well-designed and well-crafted REST API or GraphQL API on top of that, you need to work on the schema design first. So that’s what I’ve done here, I transformed the CSV files into good MongoDB documents, where dates are represented as either dates, for example, localization for the latitude and longitude that are represented as GeoJSON points, etc.

And once it is formatted correctly, I just use the MongoDB driver of Python, which is called PyMongo, and I just import everything, I do an insert many and just insert the whole thing in MongoDB. It represents about one gigabyte of data total, because I have measurements for all the regions in the world for each day, and regions in the vague term, because sometimes it’s, in the US for example, it’s a county, not the country, the county, so it’s fairly small, and in other countries, like for example Belgium, I just have one point for the whole country. So it really depends which country we are talking about, so the region term is not the same in each country, but that’s basically all the data that I get, and I get one point for every day, and when you multiply this by now more than two years, that’s what you get.

So when you process that new data, is it important and does it switch it out with what’s there already? Is that alive, or is there some sort of switch that has to happen there, how does that work?

That’s a good question, it’s actually a trick that I used, I would say a smart trick. So the problem first, the problem is that Johns Hopkins University, they are also applying updates sometimes backward in time, so it’s not a good idea to just take the last day every day and import the last day every day and increment, I could do that, but then I would have some mistakes in the past, because sometimes they apply updates in the past, because I just understand after 15 days that actually those 500 people in that hospital, they were actually not dying from COVID, maybe they were just sick of something else, and so sometimes they apply this modification backwards. So just to be safe and just to make sure that the whole data set that I’m importing is consistent and correct, I take the whole data set and reimport it entirely every four hours. I import that in temp collections, so it’s just basically the same name of the collection with underscore temp at the end, and when the whole import is finished, the last line I have in my Python script is actually a rename collection trick. And when you rename a collection in MongoDB, it’s actually an atomic operation, so I can basically do like Indiana Jones in the lost temple movie, or whatever movie it is, I just switch the statue instantly and hope not to trigger the trap. But that’s exactly what’s happening in production, I just replace, in one atomic operation, the collections that I have. And so from the API user standpoint or the GraphQL API user standpoint, it’s completely transparent for them, and they just don’t see the difference, as long as I serve the same data almost, that’s completely the same. What they should notice every day is that there is a new day that’s added every single day overnight for the previous day, of course, but they might also notice some updates in the past because they applied some differences.

And something we forgot to mention, but I’m just going back to my mind, you asked about the data source quality and where it comes from, and why I chose Johns Hopkins University. Basically if you go to that repository from Johns Hopkins University and check all the data sources that they aggregate and filter and clean for me before I use the CSV files, the list is actually super long. Basically they use the healthcare services of all the countries or regions to import all that data daily, and they sort, clean, aggregate, do all the work that’s required to provide those CSV files. So there is just no way for me on my own, because there is a whole squad working on this, there’s no way for me to do better than that. So the data I’m serving is as good as it is. I know there are mistakes in the data set, I know it’s not 100 percent accurate, and when you draw the charts, sometimes you can notice some silly numbers here and there, but that’s as good as it is. I could fix the data myself and be like, yeah, this point is clearly wrong, like there is plus 1,000 cases one day and then it just disappears the next day, there is something wrong obviously. But I don’t want to do that, because if I start doing this, I start altering and taking the responsibility of what I actually serve, while right now it’s as bad as it is, and that’s the data that every journal, every newspaper is actually using today to follow the COVID numbers, etc. Like I think if you check the real source that they are tracking today, I think it all goes down in some direction, it finishes on the same point, which is, I think, Johns Hopkins University.

Yeah, well, I really appreciate your time today, it’s been fun learning from you about MongoDB and the realm of the database, I appreciate it, thanks for your time.

Yeah, I was super happy to be here, thank you very much for listening, and thanks for having me, that’s it, and I’m super happy to be here, thanks a lot.

Thanks again to Maxime for stopping by. You can find more about MongoDB at mongodb.com, and Maxime is on LinkedIn. You can subscribe to the Breaking Changes podcast on postman.com/events/breaking-changes. I’m your host Kin Lane, and until next time, cheers.