Software at Scale 60 - Data Platforms with Aravind Suresh

Welcome to Software at Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host, Utsav Shah, and thank you for listening. Joining us today is Arvind Suresh, ex-staff software engineer at Data Infra at Uber. Now you're working on OpenAI stuff.

Again, thank you so much for being on the show. Yeah, thanks for hosting me, Utsav. It's fun to be here. yeah so i'd love to start off with your story like how did you get into all things computer science how did you get into like data infrastructure stuff i just have to learn more sure yeah so um as you already mentioned like i'm currently working in open ai but most of my background and experiences in like real-time infrasystems i graduated out of indian institute of technology madras and then i kind of joined uber right after that i got an offer from uber to join the data platform team.

And this team ended up building the infrastructure that enabled XR by scale analytics at Uber. I spent quite some time, around five years in this team, and I got an opportunity to work on various verticals of the stack, be it batch, real-time, something related to failover. And there was a lot of focus on cost efficiency as well. So I got an opportunity to work on all of these things.

And after this, I joined OpenAI. Amazing. And you joined Uber in 2018. So describe a little bit about where the data stack was, what the company was like at that time.

Sure. Yeah. I joined mid-2018. Uber was part of, was going through a scaling journey at that time, both in terms of launching in new series, as well as, which indirectly means the amount of data that comes into the system, the users, the number of users we have, all of that was going through a lot of growth.

So at that point, Uber had a lot of on-prem footprint. So most of the infra systems are managed in-house. And we had a pretty big infra team to handle all of this. So there was some amount of solid infra tools and footprint there.

More or less, the different parts of the stack, like batch, more or less all of those pieces were already there at that point there was more focus more work on the real-time stuff that kind of happened after that but more or less the bare minimum batch analytics stack was already there so that's how i would put it yeah and if you could explain even higher level like what was the scale of the data being managed or processed at that time sure yeah so i think uh i vaguely remember this number the amount of data went from around grew by a factor of 20 or something in that three four years period from 2019 and when i moved out it was more than an exabyte so the that's kind of the scale so it was starting with like uh it was around few pbs somewhere around 2018 i would say though but then it ended up going to 100 petabytes soon enough okay amazing and this sounds like it was analytics data at rest how about like you know the amount of data being read hourly or daily being written out i'd love to get to know some of that context sure yeah most of these are it's a bit of both these are at rest uh but the amount of data at which the number the growth at which the amount of incoming data was also growing in addition to just you know there were also a bunch of use cases which wanted a higher retention which means you ended up storing more data than what you would traditionally expect so that would also increase your address size but the amount at which your incoming data was growing was also higher so i would i don't remember the exact number maybe five years back but uh maybe right now i would put it somewhere around close to a petabyte or maybe even more on a daily basis that gets ingested of course this includes some kind of replication and stuff but i would put it somewhere over there on a daily basis incredible so a petabyte of data being ingested every day even if the number is like half of that or like a 10th of that that's just enormous i mean what goes into that kind of data like why is there so much data being ingested every day so uber calls the the core demand supply matching called marketplace.

So there is a lot of complex logic that goes into that. One way or the other, you find use cases to log data, you have use cases to process the data on top of it and maybe flush it down to tables and things like that. So all of this kind of forms a critical part of that one petabyte. Roughly 60-70% of that would be raw.

Raw in the sense they are just so one important thing to note is the data infrastructure itself doesn't create any data this data comes from somewhere outside which could be online systems or message buses or something like that and all of those things which are like you know directly ingested from these external systems are called raw right on top of that you have some derived data sets model data sets and whatnot around 60 70 percent is raw which means we do log a lot makes sense and if you just had to estimate how much of this data is actually used ultimately by like an analyst or someone in the stack yeah yeah i think uh more or less more or less the entire so again the answer could vary depending on which year we target because there was times where we found out that some amount of data was like maybe not even queried there were some data sets which we ended up logging more than what we require and there were some efficiency opportunities on top of it but i i think most of those problems have more or less been solved in the recent time.

So more or less what we log today, more or less the entire thing gets used in one form or the other, right? So you might not have the entire raw data sets being used on a daily basis, but one form or the other, the insights from that raw data sets helps you make data-driven decisions on a daily basis. Yeah, that makes sense. So it sounds like there was some kind of cost optimization effort, which would have culled logs that were not used.

And even after that, probably like you're saying 60 to 70% of data off a petabyte a day. And that's just enormous scale. Yes. Right.

So why does Uber need to log so much data? That's the main question, right? If you could tell us some interesting insights or some data-driven decisions that would have been made through this, like either it's like a one-time thing, it's a system based on this data. Sure.

Yeah, that makes sense. Ultimately, every company, when it starts, every product, it would start with just maybe a single Postgres or a NoSQL or a NewSQL, some kind of an online database where you store your core entity. For Uber, the core entities are mainly trebs, users, earner accounts, and things like that, and maybe cities information. But you will soon reach a point where that particular database wouldn't fit your use case.

you'll end up having a single giant Postgres where you won't be able to run analytics queries on and things like that. So the thing is, which is when all of this data stack stuff got started, they help you make a lot of these kinds of questions. Let's say you end up having some kind of questions like, how many concurrent trips are we handling throughout the year, right? The concurrent trips is one of the important metrics for Uber because that kind of determines how many workers, how many CPUs you actually need up and running at this particular point in time so that you can actually serve trips throughout the globe.

So now it will be hard to put such kind of a query. And if you want to analyze maybe the last two years' worth of data, you won't be able to serve with an online database. This is where all the use of data stack comes in. another example is let's say which is the fastest growing city in the world one important thing to note is as with a lot of other companies and products out there uber's traffic is also seasonal which means you will end up getting more traffic during the holidays and things like that which means just to answer these kinds of questions looking for last one month or two months won't be enough you'll have to look maybe last three years the more data you have the more accurate your statistics will be right another example of where this data would help us is identifying like some kind of anomalous user patterns, where let say you were looking for some insight like a single user account do you see some fraudulent or anomalous activity where that particular user account is logged throughout the globe Or do you have some kind of users Let say which city has users where they end up, you know, canceling a lot more than actual trips, right?

These might not be just one-off decisions, even key product decisions. Let's say like you end up looking at last one year worth of data and you kind of plot a heat map of all the all the coordinates in which the trip was taken, you will obviously see some kind of overlap in plots, right? Which is exactly how, I guess, Uber pool was maybe started, right? Because you end up seeing some kind of patterns in data after you crunch all these numbers together.

I'm not sure if that answered your question, but these are one of the key decisions. No, no, it certainly helps. It's like, it helps you understand that a lot of the product can be shaped just by the data that you ingest. Exactly.

And that naturally leads to a follow-up, which is how many services are querying the data stack in production? Are they reading from like the live like OLTP databases or are they actually querying from the data stack? Yeah, so that's a very good question. I think there are different, the right answer is it depends, but I can go into details.

One is anything where you need some kind of correctness and things like that in a where you want some kind of consistent behavior across multiple entities, you would obviously use some kind of an OLTP system, which is where most of trips serving stack will end up using one way or the other but there are some cases where you would be okay with okay with some kind of you know it's okay for the delay to be there for some amount and it also depends on what kind of system you're building let's say you're building some kind of a real-time system you want the data that's being produced let's say within like 10 seconds or 20 seconds you will maybe and if you have a lot of numbers to crunch and you are okay with some kind of an eventual consistency query shape, then you will end up using some kind of an OLAP system.

So an example is, let's say, restaurant manager, where we end up using something called Pinot, which is like an OLAP system. And the idea here is data eventually makes into Pinot through multiple hops. One is your serving stack will maybe push it into a message bus like Kafka, and data from Kafka will end up going into Pinot, where you will end up querying it. So this is one flow where the data stack is used in like the production flow.

There is another flow where, let's say you have the data from Kafka that ends up going into your data like one way or the other, which is like you end up ingesting this data. So Uber adopted the Hadoop stack. So of course there are different companies which adopt more of cloud offerings would have a different choice, but ultimately you will have some kind of a storage system, some kind of a distributed file system to store all of this data and stuff. so data from kafka ends up going into that file system you will maybe create tables out of it and 60 or 70 of that storage will be raw and you'll have some derived tables out of it which people would analyze and and like make decisions going forward right so these are basically two different flows one is the offline flow the other is the online flow the online flow is what i mentioned about pino the offline flow is basically what we spoke to now you can make intelligent decisions by just crunching as much data you have.

So it sounds like it was recommended that for certain non-critical, non-consistent, or like it's okay to have eventual consistency. It's actually better to query the data stack or to use Apache Pinot. Does that sound right? It's a bit different.

The thing is, it's about the amount of scale you actually see, right? So let's say you have use cases where you end up having data that goes through multiple systems or multiple hops. And for use cases that are real-time and that are where you need some kind of analytics that needs to be done, like Pinot is something that's a good fit. But in some cases where you don't need such kind of a freshness or guarantees or something, you'll end up maybe using the offline flow.

The choice between batch and real-time or let's say offline versus online is something that I've learned a lot through the process on asking very simple questions like, do you need the system to be off uh to be real-time can you not do it in a batch way right asking such questions to yourself goes a long way when like deciding which is better and the systems built the right way would obviously be more maintainable over time yeah and that brings me to you know the cost to serve right there's definitely a trade-off i'm guessing the real-time systems are probably significantly more expensive what does that difference look like like if i had to run the same set of queries on the real-time system versus what's running on like the batch system today how much more expensive would it be like 10x or 100x how do i know yeah uh so the thing is you're right that like doing everything that's done in the batch world uh it's not a question around uh so these are like completely different system right so bad systems tend to be optimized more for throughput the amount of data you can process per unit time right uh whereas real-time systems are more or less optimized for latency or response times.

So they tend to make critical decision trade-offs just based on that. So a bad system would end up reading most of this data when it requires into some kind of a compute layer, crunching numbers and things like that. Whereas real-time systems end up co-locating some of your storage with compute and there are only specific type of query shapes you can make along with it. If you are familiar with OLTP systems, think of this, let's say you end up querying a table that has a billion rows without having any indexes on it, right?

Of course, you have the same problem. You cannot crunch billions of rows in real time, right? So most of these OLAP systems end up, you'll need to carefully tune them, carefully tune what kind of indexes are there, carefully tune partitioning so that they work in real time, right? Of course, if you want me to put a number, I would say it's somewhere between 10 to 100x, depending on the type of data you have and the amount of data you get on a daily basis as well.

yeah so it sounds like the cost to serve is one thing but some kinds of queries might just not be possible at all exactly yeah yeah yeah i'm immediately thinking about your sharded data across yeah if you have some kind of sharding key trying to run a query without that sharding key is just not possible in so many systems yeah exactly so if you think about it the other way right like offline systems since they you can basically log anything you want to the event bus and you can store it and stuff, right?

And even your query systems in an offline manner, let's say you use something called Trino or like Spark or something like that, you end up reading more or less the entire data. There are some knobs here and there, let's say at the file format layer, you can skip reading a few blocks, few chunks of data and things like that. But those systems, it's fine to do all of that, right? Because they are optimized for throughput.

You can end up doing those things like Spark is gonna do all of the compute for you in like a distributed fashion across bunch of focus but of course that takes order of minutes to hours right the whereas the the overlap system is the other end of the spectrum where you want everything to run within milliseconds or worst case seconds yeah and i keep hearing a lot of these different like open source systems like wachi spark pino trino i think is the other one you mentioned yeah who who builds and maintains these systems so spark definitely data breaks and that's a little more famous but like who's working on these systems and like how is their development being advanced if you look at most of these systems would have been started by one company or the other uh which had some kind of a large scale that they wanted to solve for and over time a lot of these projects end up getting open source for instance trino started off as presto uh which was originated in facebook started off i think by a bunch of researchers but it ended up being open source and then the data breaks kind of maintain spark like they they they are like they frequently commit code and improve spark and stuff then you have pino which started off in linkedin but right now that is star tree uh but yeah pino started off for linkedin and it was open source and then there's a company called StarTree who's actually, you know, giving Pino as a cloud offering and stuff.

So more or less, a lot of these systems end up getting open source like at different points in time. But most of these end up having one large scale use case, which a company tried to solve. For instance, Pino started off with LinkedIn who view your profile rate. They want to make sure you get answered to that, you get notified to that as soon as possible And that the real nature of the problem which Pino solved interesting yeah it interesting to think about this workflow which we probably deal with every day and like they had to build a brand new database to support it that's really cool yes so how much work is it to maintain these systems within uber right like even roughly like so they're open source projects i'm assuming they're being self-hosted in in many cases just because of the scale that you're dealing with i'm sure that there's like cost considerations on like using the cloud system directly is uber like contributing code and helping is it work to maintain is it like i just have to learn like what is that like working model it depends it varies from system to system but more or less there are a lot of things that had to be done in-house to make these systems for for instance we have more or less a team for every kind of offering we use it's also not about making it work at scale it's also around there are there'll be new feature requests new kind of use cases that need to be adapted on something existing plus the organic growth which you always have to solve for so there is there is a lot of work that ends up happening along those lines for projects where open source is active uber also actively tries to open source it back for instance there have been a lot of contributions in the pino space by uber recently as well so that is something that it all depends on the timeline, but Uber still tries to make that happen.

Yeah, no, I think it's a very exciting space. I keep seeing new projects like Apache Iceberg and stuff come out. So it looks like there's a lot of advancements. So we talked a lot about like the first phase of introducing a new data tool and like using it.

What ends up being the second phase, like the scaling the actual platform, right? Like we introduce a new tool, customers start using it or like services start using it. What happens next? as the data volume increases.

Yeah, so there are, I would put it along three different buckets. One is you have some kind of organic growth that you're trying to tackle. The second is you kind of have some kind of feature requests or let's say gaps in your system. Trust me, you'll always find some gaps one way or the other.

So there's always a moving target and that would be a way for you to evolve your data stack, right? So I can go into like specific examples and let's leave the scaling part aside now that something you need to solve one way or the other, right? So let's assume you have some version of your data stack where you have something like, let's say, Spark and let's say some kind of a distributed file system like HDFS and you're able to run queries and stuff. Soon you will see there are some things that you'll need in addition to that.

Let's say you're able to run Hive queries or you're able to run Spark jobs and stuff. Is that enough? what if we can have some kind of a fancy tool, some kind of a UI where people can author these queries, share these reports, because it's not just about running queries. There would be a lot of reports or dashboards that need to be periodically run.

So you will need some kind of a BI tool for that. And that should not just allow you to run queries, maybe share these reports with others and things like that, right? And of course, there are a lot of intelligent, interesting applications with LLMs and Gen AI that you can also integrate into your stack where you give in a text and you get a SQL out of it and things like that. Secondly, you also need something similar for Spark jobs.

So Spark solves this problem where you can write PySpark code where instead of actually sequels, where you come up with some use case, you come across some use case where it is hard to represent that logic as a SQL, but it's easier to represent it as code. Now, having some kind of a notebook environment where you can quickly iterate over Spark apps. So that is something that will get added to your stack. Third thing, let's say you have a bunch of tables.

It's only a matter of time where you end up having hundreds of users and let's say thousands and thousands of tables, right? Now, you'll need a way to know what each of these tables store. Let's say I want to look at users' data per city. Which table should I choose?

So some kind of a discovery tool, some kind of a way for you to search for what data you actually want. We had a tool called DataBook in Uber, which ended up solving all of these needs. Last thing, you'll also have some kind of security compliance use cases, which will end up warrant something like a governance tag. I can go into example.

Let's say you end up logging a lot of customer PIA data. Now, there would be different kind of policies you would want to enforce on them. Let's say you cannot store this data more than 30 days, or this data needs to be deleted across all of the tables. Now, the problem here is it's easy to say to delete the data, but it's actually very hard to delete because it's not just one table you're talking about.

Let's say you log some sensitive data, some PIA data into a raw data table. Now you have thousands of tables deriving from it. Now, how will you end up deleting it from all of them? Setting retention is simpler, but deleting it across the stack is hard.

It's hard for two reasons. one because actual deletion itself is costly and you have to go through all the layers in the stack but the second is also around tracking this so we you need a way to track all of this right you need a way to say which stable derives from which so you will need something like lineage so uber ended up building an in-house lineage system just to track which workflow derives from what and you also need some ways to audit every action in the system so that you know who did what and things like that right you will also have kafka in the picture but it's not just about kafka your source of data can come from anything maybe other online stores oltp systems as well so you will soon realize your ingestion framework actually does a lot more things for instance it needs to pull in data from any source to any sync so you will end up generifying your stack to build that way so uber ended up doing that so we even have flows where we ingest from like google sheets or csvs into the data lake so that people and uh people can like access join some data with some google sheet which gets periodically ingested so these are like some ways which i have seen data stack evolve at over yeah that's that's just incredible you can kind of see the data maturity curve almost like evolve like you have a tool and you have to worry about all these hundred things that come with all the data uh tooling and like i'm very curious about the data deletion stuff because we learn about how deletes are really expensive in databases.

And I'm sure this is not easy. Like managing lineage and stuff, it feels like it solves the problem only or up, which is where you know how to do the delete. But then how do you actually perform the delete reliably? I don't know if you have any insight into that.

Yeah, I can talk briefly about that, right? So how do you... So first of all, after lineage, you know exactly what tables to touch. You're right about that.

But there will be some kind of an investment into what file format to choose and things like that way before your data stack reaches this point right so uber uses hoodie uber has open-sourced hoodie as well so what it ends up doing is a tldr version of hoodie is it enables you to write into your data in like a commit by commit fashion so that you can somehow enforce reader writer isolation right because the reason how the system scale is you cannot have some kind of a lock where every writer acquires that's just not practically possible but you can afford to do that in an on tp system so what happens is you let's say i write to a table and you are consuming from the table what if when your job is running i end up overwriting all of the data right so hoodie gives you some ways to do all of that right so we ended up closely coupling a lot of these things as some kind of an we ended up modeling it as some kind of incremental right on top of hoodie and most of this it varies from use case to use case if you are talking about deleting entire rows of data uh sure dropping it is fine but deleting like specific rows there is literally no other way other than building some kind of a reverse index which kind of tells you oh this id is kind of stored across these files and then rewriting those files one way or the other so depending on how well your data stack uh like how well your stack is you will end up using different kinds of solutions to do this but you have to do this okay and then do you perform an actual deletion itself or do you do some kind of tombstone record like some masking oh yeah so you can obviously do some kind of tombstone records if it's just about your if you just want to hide it from some other system consuming but again you will most of these are compliance requirements so the data has to get deleted beyond some point so you have to rewrite those files so let say your initial file has like 100 records and you want to remove 10 of them there literally no other way other than to rewrite 90 of them the efficient way to do this is you maybe maintain a track of what are all ids are something that need to be deleted and you do this rewrite in a batch that way at least you try to because rewriting files aren't cheap right so to make it efficient you can maybe batch your things and do the rewrite in a batch fashion but you have to end up rewriting them so it all comes from the fact that files are more or less kept immutable in this batch world so even if you have new data coming in you end up writing a new bunch of files and you somehow make sure all the queries that hit this data end up consuming the new files instead of the old files right so that's kind of the model in which all of this batch world operates which means you have to rewrite files makes sense makes a lot of sense yeah that all tracks and on the other side of the world where i'm on now there's like all these startups especially like this like vector databases and stuff is also like a very popular thing how does that track to data infrastructure and data stacks like this like are people thinking about introducing vector databases as part of this is that helpful at all like curious to get your take sure yeah i think with gen ai and ai advancements there is it kind of opened up a new dimension of search right like Initially, it was all keyword searches and things like that.

Now, embedding search, semantic search is becoming popular. So the requirements for vector databases is growing, but I feel you would have to introduce that into your software stack only when you actually have a use case that warrants it. Of course, there are different details twofold. One is, where is data that comes into your vector database?

If that comes from the data stack, you will obviously have to enhance your dispersal framework or some kind of ingestion framework to move data from one place to the other. Maybe your source will be some Kafka data or something in your data lake and your sync will be vector database. Maybe that way it naturally kind of gets added to your stack. The other is you also need to set up some kind of a vector database offering and do some kind of benchmarking.

benchmarking and and vector databases is also i would say require the same kind of tuning that you would actually need to do similar to how you do it in an OLTP or an OLAP system right you need to tie your query shape with your data and stuff so that also requires some kind of tuning so the deal here is if you have a use case it will naturally get added to your stack one way or the other the touch points where it gets added would differ on that note though like is there like demand for using that kind of vector search in data stacks like maybe that's the starting point right like maybe data analysts are asking that i want to do this kind of embed do you see use cases for that come up i see two different kind of use cases coming up one is again i'm back to the same offline versus online right if let's say you have some kind of uh a large chunk of data which you would just want to analyze and try to see what is similar and what is not right let's say you you have a set of text data and you have some set of image data right now assume they all are embedded with some kind of a model that that gives embeddings that kind of are in the same space so let's say you can the text embeddings here and the image embeddings there are more or less uh going to be closer if that makes sense right now if you want to do some kind of a nearest neighbor search where you want to find the closest 10 images in this for every text, and you want to find the closest 10 text for every image, something along those lines, where you want to join a huge chunk of data.

Of course, there are ways and there are good amount of Spark functions as well that enable you to do this, right? That is one way vector search, the abstract concept, will get introduced into your data stack. Of course, there will also be the other online variant where, let's say, you have a lot of data that comes in, where you have some kind of a query text. Let's say you end up having a large corpus of text data where you want to search through some kind of query text that will require more of an online system where you end up using again there are a lot of different implementations of vector search algorithms out there so depending on your scale you could choose whichever one fits you but all you have to do is you have to pull the data maybe from the data lake and put it into that vector database and make sure you index it right and put it into your serving stack that way let's say you get some input query text and you want to find nearest neighbors on that you will hit the online system and not the offline one makes sense yeah and on a slightly different note right like you talked about data tools and a lot of different things you need to do as you move up the data maturity stack to run like an efficient like data infrastructure like if you're advising a company today how would and let's say you don't know anything about the company so you just have to ask them a bunch of questions like what questions would you ask them if they asked you know should i use data break should i use snowflake should i build something in-house like there's so many different changes happening and how do i know what my data stack should be i see that's a good question so of course i'm gonna ask a lot of questions i'll try to maybe think a minute so you'll my first things first is i would try to understand the team size is gonna manage this infra if it's some if it's a small team then of course it's better to try some kind of uh hosted offering instead of you know hosting it yourself at least to start with then naturally there are some things that you would obviously require you will need some way to emit events you will need some way to ingest them and things like that so there are different for instance let's say you want to outsource the ingestion right ingesting data from one source to one sync there are a bunch of offerings as well like for instance The five trend is one.

So I would say the main question I would ask is what team size? And I think depending on your use case, you just need to evaluate which parts of the stack you're going to need. To start with, I'm assuming you need some kind of, you'll obviously have OLTP systems before. So you'll need some kind of an ingestion or a dispersion framework.

You'll need some kind of message buses, and you'll also need some way to store and query all of this data. So you need some kind of a storage layer and a compute layer. So I would say these are like the bare minimum to start with. and maybe going forward you can invest in other things like governance some kind of a discovery tool some kind of a nice well-integrated bi tool to query and things like that yeah and how would i learn you know what requirements i'll need to have in the future right like is there like one place where i find out about these things what's the best for me to learn this on my own so there are a few blogs which i've seen what i was a company kind of puts all of the different components they have in the data stack in a diagram and stuff if you union them across let's say that i did see some blogs from uber some from netflix and stuff so i can share them with you but if you do a union across like four or five blogs you pretty will pretty much know all of the different concepts involved of course the actual offerings that they end up using for instance you will see you will definitely see a dispersant framework out there some would be in-house build some might be a cloud offering and things like that but you would definitely see all of these concepts if you We're just union three, four blocks out.

Makes a lot of sense. And you moved on to OpenAI, which is definitely a very hyped company, especially around AI and all these advancements you've been seeing. What are you excited about in the world of data now? Yeah, I think the thing is that I got to know that there's no shortage of real-time problems.

Because as you said, it's better to do, if there was a way for me to do everything in real-time, yeah.