Insights from Jacopo Tagliabue, CTO of Bauplan: Revolutionizing Data Pipelines with Functional Data Engineering

Speaker A: Hello everyone. Welcome to another episode of Data Engineering Weekly. We took a lot of break, but we have an exciting guest today. We have echopo, the founder and CEO of bowplan. Welcome to our podcast. Why can you give a quick intro about you and Bow Plan for the audience?

Speaker B: Thanks so much for having me. Thanks so much for having me and thanks much for people listening. So 10 out of 10 for my name pronunciation, but 8 out of 10 for the company which is Bao plan because it's German, but yeah. So I'm Jacobo, co founder of Baplan. I'm the CTO here. So leading the technical team and the technical products. I am a professor of machine learning system at NYU and this is my second company. So I guess I am um, a serial entrepreneur. N1 so as my second company, the first one was doing NLP when NLP was still harder and AI was cool, but not as cool as today. And. And then we uh, decided after selling the first company and doing more data pipelines in our life that the second company would have been something about data engineering. And so here we are. So thanks again for having me.

Speaker A: Yeah, great. One of the pulling factor for me to have this conversation is that I threw the papers Bobland published that one and then it was really the question essentially I got is okay, what is the database pain points. This data engineering made you decide maybe the data orchestration engines like we are not short of any orchestration engines like we airflow and those whole families and also how aws, Lambda and like in all those functions to be the one and model and also players are there. What made you decide? Oh, this is a gate to the thing for me, data engineering that can solve this problem.

Speaker B: So I think there's a few, there's a few things there. But the previous episodes has been our first company I built, actually built the entire data stack we look at was like airflow, Spark, emr, those honestly the data lake words, uh, and sure company went well and all of that. Uh, but I also swore to myself I would never get there again if not forced at gunpoint. And then after the company got acquired, we lived to the warehouse world. So the company that acquired us, Kovia, was adopting Snowflake, which everybody at this point know at that time was a bit less known. And the moment in which I moved from Spark to Snowflake was uh, one of the best days of my life. Which tells you a lot about my life in general. But it was also like a fantastic and refreshing moment in industry. But then came with his own problems. Like Python support at that time was inexistent and still today we can comfortably say there is definitely lagging. And so you, when you have to send the data now you're locked in forever and all of that. Uh, so when the Lakehouse trends came about, uh, we thought why don't we build something like the type of thing we wanted to have for the our previous seven years of our life, but nobody ever built before. And so principle number one, simplicity. We want something that is stupidly simple to learn and use. And what is the stupidest, simpler possible abstraction in the history of computing is functions. A great fan of lambdas and we really wanted to reuse existing fast functional server runtime when we started Powerplant, but none of them actually fit the bill. We can go why later. But we really like the idea, we like the idea of withdraw function into the cloud, sort of orchestrates and run everything and do it for you. And that's one, one, one, one big part. The second part was that awareness. Like you can just run the compute, but we need to understand what's going on with the data layer. Right? And of course it's something that stuff like databricks and all data platform know very well, but it's something that no orchestrator or no fast runtime actually touch. If you run airflow, you're on your own as far as how do you cache the data, uh, where is the data sitting? Can you do git for data? Like none of this is an abstraction that is available to you and that's the same for Lambda. So the radical decision was we're going to design a runtime and a data abstraction framework all in one and that's how matplan was born, if that makes sense.

Speaker A: It's interesting that you mentioned about like function as a core construct here you can say let's take a typical ETL Persona. Uh, right. If I'm an ETL engineer, I have like my link of Franca is always okay, there's a data frame and SQL and giving the functions a construct right now. So what would be how do you pitch power plan to an engineer who knows only ETL data frame and SQL here.

Speaker B: Uh, so the general uh, uh, pitch for Bowplan is as follows. You are in charge of running the business logic of transformations. So if you want to use Pandas or Polars or uh, technically even sparse SQL. Even if we don't really, we support it. We don't super like it, but we support it, that's up to you. But moving the data around, caching the data versioning, uh, the data, uh, preparing the dependencies, all of that is for the platform. So people tend to like this very neat separation of concern. You do the fun stuff, write the code, we do the boring stuff, move the data around, build transactional capabilities, build the containers for you and all of that. Of course if your entire existence says oh, I just write hundreds of line of a spark job in a notebook, uh, and that's the way in which I build stuff, probably that's not a good, a good solution. But if you're like, you know what, I always like to have more self contained, more testable, more functional like ability to break down my gigantic logic into smaller stuff, I think then Valplian offers a very compelling user experience and of course a very cost effective way of doing that.

Speaker A: Yeah, totally. It's a dream, right? There are many tools climbing that will really give you that the developer experience that feel like local, that you're developing local and you're putting a production. We take care of all those other integrated details about that. Let's say if I'm an ATL engineer and I want to try a plan, what would be the first? Fine. If you can take in one example and say hey, you wanted to build this insight also this pipeline. These are the step one, two, three and two that you have.

Speaker B: Let's say yeah, of course you can go today on um, Babylon website and access the public sandbox which is public data center loves that. But there's no barrier to entry. Then you're going to get an API key to the API and then you open your terminal, you do PP install and that's literally the only thing you need to do to get all the power plan stuff you want to. As you said, I want to. Let's do time to insight and what happens is literally as follows. You do power plan create table and you point it to an S3 backend in which you have a parquet file. The system will automatically detect the schema and it will actually tell you if there's a conflict if you or multiple parquet file and it will create a table in a nice Berg compatible catalog for you. You don't have to manage the catalog, spin up the catalog, manage the permission, do anything. The system will do that for you. And then you're going to do bioplan import data which will actually take those data and put them in the iceberg. And all of these are committed to the lake as a multiple operation on a branch. And then if you want to run a query you can do Babel Query. If you want to build a transformation, you can do Babylon Run whatever you prefer. And when you're happy, you say Bablan merge. And you're going to merge the development branch to productions so that everybody can see it. You can learn how to do write to the publish or data pipelines, analytics pipeline, whatever you want in probably less than 10 minutes. And that's as much as you need to learn for the platform. So that's what we're seeing is like the quick start is kind of half the platform or already.

Speaker A: Yeah, that's nice. One of the things that I read in the paper also then you also mentioning right now about more emphasis on observing and branching, about the way the boat plan is working. And the paper also talked about a lot on immutable runs and reproducibility. Why does it matter even for a single job? Like, why should I really care about that?

Speaker B: A huge problem that we had in our previous slide when we have this orchestrator that was not really aware of data and then you have big scap cluster for each of the steps and there was really not like, uh, the runtime and data feel so disconnected was that when something breaks, oh wow. Now there's four different things that I need to get back in place to even be able to reproduce things. Also when I reproduce things, I need to be very aware of the fact that I don't want to uh, write over my production table, but I need to write somewhat different versions of that. And all of them may take you our. Sometimes you may able to do it, sometimes you may not be able to do it. And so when we designed valplan, we said we control the data, we control the runtime. Data is immutably stored because of our git for data abstraction, we can go into that code is stored because we run your code. So we Azure code and we have it. Uh, and so every time you run Balplan, you get a job ID which identifies forever the state of the system when you run the job. It means that for me to reproduce a job, the job ID is the only thing I need because I can point in time, go back to whatever was the state of your source TABLE when you run a pipeline, I have the code that you actually run. And so assuming your code is deterministic, which is most case with moving data around, I will get the exact same result on a development branch. Okay, I can work with exact same starting data. I don't bother production. I don't have to look into four different system, um, to sync up My development environment to actually reproduce things. One command, one ID to rule them all.

Speaker A: Okay, let's double click on that. So you mentioned about using it in iceberg table and all those things. What will happen if my code mutate that particular table and then um. How does that. I know data is now mutated. Obviously iceberg even provide you versioning capability and all those things. But somehow you have to map that into your job ID to that model. How do you do that?

Speaker B: So every time you run something in maple we version the entire pipeline. So not just one table then we can go into difference with iceberg but not just one table but the entire pipeline is run on a temporary branch which stores forever the job ID that is responsible for that run. We actually know not just that a table gets mutated by Jacopo or Anand at Tuesday, August 5th. Uh, but we also know which specific job ID actually did that mutation. And that's been auditable for everybody for the life of the entire lake and is exposed like everything else in Bablan to a simple API. There is a commit API when you can get. Can you please give me all the last modification by Yahoo Point which ID and ah. Which job they did it. And so it's always available to do that. Uh, in that sense. Yeah.

Speaker A: So this branching that you're creating, will that be a physical copy of the data that is available in my data like a base table already available there and what you have in comptables could be terabytes. Right? How do you.

Speaker B: Yeah, it's a good question. We do copy and write. So when you actually do create a branch from your current production data data nothing happens. In fact if you do it your maple and create branch this is spontaneous because nothing actually happened there. And so every time you write let's say children table by running a pipeline from your source table only the difference only the children table gets written into your branch. And when you merge, guess what? The non get copied as well. Just a pointer gets swapped in the background. So the system try to be as economical as possible in terms of never duplicating file that it doesn't have to to you as a user it feels a bit like get in the sense of. It feels a bit like cloning, quote unquote cloning. Even if nothing against clone your own copy of the system so that you can basically iterate and experiment safely into that.

Speaker A: Yeah. One interesting thing I noticed in your paper and your we talked about function as a core construct and then the uh. Constructing the DAG out of those functions There's a lot of need for the data shuffling and you in a paper talking about Apache Arrow as a caliber finder. Uh, one of the early version of Airflow is like the bigger problem with XCOM and picking office old Python objects, all those things. So how do you talk through about this system design to share the data across different functions? Why do you think arrow is critical for you?

Speaker B: So the power and functions compared to lambda for example are ah, way more performant and way better for data for a million reasons but they make an assumption on the signature. The only thing that background function accepts as input is parameters if you have them. But typically tables, source table or parent table in a dag and they will spit out a table at the end. And all these tables are arrow. They're either arrow or if you spit out a panda platform will convert it for you. But generally speaking the good, let's say the good best practice is for you to just keep them as arrow. So you can use system that do zero copy like uh, polars or duckd or actually pandas 2.0 in some cases can actually do that. Okay, and uh, this is amazing because why we restrict a bit the type of things you can do because again we're not lambda, we're not general purpose, we just do data. We can get a lot of the benefits of the Arrow ecosystem. In particular if you have, imagine having parent in airflow and then you have three children. Uh, and those are making some data quality check on airflow. Okay. Airflow will require you to serialize in this area like three times this data. Maybe you go to S3, it's super slow, it's super clunky. You have to deal with a lot of complexity yourself. Uh, when you express this in Balplan, Balplan will just pass especially if it's in the same host. We'll just pass the buffer of the table to Children Function. Children function are completely independent process with complete independent functions. But they won't duplicate even a single byte of memory to perform quality checks on this. So you basically get vectorized expectation at zero copy and it's super fast. And from the point of view of you writing the code again this is an implementation detail you don't care, you just write a function. You say that this function is apparent, you say that this function needs polars, uh, or Pyaro and the system does it for you. Arrow is a lingua franca between all the part of the system constraints what you can do, but in a good way because then it enables all this Optimization that makes the system like 20 times faster and easier to use than on purpose built DAG system like airflow, uh, in this case.

Speaker A: Okay, okay, okay, yeah, yeah. It's a clear value proportion of Apache Arrow and many other systems also adopting the similar model there.

Speaker B: I know

Speaker A: jumping onto this versioning side that you've mentioned about and then Nessie is one of the very famous versioning capability there and you could style table launches and like, you know, pretty good paper around that. How does it work in a distributed team environment where multiple people working on multiple branches and how do handling those race conditions know the aspect of it?

Speaker B: So there are two. So our own system is Forksa is base original Nessie and now it's evolved into being kind of his own thing for a few reasons that we can go into. But There are generally two big use cases and scenarios for which Git 4 data makes sense. One is collaboration, uh, experimentation like the thing that Git is known for. Okay, imagine the same thing for code but now for data. And the second one which is a bit trickier is transaction. Okay, and this is more something that has to do with the mechanics of how atomicity works in a very distributed kind of multi process, very weird database like system like Bowplanner, which sure is a fast runtime like Lambda, but at the end of the day impose guarantees in data like a database would do. So the first part which is collaboration, it really looks like git in the sense that Bauplan namespace your branches by your username. So all my branches starts with Jacopo, all your branches can start with Anant or whatever that is. And so this will automatically makes my branches available to you as read but forbid you from writing into my own branches and of course vice versa. So that's like your own sandbox environment that you can be safe with. So there's no risk condition in those writes because you can write in my own branches and I cannot write in yours. Uh, the way in which conflict get resolved is git like in the sense that at some point I would want to merge on main and you may want to merge on main like your production, let's say version of the tables. And right now the system is very conservative. So if somebody touched a table you want to, you want to update between when you fork, when you branch off and when you're merging the system is going to say hey, sorry, I'm not sure I'm. You're not allowed to do. And uh, so you will need to basically three days if you will or Just rebrand and then with run, stop. Of course, since Balplan, every transformation is a code, uh, even this operation is trivial. You just basically rebrand and run again and that's fine. So that's why the system is sort of very used to, very easy to use. But there's an argument to be made that if I add some rows to a table and you add some rows to a table and there's no other reason to believe that this is a conflict, we should probably merge both. You know what I mean? Our changes are not conflict or we should do that. But right now the system doesn't have that level of, let's say, semantic understanding. M We have however, a paper that's going to be published soon on our formal semantics for here for data. We also have a blog post with Alloy doing formal verification on the system and we have a new blog post coming out. So for people that are very interested in understanding how the mechanics of this, let's say primitives work to guarantee some consistency, we're actually doing a lot of work into that and sharing with the community as we go along.

Speaker A: Yeah, that would be amazing. I think like branching, the concept itself is not well understood in the, in, in the data engineering community in my opinion. Like sometimes it's a little bit of confusing the way that some systems are working. And one, uh, big criticism I've seen in branching is that the insights always should come from a typical main branch or anything. But there is a tendency, right? People can create the way right now the branching work, right? Like I can create like user, like anand underscore user at different table and like n number of tables, nobody know who using it, what to trust and all those things. And like when you have a branching also like you will end up like a case where like a too many branches problem, like people are creating too many branches, it's never getting deprecated or somebody have a reference to some other branches. That's a very bad situation to be here. So if I'm a data engineer, I wanted to think about branching as a concept. How do you picture a mental model regardless of you're using a program or any other system, right? How a data engineer should think about branching in general in a data pipeline.

Speaker B: So our suggestion is always to start from A real problem, that is that don't create 3,000 branches or underscore user or whatever. Start with a problem you have. And in our experience we are typically, let's say three main use cases. Use case number one. Trivial is the same thing as a Feature branching. Get somebody task you to build a new pipelines. Okay, you want to start with work with production data, but you don't want your downstream tables to be connected to formal looker or whatever until you fix them enough for the downstream consumer. You open a feature branch, you call it Jacopo, new feature, new pipeline, you develop, you iterate there completely separate from anybody else. You build, uh, I don't know, like a prototype dashboard for your downstream system. And people say, okay, this is what I want. You merge, you close it, you delete it, everybody walks away happy. That's, that's the easy case. That's case number one. Case number two. Typical thing these days, experimentation. So you have different ways to solve a problem. And I'm sorry to say LLM, but I need to say LLM M, you have three different LLMs that you don't know which one is the best one to solve this data wrangling or data cleaning problem. And so what you do is that you can parameterize your DAG in Bauplan, taking a parameter which is the model and then you can run a for loop because Balplan just Python, you can say for modeling my models, run with this model and then you write this model in each of its own branches. So you're going to have a pipeline that is called, I don't know, Jacobo, Gemini, Jacobo, OpenAI, Jacobo, whatever, anthropic. And now you have uh, three different downstream tables that you can compare with code and pick the best one and then you merge the one that you like and you delete the other ones. So this is also very powerful. Let's call it experimentation, use cases. And then finally is what I mentioned, the debug use cases. Like you come to me and say, oh Jacopo, I broke something is red in my, my alert. Ah, can you rerun this pipeline? Can you please tell me the ID? And then you tell me the ID. I'm going to create one branch which is called jacobo.debug and then I'm going to run there your pipeline and then I'm going to do the queries and do whatever I need to do to come back to you and say, you know what, this field should not be null. So there's something wrong here. That's why this crashed. And so this I think is the third type of really use case driven type of things. Uh, so developing, experimenting and collaborations are the three main use cases. And again there's one of them in baplain as the additional which is transaction but it's more of a mechanics of how we guarantee certain things. Not something you have to think about in some sense.

Speaker A: Yeah, okay. Yeah, I think, yeah. I'm also very interested to see how the branching world has a uh, default design principle in data pipeline. Stepping a little more onto the old. Right. That's my favorite port all the way since like you know, understanding like what is happening behind the scene in bow plan. So your paper kind of stresses more on the vertical scaling part of it. Right. And you also take care of the data like a scheduling part of it. And how do you want to place uh, the task around the data. So for your scheduler design, if I use Apache Span, I pretty much know it's a distributed system engine. Like it's going to spin off like interpreters programming model is pretty good versus a vertical scaling versus horizontal scaling. Like how should I think about your system?

Speaker B: This is a very good question and we thought very long and hard before, before building Bablunk. Babplane gives you this, assuming that I'm right and Babla gives you this very easy to use, no nonsense abstraction. Functions are nice to work with, are easy to reason about. It's easy to concatenate functions, they're easy to test. There's a lot of they need to isolate. There's a lot of good things about Bubble, right. Try to run two different versions of Pandas in a Spark pipeline. You cannot because the system is monolith. Well, in Bubble it's trivial. You just step one function with a Pandas version one and that's going to work. The price that we pay for this though is that we basically take whatever you write inside of this function, a face value and we schedule inside a single host. So we can scale that function inside this host, whatever this host is, uh, as much as the host goes, uh, but we can go beyond the boundary of the single host. So what it means today is that uh, BabPlan supports. I mean we have a customer that runs hundreds of gigabytes per pipeline, but there's a limit to that. So if your pipeline is a petabyte, Babylon is not the tool for you and it's not going to be in the near future if your pipeline is between 10 gigabytes and 100 gigabytes. Babel is probably the fastest and best way to do this between SQL and Python. If your partner is between 100 and 300. I think that really depends on that specific configuration and tuning that you want to be. Uh, in my experience, based on our customer, we wrote a Long time ago, a, uh, paper called you do not need a bigger boat with a reference to the jaws and will we argue that we see many customers because we're B2B company and if you chart the distribution of the queries and the pipeline that they run, 99% of them are below 100 gigabytes. So even if people adopt Babplan just for this segment, even assuming just this segment, that would be a ginormous amount of like workloads that we can run as, as well as anybody, if not better.

Speaker A: Okay. Okay. So right now the system limitation like around 10 to 100 GB, uh, which I believe like all the incremental pipelines. I don't see any other incremental pipeline runs more than that data anywhere.

Speaker B: Yeah, yeah, exactly. At the end of the day you ask yourself, but then I run every hour. I don't really process again. You can have terabytes in our catalog. It doesn't really matter. But like when you say but then I'm going to run maybe most of the hours, then it's going to be fine. There's no need to do something else. But I understand there are some very peculiar system built for very peculiar use cases that of course are great at what they do and are necessary in a sense. Let's remember that Spark gets sometimes a bad rap and sometimes I, I spent my fair share of blood and uh, sweat on that. But when Spark was created, the average EC2 was as small as a lambda. Today it's a very different one. Now it's easy to think, oh, Spark is overkill for many things, which I do believe it is. It definitely is. Uh, but the world in which that was created was a very different world than today. Today you can get a terabyte machine on AWS as a standard thing that everybody can do with not much of an expense. And again, filling a terabyte of a machine is all a pipeline that you can run at the same time.

Speaker A: Yeah, I mean there are some legitimate use cases if you want to a reconciliation or a financial accounting pipeline. Definitely need to go back a long way and see figure it out that But I feel like majority of the workload if there is not an incremental pipeline, they should really look into the uh, pipeline design for that. So we talked about the rare like a limitation the range that the data can process much faster. That brings another question is even though it is like you mentioned about each and every function running in above, but this function has to coordinate with each other. Like they have to communicate with each other. And that means like shuffling of data happening across those things which is one of the biggest pain point in Spark all out there. So how do you how your system designed to shuffling this data to be made more efficient? Well we briefly touched upon arrow how that your system efficiently you does that.

Speaker B: But yeah, so our function, this clarity interface when they just say hey give me a table and then I'm going to get all more tables and I'm gonna get you a table back at the end and all these tables needs to be arrow. But this is the function when it actually gets instantiated by its own containerized environment. Doesn't really need to know or care. You know how the arrow thing gets there. It's just the odd that it's there when it runs. Uh and then it's up to the system. If the function is scheduled next to its parent because there's room the system is just gonna get a pointer to the arrow and we leverage the zero copy mechanism that I mentioned before. So there's almost instantaneous. There's no serialization price. Uh is actually incredibly efficient. Uh but even if you schedule in a different dose and you need to send over the arrow bytes to flyte which is the typical way in which arrow gets sent between servers you can check and you verify that it's actually as fast as reading a parquet file from disk. In the sense that avoiding the entire serialization of what potentially is hundreds of millions of rows over two EC2s with the bandwidth guaranteed by the cloud is going to be faster than reading a file. Yeah from parquet just because of the encoding and decoding of all of that. So compared to the typical uh endoff in airflow or again a non purpose build back system to move data around the system is so much faster. There is not even a comparison in that sense.

Speaker A: Okay, interesting thing that you mentioned about like a parent child task relationship that do you apply any diligence to co locate those compute somehow?

Speaker B: Yeah, yeah. We so the system right now. So the business model Baltimore, which I know it sounds like a weird thing to bring up now but bear with me. A business model of Baltimore is that people typically pay for what I say the amount of time they want to. Wait, what do I mean by that? People pay, don't pay by usage like Snowflake. Uh, people pay by basically concurrent capacity. They want to be available. So for example if you let's uh say you pay for 10 gigabytes whatever there is. Okay. The system will try aggressively to beam pack queries and pipelines and Python and SQL all in those 10 gigabytes. And then when it cannot do that, it will just buffer it for you. Okay.

Speaker A: Okay.

Speaker B: It's a priority concept in the system, but the general idea is that you throw stuff to the system and the system will run queries first and we'll run, um, pipelines later. Will interleave queries and pipelines smartly. Because again, since everything is a function, you can stop a dag me there between this function and then do that. So that's what the magic of the scheduler does. And uh, this comes from. If you read the paper from XFASS at Facebook or the original Borg paper at Google on scheduling at that scale, the insight is the same. It is people need to run a lot of stuff, but they don't need to write it all right now. Actually, the things that need to run right now are a sub portion, 20 to 30% most of the entire compute. But once we provision some capacity, we make sure to provision the things that need to run now and then we use the rest of it, uh, to basically fill the rest. And this model is incredibly efficient from a, uh, price performance point of view. Because instead of imagine you have my old setup airflow DAG with a cluster for each step aside the fact that I have to wait for the cluster to go up and build the containers, it's going to take 15 minutes. But even if that was instantaneous, I have three clusters that go up and down.

Speaker A: Uh-huh.

Speaker B: With a lot of over provisioning because it's very incredibly rare that this cluster are going to be full capacity. Just not how it works. Now in bow plan, I throw the function at the system and the feature will try to fix all of that as much as possible. And if not, we just retry a bit later. So the average level of occupancy and of, let's say leveraging the actual underlying computer is way better than the other system. Again, that's how the function scheduling comes about. Is that because if you are not in a rush in most of pipelines and your pipelines are within the range that are like sweet spots for the capacity you want, mapling is the cheapest possible thing. You can run a million pipeline and it's going to cost the same as running 100 or the same as running 10 million, which is a very different, I don't know, mental model than you pay me every time you run something. If that makes sense.

Speaker A: Yeah, totally. I really like the plan. If I wanted to get the result much faster, I pay extra to get in if I wanted to relay it outflow. So the controllers are somewhere between the users and then they decide like how to do that one. I think that's a very transparent application. Google Napa Papers also talked about a singular trade off. Like you can, the developers can decide like when the recency of the data is available and also the other aspect of it. Right? Yeah, uh, yeah.

Speaker B: I think our system community has been working in very similar thing. I mentioned xfast paper by Facebook like a year ago or something like that when their insight was like xfast is basically lambda but just internal for Facebook. It's that fast but just for Facebook. And their point was like, do you have any idea how many freaking functions people of Facebook want to run and how many machines we would have to have if we need to run them on the spot? And so what they did was like we uh, purposely cap the total capacity allocated for this function and we just have a back pressure mechanism when the function is not very urgent. We're just going to schedule later. So at the end of the day we're going to have a much higher average occupancy and people are going to be happy anyway because once again some things need to happen now, but a lot of business things just need to happen, but not now. Just need to happen in the next hour. As long as they happen in the next hour, we're all fine. And this is this margin that is where the scheduler comes in and helps you become incredibly cheap in a sense.

Speaker A: Yeah, totally. We spoke a lot about the data pipeline and batch pipeline running that one. So how do you see Boplan as a system for interactive workload? Like open up a notebook somehow connected to the cloud and then trying to access some data there. And when we talked about interactive analytics, do you think Botan is a system for an interactive analytics also like purely focusing on the branch side of it.

Speaker B: Uh, so we do support. So you can launch any query or query like systems to the query like inter command to the system today and the system will run it. I mean the fun thing is that it's also a query for us as a pipeline inside of a problem because there's nowhere else like nothing exists before you ask for it. Our containers are so fast, they get destroyed on the spot, they get spun up on the spot. So if today you do Babylon query select count from my table, the system will on the spot spin up a container that contains a fork of DuckDB. In our case, it will fetch data from your catalog pointing to the same exact version that you need the same branch, whatever and then it will stream it back through flyte to your laptop without you having to do anything. This is a very good use cases for us. Uh and then the question is this where the system shines? And the answer is no. I think the system shines in making queries for the let's say gold layer or the very final tables that you produce with a pipeline. And why is that? Uh, because it's free. Again remember you don't pay by query. So in many cases once you prepare your data and your data are nice if you connect your dashboard, your looker, your stream leads whatever metabase to mapla that's free and typically a dashboard can wait a second to load. Okay, it doesn't really. Nobody really dies because a buzzword do that. So I think that's great. If you need to do large scale ad hoc querying you should use probably Snowflake or Clickhouse or a system that is for that. Do remember Bubbler runs on your bucket, the data is yours. Uh, all the iceberg data is exposed to you for free in an API which means some of our customers, what do they do? They do pipeline in Baltimore, they run half the query in BPlan and for some tables that they need to have some certain sla they register that table in Snowflake and then use Snowflake. The best of both world work. You cut a lot of the cost in transformation in Snowflake for which you need something that is more economics or cheaper or whatever. You don't have migration costs because now everything is in your bucket and you can point Snowflake to it when you need it. So that has been a very successful let's say pattern for people using Babel like the Lakehouse dream. One bucket, your storage, your data, one format and then you mix and match the compute to what it makes sense. And there's evidence now that Bubble Compute in certain Apple workloads is honestly better than much larger players so to speak for these reasons.

Speaker A: Okay, okay, I have one question on that. Um, the way they utilize the container and especially it's a well known problem in AWS functions around like a cold start problems and other things and uh the nature of an interactive workload sometimes need as I building my pipeline and let's say the container goes out, I don't want to recompute again like some kind of a state needs to be stored which I'm like again at this one there are like 30 people trying to do like systems like Docker or Filecrackers. They're trying to store that a string out of it. So how do you solve that problem? Define uninteractive. If I'm doing an interactive analytics and then when I lose a state like how do you make sure the reliability remains?

Speaker B: That's a very good question. So there's a. So because our containers are ephemeral and stateless per se, so every time you run a Vaplan anything query pipeline, import data, the container is unique in such a. But the system is so fast in building containers because for example we don't build containers like, I don't know, PIP install requirements or whatever, but we build containers with the system that we've documented in literature in some papers that we publish, which is we have this stateful cache of packages that has been installed, um, running the transdict dependencies of whatever dependency graph you need to do with Python and they get mounted as folder in the container that gets spun up. So when a new container, when a new container needs to run, the container just needs basically to do start the container itself. Uh, and all the packages and all the things are actually managed by a stateful cache, uh, that persists across invocation and then makes the invocation basically a few hundred milliseconds, so to speak. That's the same for data. Uh, again we publish a paper revealing our cache strategy. Now our cache strategy is able for example to do a differential caching between months. So if you ask for the data for January and then you run again January and February, the system will know that January is already cached and so we just request February and so it will merge the two before running the function. So all of this stateful logic doesn't impact correctness, but speed up a lot of the stuff, uh, and value propositional balance that you don't need to care or build or retrofit airflow to do any of this, which will be an outstanding amount of work and we do it for you basically. So that's, that's how it works. Stateless container but stay full cache. So we're really aggressive in caching a lot of stuff, which is why the system is so responsive.

Speaker A: Yeah, yeah, totally. Out of curiosity, Apache Spark been under five for a little more time. It's been there for a long time. Nobody really challenged Apache Spark for a long time in my opinion. There are some interesting players that are coming into this picture. Apache Gluten is one project, one project that essentially, you know, transpiring to a physical plan in a much more optimized way, at least they claim to be how do you see Borplan performance in comparison? Have you done any typical workload on RTPC benchmark? Like how do you see Borplen is performing against a typical Spark workflow?

Speaker B: I think so. So since powerplan doesn't tell you how to write your code, uh, the real question will be if you have let's say an Apple to Apple comparison. I have my, I don't know airflow or whatever databricks workflows and every, every node in a DAG is by Spark cluster with the code that I wrote for that and materialize a table pack and then that goes into the children and a typical setup for most people the equivalent in bow plan for that would be just whatever orchestrator you want to launch to bring it and then power plan DAG with Polars or whatever you want in the vast majority of cases in as long as you don't go into the distribution as long as you don't go across the Austin this system will describe and just waiting for a pilot class to go up but will basically destroy the entire thing. So I don't say it's a fair comparison in the sense that it's faster, but it's not faster because we're fast. It's faster because it's a mass Simplus fractions and we don't have to deal with all this distributed bullshit uh, that goes on the other system. If you take a query by query comparison. Again we're not a query engine per se and so the question for you will be where do you think that Polars like DB Data Fusion of all of these things will lose to spark? And I think there's a bunch of evidence now that again probably until 100 gigs there's no even point now anymore to go distributed just because the price you pay is gonna be, is gonna be too much. And then after a while it makes sense to go distributed because you're gonna, you're gonna recoup that uh, in that sense I think the Veloc Gluten gluten value Comets gluten full of testing. I think this is super exciting and I wouldn't say Spark replacement because at the end of the day the promise is keep the Spark API and the Spark scheduling, the Spark planning and just replace basically the JVM code which is a testament of the success of Spark. Like it's so resilient and so sticky that people don't really want to reinvent a system. People just want to replace the small part that can be do better but keep the Higher level thing, the same. Which again is a testament of how well designed the system was for a while. And I agree the system is the reigning champion of data. What I would say though, and going back to our previous stuff, is that is the reigning champion of data. For some type of data, uh, in other cases it will make your life harder than it needs to be. And I think now there's better ways to organize your code around this. And that's by. That's of course our entire point of view.

Speaker A: Yeah, I read this quote somewhere. It's essentially saying the world would have been in a much better position if we can solve all the world's problem in one single machine. I think duxdb and BO brands more like at least trying to get this philosophy into reality that it'll be very interesting to see this one. We talked a lot about Iceberg and I think one of the curious question for me is that is it strictly bounded to an iceberg? Like what about like other Lakehouse formats?

Speaker B: That's a very good question. So when we started like a couple of years ago, the ecosystem was not nearly as big as it is today. And I think we're proud that some of the choices we made, uh, actually paid off massively. Iceberg arrow git for data, uh, Python, like all of this thing that now seems obvious two years ago were a bit less, oh, API first, so now agents and all of that. So like all of that was great. We picked Iceberg at that time because it seems to be together with Delta two years ago, this sort of competing, competing stuff. But Delta was very bound to a certain type of client with a certain type of existing ecosystem. And we as a company didn't want to go and compete for those workloads immediately or directly because of obvious, obvious reasons. And Hasbro seemed to be the more open and let's say heterogeneous format in the sense that different companies will adopt it, running different workloads. Some of these may be easier to get from our perspective, which is actually what happens. Our first clients were not clients of Snowflake or Databricks or whatever, but we're actually AWS users. So people who build Lake house by assembling together six or seven tools in AWS or Trino or a bunch of stuff, uh, and they were just so unhappy with it. The cost of maintenance, the ramp up of a new hire. When something happens, there's six different things and into the bug like they were just very miserable. And that's our first market fit was there, it was okay. No worries. We didn't pitch Iceberg, we pitched. You know what else? We're going to put this in a format that doesn't depend on us. So you sleep Sunday night. But we're going to replace this 6 AWS tool with PP install and we can promise you that your next intern is going to learn this system in an hour, which is actually what happened. And I think there was a very good in the retrospective there was a very good bet. And I don't know how you feel in the scene in the ecosystem, but it seems now after tabular acquisition and snowflake opening, uh, that Iceberg is indeed the standards for many people.

Speaker A: Yeah, we're almost closing to our podcast. Before closing up, I just want to ask one question. You've been in the data space for a long time. What do you see in next two years or three years down the line? One problem that nobody really solved in the data engineering that you really wanted to take a diving on that what that would be and why?

Speaker B: Uh, so there are two things I think uh, there's uh, an old problem and a new problem. Maybe it's the same problem in this guy. So the old problem is as follows. Data engineering and software engineering are still not really that close and I think it's bad to bet against software. Something we learned in the past that software with all these problems has a very well defined way of working and is always going to win in the end. People used to deploy machines manually and now we have Terraform or Plumie or now even DevOps is a software problem in a sense and everybody's happier for it. Right? You don't need specialized people to do it. Most full stack people can learn it, it's versioned, it's controllable. All of that was a good idea. But it was now it's obvious. But at that time it was a revolutionary thinking and all problem we want to solve uh, is this one. Can we make that engineering more like software engineering and git for data merge, pull requests, containers, like all of these things that seems trivial. Uh, in a world that is still dominated by one off scripts and notebooks that you schedule, which is obviously like a very bad practice and all of that. I think there's still a lot of work to do. The new problem which may solve the old problem by unexpectedly is automation. I know everybody's crazy about the AI pending bullshit and all of that and there's a bunch of that. I agree. But honestly. And we publish a uh, blog post we code with together AI like a Month ago for the first time me doing AI for most of my life. But I'm very skeptical of the recent wave of hype. Have to acknowledge that seeing an agent doing write ord publish on Valplan is actually cool and ah why it's cool? It's cool because it's softer because since writing pipelines in Valplan is git for data, is containers, is isolation, is merging is all the things that you need to do anyway to have good software when an agent can pick it up because these APIs and the software is something magical. And I think we're like at day one of this marathon I don't think we have solved it. We don't solve it but we have new announcement coming to walk towards that word and that's very exciting. For the first time I seen the promise of LLM working for data engineering. After a couple of years of mostly IP or text to SQL which is nice but it's also boring and also whatever. Um, like yeah, ah, that that part is to me is like less exciting but autonomy is engineering for boring stuff that you and I don't want to do. Loading data from S3, doing write only publish quality check, all of that. I think we're closer to a world in which most of that can be automated. If you have a platform that support the rides, sandbox abstractions and all that.

Speaker A: Totally. I'm also excited to see how this LLM going to impact the data engine workflow and then git for data is always a dream for my Google. I see how they work. Um, fine. Great. Thank you so much for your time.

Speaker B: Thanks so much for having me. And if anybody listening to this want to contact me, I'm the easiest person on the planet to reach. You can just send me a message on LinkedIn, send me an email, whatever. We're very public in the sense that we share our ideas quite a lot. So anything I said today, if you want to learn more, reach out or go and check our blog posts and papers. There's going to be a ton of information on the details of how this work with benchmarks and low level system details. If you want to try the same technique on your own system or if you want to try it out.

Speaker A: Wapler.

Speaker B: Okay, yeah, thanks again.

Speaker A: Yeah, highlight all the papers in the notes frame.

Speaker B: That's awesome. Uh, thanks again was super fun and uh, to see you around.

Speaker A: Take care. Bye.