The State of Lakehouse Architecture: A Conversation with Roy Hassan on Maturity, Challenges, and Future Trends

Speaker A: Hi, everyone.

Speaker B: Welcome to another episode of Data Engineering Weekly. We have Ashwin Hashisher with us today and we have a special guest, Roy. Hello, Roy. How are you?

Speaker C: Good, how are you? Thanks for having me. I appreciate it.

Speaker A: Oh, great.

Speaker B: Yeah. Ashwin, what are we talking today?

Speaker A: Since Roy is here and Roy is quite popular, recommending Iceberg and Lake House. I think, uh, majority of our conversations are, uh, going to be structured around lakehouses. Getting Roy's viewpoint on where he sees the market going. And I'm sure in his role he's going to help us understand what's really happening on the ground. I think let's spend more time not in the basics of what a lake house is. Let's not define that. I think you've got enough content there, but more understanding. What's the reality check on Lake House. But before we get there, we should know a little bit about Roy. Uh, Roy, why don't you introduce yourself and then we go into the topic.

Speaker C: Sure. My path goes a long way. I have two parts to my career. My first career up until 2016, starting in 2000, I focused a lot on service providers. I used to work for Comcast, I worked for Motorola, I even worked for Huawei. Building networks, right? Building all the data networks that you're using today on your cable modems at home, uh, your video networks, your voice over IP networks. So I was network, uh, engineer, system engineer, working on building those networks. And then I had a point in my life where I said, I need something different. I want to move to the cloud. I want to try something completely different. I made a transition. I joined aws back in 2016. I had two parts to my time at AWS. I think the first part was highly focused on data from the perspective of go to market business development. So I spent about three years managing the Go to Market team globally for aws focus on the AWS analytics services. How do we bring those to market? How do we partner with both customers, but also ISVs and SI partners to get users to leverage our services? And the second half of my time at aws, I was a product manager leading projects like AWS Glue and AWS Lake Formation. Heavily influenced Amazon, Athena and emr. So participated in a lot of the service and product development within the database ecosystem. And I moved over to appsolver. Been with appsolver for about three years. I, uh, was leading the product team there. And, uh, then towards the end I also got responsibility for, uh, marketing and Go to Market. And that's where you saw a lot of the Explosion of iceberg and community evangelism as we took a step into the iceberg realm. And then beginning of 2025, Qlik acquired Upsolver and our iceberg lake management technology. So now I'm a product leader in, uh, qlik, focused on storage and lake capabilities primarily from the integration. And then next year, how do we modernize the QLIK platform with new technologies and modern ways of, um, data engineering and data development. So that's it. And that brings me here.

Speaker A: Wonderful. Roy. Before we join the call, Roy follows real football.

Speaker C: That's right.

Speaker A: Soccer

Speaker C: not doing great so far, but it's okay. We're going to rebound and we're going to have a great team soon.

Speaker A: Let's do a reality check on lakehouse projects. But I want to hear from you firsthand in terms of where do you think the majority of lakehouse implementations are and uh, what do you think is happening on the ground from customers and where do you see things going a little off track?

Speaker C: Sure. One thing to remember is that lakehouse, uh, is an architecture. It's not a product, it's not a specific tool or service. It's an architecture. So there are different ways that you can implement it that you can say, hey, I got a lakehouse. So I would say in particular, like databricks, pretty much every customer who is using databricks is already using a lakehouse. By simply using databricks, you're storing your data in an object store. You're using the delta lake format, you're using your querying, you're analyzing your data. On top of that lake, you already have a lakehouse, you have a very large number of customers who have a mature lakehouse implementation using databricks. On the other side of it, you have, let's say, aws. There's a lot of different services that are already leveraging data in an object store like Athena and glue and redshift and EMR to get the full advantage of these services and to be able to save you, reduce your cost, you would be putting data into that lake. Now, are you using simple parquet files or are you using iceberg, delta or hoodie? From what I'm seeing, a large majority of users are still building traditional data lakes with parquet files. But slowly you're seeing the migration of those data lakes towards a lakehouse architecture where you're using an open table format like iceberg or delta lake. What I'm seeing more is iceberg. Uh, AWS tends to push that a little bit more. But again, it really depends on the environment that the customer has and the tool that they have to use. I think from that perspective you're starting to see more maturity. And then you have other services like Azure Fabric that's built on top of Delta. So by default, OneLake, using, uh, the Delta format, you already have a Lakehouse implementation. Now it's more of a vertically integrated lakehouse where the storage is lakehouse, the cataloging, the computer is all baked into fabric. But you can say that's a lakehouse and the same can go for Snowflake. What I'm seeing is that customers are starting to use Snowflake in a more lakehouse way, where new use cases, new workloads are starting with data in S3 and objects in, in Iceberg being analyzed and modeled and, and queried using Snowflake, where the stuff that's already in Snowflake just remains in Snowflake for now. I'm not seeing a lot of effort to migrate out of that and go into Iceberg. But Net new workloads do land in Iceberg more than they would in standard storage and BigQuery. Google, they're just starting to come out with that. That ecosystem is still very new. Those users are typically running everything in BigQuery and are used to the BigQuery ease of use and storage. I don't think there's a whole lot of value right now to switch those workloads to Iceberg besides just following the trend. Right. Or just being open. So I'm not seeing a ton of adoption on the Google ecosystem, but I'm also not spending a lot of time there, like, um, talking to some customers who are on it who have interest, but they're like, yeah, we're just in BigQuery. That's really all we need and we're happy with it. But I definitely see that big time on AWS and Azure. Lots of mature lakehouse deployments. So I think from that perspective I'm seeing both of those. And then in Asia, a little bit of hoodie, a lot of paimon these days, but definitely trying to push that and maturing that those lakehouses. So in general I would say that yes, there's significant number of mature lakehouse deployments and there's a lot more coming up.

Speaker B: Um, interestingly, you mentioned about Snowflake as also a lakehouse architecture, even though internally they cannot do that one.

Speaker A: Right.

Speaker B: And when it comes to the open table format, is it open table format is good enough for an enterprise to autofill Lake house because you need to have all the Other maintenance work and like other complications that organization enter into and got into a problem with that one. So given that question, we have Iceberg and then Athena. Uh, Trino. Like a desegregated architecture where you compare with Snowflake, which is like well integrated architecture.

Speaker A: Right.

Speaker B: Which one is the best one?

Speaker C: Good to hear. Yeah. So like I said in the beginning, there's no one way to build a lakehouse. It. It really comes down to what problem you're trying to solve. If you're saying to yourself, I love Snowflake, everything is in Snowflake, I'm happy. But, uh, my company, my CEO, is saying, we need to have an open option, then I'll put my stuff inside of Iceberg and keep using Snowflake. I'm a happy camper. I checked the box. I made my CEO happy. If tomorrow they said, hey, we're going to get rid of Snowflake and move to Data Warehouse X. Data is open. I can just. So I'm happy there. If you say my warehouse costs are high, I want to change that. I wanted to take control of how my costs are managed and how I do things. Then you want a, um, more decoupled lakehouse approach. Right. You don't want a vertically integrated lakehouse approach. That's where you say, okay, I'm going to break things up and I'm going to create a more natural lakehouse architecture where, you know, a lot of my data is in. All my Data is in S3. I have a separate catalog. I have different things. But I may still use Snowflake for all the transformation. Right. I may still do ingest to the lake, clean prep, move it to a silver layer, do my modeling in Snowflake. I still use Snowflake or whatever engine as my compute, but at least I have control. I can say, hey, these workloads don't have to run at the same time as this workload, though. These workloads can be offloaded to a Python function in Lambda instead of doing them, um, in Snowflake. So I have control. I may not do it, but I have control. Right. And then you have the approach where you say, I'm just going to have a complete decoupled lakehouse. I want to be able to control everything. And then you may say, Snowflake is my query engine. Databricks is going to be my data engineering pipeline engine. And then I'm going to use, let's say, click for my data ingestion. Fantastic. You have that flexibility, but unless you think in that direction, it'll be very hard for you to get there. The best practice that I'm seeing is that when you're starting to think about modernizing towards the lake, you should think about what does the end goal look like. Right. Uh, you may not get there. You may not even want to get there. But you should know what the end goal needs to look like and what the end architecture needs to look like and start designing and building towards that. So then ultimately if your data warehouse manager in a company decides to leave and a different person comes in and say I don't care for data warehouses, I want to build something else, you're set up in a way that you can succeed and move quickly. Um, but yeah, that's one approach.

Speaker B: Yeah, totally. Speaking of this adoption of lakehouse format and then six months back we are seeing like a full flown debate on what would be the table format should look like. We got Iceberg and Delta Lake for example and Hudi. All three has its own merits and demerits. Like we run some internal poll which should run an internal poll. We found out like dental Lake is widely used in Iceberg. There's another notion of Iceberg win the war. And you also.

Speaker C: It depends who you ask. Yeah.

Speaker B: Different from you including kind of implementation very widely used in a very large scale deployments where they have a streaming upset requirement and maybe because of the nature of overall being built on similar semantics around that is fit into this one. What like is that Iceberg is like a done deal or for an organization to look into this one or uh, what should they really look when they're choosing Iceberg or any Lakers Look, I

Speaker C: think we're still very early on. Right. The debate is more of an emotional trigger for people to post stuff on LinkedIn. Right, right. These are insanely awesome communities.

Speaker A: Right.

Speaker C: Each one of them really good. The engineers working on these projects, I focus on bringing a lot of value and doing something good for the community and for the future of the data platform. I don't think there's one better than the other. Obviously we made a bet on Iceberg because from our customers we see much more demand for Iceberg. But again Delta is great. If you look at the table formats, Iceberg and Delta are very similar. Right. There's really not a lot of differences between them. If you look at the community discussions, um, there's talks about converging some of the standards and some of the ways that data is encoded and managed between Iceberg and Delta. So converging those two things you can still get the benefits of Both formats, but from a developer perspective, one person who's writing code to work with them, it's actually a lot simpler. There's less for them to know and less hairy corners to deal with. UDI, again, is great, is really good for streaming use cases. Uh, Paimon is not in that space as well. So I think that's going to eventually coalesce and we're going to have, I don't want to say a winner, but a preferred format. Uh, if the bulk of the workloads are going to be on Iceberg and Delta, I think Hudi and Pymon will handle the unique but small use cases. Or the functionality that we need will just make itself into Delta and Iceberg. Right. For Delta and Iceberg to sit there and say, we're not, we're not streaming capable, it doesn't fly.

Speaker B: Right?

Speaker C: We, we gotta do that. And as a conversation, that's a conversation up solver. Like before we join click and now with click, we're having in the community around Iceberg is like, how do we make Iceberg more streaming capable and bring that technology? What can we learn from Hudi? What can learn from Pi1? What can we learn from the other stuff that's going on to make Iceberg better? Just to create another format there? Ah, yes, great. But it just complicates life. But I think ultimately, if you look at these formats, at the end of the day, I, I judge it by the ecosystem. So if I look at, uh, the ecosystem today, there's a large majority of the primary popular engines. They just support Delta and Iceberg. They just support them both.

Speaker B: Right.

Speaker C: And they will continue adding support. So you get read and write, you get optimization, you get everything. It just, it takes time. But they'll eventually support all of them. Hudi and Paimon are falling behind a little bit, which in my opinion signals to the market there's less interest because these query engines are saying, my users, my customers aren't asking for this. Right. They're asking for Iceberg and Delta. And to me that's an indication there's a shift towards the top formats. But you still have these other formats. Ultimately, I expect these engines to just support them all. And if they converge, great. You just discontinue support for one or the other.

Speaker B: Wonderful.

Speaker A: Now I think things are on that topic of different table formats, right. I know there's an Apache project called XTable Now. And what are your thoughts on that, Roy? I think the whole idea was interoperability between different formats. I think that's how they originally started. I'm not closely following them these days, but thoughts on that in case you.

Speaker C: Yeah, I think the idea of XTable makes a lot of sense, but my feeling on that is that it's temporary. Right. Because ultimately we're saying right now there's multiple divergent formats, but in the fullness of time these formats will converge and will have some consolidation. So I think XTable is, is a nice solution for today. When you're saying I have two formats, I can't figure out which one I want. Let me just convert between them so I can support both until somebody tells me one is better than the other. Right? But ultimately somebody's gonna tell me one is better than the other or the company is going to decide we're going to pick one over the other and then I don't really need to convert between the, uh, between them. The other issue with conversion is that it adds complexity, right? It adds latency, it adds potential failure points. That at scale just is hard, right? If I have a few tens of tables, okay, fine. If those tables aren't updated very frequently, okay, fine. But if those tables are hitting a million events a minute and those are updates and deletes and I'm um, constantly updating stuff, converting that in real time just sucks, right? And it's more compute, more failure points. What about optimizations? What if you're not really converting everything because they're not converting the data, they're converting the metadata. Uh, would you convert the delete vectors? Would you lose that? Would you require me to merge the table before I can convert? It's fine, it works, but at scale it just doesn't anymore. And I think that's the line that I draw like when I bring this into a customer that has thousands of tables and these are tables coming from SAP and SQL Server, we're talking about billions, trillions of rows. It uh, just doesn't fly in my opinion. And what I've seen with my customers, they're like, yeah, we played with it, it works. But at our scale it just won't.

Speaker B: Uh, since you mentioned so many other things to be taken into account, that's one of my criticism about Lakehouse. Um, I feel like it's a very too low level abstraction to the developers that I should really know about. What is my file system architecture, which is the whole purpose of databases are very successful, is that the hierarchical database structure is not scalable. And here is an abstraction that's why nobody able to beat SQL so far yet. Now lacos again Trying to do the Hadoop era of. You need to understand what is going on internally in the file system. And the interesting comparison with, there's a paper called Google Nepal, which is an internal data warehouse system of. Google talks about, hey, as a consumer, you define what is your latency and what is the cost that you are going to spend for that. I will manage internally. Optimize. Do you think Lakehouse will ever get into this phase? Like what it takes to build that? Which is exactly what Snowflake actually does, to be honest.

Speaker A: Thanks for bringing that. Um, Snowflake is a perfect example of adoption. You drive users to the platform to embrace data in the first place, which is the ultimate mission. With open table formats and all the complexity, we go into the granular details a lot more. In most of these conversations, it gets into the weeds. Right. Compaction and all of that. It gets a little bit more tech that, uh, it may scare a, um, user who may come back and say, hey, why don't we go back to Oracle days? What are your thoughts on that?

Speaker C: Yeah, first of all, it's a great point. I would say as a data engineering community, we love to get in the weeds. We love the formats, we love the differences, we love the wars between the formats. Right. It's just what we do as tech people. When we bring this to enterprises, we bring this to our customers. That's not the conversation we're having. The conversation we're having is what are the problems we're trying to solve. That this Lake House approach, forget Iceberg or Delta, whatever, that's an implementation detail. But what is this architecture going to allow you to do? And how can it, um, unblock you, uh, for moving faster? And that's really the conversation we need to focus on. So when we were pitching this at App Solver, the way I was talking about it is shared storage. Like, I didn't talk about it as Iceberg or Lakehouse or Data Lake. We talked about shared storage. And the concept of shared storage in my mind makes a lot of sense. When I pitched this to our customers and prospects, it connected the dots for them. One of the biggest problems is that a customer isn't just going to buy one vendor. There's no one vendor wins. All right? There's snowflaking databricks in same accounts. There's databricks and dremio. There is, There's a combination of stuff. And to be agile and move fast, you need to be able to pick the next best thing and start running. So there's no. There's never going to be one vendor that can say I just have Snowflake, nothing else and then um, I'm happy in that case. None of this stuff matters to you. But in majority of cases they're going to have multiple. So the having that shared storage is the connecting factor. When I get build use architecture that you can store your data once in good quality, transform it once and then analyze it from any number of tools. Reducing cost, simplifying the complexity of your architecture, giving you flexibility and agility to jump into the next thing. If China releases another model and there's another architecture like Deep Six seven and I want to use that, I shouldn't say I got to wait for this vendor to add it for me or my data is stuck over here, I can't move it to this other tool. Right. So we wanted to create the shared storage experience for the user so they can do more with it. So we don't really have a conversation about the format because the format doesn't matter to them. We come in, we say we've investigated the market and we're a strong believer in Iceberg. We think it's a good format, we think it's, it meets all the needs of your business. Right. It integrates with the tools that you are using today. So that's one aspect of it. The other aspect of it is that you're talking about Snowflake or Google with their paper like saying how do we simplify this? How do we manage this for you? That is the other point about what us vendors can do on top of this ecosystem. To say there's a standard layer that we don't have to reinvent the wheel but what we are going to do is we're going to give you managed services on top of that will actually make it easier um, for you to, to have that warehouse like experience. And that's what Absolver was bringing us. What we're bringing into Qlik as well and what uh, I'm seeing the market is pushing for as well is don't just let Snowflake or Database do it for me. I want an independent solution that can give me a managed experience that cuts across all of my tools, doesn't just do a really good job for their own tool but does a really good job for all the tools that I'm going to use. That's ultimately what I want. Right. And ah, again I think that manage layer is a benefit. Yes. It feels like it's another thing I need to manage. But the fact that we have these standards Both Iceberg table format, Iceberg catalog format. Those standards allow us to create compatibility between the layers as opposed to what I call interoperability where we're just making it work. But there's a bunch of gotchas in there where compatibility is. I know when I plug them in they're just going to work, right? I'm going to get what I want because the standard defines those and both sides, uh, you know, following the standard now I have compatibility. So as a data engineer I know these two things plug together and they're

Speaker B: just going to work. You mentioned about very interesting, easy for data format so that any tool can come in and then give a bigger interoperability. Is that a reality though? So there is a couple of developments. One is S3 table for example made a lot of noise last month and they made an abstraction on top of the file system. Personally for me it's very appealing because I do believe that file system level abstraction, what S3 table abstraction is the right abstractions to do moving forward. And there is backslash support like Iceberg catalog support and all those things. So there is interoperability breaking thing. I don't know what's the right wrong strategy here. And the second is systems like databricks even though they don't support the iceberg but uh, they always prefer the native format which is the delta lake delta format. Same for Snowflake. Even Snowflake support Iceberg they're always preference for the native format. Their query engines are much more fine tuned towards the native format. So there is a pocket of vendors coming in with their own version of optimization that eventually impact the interoperability that what open table format that we're envisioning on that. So is that really going to be a reality or just our own dream?

Speaker C: I think it's going to happen. The question is to what extent. We have this standard table format layer that everybody knows how to use. You can think of it as a bunch of building blocks on um, how to interact with data in an object store in an efficient way. There's a definition of compaction. The iceberg spec for example or the delta spec or hoodie says compaction should be done a certain way. It's a recommendation, it's a suggestion. It doesn't tell you to do it that way. It just says this is one way to do it. Now under the hood the compaction uses the primitives, right? It uses the asset transactions, the, the snapshot commits and all that kind of stuff. Now I don't have to do it that way, if somebody says compact the data every five minutes, I can compact it every one minute. So ultimately what I think is going to happen is that you're going to have the standard building blocks and then vendors will build their managed services on top of that that are differentiated and potentially offer more features to the user. So in particular like up Solver, what we did was we took the uh, optimization engine that we built over the last seven plus years and we expose it via iceberg, uh, we implemented iceberg APIs and to the SDK and did the right thing, the standard thing. But the actual compaction and the actual optimizations is nothing that's defined in the iceberg spec. It's our own thing, right? We had our own algorithms, our own mechanisms of doing stuff. And I think that's going to start happening more and more. But that's a good thing because you can ultimately say like look, the, the subsolver engine is optimizing in this amazing way that is so much better than these other engines. I'm going to choose to use it. But the lakehouse architecture actually allow you to say turn off Snowflake Optimization, go switch over to absorber Optimization or click Optimization, use that or if you're not happy with that, turn that off and then turn on something else. So the standard, the specs we're working with will create decoupling. The user has the choice to say I like one versus the other. Let the vendors duel it out, right? Let the vendors come to the party and say I got the best optimizer, I got the best management capability. The user should be able to just turn one off, turn the add on and not have to move data around, recreate their catalogs, stop their pipelines. Like that should just be seamless. And I think we can do that if we stick to the standards of these table formats and using the same building blocks. And we've done this when we did some testing at ah, UP solver, like we turn it on with Snowflake, turn it off with Snowflake, then turn on the up solver optimizations and we just kept on going with the new data that's coming in now you can go back and fix the old data, but you can just leave it and keep doing the next thing. To your point about S3 tables, there are different pieces of this stack that you can optimize, right? So if we're simply optimizing the optimizer that uses the APIs to interact with the table, that's one thing. What S3 Tables did was change the underlying storage, uh, component to give you something different. Right. Instead of giving you a basic bucket that you just read and write stuff into, we're going to give you an abstraction bucket. You just throw an object into it and we'll figure out behind the scenes where to put it. Right. So that's great because that's also an optimization. It's a, it's a platform specific optimization because that issue exists with S3. It may not exist in ADLS or GCS. Uh, the APIs that you're using are exactly the same. S3 didn't break the put object, get object API. They just said here's a new API to create a uh, table bucket. That's not a config step, but Spark and Trino and all these engines know how to use get and put object already. So that just works now they get this extra benefit behind the scenes. Where things fall apart for me in that implementation is that I don't want another catalog stuck inside a, uh, persistence layer. I don't want an optimizer inside the persistent layer because at that point it's much harder for me to decouple myself from that. Right. The lake formation optimizer in my mind makes a lot of sense. Integrating with the glue data catalog in my mind makes a lot of sense. Right. AWS may have a different opinion, but I think if we stay true to the decoupled architecture, that's what I would expect from a solution.

Speaker A: Thanks. Thanks for that. Uh, this has been hogging my mind to ask you since the time I met you. Now with Databricks techmeter owning Tabular and Delta, uh, do you think they would bring. I need to be careful phrasing this because it's bubbling and I work in the partnership team. Based on your feedback on what you've seen in the roadmap for both of these stable formats, do you think that there will be a parity among these formats and the interoperability is no longer a question? Is that how we envision that journey? Do we?

Speaker C: So I can tell you from following the mailing lists, the delta and the iceberg mailing list. This is public mail list. Anybody can follow the threads there of conversation from that perspective. I think there are two aspects. There's the metadata aspect, like how do I manage tables through this metadata and the second aspect is the uh, physical data, like how do I store data, how do I store deletes, how do I store that kind of information? There's a lot of discussions in the community around unifying the data layer. So we're encoding deletes the same way. We're writing data into parquet the same way. If we do come to an agreement on this, and I think over time we will, because there's really no big reason to, to have two separate ways of doing it. Then the data encoding unifies between the formats, between Iceberg and Delta. The metadata layer still remains separate, enabling each of these communities to innovate, uh, and move in directions that make sense for them. There's probably going to be a lot of overlap because at the end of the day we're doing the same things. But Delta would do certain things for schema evolution and partition evolution. Iceberg will do something different that adds another feature. This is great because it doesn' kill innovation, right? It doesn't kill the communities. It doesn't say, yeah, you guys are the same as this other guy just killed this community. I don't think we want to do that. I think we want to be able to enrich and support those communities to innovate on their own way. But ultimately the data, the way we encode and store data and information is the same.

Speaker B: Right?

Speaker C: Uh, so the user will have the option to say, go from this format to this format back and forth. But the data never changes. They don't have to rewrite their data or copy their data and, uh, cost a lot of money. So ultimately, I think that's where we're going to go. And for engines like Absolver and Click and Snowflake and Databricks, that becomes a lot easier. I don't have to rewrite data. I don't have to deal with the heavy stuff. I can just change the metadata and be much more interoperable. And I think that benefits everybody longer term.

Speaker A: Got it. Thanks, Roy, for that. Let's shift gears a little bit outside of table formats for now in terms of lessons learned. Literally pulling your brains from an enterprise architecture perspective from projects that you've seen fail and what are the things that they did not do in the right, uh, step while implementing an overall Lake House project. We can take one example, let's say databricks or anything, if you can walk us through what do you think people should focus on?

Speaker B: Beautiful.

Speaker C: Yeah, it's a big topic. I think generally when people approach Lakehouse, like I said in the beginning of the conversation, you can approach it in two ways. You can say, I want to modernize my architecture and go, go wholesale into Lakehouse, I want to rebuild, or you can say, I want to enhance my existing platform by extending it with the lakehouse, where I've seen the challenges come in is when you build net new. Like when you say, hey, I want to go build a lakehouse architecture from scratch, I want to start integrating my existing tools into it. That's where people start to run into a little bit of a challenge. It's not because of the format, like Iceberg or any of that stuff. It's really about if you lived in a warehouse world, to do things in a data lake, slash, lakehouse world is just different. Right. The same tools don't work the same way. You got to think about it differently. There's more components to think about. So I think that's where I see some of the challenges, is just simply understanding what is involved in building a lakehouse end to end and then having the skills, having the experience, having the people on your staff to be able to deliver on this quickly and efficiently and also not treat yourself in a foot by building in a certain way and then not being able to scale six and six months from now. So I think that's where I see most of the challenges. When you're saying, hey, I already have Snowflake, I just want to move some data to Iceberg. It's pretty straightforward. Look at your ingest vendor. Say you already ingest data, uh, into Snowflake. Just write those tables into Iceberg and then I can create those tables in Snowflake. It's a little bit less to think about because one vendor gives you a lot of features out of the box. The other sort of confusing point is once you move into the lakehouse now, you have to make a lot of tool decisions. Iceberg, Delta, how do you write the data? Do you still use the same ingestion vendor that you have today? If it's Fivetran, for example, like, it doesn't really work well in a data lake. Right. Do you use Qlik? Do you use. I, uh, don't know, whoever, right. To write data into the lake? Is it good enough for the pipelines that you have? We worked a lot with customers who were ingesting with FiveTran directly into Snowflake and then they wanted to re architect and write into the lake and inquiry with Snowflake. And very quickly they realized that the tools they were using for ingestion just didn't work as they start to scale out and start writing into the lake. Because writing into an object store is very different than writing into Snowflake, for example. Right. And especially when you have to do cdc, right. With merging changes and deletes and all that kind of stuff. And then as the volume increases it gets harder and harder. So all of a sudden you realize the tools you used to use just don't work anymore and you got to go into different tools. What I've seen also the tools that are lake native, they don't speak the same language from a marketing and documentation perspective as the ELT tools. So the ETL tools and ELT tools, technically they're the same thing if you think about it. But they speak two different languages and that's really confusing to people who are going from the warehouse/elt world to the lake slash ETL world. They just don't understand the language. When you expand something in the documentation, like I don't grasp that. It's different to me. Right. It's foreign to me. So there's uh, we. I spend personally a lot of time just, just mapping terms and explaining. This is the same thing. Instead of loading here, we're loading here and we're transforming it a little bit differently. There's an education process here. Five, six years ago there was a massive move to the warehouse. Snowflake came out every. It was a cool kid on the block, everybody moved over and it was a well known pattern that traditional data warehouse, uh, engineers from an Oracle just understood this. They knew it. So moving to Snowflake and the ELT model, there wasn't a lot of learning. Right now when we're shifting into the lake, those users don't understand the ETL and, and lake model. So we have to teach them. There's a very small number of engineers who are really well versed in data lakes and etl. You don't have to teach them, but you do have to teach the masses saying hey, I'm in Snowflake, I want to go do this Iceberg thing or Oakland Lakehouse, help me figure this out. Right. So that's why I think there's a little pain in the market because there's a new set of users who weren't used to this model and are saying like, how do I do this? What does this stuff mean? It's not the same as the stuff over there. Right. So we have to do a lot of that teaching.

Speaker B: Um, yeah, I don't know about functional data engineering in data engineering with the newsletter by far that is the most viewed article in the newsletter there. Yeah. One of the thing is very critical for success of Lakehouse implementation is the catalogs. You also mentioned the importance of having this separation of data storage layer and the catalog layer. The catalog layer itself is very fragmented. There's a, there's a notion of like technical catalogs, like loo and there is the business catalog version of it, which is all the other vendors and all those people. Now the syncing between them and making this IT working itself is a huge task. Now I have two questions. There, there is another architecture style being proposed like catalog of catalog. Like you'll have over catalog of everything we're saying as part of it. And this particular architecture style became very important, especially the rapid evolution of AI. Like people wanted to understand the query, they wanted to understand the insights very quickly. Which means they need rich metadata information available for them to access. And many of the traditional enterprise not able to access it because of not enough proper architecture on how to do that.

Speaker C: Yeah.

Speaker B: My question is what is the best architecture's time? If a company has to adopt catalog, do they consider about like a catalog of catalog, do they still need to consider this the version of technical catalog or uh, business catalog or uh. Do you see there's like only one pro version exist?

Speaker C: Yeah. It's funny because I wrote a lot about catalogs in my, on my blog on substack. Uh, Roy on data, if you want to check it out. The terminology is overlapping. Right. Business catalog, technical catalog, operational catalog, observability. It's like a catalog with metrics layered on top of that. And a lot of tools themselves have catalogs built into them like databricks and Snowflake. You got to build some catalog into it. But how functional it is and in what way that's what differentiates between those things. Now I think the conversation around, around catalogs really came to the forefront because of the lakehouse architecture. Because before that it was built into the tool that you used. And that's it. There were business catalogs that kind of layered on top, but they were. There's like passive catalogs. You just look at them, you don't really do much with them. They don't do much besides give you additional information. They weren't usable from the perspective of driving adoption of data or accessing data. Then there's the technical catalogs which are primarily focused on driving uh, actionable connections like a query engine need to look up information so they can execute a query. Right. But users don't necessarily interact with those catalogs because they're not really user friendly. With the lakehouse, we're saying, hey, we're moving a lot of our business users to access data in the lake. We can't really use the warehouse catalog anymore. We need this other catalog. And this other catalog is now becoming bigger and stronger and has more responsibilities. Let's start to converge those functionalities into a uber catalog, right? That's going to be responsible for all of my data, um, because ultimately the lake is going to represent all of your data. So I think that's where we're moving, and that's how the conversation has started. Now, the beauty of iceberg is it also introduces an iceberg rest catalog specification. There's not a lot of magic to it. It's just basically, uh, an arrest spec and some definitions of how to operate. That says, here is a standard way for you to interact with a catalog when you're trying to read, write, update iceberg tables in the lake. And the nice thing about it is that it decoupled itself from all the traditional hadoop dependencies that we grew up with in big data. So before the iceberg rest catalog, there was a hive meta store and that had all those dependencies. Nobody innovated on it for years because it's just too much. But now with iceberg rest, it's a web service, right? I can run it on a lambda function. I can run my local machine. It's really that easy. So now you're starting to see an explosion of vendors coming out and saying, I can build a catalog too. Right. The spec tells me how to do it. It gives me all the instructions. Now I can do that now. Now we're seeing, uh, a plethora of catalog vendors coming in and saying, I can do this too. Each one of them is trying to differentiate in different ways. Right. Playkeeper is a new catalog company based on rust. Really good when it comes to security and access controls. That's where their specialty is, and they're growing from there. Polaris catalog from snowflake came out as another reference design for an iceberg race catalog. They got an amazing roadmap, but it's slowly evolving. Then you have unity catalog from databricks. Comes with all of their sort of experiences and capabilities. Open source, but also available to databrace customers. Right. And then you have data hub that recently released the data hub open source catalog, which is a really cool one. I'm, um, excited about that one because it brings iceberg business catalog and operational catalog all into one. And that's, in my opinion, where the future is going to go. So each one is attacking the problem from different angles. But eventually, like Angie jassy used to say, in the fullness of time, they're all going to be comparable. They're all going to do the same thing, right? Uh, and then it's going to come down to who is the better price, who's the better scale, who has the features that I want? So that's one answer to your question. The other answer is these catalogs are probably going to be 10% used by users, 90% used by machines and AI. And that is where things get really exciting. That's an area of innovation that if you leave the catalog to stay inside of a warehouse like Snowflake or Redshift, you're not going to get that innovation. You're going to maybe get it in 10 years. The fact that we're extracting the catalog component out of those engines, we're giving it a whole new life. And that's where things get really excited. Because ultimately if you look at agentic workflows like AI agents and stuff like that, eventually they have to figure out where is the stuff, where is this data that I need to access, where are the functions that I want to call the tool use inside of LLMs, where other agents that I can connect to. That catalog now becomes the central point for your data, your analytics, your AI and all of your workflows. To be able to say, I am the one that knows where things are. I have context, I have the ability to tell you this data set actually has stuff relevant to the prompt that the user provided. You should go query this. And here's a tool, by the way, here's a function that you can call that will execute a query against that table and, and bring the data. So I think that's where a lot of the M innovation is going to come probably in the next couple years. And so that's what excites me about the catalog itself. What we're seeing today is just table stakes, like we got to do it, let's do what we need to do just to get out of the old school catalogs, get us working, get me cataloging, make sure my query engines work with it. Give me RBAC and security, Give uh, me a little lineage. Like that's great. That's table stakes in my mind. Fast forward a year, two years from now, it's going to be all about AI context, uh, aware routing and fine discovery of information.

Speaker B: Yeah, I have one interesting. It's great that you mentioned about 90% of the workload going to be machine learable. And that is where my concern or criticism on the current version of catalog. You also mentioned the importance of separation of those catalog and the data. A critical part of An AI to be successful is that the data and the context has to be stored together and has to be up to date. Historically we've been lagging a lot like data catalogs always outdated. And one of the reason why these business users won't use data catalog is and the passing usage is likely just useless after six months.

Speaker C: Yeah.

Speaker B: So is that really from an architectural perspective, is that really make sense to have a different catalog version or just like how do we make it reality that the, the storage data and the content stay together up to date? It always gives a reflect the latest information.

Speaker C: So great point. I think the first way to look at it is not from the lens of today, but from the lens of where could it be Right. I think the issue with the business catalogs today is that a lot of it is driven by user manual entry. Users have to put in the taxonomies, user have to update the context. Right. They can, they can crawl, they can extract some stuff, they can use some AI magic to make stuff out. But at the end of the day like it's not really great and like you said, it goes stale very quickly. So I think that's one aspect. You have to look at it like where do we want to be and how do we want to do this better data? Uh, Hub actually has a really good model for this where they have active crawling of data sets. They actively go out and find data sets and crawl them and extract information out of them to be able to build this context. Now over time users can enrich it if they wanted to, or you can layer AI on top of it. That will enrich it in different ways, but it's an active crawling that constantly updates the information. Another technical lever here is in the iceberg rest catalog there's uh, something called a uh, metrics endpoint. And what it does is it allows the query engine to report information about the queries they just ran. Right. So if I'm running a ah, select star from table Users where user equals 5, the query engine, when it completes it sends a report back to the iceberg catalog and say I just ran this query, I read x number of files, blah blah blah. It's a bunch of information and we're going to keep adding more metrics to that. Now you have also an active feedback loop that says users are querying this table and they're filtering it on this particular column. Or there's a lot of information that I can teach us about the usage of these data sets as we enrich the catalog with more feedback loops from different places. I think it addresses the concerns that you bring up that these catalogs becoming stodgy and old and outdated because we depend on users to do it. The other aspect is as AI agents use it within context, they can also further enrich the catalog and say for this particular question from the user, I had to use this data set. The catalog becomes almost like a vector store. Like you, you, you go in there and the agents are searching through the catalog to find where the data is and then uh, you can extract the data and the catalog ultimately becomes that smart vector store, that smart search engine for data.

Speaker B: Yeah. I think one concern for me from an architecture perspective is that the crawling is there for a long time. You're trying to understand what it is and I don't put enough context for you to really understand that. 1. One of the thing I really liked about STF, like what recently occurred by DDT in the pipeline itself, they carry all the contact. Let's say if you're defining a PI information in a pipeline, it's forward propagation, right? This is catalog trying to understand backwards, right? Yeah, yeah. I feel like that is the right way to query the context because as you create the context, you just need to carry as folding long line and uh, catalog. I wish catalog be successful, but I'm very pessimistic about, pessimistic about the back propagation architecture completely.

Speaker C: I agree. I understand what you're saying and I think that makes sense. But I think there's two challenges with that approach. The first one is it still depends on a user to put that context in. If I didn't put the right information into my model, it's not there.

Speaker B: Right.

Speaker C: Number one, um, number two is that you're carrying context around, which is great, but at some point it's going to become heavy. Right. So this context just keeps get moved around. I add more context as I move throughout the process. Right. That ultimately just becomes heavier and heavier, latency increases, cost of memory and storage increases. Because you're carrying this around, right? And we've seen this in, in networking, we've seen this in, in other models where a little bit of context is okay, but a lot of context is not okay. Right. Um, so at some point you're going to need to have some sort of agent or, or central place to store out this context. Now it doesn't have to be a single catalog, right. It can be a main catalog and a bunch of little catalogs where there's synchronization between them. But ultimately we're routing information, right? You also have the idea of LLMs with millions of tokens in context. They can carry a lot of context as well but ultimately you're carrying context that belongs to your enterprise. Now if in your SDF model you're pushing some context, the catalog can learn from that as well. Right. And update the catalog too. So in the future you don't have to keep layering more context and more context in order to keep the history of your context. So again I want the models to work. I think the catalog will definitely we'll see it in the next two years. I think you'll see a lot of innovation around this. There's different ways to store and share context and we'll, we'll probably do multiples of those. It doesn't have to be one or the other.

Speaker A: Yeah I think catalog was still talking structured anand uh I think still doing structured in catalogs today and not a lot of unstructured has entered uh into the pipeline yet.

Speaker B: Seungji I think there's one way of doing that one solo recently got acquired by Qlik and there is like a, one of the things that I noticed like there's a three different world of orchestration data like there's a qlik of the world or like a nice GUI interface. If I know what I'm doing I can quickly go and drop and I don't need to have much of my technical knowledge around that one. Greatly liberating for certain class of users. And then there's AppFlow kind of a model like in a code first approach. Don't even touch the UI side of it. It's a serious sin to even do this code first approach. And then I also seen there are a lot of boilerplate code being written and many organization and having a platform team trying to build a YAML abstraction and say okay, you define simple steps here uh and AWS step function also in a very similar way of an approved one.

Speaker C: Right.

Speaker B: Uh, what is your thought on what is a true way to build a pipeline is a non existent.

Speaker C: I don't think there's a true way to build it. People have preferred ways to build it. The way I look at this problem is I break it up into three pieces. Right. The first part is a top level the authoring. I may be a uh code first person, you may be a uh GUI first person doesn't really matter right. There's different tools, different ways to do that. The first layer I need to think about is the authoring. The second layer I need to think about is the maintenance of these pipelines. If a pipeline breaks or something changes, how do I fix it? How do I port my pipeline from my dev environment to my production environment? There's like the maintenance of those pipelines and then the third layer is the execution of the pipeline. Now I need the thing to run and to process data and what compute cluster is it going to run on, et cetera. In that paradigm from an authoring perspective I should be able to offer users options, right? If my primary user is a GUI based user, great, I'll offer that. That's what Qlik offers today. If we want to move towards more of uh, a uh code first data engineer, we need to be be able to offer them a way to write code or even integrate into their VS code or whatever tool they use to write code. Right. The second layer around the maintenance. Now we come back to how are these pipelines really defined from a Qlik perspective? It's a GUI is that get translated into uh, a uh code that's readable and accessible and maintainable by the user? Today? No, in the future could we possibly. Right, could we take these pipelines, translate them into SQL, into YAML and then either expose it to the user and say hey user, tell me what GitHub you want me to put this into and then I will put it. And every time you change the GUI pipeline I translate that into the YAML and then I update it. Uh, I do a pull request into Git and done. Right now my code is managed through Qlik, but as a user you have visibility and access and you can always take it and go somewhere else. If you wanted to go and rebuild this pipeline in Spark you have the definition right, you know exactly what's happening and you can go and rebuild it. It's work to do but you can do it. So I think that's an important layer that looking into the future we need to be able to give our customers an option to say I want to maintain my pipelines developed inside of Qlik in a standard language outside of the Qlik environment. Now you can ask is it YAML, is it SQL, is it templates? It's hard to tell. I think DBT created a uh, certain model that people are used to templates plus SQL in the same place. The infrastructure world, like the platform infrastructure teams settled on YAML. Software engineers like their own thing. It's code. I want to put the code into gitl. I don't need YAML per se for to define my code. So I think different orgs are pulling us in different directions. I, uh, think ultimately it's a combination, right? Because code is a really good way of defining business logic. It can be SQL, it can be a Python function, it can be a rust function, its code. But then we need to be able to define configurations for this pipeline and putting that in some template. Pointless, right? Because it's one more way to define configuration. I already know how to do it in YAML. A million tools already know how to use YAML to do code analysis or security validation or uh, that kind of stuff. So in my mind, if I was to design this on the fly, I would say create two artifacts, create a YAML artifact that has the definition of my pipeline, all the configuration cluster sizes, blah, blah, blah, all that stuff. And then I have another set of, uh, artifacts that are code artifacts, right? Python functions, SQL blocks, and they're linked into YAML. You can take that Python function, test it, change it, do whatever you want. But my YAML definition says this is how you run the stuff and that's how I would manage these pipelines. Now, the third layer on the execution side, now you have the option, right? If I build my pipeline inside of Qlik, I can let Qlik run it on Qlik's compute infrastructure. I can push that down into Snowflake or Databricks or Redshift, or I can offload that and say, let's convert this model into a, uh, DBT model and give it to the user and let the user run the DBT model on their own Kubernetes cluster using airflow or whatever infrastructure. They decided that's an option for me as a vendor to say I can give you those options. I think ultimately where the industry is moving towards is that flexibility, right? We see a lot of push towards bring your own compute for a really big reason, right? I m mean it's cost savings and flexibility. So as a tool vendor, for example, if I'm representing that product vision, I should be able to say, yes, Mr. User, I can give you the option to run it on top of the Qlik compute. I can make it cheaper there. I can give faster compute, I can give you a lot of value. Or you can bring your own compute, right? Uh, and you can run your own Kubernetes. You got to deal with it. Again, I'm not speaking for Click, I'm just using that as an example. But I think ultimately that's where we're going to be and that's how I break up the thought process on this. You can do one of these layers, you can do all three, you can

Speaker B: do none of them.

Speaker C: It's really up to the vendor. How do they want to do it? Yeah, GUI versus Core.

Speaker A: I think that debate will never get over. Right. Like ETL versus elt. I think it's the DNA of the company. DNA of the tech leader, uh, DNA of the developer also factors in. But I think abstraction drives use cases. I think that coming from the field, that's what I've always seen.

Speaker B: Right.

Speaker A: And I think the more abstraction, and that abstraction could be a, uh, jitter template as well. Meaning it's totally fine. Abstraction is what we need more in the data engineering space. GUI is here to stay is what I believe in. But I think we got to embrace a code first approach also. For things that cannot be done in the gui.

Speaker C: Yeah, yeah. For enterprises, having tooling that supports both GUI and code brings different sets of users. Right. If you're trying to hire data engineers today, you're probably going to be easier for you to hire people more comfortable writing SQL and Python in VS code as opposed to click, drag and drop and that kind of stuff. Right. So you may be able to attract them more. I think companies should be able to offer both to, to their end users and it's on us vendors to deliver the tools to do it. Now, should you have 10 different tools, a GUI tool, a, uh, code tool. I think we want to see some consolidation. Just give me one tool that offers multiple authoring options and I can choose. Right? And then under the covers I can do whatever I want. I'll give you an example where I, uh, worked with a customer. Was very, just ugly, right? They didn't want to touch code, but they had a requirement from it that everything they built within the data infrastructure has to be versioned in GitHub. Right. So they didn't want to start learning how to write code just so they can check that code into GitHub. Right. They ask, hey, we want to keep the UI, but how do we translate that UI into some code that we can put into GitHub and tell our CIO that we did that? Right. You run into those situations where it's a mandate and you got to do it, but you don't want to affect the productivity of your users by making them all learn how to write Python. Now breaking up this problem is important and it's up to us as vendors to figure out which one we're going to do first and how we're going to do it. Um, but ultimately it's giving our customers choice.

Speaker B: Yeah, you seem to be fan of open source development and OpenStream. So in your opinion, what are the open source projects that you are tracking that you see in a year or two down the line with how huge potential?

Speaker C: So there's a lot. I tend to talk about them on LinkedIn as well. We talked about a couple already from a catalog side. Lake Keeper and Data Hub are open source catalogs in the iceberg space that are doing some really cool stuff. Lake Keeper around security and access controls. Something a little bit more unique like stepping out of RBAC and into Open F G A which is an open standard of creating access control through relationships, something that Google has developed. I think it has a lot of potential to really revolutionize the way that we do access controls on top of Data Hub. Like I said, it's an open source catalog project that brings a lot of things together already. So it already had the business catalog, it already had the technical catalog, it already had operational catalog. Now it's just adding an iceberg REST interface. The thing that excites me, uh, about that is what can we do with that from an AI perspective. AI agents and stuff like that. So I'm looking into that a lot more. There's another project called Page AI P A I G AI that some folks, you may know them, uh, out of the Apache Ranger community started. But that's actually a really cool technology that focuses on access M controls and governance in AI. So if your LLM is saying I want access to this record, who's going to authorize the LLM to access that record? Who's going to tell it, no, you don't have access to our record. You can only have access to that record.

Speaker A: Right.

Speaker C: So I think that's a really interesting problem that we're like glossing over. We're just saying agents get whatever they want. They're trying to solve that problem. So that's really interesting for me and something that I'm keeping an eye on, I think that's the top ones right now. There's going to be others as well, but that's the big ones. And of course the other ones like Iceberg and the Iceberg REST projects, I can go all day with this stuff.

Speaker A: So I think a lot of topics are long term. We need more time to do that. But I think, um, just in the interest of time, one final question. As most of people who read you, uh, are data engineers and architects, uh, in an AI world, um, how do you see the whole data engineer role evolve and what do you think people should do to stay Relevant?

Speaker C: Yeah.

Speaker A: Other than something, yeah.

Speaker C: To say this, in short, it's tough. I wrote about this last year. Data science was a very hot topic a couple years ago, but everybody realized that there's very few data scientists. Those are typically PhDs, MLAPs, data engineers, or just playing with models. Longer term, we're going through some fragmentation in the industry. The advent of AI and copilot will ultimately make it easier for more people to do more with data. Building pipelines won't really be a thing, it'll just be working with data. Right. Doing something with data. The fact that you have to ingest from here, move to here and clean and transform is going to be built in. It's just going to work. So there may be some people who are still dealing with this at larger companies because it's a much bigger scale and it's actually a hard problem. But for probably 80% of the industry, I don't think that's going to be really where engineers are going to focus their time. So I think the data engineering is going to split up. You're going to see folks moving towards platform engineering, building the platforms, building the RAG systems, building the infrastructure that you need to deliver data and AI and working with the infrastructure, right, the computer, the servers, all that kind of stuff and building sort of tooling, building capabilities for the end, for the engineers, the software engineers to build on top of these platforms that these platform engineers are building. And then the other set of users will ultimately become more software engineers as opposed to specific data engineers that they'll be software engineers who are data aware or have data experience. They can write SQL, they can write Python, they can write whatever, but they're going to be using data in, in, in ways that uh, are going to help them build services, products, capabilities and features. For lack of a better term build value for the business. Ultimately where we, in my opinion we're going to see and have platform team or just setting up the platform, giving tools to the rest of the users and then we're going to have a whole lot more users who are building with data and adding value to the business. That data engineer in the middle, building pipelines, I think ultimately would just be relegated to like super large companies like Airbnb and Meta M, uh, and AWS and all these guys that have very unique requirements and a massive scale that you need dedicated teams to manage those things. But in majority of cases I don't think you really need them anymore.

Speaker B: Wonderful.

Speaker A: I think that's a great way to end it. Data engineers need to step up. That's a good point. In some of our earlier sessions as well, we've discussed about this a software, uh, LED data engineer. Engineers are very close to software right now. A majority of people who claim as data engineers, including myself, we drive analytical workloads. We build to load data into the warehouse or team today. I think that role would definitely evolve, I think. Thanks, Roy, for your time today. We'll see you very soon in one of your other sessions.

Speaker C: Thanks for having me, guys. I really appreciate it. I'm a big fan of the newsletter, big fan of you guys and the podcast. Hope the, uh, the audience would love this session as well.

Speaker B: Thank you so much.

Speaker C: M. Thanks, guys.