Is Apache Iceberg the New Hadoop? Navigating the Complexities of Modern Data Lakehouses

Speaker A: Hello everyone. Welcome to another episode of Data Engineering Weekly. We are starting this series by picking some interesting articles and have a deeper conversation with the authors. One of the articles that caught my eyes, I think many of the folks of industry got little surprised with this article essentially called Apache Iceberg, the Hadoop of the modern data stack. We are going to discuss more on that. We have the author of the article, Danny with us. Hey Danny, how about introduce yourself to the audience?

Speaker B: Yeah, for sure. Uh, hi and thank you for having me. Uh, it's an honor to be here and uh, excited to talk about the article, uh, in person to unwrap the thoughts a little bit, uh, because a lot of people uh, messaged me about it. I seem to have struck a few nerves. But that's all right, uh, for an article like this, I think so, yeah. Happy to talk about it a bit more. And as for myself. So my name is Danny. I am currently the head of marketing at Estuary. I have a background in data engineering. I used to work as a data engineer for a decade pretty much or a little bit more at this point I think I keep losing track of time. But yeah, so I worked really in small startups, larger enterprises, and most recently before switching into a more marketing oriented role, I did five years in consulting in data engineering consulting as well. So I've seen a lot of projects small and big and that's what I usually use as the base for my articles.

Speaker A: Okay, awesome. One of the things that you compare in this article is about Iceberg and Hadoop. Some of the audience might not be very familiar with the hype cycle of Hadoop. They might have missed it. Give a brief overview about Hadoop and then the high frame about that, like how it became a, uh, dominant factor and more of the problem they solved.

Speaker B: Yeah, sure. So this is one of the things that some people misinterpreted in the Origin article. Um, my comparison between the two technologies were not really meant on the like strictly technical level. It's more about uh, the implementation and the culture and the buzz around the two. But there are also some parallels. The technological aspect and how, what problem they solve and how they solve it. There can be some uh, parallels around there as well. But at the end of the day there are definitely different technologies as they should be. So it's not necessarily an apples to apples comparison. So in terms of the parallels, what I see is that Hadoop came in, Hadoop became popular in the middle 2000s, right somewhere around that time and it solved very similar pain points that a lot of organizations face currently it's scaling cost speed of data insights and got uh, rid of data silos. I think these are the four big ones that existed back then and I don't know if surprisingly or not, but they still exist today. So there's always going to be a new technology that tries to solve them. Yeah, but Hadoop and uh, Iceberg are very similar in how they want to solve this.

Speaker A: Okay. One of the things that you mentioned about is very interesting when you're comparing two parallel uh, technologies, Hadoop and Iceberg is the promise of technology is there but the operational complexity is pretty high. And when the organization quickly jump onto this one, they got into the middle of this operational complexities and they're not able to get it as one. There's a lot of famous reports coming from all the way from Gardner and everyone. Like 80% of the Hadoop project fails. Hadoop, as in as a larger computing framework where you compare it to Iceberg, it's largely played around like an open table format and a table format like a very small space of it. How do you see the operational complexity in Icebergs, uh, emerging?

Speaker B: So the way I see it is that uh, by itself an open table format is not very useful. Right. So you need the ecosystem around it to actually utilize the opentable format itself. And without or with that ecosystem it becomes a bit more easier to compare with Hadoop and its ecosystem which was a bit more complete. But like just the components by themselves in either case were, were not very useful. So when I talk about Apache Iceberg, I usually also talk about uh, the catalogs or some kind of compute engine, some kind of maintenance process because without those, uh, it wouldn't make sense to use it in any production setting.

Speaker A: One of the challenges that you also mentioned about is the open table format. And we see this fragmented infrastructure kind of built on top of Iceberg all the way from a catalog and other aspect of it. And in your opinion, if an organization is trying to adopt Iceberg, what are the key factors they should consider before they jump onto? They are not going to fall into this trap. Open table format and all the problems I just mentioned about open table format itself is not enough. So what are the things that you want an organization to be successful adopting this?

Speaker B: Yeah, so there's, I think there's way more things to think about than the average uh, data engineer might think about. You have to think about automation of infrastructure as code monitoring, observability, data governance and the organization's operational maturity. In terms of things that uh, are pretty much hard requirements. But you also have to think about performance versus cost when you're evaluating even certain components inside an iceberg based data stack or compatibility with existing infrastructure and even cloud vendor lock in is a very important topic as well. Just to expand on these a little bit, uh, what I usually uh, tell. So I do help some data engineers evaluate uh, certain technologies. Not necessarily from the perspective of Estuary, the company I work for, but I do have a wide network of engineers that I work for, mainly consultants whose job is to evaluate and find a perfect solution for their client. And they obviously touch on a lot of things. They try to find the best fitting combination of components for that certain use case. So I think the first and the most important is to actually define your use case perfectly. Do you need more performance or less cost? What kind of data or data governance or security requirements your organization has? Because these kinds of parameters uh, influence if you're going to choose uh, I don't know, AWS or Google Cloud Storage or MINIO as your storage layer, they're going to influence if you going to use Trino OR PRESTO or DuckDB as your compute layer and also the data ingestion uh, part as well.

Speaker A: Okay, yeah, that's very interesting. Um, one of the first problem that you highlighted in the article is about small file problems. Small file is being there for all the way from the Hadoop world. How you think Iceberg intentionally created small file problem or how one should be aware of small file problems in the adopting Iceberg kind of a format.

Speaker B: Yes. So for those viewers who don't know what a small file, what the small file problem is, to summarize it, the overhead spent by the compute or the query engine trying to read all of the files when they are collecting all the data required for calculating the results of a query. And obviously the more smaller files you have, the more files you have to open and close, which is a uh, very fairly expensive operation for the system and it can quickly get out of hand. And the problem itself goes hand in hand with how data is being written into the data lake because that's where these files actually get created. A lot of data ingestion tools create small files because they want to serialize incremental data loads faster and they want to leave the cleanup of those small files to a later step because they for some technical reason usually they can't do it. So it's a two part problem and the way uh, Hadoop solved this that you would have to uh, either tune a few knobs on the data ingestion side or have a completely separate process and run every m or every day whatever on some kind of cadence to actually go through all the files and do some kind of cleanup. So uh, Iceberg takes a similar approach. Um it has this notion of um table maintenance which mainly the most important aspect of table maintenance is how it can compact these small uh files essentially merging them into larger files. So a lot of. A lot of data ingestion tools uh don't support this compaction operation. They essentially just append uh new records to Iceberg tables as they come in because the compaction operation is very computationally heavy. So a lot of vendors don't want to deal with implementing that. You would probably need a spark cluster to do it properly. So it's like a lot of operational overhead that is uh. Just not easy to implement. So once you have those, if you do have that kind of infrastructure that you do have a spark cluster available then you can uh schedule these uh maintenance operations yourself. Or there are some bundled tools or vendors that actually do this for you. But it's one of the more complicated uh features to implement.

Speaker A: Yeah. Secondly that is you touched upon in the article also about the metadata overhead in your seeing like seeing with various customers how organizations are usually setting up the stable maintenance do they the weightage between the bioglasses built and what is the complexity. They should an organization just should aware of it and solving those problems.

Speaker B: Uh so in my experience uh, based on recent some conversations with organizations who want to implement Iceberg or are implementing Iceberg or already have an iceberg based data stack going. This is something that nobody wants to deal with. If they can outsource it to some external system they will. Nobody wants to to tune these configuration settings and uh schedule these processes that comb through all the smart like it's. There's no business value in that. And it is one of the bigger blocker for iceberg adoption in my mind M the other being catalogs. But we can talk about it later. But this need for maintenance and the fact that it's like it is disconnected from the table format itself because obviously it's uh a compute related operation. But one without the other is as I mentioned there is just not a production release system. I think a lot of people who want to use Iceberg don't necessarily want to use Spark which is currently the best to run this kind of maintenance things and becomes a big blocker in their mind. So I think a very important next step uh in iceberg adoption and the ecosystem would be to create more developer friendly ways to handle these operations. Even organizations that are not like spark based data teams will be able to use Iceberg to its full extent.

Speaker A: Since we're talking about Iceberg, right? Iceberg is one of the table format becoming um, very popular and a wider option. There are other uh, systems like Hudi for example and Delta Lake for example. There is an Iceberg maturity in the developer ecosystem emerging. There are other two formats also available in this market. What do you think these two formats are lagging and what do you see? Do you think that Iceberg will continue growing the momentum? Um, that what we have right now,

Speaker B: I think the biggest thing that Hudi and not necessarily the Apple Lake lacks is a big vendor supporting them as much as they can be open source. These are still very complex technologies and it's very hard to maintain them and um, implement new features. So the fact that Iceberg has not just community support but huge vendors as well behind them who can dedicate like a lot of engineering hours to improving the open source standard or open source implementation, I think that's pretty much what currently looks like. It's making Iceberg win. But at the same time Data Lake is there which is also open source. But I'm not sure how much contributions dev bricks actually allows into their repository. So I think that's the biggest one in terms of like technology, uh, and features. The way I see it is 99% of use cases or maybe 90% of use cases could be handled probably by either of the three. Okay, so there's not many organizations who have that specific need that can all be solved by Iceberg. There's a few, There are a few of course and they will probably be a little bit more bias in their decision. But for most, most other players it's about support and ecosystem integration.

Speaker A: Okay, so in your opinion for any organization it's if you pick any, if you randomly pick any table format, 90% of the use cases probably would have been done because of the feature gap is like very less but depends upon the vendor support and the maintenance support.

Speaker B: And in every, most organizations already have tens of tools in their ecosystem and if they are looking for a uh, table format to build out their data Lakehouse, it has to be compatible with the rest. Like maybe they have partnerships with uh, a query engine vendor and if they don't support uh, the table format that they won't be able to use it. And these are all factors in an enterprise, if you're building from scratch, then yeah, it's a bit Easier. But if you already have 10 or 20 or 30 things that has to be able to connect to that uh, certain technology, then yeah, you can very quickly eliminate the ones that are not compatible.

Speaker A: Speaking of the runtime that brings the ecosystem of Iceberg six months down, before we had this huge all in our war about catalog, we should call it a catalog war. But to the audience, like how does catalog plays a critical role in Iceberg, uh, and why vendors are like going after the catalogs a lot.

Speaker B: Uh, yes, an Iceberg or Iceberg based uh, data stack is essentially compromised of uh, the storage layer, the catalog layer and the query engine, the compute layer. You have the storage itself, you have your files organized in an object storage bucket and you have made metadata attached to, attached to them inside, inside the object storage. So the function of the catalog is essentially taking that file based or other structure, based organization structure from the object storage itself into a more uh, structured relational database, uh, that is easier to scale essentially and handles all of the metadata and how the metadata changes and it allows you to roll back changes and a lot of other functionality that is core to the value proposition of these table formats. It is again a necessary part to use Iceberg itself in a proper data stack. And yeah, I've been rambling a bit about catalogs. The original question was about what was the cause of the catalog wars or where is it going to, or yeah,

Speaker A: first part of the question is what is the relationship with the catalog and Iceberg? And second was like why this catalog is essentially heading to.

Speaker B: Yeah, so I think there's different implementations for catalogs themselves because developers kept thinking better and better ways of where to store this metadata and how to track these changes in tables, namespaces and other resources. So there's, I don't even know how many implementations of a catalog that use different backend databases to store this information. There's one that kind of stands out, which is the REST catalog, which is an open source definition of how a catalog should look like, what kind of uh, interface it should provide to the actual developer. And the implementation itself can be any of those previous catalog implementations using whatever technology it uses, but the API that it provides should be all the same. So the idea with the REST catalog was that any service that provides a catalog for you to use with Iceberg should be pluggable with other parts of the ecosystem. So I should be able to use a catalog from any service and use the data itself in an object storage from any different services. The reality uh, is that this is not the case There are differences between REST catalog implementations as well. So all the big data warehouse players like Snowflake or Databricks, they are all, during the catalog wars they all started implementing and open, sourcing their own flavor of a uh, REST catalog. And we've done a bunch of testing between these and some other catalog options. And even the REST catalog was expecting different operations or running some, expecting some different data structure. So it didn't really work out to have a unified interface. Sadly. So sadly this is one part vendor lock in is a big risk. Uh, everywhere else Iceberg is totally open. You can use the data um, however you want with another query engine, but the catalog itself is not easily replaceable. Um, and this is partly because vendors want this level of lock in to their own system. So if you start using Snowflake, uh, Polaris or Open Catalog, I'm not sure what they renamed it to now. Uh, the managed version, it's okay. Yeah. Then they obviously don't want you to be easily swayed and maybe migrate to Databricks Unity catalog because even the REST catalog is a bit different. So I think it's an interesting aspect of this whole ecosystem. But what I feel like is that in terms of functionality, uh, for Iceberg tables the difference here is also so small in terms of functionality like that there's, I would, I couldn't say if one of the catalogs ah, is better than the other one in pretty much any core aspect. What catalog will win I think is the one that manages to integrate non relational or non tabular data sources as well under the same umbrella somehow similar to how uh, Unity is doing. For example, like you can put like images, AI models, videos, whatever into the same catalog. The Iceberg part itself I think is fairly well defined and yeah, if you want to you can implement it. Not that that complicated.

Speaker A: Oh, okay, this is interesting. So what you are saying that the role of catalog, they go beyond the structured data. What Iceberg is actually providing is much bigger than Iceberg based catalog. It has to really integrate like other unstructured data. Uh, and what is the context behind your statement there?

Speaker B: I think the most important part is to fully democratize data, uh access you need to be able to plug into all kinds of different systems. Like you need to be able to use the same data to train uh, AI models to feed an operational analytics dashboard in real time. And you also need to be able to generate uh, daily historical analytics. And Iceberg, the table format itself is uh, perfect for this. Like it can already uh, satisfy these needs but for other stuff you have to integrate other types of uh, data as well. So audio, video really anything that you can think of and use those similarly from a query engine which would also have to support these kinds of data types as well.

Speaker A: Okay, that's interesting. There are certain architecture style people are proposing what they call maybe catalog of catalog or Uber catalog. They say pick whatever they can't differentiating like this is a technical catalog, this is a business catalog. And then you can pick whatever uh, catalog that you want. Then you sync with Uber catalog that has a global access to that one. Many of the business catalog vendors right now trying to push that kind of a format. Do you think that is a good strategy to go for an enterprise? What is the pros and cons of with those kind of approach?

Speaker B: So I think from the perspective perspective of the user, the actual data consumed data practitioner who will write those uh, queries that uses this data. I don't know if it makes that much of a difference uh, or they actually should care about this. And I do prefer the unified uh, catalog approach where everything is there, there's one interface to interact with all of the data, all of the semantics are the same. And yeah it's just easier to reason about, easier to learn, easier to implement if you roll it your own. If you separate a business service catalog and the technical catalog, I think you, you also take on the risk of uh, people not being so good at handling the two and then you need the third one to actually uh, turn them together so it's extra complexity. I can see some vendors going that way because it's easier to work on them separately than by default unifying them.

Speaker A: Yeah, okay. Okay. Yeah. Anything that running standalone has an additional uh, integration complexity associated with that one. One of the interesting development happened recently in the S3 table which is break little bit of a conventional at least for now there is no ICE forecast catalog support there. But their interface is very interesting. Or write on table API get table put table get record and put record is very straightforward to more closely addressable with the typical database interface that you've seen. What is your general take on SA table?

Speaker B: Yeah, it was announced a month or two ago something like that and it made a big splash and everybody was panicking that AWS has solved iceberg and that the vendors are uh, going to die and it's the end of the world. But that didn't happen it turns out. And actually I don't even know anyone who started using S3 tables in production or even testing it out uh, for some reason. So S3 Fables are essentially a super lockdown version of an object storage combined with a table format. And as far as I remember they also support uh, maintenance operations if I remember correctly. Which is a great step in my opinion too because it's a much needed uh, part of the whole thing. But the lock in being able to access the data that lives in S3 tables is proven to be hard. As far as I know you can't bring your own catalog and then use that with it. So it's in a weird place where it makes sense as a next step from S3 that you can have a structured uh, format over your data but at the same time it's a, it's a little bit hard to integrate into other existing data stacks. So I'm still waiting to see a use case for it. Uh, but I feel like AWS will have to implement a few more features to ensure that uh, make it a bit more useful.

Speaker A: Yeah, yeah. To your point, if S3 Table make an interoperability better and the fact that it can already store the unstructured data together, it could be a good approach for an organization rolled out because those data storage can be compiled together already.

Speaker B: Yeah, there could be. I'm curious to see if uh, anybody will actually try it out.

Speaker A: Yeah. The third layer that you mentioned about query engines, we have few query engines like Trino for example and some of the query engines like Snowflake and Databricks SQLs and they're trying to support Redshift also supporting querying Iceberg format there. One of the things people are mentioning it is not as performant as the native format when you're querying mysphere uh format as well. Um, what are the key things an organization should consider when they choose the query engine?

Speaker B: So I think for. I read a few weeks ago that somebody was uh, doing a bunch of benchmarks and I'm blanking on the name but they compared uh, Snowflake's performance, the native tables against Iceberg tables and the difference was surprisingly low, like 5% or something like that. Um, I think they managed to make it usable and so good that at this point it doesn't really make sense not to use Iceberg if you have the option to integrate that kind of infrastructure. I don't know how the other uh, query engine is doing, but with Snowflake I think it totally makes sense, especially if you're starting now to use Icebreak Iceberg, definitely. Or consider Snowflake as a compute engine. You can see it can be a great compute engine. It has amazing features. You can use like uh, a lot of AI integrations as well. So it's. Yeah, it's very good. As for the other query engines, I see a lot of people using Trino, that's super popular and it has endless integrations with any kind of iceberg, uh, catalog or any kinds of storage. And that makes it uh, I think very flexible and like a lot of teams are able to adopt it. One of the cool query engines that I see more and more of is um, DuckDB and Mother Duck as well through the DuckDB extension. Uh, which I think is great for those use cases who don't have like petabyte scale data necessarily but they still want to start using Iceberg and start building out their lakehouse. Lakehouse, yeah.

Speaker A: Their lakehouse, yeah. Speaking of Captivia, one of the challenge in an organization I've seen that people don't know how to tell this is a small file I'm processing versus a large file I'm processing especially for the analytical workload because people who manage the data is completely remote from actually people querying the data and producing the value which is often business analysis or data scientists. Whereas the platform engineers who manages the table bringing and doing an ingestion are completely different. And um, if an organization trying to adopt DuckDB, what would you advise?

Speaker B: So if an organization wants to adopt DuckDB, I think one of the important steps to do is yeah, just make sure that their scale is not too big somehow. And also another important uh, part is that DuckDB is a ephemeral uh, query engine. So there's no long running process that you can just uh, jump into and run a few queries. So you need to build out some infrastructure around it yourself, some kind of automations to actually execute your queries. And there is some or like somehow run them locally if you want to. So a lot of consideration has to be taken there. Can the compute that you use for DuckDB actually handle the data that you want to query it from? Iceberg or you can use Mother Duck which to some extent takes care of this as far as I know pretty well and seamlessly. But yeah, it depends on the DuckDB extension which as far as I know still lacks a few features from like full Iceberg compatibility.

Speaker A: Okay, I always wonder, you mentioned about table format and what are the challenges about maintenance and then the catalog picking, uh, the right catalog making, input interoperability and the picking the right query engine to do that. There's so many complications for an organization to go through. This one is it Worth of a fame versus adopting systems like Redshift or Snowflake. Just it's like you take care of that. I don't need to worry about it. I just wanted to query that.

Speaker B: Yeah, that's the big question, right. I think for a lot of organizations it's not worth it because currently you need uh, a large amount of operational health or operations to spin up an uh, iceberg based data lakehouse and to maintain it as well. But as soon as this gets easier and easier I would be more comfortable recommending Iceberg and uh, some kind of combination of the other components to a wider array of users. But for now if you don't have the engineering resources to spare, or even if you have the engineering resources to spare, it probably makes more sense to start with Snowflake and then you can just start incorporating Iceberg and just keep Snowflake as a query engine because even do you really want to spend or have five data engineers or data infrastructure platform engineers spend uh, their full time for months implementing Iceberg but you can just spin up Snowflake and uh, start from there from day one. So it's always a trade off between how much work uh you can put in versus uh, are you willing to pay a little bit of a premium for uh, a more managed service?

Speaker A: Yeah, that I'm always very curious like what extended organization can afford to choose what they want and what the cost they pay.

Speaker B: I think also one of the uh, bigger aspects here is if you want a completely open source um, deployment or are you fine with using a managed service. I know that there are a lot of organizations who run their own uh, minio internally distributed across a few servers and they slap the iceberg on top of it and use uh, Trino or Presto as the compute engine and it's completely in house, completely air gapped, there's no Internet at all involved. And uh, yeah if the requirements are like that then Iceberg can be uh, a great choice. Pretty much the only choice for a scalable uh, infrastructure.

Speaker A: Yeah, totally. And um, there is a lot of value in keeping the data uh open and portable for an organization because you never know when you will get a huge bill next time from provider. One thing that interesting you mentioned in the article is about Apache X table what role you see a project like XTable will play in this larger game of like Icebergs and catalogs and other things.

Speaker B: Uh yes, XTable is an interesting initiative. I admittedly don't know uh, a lot about it but I get the idea of what it Is trying to be essentially it is trying to be a standardized interface in front of any table format, Iceberg data or Hudi. I don't know if they panel at the other ones but why not? Meaning that you only have to communicate. Your programs only have to communicate with xtable and behind the scenes those queries can be translated to iceberg hudi depending on what storage system you want to communicate with or your data lives in. I think this is f. It seems like a natural step in the evolution of we always need one more abstraction layer in all these services. I haven't seen an organization that uses multiple uh, table formats for different use cases. I think uh, in terms of reducing complexity it's probably easier to standardize along uh, one table format instead of adding another layer uh, on top of the whole thing. And yeah, just managing that as well. This is something that I think and feel but I would be loved to prove wrong by someone who actually uses xtable. So if there's anyone then please reach out. I would love to hear your story.

Speaker A: Yeah, totally. Yeah. I'm also very curious to know that. Do you predict. You predict some kind of a consolidation in iceberg table and you talked about all the layers of complexity and all those things. What kind of consolidation that you're envisioning.

Speaker B: So I think some similarly to how to what S3 Tables has done. I think that the first step in the consolidation phase, I think a lot more services are going to bundle maintenance operations first with their iceberg offerings and catalogs as well. So I think anybody who wants to provide a platform uh, that uses Iceberg we want to also have control of the catalog and manage uh, the necessary maintenance operations because uh, to create uh, a really seamless developer experience you have to take care of those and uh, yeah, they provide zero business value and uh, the average data engineer won't care honestly. So if those are done then you have a nicely packaged iceberg based data lakehouse that you can use with really any compute engine and any engine that you want.

Speaker A: Yeah. Okay. So you're predicting whoever providing the iceberg as a managed version of it with the uh, table maintenance and other things will also provide a catalog. So both will have a very good integration with each other's.

Speaker B: Yeah, I think that's 100% uh, the way the industry is heading. And after that the next step will be to manage the compute as well the query engine layer. And that will actually bring you like a totally end to end data lake house. Um, but that's a lot harder than just the Catalog and the maintenance and query engine.

Speaker A: Also like organization might want to have a different query engines for different use cases. There's no one query engine can do, for example machine learning. I might use like Pandas and all the other aspect of it.

Speaker B: Yeah, exactly. Or you might want to use Flink for real time stuff or DuckDP for small scale analytics or. Yeah, yeah, it's usually completely up to the use case.

Speaker A: Yeah. So one of the things I also mentioned article beyond analytics like Iceberg will play a pivotal role in event driven architecture and machine learning. How does this Iceberg will fit into the event driven architecture? Can you paint a picture on it?

Speaker B: I think Iceberg has the foundations of what can make an amazing storage backend for event driven architectures. Like it has time travel, it has acid guarantees that are like requirements for change data capture pipelines. It can do joins well and it can also version and branch data. So these are all amazing features that make it uh, a great backend for streaming systems or even driven architectures. For example, I know that there's a company or tool called Buffstream is like a Kafka replacement, but it uses Iceberg as the storage layer and I think that's a great example of how uh, Iceberg can fit into architectures like this.

Speaker A: Yeah, I think there are a lot of tools now emerging similar to Kafka. Writing that directly into S3, writing directly to a lakehouse bond might provide more and more real time access associated with that one.

Speaker B: Yeah, yeah, definitely. That's the future for sure.

Speaker A: Yeah. Okay. Given all those things like what do you think the remaining unsolved challenges, like what are the open challenges in this, not necessarily IceWell, but mostly for the, in this table format two years down the line. What do you want to see from these table formats?

Speaker B: I think there's. On the technological feature, technical feature level, there's a few things missing, but not necessarily from Iceberg, the table format in the authentication, but from the ecosystem. Like it's still unreasonably hard to write data to Iceberg with most of the tools. And I refuse to accept that, uh, Spark is the best way to do a lot of these things. So I think a lot of data engineers and a lot of data practitioners don't come from the Spark world. They come from the snowflake world. They come from the analytics side and they have no idea how to operate Spark. They don't want to use even a managed Spark cluster or anything like that to enable that audience or that user base. First we have to get rid of these Things and it should be compatible with all of the tools, all of the query engines that you can use for maintenance and you can use for writing and reading data in batch or in real time. That's one thing that is missing. The other one is the whole catalog mess that we really have to try and standardize, which is hard because. Because every vendor decided that uh, if you own the catalog, you own the Iceberg database. So they are trying to make it as hard as possible and make each catalog as unique to them as possible. Yeah, that's also going to be. That also is a blocker for adoption and has to be solved, I think, for Iceberg to fully become like a really an open daily cast.

Speaker A: Okay. Yeah, I definitely cannot resonate with the, uh, payoff. It's incredibly hard to write the data into this one. And the fact there's a heavy dependency, um, on Spark, which typically they will come and then you need to require a cluster to maintain EMR kind of a model. And that is also additionally expensive for you to do that one versus data.

Speaker B: Yeah, I think if your competitors as a data lake house are uh, at tools like Snowflake, then you have to be easily used by analysts or data engineers, which Snowflake and databricks, they get it right. And until Iceberg is just as easily implemented and maintained and used, they're going to have a hard time to attract that crowd.

Speaker A: Okay, so in the emerging world of this lake format, do you think two years down the line, do you see still there is a space for the closed table formats like Snowflake or redshift format, for example. Is there a room for them or all the organization should or we move to in open table format?

Speaker B: That's a good question. I think sooner or later people will demand that, uh, their data is accessible from other services, which is not going to happen with a closed, uh, table format. Uh, so it's up to the users, I think, to ask for this. As for the organizations, I think it makes sense to use an open source table format like Iceberg because it gets just so much more development effort than what they could internally put into their own. And this is true for large organizations as well. So I think switching to something like Iceberg isn't for everyone.

Speaker A: Okay, yeah, yeah, I can relate to that one. So we almost come to the end. I wanted to quickly ask, what would you advise for a new company, they're trying to get into the lake house format. What would you advise? What are the key things they should be keeping in mind to avoid the free ball and not end up Hadoop. And then I don't. We don't want Gardener to provision. Article says like 80% of iceberg projects failed.

Speaker B: Yeah, yeah, definitely. I think the first, the most important question is to really think if you need Iceberg, do you really want the features or do you really need the features that Iceberg offers? And are you prepared to take on the operational overhead that it comes with? Once you are done with that, then what I usually recommend is to just start small. Just create a PoC project, test it out, test out the integrations. It's a very new ecosystem. On the grand scheme of things, you have to make sure yourself probably, uh, you're going to run into a lot of things that nobody has ever blogged about because it's so new and everybody has so niche use cases that you really have to take your time to test it. And the other one is after this is all done, to make sure that you automate the metadata management. So you have to automate the cleanups, you have to automate the compactions because metadata growth can very quickly become a bottleneck for any organization. And so once you have these technical things out of the way, I think it's also super important to learn about Iceberg and how it works, because it's of course, like, you can write SQL and get back results, that's sure. But the way it works is different than many traditional data warehouses. So a little bit of technical knowledge will go along. Ready? I think.

Speaker A: Okay, awesome. Great. Thank you so much, uh, your time, Danny, uh, for sharing all the insights about Iceberg. Certainly. Thank you so much.