Denodo on Active Data Architecture: Trends and Outlook

Speaker A: Foreign.

Speaker B: Welcome back to AI Data Bites, powered by denodo. I'm your host, Neha.

Speaker A: Huh.

Speaker B: And today we are diving into a topic that's getting a lot of attention in both the data and AI worlds. Active Data Architecture. As organizations scale their AI initiatives, they're realizing that AI is only as effective as the data that powers it. Access to trusted, governed and real time data has become critical. And that's where Active Data Architecture comes in. Dresdner Advisory Services released its Active Data Architecture report. And for the second year in a row, denodo was recognized as a top active data architecture vendor. The report highlights why organizations need a platform independent data layer, why semantic layers are becoming essential for AI and analytics, and why data lakehouses alone can't meet all the demands of modern AI AI driven enterprises. To help us unpack these findings, I am joined by Dominic Sartorio, VP of Product Marketing here at denodo. Dominic actually sat down with Michael Moran, research VP at Dressner, to discuss the report, the growing importance of Active Data Architecture and why denodo's logical approach is becoming a key enabler for enterprise AI. So I can't wait to hear more from you, Dominic. And welcome back to AI Data Bytes again. And it's great to have you here with us for this conversation.

Speaker A: Thank you, Nihan. Thank you for having me again.

Speaker B: Okay, wonderful. So let's start with the big picture. In that case, why is Active Data Architecture suddenly getting so much attention? And why did Dresdner define it as a platform independent layer between physical data and data consumers?

Speaker A: Yeah, great question. And according to Dresner, what they're finding in their research is the business wants more ownership of their own data. Uh, they want to self serve, they want to find and discover their own data for various initiatives. They may be doing data sharing, they may just want to enable more of their own business users to self serve and get their own insights. And more recently, it's AI. They're trying to build AI agents and bots, uh, that have directly serving the business or business users. And what they're finding is data in its raw physical form, even after you move it to a central lake or warehouse, often lacks the business context. It's not understandable by the business. So that has driven a demand for another layer. You need some other layer that is between the physical underlying representation of data and more business contextual form that the business users and also AI can readily understand. The reason it has to be platform independent is because invariably the data that the business is interested in is going to be across multiple source systems. Even if an organization has a strategy of all data being copied or consolidated to a uh, single lake or warehouse, invariably there's going to be data where it does not make sense to onboard it because of the volumes or the sensitivity. Uh, or it may come from a third party or you need access to live real time data. There are various reasons why you may not be able to move it, but the business needs it anyway. So that creates a demand for another layer that's independent of any individual platform underneath, but instead can access and source data from all of them. I think this here, this is really coming to the forefront with AI. Uh, the other use cases I mentioned, like data sharing, business, user self service have always been there, just gradually increasing in importance. But now this year, AI, AI agents, et cetera are really getting a lot of budget and a lot of attention and they're finding the same problem. For the AI to make trustworthy decisions and understandable by the business need to make sure it's looking at the right data that also is within the right context. Uh, otherwise how's the AI going to know what data is relevant? So all of these reasons together have been leading organizations to look for platform independent layers that uh, present data in business context and that that really is what active Data architecture is according to Dresner.

Speaker B: Wonderful, thank you Dominic for this and it definitely explains it very well. So so and it also brings us to the key enabler in all of this, that is semantic layers. So my next question for you is one of the biggest findings in the report is that over 60% of respondents said semantic layers are critical or very important. Right. So why is the semantic layer suddenly at the center of enterprise data strategy?

Speaker A: Yeah, you've got a keen eye to notice that in the report there are various components of active data architecture, including data virtualization. So the ability to have virtualized views across multiple data sources, the uh, ability to govern at a single point of access, the ability to have active metadata which is ongoing up to date. Metadata of how data is being used, by whom and where, those are all various components of Active Data architecture. But semantic layer is the component that really is central to making everything hang together because that's where the business context is being represented. It's all about business semantics, hence the term. And part of their research, part of Dresner's research was to ask the surveyees which of those various components are most critical to you and uh, how would you stack rank them in criticality? So a very interesting was significantly more people were Placing more importance on semantic layer. So over 60% saying it's critical or very important. And I think AI is really driving that. There's a recognition that, hey, my AI pilots or uh, my AI initiatives are not succeeding. They're not delivering value because we are not sure we can trust the results or we know it's generating results that are not accurate or they're hallucinating. And so that's leading to a renewed attention on, okay, what is it the AI needs in order for us to trust its results? And uh, a big answer to that question. It needs data which we can trust. Uh, we need to make sure the right data is being fed to it. Well, how do we know what data we can trust? And you don't know that unless you know what is the business context of that data, what, what aspects of the business does it come from or is representing? So then the semantic layer is intended to answer that. So definitely, yeah, semantic layer becoming more important. And Dresdner believes, and I'm seeing this too with our own customers and prospects, that AI is really driving that. The failure rates of AI have been very high. And a big reason for that is can't trust the AI. And you can't trust the AI because you're not sure you can trust the data. But best way you can trust the data is the business contact is correct. And that's what semantic layers do.

Speaker B: Wonderful. That makes a lot of sense. Thank you, Dominic. So now with semantic layers in place, the next question is how this translates into practical real world AI use cases. So the report highlights that AI depends on active data architecture because AI systems need timely, relevant and unified data. How does Denodo enable this in real customer scenarios?

Speaker A: Yeah, good question. And it doesn't really matter until rubber meets road and you have customers in production realizing benefit. And by the way, there's a lot of vendors out there talking about AI. Recent Gartner events we've been to, for example, maybe 3/4 of the booths say AI all over them. So everybody's AI washing themselves. But there are precious few examples where they actually have successful customers in production. And, and it's great, denodo is among them. So we have some customer examples that I can share. One that I really like is with Festo. They are an industrial consultancy really based in Germany. They operate all over Europe. Primarily they advise and consult with manufacturers. Uh, imagine they have big factory floors, complex supply chains on how to operate more efficiently and, and more recently they're getting good business with sustainability because the European Union has a Sustainability act, which requires manufacturers and any other resource intensive industry for that matter, think of transportation, aviation and so on, that uh, they have to report that they have adequate measures in place, uh, to be more sustainable and that they are keeping resource consumption under control. Um, well, m. Many manufacturers were just not prepared for this. So Festo is driving really good business and providing good advice. So imagine you're a consultant working at Festo, and Festo's been in business over a hundred years. They've been at the forefront of new technologies and innovation and manufacturing all along across many industries. Consumer packaged goods, discrete manufacturing and automotive. It's imagine just the sheer spectrum there. So as a consultant it puts a high burden on you to ensure that you're providing any given customer the best advice that Festo, across its hundred year corpus of knowledge and experience can deliver. Well, that's not really possible to put into one human brain. Right. And so inevitably those consultants spend a lot of time just researching that industry. And what else, what have other Festo consultants done in the recent past? So the way they solve this is through denodo. So they built a chatbot style AI interface where consultants could just ask the question. The prompt can include, I'm working with this kind of organization in this industry. Here's what they're trying to accomplish, here are their key challenges and what advice would we give and what examples do we have? So imagine just being able to ask that question and it responds with pretty good answers based on the integral entire history and corpus of collective knowledge at Festo. The way they did this is they used denodo as a common data layer across all of their data sources, both structured and unstructured. Um, you can imagine a lot of data is just inherent in the documentation that you provide to a client. And all of that is unified with a semantic layer annotated by industry by location, et cetera. And then that powers their chatbot on top of that. So this infesto I'd like to talk about just because the sheer complexity of operating in that vertical I think is readily apparent to everybody. Uh, you know, we're not just making simple recommendations based on say, customer transaction histories or simple things like that. These are really complicated use cases, uh, for, for the most advanced manufacturers in the world. And denodo was able to provide and deliver a semantic layer, unified data layer. So they have a very good chatbot that uh, makes a big difference. So instead of spending days or weeks of researching these consultants maybe just spend hours having a conversation with their data and then being able to come up with the best advice in much less time. And so they're more productive and effective, uh, they can move on to the next client much more quickly. The clients are happier, they have more assurance that all of Festo's collective knowledge and experience is being represented there. So it's greatly helped their business. So we have other examples that are similar to that across other industries but they follow the same pattern. Another customer is Alex Forbes. They're financial services so a ah, bank, uh, based in South Africa and their wealth management division uses denodo. So their clients serve high net worth, uh, they are high net worth clients that their consultants serve. And similar thing where Alex Forbes has a lot of collective knowledge of what investments work best for different clients of different risk profiles. Got to keep it up to date with uh, current market conditions and they built a chatbot type experience. So but for similar reasons as Festo, a lot of complexity, a lot of history, uh, and how can you give the best advice much more productively. Another example is Perkins Coie, the North uh, America based law firm typically serving corporate clients. So very complex legal situations that large enterprises may face. They're multinational so you're dealing with law and regulations out of different countries, same kind of thing. We help them deliver PC chat to help their attorneys, paralegals and also their client success folks be more productive and give more effective advice. Um, and there's more coming so recently in production and you'll hear more about in the near future is nec. They're the Japan based uh, IT manufacturer and Telco among other devices, big manufacturer and they've completely transformed how they operate internally. They want to enable all of their business users right up to the CEO to be able to talk to their data essentially. Um, and so they can self serve, get better insights on how to operate more efficiently, better demand forecasting, better inventory management, uh, better post sales support of their customers, so on and so forth. So they use Denodo as a common data layer and with Denodo they have 1700 distinct business views that are in the context of the business and are understandable by people within those different business functions. And now imagine having a chat style talk to your data interface on top of that. Um, so they've just started so it's too early to see what the measurable results are. But that is the kind of scale and scope of initiatives that large enterprises are undertaking. And they're really struggling with how can we trust the AI, how can we trust the underlying data? And so by adopting an active data architecture and using denodo to implement that, they're already seeing a lot of success.

Speaker B: Wonderful. These were some really good examples. Thank you for sharing, Dominic. And across multiple different industries as well. So it really paints a picture in terms of, like, you know, how active data architecture as a concept, you know, can help organizations and how denodo can help implement it. So thank you for sharing. Now diving deeper into denodo specific features. Right. So denodo of AI tools like the Denodo AI SDK were highlighted in the report. How do these tools help AI developers discover and use data faster while still ensuring governance and compliance? I think you, you covered it as an example, but if you can speak a little bit about the feature itself, please.

Speaker A: Yeah, I didn't talk about the functionality as much. I just spoke to the use cases. Yeah. So Dresner's, uh, ADA report was focusing on the business context of data, and that's what organizations are realizing, and especially with AI realizing a lot more the importance of having another layer that includes the business context and the meaning of data. Um, and so the report talked about that, and it also talked about what does that mean in practice. So how do people in the business actually consume data, given that that layer is there with the business context? And so there are several different ways of doing that and which point to several different features in our product. So let's start with AI, which you asked about. So most AI teams, the people who are actually building these agents and bots that most organizations are deploying, they operate within the business. Right. They're like application teams. They build apps in support of the business. So they're not the data team and they're not usually sitting in it. Uh, think of them as application teams that report up to some business exec sponsor. For example, if what the agent is doing is providing a better customer experience, then the exec sponsor probably is a chief customer officer, or it could be a chief digital marketing officer or somebody like that. Um, you also see these innovation teams stood up within the business who are figuring out how do we adopt and embed AI directly within our business processes. So that's where these AI developers are. And they're not data people generally, but they still have to consume data. They're on the hook for making sure only trusted data is being fed to the AI. So the AI is trustworthy? Well, they don't know the physical data. They don't know SAP and ServiceNow and all these other systems where the data is coming from. But they do understand the business context because they're part of the business organization. So what the AI SDK lets them do is it gives them the ability essentially to do a semantic search for find the data that's most relevant for this use case and then present it in a way make that data accessible in a way that is very easily accessible by the AI during runtime. So that say a bot is being prompted or an agent is encountering a certain situation right away in real time, it can be sure it is pulling the right data and getting that right data immediately. And it's live data that represents current reality, the current situation that they're accessing. The way that's done and the AI SDK facilitates this is first you look at the semantic layer. The uh, AI asks, okay, given this prompt or given the situation, what is the most relevant data that I need in order to respond to this? So you, you ask the semantic layer which then comes back and says, okay, these are the data assets, these are the fields out of these data sources. And then we help generate the query to fetch that exact data from those source systems. And that query to Denodo looks like any other query, right? It's uh, presented through a common view, we optimize it, we pull the data out, return the result set and then the AI can further translate or act on that result set. So it's a metadata driven approach that we facilitate. And the AI SDK packages that all up the whole tool chain and AI developer needs to take ownership of their data and take full ownership of that development is packaged up in a single AI SDK. And another capability I can speak to is Denodo's Data Marketplace, which is part of what the AI SDK is looking at, but is also useful just for business users themselves. So the data marketplace, think of it as an E commerce like user experience. Imagine like Amazon for data in your enterprise where anybody in the business can go there, they can search for the relevant data they need. It's presented to them in the business context. So with all the right semantics around it, they can talk to their data. So text to SQL is a capability that's included there to get the right responses. So any business user can take advantage of this capability through our data marketplace. What the AI SDK does is it packages up that equivalent capability so that an AI can do the same thing during its runtime. Uh, so yeah, um, the ADA report from Dresner highlights the business context is important, semantics is important. And then how does the business and as well as AI consume data to ensure that you're using data in the right business context. The AI SDK helps helps make that happen for AI developers in the AI they're building. And our marketplace does that just for other business users as well.

Speaker B: Wonderful. Thank you for sharing these insights, Dominic. Now segueing towards how denodo enhances existing lakehouse platforms with a semantic layer. Uh, can you tell us if the report covers how lakehouses alone cannot meet all active data architecture requirements? And how does Lakehouse denodo complement platforms like Snowflake or Databricks and implement a real semantic layer on top of them?

Speaker A: Yeah, that question comes up a lot. Many organizations have one or more lakehouses and often those lakehouses are presented by their vendors and others as being helpful and useful for AI. And so that does create some confusion out there. The best way to answer this question is actually what Dresner analysts themselves say. The first iteration of this active data architecture report included lakehouse vendors, Snowflake Databricks and the Hyperscalers Equivalents as part of the group of vendors that they looked at. But then those vendors weren't dropped. They were not in the ADA report anymore. They were covered in a different report talking about data platforms or among other things. So, so I asked Dresner, why is that right? Why were lake houses included in the first iteration but then removed in this year's? And their answer was pretty telling. I think it's the best way to answer your question, Neha. Uh, which is the lakehouses are great for storing data and they are powerful compute engines for analytics. But the lake houses themselves are not great at these other critical components of ada. The notion of virtualizing data so that you can access live data in real time, you don't have to consolidating it. The notion of a semantic layer which provides the business context and also these different consumption patterns I talked about, like you need a data marketplace that's usable by business users. You need an experience by which AI developers can take ownership and AI itself can get trusted data. So these different consumption patterns, semantic layer, as well as the idea of virtualizing, that's not the strength there. Um, and so they felt they should be captured. They're really, ah, a distinct market segment. Data platforms is a distinct market segment. And dresdner recommends that any enterprise should have one or more products or capabilities from that segment. But then on the ADA side they're saying you definitely need something else. And having the lakehouse in their own report sort of reinforced to Dresner's readers that look for ada, you do need a separate capability. The lakehouse can't really do both. Right? You need capabilities and products from other vendors that provide these other things, that provide a virtualized approach, that provide a semantic layer and that provide these consumption patterns to direct self service by business and AI among other users. That's actually the best way to answer this question. The lakehouses are great for centralized storage and are great powerful analytic compute engines which do that class of work, that workload extremely well. But for other use cases like business self service or enabling trusted AI, you do need these other capabilities like the semantic layer. And so having then Denodo running alongside your lakehouse then gives you the complete solution, right? You have a lakehouse for the storage and compute, you have Denodo for the semantic layer, the business user consumption, trusted AI enablement and running side by side is what they actually recommend, that you have both capabilities there and that fills out the complete active data architecture as a result.

Speaker B: This clears it up very well. Thank you for sharing, Dominic. This was a great example and explanation. So, um, with a semantic layer in place now, the next challenge I think is ensuring AI moves fast but responsibly, I think. Right, so what's the best way for enterprises to align their AI strategy with their data strategy, in your opinion?

Speaker A: Yeah, great, great question. And this is another area. I think there's a lot of confusion in the market. I think the core of it is most data teams, data engineers, right up to today's cohort of chief data officers, are operating on a fixed set of assumptions of how data management should be done. That has evolved over the years, going all the way back to 30 years ago when enterprise data warehouses became popular, the likes of Teradata, Neteza and such. And back in those days you needed powerful compute engines running on dedicated hardware, being able to access large volumes of data on local storage. So you needed then to move the data to local storage. And thus was born the data integration industry with ETL and streaming products and things like that. But as we progressed over the years, you had new waves of technology like big data emerged with Hadoop and Spark, which enable more kinds of analytics, more, uh, diverse sources of data. But the same fundamental model of data management was there. Okay, we got to move the data there. And then more recently you have the whole cloud wave where everybody migrated to some cloud data platform, culminating in today's lakehouses. Uh, but again a powerful compute engine. And the presumption is you got to move the data to some environment, cloud environment, uh, that is local to those lake houses. So basically most organizations are just assuming, okay, we have an AI strategy. AI is very data hungry. Let's just amp up what we've been doing all along on the data management side and just have that be the way we deliver more data. So organizations have been doing this, have been investing heavily in cloud data platforms backed by data movement, data integration, traditional data management. But then why are the failure rates so high? Why are we, uh, continuing to see 80%, 95% according to an MIT study of failure rate? So clearly something is not working. Uh, what's not working is what AI needs from its data is fundamentally different than what these traditional data management principles, including moving your data, uh, were designed for. So how are they different? Well, first, a lot of AI is operational in nature. You have AI agents that are embedded within your business operations. You have bots that are responding in real time to prompts that are presented to it and the expectation is they respond within seconds. In both of those scenarios, you need live data, you need live situational awareness. And once you've moved your data somewhere centrally so you can analyze it, that data's out of date. And also, as I said earlier, not all data sources does it make sense to centralize. You have various reasons why that data shouldn't be moved. It's sheer volumes, it's sensitive or belongs to a third party. So any of which I might need. So you need the ability to get at live data from its original sources without losing the business context associated with all of your data in aggregate. And traditional data management does not address that very well. Another thing it does not address very well is, is the business context of data. Traditional data management does have governance for data quality, making sure your data, your, for example, your customer records are spelled correctly and are deduplicated and so on. So those traditional definitions of governance still apply, but by themselves don't ensure you're using the right data. So you might have an agent that's responding to some customer situation. Uh, but you have, you could have completely 100% clean customer data coming from your salesforce, your servicenow, your transaction order management systems, and so on. But what's the right data to use in that specific context at that specific point in time? A business contextual semantic layer will give you that. But an active data architecture does incorporate that concept, but traditional data management does not. And similarly with just governance in general, like how do you ensure data privacy protection? How do you ensure AI regulations, such as the ability to explain how AI came up with its answer, ensure that the AI is not biased in its actions and answers. All of that needs to apply across the full spectrum of data that the AI is using. So all of that is very difficult for traditional data management to implement. So I think this year we're starting to see a broad recognition in the market. Right? We cannot expect to transform ourselves, transform our business with this transformative new AI technology by using 30 year old data management principles. That's what's causing the failures. You have to fundamentally transform how you're managing data as well. So the AI transformation of your business and you transforming how you're managing the underlying data, that has to go together. Dresner believes that Active Data Architecture is the ideal way to think about how do you deliver the right data to AI at the right time, ensure it's the right live data, um, and you're staying within the guardrails, all the governance is there. And we believe Denodo, based on logical data management, the ability to access live data in real time and being able to centrally govern and have a unified semantic layer on top, similarly is the optimal product to use when you're driving AI in your business. Um, so these have to go together, right? You can't be successful with a transformative new AI strategy using 30 year old data management principles. That should not be a thing. And I think a lot of the market is kind of tuning into that. It may be after a couple of failures, unfortunately, but, uh, then they're starting to recognize, okay, we need to think fundamentally differently about how we manage data. Let's give ADA a look and let's give denodo a look and indeed, we're starting to see a lot more business from that kind of realization.

Speaker B: Thank you, Dominic, for sharing these insights and for breaking down why Active Data Architecture is becoming so foundational, especially as AI continues to reshape enterprise data strategies. And to our listeners, we hope this episode helped you understand more about denodo as a top active data architecture vendor in the dresdner Active Data Architecture Report, and how denodo's logical approach helps organizations deliver trusted AI ready data. Once again, thank you, Dominic, for joining us today.

Speaker A: Thank you again for having me. This was fun.

Speaker B: If you've enjoyed this episode, please head over to denodo.com podcast for more resources on all things data, AI and modern data management.