Episode 35 w/ Gregor Zeiler - Data Evolution: Tracing the Path from Centralized Control to Data Democratization

Speaker A: Welcome to the Data Democracy presented by renowned O'Reilly, uh, author Ole Olsen. Venue empowered by Xenia. Make your data accessible and discoverable by anyone, anywhere, at, ah, any time.

Speaker B: Hi everybody. I'm um, Ole Olufsen Bernheu, chief evangelist in Senea and the author of the Enterprise Data Catalog published with Aureli on the Data Democracy. I have conversations on new trends in data and AI with data leaders and data practitioners. Today's guest is Gregor Sila. Greger is an entrepreneur based in Vienna, Austria, heading up a company called Datastack. That's data spelled D A I T A Datastack is supporting software vendors, making their offering more precise and targeted. I wanted to talk to Gregor as he has a long time experience as an entrepreneur with an impressive record and also he is a thought leader. He recently published an article on data democracy that caught my attention. I found it inspiring and I want it to to chat with him m about it. Here are my three takeaways from my conversation with if you want to be independent in it, it's a good idea to provide both consultancy and software. Typically what you do is you start with consultancy, accumulate capital and invest in software that is expensive, but once it's up and running, uh, can sell it and scale it faster than your consultancy services. Gregor has a lot of experience in this field and there's a lot of things to understand if you want to do that. So I really encourage you to listen

Speaker C: to what Gregor has to say.

Speaker B: My second takeaway is that AI is redefining data engineering and data management, radically improving the ways we do these things. And we also need to stay in the AI game even though we find capabilities and offerings dubious at times. It's simply a matter of accumulating experience once the good products hits the market. We all need to be aware of how AI works, uh, in a lot of different contexts. My third takeaway, decentralization means data democratization and Kreger has some in depth insights from his career on how this evolved in the last 20 to 30 years. Okay, enough of me talking. Let's hear what Gregor has to say.

Speaker C: Hi Gregor.

Speaker D: Hi, Ole.

Speaker C: I'm happy to have you on.

Speaker D: Yeah, I'm happy that you have invited me to your podcast. So thank you very much, Ole, for that.

Speaker C: Oh, obviously, Gregor, it's a pleasure. Okay, so for the listeners, uh, you care to explain a little bit about where you're based and what you do?

Speaker D: Yeah, I'm based in the beautiful south of Vienna in Austria, but I'm working for clients all over the world. So let me explain my career. So a little bit more than 30 years ago. So it's a long time. During my first job I discovered my love for data management and data engineering. And that stays with me ever since. So what fascinates me uh is that constant innovation in the field. Uh so think about 30 years ago about that uh gorgeous bi frontends which helps the business to build their own reports by track and drop or uh, things like that, uh, powerful ETL tools or data integration tools which are able to move data from the data sources uh to the centralized uh data warehouse solutions uh, or that big data technologies which occurs a little bit after or currently the impact uh from AI on data management, um, that's um, never boring me. So I'm very excited about that continuous innovation in that field and that makes me proud of that to be in that field and work for that uh topics. So I've spent most of my time um in data analytics consulting gaining a lot of experience implementing data warehouse solution big data um, uh, or data lakes and data lake houses mostly in large enterprises. And uh, because I have a next um boring repetitive work I started to automate the data engineering processes. So 20 years ago roundabout and this quickly led to my first data warehouse automation software. I built together with my team at that time and uh, around about eight years afterwards I sold that company to an even larger uh consulting company and we decided together to build a new data automation tool and even better one with all the experiences from the past inside and uh, do it a little bit better. So in, in other words for the last 12 years of my career I was mainly in the role of a software vendor but embedded in a consulting company. So only the last two years um, were spent as a pure software vendor carved out from that consulting company. And at the end of the last year I decided to found a new company which is called Datastack and where I can use my experience in data management and building SaaS companies and help data and AI unicorns uh, to succeed their first steps in their journey. So that's my career. Roundabout Cities in a few seconds Compressed

Speaker C: It's a nice summary.

Speaker D: This gives a picture of what I did in the past.

Speaker C: Yeah, yeah, yeah. It's a super nice summary Craig. I a bit younger than you So I remember 30 years ago I had a very very nice Olivetti uh computer and I enjoyed like moving uh things from one floppy disk to the hard disk to the other floppy disk and uh, played around with were perfect. I had a laser printer that could spit out 200 pages a minute. Uh, yeah, so that's obviously uh, it's some time ago now. Right. All this is very interesting and I want to dive into that experience more. Uh, but first of all let's hear a bit more about what is datastack.

Speaker D: Yeah, so let me explain. I think in a world where the data landscape uh, is becoming increasingly complex, I want to help to make data in the AI more tangible.

Speaker C: Mhm.

Speaker D: So for me this has two dimensions. On the one hand, um, I want to help end user companies uh, to find the right data architecture, the right data strategy and of course the right tools for building those things. And on the other hand I mainly support um, uh, specialized software companies in the field of data engineering and data management, uh, to gain more market awareness with their products and to succeed the first steps on their journey to being a successful SaaS company. So in this case I use my experience from the past as I have explained, um, with my own products and especially the lessons learned of the last 20 years as a software vendor. Um, and I want to help my uh, clients with that experience I, I collected in my past uh, career.

Speaker C: Yeah, yeah. And I also want to get back to that. But I just feel like asking, um, given that you provide these uh, services, how like right now, in the current economic situation in the world and the political situation in the world, everything that's going on around the planet, how do you see the world of software vendors today? What are their challenges? What do they need to focus on? What are their obstacles?

Speaker D: I think that the challenges uh, are mainly what's going on, on that data democratization journey. Uh, so I uh, think uh, there's a ah, major shift in the world currently in how to handle data. Uh, so that comes from the idea that to handle data as a product and um, that data product thinking in general, moving the responsibility or the ownership of data to the more business related areas. So that's what's um, uh from my point of view is the most important challenge for companies right now to do the right things to support that uh, approach uh, in a proper manner. So using uh, data. So years ago I heard that argument that data is the new gold and everything like that.

Speaker C: Oh yes.

Speaker D: So that's clear. Yeah, I uh, think um, that's obvious over a few decades in the past that data is very important for getting uh, being a successful company. You need to have a picture of what you're doing with your company, what's more or less successful and then you have to react on that. So data was Always kind of an important thing. But uh, with all these new approaches around data, uh, and AI especially it's important to give uh, the users or the data owners the capability to use that data in a proper manner. And that's the biggest challenge in comparison to the approaches. What we experience is uh, what we experienced in the past. So let's assume we had in the past a centralized team very uh, skilled in that area of data engineering and data management. And now we have decentralized teams which don't have that high skill level in data engineering. But uh, they need to do the preparation of the data uh according to their use cases. So they need to have support in that area. And um, that's the most important challenge from my point of view to enable organizations with those capabilities that business areas are able to prepare the data for their use cases in a proper manner.

Speaker C: You strike me as uh, providing very succinct answers. But we'll circulate back to some of this. I want to ask you the lessons learned being a software vendor as part of a consultancy. You mentioned that and I also introduced you uh, in the, in this way, ah prior to our conversation for the listeners. What, what kind of lessons learned is that? Because it's a typical setup, right? You, you, you do consultancy and software and then you transition more and more towards software. Obviously it's, it's, it's harder in the beginning but then you can scale it faster uh, once you, once you get uh, get it up and running. So what's your lessons learned there?

Speaker D: So I would say it's the other way around. It's easier in the beginning but harder speeding out. So at that stage, so there are pros and cons. So let's say I'm really a fan of that approach because um, being embedded in a consulting company is a perfect starting situation for building a software product because you can test your product ID in many projects uh, so you have perfect feedback for that product idea uh in that surroundings of a consulting company. And uh, it's also very convenient uh to be able to finance your first development steps uh outside from projects uh, to being sponsored by them. So that's the big advantage of that approach from our point of view. But at some time um, there is a kind of a conflict between the interests of consulting services and the product uh areas. And uh, for instance as a small area, a product uh area in a larger consulting company you are always at the back of the queue. So let's say the second priority or the last priority, locating budget or Things like that. So uh, from my point of view this is the latest point where you have to think about the emancipation of being an independent software company outside from that consulting company. So then you should quickly change the mindset of your employees to have that less service oriented uh, mindset, more product related mindset. And uh, to find somebody who not only give you the money for expansion but above all to help you to drive that expansion. So maybe you don't need that much money at that point because you have already customer base which is able to finance your further steps. But the most important thing from my point of view is to find somebody who can help you to step outside that previous ecosystem of that consulting company and market the markets uh, uh, in a portal, um, approach. So that's the most important thing. And understanding the pros and cons of that approach and finding the right timing to step out of that approach. That's what I have Learned the last 20 years and I get a lot of uh, experience on that. And um, that's uh, how I want to help others uh, in that stage. So I'm a big fan of that approach. But you have to find the right timing and the right argumentation to, to stand this kind of approach.

Speaker C: It's almost like when I listen to you Gregor, um, it's almost like uh, it makes me think of this concept that I encountered uh, in university, um, in my studies. Tacit uh, knowledge like this is something. It's like winemaking or baking bread. Like you can write it down on a piece of paper but you don't get it before you have actually been in the situation and felt exactly like okay, this is the moment that we do this. It's something that simply takes experience.

Speaker D: It sounds like, yeah, I think uh, that's in each and every area you have to think that you cannot write everything down and transform your knowledge to, or share your knowledge by uh, writing down all those things. You have to experience that uh, for yourself and then you are able to learn uh, about that. And of course you can um, tell somebody your uh, experience and share your experience and uh, maybe do the same thing and remember what you have said at that time that they will experience. So um, I think you need a kind of help in that stage uh to make the right decisions and uh, uh, reduce the conflicts between those different interests inside that organization. So it could be an advantage even for both uh, to use the product and the consulting services together. It's a question of positioning and things like that. So um, this depends on the use case of the product and the capabilities of the product and the direction of the positioning of consulting services, um, how they can interact in a proper manner and get advantage from each other or synergies in that area.

Speaker C: Yeah, sure, I know a lot of people uh, that works in exactly this setting that you describe here. And it's just a very, I won't call it a maze but it's pretty important to get it right and it's not easy to get it right. Uh, the balance and understanding the momentum and the strategies in which directions you want to go with what service and software. Anyway, uh, you mentioned also that you are now supporting specialized software vendors with DataStack. So how do you select them and in which areas are they located?

Speaker D: So that's a very good question and it's as hockey I myself for quite some time. So let me explain a uh, little bit the background of my thinking about that. So I can't remember a time uh, where there are so many new software vendors in the market in the area of data in the AI right now. Mhm, exactly. And for me there are two main reasons for that. So uh, the first uh, is AI is opening a wealth of new product approaches that make data management and data engineering much more efficient uh, than in the past. So and the second thing is that we are currently experiencing the most massive disruptive step forwards uh, to data democratization with that data mesh or more data product thinking approach. So uh, data democratization in particular is dramatically changing the data requirements of the tool landscape for companies. So with decentralization, uh, data governance, data marketplaces or in general tools that enable the data product teams to build their data products as easily and quickly as possible are becoming extremely important. So that's the reason why I'm focusing especially on those kind of products which are currently approaching that paradigm shift to support especially the needs of the companies in that area. So that's uh, a little bit the thinking behind how do I select special vendors in that uh, area and hopefully I can help them with my experience uh, and helping to be more successful in the go to market areas.

Speaker C: Sure can, sure can. I do agree in all what you said. I think it's a very, it's also something that we have to get right, uh, in the sense that I do agree that AI obviously is making data management uh, more efficient, more powerful, more

Speaker D: easy

Speaker C: if we get it right because there is a. I don't want to sound pessimistic but there is a substantial risk in unleashing AI. I'm also obviously concerned about the regulatory aspects. But uh, more than that it's just also the functional aspect of AI, right. You want to get it right. If not you're just building, as I see it, you're just building technical debt in another part of the stack or the pipelines or whatever you want to. But wherever you want to position that new technology, it leaves a uh, positive impact with uh, potentially a negative side effect if you do not take that into account up front.

Speaker B: Right.

Speaker C: So I'm thinking a lot about that myself. I'm sure you are as well. It's not really a question, it's more of a rant.

Speaker D: I can share results on that as well. So I agree on that. Uh, but in general when you think about each new um, technology approach which occurs in the past, there's a kind of a step back in some kind of quality and engineering, uh, and so on. And after a while the optimization of that approach occurs again and you get the same level as you have in uh, the past and then it's getting faster to increase the efficiency and everything like that. And I think uh, what you have mentioned in terms of AI and the concerns of using that kind of technology during data engineering and data management, I expect as well that there is a kind of a step back. So I'm more related to the data engineering part. And when I'm thinking about AI, I'm thinking about that uh, AI could help users, especially the business related users to generate uh, a piece of code for that. Their purpose is to prepare the data. So you know that copilot, um, activities and helping users uh, to generate code by entering some words, uh, I need to have that and then the code engine generates the code in behind. So I think that will be the approach. But when you compare the results of that approach right now with the. So let's say a skilled person which is preparing the code for that. So it's a step back.

Speaker B: Yeah, exactly.

Speaker D: I think let's say a few months, uh, in the future, year in the future or whatever the time frame will be, the level of that generated code will be maybe similar to that what you have right now on a human, ah, produced code or maybe even better. So that's what I expect. So you will have a step back on that level and quality in some areas when you enter in a new technology. But if you don't do it, uh, you will lack for the far future with the capabilities of this approach.

Speaker C: Exactly. Yeah, I agree.

Speaker D: It's important to enter those things and learn also from the negative parts of that things uh, uh, that it's a step back, maybe for the first time.

Speaker C: Uh, obviously, obviously. It's like uh, it's like wearing glasses, uh, with uh, augmented reality.

Speaker A: Right.

Speaker C: Uh, it's the same thing. Why are big tech vendors staying in this game? It's not because they are necessarily very persuaded about their own products. It's simply because they need to take, they need to take the time to develop something and they need to accumulate the experience while they're doing that. So.

Speaker D: Right.

Speaker C: So we're seeing this effect of like promoting products that will most certainly never take off simply because they need to stay in the game, they need to collect the experience. And, and, and that is obviously something that is going on for generative AI at large. Right. You want to stay in the game, you want to experiment, you want to make mistakes simply because at some point we will crack the code and you want to be able to have the experience to go fast. Once you do that. Right. You don't want to jump on the train at that moment in time because that means that you've lost already. Right? Yeah, yeah, so I agree on that. I agree on that a lot, obviously. So you recently published a large analysis on data democratization in companies. Can you explain what it was about?

Speaker D: Yeah. So I think when uh, strong brands like Data Mesh and uh, even better the data product thinking emerge, many people follow them without thinking about the implications and the motivations for doing so. So that's why I wanted to use this article to create an understanding of why everything was centralized years ago and why currently exactly the opposite is the only viable approach. So um, to understand that major shift from left to right, uh, so I uh, think it's important to explain the whole data democratization journey, why we did it like this, uh, a few years ago and why it's uh, important to do it like that, um, right now. So that's why I'm writing that article, to have an understanding of uh, the whole steps and the whole democratization.

Speaker C: Uh, interesting. It has this big diagram in the article. Uh, there's a bottom layer and a top layer. So what do these uh, layers represent and how do you distinguish between centralization and decentralization in the diagram in the article?

Speaker D: Yeah, so I think the diagram is a little bit complicated but uh, the uh, whole data democratization journey over let's say decades, uh, from 20, 30 years ago, uh, till now. And the lower section of the diagram uh, is um, related and the upper section of the diagram is more the uh, business related area and the horizontal uh, bar Is the timeline from that past to the um state right now. So on the very left hand side of uh the diagram you see the first step where I started that journey and uh, with that centralized data warehouse approaches uh and even the reports at that time are built by IT related stuff uh and prepared uh for the business uh areas. So and uh after a while the first wave point of view of uh data democratization occurs with that self service BI tools which enables the business areas to build their own reports. They are even able to build additional KPIs on top of the data data for themselves uh to generate additional value out of that data. And in parallel or let's say a few years after um those big data technologies occurs and an interesting part happens at that time an additional more centralized approach so that data lakes using that big data technologies are queues. Because uh, at that time uh everybody experienced when you have a feature request changing the data warehouse it takes long time to change all those things in the data warehouse itself. Uh maybe it's a better idea to provide the raw data from the data sources to the consumers of the business areas the uncreated data inside the data lake. It's much more easier to um, pull them uh, towards the data lake and give the let's say the data scientists at that time the capability to access those data layers. But it was ungregated data. So uh. M. Uh it's not easy easy to handle those data again uh as a user you have to prepare and then do all the quality assurance of those things what you are using uh in this case. So that means that after a while the combination of that big data or data lake approach and the correlated data approach with the data warehouse architectures are combined in the lakehouse approach using the big data technology for storing the data. But you also using the curation of data during the in the course of the data warehouse architecture. So they are combining those both things to the lakehouse. It's also uh still kind of uh, uh I say more IT related or centralized approach for the data ownership and the data handling and the data management at the time. And the most massive step towards data democratization currently is taking place with that focus on data products whereby the last domain of that ID related stuff the data engineering, the data management moves now to the decentralized data teams um and they have to uh do the job of data engineering for themselves. So for me decentralization in that context means uh that the ownership or better the responsibility for data is now decentralized and that you handle Data as a product. So that means that you have to have a life cycle on that, uh, that you have to enjoy quality and the service level on providing your data to third parties or other organizational units in your organization. And those data products are now managed by the data product teams. And the storage of the data could still be on a centralized data platform. That doesn't matter. So it's of course it makes sense not to build uh, tons of different data platforms for each and every data product in your organization. It makes sense to have one common data platform with different data product teams working on that data platform, uh, using their data or managing their data for themselves. So that's what I want to point out with that uh, picture the different steps and each steps has a common reason why we did it in the past. So centralizing everything is the reason why you are ah, able to prepare the data so that it's ready to use and um, doing that same thing. And uh, in the data lake we experienced that this is not the right um, way to do it. So um, on the one hand business areas require more extra raw data, um, and then they get it and then they experience. Okay, that's easy to manage that. So what we have to learn from all those steps is that uh, if we decentralize uh, the responsibility of data, we have to care about that these teams are able to uh, prepare the data and uh, do all the jobs, uh, not only developing a solution, so that's the easiest thing from my point of view, but the daily operating and uh, ensuring of the quality of the daily operating and ensuring a dedicated service level of freshness of that data, uh, uh, that are additional things what data teams now need to do and you have to care about that they are enabled as easy as possible with those things. Because from my point of view the more people are doing data engineering, uh, the average of skill level is lower than in the past with the centralized, very specialized teams. So you have to make it quite easy to do that. So that's the major thing. But I want to point out with that picture.

Speaker C: Yeah, agreed, agreed. And I love that diagram. I encourage you to. I maybe remember it, uh, I maybe will remember it myself but if not uh, please remember to add it to the comments when we publish this episode

Speaker D: because it's a great diagram.

Speaker C: Um, okay, so finally I prepared a question that actually popped up when we were preparing this conversation because I think it's a very fun observation that you have. How mature do you think the implementation of data mesh approach is in the US And In Europe. And do you have any like I know you have a special take on that, a surprising take. Can you reveal what that is?

Speaker D: Uh yeah, I remember that. So the, the first surprising finding uh, regarding the differences between Europe and U. S was when it comes um, to the adopting of the data vault modeling approach um a few years ago or so. Although data vault as a modeling approach was founded by then Linstedt, an American guy, uh, the adoption of that data modeling approach is much higher in Europe than in the US and this was the result of a study I think a few years, uh, a few months ago. So at the end of the last year and I found at the time I also came across uh, a post of Joe Ray's uh, and he's asking for how do data modeling differ uh, by geography and the type of the company? And I'm looking at the answers to that question posed. Uh, it was clear, and this was the support of the results of that study that the adoption and the maturity of using that data modeling approach in Europe is much higher than in the US Uh so that's surprising for me because uh, it was founded in the US and the adoption m somewhere else is much higher than uh, in the region where, where it was founded. And a similar observation I have made about the adoption of data mesh, uh, the approach itself so also invented by Shamak Tiniani and American. Um, so I feel that there are far more concrete uh, implementations in Europe than in the US and um, as a European guy what I'm in particularly proud of is um, uh this week uh, the Data Analytics Summit from Gartner happens in London, takes place in London. So uh, and the slide where they named two European software vendors as standalone vendors for data product creation.

Speaker C: Mhm.

Speaker D: So I'm not really happy with that term, um, standalone vendor. I think what they really mean is uh, a comprehensive vendor for data product management and all the aspects of supporting the product life cycle in that area. So there are two uh, names mentioned at that picture. Both are in Europe and There are not 10 additional ones from US companies. There are only two and those two are European software vendors. So that makes me really proud of that. The European software market is really innovative and is approaching the needs and the requirements of that paradigm shift, uh what we are experiencing right now with that data democratization. And there are a large number of really excellent innovative products providers uh in Europe understanding that need for that uh, supporting that paradigm shift. And I know uh, that Senea, so your company as well is enhancing the data catalog capabilities with that Enterprise data marketplace capabilities following that need, uh, what companies currently need to support that. Decentralized approaches to find data products and allocate if they are useful for their use cases or build somebody and provide uh, and serve the data products via the data marketplace. So that's from my point of view really the right direction to go for. And uh, that makes me happy that European companies are on um, track with those things and doing the right things at the right time.

Speaker C: Well, first of all, thank you very much. Obviously, uh, as someone who uh, drafted, together with very, very intelligent colleagues, I have to say drafted the Roadmap for cinema, um, I'm obviously uh, thankful uh, for that comment. Um, and I think more in general. I agree. I mean I have not seen scientific studies of this, but I, I must say that this eternal like humble approach where people say yeah, we're from Europe, we're not as good at tech as the US or Asia. I don't really buy that anymore. I think, I think European software vendors are really great and I think we have a lot of great, great software vendors all across the continent. So I don't see no reason we shouldn't be proud of that. And so I agree in that, I agree a lot in that I don't think we're bad at tech at all in Europe. Obviously that's not uh, an excuse to not develop further. But I agree with you. Uh, and obviously it's quite surprising that the adoption of both Data Mesh and Data Vault um, is taking off in Europe with ah, such a substantial difference

Speaker D: than the U.S. yeah, I think the reason for that is that I think m the adoption of that data management and data engineering processes and uh, the skill level on that area maybe is um, useful for that uh, kind of development also in the software area and supporting that early adoption of those things uh, in our region.

Speaker B: Uh-huh.

Speaker D: And I fully agree a uh, um, couple of years ago it was very important to be a US based software vendor. And then a few years later you occur in the UK markets and a few years later you in the European market. So it was always kind of a step forward a few years ago, um, with those enhancements um, of your market. But currently there's no difference between being a European software vendor or a US vendor. You have all the capabilities to reach all the markets worldwide. Uh, and um, that's what you need to do. And I uh, think from my point of view we can really be proud of uh, our software vendors in our region and what they are uh, building right now and doing the right things.

Speaker C: Agreed. Agreed. Yeah. I think I let the time, uh, fly a little more than I ought to. But I got excited, especially here, uh, at the end. I think that was a very surprising and fun ending. But it's been a pleasure, uh, altogether, Gregor, having you on. I want to take a moment also. And thank you. Thank, uh, thank you, the listeners. Um, more and more listeners are joining. I'm very thankful for that. So thank you, each and everyone, for listening in. And, Gregor, it was a pleasure to have you on.

Speaker D: Yeah. Thank you very much. Ole. For having me on your podcast.

Speaker C: Stay in touch.

Speaker D: So thank you. Bye. Bye. Sam.