The B2B Podcast Index
Index
All categories
MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
MethodologySubmit
Best of:MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
An independent project byFame
All showsSearchBest episodesGuestsInsightsMethodologySubmit a podcast
Index/AI & Data/The Data T
The Data T artwork

Gleb Mezhanskiy on Rewriting the Migration Playbook with AI

The Data T · 2025-10-30 · 42 min

0:00--:--

Gleb Mezhanskiy shares why Datafold is approaching data platform migrations differently than generic AI tools. The core insight: while migrations seem like a pure code translation problem, they're actually about managing thousands of interdependent objects across multiple languages and legacy systems - something general-purpose LLMs struggle with without proper context. At Lyft, Gleb experienced firsthand how even digitally native companies struggle with migrations, taking four years to move from Redshift to Hive. Datafold's approach uses specialized AI agents rather than tools that enhance individual developers. Instead of giving data teams "better sponges" (Cursor-style acceleration), Datafold builds "dishwashers" - agents that autonomously handle migrations while looping in humans only for judgment calls. This matters because typical enterprise migrations involve thousands of views, stored procedures, and interdependencies across Informatica, shell scripts, and Teradata. Generic LLMs fail because they lack context about data content, testing requirements, platform best practices, and framework-specific configuration (like Coalesce templating). Datafold combines its existing data diff technology with agentic AI to provide end-to-end automation that includes both code translation and validation, delivering meaningful acceleration rather than incremental productivity gains.

Key takeaways

  • →Migrations aren't primarily about SQL translation - they're about rewriting thousands of interdependent objects across multiple languages and legacy systems that generic LLMs can't handle without enterprise context.
  • →Purpose-built migration agents outperform general-purpose LLM tools because they understand data content, testing requirements, platform best practices, and framework-specific configuration like Coalesce, which generic tools like Claude and Cursor lack.
  • →Data engineers will increasingly shift from writing every line of SQL to strategically designing data stacks and guiding AI agents, with AI handling construction, naming, schema design, and routine logic while humans oversee quality and business alignment.
  • →Enterprise migrations typically require tens of thousands of hours and external contractors; Datafold's agentic approach replaces human-driven work with autonomous execution, only requiring judgment calls from people.
  • →AI agents succeed in migrations when they combine code generation with embedded validation and reconciliation - the dishwasher model eliminates the manual tweaking and fixing that makes generic AI tools almost as expensive as traditional rewrites.

In this episode

  1. 1Introduction: Gleb Mezhanskiy and Datafold's Mission
  2. 2The Data Migration Challenge: From Lyft to Building Solutions
  3. 3Why Traditional Transpilers and Generic AI Tools Fall Short
  4. 4Datafold's AI-Powered Migration Approach vs. Claude and Cursor
  5. 5The Future of Data Engineering: How AI Elevates Rather Than Replaces Engineers

Mentioned

DatafoldCoalesceInformaticaSnowflakeSQL ServerClaudeOpenAIGleb MezhanskiyArmand PetrossianLyftAutodeskRedshift

About The Data T

Hosted by Armon Petrossian and Satish Jayanthi. The show scores 71.0/100 on substance across its analysed episodes. See the full show profile.

Guests

Gleb Mezhanskiy

Topics in this episode

SnowflakeRedshiftData migration automationInformaticaDataFoldCoalesceAI agents vs. LLMsSQL transpilersHiveData diff technology

Questions this episode answers

Why is using Claude or Cursor for data migrations insufficient compared to Datafold's approach?

General-purpose LLM tools like Claude and Cursor lack essential context about actual data content, testing requirements, platform best practices, and framework-specific configuration (like Coalesce setup). They can only incrementally accelerate individual developers rather than autonomously handling thousands of interdependent objects, forcing teams to manually fill gaps and still require developers to do the work - limiting acceleration to 15-20% instead of full automation.

What makes data platform migrations so difficult even for companies like Lyft?

Migrations require rewriting not just SQL but thousands of interdependent views, stored procedures, and legacy ETL objects spread across multiple languages (Informatica, shell scripts, Teradata) that cannot simply be transpiled. Gleb's migration at Lyft took four years despite being a digital-native company with great technology, highlighting the complexity of completely reimagining how data should work in a new stack.

How does Datafold's agent-based approach differ from traditional transpiler technology for migrations?

Transpilers are all-or-nothing (queries either compile or don't) and don't support modern frameworks like Coalesce with templating and configuration. Datafold uses AI agents that understand business context, handle framework-specific requirements, and include built-in validation and reconciliation - operating as an autonomous "dishwasher" rather than tools that accelerate human work.

Will AI replace data engineers?

No - AI will elevate data engineers to higher-level work like strategically designing data stacks, selecting vendors, and ensuring business alignment, while agents handle construction, naming, schema design, and routine logic. Data engineers shift from writing every line of SQL to guiding agents and overseeing quality.

What is the core value Datafold brings beyond just translating code in migrations?

Datafold combines code translation with data reconciliation and quality validation using its existing data diff technology, plus enterprise context about dependencies, framework requirements, and testing. This end-to-end automation avoids the gap-filling and manual tweaking that makes generic LLM approaches nearly as expensive as traditional rewrites.

Conversation analysis

Computed from the transcript - who did the talking, and the words that came up most.

Share of words spoken

  • Speaker B71%
  • Speaker A29%

Most-used words

data102code28stack19migrations18build17migration17tools17better14help14team13coalesce13makes13terms12customers12snowflake11super11

Episode notes

Data migrations have long been the costly, painful bottleneck of modernization, dragging on for years and carrying multi-million-dollar price tags. But AI is already flipping that script. Gleb Mezhanskiy, founder & CEO of Datafold, who has been deep in the trenches of AI-driven migrations, joins us on the Data T podcast to unpack how AI automation is finally making it realistic to move off entrenched legacy ETL tools. Drawing on his years as a hands-on data engineer and PM (including leading a massive migration initiative at Lyft), Gleb explains where migrations go off the rails: millions of lines of legacy code and lack of automation lead to massive human time sunk into reconciliation and QA. He shares how Datafold attacks the problem end-to-end, using agents and data diffing to translate and validate at scale, so teams can “lift and shift” quickly, then refactor with confidence on modern stacks like Snowflake and Coalesce. Beyond migrations, Gleb and podcast host, Coalesce co-founder and CEO Armon Petrossian, dig into how AI is reshaping data engineering itself.

Full transcript

42 min

Transcribed and scored by The B2B Podcast Index.

Speaker A: Typically, as we talk to organizations, pretty much every time the data team will say something along the lines of, hey, you know, we see Coalesce objectively a better framework than what we're using today. But what are we going to do about the 10 years of Informatica code that we have lying around? Or what about the 3,000 objects that was written by some consultancy over the course of the past five years who's now gone? And we have a ton of difficulty even comprehending what's been built and how it's been built. And so could you share a little bit more about what is unique about Datafold instead of using something like Claude or OpenAI when it comes to migrating that code from say, like a SQL Server to Snowflake instance, what are you doing differently? Or why should somebody use datafold instead of trying to go down that path? Armand. I'm Armand Petrossian, the CEO of Coalesce and this is the Data T, our monthly podcast on the news, trends and technology transforming data today. Welcome everybody. Uh, we'll get started right away. I couldn't be more excited to have you on Gleb, a close friend of mine and also another amazing founder in the data space. Uh, this is obviously the Data T. I'm Armand Petrossian, CEO and co founder of Coalesce. It's just me today. Usually Satish is here with me, uh, but he's out right now, so it's just us two. You're stuck with me. Ah, Gleb would love for you to do a quick introduction on yourself and the company and then we can jump into some questions.

Speaker B: Sounds great. Thanks for having me, Armand. Super excited. It's, uh, been great to see all the traction of Coalesce and what you've been building in the data space. Um, I'm Gleb, CEO and co founder of datafold. We automate data engineering workflows so we work with modern data stack technologies like warehouses and transformation frameworks like Cols to, um, help automate the manual work that goes and falls through the cracks of the modern data stack, including data platform migrations, of course, which is one of the, uh, most narrowest and tedious parts of the workflow. And super quick background on me. Um, I've been working on datafold for about five years now. Prior to starting datafold, I was data engineer, data pm, data scientist for my entire career, so wrote a lot of SQL, broke a lot of data pipelines, and um, get a chance to build three data platforms from pretty much from scratch. One was at Autodesk, which is a computer aided design company. Um, one was at Lyft, where I joined during the hyper growth phase and was uh, probably data person number 10. And by the time I left the team was about 300 people. And at Lev specifically, I led a massive migration from Redshift to Hive. Um, don't ask me why, why Hive, but, um, it was an absolute, absolute, uh, behemoth of a project. I was looking for the appropriate wording. I'm not sure what's the age rating? Not sure what the age rating here is.

Speaker A: He wanted to use.

Speaker B: Yeah. Um, but I. So as an architect of this migration, I made a lot of mistakes, um, in terms of how we approach this and how I approach that and we can talk about those. And also we just lack tooling to automate things and it took us over four years. Unfortunately, given that Lyft actually is a digital native company, which is very interesting. It just speaks to how hard that problem is. So I'm very excited to talk about, uh, this topic and do good justice to it.

Speaker A: I can't wait. It's one of my favorite topics. When we were behind the scenes, uh, the marketing person on our side, Alex, was like, this is Armand's favorite topic. So migrations are something that have been insanely painful, they've been a burden, it's been an opportunity cost for companies. It's never fun. I don't think I've ever spoken to somebody who was like, yeah, love the migration effort. That was a great time. Had a really enjoyable experience migrating from SQL Server to Snowflake or Teradata Databricks, whatever. Uh, it's an awesome focus for Datafold Lab. It's something that I'm personally really invested in, really excited about, to talk to you about. Um, can you, you know, just, just before we do jump into migrations in particular, is there anything else like that you can share on when you decided to go from Lyft to start your own company in the first place? Because usually there is a lot of conviction that's necessary there. There's some forcing function where you as a founder had to make a decision that was, I'm going to take a risk, I'm going to walk away from a high paying job and I'm going to go pursue this for some reason. Uh, what was that trigger point like for you? At what point did you say screw it? Um, I'm going for it.

Speaker B: Yeah, that's a great question, Armand. So it was interesting because Lyft is in an incredibly data driven business. I don't think there are that many businesses um, out there that are like quite like ride sharing. Uber is the same, a larger scale somewhat, but you basically have to balance two sided market in every city you operate in, every zip code, you operate in real time. And you are worried you're working on very thin margins and you allocate millions, tens of millions of dollars in incentives to passengers and drivers. And so the entire company ran on data, on experiments, on a B, testing on dashboards, machine learning. And it was really, really interesting to see and that was a big excitement for me. Joining Lyft is just seeing how really valuable and critical is data uh, to the business. And at the same time what was surprising is how difficult it was for us to do basic things like you want to do some really deep inside analysis of driving behavior and what motivates drivers to actually go and drive and what kind of incentives and kind of bonuses you can create. And so you want to do some basic reporting and create some reports to understand how much time every driver in the region contributes to. Um, going on m the app and drive and shipping. Something like that took like weeks. And the challenge wasn't necessarily writing SQL that anyone can learn pretty fast. It was all things around it. How you find the data, how do you test the work that you do, how do you reconcile what you get with other sources of truth, um, how do you deal with infrastructure that's overloaded. It was that realization over and over again and culminating in me driving the migration. Is that even a very advanced company and I think LIP had really, really good technology across the board with good infrastructure and great talent, you still can't really fully unlock the value of data because how much manual work exists in the data engineering workflow. And data engineering is really interesting because it kind of sits between the ultimate data consumers which could be analysts who do reporting and then create insights for the business stakeholders and driving decisions. Data scientists who are doing a lot of machine learning and experimentation and they have their own stacks and they focus on particular problems. Um, but the data engineering is all about how do I deliver the right type of data set to the right type of consumer on time with high quality. And that workflow sucked for pretty much my entire career across three companies. And after Lyft I had a detour at a uh, startup building remote control for autonomous vehicles. Also started as a kind of head of data there and build data platforms from scratch. And what was interesting, it was a 20 person company relative to Lyft at the Time was a 4,000 person company, but we ran into exactly the same problems. It was just very hard to get the data insights, very hard to build things, very hard to ensure data quality. And kind of seeing that over and over again at companies of different stages made me realize, okay, I don't think it's a problem of a company or an industry or a stage. We just need better tooling that help data engineers focus on the high value at work as opposed to on all the little details like migrations or testing or reconciling data between different databases. And so we started Dataflow with a thesis that um, we're not trying to do one thing. We try to create tools that help data engineers automate the workflows, the most tedious parts of them, and then through that increase impact and help companies unlock the full value of data because of that.

Speaker A: That makes sense. Yeah. I mean you basically are scratching your own itch from what you experienced as a data team leader. And that's always a great way to have the exposure and expertise to know what to build in the first place. And so it's awesome to see how the company's evolved over the course of the past few years. And it's been really great getting to know you on a personal level as well. I see a lot of passion, uh, in what you do. Obviously we share a similar vision on automation from, um, data engineering use cases and helping developers be exponentially more productive compared to any other alternative. As well, it's cool to see some of our joint customers using both datafold and coalesce in real life. Um, it's an awesome story. I love it. I know I mentioned earlier, migrations are something that I love to talk about and I think there's a huge potential in how AI can help assist it. And you've been on the forefront of that even from the point when we first met for dinner. I don't know how long ago it was, uh, as you were making that the primary focus for datafold. And so can you tell us a little bit about that? What is it that is unique about how you want to approach making migrations suck less for customers? And uh, what is it that got you to that point?

Speaker B: Yeah, absolutely. Uh, we've been building various tools for data engineers and one of our most famous tools is called betadiff. Essentially it helps you compare datasets or within database in the context of dev versus prod for testing or across databases. For example, your OLTP database source that you're streaming to soft liquid databricks, see if that reconciles and we had lots of customers that adopted those use cases. And one use case that we always knew is extremely painful is migrations. And what's interesting about that one is that if you improve data team's ability to, let's say test or do beta code reviews or catch anomalies faster or sooner, that is incremental value that you can quantify, making X number of data engineers at the company y percent more productive, which is great. And then for many companies, um, and for many our customers, initial customers, it's great roi. But migrations is a whole different beast because to execute a data platform migration takes tens of thousands of hours, which you as a data team don't usually have internally available because you already have a team that's fully allocated with quarter ahead roadmaps of building pipelines and dashboards and analysis. So you don't have that spare 30,000 hour capacity. So you end up um, bringing in external contractors, external talents, pay billable hours. And what ends up happening is that this is an incredibly manual process because um, if we zoom out a little bit, like what migration actually is is it's two parts. One is moving the data, which in this day and age is super easy. Even at billion row scales. You can bring in tools for moving data. There are many out there, um, you can do it in a week. But what's really hard is rewriting code because data is portable, you download it from your old system, you put it into your new system, you set up, ingest, you're good. But by the time you start the migration you usually accumulate millions of lines of code in um, store procedures and views and all sorts of legacy ETL drag and drop tooling that cannot be ported over to new stack. You have to completely rewrite them. And moving into stack like let's say Snowflake and coalesce requires not just translation of SQL into Snowflake, it requires completely reimagining how the data should work. And um, that is incredible amount of manual work if you just do it taking script by script. Uh, for a while we had customers who used our data diff technology for reconciling their migration. So they would actually embark on this very costly journey of rewriting their SQL code. And then they would use our tools to do final QA and user acceptance testing. And that worked actually pretty great. They saved a lot of time, but we always had the search that how do we help them do the entire project, the entire thing? The technology that everyone used for decades now to help migrate m the code is essentially Transpilers, there's been multiple companies started, there are some open source projects that exist for essentially taking SQL and then transpiling into different dialects. But the interesting thing is that technology is extremely expensive to develop because essentially writing a multi language compiler and SQL is simple, but it also gets pretty complex, especially if you go to transactional stuff and store procedures and compilers are hard to develop. You have to develop grammars and then it's kind of like all or nothing because SQL query either compiles or doesn't. You can have like a 95% transpiled query. It's very hard. And what's even harder is that these days people don't run uh, SQL queries. Plainly, right? The reason why COLAS is so successful is because you help customers take business logic that they want to express and ship that to production using your framework, your templating, your uh, configuration. And that goes hand with hand with code. And none of the transpiling technologies support that. And to make them support that would take a lot of time. Now obviously I'm saying all this and everyone's probably like, okay, AI LLMs, let's go. Yes, we've been looking at LLMs for a few years now, but for a while as we all know, they were quite bad at generating and understanding the code. And it's probably been around like a year and a half when we saw the kind of the third generation of GPT and CLAUDE getting really good with code comprehension, code generation and that allow us to actually start working on agents that would take the entire migration code rewrite and uh, start shipping that so that we could close the loop on um, not just helping testing, not just helping translation, but doing both within a workflow. And as we started working with early customers we saw that this generates amazing value for them because we're not giving them a tool that allows them to click and drop and maybe translate some SQL and then they have to package it into Cols framework and then they have to test it manually. We're just giving them a end, uh, to end solution. So if I'm an executive buyer, I'm a cfo, I'm head of data, I just can massively accelerate my timeline, which also is great for us because that means that we can um, get a lot of customers quick, deliver a lot of value, um, and grow the business fast. That's been the journey of getting to migrations.

Speaker A: Yeah, it makes sense. The reason why it's one of my favorite topics is because typically as we talk to organizations pretty much every Time the data team will say something along the lines of hey, we see coalesce objectively a better framework than what we're using today. But what are we going to do about the 10 years of Informatica code that we have lying around? Or what about the 3,000 objects that was written by some consultancy over the course of the past five years who's now gone? And we have a ton of difficulty even comprehending what's been built and how it's been built. Could you share a little bit about, a little bit more about what is unique about Datafold instead of using something like Claude or OpenAI when it comes to migrating that code from say like a SQL Server to Snowflake instance, what are you doing differently? Or why should somebody use datafold instead of trying to go down that path?

Speaker B: That's a great question. So there are a few things here. One, the tools like um, Cursor and claude, they're really optimized for general purpose software engineering. That means that when they work on existing code base they help you build incremental new things or sometimes actually bytecode completing new projects. But sure they kind of drive your productivity uh, as a developer and as we know for AI by 2025 the really key thing is context. Context is what makes LLMs actually very productive. In the example of a data platform migration, there is a lot of context that those tools don't have access to that includes what actually is in the data. How do we test the migration? What is the best practice for given platform? Um, how do I ensure that the model doesn't hallucinate and how do I make sure that when I convert a given store procedure I take into account the proper configuration of the larger project in colas and the setup of all the schemas and all the interdependencies. If you actually try that, then you will find that the tool helps you write code, but you end up filling a lot of the gaps manually. That's just by design of that tool. It's meant to incrementally accelerate your workflow. And now if we try to extrapolate that to enterprise scale volumes and complexity, it's usually not 30 informatica workflows, it's usually thousands. And what's more, it's kind of informatica, ah, workflows driving shell scripts that drive teradata big tech that drives stored procedures. So you have this massive code stack across different languages and untangling that one by one as a human is extremely difficult. So you may at best get maybe 15 or 20% acceleration with AI, then the other challenge is that if we talk about tools like Claude or Cursor, they are giving superpowers to a developer, but we still need developers to actually do the work. Which means that for a project that requires migrating thousands or tens of thousands of views or data sets, we need to stop the project with a lot of developers and you adapt back in the same problem of you have humans doing the work, which is really not fun and is really, um, difficult to do correctly, then the approach we take is if you imagine the developer with cursor, uh, you want to wash dishes, and Cursor is a really good soap and sponge. So you can wash faster and it's more fun, but ultimately you still need people to wash the dishes. And the approach we took is reimagining the flow into how can we, instead of saying, give people better tools to do the migration, build an agent that can do the migration for you and only loop in the person, the human, when the human judgment is required. But doing all the work itself, it's like a dishwasher, you can load all the dishes in and sometimes you will end up with, ah, uh, wooden, um, spatula that you can put in. Maybe that's where human steps in. But other than that, you just put the code in and you get the code out. And that's a very different paradigm that allows for much greater efficiency, much greater automation, much better, um, price and timeline guarantee that we can actually provide to, uh, the end user.

Speaker A: Yeah, that makes sense. I also would expect if somebody were to go down the path of just using something like Claude or Cursor, that they're going to end up spending almost as much time just tweaking and fixing the code itself, like refining that logic and validating it, that it almost makes it not even worthwhile to try to take that approach because you end up back in the same problem that you would have dealt with otherwise. Whereas for you, having the focus that you did with data diffs and data quality allows you to take that context and help accelerate it in a way that's actually meaningful for an organization. And so I think the, the approach makes a ton of sense. Uh, it's something that nobody enjoys doing when it comes to migrations. Being able to make that as simple as possible and as quick as possible is a huge pain point for pretty much any big company and also lots of other companies that are taking on data engineering use cases in general. And so, uh, yeah, it makes a lot of sense to me, uh, as far as AI goes. How do you see the roles of data engineers and data architects evolve as AI helps people handle more of these tedious tasks efficiently?

Speaker B: I think that's a really interesting question. I don't really buy into the whole notion of AI is there to replace data engineers. Um, I think that is extremely far from the reality. Even as data agents become better. I think that AI is here to reshape the job in a good way in terms of elevating the data engineer as a practitioner to a higher level of obstruction. Just like we can these days write SQL instead of Fortran and we can, um, use dashboards and frameworks like Coalesce as opposed to using stored procedures that are, I would say, far, uh, lower level, more moving pieces, harder to build, harder to maintain. AI is there to elevate the practitioner to focus on the core things. And so one of those is, um, how do we think strategically about what data we need as our organization? How do we, um, create the right stack of technologies for our organization, for our needs? How do we think about using different vendors and their strengths and having them play together? I would say that right now data engineers are still very heavily involved in constructing the business logic and creating data models, thinking about how the tables should be shaped. I think that as AI gets better and gets more context around the business and how the data is used, and I think that you have a lot of insights already into that with Coalesce and your new cataloging offer, because you can see the end side of the consumption that gives a lot of, um, ability for AI agents to help build the models and build them in a way that's really well aligned with the business. I think that takes away a lot of this. Like, okay, how should I name this column? What the definition should be, what columns should be in the stable, Right, like that can be actually automated. But then it doesn't mean that that engineer has nothing to do. They still need to oversee this process, need to make sure that all the Stack alliance, need to make sure that all the humans ultimately consuming this data are happy and their needs are met. But they, instead of writing every single line of SQL, could be guiding the agents with a much higher level of abstraction, potentially having human, um, niche language conversation with them, but not necessarily in a way like, hey, give me the revenue, but more like, how do we create the right data stack, the right pipelines for our business? So I think it's actually going to be very interesting and very fun to be data engineer over the next five years because you'll be able to get more done. Um, it'll be less tedious. You will spend fewer hours diving into rabbit holes of bugs and you will be able to deliver much more impact to the business, which ultimately is great for you. Career wise, no one really cares exact wise if you're there debugging the SQL error for five hours. But everyone wants to see great insights, uh, and data products that now we will be able to build more of with AI. That's kind of a high level take, but we can go into the technical bits as well.

Speaker A: No, I agree with it. I think the future I see is an ability for non technical users to be able to call agents to go and actually build out a lot of the objects associated with populating any dashboard that they might be trying to build. That doesn't exist yet. But it's not going to be perfect. We know that no matter what, it's not going to be the right level of data quality. It's not going to be able to run perfectly. So you still need a highly specialized data team to go in and refine that logic and validate it and make sure that it gets shipped properly. But the whole workflow should be a lot more autonomous and actually should be a lot more engaging with the consumers of data because they should be able to have a faster turnaround time on anything that they request. We're seeing that firsthand as you go through having a catalog that interacts with non technical users, as well as a development framework to automate the construction and maintenance of those objects. But it's still at the very beginning. That's still something that's happening in real time, similar to how you've approached migrations. I think that you see that light at the end of the tunnel because you're dealing with this day to day. You see it getting better and better as you go. And so by no means do I expect data engineers to go away anytime soon. If anything, good data engineers should be increasingly more valuable in the future. And the uh, value that people get out of data in the first place should grow exponentially just because the projects are able to be done in such a shorter span of time than they ever were historically.

Speaker B: Totally. And I would say that, um, I am super optimistic about AI impact on data engineering. But there's also a caveat that it's not like everyone for granted will be elevated in terms of their impact and quality of life. Um, I think the disparity between the data teams and specific data engineers who master the use of AI and also create the right data stack for themselves and for their teams and those who don't and really lag behind in adoption would be enormously big over the next few years. Like if you are still stuck on old tools like ETL and um, kind of old school, 20, 30 year technology versus if you're shipping on Qualess plus Snowflake that could easily be 10, 20 times difference in like if you have velocity, your amount of impact, organization and ultimately your career as well. So I think that's something that uh, really important to keep in mind for folks in terms of how they shape their career and what decisions they make day to day.

Speaker A: Yeah, that makes sense. What do you think migrations are going to look like in the future? If you fast forward two years from M now do you feel like it's just going to be something that's completely done behind the scenes? Datafl takes care of it and it's completely autonomous. Uh, or at least maybe uh, to what degree of effort will the data team need to spend on them on migrations in general in a year's time or two years time or three years time?

Speaker B: Yeah, that's a great question. Um, my thinking on this changes quite a bit as we go. What's really cool is that uh, our current data folds, current technology allows data team to do a migration from a legacy stack to modern stack pretty much without lifting a finger in terms of the code. So they give us all the legacy code, we rewrite it, we test it, we give them back. Let's say coalesce project with all the schemas defined, all the configuration done on Snowflake or done at Warehouse. That's um, pretty much the work we've done. Now they still need to set up the ingest into the new warehouse, um, repoint their ingest pipelines into the new warehouse and set up permissions and but usually the easy part but I think that there are certain important caveats in terms of that process and what uh is still hard. Um, currently we do migrations only in a lift and shift manner which means that we can be rewriting all the implementation code. Like if there was a store procedure with a for loop statement and a bunch of ifs, we would probably not carry this over to call us the Snowflake. We would rewrite it to be something much simpler but providing the same output so you get the same table. But now it's just way better code, way faster, way more maintainable. But also I think where the migrations go is that data uh teams often want to deal with a lot of technical debt around their data model. Because once you start the migration you see there's so much duplication out there, so many different sources of truth. And that's not something we can do right now in terms of how do we help build a better model data model for users that will actually require changing the shape of tables and maybe merging some of the business logic into a single place that ultimately would give a uh, much more maintainable code base because that requires a lot of context around the business that we currently don't ingest. But I think that's something that as our agent become better, as we get better in terms of ingesting kind of context from tools like coalesce, we will be able to help with as well.

Speaker A: Yeah, that makes sense. Uh, it's always been fascinating to me to see companies migrate from whatever large scale on premise system they have into the cloud and taking the lift and shift approach when they know that they're not super content with what's built on the on premise platform versus being able to redesign some of those components. And I think that's just a, it's just a, it's a, it feels like a necessary evil for some companies like just to get off of whatever they're using as quickly as possible. They're going to neglect the opportunity to refactor and redesign some of that previous implementation to be optimized for the future. And I think that what you're solving should in theory help them actually be more thoughtful about migrations given that they're going to buy back some time, they're going to buy back some energy to be more thoughtful on what the future looks like from that migration.

Speaker B: But until then I still believe that lift and shift is the best approach because what you want to do, the last thing you want to do is try to redesign everything and seek consensus in your organization and have the project take years. You should get over to a new platform M as soon as possible and then you will have all time in the world to redesign your coalesce data model because your developer it would be like 20 times better. So at that time you can safely do it. Um, I think is the best approach

Speaker A: to migrations that makes sense. The only time I would suggest otherwise is if there's a specific portion that makes sense to go live first that needs to be, yeah, redefined or reworked. But otherwise I, yeah, I agree it makes, it makes a ton of sense. Um, any predictions on how, any other predictions on how AI is going to reshape data infrastructure or data engineering workflows this could be broader of uh, course than just migration I would say there's

Speaker B: like, there's a, Everyone kind of dreams of the holy grail of data, at least in terms of bi, where users just ask natural language questions and they get amazing insights. And getting insights is like not hard. You can get this all day long. It's your question is like what's your confidence that those are the right kind of insights in the right context for your role and for the question you're asking me for a business, which is extremely hard. And I think that between the sort of ultimate self serve uh, and the statewide, now there's a lot of room to introduce AI into your workflow with tools and things like mcp, for example, just having the ability to um, give any AI agent that you use, be that code a cursor or codex, the ability to interact with your data stack, run queries, run um, coalesce models and jobs, goes a super long way and can really improve your user experience, developer experience, but it requires a little bit of tinkering. It's not like it's necessarily set for you. You have to uh, find your way into it and experiment. And what I found is also uptown with AI, some people are really eager to try and they just go and try things. Some are skeptics. And if you approach your AI experimentation with the mindset like okay, I'm going to give it a try and if it doesn't give me the perfect answer, I told you it's not great, of course it's not going to work. You just got to be frustrated and you will reinforce your bias. But what I found personally is that uh, yes, it's not perfect but if you keep pushing AI in terms of what you want it to do, you'll be surprised every time you push it further. And then sometimes it will grossly uh, underperform. Sometimes I had a plot code, run a couple of queries, get results that are thousand times different and say, oh, it's actually identical and like what? And then at the same time it helped uh, our team debug issues that could easily take like couple of days in about 20 minutes just by being able to interact with um, now in our snowflake environment. So that's really I think is the state we're in, is the state of adventure. And so I encourage everyone to try to build their own tools and definitely try everything that Armand and the team are putting out in terms of AI tooling. It's not going to be perfect but um, it's Definitely gonna give superpowers, but you have to lean into them and give them a chance and see, um, how they fit in your workflow. So I think that's like, curiosity and trying to improve things is gonna go a long way.

Speaker A: Yeah.

Speaker B: That's awesome.

Speaker A: Well, what last couple questions I got for you. We always do annual predictions for the upcoming year, and it's kind of funny like we were looking at the predictions for 2025 that we did in 2024 and it was all things that came true, pretty much. And so I'm curious, from your perspective, any specific predictions that you're looking forward to in 2026 that you feel like are inevitable for the data and AI space?

Speaker B: Um, well, I think that couple things come to mind, but I'm also curious what dimensions you want to push this into. But if we take um, kind of the tech stack, um, I think this year there's been this idea that modern data stack is not the thing because we are already on modern data stack. Everyone is modern. So sort of like, why call it modern? Um, and, um, I think Tristan from BVT kind of put out as a, as a, as an opinion, and I think that it's partly true, especially in certain segments of the market. Like, you know, digital natives, I think are running like, really well. But if you go to figure, Fortune 500 enterprise companies, I would say in my experience, 90% of their workflows are running on super scary, super uh, slow infrastructure. And there's so much room for them to get like, what we take for granted as startups. Right, for sure. Um, and I think that's just kind of an invisible iceberg because we don't really hear a lot from those teams or maybe even see them. Um, but the scale of that massive modernization that's about to happen is enormous. And I think that the other thing I'll say is we used to think about kind of Data Stack as its own thing. Okay. We have Warehouse, we have um, a orchestration data intelligence, ah, framework on top. Then we have BI tools, and then we have other things like ingest. But now because of the AI and because how data hungry the AI is to build applications, we need to bring a lot of things together. And so I think that the boundary of the data stack is going to be far more fuzzy. And we're already seeing this with databricks pushing super aggressively into AI and um, into agent building and into transactional databases, and Snowflake pushing super aggressively as well. And some of them have their own strengths, but it's Very clear that enterprise buyers, enterprise teams want to think much larger when they think data these days than just like I'm going to run a bunch of analytical queries. But I think that's also quite interesting for us as kind of smaller players in the space is because we also now will have to think broader than just analytical tables. Right. We have to think about how our customers are going to be building agents and what it means in terms of the client plans they're creating and how the data flows in and out. And I think that creates a lot of mess. But it's also great because now people will be able to do more with data. So a couple of predictions there.

Speaker A: Yeah, yeah, fair enough. I do agree with you. I mean I think we can get carried away thinking that everybody's on a modern data stack. But in reality, you go look at any Fortune 1000 global company in the majority of their workloads are still on premise today, even though they do have adoption of cloud data warehouses and they want to move there. Like even the most cutting edge Fortune 500 customers of ours are still moving off of Oracle, they're still moving off of SAP, they're still moving off of Teradata. And so uh, it's still a big problem for people to solve as they look forward into the future. Is there any closing thoughts or advice for the audience? Gleb, from your perspective, you've had a great career, you started a company, uh, you've found a really nice niche here with migrations, but you've been through a lot. Anything that you'd like to share as closing thoughts here?

Speaker B: Yeah, I would say a couple of things. Just reflecting on my journey. I actually come from a non technical background and I studied economics in college and then I learn computer science through um, online course from Harvard called CS50. And then I then I got into grad school for computer science and kind of started the tech career from there. But until then I kind of like didn't really know what a function is. So uh, I would say coming from this non technical to technical background was really a moment left like getting superpowers. And I think these days it's even more important for all practitioners like you know, SQL, it's great you're data engineer, you know with Python, that's great. I think if you master programming m invest in understanding how to build AI, um, that will give you 10x superpowers as well because of how powerful that technology is and you will not necessarily always build things for yourself, but you will have a really good Intuition for how to integrate the best AI tools into your workflow, into your company, and you will be able to make a ton of impact. And the other thing I'll say is, um, the success of my career today pretty much all amounted to my laziness and dissatisfaction with my workflow. I just really hated how slow I was and how much time I spent on what I thought were mundane things. And I started building tools first for myself and then for my colleagues and then eventually for customers. And I think that now, again, because of AI, because the difference in productivity in, like, the stack that, uh, let's say like Coax and Snowflake gives you versus the legacy, like, ETL tooling that, um, gives even more reason to constantly question, am I doing the right thing? Am I having the right tools, the right workflow? Is my team using the best, the best tooling? And then teams, um, like Coloss are shipping super fast. They're adding new functionality, they're acquiring companies and use cases and, um, like, okay, you are on Callisto. Great. Now you want to take full advantage of their features because they are going to be shipping a lot of AI things and are shipping a lot of AI things, and you have to try and really give it a push. And if you just stuck with your. Whatever you've been doing for the last three years, I think you'll quickly find yourself far less competitive and less impactful. So these are my parting thoughts. So, uh, quote the great. Ah, stay hungry and stay, Stay curious, I would say.

Speaker A: Yeah, that's awesome. Well, it's been great having you on Gleb. Um, it's always a pleasure connecting with you, uh, both on this as well as just in general. And this was a really fun conversation, so hopefully everybody enjoys it as much as I did.

Speaker B: Yeah. Thanks so much. Your mom.

Speaker A: Cool. Thanks, Gleb.

Related episodes across the Index

Other episodes covering the same guests and topics, from across The B2B Podcast Index.

  • Why Your Marketing Attribution Breaks on MarketplacesMarketing Analytics with Fexingo · on Snowflake92 / 100
  • The Real AI Advantage Isn't What You ThinkAI Proving Ground Podcast · on Snowflake91 / 100
  • How do you turn AI coding chaos into a repeatable playbook?The Stack Overflow Podcast · on Snowflake86 / 100
  • Is Your AI Actually Worth What You're Spending? with Parker ConradStrictlyVC Download · on Snowflake86 / 100
  • 225: The Fall of CRM gravity (The Dungeon of martech architecture, part 1)Humans of Martech · on Snowflake80 / 100
  • 044 - Synthesia: Data Director - Why Data Teams Should Stop Trying to Be in Every RoomThe Stacked Data Podcast · on Snowflake79 / 100

More from The Data T

All episodes →
  • Durable Growth in the Age of AI: Data & Marketing Lessons From Scaling Braze71 / 100
  • The AI-Ready Data Team with Erik Duffield
Explore the best B2B AI & Data podcasts →
All The Data T episodes →