The context engineering playbook (Claire Gouze)

Speaker A: Welcome to the Analytics Engineering Podcast, featuring conversations with practitioners inventing the future of analytics engineering. In this episode, I speak with Claire Goose, co founder and CEO of NOW Labs. Context is everything in data. Right now, it seems like users everywhere are looking for the same thing. A single API where you can ask a, uh, natural language question and get back a reliable answer. With this API, not only can you build conversational analytics, which is great, but also essentially any agent that needs to connect with your organization's data. The models are good enough to deliver this. The missing piece is context. Now, lots of people are talking about building context layers and hiring context engineers. Claire and her team are rolling up their sleeves and doing it. They've authored a context engineering playbook that has specific suggestions on how to build your own context layer and how to create evals. On top of it. They've built a community who are all getting their hands dirty and learning together, doing the real work of context engineering. And of course, they've built tooling to make it all easier. I appreciated this conversation for its pragmatism. Rather than talking about context, we should all get to work engineering it. Claire's playbook is a great place to start. Without further ado, let's get into it. Claire, welcome to the Analytics Engineering Podcast. Great to have you.

Speaker B: Thank you. I'm happy to be here.

Speaker A: I've, uh, been preparing for this episode and I've been looking forward to it the entire time because I saw, uh, in your, in your background, you are a, uh, I think you're from France originally. Is that right?

Speaker B: Yeah.

Speaker A: You're spending your time in, uh, in Silicon Valley these days, I think. But M, I, I, I. This is a weird thing, but I just, like, love French accents. And so I've been looking forward to hanging out with a French accent for the next hour.

Speaker B: Okay. But I'm m trying to erase a bit this French accent.

Speaker A: What? Why?

Speaker B: Because, you know, like, trying to fit in in the tech scene, you know,

Speaker A: but there's, there's a good little cohort, uh, of French, uh, folks in the data and AI space. I mean, Clem is, uh, you know, Michelle from Airbyte. It's a good scene.

Speaker B: That's true. More and more people. So, yeah, maybe I should not erase it.

Speaker A: Um, well, thanks for, thanks for coming on. Uh, this is your first time on the show. I want to give you a chance to just say hi, talk a little bit about your story. How'd you get into data? What are you up to these days?

Speaker B: So my background is a bit unconventional for Data people because I graduated from business school but I realized I wanted to learn technical stuff and I really.

Speaker A: You and me both. Um, sometimes, sometimes it could be uh, relevant. Right.

Speaker B: I think it just was very rare when I did it because like I graduated business school 10 years ago and I think that was not such a ah, frequent thing. Especially like if you want to go into data science, like ML stuff and not only like analytical stuff. Um, so yeah, I think I was like the first hire from a business school in the consulting ah, firm where I got to, because I got to BCG Gamma, which is like the data science, uh, department of bcg. And yeah, there was no business school graduate there. Um, so yeah, that's how I got into data. I wanted to learn coding and solve m, uh problems which was perfect. Um, in data. Um, so yeah, I did like 3 years of consulting so working to build ML models for BCG clients from like forecasting, personalization, um, optimization, these kind of things. Um, and then I got into a startup because I wanted to build my startup one day. Uh, so I joined Sunday, which is a scale up doing uh, QR code payments for the restaurants and they had nothing. Like they had their production database plugged into metabase and I needed to do something out of that.

Speaker A: Um, how big was the company? And they had their metabase plugged into production.

Speaker B: It was only 20 people so it was very.

Speaker A: Okay, okay.

Speaker B: It was the start and then like one year later we had like 300 people so we needed to go fast like building this data stack and giving like access to data to everyone in the company.

Speaker A: And when did you join there?

Speaker B: I joined like it was seed stage. Like it was like three months old company. It was. No, no, like um, uh, what year was like 2021? Yeah.

Speaker A: Okay, so, so you, there were tools that you could use to set up a pretty mature data stack pretty quickly at that point.

Speaker B: But we were, which was funny is that I was coming from the consulting space and you know, like BCG likes to do everything custom made for their clients. Like otherwise you don't hire bcg. So like when I arrived in this setup I was the only data people and I didn't know data engineering, I didn't know the tools of the data stack. So I just built an ETL by myself. I built Ingestion Pipeline in Python to get Salesforce data. I built uh, Python Transformation layer and then I started talking to people who told me, but there's a tool called DBT or AirByte that you could use. So then I had to Migrate all of my custom stuff to this existing stack. Yeah, that was a hard learning.

Speaker A: It's nice to have the experience of building that stuff yourself and then realize like easy peasy. I can uh, uh, sometimes I think companies uh, that get further down that path, it becomes a little bit of a dead end because then you're so invested in the custom stuff that you built that it becomes hard to kind of pull yourself out of it and adapt more standardized tooling.

Speaker B: Yeah, yeah. I think even last year I talked to like big scale ups that had their in house DPT framework I would say and they were trying to transition to DBT was taking like a year because like first like people were really caring about this in house framework and they didn't want to let go.

Speaker A: You get so invested in it. Yeah, um, yeah.

Speaker B: And then you know, you need to do the transition and nothing has to break. So that takes a lot of time.

Speaker A: So bring us up to the present.

Speaker B: Uh, so present. Yeah. So after um, my last startup I wanted to build my own startup and bring some innovation to the data world. Uh, especially because I started thinking about ideas like two years ago. It was like the start of you know, all these uh, AI tools for developers. But the data word was a bit left behind in terms of uh, AI tooling. Um, and so currently I created Now Labs which is an open source analytics, uh, agent, uh, co founded uh, IT with Christophe Blefari who's uh, OG Data Engineering influencer. Um, and yeah, basically we are an open source framework so that any data team can deploy analytics agent on their own infrastructure with the context they have DBT model, um, context from the warehouse, from their business knowledge etc.

Speaker A: Which is, and, and the work that you're doing at NAO is definitely why I uh, was excited to get you on here because I think you, the things that you're thinking about right now are right at the center of the things that the whole industry is professing to care a lot about. Although I think at times maybe not showing as much progress uh, as folks ah, would like. Um, but just because you mentioned bcg, uh, uh, our origin stories are similar in some ways. I um, went to business school, I started my career at Deloitte Consulting, um, that preceded the data science wave. So at Deloitte the work that I spent my time doing was building web systems for uh, custom web Systems back in 2003-2008. But um, it is just a fascinating question, uh, how are uh, coding agents and AI in general uh, going to impact that Entire industry. Uh, do you have friends inside of BCG or in that world? Do you have any sense of, um, how is this playing out?

Speaker B: Um, you mean for the consulting world?

Speaker A: Yeah. Do we still need consultants to do all this stuff, or are people going to do it in house? Or is the profile of a consultant changing? Or. I think it's a big, wide open book.

Speaker B: Yeah. I mean, I think there's two sides of the story. I think definitely, like consulting firms now, they can do more with less people. So I think that's one issue for them, which is like, they don't need so many junior people anymore, but they need senior people in the future too. So how do you make this work? Um, but I think we are at a time where things go very fast in terms of AI, and I think enterprise. I mean, BCG clients cannot catch up so fast with everything that's going on. And I think that's where they come to help. Like, what is the thing to put in place, the highest value case for AI in their company? I think many companies don't have this competence, um, internally, and they need consultants to help with that.

Speaker A: The change management part of it is obviously huge. The question is like, you know, does the existing business model, which often, uh, prioritizes charging for large buckets of hours from mostly people lower on the pyramid, um, does it continue to support all of that? So I. I don't know that anybody knows the answer to that quite yet. Unless you're like, uh, the. The ops people at the very top of these firms that are like, running the numbers. But it'll be interesting to see it play out.

Speaker B: Yeah, I mean, I guess you would mainly pay for the partner salary anyway. So, like, if there's only the partner left, like, yeah, you'll only get a 10% discount maybe.

Speaker A: I don't know. Yeah, right. Okay, so Nao. Um, give me. Give me the background. Uh, what was the problem that you saw? Uh, how. How long have you been on this journey? Where. Where are you at today?

Speaker B: So we actually started now two years ago, and the first.

Speaker A: Do you actually. You don't say it. Nao. You say it now?

Speaker B: Yeah, we say it now.

Speaker A: Got it. My fault.

Speaker B: Yeah, yeah, it's fine. Like, uh, I. Everyone has their own transition.

Speaker A: Um, um.

Speaker B: But yeah, so two years ago we started that and we were more focused in the, um, like, coding parts at first because, like, we called like 80 different data teams. And the main pain point that they were saying back two years ago was like, it just takes too much time to ship code like to ship DBT models. If like the business needs a new analysis, we need to build a DBT model. It takes time and all. So we were focusing more on like how we can accelerate that. So that's why our first product was to build a cursor for data. So your ID plugged to the data. Everything is in the context of the agent so that DBT model can be created, uh, faster. Um, and so when things started to accelerate with MCP and all, we saw that cursor and CLAUDE were managing this uh, problem and we're trying to solve well, honestly. Um, and that then the new thing that data teams wanted to solve was how can we make anyone at uh, the company use agents on the data? And really we realized like this was on the roadmap of everyone. And which, the thing that was funny is that we saw a lot of companies trying to solve that, like warehouses, the BI tools, like text to SQL startups. Like everyone was trained to go for this problem, but when you talk to data teams, they didn't know where to begin and they didn't have any solution. So that's kind of like what we tried to understand. Like why was that? And I think that's how we came up with the uh, open source approach because we thought like when you invest in a uh, closed source solution, um, well first you don't really know what's going on in the closed source solution and eventually you need to pay for it, maybe even before you have actually proven any value from it. Um, and our vision was to give an open source framework very transparent in the way that the context of the agent is built so that people can invest in it, tweak it to make it work. Um, and then when they see the value they will scale to a more enterprise license or whatever. But at least they can try in the open source version until it works.

Speaker A: Uh, first, just where are you on the journey? How many, I don't know, how many people, how many GitHub stars, what, whatever forms of traction you want to share.

Speaker B: Yeah, I mean we have like a thousand, three hundred stars, something like that after like six months of launch. Um, and we have, I would say like 80 companies with now in production.

Speaker A: Uh, okay, cool, that's great. Yeah, yeah, Those, those first 80 are going to be the hardest.

Speaker B: Yeah, well they are here.

Speaker A: Yeah, I tracked your, your um, transition from the cursor for data strategy to the context, um, layer strategy. I thought that was a good move. There are many startups who have talked about um, agentic data engineering or Cursor for data. This kind of value proposition and I'm not sure that I buy it as a startup. It is um, you know there, there's certainly like these kinds of things will exist, but I don't know that like there will be a discret company that like just gets it start doing that thing. Um, what, what signals did you get internally that said like, hey, maybe we want to do something different.

Speaker B: I mean, I think we just want to work on what excites people, you know, and data people just were not excited about how they code. They are excited when they can help business users and when they give a solution like a nice dashboard or an analytics agent to the whole company. They are valued by people and that's what we realized. I think data people kind of have the trauma of being seen like a support team and if we can help them being valued by the business teams, that's like the greatest value we can bring them.

Speaker A: I want to get deep into the, the context engineering playbook that you folks have, have uh, done such a great job of laying out. But before we get there, one last question. Um, uh, I just got back from Snowflake Summit and uh, the context layer are like context for agents, uh, was probably the most uh, widely talked about thing on the vendor showroom floor. Um, that is a blessing and a curse. It's a blessing because like people clearly care about the, the problem that you're working on. Um, it's also got to be a little bit daunting. Um, you know, you're a very early stage startup, um, and it seems like everybody in the data space just wants to talk about this thing and compete uh, with you for attention on this thing. Um, how are you thinking about this? Are you, does this make you lose sleep at night? You know, just generally. Are you excited to be in the center of this topic?

Speaker B: Uh, yeah, I think I'm excited uh, because I think we have a kind uh, of um, advantage which was we've been working in the IDE space before. So we've worked a lot around context and we've put the knowledge we've had with IDE in our current um, product because our context layer is a file system because we realized it was working very well in the ID and we did the same. So I think we do have a product differentiation and also like that's so interesting.

Speaker A: I want to hear that also. But like, I would not have, I would not have picked up on the idea that like the key to success in context is like very similar to the key to success in ide, because you, like, in both cases you need to be operating on top of a like, incredibly, you know, complete, um, context window. And that, uh, that is a, uh, surprising, uh, and really interesting answer.

Speaker B: Yeah, I mean that's what was working well, you know, with cursor or cloud code. Like, agents just have like a bunch of text and files and they can just like grep and search things and that goes very fast and that scales very well. So that's what we learned from it and we wanted to put the same in our product.

Speaker A: So. Sorry I cut you off. You said there was an also.

Speaker B: Yeah, and I think also, like our uh, vision is that, you know, like the context piece is actually quite simple. Like as I said, it's a file system. And what we want is to bring like. Yeah, I, um, mean, some people want to make it more complicated than it is. You know, like ontology semantics. Like, well, we want you to start simple so that the, you know, like, first step is not too high for you. And so what we are bringing is something that gives you a lot of control still on your context. Like, basically we help you create your context as a GitHub, so the GitHub is still something that belongs to you and that doesn't lock you in anything. And what we bring in value is some pricks on top of like, evaluation of like managing of your permissions of ui, et cetera. Um, so yeah, we're really playing on this open source beds of like giving the context ownership to the data teams and not to our proprietary solution.

Speaker A: You know, sorry for distracting, uh, you there for anybody who's listening on audio and not, not video, uh, I, uh, when you started talking about like, you know, let's not overcomplicate this thing, I was laughing, uh, because I completely agree and I, I see this happening over and over again where vendors want to sell something when it comes to context, but it's not super clear to me how much there is to sell. I mean, so much of this is like you said, files in a, in a git repo. Or it is, um, you know, you're gonna ask a human to write a new markdown file or what. Like, uh, do you think that there is a, a product to be built here? I mean, there certainly is work to be done like practitioners. I mean the idea of a context engineer, I think clearly has legs. But do you think that there's a product needed in this space or is it just a set of best practices

Speaker B: being value added of our products in some way? Is the evaluation part and the governance part. Because like, you know, every data team I talk to that use now, they have a different approach of what, what they put in the context. Like some um, have like a full data documentation, some have a semantic layer, some don't even like create context on the tables. They have like, it's kind of like varying depending on their context. But what they all do is that they use our framework to test the reliability of the agents and to test in the long run, you know, like in cicd, how it's going to evolve. They also look a lot at uh, conversation that users have with the uh, analytics agent, which helps them see like what are the topics of interest and how they can improve it. You know.

Speaker A: Okay, great, I love this. And when I started getting into your help docs, uh, the thing that got me the most jazzed was uh, your evaluation frameworks. And uh, because at the end of the day it is actually quite easy to stand up a conversational analytics agent, the question is, is it giving you good results? Uh, and over time are those results tending to be higher quality or lower quality? Um, and so, you know, it's just, it's a super critical part of the puzzle. Okay, so you folks wrote, and maybe like you personally, I don't know, uh, a playbook around context engineering. Um, can you share just at a high level, what does that playbook involve? I think that the reason that I was so excited to see it is because oftentimes conversations about context engineering and air quotes around that uh, want to exist at this 30,000 foot level of like, oh, let's do context engineering and then you get the details and nobody seems to have anything to say, but you actually have a detailed prescription.

Speaker B: Yeah, and I think my prescription is more around the method to use than what exactly to put in your context. Uh, because as I said, it's not going to be the same thing for a startup with 10 tables or a, uh, big enterprise with seven table. So this playbook is about what is the method to build the right context for you? Um, and so basically the principle that I give is to really try to be focused on some first use cases. Um, so for example, who is the team that asks you the more analytics or do you always have a lot of analytics on the main company's metrics and you want to solve this problem so just like reduce the scope you want to solve to maybe like 10, 20 tables and already like try to make it work for these uh, tables and then it's a lot of iteration. So What I suggest is just like plug what you have, do your testing and that gives you a first baseline of what is the reliability of your agent on the current context that you have built, which is usually like, just like your DPT docs I would say. Um, and then you see where the agent fails and you iterate, you see if you want, if you need to like, uh, redesign a bit, your data model if you need to, add more documentation, if you need to, yeah, profile your table and put that in the context. Like that's up to you. But just like you do it in an iterative way.

Speaker A: So start small, set up the scaffolding, make sure that um, you've got your eval framework running. Um, do, um, how is this process working in practice? Does this feel natural for teams, um, to get off the ground with what are the hurdles that they run into?

Speaker B: Yeah, I think so. In practice I would say people are not so like strict as I am. I would say like, I think the natural thing that people do is just like they plug their context to now for example, and they'll just test manually stuff and because when you test manually stuff you have a feeling if it's good or not. Um, but I would say like, usually Data team used to stop here, but that's also why like they were scared to roll this out to the whole company. You know, like maybe they would add one or two data champions to test it as well, but they only stop with a feeling of it's right or wrong, but not a real number to give. So then what I see is when they reach this level now they can put this testing framework in place so they can really know like, okay, on my 50 most important questions, I have 90% accuracy and it's going to stay that way in the long time. And that's like I think really reassuring for data team.

Speaker A: Okay, and where do the 50 questions come from? Do you just like, as a data person, do you kind of like guess at them and then um, you put that in the testing set, um, because ah, you have to bootstrap this somehow.

Speaker B: Well, I guess it depends on your level of uh, uh, I mean either you just like you already have in mind the most important questions and you maybe have the queries in your bi tool somewhere so you just like copy and put it in the test framework. Or you can also use a skill we have built that will build some tests for you, like looking at the main metrics of your tables. We're going to uh, suggest some questions that are really key to test and Then I still suggest that you review them, but at least that gives you a first basis and ideas of uh, tests you can put.

Speaker A: The reason that I was thinking about that is that I just recently spoke to um, Ian from Ramp, um, and they have this kind of process up and running, um, and they call their agent Ramp Research. And one of the things that they found as this got widely adopted inside of Ramp is that the questions that the data team had historically been asked were of type A. And the questions that RAMP Research was being asked, uh, asked included uh, type A, but also now included this like significantly larger population of questions that like the data team had really never been asked. The, the example that he gave was um, how many dentists use RAMP today? And like that's just not like, not a question that they used to get asked. So there is this um, certain amount of like the unknown as you deploy these types of agents, uh, into your company.

Speaker B: Yeah, yeah. But that's also really interesting because so yesterday we did like a community hour where we had like now users talk about like their experience. And what was really interesting is that they said they were looking a lot at like uh, people's conversation with the agent and they were discovering like analysis they were trying to do, but the data was here somewhere but never exposed to the business users. And so they realized there was a demand they didn't know about thanks to the agents. And I think that's really interesting because we could not do that with regular BL tool before.

Speaker A: Who does this work? Uh, if context engineering is I guess one, is that a job title at some point? Uh, or is it just work to be done that somebody has to do? Um, and if it is work to be done that somebody has to do, then do data practitioners do it? Is it everyone's job at an organization? How do you see this playing out?

Speaker B: Mhm. Yeah, I think it is a job M title. I actually know data people who were renamed context engineers. Yes. Uh, so that's actually happening already. Ah. And I think data teams are the perfect fit for this job because if you think about it like analytics engineering was about translating business knowledge, gathering business knowledge from business stakeholder and translating that into a more um, structured way and technical way. And I think that's exactly what context engineering about. It's like context is just knowledge in the company, but you want to put it in a structured way. Uh, you want it to be optimized so that it doesn't explode the token costs and you want it to be source of truth the same way as you want it to have metric, source of truth. So like data teams are already used to all these concepts and they can do it with context the same way as they used to do it with data.

Speaker A: Yeah, I totally agree. I think one of the fascinating things that I'm hearing over and over again from data leaders these days is that um, the roles in the data Org are flattening. Um, everybody's kind of expected to do everything. Uh, there's less of a separation between an analytics engineer and a data scientist and a data analyst. Um, and I would imagine that um, as technical skills become less of a differentiator because we all now have access to the same coding agents, the thing that the data team is really good at is um, understanding how um, the business uh, translates into data, how to um, you know, encapsulate that knowledge so that both humans and agents can, can get access to it across the business. Is that you think that's your mental model as well.

Speaker B: The role of data teams before was to like give access to data to everyone. And I think now they have just like to give access to agents so that they can serve um, the rest of the teams for them.

Speaker A: There's probably two big sources of context. Um, and I think that your docs actually specifically outlined this. There's kind of the machine generated context that agents can get from uh, your data warehouse and query logs and that type of metadata you can get it from. Dbt, uh, manifest all these kinds of things. But then there's human generated context, um, that you know, has to date kind of existed in all of our, our brains as tacit knowledge. Um, when you go to capture that human context, um, where do you put it? Does it matter? Like can you uh, if I have this in a series of markdown files in my DBT repo, is that equivalently good to a bunch of notion pages or uh, a series of Google Docs? Do you have uh, an opinion on the specific details of how that stuff happens?

Speaker B: At least in our framework, eventually everything just ends up a markdown file. Somehow it doesn't really matter what is the initial, uh, support. I would say like what matters is that the context is that the place where it's going to be, um, maintained. So for example, like some people ask me, should I write my documentation in the context of the agent or should I write it in the DBT docs? And I tell them like in the DBT docs because it needs to be as close as your daily work as it can be, uh, so that when

Speaker A: and then you get Like a free catalog and like uh, other stuff just kind of happens naturally.

Speaker B: Yeah, yeah, just it syncs to the agent context afterwards. But like um, yeah, when you change a DBT model then you're going to change like the documentation because the agent knows the flow so it's going to do it. And I think it's the same way for you know, like if you have your customer support processes in notion that's where you're going to maintain them and if it's also in another MD file, you're never going to maintain this file. So like keep the context close to your day to day work I would say.

Speaker A: And do you see it primarily as the data team's job to be um, editing these markdown files? Um, uh, do you imagine over time that folks throughout the business will get involved in this?

Speaker B: I think it can be uh, I think it's the same way as there was with DBT model. I think there are some data teams that opened you know like their DBT repository to. I don't like product managers that wanted to add some data models, this kind of thing. So it could be. I've seen some data teams do that with their context repository. Um, so that's why I think it's also important to have a solution that anyone at the company can contribute context on.

Speaker A: How about the machine generated context part? I um, and I saw you comment, you commented about this on LinkedIn. I thought it was ah, a, the anthropic post recently about how they built their um, agentic data stack, um, was awesome. Um, and then you kind of translated that into your kind of context engineering playbook. One of the things that I thought was interesting about their blog post was, was that they had, they shared kind of some specific uh, beliefs about different sources of context and how effective they were at raising the accuracy level. So they like they plugged in uh, query logs uh, and they, they actually didn't see a big boost to uh, the agent efficacy because maybe there's like not enough signal to noise. Maybe uh, you overwhelmed the content. I don't know the details but it just turns out that empirically they did not increase accuracy by plugging in query logs. So what do you think the most high signal to noise sources of context are?

Speaker B: It's very funny because eventually what I found is that the biggest jump into agent reliability is just like your data modeling and your data docs. Like really like I did this work of like starting from no context and then measuring where I was in reliability and step by step adding context and so by just like adding you know, like uh, profiling, query story, these kind of things, like I was stuck at like 40% reliability and I was just noticing that the agent was failing because you know, there was some ambiguity between two columns or there were ah, there was a metrics in two different table that was slightly different numbers. And so I just redid some parts of the data model, wrote some data documentation and that got me to like 90% reliability.

Speaker A: Um, and that seems totally consistent with the post that they shared recently. They were like look, just keep your house in order, uh, don't have a bunch of duplicate stuff, uh, make sure to uh, pair it back and have everything well documented.

Speaker B: Yeah, I think um, easier said than

Speaker A: done I think many times.

Speaker B: Yeah, that's true. Uh, I mean that's a deep work to make your data model very clean, very non, uh, ambiguous and documented. Uh, and to keep it that way in the long term. But at least you can see it's paying off with agent.

Speaker A: Let's just say that uh, companies adopt this process. We get good at this over the next, whatever couple of years. My understanding right now is that the, the kind of end of the process for now is now like nao, um, uh, is uh, now chat. So you like, you know, you're on your CLI and you, you type in now chat and it, I, I haven't done this but I, I understand that it like brings up a browser with a chat session. Is that right? Okay, so is that what you see as the end result for all of this context engineering work? Or is that one part of a larger ecosystem that you get access to now? Like what, where is all this going?

Speaker B: Yeah, uh, so the goal is really that now becomes the source of truth for any analytics you want to do or an agent want to do. Um, so actually yeah, you can locally have your now chat in the browser, but the goal is to deploy it so that uh, all the context work you do, you deploy it to the whole company and anyone, even some non technical people can ask questions and benefit from the context that you already curated. And the goal is not only to have this context available through an ui, but anywhere your team uh, want to access it. So like through an MCP, through Slack, through WhatsApp, through Telegram, like wherever people want to access it. But the key is that wherever you access it from, it's still logged in the same database so that the data team can see everything that's going through the agent and make sure that it's in the right direction.

Speaker A: My guess, and I Think that this is not such a controversial thing. Uh, but my guess is that we will move from uh, the data lake as analytics to the data lake as uh, infrastructure for building agents more broadly. So um, you know right now we're using this for analytics use cases. But uh, you can imagine a world where agents in order to run their kind of operational workflows, uh, they need access to data in the data lake. Um, do you imagine the type of um, maybe it's the MCP endpoints that you're describing, um, being useful to these types of non analytics but more operational agents.

Speaker B: Well the thing is what we do with the context today, it could be applied to non analytics use case. Like you could take now and just like put I don't know the list of books you've read and then ask questions on that. Uh, that could be working. Um, so yeah, that's uh, something we have in mind in vision for the long term we just try to stay focused on the analytics use case for now. Um, but we already see some data teams are actually owners of anything agentic in the company and when it's like that then they expand their agent scope to not only data requests but any kind of request on the knowledge of the company. So that's definitely the way we wanted

Speaker A: to go for that makes so much sense. I think that one of my favorite things that I saw you write, you uh, now own this idea because you've written it first. But it's something that I have thought many times um, is that uh, with agents we are in the just plug it into your production database, uh moment, uh, which like a lot of people in data today actually didn't live in that period. Um, you know this was super common in the mid-2010s before there was kind of a mature cloud data stack. Um, it sounds like you know you experienced this in 21, which I think was the kind of the tail end of that. Um, but, but the idea that you would just be like well I've got a postgres database that my E commerce store is running on. Like why don't I just plug in my um, my, my bi tool to that. It turns out that for a ton of reasons that's not a great idea. Um, and uh, when, when you give people coding agents and the ability to build now they oftentimes are like well where's the data that I need that I have access to? And they're like oh gong. Let's just go straight to Gong. Nevermind that like one gong call is going to like blow up your token budget, know, overnight. Um, so I, I, I think we are in like such early stages here and you have this idea of like the context stack being almost like a data stack. Um, there's like multiple different pieces to play. Do you, do you want to expand on that a little bit?

Speaker B: Yeah, I mean I was doing this parallel, uh, for me. Yeah. As you said, like, we are the phase where people are like, oh, let me just connect cloud code to my smells like mcp and what could go wrong? And it's the same as when you had your BI plugged to the database and rebuilt in the data stack. All these tools to like ingest context from everywhere, transform it and have this like source of truth. Um, and I think we kind of need the same thing. Maybe we don't see it now because we are building the first wave of context, but we're going to see context routes, we're going to see contradictions in the context. So maybe we need the same kind of stack of ingesting the context, that's for sure. And then transforming the context, maybe merging some old and new context, some contradictory context, I don't know. Um, and then exposing the source of truth context into your agents. So yeah, that was an idea that maybe that's what we need to do. And where are the tools that allows you to do that? I don't know yet.

Speaker A: Yeah. Do you, uh, what's on the roadmap? What, what problems do you find that uh, your 80, uh, companies have today that you want to, you want to solve for them next?

Speaker B: Yeah, well, I think the key is really about like creating the context and helping them scale this because when we rolled out now, we just had a chat replay where data teams needed to go read every chat and see what was going wrong. So that's insightful, I think, but at some point you want to automate that and we actually have one user who started automating that with a full agent, scanning all the chats and all. And he actually built a first version for us of recommendation and creation of context automatically. So that's the next big thing we want to work on is like, how

Speaker A: did you like, so, so this agent, this agent like reads the logs and is able to determine if the answer that this person got was, was good or bad and suggest like different ways to improve the context, uh, in order to get a better answer next time. Is that right?

Speaker B: Yeah, it looks, because sometimes like the uh, agent autocorrects itself. Like for example, let's say it's like going to query something, got something wrong and after eight ah, queries it figured out which field was the right one to use. Well you don't want it to do it next time you want to put that in its context. Um, so these kind of things that the agent learns itself or uh, you know, some comments that the user tell the agent like oh no, you should not do this for the definition. This is the actual definition. Well we should learn from all of this.

Speaker A: That's interesting. So almost like a memory mechanism, uh, that uh, you can build memories over time.

Speaker B: Exactly. But the tricky thing is you want to make sure that the memory is the right one to learn. Because if you use cloud code locally for example, this is you and yourself and the agent. So it can learn whatever you tell it, even if it's wrong. But when you're at the company level you want to make sure that people don't teach wrong stuff to the agent. And that's where the data team still needs to approve what gets into the global memory of the company.

Speaker A: That's so fascinating. Uh, I don't know why I had not made that um, conceptual leap before but it's really, there's been a bunch of releases recently on the topic of memory. Um, OpenAI released dreaming as a part of uh, ChatGPT. I. They really like to anthropomorphize things. That seems a little much to me but, but like it's just an indicator that like we're, there's a lot of uh, you know the labs are thinking about memory right now a lot. Um, and it's, it's funny context, the type of context that you're, you're talking about here really is you know, a shared long term memory for the entire organization. Not for the person but at the organization level. Um, and uh, there are a whole set of things that we will need to get good at about how do we form new memories, how do we uh, clear out memories that are not serving us anymore. Maybe that traumatic experience back from childhood. Let's not think about that every day. Um, you know, does seem like there's a, a lot of work to be done in that framing.

Speaker B: Yeah, I think there is but I think I try to start by doing simple things. Like for example like I built my company brain like because everyone was talking about brain. Like a lot of startups create thing on um, like brain. But like I wanted to do something simple and see if it could work. So I just like did a GitHub repository. I have all my context there. The memory is just like the agent logging new stuff we do and that works for now. So I uh, don't want to buy a tool to do that and maybe at some point it's going to not work and then I'll consider some other tools. But I think we just need to start simple on this thing and see where the clustering is before we build complex stuff.

Speaker A: Um, Sarah Catanzaro from Amplify recently wrote a, uh, uh, paper post, I think it was called something like file system is all you need or something like that. And just that like yeah, it's, it's turns out that file systems are very useful.

Speaker B: Can I ask you a question then?

Speaker A: Yeah, sure.

Speaker B: So what, I mean, I think there's this whole controversy these days about like semantic layers and like.

Speaker A: Sure, yeah.

Speaker B: If you say like file system is all you need, where is metric flow in there?

Speaker A: So, uh, uh, yeah, good, good question. I noticed metric flow as a part of your docs. Love, uh, love to see that. Okay, so if you go back 10 years ago you would, you would have Looker as one of the leading new BI tools that was being widely adopted and Looker's popularity was rising because of LookML. And um, there are a lot of good reasons in a BI world to want a semantic layer to, you know, know, blah, blah, blah. Um, the complaint that you had from data practitioners about Looker was that everything had to be in LookML. And uh, sometimes you just wanted to do some stupid simple thing, uh, and not build, you know, a whole 200 line lookML file for it and Looker wouldn't let you do that. And, and so. Okay, fast forward to now. Um, one of the points of friction that we have experienced over the last several years with metric, uh, flow is that people didn't have a good answer to when does something need to be in the semantic layer versus not. And I think that this version of the world that we're moving into, um, conversational interfaces and text to SQL being pretty good, um, and then context layers on top of all of that

Speaker B: means

Speaker A: um, that the metric layer can be governance for your most critical high value accuracy, 100% required metrics. Uh, uh, but, but it doesn't, it doesn't, uh, you don't need to first define a metric before you can do like literally anything. Like people don't want to do that and I don't think they should have to.

Speaker B: Yeah, that's true. I remember I tested the skills that you guys created, which is like querying m your data with a metric flow, uh, semantic layer and the logic you put in it was like first query through metric flow and if you don't find it, just like read the DBT docs and do a regular SQL. And I think that's the right way to do it.

Speaker A: So. Which is exactly what Anthropic referenced, uh, in that, that blog post that they wrote recently. Like if, if there's a golden path metric, then use it. Otherwise, you know, try to figure it out yourself. This world is, is fantastic. I, um, maybe I want to close on your thoughts on open source. Um, you, you kind of alluded to this. Um, you know, I, I don't know that I want to lock up all of my context in some proprietary thing. Um, and especially if file systems and markdown files are a lot of what we need here. Um, what has your thinking been on open source, uh, over the company's history? Um, why does it feel important to you that Open source is a part of this layer? And for you folks, how does this evolve into something that know people will pay you for over time?

Speaker B: Well, I think open source makes sense for a lot of reason today. I think first, like you want to be used by agents and not only by humans. And if you're open source, your code is here available. So it's very easy for agents to, you know, you want to build an analytics agent it knows about now and it's going to build your whole agent with our framework. So that's the first good strategy for distribution. And then I think you see, we do all this talking about what is a good context, um, how to do context engineering. And I think nobody has the answer today. And I think we need to share the knowledge together to have answers to this because all I see people do is benchmarks of these data sets of how to compare Clothes Opus or Clothes Fable or whatever. But it's so different when you do it on real data. So we felt like if we have an open source framework where the startup can share what are the results on, um, what is the best context based and the big enterprise can share also like how the semantic layer, um, impacted their reliability, then we all have a common language where we can experiment and learn together and like improve the whole reliability of this kind of products. So that's really what we wanted to create with the open source framework.

Speaker A: And do you think that there will be, do you already know the roadmap of commercial products you want to build on this open source foundation or is that still uh, to be figured out?

Speaker B: No, we already do. So the thing we sell is how do you scale this product that works for one team to a bigger company. Um, because our open source product gives the same data access to everyone at the company which works for small company but does not for bigger companies. So like our enterprise license is for handling data permission, context, permissions, managing tokens, budget, this kind of thing at scale.

Speaker A: Got it. That makes sense. Um, this has been fascinating and it's an area that is moving, uh, super quickly. So, um, we may have to have this conversation again in six months. But, uh, thank you for hanging out and uh, lending us your brain.

Speaker B: Yeah, well, it was very nice. Thanks a lot for inviting me.

Speaker A: The Analytics Engineering podcast is sponsored by DBT Labs. I'm your host, Tristan Handy. Email us at podcast@dbtlabs.com with comments and guest suggestions. Our producers are Jeff Fox and Dan Poppe. If you enjoyed the show, drop us a review or share with a friend. Thanks for listening.