Is RAG Dead? The Pioneer Who Invented AI's Memory Layer Answers - with Douwe Kiela, Co-Founder, Contextual AI {ICYMI}

Speaker A: Let's go.

Speaker B: You're listening to Making Data simple, where we make the world of data effortless, relevant and yes, even fun Podcast listeners. I've got Dow Keela with me today who is a top AI researcher who led the research team at Meta that introduced RAG research way back in 2020. He also has been the head of AI research at Hugging Face. He is now the CEO of Contextual AI. It's an enterprise rag company that helps deploy RAG agents production for Fortune 500 companies like Qualcomm, HSBC and others. He also has a really cool presentation on YouTube around rag agents in production like Lessons Learned. Welcome Dow. I greatly appreciate you being here.

Speaker A: Sure. Um, so, uh, I'm Dow Kima, ah, CEO at Contextual AI. I'm also an adjunct professor at Stanford. For this I was the head of research at Hugging Face. Before that I spent five years at fair, as you said, um, um, doing a bunch of different research projects including RAG retrieval, augmented generation. So yeah, um, it's a pleasure to be on the show and it's been a long time. So we recently celebrated the five year anniversary of rag. Feels like a long time ago and a lot has happened since then.

Speaker B: Intern at IBM at one point too, right?

Speaker A: That's right. Yeah. I was a part of the Extreme Blue, uh, program, uh, in Amsterdam, which was.

Speaker B: How in the hell did we let you go? That's what I want to know.

Speaker A: I think I was like 22 or something at the time, so. Yeah, yeah, a long time ago.

Speaker B: Well, that's fantastic. We shouldn't have let you go. That's fine. So rag, I do want to get into all kinds of different questions and we'll just have a fun conversation, if you will, here. But rag, is it here to stay, here to go? Is it going to m. Morph into something else? Where does it sit today? Because I know you're all in on rag, at least as far as I can tell.

Speaker A: Yeah, I'm all in on RAG in the sort of abstract sense, which is that if you want to have a generative model work on your data, then you need to augment it with retrieval. I just covered the R, the A and the G. Right. You have a generative model, you want to augment it with retrieval. Uh, so that idea, I think is not going anywhere. And so that is fundamentally what RAG is about, I think is how can I get a model to work on my data without needing to train on that data? So, uh, obviously the old idea of rag, like what we talked about in the paper, which was Even before like vector databases were a thing and language models weren't necessarily called language models, at least in the sort of in the scientific discourse at the time. So that has changed a lot. It's a much more complex system now. But uh, I think the general underlying idea is still true and that's never going away.

Speaker B: It seems pretty obvious right now. Rag, right. Everybody says it, we don't even pronounce out uh, to you know, retrieval, augmented generation, you know, we don't even say that, we just say rag. It's so commonly known. However, and the way you explain it, you know, again, non issue seems obvious, but look, it didn't exist a few years ago. What brought you to RAG in Meta? Was there a business problem that you were trying to solve that got you there? I mean, tell me a little bit about how you got into uh, the development of Rex.

Speaker A: Yeah, so at fair, especially at the time, so Facebook AI research, we were not working on business problems to be honest. We were working on really like the frontier. Right. And so this was before AI was a mainstream thing and we were just very interested in this kind of open question of could we take a model and modify its knowledge without needing to train on that knowledge? And so that's what RAG really was. We were very lucky at the time. There was this kind of first vector database phase and DPR dense passage retrieval with was just there. So it was a very good encoder for doing retrieval. And then we had uh, an amazing generative model, bard, which was an encoder decoder model that uh, had a generative component. So sort of bert but then generative, so Bart and we basically just put those two parts together and then it worked really well on open domain question answering and became uh, a Neurips paper and that just seeded this idea that you could take a generative model and then use retrieval to give it the right context. And so that's really the big open problem of our times right now. It's not really about generative models anymore. Like how can you build the best language model? I mean that's part of it obviously, but that's commoditizing rapidly. The big open question is how do you contextualize that generative model so that it can solve the problem it needs to solve?

Speaker B: But do you think RAG will be here to stay or will it develop or evolve into more like fine tuning or adjusting the parameters of the base model itself so it's kind of customized as your own model? I guess if you have the GPUs, uh to do it. But if they're smaller models or if we move towards smaller models, it's certainly possible. Do you think that's where we're heading? You say no, hey, rag's always here to stay.

Speaker A: No. So fine tuning, I think a lot of people don't actually understand what fine tuning does. So you have a pre trained base model that has been trained on basically the Internet, right? Huge amounts of data. What you're doing when you're fine tuning that little, uh, on a little bit of data that model, whether, whether it's big or small, it doesn't really matter. You are basically eliciting sort of the responses that you want to get from that base model. But injecting new knowledge into a base model that has been trained on the Internet with a little bit of knowledge, uh, that is kind of separate that you're using for fine tuning, that is just not how it works. So you can get it to be very good at style, you can get it to be very good at sort of basic kind of vibe testing things that you might want to see. But actually injecting new knowledge into a base model like that um, is just not possible. Um, and so that's why RAG makes a lot of sense so that you don't have to do that.

Speaker B: Have you ever heard of IBM InstructLab?

Speaker A: Nope.

Speaker B: It's a technology uh, we developed that actually it's a third option. It's not fine tuning, it's not rag, but it does adjust the parameters with synthetic data so you don't have the proliferation of the fine tuning models if you need that. And it's somewhere in between rag. However, having said that, I would say you've still got to use InstructLab with rag. I mean that's the best implementation. So either way it's still a key part of enterprise use your own data type of scenario.

Speaker A: I guess a broader point for me is that it's really about using uh, all of the tools at your disposal. And so RAG is one of the tools. And so fine tuning especially for stylistic elements is another tool at your disposal. And then uh, for domain transfer maybe you want to fine tune the re ranker or the embedding model. Um, so um, all of these parts together make the best system. And so that's how you should be thinking about it. So RAG will never go away. Fine tuning will also never go away. You should be using all of those things. So if anybody ever asks you, uh, if RAG is going to die, you should send them to isragdead.com. which takes you to, uh, a blog post, uh, I wrote which talks about these kind of false dichotomies. I think this really comes from marketing departments or journalists or VCs, people who like these dichotomies. But as you know, in the real world, you want to use all of those things at the same time, ideally to maximize performance. Uh, so don't believe in false dichotomies.

Speaker B: So. Fair enough, fair enough. I want to get back to some of the lessons learned because I want to go through these. You made some profound statements and I'd like you to elaborate on. But before I do, you're obviously the CEO of Contextual AI. You mentioned context. Tell me a little bit about more context. Is it more about the RAG implementation and methodologies that you're using with your clients, or is it bigger than that?

Speaker A: I think it's bigger than that. It's really, um, a language model on its own. Uh, I think recently somebody called it like this, hyper, uh, intelligent amnesiac. It's a great tool. But for that tool to really be useful, it needs context, just like humans. So you can have a genius human, uh, but if they don't have the right context, then they're not going to be able to, uh, help you with anything. So, yeah, in the same way, like finding the right information so that the language model can solve the problems that you want it to solve, that's really about finding the context. Right. So quite literally, like the context of the language model, that sort of the token window, you want to put the right information in there, give it the right context so that it can do its job.

Speaker B: What is the easiest method? I presume your company is involved in deploying RAG into production, really making it work. As I was listening to some of your talks, you were saying many of the things that, you know, I've said as well is that even with all the AI, I don't want to say it's hype because we're beyond hype. There's certainly roi, but then again, like what? One out of four implementations are deemed successful. Can you talk a little bit about what you're doing to make it more in line with four out of four?

Speaker A: Nice. Yeah. So it's also really just, um, where we are right now, I think, in the world in our AI journey. So there's a lot of initial excitement. Uh, 2023, really, there was this massive demo phase, I think, where everybody was just demoing things and wasn't really thinking about how to productionize it. And now that, uh, we've invested so much time and effort and money into AI, we need to show some results. And those results are not always materializing, not yet. Um, and the real reason for that is that. So building demos is just very easy right now. Uh, maybe you don't even need rank for that. You can just take a PDF and put it in the context of a language model, ask it some questions and say, oh, this is great. Uh, you show it to your boss and then the boss goes, okay, we're going to do this next year and we're going to change everything. And then you're on the hook to deliver and then you find out that when you actually transfer this to real data with real use cases, with real accuracy requirements, um, and you need, uh, to build this very complex system that works on very, very noisy data that is very hard to get anything to work. And then once you have that thing working, you have a very high total cost of ownership after that because you need to maintain it. You need to stay up to date with all the latest advances in AI. And we all know how quickly AI is moving, so that in itself requires dedicated engineers years. And, um, then you have to repeat the whole process for the next use case. So it's really a very, very painful journey, uh, on the technology side. And then on top of that there's change management where you can build amazing technology, but if you don't, um, help people use it in the right way, then it's not going to have the ROI that you're hoping for. So it's really about this kind of journey of putting technology into the hands of the people who, uh, would benefit the most from using it.

Speaker B: So, but why contextual AI? What's the two minute pitch there? I mean, you've had an interesting career. You went from extreme Blue intern, then you go to Microsoft Research intern, and then you're the CEO and founder of Brainstorm It. Then you, uh, spend some time at Meta Facebook, then you go to Head of research at Hugging Face. You know, that's not no small job. And something must have triggered you to say, all right, now it's time for contextual AI. Why? Why now? You've been there, what, a couple of years now?

Speaker A: Yeah, two years.

Speaker B: Yeah, two years. So why is this the place for you right now, given your history?

Speaker A: Because this is really the biggest problem, like I was saying, right, like, so language models are very kind, uh, of mainstream hypey. Everybody, uh, is interested in that. But the real big problem is not really the language model, but how you give it the Right, context. Um, and so that's what we are the best in the world at. And so RAG is a version of that. Right? So the way we do ragged is a bit different. Uh, we really started from this observation that RAG is about systems or even in general Genai that works on your data is not about models, it's about systems of models. Uh, so how do you get these models to work together? How do you make sure that each of the components is optimized for the problem you are solving? And how do you make sure that they're optimized for working well together? That's what we call RAG 2.0. We have state of the art document understanding, re ranking, retrieval, grounded generation. Um, and so all of these components are designed to work well together. And that's why uh, we are the best in the world at contextualizing language models.

Speaker B: Is it a knowledge thing? I mean, here's why I asked that, because I'm trying to get you to double click on best in the world specialized RAG agents. Because anybody can implement rag. I mean it's out there for the taking. What makes contextual AI. What's your secret sauce?

Speaker A: Yeah, anybody can implement rag, uh, as a quick baseline for a demo. Not anybody can build a RAG system that is production grade, that has very high accuracy, very accurate attributions, that works on millions of documents that are extremely noisy. Um, so our secret sauce is really the research uh, that we are doing around how to be the best at rag. So that starts from measuring how good you are at RAG evaluation, then designing all the components individually to be the best in class for what they're doing. So a re ranker, for example, making sure that that's state of the art by a large margin. And then third, making sure that all of these components work well together. So it's a compound system. If your extraction is bad, you can have an amazing retrieval system and an amazing language model, but you're still going to get the wrong answer. Or if you have an amazing language model but your retrieval is wrong, then you're still going to get the wrong answer. So all of these parts need to work well and they need to work well together.

Speaker B: But is it a methodology or is it the tech underneath or is it both?

Speaker A: It's both. So it's a methodology for how to train these systems, uh, and how to make them best in class. And then those components are all combined in our platform. And our platform makes it very easy to do this as well. So you get very high out of the box RAG Accuracy that scales to millions of documents, uh, without effort and then setting that up is very easy to do in our platform.

Speaker B: So if I am a client and I say, hey look, if I'm a client, I say I need help from contextual AI. You sold me on the ability. I want to use my own documents. Obviously that's my intellectual capital. I want to bring that with a large language model. I don't care which one. Uh, but I need to solve whatever use case, whether it's an assistant or whatever the case may be. What is your standard process? I mean what would I expect of that engagement?

Speaker A: Yeah, so you can do it yourself. Right now if you go to our website and you click on try it, uh, that logs you into the system and then you would create a data store. You put your documents in the data store and then you can build as many agents as you like on top of that data store. So it's really very easy to set up these agents in a matter of minutes out of the box. Their performance should be competitive and then can specialize all of the components and we give you lots of control and different knobs so that you can really get max performance out of your RAG agents.

Speaker B: What's the monetization strategy then? I mean what can I expect as a client? What is the pricing? What am I going to pay, uh, to do this?

Speaker A: Yeah, so we uh, charge just like a normal language model, but now it's contextualized on your data so it's dollars per token. Um, so it's consumption based pricing, uh, exactly like what you would have with an OpenAI. And then if you are a larger enterprise and you want to have provisioned throughput, then you would buy model units. So kind of the standard pricing that everybody follows in the field.

Speaker B: Can I choose any large language model I want to use or is that chosen for me?

Speaker A: You can, so you can use just a retrieval part and then just take the retrieval results and give that as context to any language model. We also have our own language model. It's our glm, our grounded language model that has been specifically trained for rag. Um, so it's not good at brainstorming, it's not good at creative writing for people in the marketing department. But it's really, really, really good at not hallucinating and making sure it gives the right answer based on the context that is provided. And if that context does not contain the right answer, it's also very good at saying I don't know which is really uh, a killer feature for something like this. If you have a low tolerance for hallucination, it's much better to say I don't know than to make something up.

Speaker B: Uh, that's the trick, right? I mean, otherwise you're going to get hallucination and things that need to come up with an answer. I don't know is a great answer. I got it. So what I think I heard you say there then is, uh, you can use. Don't let me put words in your mouth, but you can use any LLM that you want to use. But we think we have the right

Speaker A: one for you, depending on the use case. So not for every use case, uh, but if you have a rag use case with low tolerance for hallucination and high accuracy requirements, then we definitely recommend trying out our glm. And then, uh, you know, uh, you'll see for yourself that it's better than many of the other language models out there.

Speaker B: Is this all, how do I ask this question? Is this, um, you know, all manual from my standpoint, or can I solicit. I presume you've got services that can also accompany this work if need be. You do both or independently or.

Speaker A: We do both. So you can do it all yourself using our APIs. Um, but we can also help you with it. Uh, so we have a services ARM and we have some really amazing customer machine learning engineers as we call them. They uh, often help customers really get maximum performance performance out of those systems. Um, so yeah, it's really up to the customer. But I think it's important because we're really building for, uh, technical folks like AI developers, that people can also do it themselves.

Speaker B: That's a good question. Uh, so what's the Persona? The Persona is more developers then.

Speaker A: Yeah, so it's technical, so it's AI developers, which I think is a pretty broad category. Right. So it's not necessarily just machine, uh, learners. Although machine, uh, learners might want to use our individual components which are also available, and then you can just plug them into your existing RAG pipelines, uh, and optimize different components around it. But, uh, for AI developers, it's really just about knowing how to call APIs. And in this case you call one single API that does the whole RAG process for you. Um, and then if you want to use our graphical user interface, then you don't even need to know how to code. I think these people are still quite technical. Even if you're not coding, you need to understand basic, uh, AI concepts.

Speaker B: Great, I got your go to market. So let's take a step back. Real Quick, I would ask you, you've kind of already explained some of these concepts as why it's so difficult for enterprise enterprises to get this successfully into production. But I want to go back to some of the statements that you've said and get some first hand knowledge from, from you on them. The first one is, but you said better. Large language models are not always the answer. Why is that?

Speaker A: Yeah, so large language models are a part of a system that is meant to solve a problem. Um, and so sometimes that's really a quite small part. If you look at some of the RAG pipelines that we've built, they are comprised of around 12 different models and one of them is the language model.

Speaker B: Do you use all of them at once? I mean you use all 12 in

Speaker A: a single forward pass. You could have 12 different models, right? So 12 different neural networks. Um, so one of them is a language model and some of the other components could ah, be language model basis, but one is the final kind of answer, right? So that's really a small percentage of this much bigger system. And if you make a mistake during information extraction, so let's say you have a PDF with a very complex table and maybe you drop some of the rows of that table. You can never ask any questions about those dropped rows. Even if you have the perfect language model, you will just not be able to answer the question because you don't have the information. So that's why we call these compound systems, because the performance compounds with uh, the performance of the individual components starting from the information extraction. So you need to extract the information in the right way, store it in the right way in your data store. Make sure you do retrieval in the right way using different types of retrieval strategies. You need to re rank the initial retrieval results to make sure that you really get the right bits of information, then give that to the language model. Then maybe you need to do some more filtering and some output guardrails, things like that. So it's really a compound system where each of these individual components, their performance uh, affects the overall performance of the system.

Speaker B: Sorry, let me say a couple other things that you've alluded to. Uh, expertise or your institutional knowledge is your core ip, Enterprise scale is your moat, meaning if you can use that internal data, it becomes the competitive advantage. Uh, and the gap between pilot and production is larger uh, than expected. Still larger than expected right now, given how long we've been at it. A while. Why is it still larger than expected? Why is it catching people by surprise?

Speaker A: Yeah, because it's a complex system that you have to build if you want it to be really good so you can get something out of the box very quickly. Now get like 70, 80%, um, which is not bad. So like a year ago you would have been very happy with that. But accuracy requirements also go up over time. So the more AI we use and the more we see it around us, the more we expect from it. Uh, so the 80% that was good enough a year ago is not good enough anymore. Now you need to be 90 plus and that is just going to keep going. You need to get to 100%, basically. I don't think we'll ever get to 100. Exactly. But the more accurate we get, the more ROI we potentially unlock. With the goalposts constantly shifting, you just have to keep building these very complex systems and then they get more and more complicated over time. Uh, there's more and more maintenance that goes into it. And if you're a, uh, large enterprise, you don't have five of these use cases, you have 10,000 of them. And so you need to build that complex system for all of those use cases. Uh, and that just becomes very, very difficult very quickly.

Speaker B: Awesome. You also mentioned speed over perfection and work to alleviate your engineers from spending time on what you term boring plumbing like chunking, text cleaning, prompt tuning. Aren't those things necessary evils? How do you avoid those?

Speaker A: Yeah, I think a lot of them are not necessary evils. Um, they are just if you have a very good opinionated system that has been optimized for doing it the right way, then you don't have to worry about a lot of those things. So that's what we offer. You just ingest using the ingestion API and then you query using the query API. And so that's basically your entire rank system right there. And you don't have to worry about anything else. I think like the fundamental observation is that if you are in an enterprise, AI engineers are really some of your scarcest resources. They're very well paid. There aren't that many of them in the market. They have very niche skills. So you want them to work on making your company better than your competition. You want them to work on differentiated value instead. If they're working on chunking strategies and making sure that with the latest VLLM M update, uh, things don't crash because there's a dependency mess. Those kinds of basic things that they're spending a lot of time on, they should not be spending time there. They should be thinking about, how do I deliver ROI from Gen AI. How do I get it to work on my data? How do I leverage what is unique about my data and the institutional knowledge that I have? And how do I make uh, sure I design my RAG agents to capture that institutional knowledge? Um, so that's really just the main point that uh, I was making in that talk you referenced is really try to think about how you can get to differentiated value and leverage your own enterprise context in the right way so that you can do things differently from, from everybody else.

Speaker B: What is your secret sauce to alleviating hallucination? I presume part of it is just using client data through rag, the RAG methodology. And then you're able to say, hey, if it's not in that data, then we're going to say I don't know. That stated, do you find yourself, I guess a two part question, I'm a terrible host. Do you find that clients come with quality data? It all starts with the data, right? You got to cleanse the data and make sure it's one source of the truth or you'll have hallucination based on just PO data to begin with.

Speaker A: Yeah. So hallucination, the way we tackle that is by making sure we have very good retrieval. And then we have this grounded language model that has been designed to say I don't know if the right information is not in the context. So if we didn't retrieve the relevant information, the model is just going to say I don't know. And that's much better than making something up. And a standard language model is often very tempted to make things up. It will say like it's not actually in the information, but blah, blah, blah, blah, blah. Right. Um, and so that's really what you want to avoid. And then on top of that, uh, the output that we generate, we validate that separately. So we decompose it into individual claims and we test those claims for groundedness in the source material. So essentially there's an extra hallucination check that happens on top of our generations that is really important in catching some of these uh, uh, failure modes. So that's one thing. And then your second question is really about uh, the quality of data. And so everybody we talk to, they come to us and they say, but we have like especially terrible data. Everybody thinks that and, and so it's really true. Right. So uh, if your data is terrible, it's very hard to get things to work. And so there's this kind of motto that you always hear repeated, I think often by people who don't actually understand what it means, which is garbage in, garbage out. A lot of people kind of like throw that around, I think, without really understanding, uh, what that means or where that comes from. But there's a truth in it where if you. So you cannot be better than your data. So there is a limitation there. But, uh, I think in the next couple of years what we are going to see is that you don't have to make your data ready for AI. That used to be the case. It probably still is the case where if you have cleaned up data, you're going to be doing a better job. But where we're going is we need to make the AI just good enough for your data, however terrible your data is. And that's the really big open challenge, I think. Right. I think a lot of folks are investing a lot of time now in cleaning up their data that might actually go to waste. You should probably just wait for AI to be good enough to work on any type of data. And we're very close to that with agentic rag capabilities. So just having a giant blob of data and then an agent can actually try to find something. If it finds the wrong thing, it can reason about like, oh, actually I should have searched for that, or oh, actually I found these two different conflicting pieces of information. Let me gather a bit more data and then maybe I can figure out what the right answer is. So that's coming and it's coming very quickly.

Speaker B: How will you know when that's ready? Uh, is that a reasoning capability within the language model that you mentioned? Because I'm with you. I do have some clients, which makes me a little uneasy that are waiting because of exactly what you mentioned. They don't want to have to do anything to their data and they're waiting for the AI to take care of that too, so that they can just throw all their data and magically they get the right answers.

Speaker A: Yeah, I think you can already do that right now. Probably not with a general purpose solution, but with a specialized agent you can probably already get that to work right now. But I think waiting in general with AI is a terrible decision.

Speaker B: I know. That's why I like anytime, uh, I hear waiting that just makes me uneasy and I'd rather them get started. Of course they go back to where you started, which is my data is absolutely terrible. Al. Uh, I'm going to get nothing. It's going to be garbage in, garbage out.

Speaker A: Yeah, but that's a cop out from people who don't understand what that means. You can still get a Lot of value. If within that garbage data there is some valuable data, you just need to find a way to get it out. Uh, and there are lots of ways to do that.

Speaker B: You also talk about making AI easy to consume, meaning you seamlessly integrate into existing workflows and to wow your users early, uh, with some aha, uh, um, moments. Can you say more on that?

Speaker A: Yeah, so I, I think one of the, the big things that we sometimes, uh, see happening is that people have built this great AI system and then they're complaining that nobody's using it. And so, uh, you know, the technology is only part of the solution. It's really about how do you put it into the hands of people so that they can see the value. And, and people tend to not be very forgiving when the system makes mistakes.

Speaker B: So.

Speaker A: And they often have like their one or two, like pet questions. And then if it gets it wrong and it can't even count, like the R's in the word strawberry, how many there are, then it's like, oh, this is terrible. I'm never going to use this. So the way to really get to AI adoption is to make it very easy for people to get to this initial aha, um moment where they go like, oh, actually, like, I could use this. Like, I have lots of problems that are similar to what this is solving for me. Uh, so you need to guide your end user a little bit to get to that, that uh, right. Experience so that they can really see why it's useful for them. And if you do that and you integrate it into their existing workflows, then you'll actually get people to use it.

Speaker B: In line with your earlier statement to just get started, let AI, uh, take care of most of it, it's going to be better than you believe it's going to be. You talk about observability mattering more than accuracy. What do you mean by that?

Speaker A: Yeah, one of the things that, uh, I think is interesting, we talked about this a little bit. The accuracy requirements are changing over time. So 80% used to be good enough. Now it's 90, it's going to be 95, 90, 98, 99. Um, and throughout that process, especially in regulated industries, for example, or the kind of sectors that we focus on where there's a low tolerance for mistakes, it's actually not even really about the accuracy anymore. It's much more about what do you do in the situations where you might be wrong. So thinking about the incorrect answers and how do you mitigate potential incorrectnesses? So just how do you catch mistakes before they have, uh, negative, uh, impact on your end user? So that's why we do things like we have bounding box attributions in the source, so you can really double check what the exact source material said. As I said, we do this groundedness check. So we decompose the claims, we check them post hoc. So there's a lot of things you can do around audit trails, monitoring, uh, observability that are really, really important in understanding what your system is doing. And that's almost more important than the accuracy. So a lot of people would probably prefer to have that observability and maybe a percentage point, you know, less performance than that they care about maximizing accuracy at all cost.

Speaker B: So now we're not, I was going to say heading towards, but we're already here around agentic AI. And since we're talking about definitions and how people can interpret things incorrectly. How do you define an agent? Is the best implementation of agentic AI through specialized agents in an enterprise context?

Speaker A: Yeah. So how do we define agents? Everybody has a different definition. I think, uh, there's a pretty standard definition from the reinforcement learning literature, which is about something that takes actions in an environment and then has a state. But, um, I think what agent has really come to me now, um, is active reasoning Test Time Compute is really a fundamentally new paradigm that is going to change everything where you can keep thinking and keep reasoning about what you're doing and take actions, um, based on that reasoning state. Um, and then, uh, that just gets you much better answers because you're essentially just spending more compute on getting to the right answer. That's why agents are so exciting. But at the same time there's so much hype and everybody's kind of trying to rewrite what agent actually means, uh, that it can get very, very confusing for people. But one of the fundamental questions around agents I think that you're also getting at is do we need general agents that can solve everything, or do we need specialized agents that can solve very specific problems? And I think it depends a little bit on how far out, uh, you're looking. So general agents are very useful, but if you care about the best possible performance on a specific problem, then specializing for that problem will almost always work better. So I think especially in enterprises, you will have lots and lots of agents that are all specialized for different problems and they're really optimized for solving that specific problem. And then they can, uh, potentially work together, or everybody in a large enterprise will have this big team of Agents, uh, at their disposal that can solve all of these mundane boring problems, uh, that we uh, prefer uh, to outsource to AI.

Speaker B: How does the autonomy in AI agents evolve over time?

Speaker A: So autonomy in terms of taking actions. I think there's a lot of work that still needs to be done on the security side there. Just making sure that we have the right guardrails in place uh, to protect against those agents going rogue or like misinterpreting instructions or things like that. But in terms of um, like the potential autonomy, this test time compute can run indefinitely. Uh, so you can just keep that going uh, basically for the lifetime of the world. And it can just continuously keep reasoning in theory. So that might lead to very interesting uh, you know, autonomous systems.

Speaker B: So does Your platform, contextual AI's platform today use autonomous agents? Do you orchestrate apart?

Speaker A: Um, so we offer specialized RAG agents. Yeah. So you can build these agents yourself. Uh, we have a experimental uh, uh, kind of deep uh, research uh, think mode, uh, that is very good at retrieval. People uh, can build agents on top of us, uh, using mcp where we are the rag agent component in this much bigger sort of multi agent uh, orchestration. Uh, so there are different ways to leverage uh, agents in our platform.

Speaker B: Anything that I didn't ask you that you wish I would have.

Speaker A: I think maybe one thing that that is interesting could be around the different components that exist in a RAG pipeline. So there are small things around document understanding that can be really important in these compound systems. So how do you make sure that you really extract the right information, let's say from a diagram or. We work a lot in semiconductors, like circuit diagrams that are very complex or very complex tables or code. Uh, the little things really start to matter uh, in these information extraction systems if you want to get the best performance. And the other really important component in a RAG pipeline is the re ranker. Um, and so um, one research direction I'm excited about is can you give these re rankers the ability to follow instructions so that you can prioritize data. And especially in enterprises you have these kind of data hierarchies. So for some problems you care about recency, others you care about specific document types or sources or who wrote the document. Um, and so you want to be able to capture that information through prompting the re ranker, not prompting the language model. So these instruction following re rankers, uh, I think are really a very good way to get RAG systems uh, to take them to the next level.

Speaker B: Fantastic. Where can folks reach you and where can they Reach Contextual AI.

Speaker A: Yeah, my name is uh, pretty unique, so you can find me on Twitter and LinkedIn and other places. And uh, uh, yeah, the company is uh, Contextual AI.

Speaker B: Fantastic. Unlike Al Martin, I'm sure. Yeah, you're the one and only. Nice. I wish I had that problem. Hey, just a couple more before we break. Do you have a contrarian view for AI in general? Just in terms of the topic, the hype cycle, what, anything?

Speaker A: Yeah, so I think my contrarian view, I don't know if it's contrarian, but I feel like a lot of people are aiming too low, um, and they're saying, oh, I want to have rag on my data and then I can ask it who my 401k provider is or how many data of vacation I get. That's not really where the ROI for AI is. So I would encourage people to be much more uh, ambitious in the types of use cases that they're trying to solve and really solve high ROI use cases and not just go for the low hanging fruit. Uh, uh, answering simple problems that are not very valuable.

Speaker B: What would you say is a very high value use case? Quick example.

Speaker A: Yeah, so an example is what we're doing with Qualcomm where they have these extremely complex technical questions, uh, that take them a very long time to answer, um, if they were to answer that with a person on the other end. And so our AI system does a great job there. Over decades of information, highly technical content, millions of PDFs, uh, and it does a great job there. So if you had to do that manually, that would be a lot of work. Um, and these questions, they are very complex. When I see the question, I don't know what it means, let alone what the answer means, but the system does a great job.

Speaker B: All right, fair enough. Uh, let me ask you another one. I know you have a point of view. I was going to get into some your thoughts on Deep Seek China versus us. All this stuff around AI. I know you have a point of view on, but I think the way I'll kind of wrap it up is, and I've been listening to Jeffrey Hinton as of late and he doesn't paint a very good picture. I mean, it's almost like I have a hard time putting what I want to say. It's almost like shock, ah, information. Every time I see his name, I say, well, I got to listen to this just for fun. And you know, he's predicting the demise of humans for the most part. I mean, I think like 20% he says he's 2010 to 20%. We're not gonna. We're not gonna be here very much longer. Where's your head at in all this?

Speaker A: Um, I. I, uh, am a techno optimist, so I. I understand where, where Hinton comes from with that, but I think if we. If we do things the right way, AI actually has a lot of potential to help us be more human. So it can take care of a lot of the boring stuff like we just talked about, a lot of the boring, mundane work that we do every day, which will give us a lot more time to focus on the things that we are uniquely good at, which is to be human with each other. And so the kind of social interactions and things like that, um, I think we'll have a lot more time for that. And for AI to really take that over, I think is a bit of a different kind of proposition than what Hinton is afraid of there.

Speaker B: I'm with you. I mean, obviously I'm an optimist with this. I mean, everybody transformation we've done has always, I think, started this way. And I work 24 by 7. So it's not reduced our work any. In fact, it's created more work. It's just different. I know that the. There's many out there would say, but this is different. But I'm sure that they said every other revolution was different too.

Speaker A: That's an interesting point, though, because I do think that this revolution is different. It's like, I've said this at a bunch of different panels and things. In retrospect, when people, like, 100 years from now look back at our time, they will say, okay, there was the AI Revolution, which was really big. And then before that, there was the Internet revolution, which actually wasn't that important. The main reason we had the Internet revolution was so that we could get the data to get to the AI revolution. And that's really, I think.

Speaker B: Oh, I agree with you.

Speaker A: The Internet, uh, will look much less significant, uh, from the future than AI Totally.

Speaker B: I think this is game changing. It puts everything together. Unlimited data, uh, unlimited access, unlimited speed. I mean, on and on. It's just all coming together right now.

Speaker A: Yeah.

Speaker B: Uh, what. Last question. What do you do for fun? What do you do for fun?

Speaker A: I, uh, hang out with my kids. I wish I had more time for that. Uh, but that's really, uh, you know, that's. That's the most fun for me.

Speaker B: My advice to you, having three kids that, uh, are now out of the home, man, spend time. They always say it's going to go by fast, but it goes by faster than even you would envision. Anyway, Dalkila, everybody, thank you so much for being here. I greatly appreciate it. Very, uh, informational. And folks, check out Dow online. Check him in YouTube. He's got a lot of great blogs and, uh, presentations that I think you'll find a ton of value from. Thank you for being here.

Speaker A: Thank you, team.

Speaker B: Hit us on AL Martinetalkstate@gmail com. Love to hear from you. And until next time, we'll see you on the podcast. Bye. Bye.