The Current State of Agentic Retrieval - Qdrant Roundtable

Speaker A: Foreign.

Speaker B: What is up? Uh, everyone? Hope you were jamming to that music as much as I was. I can see the chat is already getting off to a good start. I'm excited because we've got a packed house today to talk all about good old state of agentic retrieval. There's so much that we're going to get into, and we've got so many incredible folks here today. But I want to set the scene before we bring out the guests of honor. And I want to talk to you about why this session feels so important. And it is probably why you actually came. You know, there's so many different challenges and changes that we've had with retrieval over the past two years. I have seen such an evolution. And so I thought, why don't we get together some people who have been knee deep in this whole retrieval game to talk to us about what it was like, what did we have back in the day and what do we have now and what are ways that we can optimize it so that we are finding success right now? So without further ado, I'm going to bring on the team from Quadrant. We've got the dev rel folks coming out. First up is my man Dylan. Where are you at? Hey, Dylan, how you doing? Jenny. We've got Eva, Neil and Andre. I think this is the most people that we've ever had on a round table. So let's just kick it off, start strong with some hard hitting questions. I'm gonna ask what has changed in the last two years? I. I've seen so much. What do we need to focus on?

Speaker C: Yeah, uh, sure. Yeah, I can take that. Uh, hi everyone. Neil here. Um, yeah, thanks for joining the session and thanks for the intro, Demetrios. Um, there's so much happening in retrieval and search. M. Uh, before I joined Quadrant, I didn't fully appreciate the field of search. And um, we kind of just expect search to work. And we definitely notice when search doesn't work well, we notice it through just getting terrible results. Um, or it takes a long time to get results. We can't find what we're looking for. The more data that we capture and from the real world, the more we need to get that information. Uh, get. Get the data, the right data we need, get the information out of our. The data we're storing. So what, Back to the question, what's changed? Um, one of the biggest things that's changed is agents. And the topic of today's, uh, roundtable. Um, agents, a lot of them don't really Search? Well, uh, I would say, in my opinion, um, there's a lot of the kind of search like a, like a beginner, uh, would just search by just like, kind of throwing a lot of things into the, into the hat and just like, trying to like, pull, pull, pull results. And they go through lots of turns and they burn a lot of tokens and they, um, try to find this context, but they're not using all the tools available for search. And if you don't have the right context in your AI, that's going to really make your AI less efficient, less effective. Um, it's going to prevent you from reaching your goals. So agents. And one of the things that agents are doing that's also interesting is like a human might search like once or twice a minute or so. Agents are searching thousands of times per minute. So they're just going boom, boom, boom, boom, boom, hitting it. And they're not able to evaluate those search results the same way a human can. Because, um, humans have a lot of more judgment that they can apply. So, yeah, I'll stop there. But those are, those are some things that really changed.

Speaker B: Dude, you're preaching to the choir on the different ways that agents search and then the ways that they interpret that data. And so I, I have some questions here about like, hey, now that we've got agents and these agents are figuring things out, they're kind of, We've all seen it. They will stumble through different problem statements or they'll try and figure it out. And they don't necessarily go about it the most efficient way possible. I wonder if you all have found tricks on nudging the agents to be like, oh, yeah, by the way, like, there's obviously the go look in that file trick. But, uh, there potentially could be plenty more that you've seen because this, at the end of the day, is a search and retrieval problem. And the faster that you can get those agents that knowledge, the faster they can hopefully do what you've asked them to do.

Speaker C: Yeah, I'm actually gonna take jump in here again, but I'm gonna pass to Dylan on this. So Dylan, get ready. Um, so, um, just to set like a little more context, there is, um, you know, it's not just the efficiency, but it's the effectiveness of search, like finding the most relevant response, uh, relevant results. And I think that, um, it goes into evaluation. How do you evaluate that the results are good enough to use? Should you try again, should you change your query? And if you should try again, when do you stop? Like, how, how Long does the agent go before it stops? What tools can it use available to it? And Dylan's actually done um, self uh correcting agent loops um, that he uh, can talk a little bit about. So Dylan, I'm going to hand to you to talk a little bit about what you found.

Speaker D: Um, yeah, absolutely. Thank you. Um, so my first points uh, that I'm going to talk about is that Neil, you just mentioned uh evaluations and I feel like a lot of people are familiar with like the general you know, evaluation frameworks and uh evals that you can run on an agent. Uh but also in the information retrieval space we have a lot of our own like evals and metrics like ndcj MRR that are like really relevant for uh, know search pipelines and that, that we use extensively. Uh, but going into your uh, what, what you mentioned about like self evaluating agents, um, so you know there's a lot of techniques uh, for, for that. You know the most obvious one is to spin up like a small agent and say hey, was that result relevant? Did it contain you know, all the information that um, the user was looking for? Um but the problem here is that you know you add another like LLM round trip, extra latency, extra cost and it makes your pipeline slower and more complex. And so something that we've been working on was computing uh, statistical uh signals that you know, very cheap signals uh based on the results. And so for example you know when you retrieve um, you know using like dense or sparse embeddings, basic basically uh, you get like uh, a score as a result. So for like 10 biddings it is between uh, you know like 0 and 1, which one being like the most confident and you know you return like the top K results or the top five or the top ten. And basically uh, you can do some statistic based on like the spread of those scores or uh, how the difference between the, the top one and the top second results. And basically from here try to create a programmatic um, you know signals that will, that that will be able to tell you if uh, the quality of your retrieval was good or not without virtually any adding any compute to your pipeline. And then once you, you have that, that signals that signal that tells you you know the, the retrieval was good or not. Then you can basically route your retrieval agents uh to different paths. Uh and you can choose for example to do more um, expensive retrieval method like a cross encoder or late interaction model or if the result was not good enough, uh, or not good at all. It's like there was a huge divergence in between all the scores that you retrieved. This is when you can invoke an additional LLM. Uh, in our testings this was like really a good way to tell apart like good from batch retrieval in a way that is really cheap and allows you to spend more money only when uh, it was necessary. Uh, and then uh, we also have something uh, on that topic on skills. So I'll pass it over to Jenny.

Speaker E: I think it's super funny. We're working like uh, actually a multi agent chain out there because, because we're passing tasks to each other. And Demetrios, you're the evaluator of how the pipeline is going. Right.

Speaker B: Nice.

Speaker E: So we're just emulating search and informational retrieval from our heads in the going and yeah, like the little addition as I think everybody knows about the agentic skills also. Right. Um, so they are also applicable in search. The problem is um, agents need to know how to use search as humans actually sometimes also do it bad because they also don't know how to search properly for something but they do it differently. With agents there is a good solution of teach them how to approach it is to show them skills. For example, we um, designed some bucket of search engineer skills which we're trying to maintain in quadrant which explained our agent which approaches our vector search engine on what to use and how to combine it to get different outcomes. For example, if you have a problem that your search doesn't show for you as an agent, relevant results at the top and you feel like there is something more to it, then there is a specific markdown part of the skill which tells hey, you should do this, this, this with us with our APIs. Um, I think we can share the link later or I could try to share the screen but I am so afraid to share. It's the hardest thing. That's why I'm senior developer relations. But I never managed to share a screen without sharing my private chat. So let's see, worst case, we're just dumping the link and showing how the skill involvement for the search pipelines is also helping our agents to use our search infrastructure properly.

Speaker B: Classic. Okay, I've got some follow up questions and there are some awesome questions that are coming through in the chat. But um, Eva, I wanted to give you some time in case there's anything you wanted to tag on here.

Speaker F: Sure. So um, Jenny mentioned a little bit of how we can teach agents how to know search. Um, so essentially when we added the uh, agents we moved from a very Static system of rag to something much more dynamic. But actually the search primitives uh, that are underlying remain the same. So one research paper uh, that came quite recently um, last month that I found really interesting on the topic is called sarah, which is the superintelligence retrieval agent. And the whole idea there is to, to um, achieving super intelligence which is when the multi round search is compressed into a single corpus retrieval. So we've essentially avoided the latency that Dylan was uh, uh, talking about the latency lags and um, what they've done that was quite relevant to our today's topic is they figured out how to use the LLM to enrich the documents. So have this iterative loop um whenever the search vocabulary was missing. So they looked at the queries that were generated, um, evaluated what queries were missing and essentially predicted omitted vocabulary and added to it before um, they passed it on to more traditional search paradigms like uh, VM25.

Speaker B: Okay, there's a lot to unpack with all three of you saying this stuff because basically let me just say what I caught real fast and then you can correct me if I'm wrong there Dylan. You're talking about different ways that you can use statistics to leverage and get better results with the retrieval and with the agents recognizing what to grab and what to retrieve and what is useful.

Speaker D: Yep.

Speaker C: Um,

Speaker B: tell me what. Yeah, tell me what.

Speaker D: Sorry. Uh, yeah, so basically it was like a cheap way to uh, get early signs into uh, bad or good retrieval without um, adding any latency or cost to your pipeline and then routes uh, based on those signals.

Speaker B: All right then from there, Jenny, you mentioned using skills and how you all at quadrant have created a bunch of skills. Are those open source? I didn't catch that one. That was the quick.

Speaker E: Everything that we do, everything ish that we do is open source.

Speaker B: So yeah, obviously drop that into the chat here and I can relay it to the um, chat or put it in wherever, whatever chat you want. But we definitely. I want to see all of those skills. And lastly Eva, you were saying that there is a paper on the super intelligent retrieval agents and the way that they're able to leverage. I didn't catch now, now my memory is starting to fade. Uh, trying to hold all these things in my hand head. The super intelligent retrieval agents were doing what now?

Speaker F: Um, so it's just a framework of how do we know that we've built an agent that essentially does successful um, retrieval.

Speaker B: Incredible. Okay, cool. So before I hit some of these questions from the chat and I'm Just looking at the chat about, big question that's coming up is like, hey, what about ground truth? How do we figure that out? What is. Because, uh, to know if the retrieval is correct, right. We also need to know what the actual thing is. The ground truth that we're looking for. Does anyone have opinions there? I see some heads shaking.

Speaker E: I would say a proper bunch. And uh, each of us has like, I would say different uh, sides of it and also about what ground truth is. I think, um, our problem is that we got the agentic systems and we got super excited and we're like, are they solving search fully? But the problem is that evaluations weren't fully solved before with the classical search. And now with agentic search it gets even more tricky because you're like, is the ground truth that the agent successfully fulfilled the whole task and, and arrive to the right point or is there is a ground truth on the edge point of the iteration of what it does in the process? And there are several approaches to it. There is a uh, ground truth in retrieval as probably people know can be created from some golden data sets per se. So some from the data that you already have and you know that this question should get this answer from your data set. Usually, um, in production it's not so easy to get it. But uh, LLMs got pretty good at helping to generate the synthetical data sets with golden truth. The one tip and opinion that I have as the person who previously worked in crowdsourcing, um, as a dev relof crowdsourcing approach. So where the humans were gathering these data sets is that don't immediately expect dumping your productional data to LLM that it will create your ideal golden set evaluation. And creating the golden truth still is the work where you need to be in the dialogue like you're in a dialogue developing a code project. So the relevance notion, the ground truth notion still comes from you as a domain expert. And that can be like LLM could be used as a tool to develop the ground truth data set that you can then inject in your evaluations. But when it comes, I know I have too much to say one little thing, one little thing. Um, but when it comes to the whole thing as um, the whole pipeline to be successful for you like solving the task, um, there are very different approaches on how to teach model to do the correct search trajectories which goes a little bit in more into reinforcement learning domain. And I feel like there is a lot of emerging there teaching agents to do the right decisions in search. And there Are the ground truth? Is the result being correct? Given the task, Given the search task, the search input. Maybe your audience knows Ralph root loops. So they're also to some extent applicable to the search. But Dylan here is coming from Arise and he was doing evaluations as the bread and butter. So

Speaker B: Dylan, this dovetails nicely into one of the questions that's in the chat. Um, someone was asking about how teams are versioning and evaluating retrieval pipelines in production.

Speaker D: Um, yeah, absolutely. Um, so, you know, something that I want to preface for, preface with first is that we are a vector search engine. So you know, we only provide only one of the cogs in the machine. And you know, this is really by purpose. We don't want to be selling like or providing like a retrieval agent. We really want to see, stay very close to the metal and provide like the, the best retrieval engine possible. Uh, but when it comes to those kinds of like evaluations and versioning, I um, will usually suggest to use um, like an uh, evaluation framework or platform. Of course, you know, I come from Arise AI, so I'm, I might, my opinion might be biased there. Uh, but there's a bunch of uh, free, uh, evaluation tooling that can help you um, do that, that versioning there.

Speaker C: Awesome.

Speaker B: And I think Andre, you had some other thoughts uh, about this too. Where are you? I gotta change the view to see. There he is.

Speaker A: Yeah, Andre's been doing uh, space on the screen. So I would like to say that uh, from my experience, what can also help if we jump back for the golden set is when you kind of have some kind of assumptions based on the human, ah, on some human manual labor work. What can be good for your golden set so that the LLM has some kind of uh, criteria that it can evaluate about. And I would like to say that it's, it's like an iterative process in which you try to improve your LLM as a judge time from time, so that the golden set that it is creating and the evaluation in general becomes better. So it's a long process, it's iterative process that just becomes better as uh, you train it more.

Speaker B: Yeah, awesome. Well, we've got more questions in the chat. Uh, see the. Let me just, Let me just grab this. Eva. Missing search terms are grafted dynamically. How does that handle prompt injection shenanigans, if at all?

Speaker F: This is a good question. End the paper and uh, feel free to share the link in the chat. Hopefully, um, you have it. I haven't found the cybersecurity aspect of it, but I think this would be like a really nice segue to um, agentic harnesses. Uh, and as you design your system retrieval, um, will not solve everything. You still need to add tools onto um, your agent that would make it more secure. But I think this is a really important design principle to consider because prompt injection is definitely a very serious, uh, serious threat. And I think especially recently we had um, openclaw and Hermes agent that came out. And with openclaw there was a lot of hype. But also we saw when we did not properly sandbox it that it was able to just go completely wild and access all of our data. Um, so yeah, I think that is not specifically mentioned in that paper, but consideration that um, every single good uh, agentic retrieval system should have.

Speaker A: Yes.

Speaker B: So let me continue with a few more questions from the chat and then we will go on to the next topic that I had in my notes. There is, um. My biggest pain point from John is he's saying, uh, biggest pain point I've had with RAG stacks is defining edge and entities dynamically with LLM analysts. Can you speak to different approaches to that problem? And I'll just throw that to the crowd. Anybody have strong thoughts?

Speaker E: I agree because I've heard it a lot on the conferences. But I believe it's uh, more of the graph ontology construction problem. And we as the vector search engine don't do the ontology and construction. We usually combine. Uh, because in many, many domains it's a very good complementary thing. For example in memory of agents it makes sense to mix both graph and vector search. That's how many agentic memory providers do, for example cogni. Um, but uh, I've heard at least from practitioners on the conferences and don't quote on me. I would ask Neo4J guys on that that um, if you atomize the task of ontology instruction in the sense you don't try to build the whole um, ontology of the domain, but you do it bit by bit, uh, in the more atomized setting that actually performs much better. Like the outcome is much better. Um, but I also know that Andre before worked with ontology. So maybe you have like a little point of view on that.

Speaker A: Uh, yeah. So can you repeat the question again so that I, I have the latest.

Speaker B: The biggest pain point I've had with RAG stacks is defining edge and entities dynamically with LLM and analysis. Do you have any approaches on that?

Speaker A: Oh, I would say that, uh. Yeah, the problem is that uh, these are uh, kind of as uh, Jenny already said, some kind of ontologies that you try to provide to the vector search and to the semantics. So you kind of try to put the explicit semantics into the vector semantics. And I think these are laying down two different uh layers. Yeah, one is application layer and another one is uh, is more of uh implementation Right. Of the technology. So I would say uh, right now what you can do is basically just uh, like the, the current uh state of art we can say is that there are lots of um uh GitHub profiles and GitHub, uh projects that are trying to combine Neo4j approach with explicit entities and uh, uh also nodes with something uh, that is uh vector based. And there are some uh successes there because uh, what does the graph tell you? It can tell you the relations, it can tell you the dependencies and in some sense it can give you additional context that semantics not always can get. You know because like you can have something that is 2000 from 2001 and something from 2014 and you want something specifically that is from 2014 to be more uh aligned on top. Uh you can, you can fix it of course with some custom um, some custom scoring, some customers, uh, relevance metrics that you provide by yourself. Maybe uh, again it depends on the domain but I would say there are these approaches that they basically try to combine uh vector and graph uh in one stack. Yeah. Or you can try to encode in some way uh information into the red. But yeah it's quite hard.

Speaker C: Awesome.

Speaker B: Well let's keep it cruising because I want to get into memory and it feels like that is a perfect segue into memory and how folks are approaching this, how you all have seen the best in the business do it and especially like I know there is a lot of talk about long term memory and the architectures that you have for those and also if they are that valuable because sometimes I will not necessarily want something to be remembered, but it gets remembered. And then for every single question that I'm asking my agent and I'll give you a concrete example of this, I told uh, I told the chat bot that I was vegetarian and now for things that have nothing to do with food it says well given your vegan lifestyle and I'm like I asked you a question about my taxes, get the out of here. Why would you need to reference my vegan lifestyle? So anyway, that's just a little bit of a tangent. Maybe we can talk long term memory or we can also. I'm going to throw different things out there like episodic memory. Do you want to go event based Versus semantic memory. How do you do the factual versus conceptual? Who wants to take this one? I'll throw, throw it up there and let anybody rise to the challenge.

Speaker E: Let me try to jump in because I relate a lot to this part about the vegan taxes. I think it's the classical, uh, you know, LLMs like please don't think about elephants. I think Dylan taught me that. And then immediately whatever you do, it's going to recall this elephants. That's why it's also remember when you do anything around memory is also to think about forgetting as a conception. But it's a very hard one, an interesting one. And there are like several toolings on how to do that. Um, so basically whatever I wanted to say is very quick is that I think people out there probably tried Hermes or uh, openclaw agents, because who didn't? It was my first wow effect, to be honest. Um, and we recently, recently been like day before yesterday actually. Uh, it's hype driven development a little bit. Released a plugin which allows to store episodic and semantic memory, um, of a Hermes agent in quadrant. And I don't have a super sick demo or anything. Um, but I chat with my Hermes agent a little bit and some of the memories of me trying to Prepare for this MLOps meetups got actually started in my quadrant cluster. I can show it. But before also like a quick note in case somebody just is not very well versed with the concept of different memories. There is a working one which is the context window. The chat you're in there is uh, semantic or factual memory. It's the one that something is true about you. For example, that Demetrius is uh, vegan and likes to do taxes. Uh, do you? No. Okay, doesn't like to do taxes then.

Speaker B: Uh, that is.

Speaker C: Right.

Speaker E: Happens to all of us. Then there is a procedure, procedural one. It's actually more like skills so what to do in order to achieve something. And the fourth one would be episodic one. This perfect thing for the vector search. Actually it's all the recalls of what you have discussed in the past and that it could help your agent to understand how to deal with this stuff better considering all its private, uh, previous knowledge. So I wired my Hermes agent to my Quadrant cluster specifically for that to show you. Let me try to demonstrate it. It's going to be a very lame demo, but we have a cooler one. So don't get, don't get discouraged.

Speaker B: Um, this is nice because people were asking for demos in the chat and so I think I just asked in our background chat. If anybody else has one Dylon, you have one too, huh? Um, that we can throw later on. All right, cool.

Speaker D: Yeah, I'll follow up with Emo as well.

Speaker E: Okay, I'm going to tell you, you're going to love Dylan's one and you're going to maybe like mine. What am I currently showing? Is it my slack or is it like UI of collections in quadrant?

Speaker B: Hold on, I gotta check it out. I see this. Uh, so yeah, quadrant you're good.

Speaker E: Quadrant I'm good. Okay. Uh, so as you see, this is a very impressive memory of 37 points approximately in quadrant. But basically what happens, uh, all of the facts that I have in the conversation with my Hermes agent, which I'm not going to obviously show share because I ask embarrassing questions all the time. Um, that's why I wipe the memory. So all the facts and all the turns in the conversations are saved, uh, with the different metadata of the sessions that they happened in. And that will help agent to recall some similar information. For example, I'm a big fan of this G Shepa approach, which Yann Lecun is recently saying that it's the, the next big thing for embeddings. So agent can find similar memories. You can see that with dense vector search, for example about this method, you will be able to find some other memories which are kind of about the same fact. Um, and this plugin, um, you can also visualize kind of the memories. There is not so much to see yet because there is not too much memories. But we can see that for example with the semantical embeddings there are similar memories about me preparing some smart facts about the Transformers attention window will being very important for context. Windows being very important for vector search. And that's why vector search won't be dead as much as Rag can become dead at some point. Um, and if I would start chatting now with my Hermes agent, for example, that uh, I will add the fact that uh, well, that Demetrius is vegan. So now all of my inputs are also going to be polluted about.

Speaker B: It was the other one. Uh, that was true.

Speaker E: I like doing taxes and Demetrius likes to do. I'm not showing you my telegram where I'm chatting with Hermes, but that's what's happening. So technically if the demo gods are nice to me, we should see that the points are gonna get updated at some point. But if they are not, the demo gods are just not nice to me and you will see the nicer demo.

Speaker B: We can come back.

Speaker E: We can come back because I think my Hermes agent obviously decided to sleep in the moment I decided to demonstrate something. But the general conception is that basically you have your memory bank because information about you that you do now will grow and grow and grow and some point MD files won't be enough to recall all of the information. And then you need some well index which will organize these memories, let them forget and let them be surfaced in the right moments. And I think this is very important where are we going? Because the information, as Neil said at the very beginning, grows and grows and grows, grows. Uh, Demogot absolutely obliterated me. But I know that Dylan has m a very cool one. So everybody can forget this. Yeah.

Speaker C: Oh, go ahead. Sorry, were you finishing?

Speaker B: I think um, we've got. Well, I wanted to talk for a minute about forgetting and that whole thing because I know that can be very difficult and there's also a question in the chat that I want to bring up uh, about like vector databases still being the preferred choice for long term memory. So maybe Eva, uh, can you talk forgetting real fast and then if you have any thoughts about vector databases I can imagine I know which way you're going to fall on that one.

Speaker F: Let's go. Um, hopefully you won't forget that one. So um, yeah, I mean Jenny showed what's like the benefit of memory and dealing with all of that. But um, I think memory is also polluting a lot our context window. So forgetting uh, is actually a massive and really interesting topic of how do we even decide? Um, being vegetarian might not be relevant when you fill your taxes but on your next session you might be trying to optimize your diet and that information will be very important. Um, so this is the world of decay functions and also relevant feedback. So agents um, have this nice thing that you can iteratively um, tell them what exactly in that particular search query is relevant. Um, and you can boost that. So this is the ability where I think vector search actually really shines because you can uh, figure out for that particular session what information should I include in my decay function and just forget. And that can be based on temporary timestamp or that could be filtered by um, keywords, etc. Uh, then there is another thing which is some memories actually over time that you're putting into the sections interior sessions might uh, be duplicates. So that's another really important thing is in order not to completely jam packed your context window and actually get out of your um, agent what you want, you can use vector search to duplicate those memories. So we have very clean context window. And next time you're filing your taxes. It's just that. And you can boost it and decay based on certain filters and scores. The information that's not relevant for the session.

Speaker B: That's awesome. Neil, I feel like you had some things to say. Yeah,

Speaker C: yeah. Um, I don't have anything insightful. I just wanted to um, hand over to Dylan with a little bit of context. We're talking about memory and forgetting and search and we weren't planning on showing this demo, but since there were requests for demos, we'll go ahead and show it. Um, this is a demo Dylan's going to show actually on device search. When we think about memory and forgetting, an AI, physical AI is becoming a really interesting area. How agents and AI on physical devices need to be able to remember certain context, um, be able to use that context and then how humans can interact with that is all something really cool that Dylan, uh, can show. Dylan handing over to you.

Speaker D: Absolutely, thank you. Uh, we're steering away from the agentic topic. Uh, but this is the good thing about vector search is that it's not only limited to rag and agentic. And so today, uh, I'm going to showcase you a project that I built for Agentic, a robotic use case. Um, can you guys see my screen?

Speaker E: Okay,

Speaker C: not yet.

Speaker B: Hold on, let me do my job. There we go.

Speaker D: All right, perfect. Um, this is just like the GitHub project. It is completely, uh, public and free and open source. So if you guys want to run this yourself. Um, so today we're going to run it on a pre recorded video, but it works with any kind of input. You can connect your camera, it will start uh, working live.

Speaker B: Can you make it a little bigger?

Speaker D: Yes, absolutely.

Speaker B: There we go. Now we're getting there.

Speaker D: All right, um, so basically we're starting with zero. Uh, that agent knows nothing about the world. It does not have like a database of like labels of what items looks like, of what is a chair, what is the floor lamp, what is a coffee table. And so we have like three kinds of models that run in parallel. So it's all running on device, it's all running locally. Um, you know, if I turn off my network right now, you would lose me, but I would not lose uh, that product. And so basically what, what's happening here is that we have first a, an image recognition, an uh, object recognition model that is called yellow, and then a second model that is an image to text model. And this model basically creates a label and also a description for each item that you sees. And then what we can do is that basically we can do semantic search on every item that um, the robot has seen before. And so you know, it started with like a completely clean memory when I, you know, when I open up that, that app, that robot has never seen anything before. And now it's able to recognize every single object and memorize them and then recall when and where he saw those objects. So you know, the applications for robotics are pretty much endless. And here basically you can, so you know, this Graph is a 2D representation of the actual embedding space. And so you can see uh, the robot's brain being built in real time and you can see uh, all the memory of basically all the concepts that are close together. So you see like all the hallways are grouped together here. We can see like there's like the dining table and chairs that is grouped together here. You know, we can see that we, we went into the bathroom so that m, those memories are pretty far away in the embedding space. And then you can really recall um, you know, everything that you has, that you have seen before. And the, the robot will be basically to uh, able to recall every single mirror that he has uh, seen. And yeah, and so, you know, this is just to. So, you know, we're not selling uh, embeddings models, we're not selling vision, uh, uh, models. We're really just that memory layer. We're just, we have a model that creates embeddings based on those image and text and we allow for very quick search and recall on those memories.

Speaker B: That's so awesome. Whose house is that? Is that your house? Oh, uh, I wish I was gonna say you negotiated a pretty good salary if that is your house.

Speaker D: No, that was just you know, like online tour from like a leasing agent or a realtor.

Speaker B: Oh, that's awesome. That is so cool. The fact that it's on device too with these, the small models is really impressive.

Speaker D: Um, yep. So, so you know, this was like a, ah, way to demo like Quadrant eight Edge, which is the embedded version of Quadrants. And this version can run on a Raspberry PI and not even like the top of the line newest one. I have one form like five years ago that runs it just fine.

Speaker B: Awesome. Now there was a question that I said in the chat I was going to ask and I totally skipped over it. Um, I told folks so we're, I want to get to it. It's um, asking about vector databases still being the preferred choice for long term memory or our knowledge graphs. And structured memory stores gaining traction.

Speaker C: Um, I would say um, from what I'm seeing it's not an either or but of both. Um, where they both have their uses um, in different contexts. But you know, like dense, dense vectors, dense embeddings are going to give you a lot of like semantic meaning. But like vector stores are not really the best for like relationships between uh, those different entities. And so using like we, we have a great partnership with Neo4J. Um and there's uh, a video I put in a private chat. Maybe you can share Demetrios. But like Graph Rag and using knowledge graphs with vector stores, ah together can uh, make a really good substrate for memory overall. And just to uh, kind of like shout out a couple other um, companies doing this is like cogni and mem0. Um, cogni has um, they use um, both transactional data or. Well they use transactional data. They use uh, vector data stores. Then they use uh, knowledge graphs altogether. So you can check those out too.

Speaker B: Nice. Uh, a question came through about the demo and it was mainly about how detailed all of these captions are. Like are you getting things down to the level of types of material and style or is it just coffee table

Speaker C: for Dylan?

Speaker D: Sorry, can you repeat that question?

Speaker B: You were looking at the chat, huh. Uh, you were busy or you were re watching the demo thinking damn, this is a really good product that I made.

Speaker A: Uh,

Speaker D: how do you know me so well?

Speaker B: That's uh, well played. So, so basically how much detail do you get from those different captions or whatever it's being labeled? Is it going down to the like hey, this is a postmodern coffee table made out of glass or is it just like coffee table?

Speaker D: Yep. Uh, so it really depends on like you know, the image to text model, uh, that you want to use. Uh, so I use uh, cglib too if I remember correctly, which is a very like you know, basic like open source model that can run on your, on your device. And so that model will usual. And I did not try to ask for much in depth definition. Um, you know, I was just looking for coffee table table here. So maybe that model has the capability to have a more in depth definition. Uh, but if not this one, I'm sure that um, you know, image to text models are very good these days. And if you want a really like detailed explanation um, of or like description of the objects, I'm sure this is something that can be done.

Speaker B: Nice. So we've got um,

Speaker C: I want to

Speaker B: bring us back a little bit to the topic du jour, which is agentic retrieval. We were going hard on that and the chat was loving it. And then we also, uh, the chat was asking for some demos and we got some cool stuff. But there's a lot that we can talk about still with agentic retrieval. And so maybe we can veer in that direction. Neil, I feel like you have some things you wanted to say.

Speaker C: Yeah, sure. Um, I can, you know, I think we got like roughly 10 minutes left and so um, as we sent her back on agentic retrieval, some general things that I think the audience should, you know, some ground setting, ground leveling things I think the audience should all be aware of with vector search is that um, even whether it's pertaining to retrieve, uh, agentic retrieval or just any retrieval, it's not only semantic search. And this goes back to the Neo 4J and graph rag kind of question too. But um, vector search is really. Vectors and embeddings are going to be the highest density format in which you can store information. You're taking in dense embeddings, you're taking tons of unstructured data and yes, you're storing that into a fixed dimensional uh, vector. But there's also sparse embeddings, there's different embedding models. All vectors are, is a format. And as you use different embedding models for images and videos and memory conversations and all this embeddings are just a way, they're a vehicle for this. And we can do keyword search and lexical search and we can do all these different exotic things, or maybe exotics, wrong words, just advanced skilled ways to find the needles in the haystack. And um, that becomes really important for agentic retrieval because just centering back on agentic retrieval, you want to have access to information the most efficient way. And we, we have looked at um, file search and other approaches and there are places for file search. There is not necessarily. You must use vector search for everything. Um, you must set up an embedding model, you must set up your chunking. You must do that for everything. It may not make sense in certain scenarios, but when you, as we start collecting more data and our agents become more sophisticated, um, their search tools become better. I think that um, vector search has a really strong place there. And I also intentionally use that term vector search instead of vector database because you want a system that optimizes for not just storing and um, not just storing and having like a blind retrieval method to get your vectors out. You want to actually have something that is focused on the Retrieval side of the equation, the search side of the equation. So like that's why quadrant we consider ourselves a vector search engine. Um, but yeah, I'll stop kind of rambling there. I don't know if anyone else wants to add thoughts to um, where they see vector search fitting into agentic retrieval. Overall

Speaker F: I can jump in on the scale. So Neil, ah, mentioned how like we're operating at millions of billions of vector scale and search will certainly get much much bigger. So early search primitives like the BM25s of the world are already five orders of magnitude smaller in terms of memory that they consume on the machine with uh, the latest one. And um, another research that I wanted to share with you comes from the uh, group research group called SID1. Um, they have a really interesting approach and they think that reinforcement learning, uh, for agentic retrieval is the way to get to that next five order of magnitude efficient um, agentic retrieval when the context window gets bigger, when the number of vectors that we have to parse through gets much much bigger. Ah, so I'll share the link so you can read uh, into their approach and why reinforce reinforcement learning? Exactly.

Speaker B: Yeah, I know Jenny, you had mentioned that earlier too. Maybe um, Eva, can you talk a little bit more to that because I like this idea. We were talking about RL too for evals and being able to create these environments, uh, which are super trendy these days I guess you could say. And so you have these environments and then you can eval them. But the RL for retrieval, what does that even like break that down for me a little more.

Speaker F: Sure. So what I got out of that paper is that the TLDR is basically they're using a multi turn RL and it's a mixture of synthetic and real questions. So Jenny mentioned before the golden data set and the reinforcement learning approach would be not that we have like human generated golden data set, but it is the environment where you learn, you interact with over time, um, with some real questions that resemble that golden data set. And some of them are completely um, AI generated. And there needs to be a reward system, uh, designed in place that rewards every single time that our agentic system hits the correct answer and punishes it whenever, um, the learning step was not made correctly. Uh, so the paper outlines how you can design those. But I won't be diving too much into detail there. So that's the um, that's the high level overview of uh, what exactly they're doing there.

Speaker B: Yeah, that is so cool to see. And I feel like there's so much potential there. The thing that I always wonder, and I also see that folks in the chat are asking this. Um, it's more along the lines of like, hey, I've got this agent. How do I either set up a framework or encourage it to do different kinds of searches or search techniques at different times? So I want to do some kind of semantic search when there's xyz. Like, is there a framework? Is there a way that. Is it a skill that we were talking about earlier that you can say like, hey, agent, here's. Whenever you need to retrieve something, here's what you should go through. And that feels like probably the easiest win I see. Jenny, you're shaking your head. You might have some thoughts.

Speaker E: No, because I think it's exactly. You answered the question. There are two ways, right? There is like the way zero shot way. You teach it procedurally, which is like the set of skills, the set of search skills. And it has its pros because it's easy enough and you can adapt it easily enough because it's just writing. Uh, I mean it's not just. But it's composing the set of instructions and we kind of know how to approach that. And the second branch that we are seeing now emerging is to actively teaching agents to search through this reinforcement learning environments. I don't see now easy ways of any practitioner to do that, but I feel like that might be the next thing in the future because everything becomes more accessible now. Right? Uh, so maybe we will get our RL gyms. Do you see my biceps? Yeah, I'm saying the RL gym, but we will get them soon where you can actually give the set of what you want to achieve and the set of the search tools and your problem. And it will converge to the like. The small search agent will be able to converge to a path which actually is, well, what you want to see in your specific pipelines. Um, we're going to stay and watch that because, uh, we represent kind of the default layer that humans or agents build upon. I wouldn't discard humans still from all of this picture, by the way, because I'm human, so I still sometimes want to search stuff myself. So I think it's just important to have the tooling which will be usable by all the categories of the search users. I think it was a tautology, but you get me, I guess. And I think what we were also trying to say that, um, vector search is also so much more than just retrieving your semantically similar facts. I feel like we kind of fell into this trap of the rag cage where you just think about it as the machine that spits out you the text chunk based on your question. But I think there uh, are other emerging interesting parts where it could be used for anomaly detection, data analysis at scale, image to text, search, videos to whatever audios. So all of that stuff. And at some point agents will be also able to use it for 3D and I'm really looking forward to see how they're going to approach this part of vector search.

Speaker B: Well said. I think that is a perfect way to wrap it up. I know that there are still folks in the chat that are stoked and asking questions. I will just mention that everyone who is in the chat and watching this right now, on your left hand side sidebar, there is a button that you can press that says match. If you go there and more than two people do it, you will randomly get put together with somebody else that is watching this. So you can meet someone that is just about as crazy, uh, on retrieval as we are and you can talk to somebody that is watching this too. So it's a great way to bond with the rest of the community if you want to stick around for more time. But for this session I think we are going to wrap. I know that there are some great places the Quadrant team hangs out. You all have an awesome discord. So I'll drop a link for that in the chat. And then of course, if you weren't um, already bought into everything that Quadrant's doing and following these good folks that are here with us today, you should definitely do that. Go and give Quadrant a star if you haven't already on GitHub and stay up to date with what all their amazing dev rel is doing. Follow them on LinkedIn and X and all the places. So thanks everyone for this excellent session. I will bid you farewell. And for those that want to hang around, click the match button on um, the left hand side bar and I'll see you all later.

Speaker D: Thank you.

Speaker C: Thanks everyone.

Speaker A: Thank you.

Speaker C: Thanks Demetrius. Bye.