Why Multi-Agent Systems Need Shared State, Graph Semantics, and Governance

Transcription provided by Podhome.fm Created: 08/02/2026 22:26:43 Duration: 3734.519 Channels: 1 Hello, and welcome to the Data Engineering Podcast, the show about modern data management. Your host is Tobias Macey.

And today I'm interviewing Ragnor Comerford about OmniGraph, a lakehouse native graph storage layer with Git semantics. So Ragnar, can you start by introducing yourself? Yeah, sure. Thanks, Tobias.

Glad to be here. Yes, so I'm Ragnar. I'm Dutch, but grew up mostly in France and Germany. And spent, I guess, most of my career, especially started my career really in kind of machine learning, computer science, because I guess the common thread was I was always obsessed with the idea of knowledge or information, or information theory as a whole.

And then kind of kind of after sort of really working quite a bit in machine learning and computer science, I kind of dove quite a bit into the life sciences or biotechs. I spent quite a bit of time working in San Francisco at Edison's a company called Q Bio that built like a medical digital twin, and built lot of their internal data infrastructure and graphs, and then also spent quite a bit of time in protein design, essentially using a large language model trained on amino acids to design sort of new enzymes and proteins.

So yeah, kind of the intersection of really computer science, ML and life sciences. And do you remember how you first got started working in the data space? Yeah, mean, as I mentioned quite early on, was I was really kind of interested in in sort of information theory and generally the power of like information and knowledge. So to me, you know, every kind of major outcome with, you know, both in research or in anything for like life science research to like hardware design was kind of downstream of how to use and operationalize information in a better way.

And to me kind of that was also fundamentally an infrastructure problem. And so quite early on when I was working in the life science, also realized essentially how fragmented all the sort of knowledge information was when if you went into a lab, you had sort of a bunch of data and data set like on local computers, and you have essentially these huge biological databases like UniProt and, and PubChem and things like that, that contain loads amounts of biological data. And yeah, really kind of across this kind of fragmentation information and the kind of the subpar infrastructure.

And so I kind of really saw that as kind of the fundamental upstream problem, and got really interested in sort of building better data infra and systems to really make use of all this information. And that brings us now to Omnigraph. So I'm wondering if you can just start by giving a bit of an overview about what it is and how did it get started? Yes.

As you mentioned, so Omnigraph is a lakehouse native graph engine built on top of like the open data format lens. And so they are kind of two lenses to look at it. One is actually working sort of backwards from essentially the new operator we were kind of optimizing for, and that is really the agent kind of workload. And so like, obviously the upsets you're making now essentially is that going forward, most of the knowledge work is going to be performed by agents, or most likely also aged working in the background, not just kind of chat assistants.

And then working backwards from essentially what the constraints that imposes on that kind of system. And I think fundamentally kind of split into kind of three different components. One is like context, the second one is coordination, third one is really governance. So context is actually when an agent is supposed to perform a piece of knowledge work, know, how to sure that the agent actually has access to the right information with the right knowledge.

And then also for multiple agents means you'd actually have access to us like a shared world model, shared understanding essentially of the world. And the second one is coordination, okay, once you go from one, two to actually multiple agents, you know, running, how do you make sure that they all perform essentially the correct global behavior rather than at the individual agent level. And then the third one is like, okay, when we have agents automating large parts of the knowledge work, you know, how to make sure that humans are still there where judgment their judgment matters, and where they can provide essentially a signal to where things are going right or wrong.

And this is kind of where also this kind of git start semantics that I missed earlier, and description of Omnigraph that became really important is like, okay, can you apply this sort of governance aspect of branching and merging also to the to the graph layer, to the structured data layer basically. And so yeah, that was kind of really the the story behind it. How do you build like the optimal substrate for a multi agent system? But then at the same time, think we talked about this earlier, like also looking at the existing ecosystem, obviously a lot of the technology or existing database that were designed for the human operator, but also because of kind of historical path dependence of how we kind of build things.

And so like the, the other sort of more bottom up reasoning or story behind Omnigrab is actually how would you build, you know, a database essentially from first principles, so assembling essentially all the building blocks of the amazing technology that has evolved, you know, fast forward to 2026, you know, and examples of these are like, you know, building a database on top of object storage, decoupling, you know, storage and compute, these open data formats, and then obviously the amazing query engines like DocDB or Data Fusion and things like that.

So really sort of the top down and bottom up sort of reasoning behind OmniGraph. And so you mentioned that one of the core challenges that you're focused on is this idea of multi agent coordination, which is definitely the major concern of the day as the number and capability of agents continues to expand and the patterns are definitely still very much in flux. I'm actually just starting to read the agentic mesh book from O'Reilly, so good timing here. And graphs have gained a lot of attention over the past year or two because of the fact that connectivity is one of the key considerations for any intelligent system because the relationship between things is one of the key things that influences the types of decisions you might make.

It adds nuance to the raw information, and there are a number of graph engines that have been available for a number of years. I think d o four j is probably the, at least, most widely known one, if not necessarily the most widely deployed. And I'm wondering if you can just talk to the core foundational aspects of graphs and some of the graph engines that are in the market and what was lacking that led you to decide that you needed to create a another new one. Yeah, no, sure.

I think maybe taking a step back, think also I see in general, there's a problem with essentially the taxonomy of databases, because obviously they were often formed for essentially forming like product categories or positioning basically, But I think we're from first principles, if we think about a database, I think we have to get a separate into like, along three axes, and one is essentially the logical data model or essentially the semantics, right. And it means like the query language, like how you represent the data and things like that.

Then there's the workload that is designed for, right, for example, let's say in OLTP OLAB, or in the case of time series for like heavy append only data. And the third is actually the physical manifestation or the physical data model that essentially aims at the kind of solving the constraints imposed by essentially the logical data model and the workload. So, and then independent of that, while one can essentially design around these things, and then the problems obviously that graphs or graph engines were kind of put into this bucket of like, okay, it comes with, you know, oh, it's optimized just for large scale sort of, you know, edge traversal.

And it's also meant to be kind of slow and not scalable, essentially, right? But to me, essentially, graphs was always more about the semantics that it describes is like, how do you represent your data model? And also what query language and then essentially how you build it in a physical layer, right? It can evolve or kind of be optimized for different purposes.

I think also because in the past, the database were built sort of in a very monolithic way, know, owning both the storage format, the indexes, you know, the semantics, query language, but now we're seeing actually, you know, decompose ability being sort of the much better paradigm, right, and decoupling search and compute and learning from mistakes, you know, that we've made sort of in the past. And so and the graph comes in as essentially at the semantics side, essentially, right, in that the information in the world essentially exists also as, you know, objects that relate to each other, right?

And this is kind of fundamentally how the world exists, and this is where sort of the ontology comes in, and this is why essentially the graph is kind of the correct representation of that world, because it's actually also in some sense the most generalized representation, because you can sort of represent any other things from a two dimensional array to, you know, like tables as a graph itself basically, right? So then it's more about, okay, you know, what workload or query patterns is it kind of optimized for the kind of underlying storage layer.

But yeah, but to be more specific, if we had to kind of compare to existing graph engines, obviously there is, you know, Neo4j, there's TigerGraph and so forth. But essentially the biggest differentiator is one on the lakehouse side that we really built this on object storage and decoupling like storage compute and building also on open data formats like our case, Lance, but you might know your Iceberg and sort of Delta Lake. That means you can actually eventually also plug in any other query engine because in the end it's just files on object storage, So there's essentially not this kind of lock in.

And at the same time, also we optimize also for the Git style workflows, because if agents become the new operators, they're fundamentally, you know, probabilistic writers, whereas in the past, you know, databases were made for trusted writers, but we have a software that, you know, is programmed to kind of write data and it has all the essentially validation logic, you know, beforehand. And agents are really probabilistic writers, so this kind of Gitstar paradigm becomes super important.

But I would have said even for humans, it would have been because since collaborating on shared data requires some kind of governance mechanism, coordination mechanism, and this is the exact one that, you know, Git or GitHub solved, essentially how do you allow, you know, hundreds of developers to collaborate on a shared, you know, artifact that needs to remain consistent at all times. So yeah. I think the two engines that I'm drawing the most parallels with right now for what you're building with OmniGraph are PuppyGraph on the lake house graph traversal side, and then the Git semantic side is the folks that are building the DOLT engine, which is a MySQL and now a Postgres compatible database engine that has native Git semantics all the way down to the storage layer.

And I'm wondering, what are some of the engines or systems that you drew inspiration from as you were coming up with the initial ideas and architectural layout for Omnigraph? Yeah, no, I mean, we are we took a lot of inspiration from many kind of different systems, including also around the kind of deployment model. Yeah, I can actually dive into that straight away. But like thinking again about the agent as kind of an operator, I think became also really clear that essentially having a declarative system became would be super important, because essentially, in some sense, you want the agents to be able to inspect the whole state of the system, without too much like searching around.

And at the same time, also you want to kind of reduce decision surface area for the agents of getting things wrong, right? And so this comes in those like the kind of query layer, right, where essentially you want your query language to be optimized for agents in the sense that you would post kind of guardrails on them. So one of these is essentially this schema type sort of query language, so you can actually validate the schema against sort of the query language. And for, you know, humans have been annoyed because every time they get an error and they kind of construct their dataset, for an age they can just recover from it in the next sort of iteration of the loop essentially.

But it's associated with declarative paradigm also kind of extends also to the actual whole operation of the system in that anything from the ontology to the queries we define, all kind of defining declarative race in order to kind of terraform, to to be kind of all inspectable. Digging into the query piece, many of the graph engines that are out there have consolidated around either one or both of Cypher, which originated with Neo four j and has become effectively the stand in for the I think it just got ratified, but the, GQL ISO standard, and then Gremlin is the other query language, and then, of course, Sparkle and the semantic web space.

And I'm wondering as you were developing Omnigraph and thinking about the use cases and the useful constraints, what are the pieces that you found to be most beneficial by having that more constrained typed precompiled query interface? Yeah, I guess when deciding whether to go with sort of an existing language or not, one had to kind of think of essentially what would be the advantage of what's actually advantage if a language is actually used by large amounts of people. And I think in the past, it was fundamentally about around the expertise or the understanding of people to write these queries, But then if we thought about essentially Asians as a fundamental new operator, this problem kind of, you know, vanished really.

And so and one could argue essentially that obviously the most have been kind of pre trained on like Cyphon this language, they should actually technically be better at writing these queries. But actually, it turns out that essentially, most of the errors in actually constructing the queries are actually because of misunderstanding of the schema or types that exist rather than the underlying sort of, sort of syntax of the query language. The LLMs are really good at sort of the in context learning, passing them essentially the syntax gives them the readability to construct like the query very easily.

And so like the actually schema typing became the most important one, because if agents start writing knowledge back, essentially having this kind of strong validation makes you the system really makes it much easier to keep the system that consistent over time. And it's actually the beauty then also of the ontology, because the ontology in some sense, it actually imposes a kind of a reasoning process on the agent, Because if essentially you are forced to, when you insert knowledge to kind of idea to the ontology, then you have to kind of follow the reasoning process, process that the anthology kind of enforces, right?

So for example, let's say you are analyzing transcripts of customer discovery calls, And now you want to essentially use a specific framework to extract information from it, like the jobs to be done framework, for example, right? Then you can include that directly in the ontology and make it typed so that it says like required fields and required edges and things like that. And so essentially when the agents want to write knowledge and essentially gets an error because it's missing a piece of information, it forces them to go back, fetch that information, and so really kind of imposes like the framework on the agent itself.

So I think this was kind of one of the major sort of benefits of doing that. And then the second one is also kind of optimizing for essentially the new workload that, that Asia optimized for, and ultimately on the database side is about assembling the right piece of context. And so that, that means that now actually writing queries is more of a sort of information retrieval task and including like search and multiple, multiple sort of modalities, like combining graph traversal, where you might look at an entity, follow its relationships, and combine that with vector or similarity search and then full text search.

So really kind of optimizing the query semantics and language for like context assembly, rather than the past kind of the reason the kind of design constraints behind like the different languages itself. Yeah, and then maybe the last bit is obviously decreasing essentially the decision surface area for agents when they actually write the query. Right? So for example, in Gremlin, have sort of imperative paradigm.

So it is kind of reason about sort of, you know, how to construct the query and execution itself. Whereas like in a declarative manner, you kind of leave it to the engine that you decide how to kind of optimize that queries. So essentially, agents are also subject to that dilution of their attention, right? So the more context you pass it, the more decision they have to make at the same time that the worse they perform.

So can you actually reduce the decision service area as much as possible for that sick task, like for example, retrieving a piece of information, and I think really optimizing for that essentially. Now, because of the fact that you have optimized for agents to be the predominant consumer and producer for the data within Omnigraph, how does that change some of the other potential applications that somebody might reach to it for, particularly in the case of graph applications. So some of the bigger ones are things like knowledge graphs, fraud detection, things like security information and event management for being able to correlate events.

Obviously, many of those applications can and do have agentic corollaries and agentic input as well as people are developing and evolving these systems. But how does it change the calculus of somebody who is looking gender generically for a graph engine? And what are the cases where they might turn away from OmniGraph because of that more strict and constrained query layer? Yeah.

I mean, that that's a good question. Because, obviously, whenever you kind of design for the operator, you kind of have also trade offs, right? So you can sort of make a system work for every type of, know, usage or workload. So in this case, yeah, we were not necessarily optimizing for the kind of traditional uses of like, you know, graphs, like as you as you mentioned, kind of in fraud detection, or, or these kind of type of queries, but really kind of an optimal substrate for agents.

So I think there's, there's already kind of really good solutions out there as well for these type of kind of workloads. But in the end, yeah, we're really focused on, on kind of, especially the semantics of the kind of shared world model for agents. And but yeah, at the same time, still like we are one has to kind of rethink about sort of the constraints that one designs for it, because existing database engines like Postgres or LTP were designed for high concurrency, you know, rights that are all trusted rights.

And now what does that, for example, that concurrency pattern look like now for agents? So for example, we think that the concurrency patterns are going be more at the branch level, because agents are going be performing like long running work independently, and then it's going to be reviewed, you know, synthetically and semantically and then merged into the truth. There's going be less of a need for high concurrency rights, you know, that they might conflict with each other.

But at the same time, you still need kind of transactional semantics, because ultimately it's also operational state that other agents rely on for coordination. And then similarly also like, it kind of merges some of the kind of analytical works in there as well, because partially some of the context assembly will involve like doing sort of more analytical queries. And also we look at sort of the evolution of the markets like Databricks, also have this kind of convergence now of you know, OLTP and OLAP.

But yeah, I mean, to answer your question, so we're not necessarily optimizing for the classical uses of graphs, essentially, because for us, it was mostly also about the semantics and not just essentially these kind of traversal workloads, although obviously, we have kind of, you know, optimized indexes for graph traversal and things like that, but it's not kind of the primary use case. So digging now into the architecture of OmniGraph, we've touched a little bit on some of the pieces, predominantly the fact that it is using Lance as the storage layer.

I'm wondering if you can describe some more detail of how it's built, how it's deployed, and what the adoption and setup looks like. Yeah, we kind of built on top of Lens, because Lens already kind of handled a lot of the lower level primitives and really well engineered. So those kind of don't know Lens. So that's a new new ish table format that's primarily optimized for three things.

One was the fast random access, so they would kind of, you know, retrieve essentially low latency like individual rows. And then second also kind of the multimodal modality of it to be able to actually blobs directly in the file format. And then third was like cheap schema evolution, so that actually adding a column, right, doesn't require like rewriting this sort of whole data set. But apart, I mean, these are kind of the kind of major components that they kind of advertise, but I mean, apart from that, I think they've done just a lot of amazing engineering on the format itself, right, handling things like obviously asset transactions, also indexes, and yeah, a lot of really good sort of primitives under the hood, and also really designed kind of like in a composable way, and so we kind of treat that as kind of the L1 or the kind of the layer one, and we kind of build the layer two on top of that, and so if you think about, you know, as a team you have kind of a fixed engineering budget, and then where do you spend it on, and we kind of really spend on that layer rather than reinventing, let's say, a new table format, and really we're focusing on essentially all the economics around like the graph, like the typing, the graph semantics, you know, the branching and merging mechanism, obviously all the kind of graph traversal and the kind of deployment around it.

But yeah, Lansen and also in sort of including also Data Fusion, who have as a kind of query layer or query execution engine, gave us a lot of like really good sort of building blocks to kind of build a really reliable sort of graph engine on top of that. One of the interesting pieces that I've seen from a number of the more recent graph engines is this adoption of columnar storage and tabular representation at the file level. And I'm curious particularly with the Data Fusion since that's largely a compiler for SQL in the the general case that I've seen it used.

What are some of the aspects of impedance mismatch that you've had to work around between this graph as the core data model for what Omnigraph is aiming for versus this tabular representation of the data on disk? I mean, yeah, one of it obviously that we had to kind of design some of our like, custom like indexes as well, because I think a large part of kind of optimizing the queries actually was around the, you know, having the right indexes. And so for example, you would kind of build our own sort of adjacency matrix, basically, or CSR, essentially for the query execution layer.

But yeah, lot of kind of the sort of query or traversal mechanism actually were handled pretty well, by being able to push down essentially predicted down to Data Fusion or actually Lens itself, because Lens also itself handles a lot of the kind of the query logic behind the scenes. But yeah, so for now, but basically for us, most of the constraints are around like not having essentially primitives for graph traversal where we had to kind of build the index there. And also we're still making like lots of improvements on that side of things.

But so far, yeah, the machinery kind of exposed by Lens already provide us like a lot of help essentially there. One of the other ways that using that columnar tabular storage instead of a graph native representation on disk can impact the overall capabilities of a graph engine. And one of the reasons that graph engines are so difficult to scale horizontally is the number of degrees of removal that you're able to traverse in a single query. And the fact that the impact of supernodes on a graph is one of the reasons that horizontal scaling of traditional graph engines has been so problematic over the years.

And I'm curious what are some of the design choices and intentional limitations that you're accepting by virtue of not having a a graph native storage representation and instead relying on the ecosystem advantages and the architectural advantages of Lance as that foundational layer, or some of the potential scaling problems that you hit if you're trying to do something like a 30 degree hop across multiple different interconnected nodes. Yeah, so so far, we haven't really kind of optimized as much for these sort of very graph analytical queries, right?

Because, as I mentioned, we were especially optimizing for the graph semantics or the representation of the data in that sense. I And think a lot of the queries also are, can be essentially represented then as like, you know, really well in terms of like SQL queries, or kind of these predicates sort of push downs, and also doing similarly for vector search or full text search. So yeah, so far we haven't really optimized for these sort of graph analytical queries, but you know, we might essentially at some point, you know, eventually even create our own format or, or do like, like further optimization.

But yeah, this wasn't currently like much of a design constraint at this stage, because again, we were kind of not optimizing for these sort of really graph analytical queries basically. And as you have been working through the early stages of building OmniGraph, discovering some of the design space around it, understanding how people are using it, how it all fits together, what are some of the ways that your early assumptions have had to shift and some of the evolutions of the design and goals of the project have grown as you've gone through this process?

I mean, less so in terms of kind of the design, but obviously, we were kind of aware that, for example, object storage give you sort of very, like, quite high latency, basically, right? And so initially, a lot of these the system kind of worked really well on fast on like a local file system, but then essentially when deploying it on object storage, so that essentially using it essentially over like multiple weeks and accumulating low version history and things like that, we kind of obviously hit a lot of the end of the limits around both the latency imposed by object storage reads.

So this is kind of on rise as well, and so this is kind of where we had to do a lot of optimizations of like minimizing the number of reads and writes and do some of the redesigns, you know, around also like caching and the over kind of write and read patterns. Of course, it's kind of obvious beforehand that, you know, object storage has this kind of latency constraints. But of course, like as you kind of, you know, as the design kind of evolves and the workloads increase, I think this is kind of where it was really hitting the limits and had to kind of redesign some of the mechanisms around that.

And so yeah, that was probably more of a kind of technical requirement, basically. But so far in terms of kind of the, the design choice around the operator, we we haven't kind of had we haven't had to do like too many changes yet, because I think a lot of the insights, you know, that went into Omnigravus who came from like, you know, many years working, you know, across like different data stacks and those having worked like with with agents and what they need and things like that.

So, yeah, we'll see going forward if if things sort of change. And then because you are building on these other established projects of Lance and Data Fusion and Arrow, what are some of the ecosystem advantages that you gain from it as far as other integration points or various other interaction patterns that come for free because of that selection? Yeah, I mean, I mean, first of all, so because it was sort of built in a in a composable way, you kind of have really this ecosystem advantage of be able to kind of plug in different parts in and out.

For example, obviously, having open data formats means you use it with other query engines, also leverage different types of storage medium from S3 and, and SEF to like, be able to run it in an embedded format as well. But then you also have the whole kind of error ecosystem, because you know, Lenses are based on error and then all the interoperability there with other libraries around zero copy with your pandas, polars, that DB and things like that. And then last as well, so we we kind of have made embeddings also pluggable in terms of providers, it can sort of interact with, or you can use like any model on top of it.

And then lastly, we kind of provide also this sort of the data layer or the data substrate, but we don't kind of impose essentially what it looks like on the agent orchestration layer. So we want to kind of design in a monolithic way, of defining also how agents are supposed to be running. So in that sense also it kind of also as we see like the actual agent orchestration libraries are kind of changing all the time and evolving, and so be able to to kind of work with them in a sort of like composable way, think is really important, because you know, lot of companies or products out there kind of give you everything from the data layer to the agents, the models.

And so what it also means is that it pairs really well with, you know, I think companies or organizations that want essentially a sovereign AI stack, be able to actually own essentially the AI stack from like using local and open source models to maybe even their custom hardware. And so this kind of really plays well with sort of the sovereignty or local first kind of community there. One of the other interesting pieces is, as you mentioned, Iceberg has a huge footprint in the data ecosystem.

There's a lot of information in Iceberg tables. And I know that Lance has at least a one way door of being able to migrate iceberg tables into Lance tables. And I'm curious if there are any thoughts that you have on being able to introspect and generate a graph on preexisting data versus only being able to start from a net new state storage and write data into the graph? So yeah, in terms of lens, for example, we're definitely able to do that to kind of use like existing lens data as well, because also the nodes and edges are actually represented as individual lens files at several lens tables.

And yeah, Iceberg is definitely sort of the most, you know, widely used format out there. So, so yeah, be able to kind of work with these, these existing data sets is quite important. But and LENS provides kind of like an interoperability that to be able to kind of migrate from Iceberg to LENS. But as the currency stands, we don't necessarily support like be able to query existing sort of iceberg data, because essentially, we designed really OmniGraph, not just as a kind of query layer or analytical engine, but but really use as an operational transaction database that for writing as well.

And so and for the right side, we rely on lot of the lens sort of machinery right around, you know, as a transaction, things like that. So to the right side wouldn't necessarily work, but we have got quite a few requests as well to be able to kind of query other, you know, formats. So there's definitely something that we kind of have on the roadmap, but haven't optimized for that yet. Because really, we're focusing on really building sort of an operational state layer rather than just kind of query layer, because I think there's also a lot, there's a lot of players already sort of under, you know, adding maybe a semantic layer on top of the existing data sets, maybe to query those, whereas our focus is really kind of on that, you know, shared substrate for for for like long horizon, like multi agent systems.

Yeah, I mean, one of the kind of primary use cases emerging right now as companies are kind of understanding that, you know, the fragmentation of their context and also the lack of essentially an explicit, you know, ontology or world model is kind of the bottleneck to really be able to automate work, has meant essentially that this is the idea of this company group brain or the context graph has become like really popular. And so this is kind of one of the primary kind of use cases we're seeing where they would use like, they use Omnigrap essentially to represent their sort of shared ontology well off the organizations, and then using that essentially as a shared substrate for agents to to get you perform, you know, background work.

And at the same time, there's essentially different instantiations of that as well of like what the company brand means. And then, you know, one of kind of the two primary domains are on one of the kind of rev rev ops, like GTM side, essentially of building sort of, you know, more elaborate than a CRM, but something actually more custom in terms of like how they do kind of the whole revenue process around like prospecting and enriching, essentially building sort of that shared sort of world model of their sort of sales organization, and then running agents to enrich prospects to, you know, running all the process around that.

And the second one also being now is like agents are now performing most of the coding work, also building essentially graphs or ontologies around the SDLC or the software development life cycle, Anything from, as we were talking about, like indexing a code base to kind of high level actually representing some of the things like projects, tasks, dependencies, assumptions, but also, you know, different code bases and their dependencies between each other, and be able to see essentially how changes propagate, might affect other systems, and the blast radius and things like that.

Because I strongly believe that actually going forward, a lot of the energetic engineering is going to be performed essentially by having this kind of shared world model of essentially the whole engineering organization and all entities around it. And, and both like Simpson in production and existing code bases, and then having agents kind of work in loops around that sort of shared world brand. But yeah, I think there's also other interesting use cases you come across around also like, you know, in my background, like around like research, R and D, or in life sciences and things like that, where you have also a large amount of entities that you know, agents need to be tracking and enriching information for, know, from proteins, molecules, etc.

And there was some people using it even for essentially like a training like ML models, and they're of using the graph, you can have more as a kind of indexing layer, then you can actually train directly on the lens files itself, because, you know, without essentially having to obviously, query the data, but essentially kind of be able to work directly on top of sort of the the lens files. One of the other interesting potential free benefits that you might get as well is because you have the storage layer and object storage, which is generally fairly inert.

You don't have to have a lot of compute running for that to operate. And the fact that Omnigraph itself is written in Rust, which has excellent support for WebAssembly. I'm curious if you've done any experimentation for being able to move the actual query interface into a browser and be able to ship an effectively serverless graph introspection to a web browser using open data in a lance store somewhere. Yeah, no, that's a super interesting idea.

And definitely kind of has been, you know, somewhere, you know, on our roadmap, you know, but haven't prioritized this yet. But I think that's definitely like a, I think it's an interesting idea. And definitely, we'll explore that. And digging into what you were discussing, as far as modeling some of the software development cycle and some of the agentic collaboration, one of the projects that I'm actually building using OmniGraph is this tool called Witten that I'm building internally for my day job and using it to track things like the agentic memory so that rather than it just being a markdown file that sits on my disk somewhere and nobody else ever sees, it is stored as a node in the graph.

It has the ability to be superseded by newer memories. So you have some evolution of some of that organizational knowledge. It has the ability to track projects that have tasks and their dependencies between tasks, the dependencies between projects, etcetera. So you can have some of that richer detail beyond just a linear view of GitHub issues somewhere of there's this GitHub issue, and maybe there's some linking between them, but it's still a fairly flat relationship.

You can actually have remote URIs from the node in the graph to some of those GitHub issues so you can actually build some of that hierarchical state within the graph. Because of the fact that it's a shared state space, you can have multiple different coding agents coordinating on those projects and tasks and memories so that as I'm working in one piece of the overall system, I'm providing input about some of the broader organizational details such as this repository is actually slated for deprecation, so don't bother doing a bunch of maintenance work on it.

So that might change the way that somebody else's agent is thinking about the priorities of the tasks that they're picking off because it can see the fact that, oh, this is actually going to be deprecated. I'm not going to put in a bunch of effort. I'm gonna focus over here instead and just the ability to turn some of this agentic software engineering from a very siloed single player mode into a more collaborative multiplayer mode and have some of that evolution of capability beyond what is reasonable within a single machine.

And even just as I'm developing it and I haven't quite gotten it deployed to that multi user state, but even just having multiple agents running on my machine, I've seen it pick up some of the fact of, oh, well, this task over here touches on this thing I'm doing over here in this other session. And this memory was stored three sessions ago, but it gets resurfaced because it's relevant to the work that I'm doing now and just some of the ways that that self improving system comes about because you do have that rich, detailed, useful history and detail about what's being done and what's been done changes the behaviors of the agents themselves.

And I can only imagine some of the ways that that will change some of the things, like, as I evolve this to support something like a operations agent that lives in a Kubernetes cluster that's triaging errors from the logs or incidents because of the fact that it's able to tap into some of the organizational memories and tap into some of the code graph, it can get some of those richer semantics and more accurate debugging than if it were just coming in fresh and only has some of the context that it's able to retrieve as part of the incident.

Yeah, no, for sure. I think like, I think also what it eventually boils down to is actually that, that agents really need more kind of explicitness of the world they operate in, right? Because I think with human collaboration, human organizations, lot of that kind of knowledge or information is kind of implicit, you know, of course, you know that the production database is, is there, or it depends on x, or like that person is involved there. Essentially, age is supposed to be automated off the work, they don't have that kind of explicitness, you know, essentially, that's where essentially they hit these blind spots that can have like severe kind of consequences.

Also on the other side, it can also like improve, improve as a massively also their ability to actually perform useful work, because they know exactly what are the dependencies between the different things. So yeah, completely agree there. I think what's kind of also interesting is that if we think about agents as kind of also kind of constructors of new knowledge or enriches of knowledge, and we can also think of the, of the graph as kind of, you know, an index or like indexing information, able to sort of like cumulatively index more and more information, because the point is that from different kind of low level facts, you can essentially construct more information that you also store in the graph.

You can essentially, I think this kind of, excuse me, maybe against like some of the system that say like, okay, we can just, we're just kind of retrieved information from the source system essentially at like query time, right? And essentially that sort of means that at query time it has to kind of construct maybe more knowledge and one can only for example here is like entity resolution, that if two different pieces of data essentially are the same, but they are not linked essentially by a foreign queue or something like that, then essentially has to kind of duplicate those entities at query time.

But if you do that essentially beforehand, essentially having an agents enrich and for example link up entities that are know, duplicate or do this kind of semantic resolution. That means that essentially now you're kind of performing proactive work and constructing knowledge that then you can use that instantly at query time. And I think I kind of view that also as like a major paradigm that's going to emerge since you have agents just cumulatively constructing knowledge that can then be built on by essentially other agents as well.

Yeah, no, for sure. I mean, because not just the open source, but also because we are kind of optimizing also for sovereignty, because I think we believe that the old models of like cloud and SaaS of kind of locking in your data into an ecosystem doesn't work anymore. There's just too much kind of, you know, market pressure right now. And it makes sense because essentially, if everybody has access to the same level of intelligence, essentially all your alpha is basically in the knowledge data you have, and then you don't want it to be either to be locked into either some kind of software as a service or essentially a cloud sort of database data provider.

So essentially sovereignty is, I mean, something that we believe in like philosophically, but I think also what the market now demands, given that also, you can just build so much internally. And so that also changes kind of like what then the business model, you know, looks like. And so the approach we took is kind of maybe similar to what we're taking is similar to kind of TerraForm in that, essentially, we kind of offer essentially the control pane right around essentially managing this whole kind of state, because you know, there's all kind of things like, you know, managing of different graphs and different clusters, and then also schema migrations, version migrations, and then the whole idea of kind of, you know, SLAs and ensuring like your system is actually operational and the observability around it.

So this is kind of what we're optimizing for in terms of like, you know, bigger companies. And we're kind of also working on a self serve offering to just get sort of like in a graph, instant graph to kind of work with, but in that sense, you know, the open source plays in well there, because I think bottom up adoptions was seen also with, you know, things on the success of Clickhouse is kind of a major, you know, driver of like, you know, adoption of new technology. And so, yeah.

And as you have been building Omnigraph, popularizing it, working with some of the early adopters, what are some of the most interesting or innovative or unexpected ways that you've seen it used? Yes, one team of you using it also as a kind of for automating like, essentially, like an automated basically trading system. It's not a high frequency trading system, basically like a more one based on like fundamental research and uses kind of the graph as a shared artifact to perform a kind of like market research, and have, you know, multiple agents like enrich information producing like signals.

Think that was kind of an interesting one. And the other one that I mentioned earlier, also one that we hadn't necessarily optimized for is actually was actually using the graph as kind of an indexing mechanism for kind of like lens data sets, and then using that essentially for essentially training like ML models, basically. So these were kind of two interesting ones, because I mean, ones that we were expecting were the ones I mentioned earlier around the company brain, the engineering or code graphs, and then also kind of a multi agent research.

I think, going forward, think there's gonna be, I think a lot of interesting work on how to kind of engineer essentially the system as a whole. So basically the combination of, you know, the ontology, the graph and the state in there, but also the agent configuration, so which agents run with what prompts, you know, at what cadence, and in that sense we're also, you know, working on the event streaming layers, you'll be able to emit events from changes in the graphs of the CDC, so that agents can react to specific changes in graph and then perform some piece of work.

I think what's going be really interesting is basically how do you engineer this whole system as a unit, know, rather than, for example, so far, like in terms of the loops engineering, the focus has been on, you know, engineering the harness itself with specific agent, but now how do you kind of optimize the system as the old that is configured by, you know, the multiple agents with the default prompts, and then also the ontology that defines basically the state space that evolves basically, right?

I I think I kind of view this loop more as a sort of like, almost like a dynamic or system, know, that, you know, has a, you know, a timestamp, and then each timestamp you have like, you know, some evolution in the, in the, in the data. And then essentially ontology or the schema forms basically the the state space that is kind of more constrained, right? Because you can imagine if you have, let's say, no constraints, or you just have a, let's say, one markdown file that multiple agents read and write to, of course, like the first problem is you will get conflicts, right?

Because in terms of like concurrency, because there's no sort of decoupling modularity. But I think the second part is actually just going to very quickly drift into like a high entropy sort of noise basically. And so, but in the case of the graph and the agent configurations, like the question would be, okay, how do you engineer this kind of configuration of the system as a whole, that you kind of like reduce entropy kind of over time and kind of get essentially, you know, a lower entropy state of that graph, but also with essentially not just like, you know, a lower like entropy or, but also that the information there is actually semantically also interesting or useful.

And that will depend a lot, think actually on the sort of the configuration of system. And maybe it also mean that we might be able to use some of that as a kind of reward signal for like post training or reinforcement learning, essentially when you have multiple agents acting on that system, and then optimizing the models to operate as a kind of joint system rather than the individual harness itself. I think like the thinking there is still like quite early, but I think this is kind of my opinion, ultimately, what what's gonna matter is, like, really engineering the the that kind of outer loop as a whole, basically.

Yeah. And because of the fact that you do have the Git semantics as part of that core capability and the versioning, I can imagine that it would have a substantial benefit for things like experiment tracking or prompt evolution through something like a DSpy and just some of the ways that that can simplify some of the MLOps space around the agent lifecycle and agent evolution. Yeah, exactly. Yeah, because one way to view the branches is definitely not just as a sort of governance mechanism of like verifying outputs and things like that, but also as a way for agents to explore like parallel solutions in the solution spaces basically, and then kind of discarding like some of the solutions.

And so you can imagine that kind of multi agent system in that you might have the graph itself that emits events based on changes in the data, but essentially that agent subscribed to changes maybe only on the main branch, but then essentially on the other branches, and the other agents react to it, so that essentially only when it gets merged, does it trigger sort of the downstream, you know, effects basically. And so yeah, I think it's really interesting to kind of maybe view these branches as kind of a way to kind of explore like parallel solution spaces.

I think similarly also to have loops itself on the branch so that you can have verification going on similar to how we do, you know, coding agent reviews on pull requests. Like right now, we're of doing loops on these pull requests, I just have a review, then we correct it, and we have another review. So So can we kind of, but mostly it's us kind of, they say like, oh, look at the PR comments, you know, and now you know, correct them, but we're doing that manually, but can you automate that kind of process itself, and then provide some kind of, you know, more compressed observability to the human so that essentially they can add that judgment in a sort of, you know, without being exposed to the high dimensional changes, but to kind of maybe what are the downstream impacts of, you know, merging that that truth, basically, and things like that.

In your experience of building this system and technology and business, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process? So yeah, mean, definitely building a graph engine is like is like not the easiest thing to to build. And also like, there we were kind of also relying essentially on coding agents, and essentially you will see them now performing like essentially super well when it comes to like designing, like building like a CRUD app or some kind of like UI and software and things like that.

But you know, in the case of like building GraphQL, there's a lot of kind of invariants that you need to maintain, For example, like, okay, what's the consistency level you ensuring? Like what are some kind of latency constraints and things like that. And so like, a lot of changes will have a lot of kind of downstream effects, and also around also essentially the complexity of the task itself. And then that's where we kind of saw a lot of kind of the limitations of the way kind of coding agents work.

And so this kind of graph like, you know, loops of like finding the context that they need. And I think this is kind of informed as well a lot like building this kind of graph on top of a code base to be able to kind of have essentially a better management of the knowledge around the code base. Because really the problem with the with the with the agents that they will see like a function, and then assume it works a certain way, or like for example, the case of lens, where we, you have this kind of function that, you know, performs the operation on the underlying fragments, then you kind of expect they will behave in a certain way, and sort of, well, mean the agent expect that it will behave in a certain way and or assumes it because it's kind of obviously optimal to not like read every piece of code file and like look at the actual source code, right?

But it actually makes a lot of assumptions or hallucination in that case. I think this is kind of where it became like actual botanite became is actually how do you represent that notion in a way that essentially when agents are coding, they can actually retrieve the right piece of information without bloating the whole context size by saying, you know, read every kind of code file. And I think this is kind of where a lot of the intuitions emerge, because of the complexity of that engineering task around, you know, how to best use like, coding agents and actually building essentially a knowledge layer for them, they're able to kind of incrementally build up knowledge about like source dependencies or its own code base, and then rely on these facts when it's designing different pieces of the architecture.

And I think this is kind of one of the most interesting ones, because we're not necessarily hitting these limits when you're kind of designing more simple systems, or especially when it comes to green field projects, but anything like when you have, you know, bigger upstream, like bigger library dependencies where we used to depend on the low level implementations of them, or your codebase itself is like brownfield and it's like a huge, you know, huge code base, then then really the knowledge around the code base, how do you optimize the representation, the indexing, such that essentially at a query time when it is performing coded, it can very quickly get the right information to make the design like optimal and things like that.

Yeah. And one of the things too that I was building in my project that relies on OmniGraph is that graph representation of the code based on tree sitter parsing of the code that exists on disk and then being able to generate heuristic connections across repositories so that for a service oriented architecture, as the agent is making changes in one place, it can have some visibility into the potential impact across process boundaries so that it can account for that either as designed to say, hey.

I need to go do something in this other project now as well, or I need to do this in more of a two phase process of this evolution to make sure that you don't introduce breaking changes naively. Yeah, exactly. 100%. I think it was the boils down to like our previous I think a lot of the the problems with Alex's kind of the the blind spots, right?

And essentially, like, it doesn't know what it does know, and essentially that's where the the edge or the link in a graph comes in, because if you link something that when it looks at that node or entity that it sees all the things that depend on it, that's how you kind of reduce the blind spots or in some sense the in kind of retrieval language, you kind of optimize them for the for recall of the information, because you kind of prepared it ahead of time in some sense. But yeah, I think like if I had brought down like the limitations of the agents, mostly boils down to really the like, blind spots or the or the recall side, maybe less so, for example, the the precision.

And so for people who are either looking for some substrate for a multi agent system or they're looking for some graph engine, what are the cases where Omnigraph is the wrong choice? So yeah, if you say specifically for kind of agents and multi agent system, then I would say it really depends on sort of the the complexity complexity level level of of the the system, basically, right? Because I think I think once you kind of start simple, and that's where a lot of people get started, right, maybe a markdown file as a kind of shared state layer, right?

And I think this works well. So if you have a kind of a single agent and not multiple agents reading or writing to it, but also that the actual knowledge you need to represent kind of, you know, fits into like a context size, because that's going to outperform like any other system, because you're able to kind of retrieve everything within the context. But I think as kind of the amount of data increases, both in terms of volume, but also in terms of the semantics that you're actually tracking multiple kinds of entities, and at the same time you're also increasing the number of automations you're running, so in that case, like number of agents you want to run-in the background, I think this is kind of where really Omnigraph starts kind of earning its, its really kind of potential.

And I would say, yeah, Omnigraph should be used kind of for very simple use case, because it's still quite a complex system to operate to understand. I think obviously now things are becoming easier to operate with, with agents, right? I think most of the users of Omnigraph actually just been using the Omnigraph skill or pointing at essentially all our documentation was optimized for agents and is able to actually guide them through it. And we have these cookbooks and things like that, right.

So the onboarding is actually much easier now. And I think this is why in the past graphs were like less popular. I used to also be kind of quite obsessed about Sparkle and RDF and things like that. And like the amount of complexity that you need to understand around us all the semantics and that that's why then there was never kind of a wide adoption.

But now essentially, agents are able to kind of really bridge that gap and actually make more complex things adoptable. But still, would say kind of to answer the question, I think Omniglass shouldn't be used for like very kind of simple workloads. And so that it also mean maybe like, I think the two dimensions, like as I mentioned, the complexity at the volume of the data, then also single player versus multiplayer. So Omnigraph makes much more sense when essentially multiplayer, like multiple agents or multiple agents and humans essentially.

And as you continue to build and iterate on the Omnigraph project and build the Modern Relay company, what are some of the things you have planned for the near to medium term, either for Omnigraph specifically or any associated projects? Yeah, so for Omnigraph, obviously, like we are kind of also quite focused on the actual deployment and essentially making deployment much easier with the control pane, right? And, and everything in Omnigraph is like declarative, so you know, the context, ontology, the policies, and so making that kind of state management much easier is something that we're working on, and also a sort of like self self deployment.

But at the same time, I think the last gap as well is around also the kind of CDC or events sort of streaming, because I think the automation eventually should be really event driven so that essentially agents can react to changes and kind of perform work and then entering that layer is going to be kind of a huge unlock to build this sort of this multi agent system because currently it relies on like crueling jobs, like polling, essentially the state, basically be able to react very quickly to events going to be sort of really like an unlock to build these, multi agent systems.

At the same time, kind of also the philosophy is like obviously again, like composability, and also where does actually UI play a role now that essentially most of the work is performed by, you know, agents and where do kind of humans come in. I think sort of the UI is just to kind of evolve to be sort of more focused on like a job to be done, right? Maybe it's like, you know, you are you where such a human judgment observability matters. And so for that, we kind of build these, we're building these notebooks similar to maybe the concept of like a Jupyter notebook or a dashboard, and that you can essentially generate this kind of UI blocks that essentially look at a single lens or slice of the graph.

And so and also perform mutations that you know, might make sense for human records. Sometimes moving like your tasks across a Kanban board is like more efficient in prompting the agent 20 times you know, to move like the different things around. So essentially, can you generate essentially individual UI blocks that are optimized for a very strict job to be done that maybe you have to do every Monday, every day, or maybe just for an event that you're organizing or whatever, you kind of look at the slides and share the UI for it.

And so we're kind of also burning out that layer. But again, it's supposed to be interoperable and sovereign, so we nothing is kind of monolithic in that, you you use everything, but essentially composable, you might, you know, vibe code your own UI on top of it, or plug into a via CLI SDK or sort of API essentially to any other sort of downstream system. So really kind of optimizing there also for interoperability. Yeah, and I haven't dug into it much.

But I've also seen references to things like MCP UI, where the agent can call into some UI generation capability that has some scaffolding, but is largely driven by the agents. That'd be interesting to see as well how that could be factored into some of the data that comes out of Omnigraph. Oh, yeah. For sure.

Yeah. So you mean that basically the MCP will generate the UI within sort of the chat conversation itself? I'm not entirely sure how it is intended to function. I just know that the idea is through tool calling, the agent can dynamically generate the UI at the time of it interacting with whatever the objective is.

Yeah, exactly. So I think yeah, there's definitely a major use case. Like our process into the notebook is also that it's kind of like in between generating, let's say an AI generating the actual source code, in like React to build like the full application. And it's kind of like one layer above, and it's based on like high level abstraction, that's also declarative that that exposes like, for example, the queries and mutations, so that you can just assemble, so the agent can actually write a whole dashboard in a few seconds on these like building blocks, rather than like, you know, building 20 different React apps that don't talk to each other, but that's generating these kind of blocks that you can then move around.

I think, and it is again, like reducing the service area or decision service area for an agent on like, on the only corp file it needs to get right, right? In this case of generating UI, what should it bother with? Or the React code and nuances that you should focus on is like, okay, what data should be represented in what way, and essentially, and what mutations are allowed. And then obviously the, like permissions and all these things are kind of handled by the data layer itself or the business logic.

But yeah. Are there any other aspects of the work that you're doing on OmniGraph or any of the related projects or just this overall space of state management for agentic and multi agent systems that we didn't discuss yet that you'd like to cover before we close out the show? No, I think we pretty much discussed everything. Yeah, that was super exciting conversation.

Alright, well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing and try out OmniGraph, we'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data or AI systems today. Well, mean, mean, just like the obvious ones, for example, like the latency of the agents or the inference.

And then also like the constraints you're hitting when you're actually running so many agents on your computer and the workload and things like that. But I guess they're not as interesting because they are kind of commonplace. But yeah, I'm curious, like, what was your what was your perspective on that one? Or what's your answer on this?

I think really the biggest issue right now is just having any coherent way to even understand what is out there and how it relates and being able to make informed decisions when you're making a technology choice because there's so much hype and there's so much overlap between the different pieces. So being able to do that selection of understanding what are the pieces that this actually solves? What are the pieces that it only sort of solves? And how do I need to think about composing solutions together to get to a complete platform capability?

I see. Okay. Maybe because there's like so many, so much technology out there, and it's difficult to make this sort of, like, right trade offs and knowing actually what the trade offs are beforehand and things like that. Exactly.

Yeah. Yeah. Yeah, I think another one is also as sort of, you know, codegen becomes so, you know, widespread, essentially what happens to kind of the open source and sort the flooding, essentially of solutions, and they also difficult to assess was is that is actually serious effort behind it? Or is that going to be abundant in, you know, in a few weeks?

Or, you know, was that kind of vibe coded, was the actual design decision being made, you know, or like RFCs and things like that? I think this is something that maybe was less, was definitely less of a problem in the past, because you know, if code was written, you knew, okay, humans had to think about it, you know, otherwise it wouldn't compile or it wouldn't have been written, or a test wouldn't pass. But now it's actually difficult to evaluate. Of course, you can tell from some of the looking at some of the commits and also the way like maybe it's written, how much effort went into that and things like that.

But yeah, I think this is gonna be like a major problem. I think now also in terms of the reviewer load, basically, when you get bombarded with pull requests, it's actually much more expensive to review it than actually just for them to generate the code in the first place. Absolutely. Yeah.

Well, thank you very much for taking the time today to join me and share the work that you're doing on OmniGraph and thinking through some of these challenges of being able to own and manage the state space across multi agent systems. It's definitely a very interesting and challenging problem domain. So I appreciate the work that you're putting into Omnigraph, and I've definitely have been taking advantage of it in my day job. So thank you again for your time on that, and I hope you enjoy the rest of your day.

Yeah. Thanks so much. It was really fun. Thank you for listening, and don't forget to check out our other shows.

Podcast.net covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.

And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.