What's Worth Knowing In AI Right Now? (with Henry Garner)

Speaker A: What a time to be a programmer, huh?

Speaker B: Huh?

Speaker A: At the start of 2025, AI just seemed to me like the latest entry in a long line of Silicon Valley hype. Cycles seemed to be like where the blockchain fallout was going to head next. By the end of 2025, it kind of seemed undeniable that somehow this was going to change the way we all work. And fast, really fast. I mean, like the World Wide Web that changed the way everyone works, but it took a couple of decades to diffuse in society. AI speedrunning it, uh, I think is going to change things massively within a year or two. So how do we keep up? Specifically us techie types? Because personally, I think we will all still have jobs in 12 months, but only if we adapt, because they won't be exactly the same jobs. I'm sure of that. How do we keep up? That's a question that's been on my mind a lot lately. And it was particularly on my mind when an old colleague of mine announced that he published an AI tech radar 68 technologies in a PDF, all from the AI landscape. And it's an analysis of what's hot and what's not, what's worth using, what's worth experimenting with, what should probably be dropped now, specifically for all of those of us who are trying to get stuff built amid this time of change. So this week, Henry Garner joins me to Discuss not all 68 technologies. It's not that big an episode, but some big swathes of it. We managed to go through things like the impact on jobs and the impact on people paying for those jobs, the people paying our salaries. The current debate at the moment, which seems to be MCP versus skills and is MCP dead or not Fine tuning, is that still worthwhile? Frontier models, can you compete? Neuro Symbolic AI? What is it and why does it possibly mean the comeback of Prologue and how that all leads into a new language that Henry's co authored called Allium. Now, I have to confess something to you and to you, Henry. If you're listening to this intro, I didn't get Allium at first. At first it sounded a little bit like a way to generate more documentation that no one's actually going to read. But I was wrong. The penny drops during this conversation and I get it now and I'm going to talk a bit more about it at the end because I think he's really onto something. But that's towards the end of the conversation. We best get there, we best get on the map and fire up the AI Radar. So I'm your host, Chris Jenkins. This is developer voices, and today's voice is Henry Gar. Joining me today is an old friend of mine, Henry Garner. Henry, how are you?

Speaker B: I'm, uh, very well, thank you, Chris. Thank you for having me.

Speaker A: Oh, thank you for coming. You, um, possibly I've got to get you on the most difficult topic to pin down at the moment, but we must try.

Speaker B: Yes. You've invited me on to talk about this, uh, sort of niche interest that you've got, so thank you very much.

Speaker A: This niche field that no one else is talking about. AI Mm m. I think, I think I'm gonna rush this episode to production because it's moving so fast. Right?

Speaker B: Yeah. Uh, I mean it's dizzying. Uh, the pace of change, uh, even when uh, you know, as I am, I'm basically paid to try and stay on top of these things is dizzying and overwhelming. The scope as well is absolutely massive. Of course, you know, uh, for us in software engineering, it's uh, changing the whole way we work, but of course it's reaching out into basically every corner of our lives as well.

Speaker A: Yeah, yeah. The thing that stunned me is my wife, who is not a technical person, is using AI every day for her job. It's like, okay, this is a bit like the Internet in that it's changing the lives of non technical people too. But it's going so much faster than the Internet dispersed through society. Right, yes.

Speaker B: You know, faster. Um, you know, I sit in the park, people walking past, I can hear talking about Claude and I know exactly what they're talking about. Clout. Um, my wife as well is using it for, uh, all sorts, personal correspondence, but also creative pursuits. Uh, uh, it's a fantastically versatile tool, uh, no matter what you do. Really.

Speaker A: Yeah. We are going to have to talk about the tech of the tool and specifically advanced techniques. But before we get to that, I'm going to have to ask you because one of the big questions at the moment is how is it going to affect our jobs? And you work for, um, our para. Paraphrase the business. Business model here. You work for a company that is basically coders for hire, right?

Speaker B: Essentially, yes. Very good coders with a lot of domain expertise.

Speaker A: Excellent coders sometimes including terrific coders who later go on to host a podcast. Um, lovely agency. I've worked with you in the past, but you must be seeing the blunt edge of this, right? It is if companies are adopting AI and if jobs are at risk. I would have thought an agency coders for Hire is the first place that sees the pain of the job market. Is that your experience?

Speaker B: Yeah, I mean, what an exciting time to be doing what we're doing. Uh, and you know, I feel the responsibility keenly as the, as the CTO of juxt. Um, we see it from a few different perspectives. You know, we've got about 150 engineers skewing towards senior engineers who've been doing the job for a while, um, and they have built up considerable expertise, um, in, if you like, the old way of doing things, you know, hands on keyboard, uh,

Speaker A: um.

Speaker B: And so to see close up what effect this new wave of AI tools, particularly the ones which have landed since late 2025, ah, is having, uh, is fascinating. Um, and this is quite predictable journey that uh, our engineers are going on from, you know, skepticism and a little bit of resentment. You know, we've all seen hype cycles play out and plenty, uh, of us for some length of time have been saying, you know, basically this will all blow over or, you know, it's spicy, uh, autocomplete.

Speaker A: Um, yeah, I was saying exactly that a year ago. Yeah, and the scales fell from the eyes.

Speaker B: You know, it's, uh, yeah, it's a, it's a well trodden path through to, oh my goodness, when you actually get your hands dirty and you start trying out something like Claude code, for example, is where I, uh, am lately advising people begin. Um, suddenly you realize the power, um, and uh, I've heard that some people, uh, have a bit of a wobble at that point and think, oh my goodness, these skills that I've spent decades building up, uh, what are they worth now when I can just ask a machine to do the job for me? Um, that may take a week or two to, uh, play out.

Speaker A: I think it was exactly two for me there.

Speaker B: You go through to the other side where you realize that it's not taking away your engineering judgment, it's not replacing the things about the role that you enjoyed. It might in fact be allowing you to delegate some of the boring drudge work that you used to tolerate because what you enjoyed doing was solving big gnarly problems or thinking through nice, uh, solutions. And those solutions have changed shape. Now we might not be specifying the precise implementation if we're delegating a lot of that to agents, but you've still got to design, uh, what your software needs to do, uh, and you get the payoff of seeing that um, it's solving real problems and um, uh, enabling people to do things that they couldn't do before. It's just, it's massively accelerated that path to done.

Speaker A: Yeah, yeah, I totally agree with that. And I've had great fun. Like all those tiny side projects that I'd never get around to in the past but would like to build are uh, things I now tick off of an evening. I've built so many new small projects, uh, just like scratching personal itches. That is a joy. If you got into this to build things, it's a joy. What I wonder is, let me ask you two things about this. So one, have you found like massive resistance from any corners? I would have thought there are some people who jump into this with enthusiasm and some people in your team who are holding out almost to the point of being left behind.

Speaker B: Uh, well, I mean there's definitely a spectrum, there's a kind of bell curve. Um, and part of my uh, objective over the last year or so has been to ensure that I shift uh, the mean in our organization towards ah, adoption. Because the writing's on the wall as far as I'm concerned. This is the new way that software is going to be built. And the era of humans writing code for most projects is over. So you've got to get on board. Of course there are contrarians, um, who are going to point out all the things that AI can't do. Um, and you know that the story's been the same since the, the beginning really where you know, you point out uh, some ceiling of AI capabilities and the next, the next model, you uh, know that's the floor of their capabilities and so on. So you know, uh, those people are a vastly diminishing, um, group of people. Um, but there are some areas where they are right, um, and it's to do with clear specification of intent. And this is really one of the skills that as engineers it's one of the core uh, capabilities that we really always had to have. Understanding what you're trying to achieve and making sure you're communicating it clearly. Um, I think it's even more important now with uh, agents um, doing a lot of the implementation. Um, but you know, contrarians might point out that uh, you know, contemporary models, very smart as they are, are still kind of constrained by the fact that they are um, autoregressive, you know, one token at a time, um, kind of uh, engines who aren't doing real reasoning. Um, and it's, you know, they are right to a point. And it's worth bearing that in mind because it does inform why they do some really wacky things sometimes. And it's important not to be completely blindsided by that kind of behavior where you've got a machine that is, for the most part, uh, highly capable and writing, to be honest, better code than, uh, I ever did. Um, but, uh, there are going to be occasions where something really strange happens. And so those contrarian voices have something. I just don't think that is, um. It's not that interesting to dwell on that right now as the primary, um, takeaway. Uh, but from a kind of security and reliability point of view, I think we've got to remember, uh, we can't be, uh, naive and too breathlessly enthusiastic.

Speaker A: Uh,

Speaker B: uh, it's important that we remain engaged in the process and able to validate that what we're getting out of these agents is indeed what we want. And if we're taking that code through to production, uh, you know, we remain accountable for what happens.

Speaker A: Yeah, Yeah. I think this is the difference between Vibe coding, where you almost wantonly take your hands off the wheel m and what we, as professional programmers should be doing, which is more. I tend to use the phrase, delegated coding, where we're still expecting to check the final architecture and output. Maybe not. I mean, what do you think? Maybe not reviewing individual lines of code anymore, but certainly reviewing processes and architecture.

Speaker B: Yes. I mean, if we're quite honest with ourselves, were we always reviewing every single line of code that, uh, uh, went through our PR review cycle? Maybe not. You know, what developed was, uh, a sense about when you needed to lean in and pay close attention because code was maybe touching something very sensitive. Maybe there was some security angle, or maybe we do a lot of work for banking and capital markets, uh, clients. So there's a regulatory, uh, angle. You want to pay close attention in those kinds of, uh, situations, but you develop an intuition for when the blast radius of a potential issue is small. Maybe the impact's going to be minor. You might not be, you know, stepping through line by line. Um, you mentioned Vibe coding, and it's not a term I particularly like because it implies sort of recklessness, uh, which I think is unwarranted. I think it's possible still to introduce a larger distance, this is the metaphor that I like to use between the code, um, and yourself. Uh, and as long as you are exercising some, uh, good judgment about when it's okay to have large distance between the code, uh, and your original, uh, uh, prompt, uh, you are not Vibe coding. You're simply making good use of your finite attention. Um, and as long as when something Is uh, critical. You've got small code distance, then I think you know you are remaining responsible.

Speaker A: Yeah, yeah, there's an analogue there I think for unit tests. I've never been a person that aims for 100% coverage. I aim for 100% coverage on the stuff that really matters, the critical stuff. And maybe it's something like that. The distance between my tests and my code sort of matches the distance between me and the code written for me.

Speaker B: Yes, I think that's a great uh, analogy and um, you know no check is ever perfect. Our uh, code contains bugs. We might have kind of static analysis that will fix some things but not all. Then you've got unit tests and you've got integration tests which will test the boundaries that the integration tests missed by stacking up these layers, um, you, there's a sort of Swiss uh, cheese um, metaphor where what you want to ensure is that your holes don't line up and that a bug can't make its way all the way through to the end to production.

Speaker A: Yeah, yeah.

Speaker B: And the more kind of imperfect probabilistic gates that you have, the less likely that is to happen.

Speaker A: Yeah, I think that's. I like that image. I've not heard that one before. I, Let me, let me ask you this then. So I have, I mean I'm not a uh, static dynamic typing zealot but I do tend to lean towards statically typed languages all things being equal, which they never are but it's a preference M and I think that leaning has become more pronounced in the age of these machines that love automatic verification, that love the instant feedback. Are you feeling any sort of lean towards certain languages for the age of AI coding? Do you think there are certain ones that fit really well?

Speaker B: It's a really interesting kind of uh, area uh at the moment and obviously it's true to say that the Python typescript are becoming the languages of uh, AI development. They're the kind of default, if you're, if you're um, running your LLMs, uh somewhat on default mode those are the languages that you're going to tend to get and so we're m just going

Speaker A: to see if you don't care, um, the LLMs will tend to recommend those ones, right?

Speaker B: Yes, that's right.

Speaker A: Self reinforcing.

Speaker B: It's not the only consideration of course and um, uh, both you and I have a background in enclosure for example and uh, with a Clojure mcp with something that links an LLM up to a repl where it can evaluate uh, the code that it's generating in real time. Um, Clojure is a very terse language, uh, and therefore it's very token efficient. And so there's an argument to say that just because Python and Typescript, the default languages of LLM development, necessarily means that they've won and those are the only languages that uh, that we should be using.

Speaker A: Yeah, yeah, I, um, I definitely think there was a very brief phase where LLM has performed significantly better on the popular languages. I don't think it lasted that long. I mean I've been using a bit of Gleam recently, which is a very obscure language in the grand scheme of things. But I wrote a skill for it, I got it to download the tutorials and create some notes from that and it writes Gleam as well as it writes Typescript as far as I can tell.

Speaker B: Uh, well, one of the things uh, I'd like to talk about today is the behavioral specification language that uh, we've built in juxt and that's a brand new language that LLMs have never seen before that's described entirely by a uh, skill. Um, it's a skill that describes the entire, the rules of the syntax of this language and it works really well. It benefits from having a uh, a command line checker, uh, which can symbolically check that the syntax is well formed because often uh, it's not perfect but it's certainly good enough. Uh, so it's phenomenal. The current wave of uh, LLMs, particularly Opus 4.6, uh, really strong. And uh. So yeah, I think any argument about uh, what's well represented in the training data, um, those kind of arguments haven't aged so well.

Speaker A: Yeah, yeah. It's funny how you can have an argument in the AI world that probably has a two month shelf life which is. And we are going to get into your specification language, we've got so much to get through. Speaking of shelf lives though, you as part of juxt, have released a kind of AI radar. What's hot and what's not in AI, which gives us a whole bevy of techniques and tools to talk about. But the first thing we have to say is like to release a radar, you have to put a uh, footprint in the sand and say these things are important right now, those things aren't. And almost as soon as you publish that PDF it's aging very fast. How do you give any advice that's going to last long enough, uh, for the effort of creating a PDF and publishing it?

Speaker B: Well, it's a quarterly radar for a start. I think when we first published it, uh, there were some voices saying should we make this annual? Uh, I mean. And yeah, the half life of the advice in it is probably six months, uh, or so. Uh, you know, very hard to say. But uh, yeah, uh, it's an ambitious task. Uh and thankfully we've got AI to assist with research and distilling our thoughts and the things that we're observing on client site. We're a relatively small team producing it. Uh, and ah, in a pre AI world it probably wouldn't have been feasible to do so, um, but it really does exist. It's uh, on our website, but also.

Speaker A: Oh look, an actual printed copy of it.

Speaker B: And a real printed copy. Yeah, actual trick. The latest version is 60, 60 pages. I think we've got almost 70 blips. Last quarter we added I think about 15 or so. Um, as I say, the pace of change is dizzying.

Speaker A: Yeah. And keeping up with what's worth paying attention to and what's still going to be worth paying attention to three months from now is really hard. Which is why I was drawn to this. It's like, well at least someone's trying to pin a few things down and expose me.

Speaker B: You know, enter it in the spirit of um, of sharing what we currently think to be true. But accepting that uh, as the industry evolves, um, we will inevitably, you know, new techniques will come in and displace the old. Uh, but we'll simply report those things as we observe them and I hope there's some use in that.

Speaker A: Okay, let me pin you down on one specific one which already I think is aging pretty quickly. I heard something saying yesterday, something saying MCP is dead and skills of the new hotness essentially. And I think, I don't think that's what you've got on your radar as it was published a few weeks ago,

Speaker B: um, end of January.

Speaker A: So end of January, so two months already. What's your take on mcp? Is MCP dead are skills the New hotness? When would you use? Which one?

Speaker B: So, um, mcps, right. Just for those who might not know, you think of them as a bit like a sort of USB interface between an LLM and the outside world. They're a kind of generic connector that uh, allows you to plug into uh, all kinds of systems, you know, Confluence, uh, and JIRA and all that kind of stuff. But also um, the web databases, uh, uh, you know, you name it. If uh, it's got an ah, API, you could write an MCP to interact with that API. A skill which is a more Recent invention. I think it's six ah, months or so ago that skills became a thing. Um, it's really just a markdown file or a set of markdown files describing an approach that you want the LLM to take. It's condensing some kind of knowledge or process or point of view. Um, and it's like I suppose, an extended prompt that you can just pull into your context on demand. Uh, and it's true, I mean you could, you could have a skill that communicates how to use a much more generic interface like uh, the command line, uh, or ah, HTTP to use uh, an API rather than having a bespoke MCP specifically for that API. So there's definitely a sense in which skills can displace some of what uh, MCPs were used for. Um, but there is one thing that uh, the MCP server provides that skills don't, which is that they're an external process. Often they're written in something like Node, so they can hold some persistent state. Uh, if you're talking to a database, they can hold that database connection. Um, and they can be uh, you know, rather than if you're using a skill where your uh, context window has to contain the skill itself. Then of course your multiple turns of what you say, what the LLM uh, says back to you, this all fills up context. MCPs can use that context more effectively. You don't have to pollute your global uh, context with kind of traffic going on, um, with the mcp. So there's a judgment call as with all these things, where it might be that you benefit from a generic

Speaker A: uh,

Speaker B: mcp, uh, and a bunch of specific skills, um, or not, uh, your mileage may vary.

Speaker A: We're still figuring out what the sweet spots are possibly.

Speaker B: Exactly. I mean as with all these things, at what level of abstraction uh, is the, is the right one, you know, are we going to have a few very generic kind of McPs rather than as. I think, you know, people got very enthusiastic uh, a year or so ago when mcps came in and suddenly there were tens of thousands of them. I think we've started, you know, initial enthusiasm has ebbed slightly and we're realizing maybe, you know, one MCP for every single API in the world is probably uh, not necessary.

Speaker A: Yeah, yeah, there is one I discovered uh, last week which I like and gave me a thought about this, which is um, the JIRA MCP server firstly. Great. Because I, sorry if anyone loves jira, but I was glad to realize I wouldn't have to touch JIRA in order to use it. But also I realized, like mcp, I will never have to be responsible for maintaining a JIRA skill because the burden of updates is on the MCP provider. And that seemed quite useful.

Speaker B: That's true, isn't it?

Speaker A: Yeah, you don't need to keep your skill and command line tool in sync.

Speaker B: There's an inverse to that. And I'm sure I've seen you, Chris, say things like, you know, I had this idea, but I'm not going to turn it into a project because now you've got the idea and you can make it happen for yourself.

Speaker A: Uh, yeah. I wrote an MCP server for Kafka and It took me 20 minutes. And I realized, well, what's the point of maintaining something that takes 20 minutes to reproduce?

Speaker B: And if it takes five minutes to update, maybe that ongoing maintenance burden is not so much of a concern.

Speaker A: Do you find that with client work that they're building in house more because it's easier?

Speaker B: Well that's definitely one of the things that clients are thinking about right now is uh, yeah, if uh, we're seeing that uh, a single engineer or a group of free engineers can make a whole uh, uh, SAS tool in a week, um, why do we need to pay vendors uh, several months to achieve the same outcome? Of course, uh, there's a lot of noise and turbulence in the industry right now as people are trying to figure out how this is going to work. Uh, it's one of the reasons that um, at juxt we are really keen to be embracing these tools and using them to their absolute best effect. We're operating as much as we can on the frontier so that um, our pitch can be, you know, it's true. AI is a massive accelerator, uh, and uh, that works in both directions. We know how to use them to achieve the goals that we need and to exercise that good judgment about making sure that we uh, are not vibe coding into production. We're building high quality code, but very fast, using everything that we know about how to constrain the behavior of LLMs productively. Your internal teams might be exploring these things for the first time. Um, and uh, there's a lot to learn. Um, and uh, an accelerator doesn't steer, uh, you know, it could accelerate you into a wall. And so I think what we're, what we will start to see over the next six, uh, months to a year or so is um, you know, a bunch of organizations realizing there's actually quite a high, um, there's a steep learning curve associated to using these Things well and uh, so I'm betting that there will continue to be a place for consultancies like Juxt.

Speaker A: Right. You give me the impression because there's a lot of headlines about uh, companies adopting AI and um, the speed and enthusiasm coming from management. You give me the impression that the companies you're talking to are more in an assess mind space than diving straight in. Is that fair? Is that because you tend to work with banks or insurance type companies?

Speaker B: It's interesting. So banks and uh, capital markets organizations, those in financial services are obviously constrained uh, a great deal by compliance and uh, so they are risk averse. They are also massive development shops with tens of thousands of engineers. And so the risks are uh, uh, clear and present but the rewards are massive too. And so we're actually seeing quite substantial uptake. Um and the reason we're seeing it of course is that we're uh, involved in these transformations and we're building the AI platforms that that uh, tier AH1 banks AH are using to exercise safe interactions with LLMs. So for example checking that they're not inadvertently leaking PII, um, defending as best we can against uh, things like uh, prompt injection, uh so I think that although uh, I think the assessing comes in figuring out how internal teams can start to leverage AI in a safe way and how to navigate the costs. If you've got tens of thousands of engineers, you don't suddenly want to give, I mean well for lots of reasons you don't suddenly want to give them all access to the latest and greatest models. So figuring out that kind of thing is I think what's being assessed right now. But there is definitely a lot of ambition for realizing the benefits of AI and yet luckily we're in a position where we're helping to uh, build those platforms that make that happen. Yeah.

Speaker A: Okay then I wanted to get onto platforms so let's go there because um, like there are things like Claude Codex, Cursor, uh, Cortex, uh, is one that the company I'm working for is currently released. Right. There are these individual programmer accelerator tools and then there are these larger, more coordinating pieces. I'm thinking of LangChain, but I'm also thinking of like Gastown. Yeah, give me your read on the larger orchestration picture.

Speaker B: Okay, so I mean so some of the, the trade offs to think through are uh, the amount of runtime flexibility you're going to give to your agents. I'm thinking here like if you're, if you've got some kind of production system, you've stood up, do you want to have provided um, tramlines for uh, everything thought through exactly what the workflow needs to be. Um, and uh, something like LangChain, Langgraph will specify the full workflow very precisely. And that is time consuming but of course carries enormous benefit that you've got a lot more confidence that you know what's going to be happening uh, when you hit go, when you submit some kind of request into uh, the public API, um, something like Gastown, uh, all bets are off. The process uh, is expressed through this kind of mad uh, management hierarchy. And Steve, uh, Yegi is great fun. Uh, he's an excellent writer, his blog posts are a good read. Uh, and he's invented this um, uh, it's a slightly Heath Robinson contraption and I think he would agree. Ah and uh, all credit to him

Speaker A: say for the American market. Heath Robinson is our version of Rube Goldberg. Goldberg. It's like an elaborate, overly complex machine.

Speaker B: Yes, uh, yeah, sorry, absolutely. And uh, you know he's very open about the fact he's vibe coding this thing and he's um, he's reaching into kind of popular culture for the names of his various components. Um, and I'm not sure that Gastown is the solution to what uh, this kind of open ended um, uh orchestration framework is going to be.

Speaker A: For those that haven't dived into it, can you break down what it actually is and then what you do differently? Um, I bring the easy question.

Speaker B: Yeah, well so what it is, it's a uh, multi layer agent uh orchestration framework which is aiming to give you as the user, uh, access to a small army of workers. Uh, and your entry point is to talk to something he calls the mayor. Uh, and you talk to the mayor about what it is that you're trying to achieve. And then the mayor is going to delegate to um, a whole bunch of sub agents who in turn are going to delegate to sub agents themselves. And because um, agents are kind of trained to be at least currently helpful assistants, uh, they, they often sit there waiting for user input. They'll sort of ask a question and then stop. And so a lot of um, the effort in Gastown goes into prodding these helpful assistants and trying to turn them into uh, huskies pulling your sled along. So they're just going to be focused on a goal and keep iterating uh, to get there. So there's um, I think there's a deacon is one of the um, agents ah, which is pinging uh, dogs and polecats and uh, yeah, I get a bit lost in the details to be quite honest with you, but the objective fundamentally is just to keep the machine running. Um, and uh, it's a slightly unwieldy uh, beast because you're fighting against uh, these LLMs kind of innate preference to um, kind of do some work and then come back to you. Um, what would I do differently? Oh boy. I mean so the way that I prefer to work um, uh, I think multiple agents running in parallel is enormously powerful. Uh, I am a little bit uneasy about simply submitting short requests to a mayor and then you know, trusting the code to be written, tested, merged and delivered without any involvement from me. I'm not quite there yet personally. Um, what I prefer to do is spend some time working on what a solution should look like which oftentimes is involving me developing my own understanding and helping to reason through what my requirements actually are. I don't necessarily know ahead of time. Uh, and using AI to scaffold my own understanding through a kind of Socratic process. Coming up with something that's a uh, well defined specification not for the whole system because I don't want us to return to waterfall where we're trying to design everything upfront, um, but a specification that is a clear ah, description of my intent for some component and then we can fan out to multiple agents in parallel and there claude codes uh, agent teams, which is only about a month or six weeks old at this point I think it's still in kind of preview um, is really useful. Uh, and I find that very robust way of having these kind of long horizon pieces uh, of work where you might have quite a substantial specification. It might take hours for agents to correctly implement and test everything that you've asked for. Uh, but you can probably decompose that problem into a few parallel units of work. You've got multiple agents to do that for you. Um, that's my preferred way of working. It's much more kind of involved in the process. Uh, and then I've got a choice about do I go away and do something else for a bit, uh, or do I stay um, looking over their shoulder and perhaps I'm off doing something else on my machine but keeping an eye on what's going on. Um, at the moment. I like to do that where I can. Uh, because it allows you to see whether actually in spite of your best intentions there was a misunderstanding or misapprehension and uh, you're starting to see files being touched that you didn't anticipate and so on. And then you can jump in Course correct.

Speaker A: Yeah, yeah. I don't know if I will agree with myself six months from now, but at the moment I find I want to be fairly hands on with architecture. I'm really quite hands off with code. And do you think, do you think there's a sweet spot for how involved you get and do you think it's going to stay that way or it's got like two months before the sweet spot moves again?

Speaker B: Well, yeah. So we talked about code distance earlier. Uh, design distance I guess is another spectrum. And uh, I'm with you. You know, I prefer to have very small design distance. I want to know um, how the system uh, is constructed. Uh, and I think that's still the place for good engineering judgment is in good architecture. Good design. I think, uh, we've seen a bunch of examples recently where for example the cursor fast render experiment where they had thousands of agents working for a week on building um, a browser by reverse engineering the specification.

Speaker A: I didn't see that one.

Speaker B: Um, and uh, well it was a complete shambles. Like there was no central organizing force. You had a bunch of, as I say, thousands of agents off doing their own thing with uh, as far as I can see, almost no coordination. Uh, and the results are somewhat predictable that there were multiple independent implementations of certain key components. Um, and uh, I think you know, some small proportion of the builds actually compiled. Uh, and I think that's a sort of illustration of uh, the risks where you don't have a, ah, clear design. You know, the um, velocity doesn't really mean anything if what you're building is this sort of fractured and entangled mess. And that there's a kind of Conway's Law analogy. We've often known with human teams how uh, they tend to build software in their own image. And if you've got one team doing something and one team doing something else, that's going to end up being reflected in the architecture of whatever system uh, they collaborate on. And uh, their team boundaries will become service boundaries and API boundaries. It's one of those things about uh, how design works in uh, in groups. And MLMs are kind of subject to the same thing if you don't constrain them with some kind of overarching design. They are polysematic, fractured and entangled. Um, machines that tend to produce uh, code along those same lines.

Speaker A: Yeah, I can see that actually being much worse than the classic Conway's Law examples because what was it he said, something like if you have two teams you'll end up with a two PASS compiler. But with LLMs you could easily end up with a thousand agents, which is a thousand teams if you do it wrong and then.

Speaker B: Exactly.

Speaker A: Yeah, okay, so this makes me think of pinning these things down a bit more and there are various techniques for this. Going to get onto your big one, I promise. But there's a particularly spicy term in your radar that relates to this, which is neurosymbolic AI on Earth, is that

Speaker B: it's a fun one, isn't it? So although contemporary frontier LLMs are ah, extremely powerful and they're called reasoning models. And as we've said, uh, it's kind of pseudo reasoning but boy does it get you a long way. But the researchers analyzing these models show that uh, well for example there was the Apple paper from, I don't know, six, nine months ago or something that was getting various different kinds of LLMs to solve um, problems like the Tower of Hanoi type problem. And it showed that there was a sweet spot for reasoning models which was sort of moderate complexity, but in kind of mad counterintuitive way. Reasoning models actually performed worse than the non reasoning models on really simple problems because they kind of overthink themselves into a bad solution. Uh, and so it's no silver bullet. Um, and when people uh, like Anthropic look at what's going on inside these models, really interesting research. Um, it's the kind of research that shows that they're polysematic and that neurons represent multiple concepts. And you've got to look at clusters of uh, neurons in the LLM in order to understand uh, what the LLM is thinking and what was it about 18 months or so ago Anthropic built Golden Gate Claude, where they kind of proved that they'd identified the cluster of neurons that represented the Golden Gate Bridge and they were able to dial those up. So that basically every question you asked of Claude, it would find a way to reference the Golden Gate Bridge. So if you asked it, you know, how you should spend, uh, $20, it would advise paying the Golden Gate Bridge toll and uh, that kind of thing. Um, so you know we can interact with these um, models. Um, but they're not doing symbolic reasoning, they're doing kind of vibes based um, pattern matching. You know, they're kind of metaphor machines. They're looking at things that are like something else.

Speaker A: Um, of course, this too though that's not necessarily.

Speaker B: Well we've got kind of two modes, haven't we? We've got the kind of um, system one and system two type thinking and the analogy is that the LLMs are this kind of fast intuitive kind of way of thinking even when you're, you're forcing them to reason. Uh, and you know it's really cool that it works but there's this, there's something that researchers spot called motivated reasoning where if you, if you kind of ask Claude, uh, or indeed any reasoning model, um, a maths question but you imply that the answer is something that it's not, you're often going to get the model finding a way to make you right and it'll say oh yes, the answer is that and this is how you get to it and it will construct some kind of plausible um, chain of thought to get there but it's just not true. Um, and uh, uh this is a problem that LLMs have and it's sort of fundamental architectural limitation but you can pair them with some kind of symbolic reasoning which is doing um, proper

Speaker A: consequence,

Speaker B: uh, analysis, maybe something like Prolog, that's solving a well formed problem and providing you with exactly the steps to get there, you know that it's correct.

Speaker A: Um, and how that would work because the return of prologue to the conversation is not something I could have possibly predicted.

Speaker B: Well, isn't life funny? And it doesn't have to be prologue, of course, it could be giving access to a calculator for example. Give the LLM a tool that does execute symbolic reasoning in a way that LLMs will try but potentially fail to do and then this combination of the two, uh, can give you much more confidence that you're going to get good results and it opens up the possibility for applications in places like regulated environments where you need to show exactly the steps that were taken. But perhaps you want to have some kind of fuzziness to your inputs. Maybe plugging Prolog directly into uh, whatever upstream systems you have isn't viable. But you can use the LLMs for what they're good for there, which is translation, um, and then ensure that the steps that need to be correct are indeed correct.

Speaker A: Are you saying that we're going to, I'm trying to picture this. Are you saying that uh, we're going to write some kind of Prolog model of the system and then get the LLM to say here are my facts, are my conclusions correct?

Speaker B: Uh, well it might not be checking, it might not even be attempting to uh, to come up with its own conclusions. You know you can in creating a hybrid neuro symbolic system. Um, the idea is, and of course LLMs can write code. So this is, this is another thing. It doesn't Necessarily need to be that um, uh, you know you've written your system in prologue and again doesn't have to be prologue. I just think it's really funny that this, this language from the 70s is suddenly uh, you know, finding a potential application again still very early days. Um, but it could write Python for example. Um, and so you uh, can have your LLM writing some code that can then provide an answer with a kind of audit trail for how it got there and then take that answer and run with it uh, with each part providing what it's good at.

Speaker A: Maybe I'm, maybe I'm getting confused. Uh we saying like it will write programs to assist it in the act of writing programs or is it more like when you ask it a non programming question it might lean on a programming language to do some of the rigorous parts.

Speaker B: Exactly that. Yes.

Speaker A: Right.

Speaker B: That's right. So uh, you know it's tempting to try and use an LLM as the solution to every problem and bless them, they will try. It's rare but you start to see it these days and I guess there's a lot of work going on in the um, in the uh, at the post training to guide LLMs to be transparent about when they can't answer a problem. And I've only had it a handful of times but uh, I might get a response that says ah, actually I can't help with that, um, which is useful. It's better than the alternative where you get some kind of motivated reasoning or hallucination uh, or confabulation and uh, uh don't realize it. But better still would be if the LLM goes oh well I can't do that because I am a semantic uh next token predictor machine. But what I can do is write a little helper function that's going to provide the right answer for you. So I'll go away and do that.

Speaker A: Right. Is there a particular field that's being applied to or is this all just experimental?

Speaker B: It's still really early. You know, it's a sort of emerging architecture, um, but it's a very general principle. So although I can't think of a particular field uh, where this is um, becoming prominent anywhere where you've got this mix of requirements and you need some kind of uh, auditability, you want to bring some kind of symbolic reasoning uh into play. It's another tool in the toolbox and uh, uh, it'll be really interesting over the next year to see uh, whether this takes off uh, or whether we find ever more ingenious Ways to juice the ability of LLMs to the extent that uh, people get away without it.

Speaker A: Yeah, yeah. I have been feeling recently that even if the sophistication of LLMs themselves plateaus right now, there's still an enormous amount to be discovered in the way we use them and the way we orchestrate them with other tools. That's right. M. This is actually one reason why I invited you particularly to talk about this topic because as well as like using off the shelf. Off the Shelf LLMs as many of us are, you've done lots of training of your own models in the past and I want to get your read on. To what degree is it now worth uh training a specialized model from scratch? Taking an existing model and tweaking it like post training new weights? What, to what degree do we distill large models into small models? Is there space for all these techniques? Because it's, it seems like apart from cost of tokens you might as well just throw a large model at everything and be done with it.

Speaker B: There's a lot of truth to that. Um, you know trying to compete with the uh, the speed of development of the frontier model providers uh is almost certainly a losing game. Um, so you need to have quite a high threshold I think now before you decide to uh, do your own model. Um but as you say this has been something that uh, I've done in the past and where it is still useful I think is um, if you've got a lot of. So one of the use cases for example is classification uh and uh prior uh role. I worked for Regtech and what we needed to do was classify regulatory text. Um and there's a rich and storied history about uh text classification and we reached for some of the uh, really good classical ML techniques early on. Things uh, like support vector machines uh for classifying regulatory uh text. Um this was around the time that LLMs were coming to the fore and so we just decided to see how well an LLM might perform uh, and it outperformed the classifier uh as long as we fine tuned the LLM on our training data and provided it with the hundreds of thousands of training examples that we collected over years that gave it a kind of intuitive understanding about how legal text related to the classes that we wanted to apply. Uh, and so that fine tuning provided this sort of uh, you know intuition is probably the best word actually. It meant that we didn't have to have massively long prompts that described exactly how you should classify each piece of legal text. It had Learned by example. Um, if we had tried to put all of that into a prompt, we'd start to hit the kind of uh, lost in the middle type challenges that happen when you've got very long prompts. And LLMs tend to be much more sensitive to what's at the beginning or end of the prompt and forget uh, what's in between. Um, or we'd sort of hit context window limitations and that kind of thing. So there's still a place if you've got this kind of repository of your own information, valuable um, business knowledge, uh, that you want to give the LLM a kind of intuitive knowledge about fine tuning still has a place. It's more common though that you'd plug them into some kind of rag pipeline. And if what you're actually doing is information retrieval rather than classification or some other kind of machine learningy thing, if what you're asking for is fetch me the relevant information, you don't really have to fine tune a model. You can uh, vectorize your data, you provide a, uh, means of looking up based on a query, the potentially relevant data, then you can provide that to an LLM. And you're unlikely, as long as you're not finding uh, hundreds or thousands of responses, you're not going to exceed the context window of an LLM to be able to sift through that and come up with uh, a good response for you. So, yeah.

Speaker A: So modulo cost, you really are leaning towards just frontier models with the right techniques and orchestration around them, yes.

Speaker B: For most people there's a maintenance burden. As I say, trying to keep up with frontier models is um, is probably a losing game right now. But also you need a bunch of embedded expertise in your team in order to do things like fine tuning. Well, uh, you need to be hosting um, your own models. So that carries a cost and uh, an overhead that's uh, not insignificant.

Speaker A: Fair enough. That reminds me of something I was going to ask you. Hosting your own models as a company, that's a big ask. Right? You probably want plenty of servers with 300 gigabytes of video RAM or whatever, but there's also the hosting your own locally, which, what's your experience here? Because my experience is it's impressive how much LLM I can run on my 32 gig MacBook, but it's still kind of brain dead to what I would get if I go for the cloud. Is that your current experience and do you think that's likely to change anytime soon?

Speaker B: Well that is indeed a probably billion dollar question.

Speaker A: Uh, yeah, Many large companies fortunes are riding on um, being. And no, you'll always need the cloud.

Speaker B: Yeah, exactly. Uh, and my hunch is that uh, the opportunity of being able to run something locally on commodity hardware is too great for us not to find a workable solution, uh, in the next, uh, year, uh, or two. Um, and yet at the moment you're right, we're kind of spoiled when we can go to the cloud, uh, and get this really sort of emotionally intelligent, uh, AI, uh, kind of uh, coming back to us with questions that we didn't even think to ask. And then you go to a local version and uh, it's much more sort of pedestrian and uh, uh, slower. Um, and so that introduces kind of friction and it means that uh, most of us aren't doing that right now. Um, but uh, just imagine if uh, uh, if we find a way to make these models much more efficient and provide for, at least for certain domains, um, enough capability that you don't feel that kind of friction, uh, anymore. Uh, of course that's what we would want to do. Our data stays local. We don't have to worry about inadvertently um, leaking it. Uh, costs are much more predictable. Uh, you know, um, at the moment if you, if you really want to do it, uh, you do need a lot of RAM more than, more than 32 gigs. I mean even more than 128, uh, gigs. Uh, really, which is what I've got. Ideally you have like uh, 500 gigs would be great. You could run Kimi, um, locally and you get a pretty comparable experience to uh, running Claude, uh, Opus. But uh, yeah, that's going to be outside of most people's uh, budgets right now.

Speaker A: Yeah. And sourcing availability.

Speaker B: Indeed, yes.

Speaker A: It's hard to get hold of memory at the moment.

Speaker B: But anyway, you know, there is something I'm following with very close interest right now and I'm sure it will make its way into the next Radar is. I think it's a Finnish company called Talas which has built uh, a chip, um, which has the weights of. I think it's uh, one of Meta's models, uh, one of the Llama models, uh, encoded in it. So it's a sort of LLM as a, as a piece of hardware, uh, and uh, reportedly I think it's 17,000 tokens a second. So it's basically instant.

Speaker A: Wow.

Speaker B: And so current numbers are something like

Speaker A: what, 20 to 50ish, something.

Speaker B: Yes, yes, depending. So you know, it's not inconceivable for uh, you know, for some, um, those with enough budget you could have Claude uh, on a dongle, uh, you know, before too long. So maybe that's another solution.

Speaker A: Yeah, they're going to really struggle to keep them up to date. But again maybe that won't matter. Maybe if the LLM power plateaus and it becomes about orchestration, then you find that really what you want is enough USB ports. Plug in all your Claudes and a donkey. I've only got three and one of them is being used to attach my camera.

Speaker B: So that's no good.

Speaker A: Okay, um, let's go back to production then and stop speculating. The other topic I really wanted to talk about, which is very much at the coal face is um, LLM M observability. Like we know it goes well, we know it goes badly and I know you've put on the radar like tools for keeping an eye on them. Is this like monitoring pipelines into production or auditing whether your users, whether your developers are using it Right. Or what's the state of the art there?

Speaker B: So yes, I mean it used to be that uh, we were much more concerned about training our models and evaluating them during development and then you push them to production and they just kind of worked. And uh, if you were very diligent you might be looking for data drift and things like that over time where you'd be checking that the distributions that you were training on were still correct uh, in production. Um now of course if you've got sort of agent uh orchestration going on, you need to be paying very close attention to what your system is doing. You need to be keeping an eye on kind of tools, tool use that uh, agents are doing, how they're talking to each other, um, whether they're stalling, crashing, uh, and so yeah there's a whole emerging uh, set of tools uh, which will provide you that kind of observability. And I mean Phoenix is quite a good one. It's very easy to get started with. Um, you get to see traces, you get to um, see what's going on, uh, and provide nice metrics and graphs and so on that you can start to see if something is not going how you anticipated. And it's kind of paired with uh, workflow tools as well, like temporal uh, for example. So because uh, if you're talking to uh, the uh, SAS model providers uh, at scale, you're going to start to become really aware of some of the stability issues that they are struggling with as the world tries to use uh, their API, uh, and so getting really robust around uh, when the Service is unavailable, um, back off and retry. Uh, if your agent crashes in the middle of a long multi step workflow, you don't want to start back at the beginning, uh, you want to pick up where you left off. And so uh, temporal is in our radar as well. For that reason it provides the ability to temporal like save game, uh, in your uh, workflow.

Speaker A: How does that work? What is temporal in detail?

Speaker B: So if you think about uh, kubernetes, which is orchestrating machines, uh, temporal is orchestrating your agent workflows. And so it's providing a way of keeping durable state in a way where if your machines crash, if your agents crash, things will always be able to pick up where they left off. And some of these things are very long running. And so uh, it might be that human approval steps are part of your workflow in which case perhaps there are days where something is sitting in a particular state waiting for some new thing to happen. Temporal provides uh, uh, a framework for um, specifying that behavior in your own programming language. I think there's Python, Java, uh, Typescript, uh, and so on. And so it's a nice way, quite good ergonomics of writing these uh, robust workflows.

Speaker A: Is that as simple as, fundamentally as simple as persisting all the context windows from the various agents? Because the nice thing about an agent sounds so simple. I do and I'm trying to get, and I'm not trying to dismiss what they're doing, I'm just trying to get to the hub of the architecture. Like the nice thing about an agent is fundamentally it has state, but its state is just a big long text transcript, right? Mhm.

Speaker B: Well yeah, from the outside I think there's all kinds of mad science going on for efficiency. Um, but yes, from the outside, that's right. Uh, big long context string.

Speaker A: Okay, so managing those at scale and doing a simple task at scale is always harder than it sounds.

Speaker B: Yes, that's right. You've got these uh, essentially an arbitrary number of long running processes you want to pick up um, based on triggers in your workflow, other events and things

Speaker A: and just for my local version of that, because I often find I have um, companies have orchestration problems, we're finding developers have orchestration problems where I've got three or four instances of claude code running and I'm just waiting for the next one to ask for the prompt.

Speaker B: Yeah, we're moving into management, aren't we? Yeah, we thought we were ics, but it turns out now we're responsible for a team. Uh, and the Analogy I quite like is that we're going from being uh, kind of pilots to air traffic controllers. A completely different way of working.

Speaker A: Yeah, yeah, I can say that that's quite a good way of thinking about it. And sounds more glamorous than going from contributors uh, to engineering managers, doesn't it? Right. I imagine air traffic controllers would say it's not that glamorous. But are there good local. I mean would you use Temporal or Phoenix locally? Or are there local management things you would recommend instead?

Speaker B: That's a really interesting question. I mean, so as I say, my preferred workflow is basically an agent team and um, fanning out and then back together again. Uh, what would be the circumstances in which I've got genuinely long running process at the moment? I think probably four, six hours is probably the longest I've ever left agents running unattended. And that will have required quite a lot of setup to make sure that uh, uh, I've got some idea of what's actually supposed to be happening. Then. Um, so a lot of uh, you know, upfront specification and then uh, a RALPH loop. I suppose that's the dumb answer where if you've got some hairy audacious goal, um, for example, you've spent as I have a day with an AI iterating in a lot of detail on the specification for a full distributed system, uh, spread across 10 files. It's like 3,000 lines of specification dense um, code like specification, not human pros. Uh, then you're asking a lot of uh, even an agent team of LLMs to go away and implement that. That's going to be a lot of code. And there's this um, quite sort of. It's annoying but um, uh, a quite sort of fun, whimsical quality of LLMs of being like a sort of genie that will often find a creative way of telling you that uh, uh, they've done what you asked, but it was never quite what uh, you intended. Maybe it's uh, the letter rather than the spirit of your request.

Speaker A: The classic one is I've got the tests working by disabling all the tests.

Speaker B: Yeah, exactly. Um, but a uh, Ralph Wigham loop is uh, a very simple way of combating that. Uh, which is to you know, if your prompt is something like I've got a load of specifications sitting in the specs directory, read them and implement the code that they describe. M. You give that to an LLM and it will go off and it might run for several hours and it will come back and say yep, I've done that and on closer inspection you'll find that maybe it went really deep on a particular component and that is great that's been implemented but there's whole other areas that have just been completely ignored. Or maybe it will have taken a very broad brush approach and implemented everything a bit but missed m lots of the detail. Um, but the uh, genius insight is that you could submit exactly the same prompt again and say take a look at the specs directory, implement the system that they describe and when the new LLM wakes up with a fresh session it's brand new contributor. Uh it will take a look at the code you've already got and say oh that's not quite right. Uh this doesn't match what the user has asked for. So I'll go away and I'll start on that. That next LLM will take you a bit closer and of course you can just loop as many times as you need to. And this maybe is where the new engineering judgment comes in is knowing how to specify that prompt, how many times to iterate, how much upfront work to do. Um, it will get you eventually after some number of goes uh, perhaps towards broadly what you wanted and this has been my experience even with very sophisticated distributed systems that you do eventually get what you want.

Speaker A: So you've tried this with something meaty and it actually works.

Speaker B: It does work, yes. Yeah, absolutely. Um, I've written about it. So uh, distributed uh, strongly consistent, very high performance system uh with uh ah, Byzantine fault tolerance. Uh uh it took a weekend of elapsed time and probably about 20 hours of LLM time uh to implement but uh, it worked. And of course you know the fact that uh, you can, if you specify the intent that you have for your system but not the implementation, give CLAUDE or your LLM of choice a lot of latitude to solve the problem. But whilst being constrained by you know, your invariance the things that you that must be true like the fact it needs to be fault tolerant and uh, the kind of error conditions that uh, that we need to uh, defend against. And I had the experience of um, CLAUDE realizing uh, that uh, there were bugs that only happened under certain crash scenarios and repeatedly uh, setting things up, killing them and looking to see what happened and then maybe adding instrumentation, little bits of code that would be logging, doing things again in a way that I would have been very, it could be quite tedious drudge work to do that. Uh, you're sort of trying to reason through complex race conditions in a distributed system. Um but if you've Given the LLM enough context about the goal, and it's a goal that can be verified, uh, then my experience has been they're a fantastic tool for just, um, patiently diligently working their way through a solution. The same is true for, um, latency. I set a really ambitious latency goal of, uh, uh, our P99 latency being under 100 milliseconds at 10,000 requests a second. And again, it had access to the load test rig. It obviously had access to the code. Uh, and our first, uh, attempt, our P99 latency, was 30 seconds, I think it was 31 seconds in fact. Then over the course of some hours, we got it down to something like 25 milliseconds. Um, and part of that was Claude realizing at some point that it had noticed that the P99 was stuck on 208 milliseconds for ages. In spite of what it did to the code, nothing seems to shift that persistent tail latency. It added some instrumentation to the code separate from our load test rig that confirmed, um, that actually nearly all the requests were happening in under 100 milliseconds and realized that, um, the latency was being introduced by our Docker networking bridge and decided all on its, um, own, okay, I'm going to move the load test rig inside docker. And then suddenly, you know, the latency drops. And that was all completely hands off. I just said the overarching goal that I wanted to achieve and, uh, left it.

Speaker A: I mean, that's insane for building large systems. I have to ask before I completely react to that, how much do you now trust that that system that's been built?

Speaker B: Well, I have a lot of tests. It is better tested, uh, than, uh, most systems that I've been involved in. And I think that's the fascinating thing about AI assisted engineering is we always knew that comprehensive testing of at least the important critical parts, uh, is important, important, uh, and yet we were always constrained by what we could reasonably achieve and the time that we had available. The calculus has shifted with AI and suddenly it's feasible to implement tests for every conceivable, uh, scenario that you think you want to defend against or that's quite sort of, uh, reactive. Maybe you're writing tests about code already exist. You could spend equivalent amounts of time making sure that you absolutely understand how the system should behave in all kinds of, uh, adversarial scenarios and make sure that behavior is really well specified in a way that theoretically you could have done before, but maybe you'd have been working on a whiteboard, and that's got kind of finite space, and then somebody comes along and rubs a bit out. Now you're working with an LLM on a behavioral specification file that's durable. You can be interacting conversationally, and so you can be asking questions of like, what happens if? Kind of thing that previously might have been quite hard to reason about, but now you've got the LLM there for signposting and assistance that mean you can be much more rigorous. And uh. So in a funny sort of way, we're all, I think, uh, not unreasonably. We've been worried about vibe coding and the spaghettification of code with LLMs, but. But in diligent hands, you've now got the opportunity to do the things that we always should have been doing much more rigorously. And perhaps we're going to see certain types of software that have been built along those sorts of lines being much more resilient than they ever were before.

Speaker A: Yeah, yeah, I can see that. I genuinely can, with the extra capacity and diligence. I have to ask you. I have to ask you, the halting problem. How did you know when to stop looping?

Speaker B: Well, at some point you're going to get, uh, a few consecutive responses that say, yep, there's nothing more to do. Everything looks fine to me. Eventually, yes. I mean, it might take some very long number of loops and I wouldn't ever trust the first one. But once you've done that, uh, a couple of times probabilistically, you know, um, on the balance of likelihood, you go, okay, I think we're there.

Speaker A: Okay. Okay. Then I have to ask you, for building a system of this kind of size, you talk about pinning down the specification and the constraints. That leads me to your contribution to this field, which you've been working on, a system for pinning down constraints for the sake of LLMs. Right. And for the sake of Agent. Tell me about that. What's your novel addition to the world?

Speaker B: So, uh, it's called, uh, Allium. It's a language for behavioral specification. It's an LLM native language. What does that ah, mean? It means that there's no runtime except for the LLM. There's no compiler except for the LLM. Um, it exists as a, uh, specification of what the language is, uh, and it has an accompanying skill which teaches your LLM how to read and write files in this format. And the language guidance provides, um, a lot of information about how, uh, a behavioral specification should be elicited from a stakeholder. There's also another mode that you can run in where it looks at uh, an existing code base and distills that code into a behavioral specification. So it looks at the how of the code that actually exists and then attempts to reason about the what, what it was trying to achieve and to encode that as a behavioral spec. And the thesis behind this is that um, we're seeing a rise in spec driven development. You've got things like Spec Kit and Kiro and Claude plan mode and generally the solution is um, markdown files going to write a long human prose description of what we're trying to achieve. There's definitely value in that. It's certainly better than nothing at all. Um, but human language just naturally hides contradictions and there may be ambiguities uh, there um, in a long specification, uh, a person might miss the fact that you're a bit ambiguous around uh, the kind of authentication model that your system needs to have, or whether guests are allowed to do certain things or what the uh, um. It's not just about security of course, it's also about any aspect of the behavior of your system. Um, and so the thesis is that providing a formal way of expressing your behavioral intent uh, provides much less opportunity for that to happen because you can start to leverage the LLM's ability to spot bugs in code if you're expressing a behavioral intent. A bit like code. Um, right.

Speaker A: Give me a concrete example. What does it look like when I write some allium?

Speaker B: So it's not designed to be written by humans, it can be read uh, by them because uh, it's designed to look a bit pythoney typescripty. So in fact we are trying to leverage that point around um, well represented in the training data, um, just to give the LLM a leg up, uh, given that we're teaching it about a brand new language, it's literally never seen before, um, but it arises through conversation. And so you would start by saying something like I want to create an Allium specification for my system. I want it to do X, Y and Z. And the skill around, uh, the specification language will be the one that teaches the LLM to say hmm, interesting. When you said customer, what did you mean by that? You know, is the customers map to accounts, uh, what can they do, you know, uh, and through this kind of conversational uh, iterative process, a lot like you know, a product owner uh, might have with a senior engineer.

Speaker A: Yeah.

Speaker B: Or so on, you get this rich and quite um, rigorous specification of what the system needs to be completely independent of implementation. It's got nothing to say about the database that you're using or the web server that you're using. Uh, it's pure, uh, behavior. Um, and so this is how we can kind of avoid getting tangled up in implementation concerns. And it's really powerful because LLMs kind of snap to the code if all you've got is the code to look at. That is both the description of how the system should work and what it's trying to do condensed into the same kind of substrate. And yet there's no reason those two things have to be expressed in exactly the same way, because the how has all kinds of other things to grapple with. You know, implementation, uh, constraints, uh, libraries. And, um, maybe there are kind of expedient workarounds or, you know, bugs lurking in the code. Having a completely separate way of specifying what the system should do isn't duplication, to my mind. It's providing the kind of resilience that you get from also having tests which are, uh, expressing what the code should do in a different way. The trouble is that because LLMs tend to update both the code and the tests at the same time, they're both kind of evolving simultaneously. You've got no fixed point around which to check that those changes are correct. Uh, whereas if you've got a behavioral specification as well, then if you're only ever changing two of those at once, you've got that kind of fixed point. It's a bit like three legs of a stool.

Speaker A: Right? So are you saying the workflow here is I have a chat with my agent to produce Allium files as an asset which are then fixed, and you would then go on to have it implement those, but you would, you'd expect to get a change which changes a lot of source code files, but doesn't change the spec files exactly.

Speaker B: You will tend to be changing one or the other, um, at any given time, um, and you can do as much upfront specification as you want to do. Part of the reason for wanting to build this language, uh, is to work, to have it built entirely for that way of working with LLMs, where you want to start by speccing out a, uh, prototype, um, but you want to put that prototype under some kind of behavioral specification because then that allows you more easily to sort of grow it, uh, in a kind of rigorous way, uh, where you're not going to end up in the situation where you try and change one bit of the code and another bit breaks and then you try and fix that bit and the first bit goes Back to how it was because it can be bit like playing whack a mole sometimes. So it's designed for this kind of iterative way of working. You start with a conversation, as I say, unless you've got code that you want to distill into a specification already and you do as much as you need to. And the specification has provisions for open questions, uh, and deferred specifications and other bits which are explicitly acknowledged to be kind of open at that point in time. And so when the LLM m, a different LLM probably is coming along to actually implement that behavior, it can see, oh, this bit is the bit that has been well specified. This is the bit I will um, write good code for. And that bit is currently undefined. Uh, so I won't do that. Uh, and so yeah, when you hear spec driven development, often people think oh, we're back to big design up front and waterfall thinking. That's exactly not the approach that I uh, advocate for. This is definitely an incremental and iterative process. It's just providing you with that fixed point that means that you've always got uh, some stability, uh, in um, your iterations with your LLM that allow you to uh, keep walking forwards rather than get caught in some kind of doom loop, which is probably an experience that lots of of us have had with LLMs where we reach a wall and there's a massive drop off in uh, our velocity. Um, the thesis of Allium is that we can achieve velocity through clarity of intent.

Speaker A: And you're also saying that there's a gap in the design space between the flexibility of human language and the detail of programming language then?

Speaker B: That's a nice way of putting it. And is it between human language and a programming language? I say what it's between is human language and perhaps the most rigorous specification language, TLA for example, which is going to provide you with formal guarantees of correctness. Um, Allium sits in between those two kinds of uh, things. You know, we can't offer formal guarantees. We're living inside the LLM. We're therefore subject to that kind of semantic, um, kind uh, of metaphorical similarity rather than symbolic correctness. We do have command line tools and other things that will check the validity of syntax. But even so, you know, we are uh, we're in between those two extremes. We're more m rigorous than prose. But uh, but yeah, we can't offer the formal guarantees of tla. However, it is a programming language in the sense that it's got very well defined semantics. Uh, you are describing exactly what you want to have happen. And so in that respect, uh, it's not inconceivable that in the future there could be an actual runtime for Allium. And one of the directions that we're moving towards is to uh, to make sure that we capture enough information to be able to implement property based testing for example. And so we're sort of moving in that direction. And in spite of the fact that we are uh, LLM native, uh, it's important to me that it is a very well defined language, uh, with clear rules about its use, even though we can't guarantee the LLMs will always follow them to the, to the letter.

Speaker A: Yeah, ah, but you can guarantee that about human programmers either. So what kind of things can I specify concretely? Can I say like this project should use OAuth and support Google, this project should save all data persistently to postgres, a durable database. Can I say your thing of the P99 for response time should be less than 100 milliseconds.

Speaker B: Uh, you can specify all of those things because uh, there's an implementation guidance section for uh, every rule in the specification provides the ability to use human language as well. And so there's nothing that you can't put into your specification. You can add non functionals, you can say I'd like to use Google. So you can say I'd like to use postgres. That's not the core objective of a behavioral specification though. What we instead want to talk about are uh, the domain entities that we're operating on, what their attributes are, uh, um, for example, so we're not going to implement the schema uh, that ends up in the database. But we are going to say we've got customers and line items and we've got quotes and so on. These are the things that our system knows about. You're going to have rules that describe how transitions through your system work. There are going to be preconditions, post conditions, triggers, um, surfaces is uh, something that gets specified as well. So those are intentionally not talking about a web page, uh, or a CLI interface or an API. It's simply a surface. It's a way that the outside world can reach your uh, domain, um, and so surfaces expose the things that the outside world uh, can read and uh, the things that your code requires those people, those actors who are interfacing with your code to provide. But it's intentionally expressed in this quite abstract way because we don't know is this an API or is it a Rich uh, web front end. We don't really want to because the same behavioral specification could be used to drive an implementation of both of those things. And so when you're um, asking your LLM to implement you're going to have to. Well ideally you're providing some kind of uh, additional context which is going to talk about your technology choices uh, and those kind of non functionals and other uh, things like that. Uh, otherwise you're going to get some kind of default uh, choice. Uh, as we said it's probably going to end up, it's going to be

Speaker A: tightened up to the REST API. Right?

Speaker B: Yeah. Right.

Speaker A: This sounds like you're describing a language for business logic then.

Speaker B: Uh, well so in a sense, yes. So kind of state machine. Yeah, transitions is uh, is a key part of it for sure. Um, I mean that's a lot of what systems need to do isn't it? Manage interactions with data and uh, ensure that uh, they're correct.

Speaker A: Absolutely, yeah. I'm just wondering, I mean after we finish this discussion I'm going to go and run it on one or two of my existing projects. I'm wondering what will be distilled out of it. Like will it boil out some of my hard external requirements or is it going to boil out what my core business logic is?

Speaker B: Well it's fascinating isn't it? There's inherently a judgment to be made about level of abstraction because if the specification ends up being too close to the code it's not really adding anything if it's much too high level. Of course every website's the same. I go and I make a request and I get some data that's not useful. So part of the skill is around um, and I mean this in the sense of uh, markdown skill. The way that we guide the LLM is to make uh, a sound judgment based on you kind of um, senior engineer type expertise about what the right level of abstraction is. Of course you can always guide that in the prompt that you provide along with your distill request. But our experience is the default is really good and we've used it to find bugs in existing code. Um, because in producing a behavioral specification that's attempting to communicate the, the what of your system as well as the how in a kind of compressed way. Uh, because a spec is Normally somewhere between 5 and 50% of the size of the code depending on how uh, complicated your semantics are or how verbose your programming language is. Uh, so it's always introducing some kind of compression and then by comparing the specification against your code, you can start to go, well, that's funny, that code's quite complicated for the specification that has described it. I wonder why that is. Perhaps the code's doing more than it should. Perhaps there's some kind of tangled intent going on there. And so although you can never say for certain that the code is wrong, uh, or the spec is wrong without getting your hands dirty and taking a really close look, it provides this kind of, um, topology of your code that allows you to go and look at the areas where you might find something that's wrong. And as I say, we've used this to discover bugs in code that people were using very happily they weren't aware of. Um, and so it really does add value and is part of this story about how I think LLMs can introduce more rigor to software engineering than, uh, we've had to date.

Speaker A: Yeah. Yeah. I wonder if you're going to. So I've done a thing with a few projects where I've, uh, run an agent over the code base and said, well, draw me some mermaid diagrams, right, so that I can try and visualize what's going on. It does quite a good. Excuse me, it does quite a good job. And sometimes I've looked at these diagrams and thought, well, that's not, not right. That doesn't seem like the architecture I'd want. And trying to prompt for it to re architect things the way the diagram should have come out. Are you finding that with Allium? Are you going to be doing that where you distill out spec files for a system and look at them and expect a human to read through them and find actually contradictions, change the spec and have the LLM reproject it back into the code base.

Speaker B: I mean the shape of that way of working is correct. But again, the specification files themselves are. You, uh, can look at them, um, to check individual rules, but I don't think anyone will look at a whole file and attempt to know whether that file describes the whole behavior that they want from their system. That's a lot to try and extract out of, uh, you know, potentially a long file. Um, but for, you know, taking a look and checking that, oh yes, the right attributes are being exposed for my domain entities here or. Well, yes, that rule has, uh, the right preconditions and it triggers the right thing. Those kinds of things for sure, you could have a domain expert to, uh, look at. But I think for the kind of workflow that you're talking about there, the conversational one that we're all getting used to with our, with our prompts, uh, is where people are right now. And so kind of Allium is designed to work with that. Um, what it tends to create is um, this dynamic where the LLM is asking you really sharp questions about what did you mean when you said that? Or I've noticed that this part of the specification, that part of the specification are ah, you know, slightly inconsistent. What do you want to do about it? Um, so kind of there's a sort of sapien whorf kind of quality where you know that, that theory that the language that you use describes your reality. I think by teaching the LLMs about this structured language of behavioral uh, specification just puts it in that frame of mind where those are the kinds of things it's asking about and it's really powerful. Um, and I am constantly surprised by uh, the quality of the conversation that we have and how quickly you can get to a really good description of the problem space. Having been told about the potential ambiguities and conflicts and edge cases that you might always just have shrugged off before because it was a bit, you weren't really aware of them. Maybe your sense was they might, they might pop up. But, but uh, there are more important things to worry about right now. You know, the Allium, um, skill teaching an LLM to patiently, in a kind of Socratic way, work through these things with you makes it almost pleasant. Um, because again this uh, the speed uh, that you can realize means uh, that it can be quite intense to be uh, being asked in quick succession lots of very detailed questions about how your system should behave. But an hour or two later and suddenly those problems are solved. And uh, maybe an hour or two after that you can see the system actually running. It's very rewarding.

Speaker A: Right, Right. So I've got the wrong end of the stick with this language. I thought it was a language for us to use to drive the LLM. But really you're saying it's a language to help the LLM pin down its own thoughts ready for when it talks to us.

Speaker B: I think that's a great way of phrasing it. Yes.

Speaker A: Right. Okay, so I've got to go and try this and see what the results. That's the only way I'm going to um. How do I get started? Give me that beginner's guide.

Speaker B: Uh, so it's open source. Uh, it's on GitHub. Uh, you'll find it at uh, Juxt Allium. Um, and it's very easy to install. It is just a set of markdown files. Uh, so, uh, there's a CLAUDE Marketplace, uh, interface. So even within claude, you can do plugins and add the juxt marketplace and grab it that way. Or if you're using uh, anything else, there's a sort of npx, uh, skills installation as well. It's all described on the readme. Once you've got it, uh, available in your LLM, uh, of choice, uh, just type Allium, uh, and uh, it will take a look at your code and probably ask you do you want to add some features, do you want to distill a specification, uh, and away you go. You're immediately in that conversational interface informed by uh, the kind of behavioral specification language.

Speaker A: Okay, I've got a couple of projects. One that's been going very well and I'm curious and one that's been going a little bit badly and maybe I could use some assistance on it. So I'm going to go try it and I'll put the links in the show notes once I've done it. Henry, thank you very much. This is interesting. Pinning down an LLM's thinking, I like this, I really like this. I'm going to give it a go. Henry Garner, thank you very much for joining me.

Speaker B: Thank you. My pleasure.

Speaker A: Cheers. Thank you, Henry. So I did try Allium on one of my projects, by which I really mean I downloaded the skill files and said to Claude, go on then. And it did actually work. It found a whole bunch of corner cases in the design, like genuine problems that I hadn't thought to think about. It found unknown unknowns. And then I tried it on a couple of other repos and it did it again. And then I tried it on a friend's repo and it worked for that. So yeah, genuinely, I think he might be onto something. I haven't tried it for a long run. I haven't tried it over six months. But there is something here. So give it a whirl if you're interested. You'll find the links as ever, in the show notes. Um, and while your agent is processing them in the background, please take a moment to like, rate and share this episode and make sure you're subscribed. Because whatever the future holds, we'll be back soon with another interesting mind from the world of software. Until then, I've been your host, Chris Jenkins. This has been Developer Voices with Henry Garner. Thanks for listening.