Masters of Automation · 2025-08-15
Key moments - from our scoring
Substance score
66 / 100
Five dimensions, 20 points each
Maxime Labonne outlines Liquid AI's approach to running language models on edge devices, addressing the core challenges of inference speed, latency, and model quality that have historically made on-device LLMs impractical. Rather than treating small models as afterthoughts, Liquid treats them as first-class citizens, using specialized post-training strategies tailored to their constraints. The conversation covers LFM2's architecture - which incorporates convolution layers alongside attention mechanisms to reduce VRAM scaling issues - and concrete examples like LFM1B Math, a specialized 1B parameter model fine-tuned for mathematical reasoning that compresses reasoning tokens from 32K to 4K while maintaining accuracy. Labonne argues that while context windows and RAG aren't dead despite marketing claims, the real future involves complementary approaches: cloud-based models for complex AGI tasks and locally-deployed, interpretable models for privacy-critical applications like navigation, healthcare, and offline scenarios. He brings experience from Airbus (network protocol understanding with BERT models) and JP Morgan (SpamT5 for email security), framing edge AI not as replacement for frontier models but as essential infrastructure for use cases requiring offline capability, low latency, and user privacy.
Liquid AI uses an optimized architecture (LFM2) that incorporates convolution layers alongside attention mechanisms, reducing VRAM scaling issues compared to vanilla transformers. This allows models with 350M to 1.2B parameters to run locally while maintaining usable inference speed and latency below 10 seconds for initial token generation.
Small models cannot be trusted not to hallucinate and lack pre-existing knowledge, so post-training must adapt techniques specifically for their constraints. Liquid treats small models as first-class citizens during post-training rather than as scaled-down versions of large models, enabling specialized fine-tuning on tasks like math reasoning that would be impractical with generic approaches.
LFM1B Math is a 1.2B parameter model specialized for mathematical reasoning through large-scale supervised fine-tuning on 100B tokens, followed by reinforcement learning with GRPO. It compresses reasoning traces from 32K tokens to 4K tokens while maintaining accuracy by selecting samples where success rate is moderate - neither too easy nor too hard - making it practical for edge device deployment.
No, RAG remains essential despite marketing claims otherwise. While long context windows like Gemini 2.5 Pro are useful, models still struggle with effective reasoning over entire context windows, and providing pre-filtered relevant information via RAG outperforms dumping all data into context, especially on edge-constrained devices.
One endgame is cloud-based AGI (OpenAI's approach) with hidden model routing behind APIs, while the other is locally-deployed models with high interpretability and user ownership. These are complementary rather than competitive - cloud models handle complex reasoning while edge models serve offline scenarios, privacy-critical applications, and situations where connectivity cannot be guaranteed.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode contains substantive technical insights about edge AI, post-training optimization, and LLM architecture (e.g., the LFM2 design with convolution layers, token compression from 32k to 4k for reasoning models, RAG vs. long context tradeoffs). However, there are stretches of meandering discussion and self-promotional content that dilute the density. The concrete examples (LFM M1B math, Spam T5) are valuable but don't fully compensate for filler segments about passion and career trajectory.
So we did a very large scale supervised fine tuning uh round and we really adapted, selected the prompts from open source data sets to make sure that this was a very good blend for these models that could really Leverage its existing knowledge and add some more. So in the end we trained them all on 100 billion tokens.
For example, with LFM2. What we did is that we introduced specific convolution layers inside of the architecture. So our three models, LFM2 models have 10 convolution layers and only six attention layers. And that really, really helps us in terms of scaling because while a traditional transformer will really use more and more VRAM with longer context, the LFM2 architecture allows you to, it will still increase, but just not as fast.
The guest articulates a contrarian view on edge AI's future and challenges common narratives (e.g., 'RAG is dead'), and his focus on small models as first-class citizens rather than afterthoughts is reasonably fresh. However, much of the framing - context windows, post-training, fine-tuning - is now well-trodden ground in LLM discourse. The gaming application idea (AI as game master rather than dialogue generator) is creative but underdeveloped. The two-endgame framing (cloud AGI vs. local interpretable models) is intuitive but not novel.
And we took the opposite approach and we tried to treat these small models as a first class citizen and that's why we're also able to squeeze so much performance out of them.
But it has some limits. Right. And we know that having a long context window doesn't mean that it's very effective. There are benchmarks related to this such as long bench, v2 or ruler. And you see that often there's like a claim and there's the reality. And the reality is often quite ugly compared to the claims.
Maxime Labonne is a legitimate practitioner with tangible shipping experience: PhD in applied AI, work on real projects at Airbus (BERT for network protocols), JP Morgan (Spam T5), and now Head of Post Training at Liquid AI. He has built actual production systems and released open-source work (57k-star GitHub course, LLM Engineer's Handbook). This is solid practitioner-level caliber, not a pure theorist or talking-head. However, he's not a household founder/CEO running a major scale operation, which caps the score.
I did it in um, applied AI or more like machine learning applied to cybersecurity... I really did not make the connection between my own research that was more focused on network based intrusion detection... But near the end of my pg it became like very obvious that it was actually the same problem and that the protocols I was looking at were just another form of communication.
I released a paper there called spam T5 and it was a T5 model that was specifically fine tuned to detect spam emails. And the idea behind it is that JP Morgan has a lot of super wealthy clients including Bill Gates I think. And these super wealthy clients get a lot of attacks all the time.
The episode includes concrete examples: LFM2 architecture (10 convolution layers, 6 attention layers), token compression (32k→4k for math reasoning), model sizes (350M, 700M, 1.2B parameters), 100B tokens of supervised fine-tuning, Spam T5 at JP Morgan, BERT for network protocols at Airbus. However, many claims lack supporting metrics: no benchmarks cited for LFM2 vs. competitors, no concrete numbers on inference latency improvements, no specific game developer examples, and much of the discussion about future applications (agent communication, IoT) remains speculative and unsourced.
So let's try to compress this into reasoning trace. That would be max K tokens, right... And this allowed us to compress the Reasoning tokens from 32k to, to 4k tokens uh, while minimizing the performance degradation.
So our three models, LFM2 models have 10 convolution layers and only six attention layers.
The host asks reasonable setup questions but rarely challenges or pushes back substantively. Follow-ups are mostly clarifying rather than probing: 'so tell us more' or 'how does that work?' The host does attempt some deeper questions (context windows, agentic frameworks, industry applications) but accepts the guest's answers at face value without pressure-testing. The guest admits 'I really don't know' and 'it was kind of underwhelming' on agent-to-agent communication, yet the host doesn't dig into why or what would make it better. There are soft transitions and a career trajectory tangent (books, GitHub course) that feel more like an interview than a working-through of ideas together.
And one thing that I see is that like it takes some um, configuration work from like to create an agent to, to, to tap into designing those agentic workflows and then maintenance observability and then designing all that... how much of it is a dream that is Getting sold versus the reality, actually building them.
Yeah. So to the basic examples are like having a phone, consumer electronics in general, everything can embed AI. It can be a PlayStation, it can be an iPhone, it can be really everything. But yeah, as you mentioned, there's also more creative industries.
Computed from the transcript - who did the talking, and the words that came up most.
The following is a conversation between Alp Uguray and Maxime Labonne. Summary In this episode of the Masters of Automation podcast, host Alp Uguray interviews Maxime Labonne, discussing the challenges and innovations in running large language models (LLMs) on edge devices. They explore the importance of post-training techniques for enhancing small models, the future of local AI models, and the integration of AI into everyday applications. The conversation also touches on the role of context in AI performance, architectural considerations, and the dual paths of AI development. Maxim shares his journey from cybersecurity to AI, the use of AI in spam detection, and the potential of agent-to-agent communication. The episode concludes with insights on the future of AI in gaming and the importance of community in AI development. Takeaways Running LLMs on edge devices presents challenges like latency and model quality. Post-training techniques are crucial for enhancing small models' performance. Local AI models can provide privacy and customization for users. Agentic workflows can enhance AI's functionality in applications. Context windows are vital for AI reasoning and performance.
Transcribed and scored by The B2B Podcast Index.
Speaker A: Foreign.
Speaker B: Welcome to Masters of Automation podcast. Today I have the pleasure of hosting, uh, Maxim Labon. Maxim, welcome. Great to have you here.
Speaker A: Thank you. Thanks a lot. Hi everyone, and thanks for the invitation.
Speaker B: Yeah, of course. It's, uh, it's going to be a fun conversation to get started on large language models and post training. One question that I've been thinking hard on is running LLMs on devices. So can you tell us a bit about how does liquid AI work and also what is the most important thing and aspect of post training, um, that makes it so much more powerful and different?
Speaker A: Yeah, it's a good question. So running LLMs on edge devices has always been something that people try to do. I don't know if you remember all these videos about Raspberry PI being able to run like, powerful language models, but of course in practice when you try to really do it, you are confronted with a few challenges. The first one is inference speed. Of course, these edge devices are not very powerful, so they will be very slow. Uh, related to that latency is also a big problem if when you enter your prompt, you have to wait for like 10 seconds to get the first token. Most users will not be very happy with, uh, the user experience in general. And finally, another problem that is very current is the quality of the models that is very low. So in this space, I think the architecture that we've designed at liquid for LFM2 addresses the inference problems quite well because it's really optimized for, uh, these use cases. And finally, post training is mostly here to recover as much as possible and improve the quality as much as possible, to squeeze all the knowledge and all the reasoning capabilities that we can in these very, very small models, because they're just like 350 million parameters, 700 parameters and 1.2 billion parameters. So those are very, very small models that can run pretty much everywhere, microwave included.
Speaker B: So do you envision a feature where even our microwave will be able to run the LLMs and we'll be able to talk to them?
Speaker A: Yeah, hopefully not the microwaves. Um, but I, uh, do think that there's a future, and I see it mostly, for example, on the phone, where instead of running, um, GPT4O and VAFOR language models to do very simple tasks, uh, we can have offline, local, private and customizable models running directly on the phone and you can really touch their weights, change them if you want. And that I think is really, really exciting. And that's the kind of future that, yeah, I would like to see, uh, happening in real Life and from the
Speaker B: perspective of the model training and preparing them to, to be able to run on a machine and then on that state like what are some things like why is, I know this has been discussed a lot but why is post training and uh, teaming and looking in model is so important?
Speaker A: I think that the main thing is that usually these small models are uh, an afterthought. You know, for example you have quen and they release um, a family of models but you already know that the um, favorite kid is actually the biggest model and then you have smaller versions of big model basically. And we took the opposite approach and we tried to treat these small models as a first class citizen and that's why we're also able to squeeze so much performance out of them. So to me this is the right approach. Uh, you need to really craft all your post training data and efforts to really target these specific small models because they're not that smart. You cannot trust them not uh, to hallucinate. You cannot trust them to have a lot of uh, pre existing knowledge. So you need to adapt your um, techniques. And I can go into more details uh, with a concrete example if you want. But yeah, basically you need to really do this adaptation strategy to make sure that you're not training them the way that you would train a frontier model. Because that wouldn't work.
Speaker B: What would be one example like to that will make them more focused and for that specific task.
Speaker A: So an example that we made is LFM M1B math. And this isn't a model that is very small, it's a 1B model and we trained it specifically for uh, math reasoning. So it's quite funny because the model is great at math but it's bad at everything else. Right? It's only focused on math and that's it. And the challenge was okay, how good can it be with 1B parameters? And the second challenge was, you know these models, these reasoning models, they tend to be extremely verbose and especially for math they tend to take like 32k tokens to answer a single question. And we thought well that's impossible. Realistically speaking if you deploy it on an edge device, 2K tokens is like very very long uh to generate. So let's try to compress this into reasoning trace. That would be max K tokens, right. And so to do that we did a very large scale supervised fine tuning uh round and we really adapted, selected the prompts from open source data sets to make sure that this was a very good blend for these models that could really Leverage its existing knowledge and add some more. So in the end we trained them all on 100 billion tokens. So it's like mid training in terms of volume. This is really, really big for supervised fine training. And then we had this reinforcement learning stage with grpo and here we really looked at the success rate of this prompt. If we ask our supervised fine tuned model to solve a problem, what's the accuracy if we ask it to generate like 10 answers, is it like 10%? Is it 90%? So then we tried to really select the um, samples where the success rate was not too high and not too low either. Um, because those are the samples that you can really try to learn better and do a better job at answering. And this allowed us to compress the Reasoning tokens from 32k to, to 4k tokens uh, while minimizing the performance degradation.
Speaker B: So in a way it's like getting, specializing the model to say more by saying less and capturing the value and what it's trying to say in a few words, a few tokens.
Speaker A: Yeah, they tend to rant quite a lot with reasoning. Right. If you look at the reasoning traces of some models, you can see them double checking everything or like doing like calculations but very manually and showing every possible combination. And that's exactly what you try to remove and adapt in your model. So the traces are a lot more concise and they're also a lot more focused which helps the model in the end producing better answers.
Speaker B: It is very interesting. So then, just so that I'm imagining where we will have devices and then the models fully hosted on the device where I don't need to make a call to the cloud or somewhere else. So I could ask my questions to, to what extent that reaches to a diminishing returns when it comes to intelligence. Are there certain questions that I ask unexpected of the model that it runs on a certain machine where the expected output is not as uh, as I expected as a user. Right. Like I wanted to make a phone call for example or I wanted to analyze all my docs in the Google Drive and give me the, give me one analysis like what is the spread today that things that work or like things that you see will work maybe two to three years down the line. That in a way also makes it a uh, competing case with the frontier models as well where they throwing compute and running everything on cloud.
Speaker A: Yeah, this is a very good point. I think that right now the situation, if you try to use a local model on your phone, you're going to see a Lot of chatbots. And chatbots are nice. Like for example if you're on a plane and yeah you're offline. Um, so this is like the best solution that you can have at this moment. But I don't think that chatbots are enough. We can be a lot more creative in what we do. Something that small models can do very well, for example is translation. If you take a model and fine tune it on translation task, it can get really, really good at this. Very competitive with even frontier models. And that would be a way to do it. I think there are a lot of different tasks that you could try to do with this. And in your question you talked about kind of agentic AI if I understood correctly. And I think that for this specifically you need to have the framework and the tooling to do it properly. Which exists currently um, for like bigger models. For cloud models you have like all these LangChain and other popular frameworks that allow you to create these agentic workflows. But currently with edge models I don't think that there is good solution that are ah super focused on this. So at liquid we try to create um, our own solution uh, to deploy edge models easily on the phone. It's called Leap and we have an Android SDK and we have also a iOS SDK and the goal is to provide all the tools to app uh, developers so they don't have to focus so much on machine learning specific stuff and they can focus on what they want to build on the app itself because this is what provides value. Right. So I think this is currently something that is being, is work in progress basically. But at the end of the year I can definitely see like very, very small agentic models that uh, are super specialized to do like this function calls and be able to really crawl uh your Google Docs and connect it with your Google Calendar for example. That is something that I think a lot of people would be interested in.
Speaker B: And one thing that I see is that like it takes some um, configuration work from like to create an agent to, to, to tap into designing those agentic workflows and then maintenance observability and then designing all that. And in one rhetoric is that when the models reach maybe unlimited context window or like trillions of context window at some point and it could do a lot of complex reasoning like is that a, actually a realistic thing? Like at that point we would still have to have this agentic scaffolding and then making sure things work and like tied together like how much of it is a dream that is Getting sold versus the reality, actually building them.
Speaker A: Yeah, this is very connected to the question of RAG is dead, which is something that we hear about like every two months it feels like.
Speaker B: Yeah, it comes up a lot.
Speaker A: Yeah, context windows are great and I really enjoy a model like Gemini 2.5 Pro for example that is really, really good with this huge context window. But it has some limits. Right. And we know that having a long context window doesn't mean that it's very effective. There are benchmarks related to this such as long bench, v2 or ruler. And you see that often there's like a claim and there's the reality. And the reality is often quite ugly compared to the claims. And in general this is a problem with having a lot of information in your context window is one thing, but being able to reason over the entirety of the context window requires like very specific training and quite powerful reasoning capabilities. And we don't have that right now. Gemini 2.5 Pro is probably the model that is the um, best performing in this category. But if you look at even Claude 4, um, the models are not very good. Like they, they can only handle I believe, 2k or something like this. So it's, it is quite limited. They're very good with this context window. But what I want to say is that it's probably not enough to just rely on this and it means that we probably still need RAG or some kind of, you know, retrieval and then injection in the context to also help the model and maybe do a bit of the heavy lifting for them. Uh, so we pre process the data and then they can um, more easily uh, process it to answer the question.
Speaker B: And it's like giving them a point in time in the large data set to focus on and then double down on reasoning to focus on that parts of the data. And Right. Like to that end it's, yeah, it
Speaker A: will always outperform, uh, just like dumping all the data in the context window. We know that, we have quite some experience with it. Despite all the claims that RAG is dead, RAG is still not dead. People still use it. And the answer, uh, the reason why is very simple to understand. Right. It's because if you give like all the relevant information to the model instead of the opposite and providing a lot of irrelevant stuff, it helps the model. So I don't think that will change in the um, coming year, at least
Speaker B: in terms of the context window. What would really influence getting a model to go from having K to 2 million tokens of context window is that just model having more parameters and More compute power behind it. Like what is the, what is the parameter that uh, will take the next leap of faith?
Speaker A: It's a very interesting topic. So there are different ways of approaching it. For example, you mentioned the number of parameters. It does play a role, like it's easier to have long, at least effective long context windows with bigger models and it's a lot more challenging with smaller models. So there's an effect here in terms of compute. It's true, you also need a lot of compute. But I think this is also related to the architecture that you use and the operators that you have inside of this architecture. For example, uh, the vanilla attention mechanism, the way that it was introduced in the original transformer paper, is terrible in terms of scaling. So yeah, if you want to have 1 million context windows with the vanilla attention mechanism, it's going to cost you like really a ton of money and a ton of compute so much that it's too expensive. It doesn't make any sense. Since then, we patched the attention mechanism quite a few times and now we're even improving on the architecture of the models. So processing long context windows is not as expensive as it was. For example, with LFM2. What we did is that we introduced specific convolution layers inside of the architecture. So our three models, LFM2 models have 10 convolution layers and only six attention layers. And that really, really helps us in terms of scaling because while a traditional transformer will really use more and more VRAM with longer context, the LFM2 architecture allows you to, it will still increase, but just not as fast. And this is very, very noticeable. And we did it because we wanted to use it on edge devices where you don't have all the VRAM in the world, UVA constrained. So that's a good example of how to use it. So yeah, I would say that the number of parameters is very important, the compute is very important, but more importantly the architecture is very important. And having some operators like recurrence or like convolution or some local attention is very, very important. If you want to really scale your context window and it makes it more
Speaker B: specialized than the device that it runs on and then the use case, like the data that it sees as part of the context window, then that makes a lot of sense like from the, from the perspective of the, like the future you see. Right? Like, like especially. That is one topic that I'm thinking about deeply. Right. For the some, like some LLM providers are tying themselves to social media, right? Like for example, XAI is with integrated to Twitter now they're integrated with Kalshi with prediction markets as well like feeding that data in to make smarter. OpenAI has this hardware angle now coming in as well like with uh, Jony I've and working together like what's the end game that you see that all the uh, models, we want them to be smarter and then help us to be better. But like in terms of usability or how we consume them changes a lot from user interface to just the hardware or computer. Human design interface comes with like based on your research and then the work that you do like how do you see that evolve?
Speaker A: I think that two different endgames, right there's one that is AGI and that is like the, the most popular and mainstream one. This is the one that OpenAI is going after. I believe that the hardware will just be cloud based and it will still require an API. And then the model behind the API, they don't want you to know what's there. They will do like some routing and like just like trust us, no worries, we will give you the best answer but you don't have to know what's behind it. And the other end game is the exact opposite. It's having models running locally and with very high interpretability because you know exactly what's running there. So obviously like this is more like my approach and I think this is overall more connected to the open source community. Being able to have ownership of the model is something that is very powerful and I don't think that these two end games and approaches are necessarily incompatible. On the contrary, I see them as very complementary because you will not be able to run AGI on a phone, at least not in the near term. So you will still need to have like some cloud based API. Um, and on the other hand there are a lot of applications that you just cannot have with a cloud model. For example, if you have a car, you cannot assume that you have online connectivity at all times, right? So this is one of these examples with like Edge AI is extremely important. It's required. There's no other solution. So I think that we will see the development of these two approaches going forward. And it doesn't mean that one is better than the other. I think they're just complimentary and that's,
Speaker B: that would be a huge like imagine just being hiking and then, and then falling and having an injury and just trying to ask hey, how can I get better? And says no connectivity retry again. And I think that that would be the point of, of failure.
Speaker A: That should be a great marketing material for edgi. Yeah, I can see the video, can see that.
Speaker B: And we could generate it now as well with uh, the products out there. In terms of your background, like what drove you to this field while in your research and while you were in school to your previous research experiences.
Speaker A: Yeah, so I comes from cybersecurity. Like this is what I studied at uh, uni. And then I decided that I was willing to explore something a bit outside of my comfort zone and AI looked very exciting. So that was in 2017 when I started my PhD. I did it in um, applied AI or more like machine learning applied to cybersecurity. And uh, that was a great way for me to combine like my experience with cybersecurity and also discover AI step by step, starting with really like um, basic neural networks and then going forward like implementing more complex solutions. 2017 was also the year that the uh, Transformer paper was released. So that was an interesting time in nlp. At the beginning I really did not make the connection between my own research that was more focused on network based intrusion detection, so finding attacks in a uh, computer network with what happened with nlp. But near the end of my pg it became like very obvious that it was actually the same problem and that the protocols I was looking at were just another form of communication and you could totally represent them and learn them with a uh, transformer architecture. So this is what drove me to abandon cybersecurity and really embrace AI, to do it first with computer networks a bit more. That was my work at Airbus and then outside of it really like full blown LLM and nlp. And this is why I wanted to leave Airbus and I joined JP Morgan afterwards, uh, to really focus on LLMs because that was my favorite thing at the time. And just after that ChatGPT was released and suddenly everybody was very excited about LLMs.
Speaker B: Yeah. And then, and do you see like some cyber security is really interesting I think when, and also like you, you touched on the problems that both touch are very similar like at times as well. Like the. For example do you see LLMs also being an, a pathway and an interface for a new infrastructure that maintains the cybersecurity all around us just because as
Speaker A: reasoning and computeron is getting it's super interesting idea actually. Yeah, uh, in some way I see exactly that. I would even say that it's not just about cybersecurity, it's about computer networks. Something I worked on at Airbus was this BERT model that was especially fine tuned. I would say now Back then it was called transfer learning. Now we would just call it training. You know um, it just train a BERT model from a checkpoint to do like ah, a protocol network protocol understanding. And having a model that can do network protocol understanding allows you to build a lot of applications on top of it. So you can do routing based on your understanding of what happens in the network. You can try to detect attacks, you can try to classify the network flows, you can do a lot of different application. It's really like a neural way of doing computer networks. Back then I think it was not fast enough basically to do it really well. But I was like really excited about it. I'm not too sure actually if this is now something that people do. But yeah, that was like the first BERT model fully dedicated to network uh, protocol understanding and that was a really great time.
Speaker B: Yeah because as, as the infrastructure gets better I could see that even the technology relying on like the spams are going to go up, increase over time. Um and then there will be also bad players as well as a good players using the models to like cloning someone's voice and then calling their grandmother to uh, get their money. Like as basic as that to large scale of M network.
Speaker A: This super interesting project that JP Morgan talking about spam. So I released a paper there called spam T5 and it was a T5 model that was specifically fine tuned to detect spam emails. And the idea behind it is that JP Morgan has a lot of super wealthy clients including Bill Gates I think. And these super wealthy clients get a lot of attacks all the time and this is really bad. They could see a lot of these attacks and yeah some of them they're super difficult to detect. Even as a human when you read it you're like okay, everything sounds good but actually no, there's just one thing that is changing and that is really the attack. So yeah, that was a super fun uh, projects to do. I hope they still use it. But uh, yeah like this uh, T5 fine tune for spam, um, Spam detection was great.
Speaker B: Yeah, yeah I can imagine it's like just everyone trying new ways to get attention or get them to click once and then and it's in a way like AI agents speaking to AI agents. Like you have an AI agent spamming and then another AI agent trying to detect if it's an AI. And then like there's like this phenomenon of um, A to a connection and then like tying to that actually to the A2A. Like Google released that framework A2A to enable that agentic communication between two AI agents. And of course OpenAI has their own agents SDK and then other platforms have their own SDKs to communicate. But at the end of the day, what does it really matter in that communication protocol? Um, because everyone has their own framework. But at the end of the day, what is the, what is the unique niche there that will actually make it special? To that end, there's uh, a project at MIT called nanda. On the project they try to build a, uh, decentralized network, AI agents. You could bring your own agent and then connect it to another agent. So like, from that perspective, where is it going?
Speaker A: I really don't know. Like, um, agent to agent communication hasn't been like, um, a big focus to me. And like every time I tried it, it was kind of underwhelming, to be honest. I'm sorry, like people like hyping it up on LinkedIn and Twitter. But honestly I find it quite underwhelming. I'm not saying that it doesn't have legs. I think that we're just not there yet. And the main problem might not even be the models themselves, although they, they're not the best in terms of function calling in general. But the tooling and the um, engineering that goes behind it is super important for this kind of stuff. And we're just very early. Right. So I wouldn't necessarily discard it. Uh, I would just say I'm waiting, I'm waiting to be surprised. Um, I'm waiting for like a killer application of it. But right now I don't see use cases that really benefit from it.
Speaker B: And I thought the same thing. It's just that it's I think just an object that speaks on other objects that is really not like as complicated protocols going on in there from, from the perspective of now to liquid AI, once the model works good at the device and then it could be any device. What, what is next? Like do, do, do you, do, do you think more the models in different devices will speak to each other as a different protocol? Or like the microwave example, like my MacBook talking to my microwave to cook the food. So how is that models hosted on devices will then scale up, be that Internet of things around us and then make it work actually this time.
Speaker A: Yeah, that's super interesting. Actually, I haven't really thought about that, but uh, I'm writing down, I think that the microwave Talking to the MacBook is a great idea. Um, I would like to see it happening in real life. Yeah, it's a good point. You could have Some agent fusion communication with a kind of IoT framework. But I'm not sure I see the point in doing this unless you have super specific niche cases. And yes, the MacBook will tell the microwave to maybe do something, but honestly my microwave is very dumb. Uh, it's not the smart microwave. So I'm not sure this would help me in real life. What we want to focus on first is really enhancing the capabilities of what these edge devices can do. So we're interested in text, but also in other modalities. We're interested in vision, we're interested in audio. We're interested in scaling it up too so it can run on laptops with like very good quality. Yeah, there's a lot of things that can be done. And something else that we are exploring is providing models that are already fine tuned for specific applications. For example, you want to do function calling. If you just want to do function calling, it's better to use a model that is just trained on this, right? It doesn't need to have chat capabilities, it doesn't need to be able to translate languages. It just needs to be very, very good at following your instructions and doing function calling. And we think that providing these checkpoints that are very specialized in one task is another way to help developers making LLM powered applications that are truly useful because they reach a certain level of quality that you really need before relying too much on these M models and
Speaker B: you will know their specific use case. Why? To leverage it and capture it. Does it then do I have to have more of an agentic scaffolding, agentic framework where maybe one model speaks to another model to go and do the function coding and brings back the results. How to think about that full framework?
Speaker A: Yeah, you could definitely have this kind of agenting framework where you have this centralized model that will just route the queries to other models that are more specialized. A model for um, data extraction, a model for functional calling, a model for rag, for example. My problem with that is in practice the bitter lesson tells us that just having one bigger model is probably going to outperform this super complex framework. So usually it's more recommended to have one bigger uh, model. But if you have one application that is very narrow and all you want to do is just rag, for example, a tiny rag model is probably what you need because it will probably outperform the bigger model for this precise task. It's just not going to outperform the bigger model on all the other tasks.
Speaker B: Yes, it's like there's a trade offs to Win and lose by leveraging each of them. So when it comes to different industries right now, and this is like beyond your work at Liquid and or jp we talked about before about gaming as being one area as well. Where do you see the AI influence other than enterprise stuff that is going on, like influence the world?
Speaker A: Yeah. So to the basic examples are like having a phone, consumer electronics in general, everything can embed AI. It can be a PlayStation, it can be an iPhone, it can be really everything. But yeah, as you mentioned, there's also more creative industries. And for example having models for game design is something I'm really excited about due to my past making video games. I can see the potential of it and how you could leverage the models not in a chatbot way, but really as a game master way. So the model would not just Write dialogues for NPCs. I'm sorry but I do not want to read these dialogues if it's generated by AI, thank you very much. I don't need to play a game to read that. What I would be interested in is having the model really tuning the internal logic of the game to create challenges or even to review what the player does. For example, if the player inputs some kind of description of what they want to do, for example, they say, oh, I'm going to use my sword and I'm going to slash the dragon, blah blah, blah, blah blah. The model can then review this answer and assign it a score and say, okay, you have a good answer, this is what's going to happen. And this is what you have. If you play tabletop rpg, uh, games with your game master, you have these kind of exchanges where you kind of negotiate like oh, can I do that? And the game master will see if it's logical, if it's consistent and then approve or disapprove. And this is what the models can allow you to do and use that way. I think yeah, it's a lot more creative and it brings a lot more value than having just another chatbot or having synthetic dialogues that nobody wants to read.
Speaker B: Yes. Like the game is evolving with you and based on how you behave in the game, giving you like new challenges. And then even if you have a sword and you're trying to slay a dragon, next thing, dragon has a shield or like next. And it's not part of the game and ah, it suddenly adapts to it to create more difficulty.
Speaker A: No, absolutely, yeah. No, I think this is a good example indeed. The game like can evolve and you know, I think as a user, as a player. You don't even need to know that it's LLM powered at all. It's just the game and you just play the game just having fun. And it doesn't have to be like tech first, it has to be like gameplay first. The game has to be fun first before the tech is interesting. Otherwise I probably don't want to play it.
Speaker B: Yeah, otherwise nobody uses it in the first place. The game is an interesting one because I feel like there's a storyline that the directors write and then the AI then makes it more personal, personalized based on how you are and then how you play, the experience you want to get. So then everyone would have a different piece of the game for themselves, uh, unique to them and that will make it very special. Of course. What, what, what's the limiting factor there like today? That's. Is that more on the 3D graphical design generation or is it more on just having such a strong model, high reasoning model to actually compute those variations that could happen if you slay the dragon or not slay the dragon. What are some things there that are limiting today the industry?
Speaker A: I think there are three elements. Those are the ones that I listed previously pretty much about AGI. It's really about the quality of the model. Like you say, um, you need to have a certain level of quality for it to be useful. Otherwise if it's unreliable or if it's like even 10%, 5% unreliable, it's probably not a very good experience as the player. Then you have uh, inference performance in terms of throughput, in terms of latency and it's something that is reactive and it's something that is fast enough to be able to have this real time interaction with the game and with the model. And finally this tooling, tooling engineering that takes time, that requires a lot of iterations to converge to like an optimal solution. I think we getting there but we're still very early. There's a lot of tooling in terms of how to run the model on a lot of different platforms and ensures that it's just going to work right. When you install a game on Steam, you don't have to compile llama CPP to make it work. It just, just works for you. Right. So you need to be able to deliver the same kind of experience to the players. And there's tooling for the um, developers themselves because video game developers in general are not machine learning people and they cannot like the complexity of developing a game to me is already like much higher than AI. They cannot do machine Learning on top of it. So they need to be helped and guided and have like higher level abstractions to be able to develop it as efficiently as possible. And so it doesn't become like the main hurdle of the video game development. It has enough complexity as it is.
Speaker B: And to, to that perspective, when, when I attended your lecture at mit, I was thinking about what would it take someone to learn the basics of fine tuning and training a model. And maybe now training, they need a lot of money but uh, at least fine tuning a model for a specific task and then creating a gentic application for gaming, music and whatever it may be, that that's their passion but also like they want um, to bring value to the world by solving a problem. Like today most of like and like an entry level task is I think pretty automatable right now. Right. Like I could spin off an agent or use it in tandem with myself to get to a point. What, what is the best way for a student like who's graduating today, um, to pick up on which tasks like how, how to get started. Is it just like building, learning, understanding the nature of the models? How do you see that evolve in the uh, for the young?
Speaker A: Yeah, I have a good recommendation. It's a bit of shameless, uh, self promotion but I made the LLM course on GitHub. It's a very, very popular course about large language models. It has over 57k stars. And this gives you a overview of pretty much everything that you need to know either in terms of LLM, um, science or LLM engineering. And if you want to dive deeper into LLM engineering, I also wrote a book, the LLM Engineer's Handbook on that precise topic. There are a lot of resources that are linked in this GitHub. It's not just about my work. Uh, no worries. This is a lot about curating high quality educational resources to give a path to people who want to get into this field so they understand like they can map already what's, what's there, what they need to learn. And then you can have what uh, I would recommend is having like a high level overview of the different categories. For example, you want to know what quantization is because it's very important. You want to know what fine tuning is, you want to know what inference optimization is about. But then you can specialize into something that probably resonates with you, that is useful to you. And here you can start having a project on this. You can try re implementing stuff from scratch. This is a great way to learn when you're a bit more comfortable with this concept and yeah like really specializing and creating your own niche because this is what we want the most in the industry. It's people who are specialized enough. So you specialize for example data generation, you specialize in post training, you specialized in evaluation, you specialize in inference optimization infrastructure. Um, I think this is a great way to really map the entire field and then find what interests you the most.
Speaker B: And in terms of the post training, what drove you to post training Pro
Speaker A: ah training is really connected to my early work. I talked about this BERT model that I trained that was already post training. I didn't know back then didn't uh, have the name but it was already about that. To me prostrating is really nice because you can really interact with the model and that's something that is like really important to me. You can take a checkpoint and kind of make it yours. Um, there's a lot of nice things since the release of Llama that you can do with post training. This is where I really started doing fine tuning as we would call it right now. Before that I did it with GPT2 quite a lot. It wasn't that good honestly. You could like really quickly reach the limit. But it was really interesting already. One of my first projects with transitive AI was, was in like 2019 I think and it was a GPT2 model fine tuned to write scientific articles. And I had the dream of uh, having this GPT2 model helping me to write my own articles. It was half consistent like um, it was mostly hallucinated so it was really, really funny to read. You could prompt it with like very important questions and get like surreal answers but it was not uh, really useful in the end. So when the models got to a state where the quality was high enough for them to be truly useful. Yeah this is what excited me the most.
Speaker B: So you will have that interactivity with actually something that's like it's talking, it's there that it's um, set a state that it's giving responses and yeah I feel like the also the progression sometimes we don't see it but like maybe GPT2 to GPT3 or maybe another model that comes up a little better then we completely forget what it was like before. But then actually looking back like oh two, three years ago they were hallucinating 50% of the time and then making stuff up and now it's in a much better uh, world with all the uh, like to like to your point specialties that people have and then that Investment specialty, then goes into investing into model development, refinement and release as well, which is huge. I know a lot of people ask about that, like, well, how can I be relevant today and do things?
Speaker A: Yeah, an example of this is, um, I had this project where we tried to automatically generates unit, test, Python, Java code, uh, that kind of stuff. And we worked really hard, like fine tuning a model. We work really hard, like, uh, benchmarking against baselines, like vendor solutions and stuff. And then ChatGPT dropped and it was much better than anything else. No comparison possible. Right. Um, and that was the original ChatGPT, which is now really, really bad compared to everything else you can find online. So it shows. They have the, the speed of progress has been quite unreal. And even now, like, the models keep getting better and better, more and more efficient. And yeah, that what really excites me, I think in this field, uh, it's,
Speaker B: it's never stagnant, I think in a way that, like, it keeps going and then keeps evolving. So for the last part of our, of our conversation, I want to ask you about, like a few things that taught you, things that made you choose this field. Like, for example, what is like one book or one person that you met that made you, oh, I want to take this path and I need to change my direction into where I am today.
Speaker A: It's difficult to really, um, find one person in particular. To me, coming from cybersecurity, a thing that really surprised me with AI, and that was in 2017, is how the AI community was super open, welcoming, and they had a ton of great educational resources online, which is really not what you would find in cybersecurity. People were not super welcoming, to be fair, and knowledge was also, like, quite gated. So I think, yeah, uh, I would say, like the entire AI community to me was what truly drove me into this direction because I could learn very fast. If you're motivated, you have everything that you need online to be able to do whatever you want to do. And that's really, really amazing. I think that we forget how incredible Wikipedia is, how incredible the Internet is in general. And even when you have that community, like the AI community is really fantastic if you have this need to learn because everything is available. And to me, that's what really motivated me to delve deeper and deeper. And yeah, this is where I am now.
Speaker B: That's a very good point because I remember like three, four years ago, the main conversation was that you could watch anything on YouTube and learn it now. So now you can watch anything on YouTube. And you also have an AI assisting you to do things on top of it. Just, there's so much accessibility to actually not to have any excuse to, uh. Yeah, maybe the only thing that, uh, is missing in the puzzle is since the barrier to entry is really low now, like, and everyone is welcome, everyone's helping each other out. The community is growing exponentially. People are helping each other, and AI can do things. Perhaps the biggest thing is like, how, like, what is the passion of the person and then how creative they want to be to handle a, uh, problem probably that's dear to them.
Speaker A: That's a very good point. I think that if you're passionate about it, you will have this drive to learn more and explore, like, all the resources that are available, which is why it's probably important to try to find your own niche, something that you own, something that is really yours and that you feel comfortable, uh, about, which is probably not what everyone can do. Right. Maybe you realize that you don't like AI that much, and that's completely fine too. But I think it's a good point to reiterate. Yeah, it's, um, it's something that you need to find for yourself, I think.
Speaker B: And the last question is, if there, if, if. Let's say we reach to the point of AI and then it runs on cloud, machine, wherever it could be, but it's very smart, what will be the number one. And it could solve one problem because maybe we have only one data center left. What will be the number one problem that you think to ask it? So it will go ahead and solve
Speaker A: it in today's world, how can I make it run on a phone?
Speaker B: It will make the job easier.
Speaker A: Yeah, I think in terms of efficiency, this is like, I joke about it, but it's really important. It's really important for environmental reasons. It's very important for, like, just like overall efficiency reasons and also kind of, um, privacy and governance issues with AI. We don't really want to live in a world where this AI would indeed be just in one data center and gated by a company. So, yeah, that, that would be my answer.
Speaker B: Definitely agree. I feel like if that problem is solved, a lot of maybe thousands other problems are solved with it,
Speaker A: hopefully.
Speaker B: Thank you very much once again. It was great to have you and it was a great chat.
Speaker A: Thanks a lot. Yeah, really enjoyed it. Thank you for the invitation.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
Maxime Labonne: Designing beyond Transformers | Learning from Machine Learning #12Learning from Machine Learning · features Maxime Labonne86 / 100
Where's the Smart Money Going in AI? Rob May, Co-Founder & CEO of NeuroMetric AI, on Inference, ROI, and the Bets That MatterMaking Data Simple · on Inference optimization91 / 100
Drive Impact Through Systems Like an AI-First PMMProduct Marketing Adventures · on RAG (Retrieval Augmented Generation)87 / 100
Public Intelligence Limited: A Punk Rock Guide to Sovereign AIData-powered Innovation Jam · on Post-Training87 / 100
Governed Autonomy: Why Edge AI Needs GuardrailsIndustrial AI Podcast · on Edge AI87 / 100
Adamma Ihemeson on Building Autonomous AI Systems You Can TrustBack on T-R-A-C-K · on RAG (Retrieval Augmented Generation)84 / 100
ex-WHOOP PM, YC Founder Ben Katz: AI-Native Sports Training for Everyday Athletes & Why Coding Agents Are the New 10x Workforce76 / 100
Snowflake VP of AI Baris Gultekin: No AI Strategy Without a Data Strategy & Why Skills Are the New Apps67 / 100
MIT Prof. Ramesh Raskar: The Internet of AI Agents: Why AGI Won’t Be One “God Model”?82 / 100
MIT Prof. Stephen Wolfram: Computation, AGI, Language & the Future of Reasoning.97 / 100
Part I: From MIT Researcher to YC Entrepreneur. Building Workflow Stack with AI w/ Bernard Aceituno63 / 100