How Inference Layer Innovations Are Changing AI Efficiency and Costs | Sudip Roy Cofounder & CTO of Adaption Labs

Speaker A: The inference costs have gone down by 300x but the demand has gone up

Speaker B: by almost like a million x inference APIs. I think what we are seeing is pricing changes all the time.

Speaker A: Fundamental property of large language models is especially large language model inference is that it's autoregressive. High variance in the uh request and response profiles and the autoregressive nature of those combined make it like really hard to build robust and reliable systems.

Speaker B: All AI problems are looking like um, the self driving car problem. Like we are making 80% progress very quickly, 10 to 15% progress in six months but the last 5% is taking forever.

Speaker A: Prices would basically want to integrate AI into their products and in uh, many cases they are used to having that 99.9% reliability like when they were not using AI, right? And then now suddenly they're like okay, Even getting to 90% is really hard. If the AI system itself was more dynamic and could continuously learn, then I think we suddenly have an opportunity to bridge that last 5% gap.

Speaker B: Hello everyone. Welcome to Startup Project. Um, my guest today is Sudeep. Uh Sudeep is the co founder and chief technology officer of Adoption Labs. Um, I think we are seeing two trends simultaneously playing out in AI. One is uh scaling uh just purely with compute, uh and adding more cluster of GPUs put more data into large training models. But then there is a set of new efficiencies that are coming up at the inference layer. Uh and I think Sudeep and team are working on um, adapting to those trends. Um, so Sudeep was previously ah director at Cohere, uh working on inference and shipping the serving and fine tuning infrastructure there. And I think before that um, he spent considerable time at Google Brain. Uh he co authored tfx, the platform that powers uh production ML inside Google, help build pathways, um, and now is working on adaptive uh labs. Uh Sudeep, welcome to Startup Project.

Speaker A: Thank you, thank you so much for having me. Looking forward to the conversation.

Speaker B: Yeah.

Speaker A: Before jumping uh into adapto I want

Speaker B: to talk a little bit about your background, you know your experience working at Google bring. Can uh, you talk a little bit about you know, what types of problems were you guys working on uh when you first joined at Google Brain and the follow up is

Speaker A: did you guys

Speaker B: imagine the space would go this parabolic in this short time? I guess. Right. So that, that always sort of caught uh, everyone off guard in some sense.

Speaker A: Yeah, um, so I actually come from a data management background. I went to um, Cornell University in upstate New York I did my PhD in data management systems working mostly on more traditional systems like transaction processing, um, distributed transaction processing. And then um, when I joined Google I joined as a researcher, uh, working on um again like in the data management space. But over the years I uh, saw back then it was more um, machine intelligence and machine learning, um, being um, as I guess pervasive in all products. And I decided that that would be an interesting area to stretch myself into. So I moved more towards just doing end to end machine learning. And this was still in the era of you know every task gets its own set of models you train effectively, like thousands of models. And um, so uh, the product that we developed, TensorFlow Extended, was effectively a harness around TensorFlow. TensorFlow was the core infrastructure that was used for the training piece of it. But then end to end machine learning involves a lot of other pieces right? So starting from data processing to um, data validation to the model training itself to post model validation and evaluation and all of those. And TFX basically just standardized all of that end to end processing. And um, then in my last um, three years at Google DeepMind, um, I got an opportunity to work on a really interesting project. Um, it was ML Pathways and the idea was to do, develop the infrastructure for the next generation of AI models. And this was also when the field had pivoted towards foundation models which were substantially larger than the earlier generation of small machine learning models. And um, so we developed Pathways which would be um, a system that can run across multiple TPU pods spanning possibly even different uh data centers. And um, so there were a lot of interesting systems, systems challenges uh, that we encountered in developing the system. And I think the system is still used for training and uh, serving like the Gemini series of models. Now um, during these last three years I also wanted to get the flip side experience of effectively uh, viewing these systems problems from the point of view of a um, machine learning researcher. So I also tried to explore a lot of um, uh AI models, mostly transformer mix models and how they can be used to solve complex optimization problems and systems um, like ML compilers and such. But yeah, that's briefly my journey at Google DeepMind.

Speaker B: What, what was your experience working at Cohere? I think your last job was at Cohere, right? And you focused primarily on inference. Talk a little bit about.

Speaker A: Yeah I guess uh, yeah, uh, your previous question was also like did I actually anticipate large language models and found models taking up. And um, I would say that I could see the potential of large language models and that was Primarily I guess one of the motivations of switching to Cohere, which was still early in its days, it was around like a seed stage startup and I could see the impact that this technology could broadly have. But I think um, yeah, I would definitely not claim that I could anticipate the scale of the impact that we uh, have seen over the last three years. Um, yeah, more so even within just like the last six months or so. Right. The, the pace of um, technological innovation, the um, how pervasive it has become in our day to day lives, those were definitely like very hard to anticipate. But yeah, one of the reasons why I switched to GO here was largely because I wanted to work in a really fast paced environment where the gap between like research and um, end to end product was uh. Yeah, was much smaller.

Speaker B: I want to point out that you know, when previously before foundation models you mentioned we had models for each purpose. Right. Like if uh, a company like Intuit wants to like build uh, you know, all the tax forms that convert into structured data, they used to like, used to take all the previous data uh, of uh, you know, tax documents, scan them, create data sets, tag them and build a very specific model or like use an existing model that's hosted on AWS or Gemini specifically for image recognition and then you know, try to optimize that. So we were focused on very specialized models. I uh, want to get your view on like how long would, you know it it took us uh, to get a project like that online versus do the same thing with now with large language models. Like how both in terms of time but also like how much resources we were spending. Because we always think of LLMs as like this costly. Now we are spending hundreds of millions of dollars in training clusters, you know, GPUs, infrastructure. But I think there's also this angle

Speaker A: of what we used to do before

Speaker B: and a large portion of that is now being outsourced to large language models to do it. Right. Like as whole set of subtasks we would otherwise do specific ML models as contour LLMs. Right. What are your general thoughts? Like how that changed?

Speaker A: Yeah, if you think uh, of uh, maybe I don't know, seven or eight years ago when we're still doing these very bespoke models for specific tasks. Um, I take the end to end time from okay, envisioning that we need to solve this problem with a machine learning model to having something which is deployed part of the product. It still used to take let's say one or two quarters like, especially at like uh, you know companies like Google where there are a lot of like production issues that also need to be resolved. It's not just about the training piece of it. Um, the resource, uh footprint of training these models was fairly small. Um, but um, um and the, the infrastructure that was built around was primarily TensorFlow which was still also in its infancy. So there was a period of like co development where the hardware um, uh increasingly the hardware for computing increasingly moved more and more towards accelerators from just general purpose compute. The infrastructure in terms of the frameworks like TensorFlow, they were also developed. Um, so all of that definitely helped in reducing that quarter to two quarters to possibly like one quarter. Um, but um, it still took a lot of engineering hours in order to actually productize a model. Uh, with foundation models if you look at like the current iteration velocity of new models arriving in the market, it's still um, six months to a year gap. Right? Um, I mean it kind of feels like it's actually much shorter that every week there is a new model in the market. But these models are produced by different companies and different groups of people. But if you focus on one particular lab, the gestation time is almost like six months to a year. Um, and um, it kind of makes sense, right? Because now you're training this one model that is good at thousands of tasks. So you kind of can justify that. Okay, I will actually spend massive amounts of compute, I'll spend massive amounts of human resources and ah, data in order to get this one model because it'll solve thousand different tasks so I don't have to train those thousand models. So that's kind of like the justification for why um, it makes sense to train these really large foundation models. And um, over the last three years or so what we have done is we have basically said that okay, we'll spend humongous amount of resources on the training and then we'll optimize surfing as much as possible in order to reduce costs.

Speaker B: Right.

Speaker A: Um, but now what we are seeing is uh, we have run into challenges there as well because the models have gone into like the trillion parameter regime. The inference costs have really skyrocketed because of uh, multitude of factors that we can go into later. And we are slowly now, uh, maybe almost changing uh, to the world where smaller verticalized models which can deliver um, equally as good results are suddenly more appealing than just the general purpose foundation models.

Speaker B: Can you talk a little bit about like why inference costs are still high? Like I mean what are the different factors that are driving the cost?

Speaker A: Right?

Speaker B: Why Is inference cost so high?

Speaker A: Yeah, actually if you take a broader look at um the inference cost just on a per token basis the inference costs have fallen right by orders of magnitude. By um, some estimates it's around I think 300x uh drop in inference costs per token um for even like the most premier models. But overall the amount uh that is being spent on AI has dramatically increased even though the inference costs have smaller. And that's because of a few different reasons. One the model sizes have become bigger um so in order to extract each token you need to spend much more compute. Um the second is uh, the um number of tokens used for any different any particular task is substantially higher because now we have reasoning models which like to actually think um, uh a lot in some cases and that contributes to it. And the third is with agentic workloads. It's a single task is, can now possibly be served by you know a parallel set of requests which go on in the background um, and some of these background jobs can run for hours uh, uh, almost like cron jobs in the traditional uh systems uh sense. So all of these have effectively contributed to somewhere between like a thousand to a million x increase in the number of tokens that we are producing. And so the inference costs have gone down by 300x but the demand has gone up by almost like a million X and is still accelerating. So that has created this almost like ah, a demand supply gap which makes inference feel really expensive.

Speaker B: And there's this factor of probabilistic nature of the output. Right? Because um, we both can try to do the same task but we will slightly give different prompts and one prompt might get an efficient immediate answer. Another prompt might uh, take a longer time or take more tokens to get the same output. And it's sort of like we're taking two different paths to get to the same destination. And one path could uh, you know, because they're part and uh within how the user sort of like is efficient or not efficient in terms of like purely prompting or just the way you know systems are interpreting a particular prompt. I think there's also this of uh, because previously when you put out an API we used to be able to sort of estimate the cost of uh, you know, how many API calls means how much cost. Right. In the cloud era it was a little bit more reliable way of for us to calculate cost when we put out API as a product out there. And with inference APIs I think what we are seeing is, and that's why we are also seeing this pricing um, changes all the time because uh, I was telling you about this product that you know I was using. But that product actually it started out as a subscription product but now they had to overlay a credit limit on each subscription, uh which is in some sense not new. We had like busting credits uh, you know, in APIs, right? The concept of bursting and you know when you have like peak volatility in some sense we had that. But bursting was built in for like uh, at 9:00am M, everyone is logging in. So you have you know, some API calls going through the roof at particular time. But now this we have to combine either a provision model with a PECO model versus uh and this is coming out from the fact that these are probabilistic in the nature, right? Like in some sense that's also part of why you have higher costs in inference.

Speaker A: Yeah, I think uh, partly it's the probabilistic or non deterministic nature of it. But I think the fundamental I um, guess property of large language models is especially large language model inference is that it's autoregressive, right? In the sense that um, each new token depends on the previous token. Uh so you cannot just emit the entire generation at once. It's really hard to parallelize the entire generation at once as well. Right. So it has a very sequential nature. What that ends up, um, the implication of that is um, now suddenly the APIs, most standard systems, APIs were built around the fact that each API call is more or less deterministic and homogeneous in the sense that I can treat each API call as like one individual um uh atom of uh atomic set of work. And um, it's you know I can scale up my infrastructure horizontally and parallelize things. But now with this autoregressive nature of LLM inference that um, doesn't uh, hold anymore. And that introduced a lot of challenges in um, especially even two years ago we ran into like a lot of challenges and there was a lot of innovation around okay, let's do continuous batching, let's do things like page retention and flash attention in order to optimize um inference. There were other techniques um, that where like for instance recently Turboquant was published that was all to optimize like the KV cache movement. But um, a lot of that is not just about the probabilistic nature of the LLM entrance. Uh, in fact like even for the traditional ML models before the foundation models, right, um, uh whenever we used to deploy machine learning models it was probabilistic Right. Uh, if it was a classification model then you know, depending on the model it'll give you one class or the other. And um, there was a period through which like you know, uh, a lot of the SRE tools especially you know, went through this change to um, make themselves adapt to this more statistical way of uh, building systems. Uh, with foundation models. I think the shift is not just the probabilistic nature of the models but more the um, high variance in the request and response profiles and the autoregressive nature. And those combined make it like really hard to build robust and reliable systems. And that has implications on cost as well.

Speaker B: Right?

Speaker A: Um, because suddenly it makes uh, it challenging to pack these requests efficiently on ah, a given set of hardware. Um, so a lot of GPUs are actually more or less like underutilized because we need to over provision them because we cannot handle surges. Um, yeah. And that, yeah combined with the fact that serving the foundation models itself is so expensive means that that over provisioning has an associated cost with it. Right. That has to be passed on to the consumer somehow. Which is why a lot of the subscription based pricing structures are breaking now. Um, and people are increasingly moving towards more usage based pricing where they can pass on those costs to the end users.

Speaker B: Uh, I think it's a good segue because we talked about some of the inference serving techniques like kiwicaching, quantization I think Lora, uh, um, I think. So what led to adaptable apps? Like what is the main thesis that you guys thought okay, we need to start a new company, uh, and what are you basically trying to attack it?

Speaker A: Yeah, um, with adoption we uh, effectively saw uh, two sets of users who are being fairly limited in their accessibility to AI. The first set is um, users who are working in underserved communities or underserved languages, uh, who don't necessarily have the resources to own their AI, uh, and are limited to whatever the Frontier model APIs provide them with. So that's one class of users that we wanted to solve the problem for. On the other hand, enterprises had a lot of private enterprise data, uh, that they were not able to leverage properly in order to build a moat, uh, for themselves by having custom vertical models. And um, largely the reason why that was the case was the foundation model training or customization was considered to be a bit of a black art which was limited to only the frontier Labs. And our mission at Adaption is to enable all of these people to have more control over their AI, uh, end to end, um, and um yeah that was uh the long term vision and mission with which we uh started adoption. Um and our technical approach is to invest in gradient free continual learning where we want to enable intelligence to evolve as the world around it changes and we want to do it in a gradient free manner. And what that means is um, uh we want to make sure that the learning or the change in behavior of the AI stack feels almost instantaneous. So you shouldn't have to wait for weeks or months in order to see the change. But as the environment around it changes, the model or the AI stack should be able to interact with the environment and evolve very naturally and gracefully over time.

Speaker B: So uh, gradient comes from gradient descent uh which underpins everything in uh MLM

Speaker A: AI uh pretty much. I guess gradient uh comes from the fact that um you'd want to effectively update the weights of the model. But uh, one of the I uh guess uh positions that we are taking at uh adoption is that we uh want to take a more full stack view of AI as opposed to just considering the model to be AI. Uh increasingly even today if you have seen the wider field a lot of the innovation has actually moved uh out of just um the model itself to the systems that are built around it, to the interfaces that are designed around it. Um and yes like the model is still an integral part of it. But um to have a successful AI system you need to effectively like invest and consider what you are building around it. So we at adoption want to take that approach where we want to innovate across the full stack and co optimize across the full stack starting from the interface um, or the environment uh and thinking about how the model interacts with the external world, whether it's humans or other agents, to thinking about the harnesses that are built around the models themselves to thinking about the models themselves. And we strongly believe that by co optimizing across the full stack we can uncover solutions uh, that are just not possible to do by focusing on one of these layers in isolation.

Speaker B: So in some sense how I'm understanding it is um today to like coding for example I think since um, I would say last November.

Speaker A: Right.

Speaker B: Sort of like spread like a wildfire coding with AI. Right. Um, but theoretically like even when GitHub, GitHub Copilot was launched, I think the GPT1 or GPT2 I might be wrong but very early on uh, and what really changed from there to here is just model quality. Underlying model quality definitely helped the change. But the form factor or the harnesses that are built around for like reasoning and um, you know, just the agentic form factor and some of the changes that really got the advantage and made more people to use it. It's just sometimes it's simple UI change

Speaker A: actually improved a lot.

Speaker B: And then you added multiple reasoning capabilities which is not. And there is obviously the model capability itself, but just making multiple calls to plan better, to reason better and come and store it as a plan and then retreat on top of it. Some part of it is just pure technique on top of the model. Right?

Speaker A: Like you could do a version of

Speaker B: this on top of GPT2. It might not be as good, um, because you still need a model that's better at coding. Um, but it was already like you could see that this evolution coming in some sense. So you're almost saying that hey we'll have these models coming out from large labs, but you still require um, just by building new things on top of it. Whether it's like inference level, but also like um, just how you think about these interfaces on top of models can be a huge advantage. Is that the right way that think about what you guys are approaching?

Speaker A: Uh, yeah. So we definitely want to think about like the interface, um, and uh, everything else that you just said. And um, one of the reasons why we want to emphasize that is as I mentioned, we want to do continuous learning. And the way to do continuous learning is to be able to interact with the environment. And in this case the environment is either the human user who is providing you with the feedback or possibly like other agents, um, or sensors that you are interacting with. Uh, which is why the interface is important. Right? That's your primary like feedback collection mechanism. Um, and we want to use those signals to optimize long horizon tasks. Right? So tasks which may take um, hours to um, possibly even days in the future. Um, and uh, it's only possible if you think about the mechanisms or the right appropriate ways of collecting that feedback and how you can actually fold it into the model layer through the harness. Right. So that's why we believe like it's important to co innovate across the full stand. Um, I would say um, uh, it's uh, one of the reasons why coding especially uh, has been really successful over the last six months is uh, not just um, because of the interface or the harness, but also because it's actually an easily verifiable problem, right? In the sense that the code either executes and does what it intends to, what it should do or it doesn't. And um, so a big part of it was the data generation and collection, um, and the feedback loop of what happens when it makes a mistake was really fast. But there are a plethora of other use cases where that mechanism is not as binary in the sense that it's not just true or false. It's mostly a human. How does a human feel about ah, a particular content, uh, for example, and those are much more like softer signals. Right. So, and I think there's still a lot of interesting problems that are to be solved in how we can actually make those feedback loop around softer signals. Um, uh, yeah, uh, help the system open, improve over time.

Speaker B: What would be some of those use cases look like? Uh, I mean are these like the same use cases in some form? We are doing not so good. Like we're solving 80% like say with Cloud cowork or 365 copilot or glean. Uh, because I'm talking about your second set of users, right? Uh, in the enterprise context we have some tooling, they're good. They have some use cases that are unlocked. Uh, give me an example of like, what could be better if like adaptive really succeeds. Right. If you guys. That, that last signal, if you fill it out.

Speaker A: Yeah. Um, so let's say that you actually um, I think the most canonical example is let's say a customer support agent, right? So let's say that you use uh, some form of AI to uh, have a chat interface, uh, that is driven by a customer support agent. Um, today if, let's say um, you know, a user asks for something, the agent goes, it spawns up sub agents, it tries to do the task, it comes up with something that the user does not like and the user actually says that okay, this is not what I wanted. Um, that's usually a bit of a dead end, right. Uh, and the system did not learn from that. Right. The next time another user comes and funds the same request it is going to give it give you the same results after making those, you know, possibly hundred inference calls. So I think what we want to do is like we want the system to be able to learn, right? So sure, it failed in that instance, but how does it adapt now that it actually has an explicit signal from the human that this is not what they intended? How do we fix it automatically? Um, so that's an example um, of what we want to solve.

Speaker B: This is something I encounter even in like when I'm coding with AI is that it forgets the last three chats we spent almost 20 minutes figuring out the problem and somehow next time I deploy it, it deploys it in the wrong way that we've already figured out and doesn't actually keep track of it. So now after I explicitly tell that when I found the right solution to document it and look at it, or like create a readme file or like a deploy file so that it doesn't forget the fact and also note online we made these decisions so that would help reinventing this.

Speaker A: Yeah, that's definitely, um, I think we kind of went through this journey with prompt engineering, to be honest. Right. So we had this, um, you know, LLMs which understood instructions in a very specific format. And um, as humans we basically uh, almost like adapted ourselves to give this really elaborate instructions. Right. So, um, uh, and um, many. If you, if you have seen like the prompt structure of any app that is using AI, it could be like multiple pages of do this, don't do this, do this, don't do this in a specific. And that makes the system very fragile as well. And it's a kind of an unnatural way, uh, of communicating with the AI. And we are kind of seeing something similar with the example that you said. Right. I write code and I kind of experience the same thing where, okay, I'll have like some. Okay, I need to give it the context of the overall system architecture that it has to keep in mind. Uh, I need to give it context around the current design that I'm actually working on and then it may do the right thing. But um, yeah, it would be much easier if the system just learns about that, keeps that context and keeps the relevant context, um, for me, um, and learns from it over time.

Speaker B: Uh, I think it also, for me it's a proof that LLMs are not conscious because a conscious nature by itself is contained. So that's always a proof like alums are not conscious.

Speaker A: Yeah, I guess that's, yeah, that's going into more like a philosophical conversation now. But uh, yeah, so for the first product, I think you mentioned Adaptive data.

Speaker B: Right. Um, so what does this adaptive data do? And like how does it solve this gap, um, of that last mile Context.

Speaker A: Yeah. So, uh, Adaptive Data was the first product that Adaption launched. Um, we launched it I think around four weeks or so ago. And we have gotten really strong, um, response from the community. In fact, we have had more than Roughly, I think 25 million data points that have been processed within the last, um, four weeks or so through the product. And what it enables you to do is, um, effectively if you have either low quality data or if you just have an intent, but you don't even have um the data in the right structure or the format to be able to connect it to your downstream AI systems. Adaptive data can makes that transformation really seamless and um, the product um today uh enables you to do that for downstream uh training purposes. But in the future we'll expand it for other integration points with AI as well. So the idea is if you're an enterprise and uh, maybe you have some high quality data which is really sensitive that you cannot use as is or um, you want to customize a model but you just are starting with like zero data or very low data and you want a larger dataset in order to create that custom model then adaptive data can um, help you seamlessly achieve that. Um um uh we recently also announced Autoscientist which is a step more towards the direction of why stop at the data itself. But uh, we will also solve the model problem for you and um, Autoscientist allows us to co optimize both the data and the model um so that you ultimately get that custom model um which uh is trained on that adapted data um end to end. Um and what we have seen is by uh doing this co optimization across data and model we can achieve much higher um quality both in terms of like the model generations um as well as fidelity um than what you would be able to do with just iterating on the model itself.

Speaker B: What types of customers are finding this useful for them?

Speaker A: Yeah we have seen a broad variety of um users uh there we are working with some financial uh companies who are, are using it to create uh vertical models that are really good at trading. We are working with um, uh some companies who want to uh, who have customers uh in low resource languages across the world so that are usually like not very well catered to uh by the standard set of models. So they are using adaptive data to um you know generate high quality data in those low resource languages and train custom models. On top of that um we have customers who are using it to generate really long context uh uh data because they want to fix long context issues that they're encountering with the models. So um, yeah really a wide variety of uh use cases.

Speaker B: Um so uh, if I could look at like the abstraction layer you are sort of like between post pre training before inference. So you're basically offering techniques at inference there. Is that a good way to say it?

Speaker A: Uh so with adaptive data it's mostly offering uh you high quality data that you can use to customize models either at med or post training stages. Um yeah so but Data.

Speaker B: Is that how I should understand this or.

Speaker A: Yeah, it's. The output is going to be a high quality SFT data set.

Speaker B: Got it, got it. And you also talked about like uh, adapter intelligence. Right. I think and adapter interfaces. What is the next step going to be here, like in terms of like providing more intelligence?

Speaker A: Yeah, so the next step which I mentioned was like autoscientist, which is a step towards adaptable intelligence. Right. So there we are not just stopping at, okay, here's a uh, really high quality output data set, but rather uh, we will also take care of the training process for you where we are going to uh, co optimize both the data and the model and deliver you with a model that is very highly tuned to your particular tasks. Um, uh, with a guarantee that it does really well on that task. Right. So, so that you don't necessarily have to have a really uh, in house uh, set of expert users or expert researchers in order to get to that model. Because that's kind of one of the unsolved problems. Um, I think there's a lot of players who offer like fine TNA APIs which remove the infrastructure complexity uh, from training itself. But a lot of people run into hurdles and using the APIs because they either don't have the high quality data that is needed in order to do the fine tuning, uh, or they don't really have the enhanced knowledge in order to tune the training process itself. Uh, and with adaptive data we are solving the first piece of the puzzle. With Autoscientist we are solving uh, the second piece of the puzzle. But more importantly we are co optimizing m it with the data itself as opposed to just doing the second piece by itself. Um, so that's a step towards adaptable intelligence. Uh, if you think about more broadly what model training is, it is an example of a long horizon task. Right. The training process itself can take hours to days. Um, so it's an instance of a long horizon task that we are now optimizing. And with adaptable intelligence we want to uh, push more in that direction of optimizing other long horizon tasks in the future. Uh, and with adaptable interfaces, which is the third pillar that we are focusing on. We want to also explore what intuitive interfaces that are very task specific look like where it is easy for users uh, to consume the information in a very native interface that is mapped to their specific task, but also easy enough for them to provide feedback in a seamless way so that they can um, using adaptable intelligence, see the system correct those um, behaviors or errors over time.

Speaker B: Give me an example for adaptive interface. I mean I know it's still like you guys are thinking about it, uh, what would an adaptive interface look like?

Speaker A: Um, I can't go too much into the detail because it's still roughly in the works but hopefully we'll soon be able to actually show more about it. But um, think about it that today if you go to a chat interface, chat is like primary way of interacting uh, with uh, uh, with AI. It's not very task specific, right in the sense that you are and you get the same interface irrespective of what you want to accomplish out of the model. But um, as humans we find it much more palatable if information is organized in a specific way format that makes it easy for us to like grasp what it is. Right? It may be charts, in some cases, it may be images, in some cases, it may be a combination in some cases. Uh, uh, so adaptable um, interface basically builds up on that thesis to create like really task specific interfaces that make it much easier for you to consume um, information.

Speaker B: So in some ways it's almost like a full stack where you start with you know, providing if you have data, good, bring that data. If you don't have data, we are starting with providing you high quality data for your specific tasks that you're optimizing for. And then once you have the data then you sort of like build the intelligence layer. So with all the Amazon, all the post training techniques and correct me if I'm wrong and then once you have all that techniques in your system then you provide interfaces from which that will again feed back into your uh, techniques that you can implement and then again give back uh, more optimized results. Uh, I think how it looks like it's a full stack approach from starting from data and then intelligence and then interfaces.

Speaker A: Yeah, absolutely, you captured it really well.

Speaker B: Uh, so it also like looks like if I'm like trying to create a new let's say cursor computator, uh, and I don't have a lot of coding data, I could potentially create some data, get more data from you guys. I'm not into the training business but I want to like take an open source, really good trained model. Um, and now I'll use all the post training or uh, you know, fine tuning or inference layer techniques and then have uh, you know, your intelligence APIs incorporated in my product and then get that feedback cycle going on and improve further. Like is that a good way to think about it?

Speaker A: Yeah, absolutely. I think Our overall mission is to enable a much broader class of users to be able to uh, exercise control over the AI across the full stack as opposed to just being relegated to build products on top of AI on top of like rigid APIs. Right. So that's uh, that's what we enable like think about. So right now. Yeah, sorry, go ahead. No, right now if you think about it, there's maybe, I don't know, um, 100 to 1,000 people in the world who you know, have the knowledge to train frontier models. But um, you know, what if we are able to like 10x or 100x or thousandx that number, uh, and that's kind of what we want to achieve, um, in the long run it almost

Speaker B: feels like this is the next version of fine tuning. I think if you just like map down what different types of people are different doing things like There are only 5 to 10 real companies which are pre training. Right. And then rest of them are doing some version of fine tuning that's uh, also a small set of class. Uh, if you think about like broader market like Fortune Thousand, I think 80% might not even be doing fine tuning yet. Uh, right. And then you're creating a new class of techniques on top of fine tuning. Right. Um, how do you think like this versus fine tuning or what is a place for when you do fine tuning versus when you do more inference layer adaptive techniques that you're talking about?

Speaker A: Uh, I guess in the long run we almost don't want the user to also be aware in some sense um, of when to fine tune or why to fine tune. And the system basically does the right thing for you behind the scenes. Right. So it's almost a uh, I guess in some ways a bug in the system that really the developers have to be so exposed to a knot of the nuts and bolts in order to see the behavior of the system to be changed. Uh, of course there will be like incremental milestones that we take in order to get to that long term vision. But yeah, our long term vision is that the system is going to take care of the adaptation for you. Whether it's fine tuning, whether it's through something else, whether it's a combination of the two, that those are all like implementation details in some ways.

Speaker B: One similar question, where are the new techniques coming from? Are like, you know, I think once sort of like the whole inference and optimizing inference started, you know, people shifted towards KB caching, uh, Lora, uh, and quantization and all these techniques and the company started sort of building on top of those techniques. Where, where are the new techniques coming from these days? Like is it mostly the big training labs? Is it you know, uh, open source, uh, we're very getting or obviously internally you might be aggressively looking for what's the next technique that is coming out.

Speaker A: Right, yeah, I mean we are a Frontier Research Lab and I think um, the um, I guess the term that is popular in the industry these days is Neolab. So we are at one of the neolabs, but there's uh, a host of other neolabs. Right. I think a lot of them are innovating in various different spaces. Um, some of them are more specific for, okay, we will solve, um, AI for sciences for example. Uh, so they're innovating in that we are definitely uh, innovating as well. We have our own in house research, uh, teams that are constantly looking for new approaches and um, techniques. Um, uh, so yeah, and obviously there's also the broader research community which is outside of the industry that is constantly innovating especially well. So um, it's a broad enough set of problems that there's plenty of space for research and innovation. Um, um, but we do think that it's a very valuable problem to solve

Speaker B: in the long run. Are there like techniques that maybe you know, the SF research community knows that are not yet popular but you guys are like this thing is the next, like are there any like techniques like that that are, that you see coming up?

Speaker A: Um, I guess it's kind of hard to um. So there's definitely like a lot of

Speaker B: techniques which are that you don't want to believe.

Speaker A: Uh, I think there's definitely like, you know, some, I guess, um, in house techniques within Frontier Research Labs which had not been widely available and with our products, uh, I think partly, you know, having been at some of the Frontier Research Labs, our mission, uh, in some ways to also productize it so that a broader class of users can actually have access to those techniques. Um, but um, yeah, part of our mission is also to just consistently continue to innovate and bring these new techniques um, to the world as well. Um, I would say that uh, are there techniques? Sure, I'm sure there are like unpublished works that many companies consider to be valuable intellectual property. But the space of innovation is so rapid enough that um, I would hope that those um, techniques don't remain limited to only a small set of individuals but are more broadly accessible over time.

Speaker B: One of the things, uh, I found use this phrase of like last 5% reliability gap, um, and one of the things it really resonated with me because in some sense all AI problems are looking like um, the self driving car problem, um, like we are making 80% progress very quickly, right? And then 10 to only 10 to 15% progress in six months. But the last 5% is taking forever. Uh, the last 5% is where like oh, I wish this last 5% change and it changes the BNB. I feel like I see this in coding. I see this in like agents that are doing Excel updates. Um, whenever I'm trying to more these use cases it's like ah, uh, the last 5% is where like it's a breaker actually the last 5% is where all the real value is. Uh, because that tells you whether it goes from demo to like day to day use case to production. Right? Like that's where the value. So talk to me a little bit about that. Um, you know, the last 5%. Did I put the cap thesis?

Speaker A: Yeah, we see this fairly often in enterprises, right? Enterprises would basically want to integrate AI into their products. And uh, in many cases they are used to having that, you know, 99.9% reliability like when they were not using AI, right? And then now suddenly they're like okay, Even getting to 90% is really hard because. Because AI is very stochastic in its nature. I would say a big reason why we have that last 5 to 10% gap is also because a lot of it is uh, very um, dynamic because the context uh, changes over time. The relationships that uh, you need to be aware in order to make an accurate deduction change over time. And more uh, or less like AI is static, right? So it doesn't do well with that dynamic drift in the environment or the data around it. Which is why I would attribute like a lot of that last 5% failure too. But if the AI system itself was more dynamic and could continuously learn, then I think we certainly have an opportunity to BRIDGE that last 5% gap. If the user, if it can learn based Q. It's not that it's not going to make mistakes. Let's say that if make some mistakes and you say, you tell it today that okay, going forward this is not what I want to do because you did it wrong. This is uh, and then it keeps that uh, and it internalizes it and then the next time it actually learns from it, then over time that 5% becomes 1% and the 1% becomes I don't know, 0.1 uh percent. Right, but, and it has to be through systems that continuously learn and evolve as opposed to static systems that just you know, have a fixed set of behavior um, that don't change with the environment.

Speaker B: So I work in storage and one of the things that you know people are surprised that most of the storage systems still use HDDs, not SSDs. And I think it also underlies a point that you know we in all the tech companies spent a lot of time in making those systems so efficient that even though the hardware is not the cutting edge, you could get a lot more efficiency in terms of how to store software, defined storing and like more techniques uh, on software rather than the hardware. I think we are still in AI, we are still dependent because things are so moving rapidly. We, we've not gotten to that stage of optimizing systems on top of uh, hardware layer. Uh, I think now I think you're one of the examples of that shift slowly happening of optimizing systems on top of just raw training. Because still last three years we're getting the benefits as much as on top of training and we've seen only one or two form factor products, primarily chat defined products that came out of it. Uh, but there's a whole set of products that will come out of just to optimize things and that will heal a lot of value. I think you're sort of falling into that next era of products that we will see.

Speaker A: Yeah, absolutely. I think the hardware analogy is actually uh, the storage analogy is really good. Right. And uh, um, it's a sign of maturity in the landscape that we are trying to now focus on um, broader optimization of systems as opposed to just one component of the system. I would say a lot of it is also driven by the fact that there's this big demand supply gap in the industry right now. Right. Um, in the sense that because of all of the agentic workloads the demand for AI inference has gone up by multiple orders of magnitude. But um, on the hardware side there's still a lot of supply constraint which is driving up like the compute prices and the industry. And um, that supply constraint is going to take a while to resolve. Right. It's not something which will go away in the next six months. It's probably you know, optimistically 12 to 24 months. But conservatively it can take up to five years because it starts from like right at the level of energy and uh, acquiring land to build data centers, uh, to um. Yeah. So there's a lot of just fundamental core infrastructure issues that need to be solved in order to bridge that demand supply gap. So um, if you have that demand supply gap, then you need to innovate at the system level to extract as much efficiency as possible with the capacity that you have. So um, that's something which um. Yeah, so the gap is in essence creating an opportunity for a lot of innovation at the systemic efficiency level.

Speaker B: And it's pretty much also reminds like pretty much every new uh, product sort of goes through this cycle in some sense. Like you start with some foundational change

Speaker A: and it's directly correlated hardware.

Speaker B: Uh, like even like for example Apple iPhone. Right. Like initially it was like okay, all this form factor innovation, but then post that uh, it's like relentlessly improving the system. Uh, right. The full stack system system is what led to what we see today.

Speaker A: Yeah, yeah, absolutely. Um, I think the only difference is just uh, how fast things are moving. I uh, think in the previous changes, um, there were still the changes happened over a period of few years. Uh, but now you have breathing space to do some. You can take a step back and

Speaker B: come out like that. I know everything is right now now.

Speaker A: Yeah, exactly. So it's much more relentless. Uh, yeah, uh, I think that's a

Speaker B: good note to end the conversation.

Speaker A: Um, sleep.

Speaker B: Thanks for coming on the show. I think one last question I would say is uh, a couple of years down the line, would we still be 10Xing our GPU cluster for training or do you see that die down? What's your take on what does the next three years look like?

Speaker A: I think we have already seen a uh, really major shift from training to its inference. Right. And the last three years, um, before last, before three years, uh, more than two thirds of the compute was actually going to its training. One third was for inference and the three years it has shifted and now two thirds is going to its inference and one third is going to its training. And that's a sign of maturity. Right? Because it means that people are actually using those models much more. I would consider that trend to accelerate much more. But broadly I also expect COMPUTE to become increasingly decentralized. So um, because um, so the AI is going to become like much more pervasive. It will move to its own devices towards the edge. Um, and um, that again will um, enable um, innovation uh, at every level of the stack. But also um, requires a lot of continuous learning because if you have AI deployed on an edge device, you kind of want, want it to just continuously learn when it is deployed. So in that sense like the gap between, okay, this is training, this is inference will also blur over time.

Speaker B: Um, yeah, yeah, um, Sudip, thanks for coming on the show. Um, I'm looking forward. Um, what you guys will launch next. Great.

Speaker A: Yeah. Thank you for having me.