Reinforcement Fine-Tuning and the Future of Specialized AI Models

Speaker A: Foreign. Welcome to Databrew by Databricks with Danny and Brooke. The series allows us to explore various topics in the data and AI uh community. And whether we're talking about data engineering or data science, we interview subject matter experts to dive deeper into these topics. And while we're enjoying our morning brewing, my name is Denny Lee. I'm a principal developer advocate here at Databricks and one half of Databrew.

Speaker B: And hello, I'm Brooke Wenig, senior director of our machine learning practice and the other half of Databrew. And today we are thrilled to invite Travis Adair, co founder and CTO at Predibase to Databrew. Uh, for those of you who aren't aware of Predibase, it is a platform to help you train the best models, serve them at maximum speed, think about SLMs with the highest quality and fastest throughput. And Travis is also creator of Lorax, an open source platform for serving multi Lora attention heads, as well as Horovote, which I have used many years ago. So thank you for your contributions to the open source community as well, Travis.

Speaker C: Yeah, absolutely. Thank you. Um, it's great to be here. And um, yeah, I don't have my coffee, but I do have my tea.

Speaker B: I'm joining you with the tea as well.

Speaker A: I feel outnumbered right now. Just joking.

Speaker B: So, to kick things off, Travis would, uh, love to hear from you. There's continually new models, new techniques being released. What's the current lay of the land?

Speaker C: Yeah, um, well, definitely, I think it's uh, a really exciting time to be in this space. Whether you're like, working with the proprietary models or with the open source models. I mean, with our company Predibase, we made the decision very early on to bet on open source and on open source being kind of the future. And I think that what we're seeing is that, um, the original kind of bet that was being made by a lot of people is that there might be this convergence where there's this one model to rule them all. Like everyone's using a single provider. I think the good news for companies like us that that's definitely not the direction things seem to be going. We're definitely seeing a much bigger diffusion of different models that are better suited for different use cases. And the landscape is constantly shifting in terms of which companies are putting out the best models. And I think it's actually great for people who, um, are wanting to build applications on top of these systems that things are constantly improving and getting better. And there's no clear horse, uh, that you should be betting on, I think. Just earlier this year we had Deepseek, which was this amazing model release. Uh, very recently we had Quinn 3. Um, we also had Llama 4, which maybe didn't quite meet expectations on some levels. But I think that's kind of the nature of the game is that, um, there's constantly new things coming out for people to try, and some things will resonate and some things won't, and some things will work well for some use cases and better for other use cases. So I think this diversity is actually really good for the industry and for the community. And that's part of what we want to try to help people navigate with our business, is which models work best for you and for your use case and kind of help get them into production.

Speaker A: Awesome. So one of the things, based on, like, what you're seeing, and especially with your background, right. Uh, uh, of building Horvod and talking about the various techniques and you had mentioned, like, that DeepSeek R1 paper, like, there seems to be a lot of techniques for fine tuning, like, including supervised fine tuning, reinforcement fine tuning, um, like the deep seq R1 paper. So I guess the first thing's. Can you tell me a little bit about what is rft? What is reinforcement fine tuning? That seems to be an important construct for a lot of the current wave of what's happening right now.

Speaker C: Yeah, absolutely. So I'd say at a high level, um, kind of the way the taxonomy of these different concepts is evolving is that we kind of have this broad bucket of things that we call post training, which is like, okay, we've trained the foundation model. It's good at a lot of different things. Now we want to tailor it a bit more for specific use cases and specific tasks. How do we do that? And historically, one of the most common techniques was what we call supervised fine tuning, um, which kind of has its roots going back to kind of the data science days of the 2010s, where obviously before that too, that was. There's a big interest in the technique than for taking like a BERT model or something like that. And then you would give it examples of the output that you want it to classify, et cetera. And then you would get it to learn through some form of mimicry or extrapolation, like how to reason about how to arrive at that right answer. So basically learning by example. And I think that this has been kind of a common tool in the toolbox going back to the beginning of the LLM fine tuning era, like when we had the llama model come out, it's like one of the earliest really production usable LLMs. Um, but it has some limitations for sure. And definitely I think a lot of people um, recognize that getting a high quantity of really curated labeled data was a big long pole and getting supervised fine tuning to be effective and ended up being a reason why a lot of people decide not to do it. Instead just see if they could use in context learning or rag based approaches and kind of get all the benefits, benefits that way. Um, separately we've had this other kind of area of reinforcement learning that has been um, popular in the industry and primarily with LLMs. Through this technique called reinforcement learning from human feedback, we are trying to align the model more towards human preferences. So something more subjective, not something that's right or wrong, but more like humans prefer this or our company policy prefers this, et cetera. Um, and that's where you, instead of giving it an objectively correct answer, you'll often say like here are two examples, which one is better? Right? So it's like a very subjective uh, quality. Now that we're talking about this new technique like reinforcement fine tuning, um, in some ways it's a bit of a marriage of those two concepts where you're saying I want to take the techniques of reinforcement learning but apply them to these tasks that are more objective like you would normally do supervised fine tuning for. So that's the term got coined by I think OpenAI back in December of 2024. Uh, the technique has been kind of pioneered a little bit before that when it was called reinforcement learning with verifiable rewards and then definitely was heavily popularized by deep seq. Um, um, when they published their R1 paper and kind of talked about their method called GRPO that they used, which was a specific way of doing reinforcement learning. Um, and so in particular what we found that's really exciting about this technique is that it lets you get a lot of the benefits of something like supervised fine tuning. Um, but on tasks where there might not be a very clear labeled ground truth example of what you're looking for, like in code generation, where you might know if something's good or not, but you might not be able to say this is the absolute perfect function for doing this task. Or reasoning, which is another area where some reasoning can be better than others. But it's hard to say this is the perfect reasoning, which is I think where the deep seq um approach came into play. In also cases where you just don't have a lot of labeled data, we Found that this tends to work quite well in those cases as well. Um, that's a little bit of background on why we started looking into this technique and how it came to be.

Speaker A: I'm curious then, is there before we dive into grpo, because I think either myself or Berko and I have a ton of questions just related to that specific topic. Is there an implication that basically you still need to actually have high quality label data or just you really need that feedback loop in order to take it to the next level? Like even with the reinforcement training, the reality is that combination of the two really takes itself the next level. I'm just curious from your perspective.

Speaker C: Yeah, I definitely think that um, the quality of data is always going to be an important factor. I think it just changes with reinforcement fine tuning. The big challenge with supervised fine tuning was always collecting the labels with reinforcement fine tuning. Now we're saying we don't necessarily need labels. You just need a specific way to assess correctness like ah, an objective function that can be used to say this is a right answer, this is wrong, etc. Um, but that still implies that you have good data. It's just, it doesn't necessarily need to have like a perfect label associated with it. Um, so I still think that there's an element of bad data in bad model out. Like that's not going to change, but hopefully at least the curation of the data is, you know, less labor intensive because you know, you don't have to spend as much time hand labeling things.

Speaker A: Right, right. It's just more like, it's more like providing feedback on uh, it's recording the feedback that's provided instead. So maybe that might be an easier way to get higher quality. I guess the one thing I'm curious about is like, because it can be very subjective at times too. Right. Versus objective. Uh, do we run into this issue where basically too many people providing feedback that actually conflicts with each other?

Speaker C: Yeah, yeah. So um, this is in the case of like, you know, you're trying to steer a model towards um, like some kind of like human preference based, um, direction. And I think that's definitely something that can happen. I guess maybe to take a step back for a sec, like talking a little bit about use cases for a moment. Um, there are a few different use cases that I think this comes up where reinforcement fine tuning can be very useful. Um, one of the earliest ones that we explored and had a lot of success with was uh, code generation. You can imagine natural language to SQL or transpiling, um, from one coding language to another, something like that. We found that tended to work quite well with something like reinforcement fine tuning because one, the task was verifiable in the sense that you could say this code doesn't compile, therefore it's wrong or this code doesn't give me the expected result, therefore it's wrong. Um, but it also didn't require having labeled data because I don't necessarily have the exact uh, SQL command that I want to run for this intent or I don't have the Python code that I want to generate from this Java code, but I can generate it, I can compare the outputs of the two systems and see if they're the same. Um, this ended up working pretty well for a use case like that. Um, additionally I think people have had a lot of success with the math, um, and side of things as well, which is where you know, Deep SEQ had a lot of success. And then I think that um, there's also this bucket of these more subjective tasks um, that are, that's very interesting. Like you could maybe say maybe not subjective but like um, you know, like ah, agentic systems as well where there might be like multiple steps that need to be taken and then you get like some final outcome at the end which is like, did the system do what it was supposed to do or not? Right. And then you need to kind of back propagate that through the entire reasoning chain to figure out, to update it, improve it. Um, so those are cases where we've seen a lot of success with this technique. Um, I think an area of research that we're very interested in as well though is this idea of learning from feedback. Um, because you do have a lot of situations in the wild where people deploy these systems to an end user, to a customer or something like that, and the customer might be able to give some feedback like good, bad, this was a good interaction or a bad interaction rated on a scale from one to five, et cetera. Um, but they usually won't give it much more detail than that. And so what we're um, looking at doing, and this is in general a direction that the industry is looking at, is can we take that data that comes from a human or a set of humans and build a reward model that captures the essence of the human preferences, um, that can then be used within an RFT process to score the generations from the model. It'd be like models trying out different generations. And then this reward model that acts as a proxy for the human is saying this is good, this is Bad and steering the learning accordingly.

Speaker B: Yeah, the verifiable outputs I think is a very interesting point that you brought up. Uh, um, because like that is one of the critiques of the Genai community is we're just building tools that make programmers more productive. Um, um, but so many of these problems are hard to have verifiable outputs of. Does the code run successfully? And so I'm just curious to get your take on to what's, what's next. Um, so so many of these problem domains are focused on coding and math. You had alluded to agentic systems, but I'm curious to get your take on what's the next big challenge to tackle.

Speaker C: Yeah, it's a good point. I mean definitely think, um, you know, coding has been one of the first like kind of applications that's had a lot of success um, with both RFT and I think LLMs in general like taking on a lot of that workload. Um, I do think that we're starting to see a lot of interest in um, scientific domains being used with this technique as well. Like you know, biology, um, kind of drug discovery, um, chemistry, like various medical fields where there's still like some groundedness and you know, like there's things are right and wrong, like there's measurable ways to say like whether something worked or didn't work etc. Um, and where you can have these domain experts come in and impart their domain knowledge into creating these reward functions that define how you're going to objectively score correctness or incorrectness. Um, I think that's like a big area of investment right now that we're seeing um, uh, a lot of companies explore. In particular, I'd say, um, talking with some folks in the space, this idea of the long tail of applications is something that a lot of people are looking at. For example, modern LLMs are pretty good at generating Python code because it's a very popular language. And so there's a lot of coverage on the Internet and also a lot of people who know how to write it that can help improve the systems. But there's also this long tail of domain specific languages, languages that maybe um, were popular decades ago that still have legacy systems that you want to convert to new systems, um, that don't have as much coverage. And so the models aren't very good at those tasks. And so that's, I think one area that we're seeing a lot of investment in the short term is like can we get um, more coverage on these like long tail um, coding Languages and applications and help get a lot of that uh, stuff modernized and you know, um, get these LLMs to be able to understand those types of tasks as well. And then even outside of coding, I think expanding to other verticals, other domains like the life sciences and so forth. And then I think where there's a big opportunity going beyond is stuff like multi, um, hop rag agentic planning systems and execution systems. I think we're going to see increasingly more systems that are being deployed that are not just one LLM call, but some complicated chain of many LLM calls. And then being able to introspect into that and say like where did it go wrong? Like where did it make a mistake that needs to be corrected? That's I think going to be a big area of investment for RFT in the future as well.

Speaker B: You actually prompted my next question which was about reasoning models since you've been Talking about deep seq1 which naturally brings up chain of thought. How does RFT help in situations like that where you have these very complicated multi step workflows?

Speaker C: Yeah, yeah. So I think um, I think chain of thought is like a really interesting example of where RFT is particularly useful because um, if you imagine like wanting to fine tune a model for a specific task and having some labeled data, I think we can kind of like understand how you would go about collecting that data. But if you want to try to teach a model to reason better, it's like how would you do that with supervised fine tuning? Right? Because it implies that you have some way of getting perfect or nearly perfect reasoning data, um, to learn from um, that you'd want to say here's an example of perfect reasoning for getting the correct answer. Go learn that. Um, certainly we do see people do this through distillation where they'll take a um, strong reasoning model like R1 and they'll say okay, let's have R1 go and do the task and spit out its reasoning trace and then we'll fine tune a model on R1's reasoning, um, like using supervised fine tuning and like maybe you'll do something called rejection sampling as well where you'll like discard all the generations that R1 had where it produced an incorrect answer. Right. Um, and that works pretty well. But like the one challenge you have there is like you're never really going to do better than R1 with this approach. Right? Because you're learning directly from R1s reasoning and R1's answers. And so if your goal is to ultimately like push the frontier of like what is possible or like improve accuracy even further. Like normally the way that we would do this with supervised fine tuning is like collect more label data. But you can't really do that with this distillation technique because you're inherently bound by, you know, this upper limit being the performance of the teacher model that you're distilling. So this is where I think RFT and is particularly powerful is that it's a way that you can go beyond what is possible with the current models today, um, for these kinds of tasks, like reasoning based tasks, where there's no clean way to just collect more ground truth data. Um, and so this ultimately also is what was um, really insightful from the Deep SEQ paper is they call out what they call this, um, aha, uh, moment where they ran this GRPO process, this RFT process on their model and it suddenly went from not really understanding how to do the task particularly well to having this moment of clarity where it's like, oh, I just found a really clever way to think about this problem that's different from the way I thought about previously. And then it was able to solve the answer. And that was not something that came from ground truth. It was purely something that came from this self discovery, this self play of, uh, reinforcement learning.

Speaker A: Oh, this is really cool. Okay, but I got to ask the first question because, uh, what's GRPO like? You know, you've mentioned the Deep SEQ paper, the aha moment, which is awesome. But yeah, what is this GRPO thing?

Speaker C: Yeah, yeah. So GRPO is pretty simple. It's just one of many different reinforcement learning methods that exist. It was one that the Deep SEQ authors pioneered for themselves. And there is like maybe um, one or two things that are noteworthy about it though. Um, one is that compared to what was the standard reinforcement learning algorithm before grpo, which was ppo? Um, PPO was um, notorious for being a bit unstable in the sense that you had not just one model that you were optimizing, like the policy model, but you also had another model that you were jointly optimizing called the value model. And this whole system, because you have multiple models being optimized at the same time, ends up being both very memory and resource intensive. So you have lots of stuff occupying GPU memory that needs to be accounted for, but then also very unstable because if one of the models isn't learning correctly, that causes the other one to learn poorly. So you have to get everything to just be configured just right to get to work. So it often ends up being Like a very big, um, hyper parameter search space problem where you have to find the exact right configuration to get everything to work. And so um, with grpo, they recognized that they could kind of do away with a lot of this complexity if instead of having this separate value model that needed to be trained, you could instead just have these reward functions that you define where some human, for example, could encode their domain knowledge and say, here's how you score a correct answer. Write a function that assigns numerical scores to an answer based on certain properties and then based on some sampling of different generations, um, in a group, which is where the name GRPO comes from, is the group Relative preference optimization, policy optimization, then you have a, um, way to kind of assess like, all right, so among these eight different candidate solutions, which ones are relatively better than the others? And then based on that, you're able to compute a loss that then determines how you should update the model. So it's all based on like the relative performance of different sample generations within a single, for, uh, a single input.

Speaker A: So the single input basically generates multiple potential outputs. Those multiple potential outputs. Then it's like almost going back to consensus algorithms all over again, where you just basically use a different type of weighting in essence. But then because you're using that relative output as you're denoting, then you're able to use that as the basis for going ahead and um, judging what's the correct answer and that it's not nearly as memory intensive. Does that imply that it's actually more network intensive though, or in terms of running multiple, um, outputs at the same time and then, uh, having some consensus across all of them?

Speaker C: Yeah, so there's definitely an element that um, you do have more computation potentially because you're generating in different generations instead of just one generation at a time. Um, so there is that element. Um, but yeah, you are saving a lot on memory as a result. Um, but yeah, to your point, it is very similar to a consensus algorithm or something where you take all those generations, you score them all, you compute the relative differences, which is what we call the advantage, and then that advantage translates directly into the loss. So that you can say if this particular output, um, had correct formatting but incorrect syntax or something, then it gets a score of. And this other one that got everything right, got a score of 0.1. And this other one that did all these other things incorrect, got negative one. Then it can learn directionally just from that. Like, okay, like here's, you know, I should try to encourage myself to do things that look more like the answers that got high scores and do things that look less like the answers that got low scores.

Speaker A: Okay, well, then. Oh, sorry. Go ahead, Brooke. My apologies.

Speaker B: I think we're about to ask the same thing, Danny.

Speaker A: Yeah, yeah, you're probably right. Yeah, yeah. Because. Yeah, this is super exciting.

Speaker B: Yeah, sorry, of what you were talking about, of how do you get it to produce higher scores? How do you avoid it from producing lower scores? How do you prevent reward hacking?

Speaker C: Yeah, so that's. That's a really good question. Um, so one of the big problems that does come up here in this process is you write these reward functions and you, um, you know, try to encode, like, your understanding of, like, what good looks like and what bad looks like. Um, but the models can be kind of clever in their own ways. They can come up with ways of, uh, getting a good score on the reward function, um, that might not actually, um, achieve the objective that you have as the programmer. A good example, and we call this reward hacking. A good example of this is, um, we're trying to do a code translation problem. Um, where we were going from this, um, framework called Pytorch to a framework called, um, Trident. It's like a Cuda like language. And what we were doing is we were writing, um, these reward functions that said, okay, does the code execute successfully? Does it, um, give you the same answer as the original Pytorch code? So kind of comparing them directly, and then if so, let's give it a score of one. So one meaning good. And we saw that the model was learning, it was getting better and better. And so, uh, great, this is clearly working. We're really happy with this. And then we looked at the generations, like, what the model was actually generating, and we saw that it would write this function that was supposed to be in this Triton dialect. And what it was doing is it was just calling the Pytorch function and then returning that and not actually doing anything in Triton at all. And so we're like, okay. So it's basically figured out that it can game the system by just, um, writing the original Pytorch that was in the prompt to begin with. So it didn't have to do anything clever. It was just like, grab that, return this. And then, um, it was clearly an example where the model figured out a way to kind of game the system and work around what we had originally instructed it to do. So we went back to the drawing board, we took our reward function that we'd ran, and we added this extra Check to basically mock out um, the function that it uh, that's supposed to, the Triton function that's supposed to actually do the work and compare the output before and after mocking and said if the final uh, answer is the same both with and without stubbing out or mocking out this function, then that's evidence of reward hacking because the function should be doing something, it should be materially changing the output in some way. Um, then by introducing that check, we were able to get it to stop doing that behavior and start doing what we were expecting it to do. But it was definitely evidence that when you are modeling these systems, when you're building these kinds of reward functions and so forth, you need to be cognizant of the edge cases, what the model could do to kind of circumvent your logic and then also kind of be on the lookout during training to just be looking at the outputs, make sure they look good and then be willing to kind of go back and update things if it's starting to go off in this kind of reward hacking direction.

Speaker B: So this overall seems like a challenging system to design. Like how do you define the rewards? How do you guard against reward hacking? How much effort is it to use RFT to do post training on a model?

Speaker C: Yeah, well I'd say that um, it's definitely very task dependent, like how complex it ends up being. I think we have some pretty good recipes that we've come up with for a lot of code generation tasks at this point. Um, for other tasks like maybe summarization named entity recognition, classification, they're pretty good standard templates you can use for a lot of things. Um, but it's definitely the case that once you start getting the things that are more um, domain specific, like some of these agentic planning systems and so forth, it does end up taking a bit of effort to get the solution right. And so I think what we found is a good kind of general framework is that you want to try to start with something relatively simple, like start with like the easier examples. So this is kind of like a technique, they broadly call it curriculum learning. It's like you have all these different data points that you want to learn, like start with the ones that are easiest and like try to throw that at the system and then try to get it to learn some of the easier um, attributes as well. Right. It's like what are some things that are clearly right and wrong, like formatting of the correct answer, like that's something you can get it to learn uh, relatively easily. And then I Think what you'll typically want to do is once you start getting to some of these more complicated things like correctness, like for code generation, like is the code correct? Um, what you would typically do is try to break this down into not just like a binary, like 0 or 1, yes or no. But can you give it partial credit? And if you can find ways to kind of give it partial credit, then what you can do is you can start to steer it in the right direction, even if it's not all the way there yet. And this is really important because this RFT process is what you call an on policy learning algorithm, which means that it's learning from its own outputs. And so what that means is that if you're trying to get it to learn something it has no idea how to do at all, it's never going to get there right away because the model doesn't know how to do calculus or something. You're not going to get it to learn calculus by just telling it, nope, you're wrong, you didn't get it right. Like you're going to have to teach it small baby steps along the way to say, okay, this is the correct notation versus this is not the correct notation. Slowly start to get it there and it'll eventually learn what you want to learn. But you have to give it that partial credit in order to prevent it from just getting stuck and unable to make any forward progress.

Speaker A: Wow. So we're basically as per curriculum, learning going back and teaching our models like a full elementary, middle school, high school course, college course load basically. But yeah, I like the analogy quite a bit, um, saying that, um, all right, you've built all this. This is really cool. Can you tell me a little bit about now that you have this model, how do you optimize performance of the system for things like model serving? I'm just curious.

Speaker C: Yeah, so, um, this is definitely something that we spent a lot of time thinking about for, for us as well as we do both post training as well as serving optimizations. Um, and so one thing that um, we do is we also um, not just I guess do the kind of quality based fine tuning, but we also do some post training optimizations to help with speed as well. And so one thing that uh, we've invested very heavily and that we've found has been very successful is when you have these models that generate a lot of tokens, um, you can use techniques like something called speculative decoding to speed up the inference process so that instead of normally they'll like maybe to just Kind of briefly give a primer for folks who maybe aren't as familiar about LLM inference and how it works. Normally you're generating one token at a time. So you generate a token, you feed it back in the input, you generate the next token. That's the process. And that tends to be why LLM inference is slower than most other forms of inference, because they have to generate one token at a time. And so what you can do with speculative decoding is if you can generate say four tokens at a time, um, and then just validate that they're consistent with what the model would otherwise have generated, then you can speed up the whole inference process by 2-3x in some cases depending on how expensive that verification ends up being. Um, and that's like, you know, a big win when you're you know, like trying to do this as fast as possible in production. So what we often do with um, you know, when we do these engagements is, you know, we'll do some fine tuning for quality initially to get the model to output what we want and then we'll do some post training optimization for this speculative decoding as well and other kind of quantization methods and things like that in order to get the latency where the user needs it to be and then ultimately deploy that into production. So it's usually a multi step process like get the quality where you want and then get the performance where you want for production.

Speaker B: And so I know you created Lorax. How does Lorax help with serving these models as well too?

Speaker C: Yeah, absolutely. So the kind of background behind Lorax as a project is that um, when we were starting to look at doing fine tuning and serving, particularly like this parameter, efficient fine tuning and serving, where you have these small adapters that um, Maybe only constitute 1 to 10% of the total parameters in the model that are being modified. Um, the solutions that were available at the time, um, circa like 2023 were that you would have to either merge those weights back into the base model and then serve that, or just serve this model plus these adapter weights together in a single deployment and then do that for every single adapter that you have. Um, and so what we realized was that you could serve um, many of these adapters on a single GPU because the amount of memory that they take is so small relative to the, the model itself, um, and then effectively swap them in and out um, at inference time. And this ended up being very valuable for RFT as well, where you have, you can imagine constantly new Checkpoints that are being written. Like every time you update the model you write a new checkpoint. That checkpoint needs to get loaded into the inference server to generate the new candidate generations and then unload it at the end. Um, so this idea of swapping loras, swapping these adapters, um, was what ultimately led to the creation of Lorax, which was an inference server designed for serving many fine tuned adapters simultaneously. Um, and ah, since then I think we've obviously improved it quite a lot so we don't even have really the notion of swapping at all anymore. Everything can be done in a single batch. We have custom CUDA kernels that um, have been written in collaboration with from some folks at University, uh, of Washington who did the Flash infer project, wrote the original version of these kernels, um, and they let you process um, multiple different Loras in a single batch together without having to pay any kind of switching costs. Effectively what we've been able to do is scale this up to now the point where you can have hundreds of different fine tuned models on a single deployment without any noticeable degradation, latency or throughput. Um, and so that ends up being very valuable for customers that want to have um, many different, very narrow task specific um, RFT models or fine tuned models. So what you can imagine ends up happening in practice is instead of trying to fine tune one model that does all these different things really well, you can fine tune models that are very specialized to specific things like function calling or handling guardrails, uh, or something like that. Um, and then as long as you have a way to route to the appropriate fine tuned model at query time, you can have this very highly performant, very efficient model to service that request. That's all running on one gpu. So your costs are very manageable as well. Um, that's ultimately a better production system at a cheaper price point.

Speaker A: This is awesome. Then I think that naturally we're talking about RFTs, we've been talking about Lorax. Pretty cool. How does this fit into Predibase, your company? You've got a cool company, um, you've been building these cool systems. Love to know better, a little bit better about how this all fits together.

Speaker C: Yeah, absolutely. So with Predibase, um, we're a platform that helps people to customize and Productionize open source LLMs. That's like our mission. And so as part of that we have kind of two broad offerings in our product. One is the inference capabilities and the other is the kind of post training capabilities like fine tuning and reinforcement. Fine tuning, etc. Um, and ultimately we see this as being ah, you know, like why a product that does these two things in particular is because we see that there's really like this um, flywheel that exists between these two things, right? It's not just like that. You know, you want to post train and you want to serve and these are like two distinct steps, but rather you want to put something in production and then over time you want it to get better, right? You want it to improve. And so to do that you need to collect more data about how the system is performing. Like what are the requests and responses that are coming in, what's it getting right, what's it getting wrong, what's the human feedback on those responses? And then as that data comes in, you want to curate it and refine it and then help turn that into the next round of fine tuning that you'll do to improve the model further. So we already see today that our customers use our post training speculative decoding technique turbolora, to automatically make their models faster and faster over time. So as you get more and more request and response data, you can fine tune these speculators to make the model faster. And then we're also starting to do more with fine tuning for quality as well over time where you collect more and more of this data, you ah, refine it, you curate it, you build eval sets, you do a B tests and you can get the model to be better and better in production, um, on a quality dimension as well.

Speaker B: So how can listeners get started with Predibase?

Speaker C: Yeah, so we have a free trial that's available for folks who want to um, check it out. So we give um, about $25 in credits to every new person who signs up to the platform. Um, and with that they can fine tune models, deploy them, kind of get a sense for how uh, their production workloads will work on our platform. Um, and from there um, there's a full developer tier available to help you kind of deploy your own models, whether that's any open source model that you want, like Qin3llama, et cetera, they're all available for fine tuning and serving. Um, and then from there, as well as you get more serious and you want to go to production, we also have support available for uh, helping people do dedicated deployments with reserve capacity and helping to build customized deployment solutions for the exact demand, uh, the exact number of requests per second and latency targets that they want to hit. And then once that goes to production, we um, help them with uh, providing SLAs on that and making sure that their Deployments are working 24 7. Um, but getting started is very easy. Just sign up with, uh, your Google account, et cetera, and you can get started without even having to put any credit card down today.

Speaker B: That's amazing. Awesome. Um, so we just want to say thank you so much for joining us today on Data Booth. This was a fantastic conversation about the lay of the land of current models on RFT Lorax. And once again, thank you for all of your contributions to the open source community both with Horvode and with Lorax. Um, and we hope everybody gets a chance to check out Predibase. As Travis had mentioned, no credit card required, so no blockers to getting started today.

Speaker C: Awesome. Thank you both for having me.