Challenges and Solutions in Managing Code Security for ML Developers - ML 175

Speaker A: Stop wasting your nights on a mattress that doesn't get you experience. The most comfortable mattress in the world. The Sleep number Smart bed. At the touch of a button, you can personalize your comfort. Choose firmer or softer, adjust cooler to warmer, and right now save up to $2,500 during our massive Labor Day event. Hurry into your local sleep no store today because we have your number. When you're a maintenance engineer in a beverage manufacturing plant, you keep production lines moving and quality on track because there is no room for slowdowns. With Grainger's vast selection of high quality motors, sensors, belts and hard to find parts, you can get what you need fast and all in one place. So nothing gets in the way of getting the job done. Call 1-800-GRAINGER click granger.com or just stop by Granger for the ones who get it done.

Speaker B: Welcome back to another episode of Adventures in Machine Learning. I'm one of your hosts, Michael Burke and I do data engineering and machine learning at Databricks and I'm joined by my extremely well dressed co host Ben Wilson.

Speaker C: Uh, I work on open source stuff at Databricks.

Speaker B: Today we're going to be talking about a real world case study and it's something that I'm currently working on for an unnamed customer. And we're basically just going to riff about this use case. It's a very popular one, it's a very powerful one. And Ben is also doing some feature developments that are adjacent to this as well.

Speaker C: Yes.

Speaker B: So the, the use case today is building an internal code oriented chat assistant. So what that looks like is I'm a data scientist, I type in my question, how do I pull this data? How do I build a model, how do I do this type of analysis, how do I access this one S3 bucket that's all the way over there. And the chat assistant will have context not just about code, but about your organization's code. And then it will give you concise answers that will hopefully make you more productive. So that's the setup. What we're going to be specifically focusing on today is not rag or retrieval, augmented generation, but fine tuning. So all these big LLMs, they've been trained with millions of dollars worth of compute. And what we're looking to do is modify those weights a little bit so that they know a bit more about your data while still having understanding of the English language, python, best practices, etc. Sound good to you Ben?

Speaker C: Mhm.

Speaker B: Cool. So let's start it off with what I think is potentially the most Important piece, which is evaluation. Evaluation is the thing that will allow you to iterate in a stable manner and know whether the things you're trying are good, bad. Okay, so Ben, how would you think about evaluation?

Speaker C: Safety first. Um, so when you're talking about code, it's amazing how much damage that stuff can do if you run something that you don't intend. It's all about that level of security. And there's multiple layers of security associated here. Um, but stepping back from that, the first thing, when we're talking about eval of anything that's coming out of Genai, uh, you need to be able to have a metric that is actually valid, uh, that is custom designed for the problem that you're trying to get this thing to do. So you start asking yourself like, well, if this thing is supposed to generate code, what do I want to evaluate its quality based on? Is it important that it's creating code that is legible, something that a human can read and grok pretty quickly? I, uh, would say that Most commercial grade LLMs out there are exceptional at that. Whether you're talking about right before we were recording, we went over and played around with 401 preview a little bit about doing exactly this. And when it's evaluating code and potentially giving an example of something that you're asking for, like, hey, rewrite my code for me. It's really good at it. It's not perfect, but it's pretty good. Claude 3.5 also fantastic at cogeneration. But when you were talking about evaluating something for fine tuning, would want to be be able to rate this as how good is it syntactically, how good is it for a human to read? Maybe there's performance considerations like, hey, uh, do a big O analysis of this and optimize for minimization of memory footprint or CPU complexity. It could be some things that you need to evaluate there, but a lot of that stuff can be determined by just evaluating the text of the code itself without having to run it. And you can get metrics associated with that. Use an LLM M as a judge and say, rate my code for these things. And then there's another level of this where you can use linters and use something like Rough, for instance, written in Rust. It's super fast. It can parse ludicrous amounts of Python code and adhere to a set of predefined stochastic rules like, hey, I have this rule set to make sure that my code is formatted in a certain way that I'm not doing Things that are against Python development practices. You can enable, disable different rules, uh, create your own if you want, and you can execute that in a very safe manner because it's just parsing text and giving you an evaluation of what it finds. And you can even have it auto fix it. You wouldn't want that for this case. You just say check this code and report out all of the violations that it has or whether it has violations, yes or no. And that can be a metric. But when we move away from that into does this code actually run? Does it do what I intend it to do? That's when we're in the security world.

Speaker B: Yeah. So giving a little bit more context and I would love to deep dive into that in just a sec, but we're pretty comfortable with basically leveraging either LLM as a judge or deterministic metrics with or without Python execution to see if the code is like stylistically reasonable, if it's concise. Most, um, LLMs, especially open source ones, are fine at doing this and analyzing code. So what we built is we built an ML flow evaluate framework where we basically create a bunch of metrics. Some of them are custom, some of them are Pre built by MLflow. We run via the MLflow evaluate command and then we get a bunch of summary statistics of each metric. So percentiles, etc. And this works great because then we can create custom functions. The question to you first is, let's say I'm a data scientist, let's take a very specific use case. Let me think of one. Let's say we're looking to do a cohort analysis on a, I don't know, frickin shopping cart or something like that. Like we have a bunch of products, we want to see how users are buying them. We're going to do a cohort analysis on the users that buy X number of products. So it's effectively like clustering or something like that. Let's say there are a few APIs that are specific to the organization and those APIs need to be called correctly to get the data. And, and then we also have a pre built sort of notebook example of how we would expect this should be done. So the organization has all this source code. How should we go about executing all of this very complex read logic, um, and evaluating that the output is actually correct in a safe way.

Speaker C: So if I were the one designing that system M, I wouldn't use like a base LLM to do that because there's so many things that can go wrong and it's so hard to teach that, to use those properly and to have enough examples for it to understand. So I would instead not look at the LLM, um, fine tuning as the means of doing that. But I'd rather go into like an agent framework where if I have an API that I know that I call this service and I'm expecting to get a response, I can make that deterministic. And the way that I make that deterministic is write the interface and register it as a tool. So I know I can test that. I can say, here's the example input for this. Here's my, like, my arguments that I would pass to this and I would get back a deterministic response based on what the conditions are of what I'm submitting to that. And then the agent would just have that tool available. So I would write a very thorough description of what the function is for, what the tool is for, and what the arguments are and what the arguments can and can't be for its interface.

Speaker B: So effectively you'd mock it?

Speaker C: I would just would push that to the location that it needs to be in, which is a tool like it's a function call that you expect consistent behavior of retrieving this data or contacting this service and not leave it up to an LLM to have to learn that.

Speaker B: Sorry, I'm not following. So LLM, um, generates a piece of code that says, read data X, read data Y. Do cohort analysis. How do we execute that?

Speaker C: I mean, if I'm building a system that is a human interface to ask a question about this analysis, that agent would do that one thing like, hey, here's your tool that you can use use to get the data for cohort A. Here's the tool you can use to get the data from cohort B. It's probably the same tool with just different arguments passed in. And then the analysis function could be another tool that takes in those data sets and does whatever analysis we want to do and returns a result. So now I have this agent that can answer a question. Hey, I'm interested in, like, I want to know the difference between people that are buying milk and people that are buying chocolate milk. How many people? And what's our sales for them? Uh, and forecasted sales for the next six months. Should we buy more white milk or chocolate milk?

Speaker B: Yeah, the use case is slightly different. We're not actually going to answer the question. It's not like a function calling paradigm. It's an assistant paradigm where you say, how do I do this? And it generates text and then the user will go execute that text or Python code.

Speaker C: Uh, that's complicated man, like really complicated.

Speaker B: Um, it's like effectively chatgpt without running the execution where you say how do I do a group buy in pandas? It'll send you the text and then you now go use that in your environment. How do we validate that that generated Python code will produce the correct result without all that agentic tool function stuff?

Speaker C: Gotcha. Okay. Um, yeah, this is like science fiction level stuff. Don't think it's possible with today's technology to uh, get something like that that would be fine tuned to be really good. You would need to maybe OpenAI can do it with a fine tuning data set that is fairly large that has all of these different things that you want to do. Um, it'll be expensive uh, to do that and I would not be able to tell you what that quality would be, uh, because this would be very complicated.

Speaker B: Um, well, even just on the evaluation side, let's say I have three lines of Python code, I want to execute that as if I was a user and evaluate that the output of that code is roughly correct.

Speaker C: Right? Yeah. So if, if we're setting aside the sci fi aspect of this and how possibly impossible this is to get something acceptably good, just executing Python code in a safe manner, it's super dangerous and there's a number of reasons why you, um, unless you know exactly the data that it's been trained on and also have some sort of guardrails put into it to say here are the libraries that I do not want you to ever use and here's some more instruction sets of operations that I am not permitting you to generate code that uses these things. So I would have a, basically a disallow list of libraries within Python that I know can cause some very serious problems. Like maybe I don't want it phoning home, uh, like I don't want it connecting to the Internet for when it executes code. So I'm not going to use any like httpx, I'm not going to use urlib, I'm not going to use uh, I don't even want it to parse, you know, uris. So I don't want it to use URL encode, I don't want it to use um, like any of the base networking libraries that are part of core Python and I would instruct it that as part of like the system prompt during training. And then I would probably also say don't use shellutil uh, that's operating level system stuff. Uh, if you've never like really messed around in a virtual environment with what you can do with Python with pseudo root access, I highly recommend everybody try that out. Take a Friday afternoon sometime and see if you can completely break a computer with Python. Uh, promise you you can, um, you can do crazy stuff like start deleting user directories. And yes, that is recoverable in most operating systems. There are ways to get that data back, uh, because it's usually a soft delete. But there are creative ways that you can turn your computer into either the world's most expensive space heater or the world's most expensive paperweight. Um, that's why you should always do this in a virtual environment. Uh, because vm, um, who cares what happens to it, Just start another one. Um, but I recommend people try that out, like, see, can I obliterate this compute environment and make it just unusable? And the answer is yes, you can. Um, don't do it to your own file system though, because if you do some of these techniques, like I'm going to RM minus RF on this folder that contains your docs folder or something, and then you can write code that will, uh, just fill that entire space up that you just deleted with just random bytecode.

Speaker A: Stop wasting your nights on a mattress that doesn't get you experience the most comfortable mattress in the world. The sleep number smart bed. At the touch of a button, you can personalize your comfort. Choose firmer or softer, adjust cooler to warmer, and right now save up to $2,500 during our massive Labor Day event. Hurry into your local sleep number store today because we have your number, and

Speaker C: then delete all of that again and then fill it up again with a bunch of random bytecode. You do that enough cycles, the operating system can't recover your underlying data because it's been overwritten and the blocks are all now corrupted. So uh, yeah, you can do stuff like that, or you can go into your disk recovery, uh, directory on your operating system, Mac or Windows or whatever, just wipe all that stuff. So you can't safely restore your computer, uh, from a backup.

Speaker B: Sounds like you've done some of this before.

Speaker C: Maybe, uh, you can do crazy stuff like, um, generate, like generate so many files on an operating system that you actually crash the operating system. Like it can no longer index files. So like, yeah, you're still looking at a screen that's on, but your CPU is pegged to 100%. And the. If you open up any file browser in Your operating system, it doesn't take, do anything because it's sitting there trying to index the, you know, 37 trillion like files that contain hello world, uh, that you've just written to your hard drive. So now your, your actual, you know, actual index tree, uh, for your computer is just filled it like it's a bad day, Bad day for Michael. If you, uh, do something like that. I don't recommend you do that.

Speaker B: Um, I was going to, but now I won't.

Speaker C: Modern, uh, computers handle that a little bit better. I think the last time I did that was on like, uh, I think it was on like a Windows 7 computer, uh, that we weren't using anymore at a job I was working at many years ago. And I was like, hey, I bet I can just brick this computer. And guys I was working with are like, well, yeah, you can like, you know, delete the root directory. I'm like, no, no, no. You want to see something cool? So I just wrote like a recursion algorithm that's just like using like file writing of a very small file just billions of times. And we were just sitting there looking at the actual task monitor of it and everybody's like, dude, what is going on? I'm like, file indexing, like, this is insane. Like we can't even do anything. I'm like, yeah, we gotta fdisk this thing in, uh, order to recover it. Reinstall Windows, they're like, whoa, yeah, don't do that at home. Uh, but yeah, code is potentially dangerous. In most base libraries you have the ability to do crazy stuff. Usually people who are on that computer, they're working in a job, they know like, I'm going to get fired if I start doing stupid stuff like this. Or it's your own computer that you paid money for and you don't want to, you know, wreck that. So you're not going to do stuff like that. But we're not talking about code that you're writing. We're talking about code some gen AI is generating and you're just blindly executing it. You have no idea what that thing is going to come up with.

Speaker B: So how do you do this safely?

Speaker C: You don't. I mean, the safest thing is don't run it and have a human in the loop. That doesn't scale. And that's not a good answer. There are safe ways to run arbitrary code. And if you ever check out like any sort of like devcon, like the black hat hackers, you know, symposiums that they do, they have one like, uh, they have a bunch of them. But you can check out YouTube videos if you want, or any of the white hat hackers stuff like presentations that people are like, oh, this is how we do penetration testing and this is how we evaluate these things. And some of the stuff that they're running is dangerous stuff. And if you're in cyber forensics, you're going through and trying to debug what somebody wrote in order to exploit the system. You don't ever want to run that on your computer. Like, oh, this looks like maybe this is a virus. Uh, I should install this at my computer and see how it works. You create a sandbox environment. And that sandbox environment is just like we were saying with these file system access stuff. If it's in a virtual machine that's running on your computer, who cares what it does? Create that vm. Um, do not allow it to have outside access to any network. Disable all ports on the thing other than the fact that it's reading from a file system where you're submitting that code, or you have this one port that's input only, like during the instruction set and you're saying, execute this, but you want to make sure that you're executing it only within that environment. And it has no access to your file system. It has its own file system. And if it blows up or does something super dangerous, kill it, delete it,

Speaker B: kill it with fire.

Speaker C: And it's fine. Just delete everything associated with it.

Speaker B: And you're cool going one level deeper because I will implement this in the next, like four days. What exactly would you do? Let's say we're using MLflow evaluate. We get a string of code. Let's say it's compilable, ast parsed everything and it will run, but it has a drop table command and then it also has a vacuum command to remove all of the underlying data for that table. Let's say I have admin access to the workspace.

Speaker C: Uh, don't do any of that stuff. Uh, if you're getting into the world of SQL execution, there are libraries out there that can clean and parse dangerous commands. You could write your own parser. I don't recommend that. It's really complicated. Um, but at a bare minimum, there should be an exclusion list that you're evaluating whether this is safe to run or not. And you can just put in, uh, for like dummy parsing. Like, is there a. Is the word drop in here in either uppercase or lowercase? Is there any command that's in this SQL execution that will execute something other than a query.

Speaker B: I don't think whitelisting or even blacklisting would work because we would be leveraging APIs that are custom built by this organization. So we don't always know what is happening within those APIs.

Speaker C: So the APIs have delete statements and truncation statements.

Speaker B: We don't know. I would not be comfortable like creating this.

Speaker C: Okay, just don't build it.

Speaker B: But is there a way to spin up um, a Docker container or something where if we execute the most horrible evil python code, it'll just kill the container?

Speaker C: Python? Yes, SQL. No you don't. If you're calling some code that then has access to a table somewhere, you have no way of knowing what that it's in an external system. Yeah, but what you would have to do is replicate that system internally within the container. So you would basically take a snapshot of that data in whatever mechanism that it's stored in and have your local code connect to that local instance. That's a lot of work though. Like a shocking amount of work for something like so silly.

Speaker B: Could you give it read only permissions?

Speaker C: Uh, you could set access control for this execution that this is a read only user.

Speaker B: Sure.

Speaker C: But you'd still have to be kind of careful about exposing a production data source to something that you don't trust its code execution. Do you have robust injection attack protection where somebody writes some crazy SQL, they close the statement on the select and then they write like some sort of transact statement.

Speaker B: But the whole point of this read only permissions, how could that. So to be crystal clear, what I was like my cursory proposal as of like yesterday, so only spent like a tiny bit of time thinking about it would be spin up a execution environment via a uh, service principle that has read only access to dev. Obviously not prod.

Speaker C: Theoretically. I don't know, I'm not a security professional.

Speaker B: Okay.

Speaker C: I wouldn't do it, uh, just because there's a redesign that would make more sense. So if you're is not abstracting the actual command that you're going to be running. So if you're calling some library that has some code in it that could potentially do dangerous things, then you would have to evaluate what the impact of that is of like, okay, this could call this admin API and even though I think I have everything configured and this is safe and secure, and then you run it and then you find out later on like 10 minutes later like we just dropped the entire catalog because we forgot to do this one like make sure that it didn't access this one thing. You have no idea.

Speaker B: Sorry to harp on this, but isn't the solution environment separation? Like if I go to my personal laptop and run drop catalog from like on any workspace, nothing will happen because it's my personal laptop. I don't have the workspace URL, I don't have the token and there's no way that the LM will guess both of those things.

Speaker A: Stop wasting your nights on a mattress that doesn't get you experience the most comfortable mattress in the world. The sleep number smart bed. At the touch of a button, you can personalize your comfort. Choose firmer or softer, adjust cooler to warmer and right now save up to $2,500 to during our massive Labor Day event. Hurry into your local sleep number store today because we have your number.

Speaker C: So it's guessing that it's that you writing connect to. Like let's say we, you set up this, this unit catalog and there's data in it in this particular, you know, table within a schema. You connect to it, you set it up to be I only have read access to this and I send a drop command and then you know, truncate my catalog or something. That system processing your instruction will reject that and say you don't have this permission to do this. You're not an admin. However, we're not talking about that. We're talking about you're calling a Python API that you don't control. You don't know like uh, what that thing has. The first instruction in that Python command would be use service principle with admin access. Somebody could have written that. You have no idea. That's why I would never do that.

Speaker B: So could I spin up. Oh, clicked. All right, cool. So in summary, if basically you're using a uh, organization built API, you don't know what it does. Let's say you have do cohort analysis and it needs to create a new location. And by creating that new location it wipes everything at the specified location. The default is the root and it automatically authenticates via uh, an admin service principle. I see how now that is potentially a problem.

Speaker C: Yeah, like their, their statement for creating a table could be create or update table at this location.

Speaker B: Mhm.

Speaker C: That's going to wipe the data.

Speaker B: Yeah. Oops.

Speaker C: So if you don't know what the API is, it's super unsafe to do that. Cool. Now the other thing is like okay, we're talking about just Python code execution of core Python libraries and that Function, we're just arbitrary executing Python code. Maybe it's a function that adds two numbers together is the intention. But then we're like, well, we don't want to create like 10,000 different functions to do all this stupid stuff that's basically re implementing like base Python operations, right? Instead we're going to do the cool thing which all the cool kids want to do, which is, hey, Genai, just generate my code for me and I'll run it for you. That's where we're talking about. We need safety of an execution environment because we're not talking about interfacing with external systems where we could potentially create a lot of havoc. We're talking about the easy thing to get going right away is just call exec on that. Like, hey, I have, uh, Python code in a string that's properly formatted, ast node parsed, it's all structured correctly. And then I just call eval in my main process. That's where we're like, we start talking about shockingly unpredictable things could potentially happen, right? The best case for garbage code is it just throws an exception. You get a stack trace and you're like, yeah, it sucks, but nothing really happened. The worst case scenario is, oh, it ran, it ran really well. And now the entire core contents of every user's directory, uh, because you're running it as an admin in the main process or running it as your user, which you may be an administrator, while doing like exploratory dev work here and now you can see every bit of data that everybody has in all of their notebooks in the execution environment, just printed the standard out and logged and you're like, I hope nobody had keys in their notebooks.

Speaker B: Of course not. No one ever has keys in their notebooks.

Speaker C: Yeah, and now you have to start a security incident and involve a bunch of people at your company saying, hey, we now need to do an audit because I did something dumb and we may have leaked every key that people have accidentally put into their notebooks and rotate them all before we have a data breach or something. Who knows where that data might have gone and scary bad times will happen. It's a safe environment that has no ability to talk to the outside world. And you just get an evaluation of what the return result is. And then you can compare like, hey, uh, it. The instruction for this thing to generate code was to do this task and it returns something that proves that it did that task. But if that task is, hey, go pull some data and, you know, tell

Speaker B: me, manipulate it yeah, yeah.

Speaker C: That's when you're in scary territory. I would change it more to hey, generate the SQL query that I need to do this and ensure that you're running only a select statement and then it goes and fetches the data within a secure sandbox environment.

Speaker B: Mhm.

Speaker C: And make sure that

Speaker A: stop wasting your nights on a mattress that doesn't get you experience the most comfortable mattress in the world world the sleep number smart bed. At the touch of a button, you can personalize your comfort. Choose firmer or softer, adjust cooler to warmer, and right now save up to $2,500 during our massive Labor Day event. Hurry into your local sleep number store today because we have your number.

Speaker C: Yeah, this data is the right data that was requested.

Speaker B: Crystal clear. All right. I think this sort of wraps the evaluation topic. Um, anything else before we move on to the actual fine tuning?

Speaker C: I mean it's a big topic. We could talk for days, but yeah, uh, this is the stuff that I'm currently working through uh, for a project like not just the concept, like we just explained why this is important to have a safe execution environment. And that informs why I'm working on what I'm working on. But what I'm concerned with now is how to do that in a very safe way. And there's a lot of options that you have um, for safe execution. And it's really, it's a difference between. Ease of development and uh, for the person building that solution versus performance and security. Uh, and then like just how complex is this? And you can go on one extreme end is the most secure and potentially least performant which is creating a brand new VM for each execution. It's going to be terribly slow, like really slow, but it's super secure provided uh, you define the characteristics of that virtual machine and set it up in a way that your users can't mutate. So it's like a protected configuration and when they want to run it, they get the safe and secure way to do it. It's going to take seconds to do a single function execution. And then the fast way is, and still pretty secure is doing container services like hey, I have a base container and I need to execute a number of these function calls. Well that container, provided that you set it up correctly, doesn't have access to the file system or networking on that computer. And it's like an isolated sandbox, uh, and it's quick and easy kind of to build. And then there's a bridge between the two of those that gets the very fast performance but also the security of virtual machines where it's like micro VM processes and that's how cloud providers do it. So when you interface with like AWS Lambda, they're running micro VMs on Kubernetes and they can spin up what is a micro vm. Huh, huh?

Speaker B: What's a micro vm?

Speaker C: Imagine snapshotting the state of a base image for a virtual machine. And you can replicate it, but sanitize the state of it. So it takes a little while to spin up the first instance of it. That's why there's a cold start issue with Lambda. But once that VM is ready and active and held in, uh, it's like something that you can just submit code to. Each individual request that comes in gets a replication of that container to execute and you can replicate like 100,000 if you want and it'll scale to however big you need it to go. They have a queuing system that's you know, handling request volumes that's coming in and making sure that you're reusing. Like when a VM is done and it's ready for destruction, you can potentially reuse that by wiping state and then it's ready for another request immediately. That's how Lamb AWS Lambda. If you put a lot of volume at that thing, it is shocking how fast it is. I got it mind, uh, blowing how many requests that that service can actually handle. Like AWS did a fantastic job building that and all the other cloud providers have something similar. You can, Azure, gcp, they all have services like that.

Speaker B: Got it.

Speaker C: That's the Firecracker API. Yeah, it's an open source package but if you look through the setup for it and the configuration, you're like yes, a little bit more complicated than uh, Docker. Just a little heard, but it's cool.

Speaker B: Okay, cool. In summary, we're evaluating our stuff. Um, the way that we're going to be doing this is maybe not with code execution, tbd, uh, um, but just doing sort of heuristics on is the code looking clean, does it compile, does it pass the linter checks, that type of stuff. And let's say we have a suite of, let's say 10 metrics, um, that properly evaluate our code. The next problem that I wanted to run through is, well, what we have here and again we're not going to be using retrieval augmented generation, we're just going to be fine tuning, um, we have a bunch of internal code bases. So repo one, repo two, repo three, they all have Custom logic and let's say it's relatively good Python code. So there's doc strings, there's a few markdown examples, there's readmes, um, and the APIs are fairly like, well written. How would you go about fine tuning? Would you go create a data set manually? Would you leverage a synthetic data generation mechanism? How would you actually create the data to fine tune?

Speaker C: I would, I would do the same thing that OpenAI did, which is use GitHub public repos. So it's a volume problem, right? You want to have enough flexibility and referential, uh, intelligence of how to solve problems in an abstract way. And you need a bunch of examples of ways that people have done that. If you were to look at the entire like extract of public GitHub, uh, repos, you're probably going to find so much duplicated code. When you're talking like going down into the base level of what this function or method or class does, there's certain things that when you're building frameworks or you're building applications, everybody has to go and do it. You need some way of handling, you know, conversion of JSON to dictionaries in Python, or you need some way of showing proper ways to use regular expressions in Python. You need loads and loads of examples of these common things, like how do I make it a rest request? And there's probably millions upon millions of examples in every major language there on GitHub. In those examples, there's probably going to

Speaker B: be,

Speaker C: let's say 60% of it is hot garbage. Like somebody doesn't know what they're doing, they create their own repo, they write some code and no other human has ever looked at it, probably for the best. And if you were to look at it, you'd be like, what are they doing here that's so unsafe? Or why would they think this is performant? Like, there's no way this would, like this function is meant to iterate over a collection of data. And the way they wrote it, it probably works great if I just have 10 elements in that, uh, but what if I pass a million elements into that? It's going to throw a recursion error or it's just going to blow up the stack because of all of these objects that they're creating unnecessarily. So there's lots of ways to write really, really bad code that's going to cause lots and lots of problems. So the way to get around that issue is either manually curate everything and have experts go and just review it all and Fix everything and make sure that your training data set for fine tuning is the most immaculate examples and enough examples of this concept so that can kind of grok what the heck is going on. The other way is throw everything at it and it'll theoretically figure it out. And that's, that's base model tuning, like foundation model tuning. And as you said at the top of the episode, that's large amounts of human capital, large amounts of, you know, just pure capital that needs to go into that. It's super expensive to train these things. And if you don't have the resources that Microsoft, uh, and by proxy, OpenAI or Google or Meta, if you don't have data centers like that, you're not getting in this game. It's so ludicrously expensive. We're talking tens and tens and tens of millions of dollars to train these things. So fine tuning is an option, but you gotta be careful about what you're teaching it. On fine tuning, uh, are they actually the best development practices? Do you have some, you know, distinguished engineer at your company who has gone through every single example that's going to be done and making sure that they adhere to the standards of best practices? Do you have an entire team of people at that level who are going through and nitpicking every little thing and be like that's not efficient or that's not the way to do this. Here's a better approach and these are the standards that we want to set for our code base. If you work at a software development company that's all about building high quality code, you probably already have that data set because everything's peer reviewed, everything's gone through optimizations, maybe snapshot the state of a repo right after a major refactoring has happened by multiple teams and people have signed off saying this is a fantastic state of our repo right now. Or you just go and custom select certain things where people have agreed on. This is an awesome implementation. We can read it. It's performance, it's maintainable, it's testable. There's separation of concerns here. We don't have this bloated method that does way too many things, just like clean code. That's what you would want to use for fine tuning.

Speaker B: Um, we can't do the manual effort because we're lazy.

Speaker C: Hope for the best.

Speaker B: The subsequent question is we were looking to generate a synthetic data set. We're not going to use the GPT models because that's against their license. Um, but we would use an open source model to essentially take the context of the API specs and build solutions based on examples, DOC, strings, etc. What are your thoughts on that approach?

Speaker C: Which open source model?

Speaker B: Llama.

Speaker C: Probably like the largest Llama model is your best bet.

Speaker B: Yeah, whatever.

Speaker C: Like somewhat good at code. Uh, it's probably going to struggle. Uh, there's a lot of any of these commercial grade really great code generating LLMs that are out there. Uh, it's not, you're not just talking to like a single model. You're talking to an agentic framework that has loads of sophisticated logic that is around everything that's operating in the back end. You know, you talked to 401. You start seeing its chain of thought in real time as it's through.

Speaker B: Yeah, that was super cool.

Speaker C: That's an agent. It's a very, very, very complex agent that's generating new tasks that it needs to do based on how it's analyzing a problem and then it's telling you like what it's doing right now. So 4 uh, O doesn't do that. 4.0 is just when you're talking to multiple models that have all been fine tuned to do different tasks. It's like, oh, this is a question about code, I'm going to go talk to my Python expert here. Or this is a code about Java code, I'm going to go talk to my Java expert here. And then you'll get an answer that's probably optimized for that particular use case. So that's the complexity you're going up against when you're talking about, well, we'll just use an open source model. That's just one model. It's not, you know, this, this fleet of sophisticated models that experts uh, have sat there and like fine tuned the hell out of to do certain tasks really, really well. So it's a big mountain to climb with a small team and I do not imagine you're going to be successful at it.

Speaker B: Challenge accepted.

Speaker C: Yeah, it's like a cool research project I think, to like see. It's like a hackathon project, like how good could we make this? But then the question is, is that good enough for people to use this in production?

Speaker B: That is the question.

Speaker C: Or is this going to be so buggy that people are going to be like, yeah, it's cool but it sucks and can you make it better? And you come to a point where it's like a law of diminishing returns, of how much effort you need to put in in order to get the architecture that you're dealing with to compete against what people expect. And the expectation is a bar set by OpenAI and anthropic right now. And that's some of the finest minds in this industry that are tackling these problems. They've been working on it for years and that's uh, the only thing they work on. And they have very large budgets to work with.

Speaker B: That makes a lot of sense. I think that we 100% can get some sort of customization down via fine tuning. The question is, will it be good enough? And I think that'll just come down to the synthetic data set or the manually curated data set that we're fine tuning on.

Speaker C: Yeah. So how I would tackle this project, and by the way, we've done stuff like this as hackathons internally in engineering to uh, try to do this exact thing. And you handle it like a hackathon. You're like, here, uh, are my principles of behavior that I'm expecting from this block of time that I'm going to be given to this. So like, hey, I've got one week to see what I can come up with. And you're setting yourself hurdles that you need to jump across at certain days. Like, okay, after day two, I should have my first iteration ready to go. The code does not look nice. It's like hacktastic, man. Just like make it work and you get it so that it can execute and you're evaluating the responses manually. You're not going to like evaluating with an LLM as a judge because you don't trust any of it. You want to see what code is it generating and then look like, read through the code and say, did it actually learn anything? Is this good? Does it understand the context of this environment that it's providing suggestions for? And you'll be able to give thumbs up, thumbs down pretty quickly so that after that two day mark you can either pop smoke and get the hell out or, or be like, I think I'm onto something here. Now I need to go to the next phase, over the next 48 hours of giving it, like identifying what's wrong in the responses that it gave the first time and making sure that you have examples or enough examples of the right way to do that. And then kick off another fine tuning. And then maybe you have some metrics that you've written that can do some sort of automated human augmentation to the evaluation. But then you still have to read through every response and get an expert who knows what the hell they're doing to look through this code. And be like, yeah, that's good. Or uh, uh, this is broken, man. Nah, we can't run this after that point when you have a go, no go of saying, hey, it's 85% pretty good. Then you go into the whole, maybe we should automate running this code and see if it works. Because now we need to do, we need to evaluate. Because those first evaluation stages, you're testing like 50 examples or something, something that a human can read through and not get exhausted. And then when you're at the point where you're like, I think we're onto something, we have some ideas of how to make this better. Now you have economy of scale. Or it's like, okay, we need to actually automate evaluation of this and get some sort of metric because it's just too much work for people to do. Like, hey, we're going to evaluate 100,000 questions. Like, no human's going to be able to do that. So now we talk about MLflow evaluate with Genai as a judge and then safe execution of Python code. And yeah, that's not something that I would do in a week though. That's like I did my hackathon and I've learned all these things throughout that process that now informs a product design that I can do. If I'm going to make this into a product and propose it and say like, here's all the pros and cons of this approach and here's the ideas, the things that we learned while doing the prototyping of this. Business leaders, should we invest our time and money into this? And if they say yes, this is awesome, let's do it, then it now becomes a full blown project where you're like, okay, we're doing design docs, we're going through proper engineering procedures of building an actual new product and there's stages of that as well. Prepare for private preview before we're going to get some test customers who are okay with some garbage. Uh, every so often we learn from that, improve, fix all the problems until we're ready to generally make this available to the public or to our customers. That process for something that Databricks has now, uh, the GENIE service. Ask a question, it'll answer from your data. Uh, it took over a year of a fairly large engineering team working on that. And that thing is awesome. That team did a fantastic job. But that's a fine tuned service that is very sophisticated and it used a lot of training data to get it as good as it is.

Speaker B: Yeah, so that's part of My question is like for the MLflow docs, you guys use a service called Run LLM, uh, that is a third party that goes and actually scrapes your APIs and then builds a chatbot on top of that. And I was just testing it at, well, we were just testing it out prior to the call to see how it basically does really good API lookups. It's a better search, but it doesn't critique code that well. It doesn't go that much more in depth. So this is a solved problem. They basically want that functionality.

Speaker C: Do you know how that works though?

Speaker B: I have no idea.

Speaker C: That was how, uh, run LLM calls GPT4. Oh, that's the LLM.

Speaker B: But does it.

Speaker C: How does it.

Speaker B: Got it.

Speaker C: That's all.

Speaker B: That's okay. This is a rag problem. That's what I've been saying.

Speaker C: This is a rag problem. So it's using what a really powerful open, like not open source, but a really powerful proprietary model commercial grade can do, which it knows how to write code, it knows how to read code, it knows how to understand the best practices and then it just gets contextual information about our product and it can generate code that you can actually run and it works. It's pretty slick. Like they're. The suggestions that it comes up with generally are pretty top notch.

Speaker B: Yeah, I use it a lot and

Speaker C: they tie that into 401 whenever they're doing that. I think next year it'll be just shockingly good at what it does. But yeah, this is a rag problem in my opinion. A quarter if they want to make it sophisticated and extensible for users to automate super annoying stuff that they got to do. This is an agentic problem where you use that concept of deterministic behavior and have the LLM safely interface with something that is safe because you don't have to. It's not generating code per se. You're controlling what it can run because it's calling a tool that that tool is defined. Underlying that is a function that's executing in a sandbox to safely do something that could potentially be very dangerous.

Speaker B: Crystal clear. So, um, in summary, when you're looking to build these fine tuning use cases, focus on evaluation. Uh, use LM as a judge. Linters deterministic rules. If you're doing code execution, be very, very careful. Slash. Don't do it for data creation. Lots of the input data will be bad. So either manually curate, put everything into a model and rely on a full training run, or try to generate synthetic data. Good, perfect. Cool. Until next time it's been Michael Burke. I'm my co host, Ben Wilson. Have a good day, everyone.

Speaker C: We'll catch you next time.