Chai Discovery's Bitter Lesson: Drug Design Is Another Scaling Problem

Speaker A: One of our big guiding principles is just simplicity. Um, so, like, when you look at a model, like, let's say chi one, I think there are 23 distinct sub modules in chi one. Um, and like, when you're trying to iterate on something like that, it gets really hard because you're like, I kind of need to understand each of these sub modules independently. I need to understand all of their behaviors, their dynamics, and like, that doesn't actually scale that well. Um, so, like, you can start to think, okay, how do I simplify this? How do I identify, like, what's, what's really important? And once you have that, like kind of the whole research process and like, identifying these types of scaling directions, uh, becomes a lot simpler.

Speaker B: We're here with Josh and Matt, two of the co founders of CHAI Discovery. CHAI is engineering molecules with AI. So it's something of a foundation model lab for biology. Sounds like a big idea, but let me just start there. What is the big idea?

Speaker C: Thanks for having us on the show. Um, one of the exciting things we're trying to do at CHAI is to make the drug discovery process look a little bit more like engineering. We've seen all the work happening with LLMs for code generation, for instance, and that works really well because code's a very simple abstraction to get across what you want to do. Biology, uh, doesn't look like that today. It's a lot of trial and error. Um, I think if we go back to, uh, the early days of, uh, just like modern biotech, a lot of the medicines that we have today were kind of discovered quite randomly. It was a bit serendipitous. And what we're trying to do is allow us to industrialize that process a little bit more, uh, and try to come up with the tools that we'd need in order to have abstraction layers. The same way we have that in code and in modern engineering allows us to iterate really quickly and try to bring that, uh, into biology and into drug discovery.

Speaker B: So let me ask you a question on that then. So there's. And tell me if this is a reasonable way to frame it. There's almost this boundary between that which can be engineered and that which needs to be tested in the real world. And it feels like that boundary has been moving over time. Like the, the, the proportion of the drug development process that can be engineered as opposed to trial and errored, seems to be increasing. Is that a reasonable way to think about it? If so, can you talk about, like, what specific developments have pushed that Boundary over time.

Speaker C: I think that's one way to think about it. Um, the way that we often uh, look at this is like the lab is an important part to verify that what you're doing is correct. And actually verification is a very big theme in AI as well. If you can evaluate that your model works, you can verify it, then you can start to hill climb that and you can make progress on it. So I think the lab is a very important part of this. And the question is how do we take drug discovery and make it look a lot more like drug design? So one of the reasons why we call this field drug discovery is we're often looking for a needle in a haystack. We'll screen millions, billions of molecules, try to find one that works. And, and if we can instead put in what is the dream state of the molecule you want and then the model can materialize, that, that's going to be really powerful. So it's not even a question of reducing lab testing. I mean maybe that happens as a result. I actually might even take the opposite side of the coin and we can talk about that where maybe we'd actually do even more lab testing because the ROI will increase the same way. There's more software engineers, uh, or there's more demand for software engineers now that they become more productive. There might be more demand for the lab. Um, but, but I think the key part is how do we just change the paradigm here and how do we make it more design oriented?

Speaker B: Can we talk about what is state of the art today? And then maybe let's take a little trip down memory lane. Five years ago, three years ago, one year ago, like what have been some of the major breakthroughs and like how has state of the art changed over the recent years?

Speaker A: Yeah, I think the field has like, it's really evolved, especially like over the last decade. Um, so it really wasn't until like, uh, you can actually look back, there's like this biannual protein folding competition. So every two years, uh, usually a bunch of academic groups who compete in this protein folding competition. Uh, what would happen is like you kind of hold out these protein targets that no one's ever seen before. They never get deposited publicly and then all these groups compete so you can predict the proteins the best. Uh, and like really it wasn't until 2018 where you started to see this like big step change in performance and then finally again in like 2020, uh, with Alphold 2. But really it was kind of like the advent of deep learning that really kicked off this whole field. Um, so at first what you do, um, these are kind of like bring me back to my original days in my advisor's lab. So we worked on one of the first systems to do this with deep learning. What we would try to do is predict the distances between amino acids and a protein. And then we'd have some render, which then took in these kind of noisy, incomplete looking distances and then emitted some protein from that. Um, and what's pretty remarkable is that with relatively minimal information, like you could build a machine learning system that could actually produce something that really looks like a protein. And for the most part was correct. Like there was still pretty big gaps. It didn't quite have the resolution of what we have today. Almost like with the early image generation models, where it was like a little bit grainy, a little bit blurry, and now it's like, oh my gosh, this is crazy high definition images, like in seconds. Uh, so we kind of like saw that same evolution happen over time. So it started with protein folding, and then I guess once that became, uh, more realizable, uh, there were kind of these other sub problems that people wanted to solve. So now given a protein structure, can I design say like a sequence which might fold to that? And that's getting closer and closer to like the drug design problem. So you're kind of like now thinking, okay, how do I start engineering proteins that like match a certain shape or might perform a specific function? Um, and so people kind of studied that problem independently. And that was kind of like 2021, 2022 time. And then kind of these ideas began to merge together. It was really like the advent of diffusion models where we started to be able to like, okay, I can now generate a protein structure and a sequence kind of simultaneously. Um, and I can start making these like what we'd call prompts more and more realistic. So I can now prompt these models on a target that might have a certain shape and I can say, hey, I want to bind over here. Kind of like add, add more real world constraints on the problem.

Speaker B: What did you guys see in 2024 that made you think that was the right time to start the company?

Speaker C: Yeah, there were a lot of discussions that went into it. I remember one of these early, uh, discussions actually back in Matt's house when we were, um, you know, Matt was, was uh, showing me some of these results on like antibody, antigen, like structure prediction. Matt was just talking about protein folding. Um, but for, for a long time people thought that protein folding for antibodies was just like too hard of a problem. People were like, there was not enough data for antibodies, like in the protein data bank, for instance, to solve this problem. And people, as a result, thought that antibody design was going to be out of reach. A lot of the work that people even did on protein design in the early days with deep learning, it wasn't antibodies. It was, uh, these different class of proteins called mini proteins, um, which are, uh, actually really interesting in their own right, but they're not what most of the drug, uh, industry is looking at. So the holy grail was whether we could design antibody proteins on the computer, especially ones that had all that therapeutic function. And, and our thinking was that if you couldn't predict what an antibody looks like, how are you ever going to design one?

Speaker A: Traditionally, people have thought like, biology is scary. Um, there's so much to know.

Speaker B: I'm terrified of biology.

Speaker A: I am as well, honestly. My background was never biology. I studied pure math and started my PhD in, uh, theoretical computer science. And it was only after my third year that I ended up switching into deep learning. Protein structure prediction, all of this stuff. So it was totally new field to me, seemed insane. But at the end of the day it's much simpler and the problems are much more interconnected, uh, than one might think. So I think people, when they start going into the field, they're like, oh, man, what's an antibody? What's a mini protein? These are all just sequences of amino acids. At the end of the day, these are just different types of prompts for the model. Um, but in the same way where you might have a math problem that goes into ChatGPT, ChatGPT can both answer your math problem and help you with your English homework. Uh, so really we have the same type of thing going on with our models. We just have some way of representing these sequences of amino acids. Then we have a way of designing, predicting those as well. Um, and I think in that lens things become a lot more clear.

Speaker B: And Matt, you mentioned your background a little bit. Josh, um, talk a bit about your background and then more broadly, in order to pull this off, you have a bunch of different disciplines that come together. So can you just talk a bit about your background and that of some of the other core members of the team and how these things all fit together.

Speaker C: Yeah, I've been excited about AI and biology, uh, since I was a kid. So I guess I'm like, Matt, I didn't start with theoretical CS and get into that way, but I went to a high school with a stem cell lab, so I Was just always excited about biology as a kid. And I grew up as a programmer. Um, uh, I really started my career at OpenAI. So it was on the early team there. It was a nonprofit back then, so it was a pretty good time to be there. We did GPT1, GPT2, scaling laws and M. And it was uh, the question was like, if the models can learn to speak English, German, French, why can't they learn to speak DNA and protein? Um, and that was kind of my research agenda since then. I think that sort of intersects with around the time like Matt got into the field as well. And it was a, uh, I think it was a pretty important time as well. Right. Because if you look at the kind of methods that we were bringing in, like there's been a lot of these, you know, changes on the edges if you will. Right. And you know, Matt talked a bit about the history of what's happened in the fields here. Um, but it's all about how do we find the right deep learning architectures with the right compute configuration and the right model architectures to make this happen. What are the right tasks to apply it to? We were talking about how we even knew that 2024 is the right time to start the company. As Matt was saying, these are all different kinds of amino acid sequences and people thought that antibody class of problem was going to be too hard and we started to see the first signs of life that actually this was starting to work. I think a lot of it fueled by um, some of the new architectures we're bringing to the problems where I think it was diffusion models.

Speaker A: Back then the first time anyone was able to generate reasonable looking proteins was uh, the advent of diffusion models. So it was pretty crazy. There were a bunch of generative modeling approaches that would kind of work if you had a bunch of data. So people got these working for images. Um, there are a bunch of tricks to make this better and better along the way. We've definitely borrowed a lot of those ideas in our domain as well. Um, but it was really like once diffusion models came around.

Speaker B: Is there an intuition for why diffusion models work?

Speaker A: Uh, yeah. So diffusion is not a one step process. So like uh, I think kind of up until this point, like the main generative design paradigm was like called variational autoencoders. And uh, in that case you're like saying I want to just like compress my input distribution. So like you have some like proteins, you want to make these look like kind of like fuzzy Gaussian vectors. Uh, that task is just like really hard. And maybe today if we tried super hard, I think we'd crack it. Um, but at the time I think it wasn't really as developed enough to work for our problems. What turned out working really well was just kind of giving the model more time to think and showing it more examples. Like, here's a slightly broken looking protein. How do you make it better? You can kind of break that protein more and more and more. And you can make it look more and more noisy, more and more broken. Uh, and teach the model just quick little shortcuts. All right, here's how I make it slightly better. You can just keep asking over and over again, make it slightly better. Make it slightly better. And kind of like breaking the problem down to that scale worked really, really well for biology.

Speaker B: Yeah, the make it slightly better reminds me of a game that I like to play with my daughters where we have ChatGPT give us a unicorn and then we make it stronger. And we just keep telling it to make it stronger. And by the time we're done we have the strongest unicorn in the world. So about the same, right?

Speaker A: Yeah, uh, it's that easy.

Speaker B: Okay, maybe not the same.

Speaker D: Yeah. So I feel like in this domain you need to assemble kind of like a quadrilingual group of people, like an avengers squad of chemistry people, biologists, AI folks. And so that's a challenge. Um, how have you guys gone about finding people, convincing people to join the team and who are your superstars?

Speaker C: So we've been really pragmatic about this. At shop, you look at the founding team, it was mostly AI researchers. Um, so people who had, uh, worked on either scaling models or getting them to work in this domain. But really with each generation of model, uh, the kind of people we've needed for the next milestone has changed. Uh, or I'd say probably has expanded. Uh, so if you look at Chai 2, that's the point when we started to bring in, um, some of the most incredible antibody engineers and scientists in the world. Um, uh, one of the scientists named Andy Young, uh, actually when we hired him, uh, people asked us if we had pivoted into building a full stack drug pipeline. Cause they're like, you'd be crazy not to do that if like Andy joined your team. Um, but uh, I think Andy has enjoyed running more antibody campaigns in the past couple of months than he's probably run in his whole career. Uh, which is very cool to see. You have folks like Nathan Rollins on the team. Nathan was actually homeschooled and then started College very early on. So you joined like David Baker's lab, who won the Nobel Prize for protein design, like when he was 14. Started uh, his PhD when he was 18 and has so many creative ideas. Uh, as the models started to get better, we needed to build up a product team. Uh, because while the researchers might get the models to a point that uh, they're very powerful, you need to build the right product interfaces that the models are actually useful. And uh, that's where we started to bring on uh, people who have built some of the most exciting products that we know about today. Uh, like our co founder Jack worked at Stripe Munaz, who was one of the top 10 code contributors at Stripe. Neil, uh, who ran his own cybersecurity company before security started to become very important. As we deployed this to our big partners, as we started to scale up, we brought in people uh, who've uh, uh, really done a lot of the GPU hacking if you will, in order to scale up our systems that they don't break when we're running them at scale. We had an email from uh, or a Slack message from one of our hyperscalers the other day, uh, where we had a cluster, I think that's some issue too. And they're like, oh, the GPU's got too hot. I think you guys are running too many. And we're like, isn't that the point? Right? That's probably. I was like, good, we're doing our job at least. Yeah. It's like, how do you convince those people to join? I think again a lot of it comes back to uh, the results and like a clear need. We didn't hire antibody engineers before in an antibody design model. Like, what are those folks going to do? We were even I think worried when we started that trend because for some of the um, next generation formats, the complicated antibodies, like multispecifics, like they didn't even work with CHI two. Uh, so it actually took a couple of weeks, uh, when some of those people showed up before the models could work at a point that they could work on some of these interesting case studies. But uh, fortunately the progress was fast enough to kind of bring that online. So I think we're always like evolving that team and going for, you know, that next milestone. We've got the team very small as a result too. Um, so this way, you know, everyone is a little bit like slightly over capacity I think, which means we have to prioritize. It forces us to work on the things that really matter most.

Speaker A: I think on the research side as well. One of the founding engineers, Kevin Wu, he had the first, uh, I think it was the first protein diffusion model like ever. Uh, and that speaks to Kevin's speed of execution. Like he is a heck of an engineer and like, I think engineering has just always been important since day one. Um, so really even our researchers are all excellent engineers and we really care about building a high quality code base. At the end of the day, we are technically a software company. We're AI researchers, we're protein designers, we're a lot of things. But uh, our deliverable is some piece of software. So we've kept that bar really high while also trying to know, level that with great research talent and people that can actually like push the frontiers of what's possible.

Speaker D: And you've had a number of, it seems, like aha moments in the field. Like AlphaFold was an aha. We can figure out how a protein folds and then the diffusion models, aha. Like we can generate proteins. Um, and it seems like your latest models are a kind of another aha moment where you're not only generating molecules that look like proteins, but they also have therapeutic properties. Like they can bind really tightly. Um, they have really high hit rates. Can you talk sort of about the quality of the molecules that your models are producing now and everything that sort of went in to those models to make them able to do that?

Speaker C: Yeah, if we look at the quality of the molecules that's coming out, if it goes back to one of the theses. When we started the company that uh, we really wanted to focus in on this de novo generation of the molecules. If you look at what a lot of the drug discovery, AI work was at the time, it was about how do I take a molecule and just make it a little bit better? Which we just talked about in a sense. But it was doing it with a lab in the loop, uh, style, right, Where I take some data, trying to make it better that way. And the question, uh, we started with is, well, can we actually just do all that on the computer? Right. Is there a way that maybe there would be enough data or we could collect enough data, um, so that we could just zero shot a molecule that has a lot of these properties. So the first thing we needed to do to get there, um, was to design molecules with really high success rates. When we started the company, the state of the art for anybody design was about like a 0.1% binding rate. So one in a thousand of the molecules you design would actually bind in the lab. Um, so first of all it means you have to screen a lot of molecules to find some good ones. It also means that the gradient you get on your process is actually quite weak as well. So for many targets you won't get any hits. For the ones that you do get some hits, you won't have enough to actually see whether you're having the drug like properties. So we really focused in on how do we just make this process more accurate. We got to uh, with our Chi 2 model, about like a 15% success rate. So uh, now if you screen 1,000 molecules, you're getting 150 back. Now you can start to get some interesting statistics on the properties of the molecules. And that allowed us to iterate on that, build the right evaluations around that in the lab and really try to hill climb that as well. And we're getting to a point now where we can actually bake in a lot of these different properties from the start into this engine. And then maybe if you think about how does this happen or how are we approaching this as well and why do we think it's going to continue improving?

Speaker A: Yeah. So um, honestly I think Josh and I were both surprised at how quickly this worked. Um, when we were originally budgeting this we're like ah, maybe like 20% hit rate, like three or four years. We were like really shooting for like a 1%. We thought one in 100 would be amazing. We're like this is going to be a groundbreaking thing. We had a philosophy on like an approach that we wanted to take and it just like ended up working really well. And that approach is like very similar to what's worked in the rest of machine learning. So people kind of treat biology as this like bespoke problem or like bespoke field. But really it's like the same principles is like self driving LLMs.

Speaker B: Well, same principles. But one of the things we've talked about before is how you manage to find scaling laws. And it's one thing to tokenize a string of text, it's another thing to tokenize biology. Can you say a couple words about uh, without giving away any of the magic, you know, can you just say a couple words about that challenge and how you guys solve that?

Speaker A: Yeah. So uh, one thing that I really liked about Chai is like we're, we're a very bitter lesson pilled company. So like we really believe in like scaling data, scaling models, scaling compute. In order to do that obviously you need to identify scaling laws. Otherwise you're just kind of like wasting time and resources. Um, And I think without giving away too much, one of our big guiding principles is just simplicity. Um, so when you look at a model like let's say chai one, um, so there are, I think there are 23 distinct sub modules in chi one, um, and when you're trying to iterate on something like that it gets really hard because uh, you're like I kind of need to understand each of these sub modules independently. I need to understand all of their behaviors, their dynamics and, and that doesn't actually scale that well. Um, so you can start to think okay, how do I simplify this, how do I identify what's really important? Um, and once you have that the whole research process and identifying these types of scaling directions becomes uh, a lot simpler.

Speaker C: One of the other things too I think it's interesting about this is we take a lot of these lessons from what's worked in the rest of the deep learning space. But as you're pointing out the data itself is different. The models at Chai are completely built from scratch. We're not fine tuning GLM or something like that on some protein data. We're, we build everything from the ground up. I think a lot of the company building process though is taking a philosophy and actually sticking with it and iterating on that and just having some guiding principles. When you're building like a blue sky research company if you will, you know Chai is almost like one of these neolabs, right? Like we have this big AI problem we're going after where as we make progress on it, you know that opens up opportunity for our customers. Um, but if you're going to work on something so open ended that way, you need some principles to guide you. And I think we've done a very good job on like tracking those principles in the company working on it. So a lot of the things that

Speaker B: Matt, can you share them, what are the guiding principles?

Speaker C: So I think simplicity was one of the ones that Matt mentioned. It's this like bitter lesson pillness of like you know, scaling compute and data and models, uh, it's being really rigorous. Uh, that's something that's so important in this space. You can fool yourself so easily in biology. Like the error bars in the wet lab are actually quite large as well. Um, so it's actually a little bit different than uh, if you look at like cogeneration for instance, if you look at SWE bench people were like oh I like this is ah, maybe like a year ago people are like oh I got 16% then 17% then 18%. I mean, in biology, if you're like plus or minus 5% in your lab, like, that's all. That might all be the same. So it actually just means the, uh, bar is really high in terms of the step changes that you want to see with the models. But you also need to be really honest with yourself about whether you're making progress or not. Um, so you could come up with some fancy model that looks like it works well in, like, one or two new tasks. But it's very important to show that that works more generally, uh, if you're actually trying to build a product that can bring the field forward.

Speaker D: Yeah. And that's, I think, pretty interesting because biology is one of those inherently not so verifiable domains. And you guys have been really good at, um, sort of showing your progress to customers, um, and to people like us who know very little about biology. And so can we talk a little bit about the evals and the verifiable part of the model progress? Like, how do you guys know that your models are getting better?

Speaker C: Well, I would say, first of all, that I actually think this is one of the more verifiable domains. It's actually very objective readout. If you look at something even like CodeGen. Right. Maybe as a verifiable task, like, did my code compile? Did it solve these unit tests? But how do you think about the taste? Right. Did I write some really sloppy code that can't be maintained? What does that look like? When we think about designing a molecule in the lab, uh, we can actually be quite specific about many of these properties. Maybe we get a molecule that binds the target, but can I manufacture it? Right. That might be your version of some tech debt, but you can measure that. Uh, and I think those evals, again, actually make this domain more verifiable. Maybe it takes a little bit longer to validate it. It's not like five seconds to get a readout and run a unit test. You might have to spend a couple days, a couple weeks in the lab to get that readout. But at least you can be honest with yourself.

Speaker D: Yeah. When we talk about progress and how good the models are, there's a domain of targets in biology, um, that you can, as you mentioned, sort of address with traditional screening methods. They take a long time. They're very slow and rudimentary. Um, and then there are targets that just aren't addressable with existing methods. They're not druggable for whatever reason. Um, and so where are we in terms of model progress? In terms of working on existing targets and generating molecules faster, that's one end of the spectrum. And then the other end of the spectrum is unlocking novel biology, new targets and things that we couldn't drug before.

Speaker A: This kind of goes back to, uh, our core modeling philosophy. Um, so there have actually been plenty of times where, man, if we add a 24th module, we can actually unlock that new target. Um, and we're like, is that really something that we want to maintain long term? Is this incremental or is this actually a compounding improvement? Will this actually help us generalize to the broader class of this whole class of targets that we really can't hit? And so our philosophy has been, all right, let's just continue to focus, identify your scaling loss. There comes a point where if the model is able to push loss down even further and it has to understand something very, uh, intrinsic about the target that it's operating on. So one example we were talking about the other day, Paul and I, maybe the way that we're looking at certain glycosylation sites on proteins, we're like, oh, we might want to represent them differently or something like this. And we're like, well, even if we didn't represent them, there are certain sequence motifs that will tell the model, there should be a glycosylation site here. In order to drive loss down further, the model should just have to learn that, uh, so there are all these hidden features of targets where if you really believe in scaling laws, you believe the models will get there. These types of targets should just unlock with better models. And of course, you still have to take this very seriously and you still need all the proper validation. Um, you need to really challenge yourself and make sure that this is truly working. But I think our approach has always been with better models, we should be able to unlock a lot of these targets.

Speaker C: One of the other interesting things is if we look at, we talked about, like, the interdisciplinary nature of this, if we look at the different teams at chai, what people will call a hard target is actually different in literally every team. So on the science team, it might be, you know, like a undruggable, uh, GPCR target. No one's gotten something that has, like, you know, modulated that in a functional way. On the ML research team, it'll be something like, oh, there's like, the model just can't fold this thing up. It doesn't know what it looks like. And then on the product team, it might look like, oh, I've got all these, like, you Know, modifications like my glycosylations, and it's a membrane protein. How do I represent that to the user? Um, and actually the fact that it's different for each of these groups, I think is a feature rather than a bug. And it means that, uh, if we want to make, like, broad progress over here, everyone is kind of pushing in parallel on these different ways. And that means that there's very smooth progress that we can make all the time. And there's usually not one bottleneck at chai. It's not like, oh, if we only had that one extra module, things would work. Or if we only, uh, you know, like, try to push this into the product in some way, we can unlock it. Uh, we're trying to build this unified solution because at the end of the day, the goal of the company is to build a computer aided design suite for molecules. It's not to make one or two molecules. It's not to get a pipeline of, like, five interesting therapies that we bring to market. It's to change the way that medicines are discovered. And we're, if we're going to do that, we need to work on all the hard targets, regardless of how you define hard.

Speaker A: Yeah.

Speaker B: Say more about this idea of computer aided design suite for molecules. What does that mean?

Speaker C: So at its core, uh, it goes back to this point about making biology look more like an engineering discipline. So we're not going and fishing something out of, uh, a large library or doing a ton of trial and error. You want to be able to specify upfront the principles that go into designing your molecule and then have an engine that can actually realize that into some molecule that we're going to go and create in the lab. And look, you still might do some iteration on the lab and on your model because maybe your hypothesis was wrong, but we want to speed up is actually again, have that computer aided design suite so that you can go from idea to testable hypothesis very quickly. And if that loop right now takes something like nine months, I don't know, to go and discover your molecule versus if it takes nine weeks or it takes nine days. Each order of magnitude just scales in a very big way, the number of ideas you can really sort through. And I think that's ultimately how the field is going to converge on better medicines. It really comes back to people sometimes talk about, do we care? And you kind of noted on it, Sonali, do we care about speed? Or do we care about the difficulty of the targets? At some point they converge in this way as well. Uh, Because a hard target. If we can iterate through hypotheses a lot faster, then maybe it'll be easier to crack it.

Speaker B: Uh, so if we can maybe detach ourselves from the present reality and go far enough into the future that we're not thinking present forward, we're actually thinking future back. 2035, 2040, 2100, whatever, whatever you want. What does the industry look like? Like, let's imagine that computer aided design suite for molecules has become a standard. Let's imagine a lot of innovation has flowed downstream into some of the wet lab parts of the process. Like, can you paint the picture of what the industry might look like 10 years from now?

Speaker C: Yeah, I think it's going to be a really exciting time. Um, and we can look at this on a couple of angles. So first of all, the quality of the medicines that we develop, uh, will hopefully go up. There's a lot of molecules we put into the clinic today that there is. It's really hard to discover a molecule. You find something that's like 80% of the way there. Maybe I'm just going to advance it anyways to like, hit my timelines. It's probably going to benefit some patients, but then you get beat, you know, a year later, uh, by someone else. And it's really just not the most efficient spend of resources in the industry. You've got the kind of diseases that are just too hard to go after today. People have been trying to drug Alzheimer's forever and unfortunately, um, you know, haven't made as much progress as we'd like. You have things that just aren't economical to go after today. Think about, like, personalized medicines, rare diseases, things where maybe the patient population is going to be smaller. But again, if we can iterate through those ideas faster, if we can launch m something faster, if we can do it in a less expensive way, uh, then, uh, those probably come into reach as well. So I think there are just so many different axes, um, that, uh, we're able to push on. And, and I think that means that the future is really bright.

Speaker A: It used to be like, you either want to be first in class or best in class. Now it's like you want to be last in class because you actually just want to be the final answer. Uh, so I think a lot of what you'll see is just way more intentionality in the types of drugs that you're designing. This drug will be super specific to the disease of interest. It won't have certain interactions, certain negative interactions that a lot of drugs today Do a lot of this stuff is actually like you're able to model most of this, um, computationally, like maybe not today, but, ah, there's definitely a path towards getting there. And I think that's like one of the most exciting things for me.

Speaker B: Very cool. Can we talk about a business model decision that you guys made? Um, because I think a lot of times folks think about Chai and Isomorphic in the same neighborhood. Isomorphic is developing drugs. You guys are enabling the existing industry to develop drugs more efficiently, better, faster, cheaper than they have before. Why did you decide to go down that path versus the Isomorphic path?

Speaker C: Yeah, first of all, I think both of these paths are great and uh, uh, they can create tremendous value. We've always been really excited about building infrastructure for the industry. Our bet when we started to go back to those results in Matt's house a few years ago was, um, that this is how most future drugs are going to be discovered. And if that's the case, somebody needs to go and build that infrastructure to make it happen. I think part of this too is a lot of our founding team, uh, like including Matt and I had worked in companies before where we had built these full stack, uh, drug pipelines. Right. And we were building AI models, we were putting those drugs into the clinic. Again, we're pretty bitter lesson pilled as well. And our thinking was as the models get better, we want to be spending more of the money making better models as opposed to diverting those resources into clinical trials and things like that. And one of the interesting things about our business models, as the models get better, actually wins us the right to continue investing more in them. Right. And you have partnerships with these pharma companies that are paying off today and allow us to continue to invest in it. So it's a much more scalable business in that way. And I just think about the ultimate impact that we can create for the world is a lot larger. We've always wanted to just partner broadly with the ecosystem. It goes back to this point about fooling yourself in biology. If you work on a small number of drug targets, you might come up with the most exquisite molecules, creating a ton of value for the world by doing that. Um, but you might miss the forest for the trees because maybe your model doesn't generalize to the other 500 molecules that people are going to make that year. And if you go and partner with people, you just, you just can't fool yourself. Like you look at our partners, Eli Lilly, Novartis, Orgenx, Pfizer, like These are not companies that are, you know, uh, they, uh, they take this stuff for granted. Like, you have to really deliver on these partnerships for them to take you seriously. And that means it can't just work. Like some of the time, like when we ship models at chai, they really have to work, they have to deliver value to our partners. And it's not like, oh, we made some molecule, it doesn't fully work. We're going to have our chemists clean it up a little bit. So it's almost a harder business to pull off. I think that's one of the reasons why you haven't seen it pursued many times. If you work on a drug pipeline again, there might be ways to fix things up later. There will be the proof of what happens in the clinic, of course. Um, but, uh, when you have this partnering based model, you have to be really rigorous about your models and your models have to work really well because otherwise those partners are not going to come easily. So it's made our life harder, I think, in many ways, but I think it's also the more rewarding path if we can get it to work well.

Speaker B: And what have you learned? I mean, you know, you're out of the lab, so to speak, and in the real world, you know, delivering real value for actual pharma companies. What have you learned as you started to work with these partners in terms of any surprises, in terms of how their needs might have differed from what you expected or how their level of sophistication around this might be different than what you expected? Have you any surprises or any learnings from working with these partners?

Speaker C: Yeah, so when we went into these partnerships, I think a lot of people told us that like pharma doesn't know how to use AI. These are not, it's not like a tech forward industry and things like that. And to be honest, that hasn't really been our experience. I think that these are again, they're very rigorous partners, very rigorous customers. Right. And they're going to test every one of our claims right. Before they start to deploy these things. But when they see the data, they go all in. Right. Because pharma is an innovation industry. Um, I think it's interesting just to think about even the whole economics of the pharma industry. If you build a product in pharma, right, like, like a drug, right. You only have exclusivity on that drug, uh, for a certain period of time. Right. And you have to continue to reinvest. Eli Lilly is a trillion dollar pharma company. Right now, if they don't get more blockbuster drugs, they will not be a trillion dollar pharma company forever. Um, and I think that forces these companies to really be on their game of adopting new technologies and deploying them and trying to stay ahead. Pharma is a very competitive arena, so a lot of people trying to bring these drugs to patients, by the way, is great for patients, but it means that you have to be on top of your game here. Uh, and I think that means that, um, once you're through that door and your models are working, we've seen an upscale this adoption very quickly and people thinking about how to use the models in incredibly creative ways.

Speaker A: I really like the incentive structure that it creates as well, like taking more of the partnership model. As our models get better, we get better results to our customers and so on. Um, so I think that's just a nice side effect for us as well. CHAI has been incredibly focused. Um, and really as the models get better, you can kind of iterate on a better model with better data and so on. So a lot of what we see and a lot of what our partners are using the model for, like generating binders, antibodies, so on. We're also, uh, doing a lot of dog fooding in house. We have a whole science team that's using the models and just trying to understand what they can do. And a lot of that comes back to like, okay, now that like the model, now that we've unlocked use case X, like what type of new data can we generate? How can we make the model better that way? Um, and really like that's been the long term vision of Chai is like we never thought of like we're going to build this one model that's going to solve all of this. Like, we know that they're just like, uh, in other fields. Like they're going to be multiple iterations of this model. And as you get better and better models, like you get this flywheel effect, I guess.

Speaker B: Um, well, and you mentioned what data can we generate? Can we touch on data for a minute? You guys can't exactly just go scrape the Internet and have all the data you need to build your models. Where does the data come from? How does it kind of compound over time? Can you just say a word about data as an input to your models?

Speaker A: Yeah, so I uh, think the primary source of data or the gold standard source of data is this protein database. Um, so this is like, it's actually just legit lab scientists who since 1970 have just been depositing crystal structures of like proteins and other molecules over time. Uh, and like really without that like structure prediction and design wouldn't have been a thing. Um, so there are these. So that's like one source of structural data. When I started in the field I really took like a structure build approach. I was like super interested in predicting structure. Like how do you think about a machine learning model that can output 3D coordinates? Like that's, that's not, doesn't look like an LLM, it doesn't look like an image model. This is really in its own class. Josh interestingly was taking like the exact opposite approach. Uh, so he, he was like one of the original authors on esm and that was like really like a seminal work and understanding how to apply language models to protein sequences. Uh, what's really cool about that work is if you can train a language model to understand protein sequences, what ends up happening is it ends up kind of like representing the 3D structure internally. And there's like a really interesting reason why this happens. So like in order to predict like missing amino um, acids, like same way that you'd predict like next word in a sentence, I want to predict next amino acid in a protein. In order to do that effectively you really need to understand, okay, what does amino acids immediate microenvironment look like? Uh, because that kind of tells you, okay, what are the compatible amino acids with everything surrounding it. And in order to do that, well, you need to understand the protein's 3D shape. Um, so I was taking this really structured based approach and Josh was even more bitter lesson pilled than me. He's like, we're just going to look at the sequences and this is just going to emerge. Um, so yeah, the two main sources of this data, again like protein database for structures and then these massive, massive, maybe even like order of like trillions of token sequence databases. Um, and what you can do once you have really good models is just run them on the sequence databases to get new structures out. Um, so again like you have this compounding effect. As your models get better, they get more and more accurate at predicting these structures. And uh, then you have more and more training data for the next series of models.

Speaker C: Sometimes people ask us at CHAI about these days, at chai, like which paradigm are you actually going after? I think one of the things I love about our team is that it's actually just like neither. We're very pragmatic, like we want to solve the problem, we don't really care. Is it a sequence approach? Is It a structure approach. In practice, it's going to end up being like both. Of course, Um, I think you'd be surprised, but there might even be more biological sequence tokens on the Internet than English language tokens. Uh, now a lot of that data is not that useful. It might not be redundant, might be very noisy, but there's a lot of data out there. There's a lot of art in bringing it together. I think also the exciting thing is as the models have gotten to a point now where we can design things in the lab, right, to new targets. For instance, we can actually use the models to generate data as well. So there's a lot of exhaust from all the experiments that we're doing at chai, which also helped to make the models even better. So it's a similar kind of takeoff that we saw with LLMs. When I was at OpenAI, we worked on reinforcement, uh, learning of a GPT1 architecture. It did not work because the models weren't good enough. Uh, but once the base model got good enough, then you could start to do those kinds of experiments. I think there's a similar analogy that's starting to happen in our world now as well, where the models have reached a point where there's actually a renewed interest in data. And how do we actually bring the models into the loop on making that happen? I think that creates another really interesting cycle on compounding improvement of the models.

Speaker D: Yeah, we're talking a bit about compounding improvement in the models. And you said something about how pharma as an industry is an incredibly competitive landscape. Um, and it's interesting because I think there's been a renowned interest in using AI and using, um, ML to generate proteins and molecules. And so your arena has actually become quite competitive. Um, and you guys have obviously done an amazing job at staying at the frontier. It's been the year of deployment for you. You've locked up a number of pharma partnerships that are making your models better. But how do you guys think about the competitive landscape, um, and staying at the front of the frontier?

Speaker C: Well, first of all, I think it goes back to looking at the results and not fooling yourself and being rigorous. So one of the reasons why we do a lot of evaluation of the models, we mainly do it just to hill climb in the models itself. I think if you look at many of the capabilities that we've brought in have kind of been like, first in the field, you look at our Chai 2 model, right? Like getting to success, uh, rate of the models that you didn't have to do large library screening anymore to see results, um, a couple of months later showing how we could bring in a lot of these developability properties we've talked about before, like the manufacturability of the molecules. So in many cases we are pushing the model forward and trying to see these capabilities emerge and then we try to quickly like lock those in. Like, how do we uh, how do we make those capabilities like even more pronounced that they become production ready and we can like ship them to our customers? I think one of the, one of the things I like to tell the team is that uh, you know, it's not like we are head to head like with other model providers or something like that, where all of us I think are working against nature. Like nature's actually been a pretty good baseline. People have done drug discovery a certain way for a long time. And you know, Matt talked about how like, maybe you don't want to add like module 24 to the chai model, but people have added like module 240 to like the existing wet lab protocols and they have been like tuned quite considerably. Um, one of the scientists on our team, Andy Young, uh, was uh, uh, he was one of the first people working on yeast display at MIT, actually like two decades ago. And he's got 20 years experience and like Pfizer and Genentech, like really honing in these methods, has a drug approval to his name and antibodies. And I think you look at someone like that and like, you know, and he knows how to make a good antibody with existing tools. And uh, that is actually the bar that we need to clear now of course, I think the ceiling in AI is going to be a lot higher than what we've managed to do before. Otherwise what would be the point of, uh, of doing this? We didn't start the company, uh, just to make, you know, a ten times faster mouse. Right. We started this to make breakthrough medicines that weren't possible before. Um, but ultimately that is the bar that we needed to clear in order to get adoption. I think we hit that inflection point a couple months ago. That's why you've seen a lot of these big pharma announcements. Uh, but now we just need to continue to hone in on like making these things even better.

Speaker B: For somebody who's listening, who thinks, wow, this sounds pretty cool, I wonder what it would be like to work at chai. What is the best thing about working at Chai and what is the worst thing about working at Chai?

Speaker A: Um, yeah, I can Speak to some of this. I'll think of this on the fly. The worst thing. So I think the best thing is just how actually mission driven everybody is. Everyone is so dedicated to what they're doing. I've worked at other companies. Uh, the closest I've ever seen to this is maybe some of the guys in my pen PhD lab. Um, but like, everyone is just like, incredibly, incredibly motivated. We all work really hard. There's like an obvious shared goal. Um, and I think that's really rare to see. And I think this goes back to just like the focus that we've had since the beginning. And like, we've always had, like, kind of a clear philosophy, a clear plan on how things are going to get there, how things are going to get better. And really, like, everyone I try is very bought into this. It's pretty amazing just to see, like, the amount of dedication that everyone's putting in. Least favorite thing about Chai, Uh, not directly on top of Dandelion Chocolate, maybe?

Speaker C: No. Maybe the next office.

Speaker A: Yeah. Uh, actually, so probably the least favorite part now, uh, is just like, uh, I guess it's getting things to work at scale. Um, and really, I didn't even know what that meant. Actually. When we started Chai, we had 128 GPUs. And I was like, this is like the most scale possible for a lab. This is crazy. Uh, I came from my PhD group where there were four of us sharing eight GPUs, and I was like, I just felt GPU rich. It was crazy. Um, so now at Chai, we have a lot more infrastructure to maintain. We have a lot more compute resources. Luckily, GPUs are parallelizable. Um, but that also kind of brings up its own set of problems. So just like, how do you keep a cluster healthy over time? How do you get that large training run to. How do you keep that running for months on end? Um, and even when it does die, how do you automatically resume these things? How do you keep all of your communication down? How do you optimize the models and make the best use of the resources that you have? So I think these are a lot of problems that they continuously pop up. They're good problems to have, but I think they're also really difficult to solve. Um, yeah, and I'm just excited to work on this.

Speaker B: Yeah, I remember one of the CEOs that, uh, we've worked with a couple times, Frank Slootman had this line about, you either have the pain of failure or the pain of growth. You'd Much rather have the pain of growth.

Speaker A: Yeah, that's exactly right.

Speaker C: I think my favorite part is, is probably the results. Uh, uh, and that, that sounds a bit cliche, but there's nothing, it's, it's working right? And just like knowing that you're, you know, uh, I think many of us in the company, right, like we've been working in AI for a long time, right? There's all this experience you've built up and, and to know that you're applying it to something that really matters, like just even I think just take Matt and, and, and I like we've been working on this problem for like 10 years, right? And a couple of years ago I'm looking at some like, yeah, we're writing some cool papers. Like everyone is like celebrating this. Are we actually making the world better? Is this actually going to impact some patients? And I think now the answer is actually yes. We have reached the point where this is going to make a big difference in the world. And every time you get one of these breakthrough results, anytime there's a new feature on the product that makes uh, lives of our customers easier, whenever there's new lab results, uh, coming back, uh, from the science team, it's just always so uh, honestly exhilarating to realize this is actually going to change the world in a pretty profound way. I think that's also then uh, maybe comes to the least favorite side, right? Like, you know, we're running the company and it's like we have real partners that are relying on this and like things have to work right? And you ship a new model generation, how do you make sure that there's no bugs in that? How do you make sure you don't have regressions? Right? This is no longer just like a, again the blue sky research problem of like, oh, we got some cool results and we move on. Uh, we've had to have really high priorities on like, you know, having production level code bases. As the team grows, how do we make sure that the code base is in a state that more people can contribute to this? So um, something that uh, one of our other co founders, Jack likes to say is that if you want to move fast in the long term, you sometimes have to just move a little bit slower in the short term and make sure that you are building something. Uh, again that goes back to that compounding idea. It goes back to we don't add module 25 to make the next thing work. Uh, so sometimes uh, you're so excited to get to the next result and you just Want to jump into it, but, uh, we have real partners, some of the biggest companies in the world that are now relying on us. Uh, and it's important that we, we realize that we take that responsibility to heart. Uh, and we make sure we're building systems that, you know, continue to work.

Speaker D: I have a burning question. Why are you guys called Chai Discovery, Chemistry and AI.

Speaker C: But we love Chai Chi as well. There's a lot of Chai themed stuff in the office.

Speaker D: That's right.

Speaker B: That's a good question. I didn't know that either.

Speaker A: Josh is the visionary. That was all him.

Speaker B: That is a very user friendly name.

Speaker C: Yeah. We also wanted to name that. Like, like biotech companies have such complicated names. We wanted something that's going to be much simpler. We're trying to make this whole thing simpler. Right, yeah. Um, so, so we need a simple name to go along with it.

Speaker B: Awesome. What are you guys most excited about in the next six to 12 months?

Speaker C: I think for me it's just the deployments, uh, that are happening. So we've announced a couple of these partnerships and I'm really excited, uh, just to hear about the results that uh, our partners are bringing online. It goes back to this point of making a real difference and also why I'm, uh, so happy to see how these partnerships are going, even post the agreement and as we're working with these folks, that the models are not just sitting on a shelf somewhere, they're actually being used on real programs, uh, people trying to approach devastating diseases where if Chai could give them a molecule that works, it could really change the lives of patients. So I'm really excited to see how that goes. Just the pace of progress here is incredible, but also just the pace of the models. A year ago you could not zero shot a molecule and have a good sense that your program was going to work. Now that's changed. Someone might zero shot a molecule and be like, I think we're going to bring this program to the clinic now. And then a year later, uh, you know, they might even have some of those first, uh, molecules going into patients. And just the speed of that is just incredible. And it's uh, you know, sometimes you get some shivers thinking about this, that like, okay, my model is going to like. Matt has some patents from our last company about, you know, like just generating the molecule on the computer and these things are now in patients. And just to think about the scale, like, I don't know, a few years from now, do we have dozens, do we have hundreds of like chi molecules going into people. It's a bit mind blowing to think about what that might look like for patients.

Speaker A: I think one of the things that, again, what made Shai unique, the dedication. People are like, man, you work a lot, um, and aren't you burnout or whatever? It's actually really easy and it's very motivating when you're making progress, seeing the progress they're making and just thinking, man, the next M model is going to be even better than the last. We identified this new thing. So on, um, that's so incredibly motivating. For me. It's more of just like, what can we unlock next? And how do we make these things more controllable? And when someone comes to us with a certain target, instead of just hoping we get good affinity or something like that, can we actually control this? Can we say we want exactly a 10 nanomolar binder, things like that? There are a lot of technical things that I think we're kind of right on the brink of solving. Um, and for me, it's really motivating just to pin those things down and just get all of this over the line and see of where that leads to next.

Speaker B: Awesome. Matt, Josh, thank you for engineering biology, and thank you for sharing your story with us today.

Speaker A: Thank you, guys.

Speaker D: Thank you.

Speaker C: Thanks for having us.

Speaker B: Sam

Speaker D: m.