Lukas Biewald | You think you're late, but you're early | Learning from Machine Learning #13

but look honestly if you zoom out even a little bit all this stuff is so underhyped like you can't you can't hype it enough right i mean it's just like it's just amazing it's incredible this technology works really well it affects so many industries people are like maybe disappointed that like it didn't solve every single one of their business problems like right now and it's like relax like obviously it will soon just wait wait a month you know How did the best machine learning practitioners get involved in the field?

What challenges have they faced? What has helped them flourish? Let's ask them. Welcome to Learning from Machine Learning.

I'm your host, Seth Levine. Hello and welcome to Learning from Machine Learning. On this episode, we have a very special guest, Lukas Biewald, the co -founder and CEO of Weights and Biases. the host of one of my favorite podcasts, Gradient Ascent, one of the earliest AI entrepreneurs starting CrowdFlower Figure 8 back in 2007.

Lucas, it is such a pleasure to have you on the show. Thank you so much. Thanks for having me. We're going to get right into it.

So what initially attracted you to machine learning? I was always interested in AI. generally, like, you know, even when I was a kid, I think I really liked to play games. And so I thought a lot, you know, try to program, you know, my own kind of like, rule based systems to, you know, to win it, like connect four and stuff like that.

So I was thinking about that a lot. And, um, you know, my dad actually read me that book, uh, Gertlischer Bach. And I didn't understand much of it at all, but it has these great parts where there's these sort of stories. And so the stories are pretty compelling.

And then I think with that kid mind, I was like, you know, AI does seem like humanity's last project and kind of the most interesting thing you could possibly work on. And then I went to Stanford and I... I actually reached out cold to Daphne Kohler, who was kind of a young professor there at the time, and asked her what she did. She's like, okay, take my class in machine learning.

The class was so much fun because it was kind of the fun parts. And actually, the best part of the class was this kind of reinforcement learning piece where you train a fellow program through reinforcement learning to... to get good at Othello. And yeah, from that point I was hooked because that's actually, Othello's a really satisfying reinforcement learning project you could even do back then where, you know, I kind of watched that program, you know, over a few nights, basically go from terrible where, you know, it couldn't beat anyone to just like, you know, crushing me every single time.

And actually it's funny, you know, I got way better at Othello playing that game. I've actually never lost at Othello since training that computer because the computer's kind of teaching me how to play um towards the end of it although i've never played like a i don't know professional or like you know skilled a fellow player but um but yeah i mean that that was like the point where i got i got like you know really really kind of hooked on it and then it kind of went through like a sad part so i you know i started doing research with um daphne i kind of thought okay maybe you know i want to be like a professor of this but you know this is back in like 2004 2005 there really was not a lot of stuff Um, working, I mean, Google was like working really well, but a lot of that was kind of like page rank and, um, you know, a lot of the, the energy was around sort of like, you know, ranking ads that didn't, you know, that didn't feel like that satisfying.

And I interviewed like some hedge funds and, you know, I was like, okay, like, I don't know if that's really like, you know, what I wanted to dedicate, you know, my life to. And I was actually thinking like, okay, you know, label data is kind of driving all the research, which is why I got into like the data labeling business. So I thought, okay, you know, like if we could create more label data, we could create more interesting applications, which is kind of like what I always, you know, what's kind of the most fun part of machine learning.

It's like, you know, really seeing what it can do. And man, I've like lost even what the question was, but I'm no, you know, it's perfect. So that leads you right into 2007. Founding your was that that was your first company that was founded, right?

Yeah. You were way ahead of the game. You were way ahead of the game. Yeah, although you say I was one of the first AI entrepreneurs.

I don't think that's actually true. I mean, there's been a long... People have been building AI companies since the 70s, 80s. So I think the definition of AI kind of changes.

Once we get something working, we no longer maybe consider it AI all the time. Right. Yeah. But yeah, that was actually sort of like an AI winter.

It was also sort of like an economic winter where there's... You know, like Y Combinator just started, but if you look at those early stuff, you know, those are kind of much weirder ideas. And now I raised my first round. It took a year to raise a round on, um, $2 million, you know, valuation and we had revenue.

So it's really a different time. Like Facebook was just starting to take off. And, and, and so another thing that was funny back then was I, I was basically, I was building an AI labeling company, but I was kind of coached to take AI, you know, out of the pitch. Cause that.

It was such an AI winner. Investors did not want to invest in anything related to AI. That just seemed like a bad science project. That was a real sign of the times.

It's funny because what was happening actually behind the scenes is these companies were starting to find success. Our labeling product got some quick traction with actually eBay was our first really big customer and that really pulled us along. Yeah, you mentioned a couple of really interesting points that I've been talking about also recently, like the moving goalposts of AI, right? Because like once it works, it's it's just an algorithm.

It's like, I always think about computer vision. And I think about like face detection and things like that. I mean, those are just algorithms. But if you thought, you know, whatever decades before, if there would be a thing in your pocket that can identify where your face was, right, that would be obviously magical.

Now we just think about it like it's just a given. Like, of course, that's how That's how things work. So I guess the modern era of AI entrepreneurship, I guess what I was saying early to the game, there are still companies that are trying to solve those same data labeling problems that you were addressing in 2007. So you could really kind of understand you had some inkling of something amazing happening.

I mean, think about it. That was nearly nearly 20 years ago. And we're still trying to solve these problems. Right.

Yeah, I usually ask like was there a moment that convinced you that this was the field that you wanted to build your career in? But it sounds like it was pretty early on or was there something later that happens that where you were like, yeah This is definitely what I want to be doing No, I mean, I think it was really early like I mean like really like a little kid kind of thinking about AI I think I remember somebody's dad I think told me about like simulated annealing days to call it which is kind of like a gradient descent And he told me something was designed in that way where they had the computer try different weights and then gradually make the weights better.

It really just captured my imagination. I was writing code. I was like, what if computers could write this code? When you're a kid, it's not obvious that facial recognition would be hard, but multiplying two huge numbers together would be easy.

you kind of come to that, you know, over time. And so I think I had like a lot of optimism, which has sort of like slowly, you know, crushed out of me through like, you know, actually like trying to try to do it, you know, from like, you know, 2003 to maybe like 2016. Actually, it's funny. I was, I was a big Go player.

Like I loved playing Go. I think one of the things I liked about it was that computers are so bad at it. And if you, you know, if you do play Go, you kind of... Understand why like it's it's like it's interesting.

It's like it goes like really designed for our brains to do it Well in a super cool and satisfying way, but you know, I remember when I was running I was running crowdflower figure eight when alpha go be Lisa doll. Yeah I mean, because I'm like a huge Go fan. I was like, actually, you know, I was like watching those games live and like, you know, watching the commentary and really thinking about it. And it was like, I actually had not really paid attention to the advances in deep learning.

Like at that point, if anyone is ever like, hey, we have a new algorithm that does things better. You just have a huge skepticism because people kept saying that, kept saying that it wasn't true, wasn't true, especially in like the entrepreneurial space. It's a good pitch, you know, but it's actually like generally not true. So I was like super skeptical that, you know, AlphaGo was going to beat.

uh, least at all. And then, you know, when it did, I was just like, Oh my God, you know, like this is like, that's kind of what made me realize that, okay, I need to, um, you know, kind of get more technical again. And I was like, look, and that's like how I ended up, you know, doing an unpaid internship at, at, at opening up. This is like very brief and wasn't on their payroll or anything, but I was like, I just need to go somewhere where people are doing the latest stuff and, and work with them because I, you know, running a data label and company, you work with like AI customers, but you're not really doing.

a lot of AI internally. So I kind of was, it's funny. I was like, you know, my thirties and kind of feeling bad about myself. Like I had just become this like totally like deeply non -technical person and just sort of like a mediocre, you know, like entrepreneur and like just didn't even really know like, you know, what was kind of going on in the space.

And so I'm so happy that I did that. Cause I think that like led to me. Getting a lot more technical again and and like actually like getting to enjoy This new era where like all the AI stuff works and like things surprise you by how fast they come rather than like how slow they come Yeah Going back to the point that you were making earlier, it's like back in whatever say the mid 2000s when you didn't want to say AI in your pitch, thinking about what that's like now, where every single entrepreneur, whether it is AI or it's not AI, they'll just say it, it's become more of like a marketing term.

And then yeah, the other point that you're making with Go, that's like one of the main stories in the book Genius Makers. And they talk about that like move, move. Yeah, yeah. Yeah, where it was not intuitive for a human to make that move, but it ended up being the right move, which is just such an interesting idea with AI in general, just like the ability for it to pick up patterns, but it's more than just picking up patterns, it's picking up patterns that humans might not be able to detect, which, yeah, unless you're kind of really in it, it's hard to understand the power of it.

And I'll tell you, for the true Go aficionado, which I'm not like a... I don't know. I don't play a lot of Go these days, but I am pretty interested in it. As a kid, I played it a ton of Go.

And I would always think, OK, what would God do? You know what I mean? I wish I could know. Because it feels so weird.

The kind of optimal first move is the 4 -4 point, or maybe the 4 -3 point, which is just not the center of the board. It's not the edge of the board. It's just like, OK. And then it's debated.

Is 4 -4 better or 4 -3 better? And all these openings, there's a lot of tradition and like, You know, you're kind of like, okay, how true is it? Like, and there are people who come up with these like wild styles and they'd like work pretty well. And I just always was wondering, like, could we find out in my lifetime, like what the best, you know, moves are.

And then when they trained it, like the, the alpha zero thing where they trained it with like not unprofessional games. I think the one that beat least at all was mostly trained on professional games and a little bit of self play to get a little bit better, but it kind of had a more. like a style that like came from humans, but then the ones trained without any human input are so interesting, both where they find like, you know, kind of famous human openings and like those, but also there were a few things that were missed, like really like obvious things that you could go back, you know, 20 years and actually explain them to like any Go player.

And the whole world of Go players, like, you know, like all these people just missed like these like Choices that that AI found I think it's like just so cool and and so profound Yeah, I mean it a little bit happens in chess, but I think I think go is like more weird Yeah, I mean like I'm also kind of into chess. I'm less proficient at it and know it less well. But it's pretty fun watching, you know, the new chess programs that are trained off of again from scratch. And there's so much more aggressive than like the deep blues and stuff.

And that makes me feel better about my unsound chess play. I can't speak to my go, but chess for sure. I used to I mean, I used to love playing chess. I wish I had more time.

And then I also think about like poker and things like that and how much there is. Yes, there is like the mathematical statistical statistically correct, you know, move to make in every position. But there's also the game that you're playing, and there's the styles that you can have. So it's interesting to think about that.

And the parallel that I'm thinking with, you know, frontier models and LLMs and things like that is like personas. And I'm thinking about that also almost as like the style in which you want something to be, you know, behaving in whatever environment you're in with them. So it's interesting, dependent on the context and dependent on like, you know, what the what the prompt is, you could have two very intelligent models, and they could respond, you know, they could respond differently.

Yeah, well, but just making a different I mean, like, so go, there's one best style. Oh, it is. Okay. Oh, yeah.

So like, you know, Because it's a full information game like chess, right? So like there's a best way This is best move at any point obviously and so I think you know, it's just I think that The models trained without human input now find it, you know find a better More effective style than the ones trained on on humans to sort of reveal that you know humans do some suboptimal Oh, I see what you're saying. I got you. I got you I guess that the thing that I was thinking about with chess was you could have the best move or in poker, you could have the best move given that you think somebody is going to then respond to you thinking that they're going to respond properly.

But you never you don't know exactly what they're going to respond. So in chess, you could have a move where you're planning one, two, three moves ahead. But, you know, some of the best players are thinking multiple 10, 15, you know, 10, 15 moves ahead. Is there is it like that with go also?

Maybe. I don't know. I guess I don't like to. Yeah, it's okay.

To play that way. I think generally relying on your opponent to make a mistake is a dangerous mindset. So what I'm talking about is the programs that try to play optimally. I think chess programs sometimes they try to train it to play like a human and make mistakes like a human.

And I'm sure you could take advantage of human tendencies more effectively and crush them even more thoroughly. But I think these programs beat humans. 100 % of the time, you know, and like by just playing optimally, too. So, yeah, it's pretty amazing how the how how fast that has happened.

I mean, like, I guess, you know, decades ago, people, I guess there were a lot of different feelings on the matter, but these sorts of games. Yeah, the the machine, the machine, the machine is going to the machine can win in all of these now. So was it the idea of the magic behind gradient descent, the ability for machines to get better over time, that was the thing that kind of attracted you to it? Or was it more than that?

Well, in a way, that's all there is, right? But I mean, there's like what you draw from that, which is like, if computers can program other computers, then you can solve literally every problem that humans face, right? I just think when you have that recursive ability, Yeah, to automatically prove algorithms. I think that's just the most powerful technology that you could possibly build.

It sort of subsumes every other endeavor that you might, you might try. Yeah, I kind of go for a hard pivot into fast forwarding to founding of weights and biases. What was the initial impetus there? Well, you know, I think it was I think in some ways for me personally, it was a reaction to what I liked and didn't like around CrowdFlower Figure 8.

So I think I felt like CrowdFlower Figure 8, it's a good business. I mean, Scale AI has taken it and run really far with it. It's a good business, but it might not be the perfect business for me to be... You know, running like, I think I'm a little more oriented towards wanting to sell to developers, like wanting to make like high quality products.

And, and I think I also just felt like a passion around it, like more than like, this seemed like a good business models. Like, you know, I really respect the people that are building these AI models. I'd like to help them. Like, I think like, you know, what I have to offer is like kind of understanding like how they work and like how to make their, um.

their work work better. You know, my, my co -founder Sean and Chris and I, we were just really passionate about like moving forward to the state of the art of AI. We felt like what we had to offer was good, um, developer tools. And we had been pretty close to the GitHub founders and kind of watch them, you know, be successful.

We also, I think one thing that was helpful to see was the GitHub founders at first, everyone's like, oh, developer tools. That's a small market. That's a stupid thing to work on, but it's sort of like developers became really powerful within the organizations. Cause like what they're doing is automating companies.

I think what we were thinking. with Weights and Biases is these AI developers, even smaller market, it's like a subset of developers, but they're even more powerful within their organizations because they're automating the automation. So let's get on their side. Let's make stuff that's good for them.

And that's going to be a great thing to offer the world. Yeah, definitely. So Weights and Biases was founded back in 2018. So it was a different...

machine learning landscape then. I fell in love with Weights and Biases back in 2019. I used it to track experiments. I think I'm not sure if it was even earlier, but definitely one of my first one of my first big projects.

I was like, I need to figure out a better way than writing all of my results down. And I want this to be automated. I was like, oh, OK, I could just add this these three lines of code. That sounds great.

And now what it has evolved into. Oh, my God. It's like. So at the time, I was just, you know.

logging classification reports and confusion matrices. Now I don't even know if it's like a standard thing. I put like my visualizations. I have like these data map plots that I put into weights and biases.

Yeah, I have a lot of fun with it. I've told every machine learning person that I know how much I how much I really weights and biases has made my career so much better, like just so much better. So thank you. Thank you.

Thanks for that. That kind of feels really good. Yeah, definitely. I hope you hear that because it's an amazing tool.

Yeah, I guess maybe what happens is people start taking it for granted, but no, it has become ingrained in the way that we do work, 100%. So yeah. But what I was going to talk about or ask you is basically like, yeah, the machine learning landscape has changed. 2018, I think it was, I don't know what I was working on a lot of just like classification stuff.

And now 2025, we have the LLM for the last like, what, three years, I guess. It's a very different landscape, the generative landscape. So how has the vision evolved for Weights and Biases? Can you talk like, yeah, just with the landscape changing and...

Yeah, just you touch upon that. Yeah, I mean, the you know, the biggest change by far to the landscape, I think, is that, you know, when we started like. you'd mostly build your own model and like fine tuning was like some people would do it seemed like maybe a good idea but it was actually kind of unusual and now i think many many tasks you could just you know use an off -the -shelf lm you could go to open like if you want to translate something you don't need to build a translation model you can ask um opening literally just ask it to translate something right um and so so many of the the tasks now are just asking lms the right way and then we have like these new even more exciting tasks where it's like chaining the stuff together using tools which you call like agentic systems i think that's where um you know, most of the interesting innovation is I do think deep seek in particular cause the resurgence of people wanting to do their own reinforcement learning.

So we are, you know, seeing actually more users coming on board to like weights and biases, original product and than ever, which is like really satisfying. But, you know, it's like a different style of thing, right? Your models are enormous. You're typically, you're always running on like GPUs.

Often, you know a lot of GPUs running in a distributed way like that's like kind of table stakes like it was kind of exotic when we started like opening I was the only one doing distributed training for a long time on weights and biases and and now I think you know any Anybody really trying to do like meaningful scale is doing any distributed training of some some variety So so yeah, so look at what happened was we have this one kind of product that does experiment tracking that was really popular that we now call our models product, just to give it a name.

That's a WB Python package you download. But we added a product called Weave that's designed just for running LLMs and kind of all the work that happens around that. And the two are actually great together, right? Because a lot of people, you want to build a model, then you want to run it.

But I think that the number of people who would have a need for a product like Weave that could track an LLM that they got off the shelf is much bigger than the number of people building AI models. So I think what we actually saw was this democratization of AI that everybody had talked about and where anyone can really harness the power of AI. And then, you know, they get all the kind of drawbacks of AIT, right? Like, you know, just because you're running a third party LM, it's like still non -deterministic.

You know, it's still like really hard, you know, like to know how to test it. Like, you know, you're never going to have a hundred percent test coverage and you have a non -deterministic, you know, LM underneath, you know, your system. And you're going to have to think a lot about like, okay, like what are the metrics? I'm actually trying to optimize here.

How do I make them better? You know, what's different is like, you know, people kind of coming to that new without any machine learning training. You know, they're, they're not as comfortable with statistics as the average, you know, like ML person, and they don't even necessarily in the beginning, see the value and evaluations, right? Cause it's kind of annoying to set up evaluations.

And you just wouldn't think of it. You'd be like, ah, just I'll see if it's good. And then I'll like, you know, ship it. And so I think in the early days of waste and biases, everyone, no one would be like, I shouldn't do evaluations.

And we're like, how do we do. Right. I think with LMS, you kind of actually have to convince people to do evaluations. I do think there's a flip side of that though, like.

I think ML researchers had this tendency to just look at aggregate statistics. And a lot of what our product would try to do is get you to look at individual examples and see what's happening, because there's a lot there usually. And usually, ML researchers don't do enough of that. I think you see the flip side of that in this LLM world, where people talk about testing by vibes.

But testing by vibes is actually trying the thing, looking at some examples. It's a really good thing to do. So now that happens enough, maybe too much. And there's not enough to look at the aggregate statistics because, you know, it's like, Vibes will get you really far.

Vibes is really good at preventing you from doing crazy things and having like these horrible, you know, like deployments and stuff. You know, if you're trying to like nudge inaccuracy higher, which is kind of where every project ends up and you want to get like from 69 % to 71 % accuracy to 73 % to whatever, that's where you actually really need evaluations because you can't kind of make incremental steady progress without, you know, clear metrics that you're improving. Yeah, 100%.

So I guess a couple of things. Yeah, so I loved When you guys came out with tables, I thought that that was really cool, because I was able to in weights and biases actually look and group my data and find and see particular examples. And I built and I've used weave. You know, we've yeah, we've is so cool.

Some of the things that I really like about it is the tracing. So when you have, say, like multiple LLM calls, you have, you know, certain tracing. And what I really like is that you can see where the what parts of the code were touched with that area. And I haven't really been able to get that sort of detail anywhere else.

So that's that's one of the reasons why I like I like using weave. And then, yeah, just to get into like evaluations, I think people the hard thing is it's just it's very hard to evaluate generative outputs. What makes a summary, one summary better than another summary, and you have to sort of think about different criteria of what it's going to be on, and then to go into your point. Yeah, like you have to have a sufficient, you have to have a robust evaluation system, because anytime you're making any kinds of changes into the system, whether it's prompt engineering, or whether it's using a new model, whatever changing hyper parameter, whatever you're doing, there are always trade offs that happen.

And if you don't have a robust way of evaluating it going from something that looks good. You know, like, oh, I think it's good. I tried it out on 10 prompts. I tried it out on 10 examples.

It looks good compared to like, oh no, like I need to get this performance up from 75 to 85. Any change that I'm making, I have to make sure. So what I've been doing is I just keep building my evaluation set, right? Just I get it done end to end and then I'll test 10 things and then I'll make sure it still performs on those 10 things and it'll be 20, 30, 40.

And then like all of a sudden you're like, whoa, I have like 150 things that I'm testing on this thing. saying, all right, it's still not gonna capture everything when I bring it to an actual user uses it because you can't ever, you have to continue to just kind of figure out how your users are using your model. But at least I understand there's no regressions in some of the capabilities that I wanted it to have from the get -go. So I think LLM observability is really hard.

It's a challenging problem and I... It's hard because I know what it's like from your perspective, what Weights and Biases does is a general tool that can be applied to specific problems, which I guess is a lot of software obviously, but particularly for something where there's just... no real rule book, there's no like, oh, like check the F1 score, you know, for like traditional machine learning. It's not that simple.

So it's hard to know what to observe when it's unclear exactly like what people want to be building with it. So it must be a very, that must be very challenging. I would say I'd say I don't know, how do you deal with that? Like, where the goals of the people using your product are evolving so fast?

Is that something that you think about? Totally. Yeah. I mean, I think it's I mean, it's exciting.

It's like, you know, I think the, the entrepreneurial gets like really excited. You know, I think it's, you know, it, it kind of favors smaller companies in a way, right? Like when you need to like really iterate quickly. So I think like one of the funny challenges that I find myself in is trying to, you know, make like, um, you know, 250 plus person company, you know, recently acquired by a thousand person, you know, company, like just keep like, you know, thank you.

Thank you. Um, yeah. You know, like I think we have to move really fast. And we have to like listen really carefully to what people are doing.

But, you know, I think that you don't also, also you don't have to solve every problem, right? So like, you know, I think we talk to our customers all the time and we like look at their workflows and you see like, okay, people want to make these like small sets. They can quickly look at, you know, like LM is a judge is like a really popular strategy. There's different kind of patterns around that.

There's also like, you know, making sure that like users can actually like, you know, put popular, the feedback back into weave. And there's often like. you know, sets of data where it's just like, you can never like get this one wrong. Like we never want like anything, you know, bad about our brand or like, you know, we don't anything like sexual in our, in our result, you know, like, so you get, you get like what ends up happening, I think in the same way that like, you know, we'd kind of know that we were dealing with, I think a serious like ML company when it'd have like thousands of metrics, right?

Cause he's sort of like over time, you just get more and more metrics that you're like tracking around whatever you're doing. Cause it's like, in some sense, you're always optimizing one number. But nobody really thinks like that. You know, like they, they want to see all the different things that are happening.

Cause they don't really know that what they're exactly optimizing. So they want to look like the same way as like with a balance where it's like, you know, over time, you know, there's all these different considerations that you have to live. What means good. And so you start adding more and more, um, you know, evaluation sets.

And so I think that's, that's, that's where things move in. Like, you know, we don't have to. do everything for you. We want to do the most helpful stuff, but I think a lot of people should be writing their own evals.

There's a lot of great third party open source evals libraries that we try to support really well. And then we also try to make it work easily out of the box so people can at least get a taste of the power of what we have. Very cool. We'll go into this one.

What are you most excited about the future for Weights and Biases? Well, I think we're now inside of a public... uh, company with really big ambitions. And I think like, you know, one thing that's exciting is that, you know, they have, you know, they have a ton of resources available to build, you know, lots of stuff.

So I think like, you know, we had to be really careful about like, where are we like aimed our, you know, resources, but we knew there's like tons of other like steps in the, in AI workflow that like, where we felt like, you know, we could do a better job of, of building great stuff, but you know, we also wanted to stay focused, but I think now it's like, there's a bigger scope here where kind of any any step in the AI workflow is kind of like fair game or something like we could build.

That gets really exciting to be able to build more stuff. And I think we obviously don't want to only work with core. We've been really careful about always saying that we're going to work with every cloud. It's always going to work with all the different infrastructure providers out there.

But it's kind of interesting to actually see real life infrastructure and hardware. Um, deeply cause it does show you, there's like more stuff you could surface. Um, that I think would like kind of help the end user, especially the AI researcher. Like I think we could do more to like, let them know like, Hey, you know, weird things are happening in your, in your hardware stack and your training that you may want to take a look at.

Um, we have a conference coming up in a, in a couple of weeks called fully connected. Um, and we're going to announce a whole bunch of like integrations there that that's kind of the tip of the iceberg of what I'm, I'm excited about. Cool. Yeah, that's exciting.

We met last year at Fully Connected. That's a great conference. Yeah. In terms of, you know, joining a larger company and, you know, understanding more of the integration and scaling and things like that.

So, yeah, you're thinking there's just a more diverse problem set or more areas where you could kind of help your end users or your understanding things better. Yeah, like what excites you about that? Well, I think it's like, you know, we go head to head against like Amazon. Amazon, you know, we're friendly with Amazon, but they have a competing product for like most of the stuff.

that we do, and mostly their product is garbage. I don't even know if they would like, if you privately meet these people, they'll tell you that they think the weights and biases product is better. Boy, I'm not making any friends with this comment, but honestly, I think we've picked off a few things that done them really well. I think at an executive level though, people want to buy a complete solution.

No developer wants that, but for some reason... I shouldn't say for some reason. I understand why. Like executives and companies kind of want to buy like one complete solution, have it be like cohesive and sell it.

And I think like, you know, it's kind of a danger for weights and biases, the product, right? Like, you know, people would use like MLflow because they're like a Databricks shop and they're just like, look, I wish I could use weights and biases, but like, you know, Databricks comes with like MLflow for free and my boss is saying I have to use it, you know? Like, I think there's a lot of power immediately. in bringing these two companies together, like CoreWeave and Weights and Biases, now we can offer more complete solutions.

CoreWeave is also incredibly fast, like revenue growth, far faster than us because of this amazing demand for hardware and they execute really well. And so we can also use some of their resources to build a more complete solution, which is probably where the world is ultimately going. So I think we could have gone on Um, alone, but it's not free. You're always kind of like looking for ways to, to like get an edge because it's these markets tend to be like winner take all or winner take most.

And so like, you know, I'm pretty focused on like, okay, how do we like completely dominate this market? Like not end up in some sort of niche, um, category. Yeah. Sort of transitioning into the entrepreneurship, uh, parts of things.

It's interesting thinking about. tech solutions and how there are many solutions obviously for many problems that's like, you know, the exciting thing about being in this in any, you know, market, but Basically, like Weights and Biases has a lot of competitors. There's open source competitors. There's other third parties.

There's complete companies around it. There are companies where they're just doing things. I mean, you know, you spoke about Amazon a little bit. It's like, yeah, like Amazon has their own foundation models, too.

You know, but like, I don't know a single person that's using them. And they also invested in other foundation models. So they have these huge, huge incentives. But to go back to the other point, it's There's an interesting thing.

If you think about all the problems that are flowing down, you try to get a bucket and you try to find that bucket that you're going to capture a lot of the problems that you're getting. But then at a certain point, if you're a company that's buying these vendors, it's like, well, how many buckets am I going to have? I want one solution that's going to solve all of these things. And then sometimes you might not be able to focus narrowly on the problem.

I know you definitely think somewhat about this, but I'm curious, like, what's your take on this whole there being many solutions for things and like tech consolidation? Yeah, like you sort of touched upon it, but do you have you do you think about that in the industry that you're in? Yeah, totally. I mean, I think what Marcus tend to do is like when there's some like new problem and you need new capabilities, that tends to favor smaller.

companies. So you get this kind of Cambrian explosion of different options and the sort of biggest ones tend to consolidate. And so I think we're kind of seeing consolidation of some stuff. I think there's a time when MLOps is all the rage.

You don't hear about that anymore. I think that's definitely kind of consolidating market. But then there's also a lot of new like LM, you know, functionality around like, I don't know, like, like routing, like, you know, how many companies need like a router today? Like, probably not a lot, like, will they need a router in the future?

Probably most will want one, right? So that's like a really interesting, you know, category of thing. And there's a bunch of companies there. And, you know, I think There'll probably be more.

And then over time, there'll be less, right? Because maybe that's not really a standalone business. I'm not sure. Maybe the ones that are doing it will expand into other categories.

I think that's just life and technology. But I think with AI and the speed of innovation, it just happens way faster than it used to. Yeah. I think I'm stealing a question that you asked your guests.

What's a free idea? You asked that, right? What's an idea if you had any time in the world that you think... Yeah, it's funny.

I feel like I ask it more in a non -commercial context where I'm actually just curious. Because I always think, if I'm not doing this, what would I really do? It's a good question to ask yourself what's interesting out there. In terms of startup ideas, Man, it's like, I don't know.

It just seems like such a fun time to be an entrepreneur right now. Because I feel like, I mean, this is not a secret, but you take this into any application and it just still blows people's minds. People just do not connect the dots between LLM generally and LLM applied to a particular problem people have. And so I'm just seeing all this.

You know success in that enthusiasm. I think people kind of think they're late Like my friends will talk to me be like, oh am I too late to this? No, man, you're like still early It's amazing, you know, just like right like I think lms applied to anything. It's probably probably like a good idea honestly, um, you know, I don't know like I think it's like good to pick ideas that you kind of care about um, I mean Again, i'm not telling anybody the things that they probably don't know but it's like um The progress in robotics right now is unbelievably exciting.

That's obviously going to be this huge thing. Again, I think you might think you're late, but you're early if you're looking into that. That's what I'm intrigued by. When you're doing a company, you really don't have time for anything else.

I know. The potential for robotics is just too exciting. I mean, being able to combine sort of all of the power of LLMs along with, you know, computer vision and all that, I think that's a really exciting, really, really exciting field. I'm so curious to see what's going to be coming out in the next couple years and what people are currently working on.

Zooming out to just like kind of machine learning in general and you know, maybe we touched upon it Maybe we didn't but what do you think is an important question that you believe remains unanswered in machine learning? I mean, there's a lot I think like I Don't know what then I'm like kind of like intrigued by is like you know if you ran if you ran back like history like a thousand times, you know, like how much of, you know, what we do around like a transformer architecture would be consistent and how much is a sort of product of like, you know, how we, how we do it.

Like I think like, you know, there's just not a lot of like appetite for experimenting with stuff and people don't tend to write it down when they get just the same result. Like I feel like. There used to be LSTMs, GRUs, all these different architectures. I even remember trying to program these things by hand and getting it a little bit wrong, and it's still fine.

So I wonder a lot about what are the core bits of this architecture that really matter. And then, of course, we now design our chips around this transformer architecture, so it's still locked in. You wouldn't want to change the aspects of it. But I think that's maybe an intellectual...

Question that I I wonder about a lot and I think and there's been an explosion of Research and reinforcement learning right now, which I think is totally makes sense, but I'm having trouble Comprehending I'm talking way like this is sort of trying to like understand right now. It's like what what about these different? Like reward functions make them effective or not. I Don't know.

I'm confused. Maybe somebody understands it. But yeah, I'm more confused these days than I was in the past Yeah, we have a mutual friend that would understand. John Schulman would understand proximal policy optimization.

But anyway, going back to the other point, Transformers, and I think about it too, I think about this with everything in life also, is it just a local maximum or is it a global, you know, or, you know, is it a local minimum or is it a global? Like, is this the most optimal? And then I just was speaking about it, but also like, yeah, like, The 2017 architecture is not the 2025 architecture and that's because the bet was placed and the bet was placed and the infrastructure was put into place and the and the chips were made and the ecosystems were created and then you start iterating on something and if you make a choice and you iterated on on it for eight years and You start to solve all of those problems that people were having it's like you're gonna get it to a place where it's really good and then we are seeing that we're that we're probably going to hit a ceiling but I think that already it's unlocked so much it's unlocked so much and just like what going to your previous answer just applying LLMs anywhere is there's you're going to be you're going to be able to you're going to be able to get value so it's going to be very interesting to see sort of what what happens this is another one that I like to ask how do you view the gap between the hype and the reality of AI?

Yeah, okay, so like hype is tricky because I live in San Francisco and I Don't really know like the level of hype Like out in the world like it seems like very like uneven like I'm always kind of surprised that like resonates with my friends You know another in other Geos and and not But look honestly if you zoom out even a little bit all this stuff is so underhyped Like you can't you can't hype it enough. I mean, it's just like it's just amazing, you know, like it's incredible This technology works really well.

It affects so many industries. People are maybe disappointed that it didn't solve every single one of their business problems right now. And it's like, relax. Obviously, it will soon.

Just wait. Just wait. Wait a month. Yeah, exactly.

There'll be a new model next month. I can do it. I mean, one of the exciting things, I don't know if you've had a chance. Have you had a chance to play with any of the AI assisted coding like cursor or things like that?

Yeah, of course. Come on. Yeah. If you're not doing that, what's wrong with you?

I honestly like when I come just an engineer. Yeah. Yeah. Yeah.

I mean, I think there's like, I think cursor code is like so much fun. Yeah, I think that. I think that really is a really well -done product. Sorry, I mean, Cloud Code.

Oh, Cloud Code. Yeah, Cloud Code, I think it's really delightful. I think Cursor is also really well -done. I mean, Windsurf.

I don't know, I like them all, honestly. I think they change so fast and they're just awesome. It's funny, I'm coming into... You know core weave and I'm and they like, you know that for them like kubernetes.

They like know it so well, and I'm just like Like they're like, oh here's some machines, but it's just like a kubernetes cluster. I'm like, oh man Like I never learned how to do this, you know, and so Yeah, I started asking like cursor like hey just like to play my stuff like in this kubernetes cluster And it's just sort of like okay, you know, like has a couple questions and then it's just like firing off jobs I'm like, yeah, that's amazing. I love it. Yeah, and then meanwhile my daughter, you know, it's me is like Claude code together To just like vibe code like weird games, you know, and and I was like, you know, I tried with cloud code.

I mean, this is like going to be so dated in a few months because everyone knows. But I was like, hey, like, can you like deploy this so my daughter can show it to her friends? You know, and it's just like, oh, yeah, make a Netlify account and I'll like put the app in there for you. And it really did.

Like one, you know, like what's good is so cool. It's so cool. Yeah. Yeah.

I find myself like. anyone in my company that wants to work on something or like has an idea, I'm just like, can you just download like, can you just download cursor? It's not that simple. You need to know how to code to be in a code editor like you do, like you still kind of do.

But there are other tools that are out there. What is Lovable? I don't know, my dad uses Lovable. He was making these crazy games and showing them to me.

And yeah, I mean, like, yeah, Lovable is great. I mean, there's other ones too, but that's the one that he was. Yeah. Replet has one zero V zero.

There's so many and I'm sure in like, you know, a couple of weeks, there'll be another there'll be another dozen that'll be doing something that's mind boggling. It's it's it's amazing. I mean, the nature of work is changing. I just there are so many people that view that look at the negatives of it, but there are so many positives to it.

Also, the democratization, like you were talking about earlier, many more people can do many more things. And it's just it's so it's so much. It's such an exciting time. It's it's such an exciting time to be in.

Yeah. What what's your daughter deploy? No, it's in some links. I mean, we're we're just five.

So the mind of a five year old is like a weird. weird place, you know, so yeah, but that's so cool that she was able to do it. Very cool. Where do you see things in three years or five years?

What's your take on like software developers? Do you think that there's going to be like you think there's really going to be a real hit to software developers? Or do you think the nature of work is going to change? What's your take on there?

Look, I don't have any special insight. I do think like one thing that I'm seeing that's kind of I've been thinking about a lot and this is it seems like You might have thought that this sort of be like more of a democratization of Code and certainly that's true, right? It seems like non developers can like make stuff But it seems like the more dominant effect is it's making the best developers even more productive Than not the best developers like I don't know when you like meet the teams at some of these companies like, you know cursor You know kodium or madness or something?

It's just like it seems like a small number of like incredibly hard -working effective people and so that's like sort of seems like the winning Strategy like it sort of seems like everything we do kind of increases Inequality like unfortunately and that and that it seems like that's what's what's happening, you know right now and it's very hard to predict like I think like if you told me It would be able to automate a lot of code tasks. I think I would have thought, you know, the upshot of that would be like, wow, now I can like hire like more developers, be even, you know, more effective.

But I'm not so sure that that's like the right thing to do right now. Like even with, you know, with what we're seeing, like it might be, you know, now like really small teams or even like one person can be more effective. I definitely think like. you know, engineers have like more of a product mindset can be more effective.

Cause the way a prop, probably just kind of like a prompt engineer, you know, for fresh years, right? So, um, it seems like it seems like moving up the, the sort of stack is probably, you know, probably, probably like favors that. Um, but yeah, I mean, um, I don't know. I think about it with like my kids too.

It's like, okay, like should they, Like learn to code and stuff like I'm kind of like okay like learn to code as we do today If it's like fun for you in the same way that I think like woodworking projects are like sometimes fun, right? Maybe we'll you know, we'll have like artisanal You know libraries like charming bugs Yeah That's funny. No, it's a couple of good points there. I mean, well, I'll start with the last one.

It's like, you know, like I had to like build a clock at one point in my life. And no, I never need to know how to actually build a clock, but I learned so much doing that. What kind of clock did you build? It was like at camp and it was like a wooden it was like woodworking and then we put like the motor in and it was able it was able to do it yeah it was it was pretty cool I remember like staining the wood and you know cutting it and sanding it and doing all that stuff like no like do I ever need to do that stuff but it's like it's like being able to do it I think there's like um One of my friends in college always used to talk about this time that he was working as a farmer, basically, and how much it changed the way that he just thought about every like everything like what it takes to maintain a garden or a crop a set of crops.

It's like, those are the lessons that are very important that you need when you're doing anything that has any anything that's not just like a one time thing right like which code very much is it's something that evolves over time so being able to maintain and take care of your code is is something that that's that's very important And then you also raise a really interesting point. It's like, yes, there is the democratization, but there's also the concentration of power. You know, whether you believe in like the 10 X engineer or not, but whatever, a very highly effective engineer that's then able to automate more of their work is just going to become extremely, extremely effective.

Doesn't everyone believe in the 10 X engineer? Like, how could you not? Like, if you actually work with engineers, I think that's like. I mean.

I don't know, I just didn't want to assume that you did, because some people are very much like, I don't know, I've seen a lot of polarity on that. Really? Yeah, some people really don't and then some people really are. Maybe you're right, maybe I'm just not talking to the right people.

I think that if you know engineers, you know that there really are people that - I can't imagine working with engineers enough. No, 100%. No, there are people that are just capable of doing that are capable of architecting things that just other people are not capable of doing and make an impact that's just much larger on things 100%. Yeah.

But yeah, so I just didn't want to assume that that's what you thought too. I'm moving into like advice and things like that. I'm going to switch this one. What advice would you give yourself when you were starting your career?

Yeah, I think the most important advice I'd give myself is like, you know Lucas like you think you're late, but you're early like I just always felt bad like I you know, I Like it's funny. Like I I graduated, you know, like 2005 and I got an offer to go to Google and I think I didn't take it honestly cuz I was like all my friends went there like You know, it's like, it must be like late. You know, I think, yeah, this is like my whole career is always like feeling bad that I was like late to stuff, but actually like, you know, I was like, um, like early to everything.

And so, um, yeah, I don't know. Maybe that would make me just feel better about my career advice is tricky, right? Yeah. What advice would I give myself?

It's probably something I just think about more. That's a good one though. And I think another thing. That I think another thing that I realized later Is it carving out like huge blocks of time to stay technical?

I kind of felt like bad about that actually when I was like first You know for like a long time I felt like it was sort of like I you know I had these coaches like executive coaches like if you go off for a week You need to like come back with like something you produced and like a message to go like you can't just like Disappear for a week, and that's like you know fuck it like I disappear for a week Like if like, cause I would go in like paternity leave and I would come back with like so many ideas and like you can't really concentrate on paternity leave, you know, but it's like, I think like saying technical, it's going to like be like lots of ideas.

Actually it was, I met the new relic CEO and he was running a public company at the time. And he was like, you know, I spend one week a month just like basically by myself or with the smaller people just like writing code. And that like, it was pretty inspiring. And I think another thing it's like, you know, I think when I I think for a long time, people have this stupid idea that you can have these executives that don't stay technical or able to do the IC work.

Because management is its own skill. And so you should hire good managers, not good ICs. And I just think that's so stupid. How many times do you have to see you hire someone who seems like a good manager, but they can't do the individual work and watch them fail?

Before I said, give me a fucking break. Like, you know, like, you know, if you're going to like work for me, you better be able to do the IC job. And like, I do not know how the company's function without that mindset. I'm just like baffled.

I thought maybe it was just like me, but I actually just think there's like a lot of bad executives that just kind of like keep their head down and somehow, you know, keep like shuffling around and like, you see like VPs of engineering that are pissed if you like ask them to like code in an interview. It's come on. I'd be like. Excited if you give me like a like a leak code Google code interview I might not be like awesome at it, but I would be like pumped Yeah Corvish has done that when they when they hired me Go back go back to the due diligence and tell me you want you want me code See you better be able to do the leak code I love it.

I love it Yeah, it's it's huge. I mean getting shit done I mean like that's what it comes down to is You can talk about it and you can plan and you can do whatever you want But it's like are you able to actually execute and are you able to actually get this done? And the amounts that you can get done in a short amount of focus time is like is unbelievable So you better have people around you that are able to do you know do do things? I think that in what I love about startups is that you don't have the time, you don't have the fact, you don't have that, you can't, you won't survive, right?

If you're someone that can't execute, if you're somebody that's not able to get the job done, it's just like you don't have a, you just can't really have a role at a startup. That's what I love about entrepreneurship. Yeah, and thank you for the real answer. That was great.

I like that a lot. So getting into just, you know, this is learning from machine learning. So I have to ask the question. What is a career in machine learning taught you about life?

Well, I think like one perspective that I have from the machine or whether you think about a lot like when you do machine learning. This might not be true now. I think most of my machine learning was done like building and fine tuning models. This is not like the new, uh, you know, engineer stuff.

But like, you know, I think when you're like training models, you, you kind of don't know, like when you should stop, like when something's like not working, like when you have, you know, an idea. So I think a lot about, okay, like when do we like pull the plug, you know, on this thing that I'm doing, like how much information do I like really need, you know? Um, and I think another thing is like the, the feedback loops are kind of like your unit of um, your unit of work. So I think like the faster you're like getting feedback and the more feedback that you're getting, I'm sort of obsessed with getting, you know, feedback, whatever that means, you know, for whatever I'm doing, kind of getting that like quickly to try to get, you know, better, whatever that thing is.

I think that kind of comes from doing machine learning where you're just trying to get like the information on like, is this working or not? So I can like move on to the next experiment as fast as I can. Yeah, that's interest that that's interesting so going to the first one where you're saying like knowing when to stop are you particularly talking about like you're training a model and you're Iterating on it and you're like when is it good enough to go into production or like?

Like I was thinking of it's like okay. You have a model There's usually like an infinite number of things you could try to make it better like get more training data different types of hyper parameters different architecture, you know, all these different things, you know, and you're kind of like, okay, like, what should I try? And then when should I like move on to trying like the next thing? And I think that's, I mean, it's funny, I was starting to say, I think that's like effected my life, but I think only the fact that I think about that a lot kind of comes from in machine learning, like think about a lot.

And it's like, it's hard, you never really get that feedback on if you like cut that experiment short fast enough. And I think different than like more like statistical practices, you kind of need to make these decisions. within perfect information, right? Which is more like life.

You know, like, I think like medical stuff, I felt very familiar having like medical issues where I'm trying to look at a bunch of studies that don't kind of line up and none of them are like seem totally statistically significant, but yet you have to make a decision about your health that's important. That felt like a very familiar situation to me from machine learning where, you know, you've done a bunch of tests, none of them are perfect. Probably there are some bugs in your code one time, so it's questionable if you can trust it.

So you have 30 messy data points, and then you have to decide what information to collect next or what to try next. I don't know. I think that's a common situation in life. I think machine learning gives you a lot of practice handling that.

But I'm not sure I would claim to be good at it. I just, I don't know. I think I just, you know, you counter that a lot to get that perspective. And you know, like running a company, it's like, okay, you know, you try some like new growth.

Like, it's funny, our head of growth, it was actually, she was a machine learning person. And it was like interesting to watch her. You're kind of thinking that it was like - Lavanya? Lavanya, yeah, yeah.

So her growth stuff felt like machine learning was, okay, we're going to run these like 10 experiments. you know we're going to try to like cut them off with like imperfect information get the like 11th experiment in there it kind of made me realize it's actually a really similar thing to trying to find the best um you know ml model for some task yeah that's what i was going to say is that it while in machine learning, you're dealing with uncertainty, also in entrepreneurship, you're dealing with uncertainty and making decisions, yeah, with imperfect information.

And you can just make the best decision, you know, in that moment. And then going into your your other point, yeah, about feedback, it's, yeah, I mean, it's it's crucial being able to collect the right feedback, being able to understand how people are actually using the thing that you created, and then trying to incorporate that back into the model, whether it's some Sometimes it's like, yeah, obviously, you could change the data set that you train the model on, or you could change the way that the user interacts with your model and things like that.

That's great. Is there anything else that you'd want to share about that? Well, I don't know. I guess I was thinking about you had an earlier question about how do you...

like what advice would you give yourself? And I think like one of the funny things that I've experienced in life is that you get a lot of advice that's like really good and you just don't like do it enough, you know, and sort of like over time you like learn to just like do that like obvious thing that you'd obviously like tell yourself just like even to a more bigger degree than um than you're getting. It's like I feel like asking customers for feedback is such a funny one where like You know, just no one, no entrepreneur does it enough.

And it's like, it's not the only thing, right? Like, you know, like, I mean, just sort of blindly doing what customers want is kind of a stupid way, you know, to operate. But I think like understanding customers deeply and asking them like what they're thinking and like, you know, like what they would want. It's like, I really think that that, I think all my co -founders, we really, like what I think about like all my co -founders, my two other co -founders, we all really had in common was.

We've really felt like super bad when customers were unhappy and we like worried about it a lot. And it would, it really like hurt our feelings. You know what I mean? And so like, we would, I feel like we all kind of just like maybe a little bit of like an anxious style.

Like we'd be like asking her feedback. Cause I think we'd be just like worrying like, okay, like, is there some problem? You know, like why aren't you using our product? And I think like getting that feedback from customers like all the time.

you know, just like really, really helped us be successful. And it's so basic, but it's like, I just, I know I like watch a lot of entrepreneurs like not do that. And I think even in my first company, I probably didn't do it enough. Um, honestly, as I'm talking about this, probably should do it even more, um, you know, now, but it's funny because everyone knows you're supposed to do that, but I think it's just, it's like pretty annoying to do it.

It's like a little bit stressful, you know, um, you know, we just don't do it enough. Yeah, well, you make a really good point in Well, you made many good points, but the the point around like you don't just blindly take customer feedback and then apply it. It's the step of deeply understanding the pain and the problem and the need of your customers to then inform your roadmap and then to inform the decisions that the decisions that you're making. And I think that that's where the challenge is, is that you can get tons of feedback from your customers.

But the hard thing is to really have the empathy to put yourself in the shoes of your customers to understand what their problems are what the incentives that they're dealing with are. I mean, I know that from my perspective, I think about the problems that I'm trying to solve, and they're often not the problems that my people who are going to pay for the software that I'm creating are going to are dealing with. And it's that gap that makes that such a hard, such a hard, such a hard thing to do.

Yeah, totally. Lucas, this has been like just like unbelievable. I said to you offline, but you're one of the people that I wanted to get on this podcast for so long. Yeah, it's it's incredible.

Maybe you think that you were late or whatever. You were early to the game. You know, you've had such an incredible career to successful exits, not exits, but you know what I mean in terms of your your startups. It's it's really, yeah, it's really admirable.

It's it's it's really admirable. And also just the type of person that you are. It was so amazing getting to know you a little bit at Fully Connected and just how real you are as a person and not the usual CEO. And I really just appreciate you, so many of the things that you're doing, the incredible software that you're doing, but just the force that you are in this field.

So thank you so much. It's really amazing to have the opportunity to talk with you. No, thanks, Ed. Really appreciate it.

Yeah. Is there anywhere where listeners can learn more about you? Well, just go to WMB .com.

But if it's before Fully Connected, then they should come to Fully Connected. We'd love to have your listeners at our conference. It's a great conference. I'm in New York.

I don't know if I'll be able to make the trip to California this year, but I'm sure I'll be there again in the future. I would encourage people to come. Lucas, man, what a pleasure. Thank you so much.

I really appreciate your time. Thanks, man. On this episode of Learning from Machine Learning, I had the privilege of speaking with Lucas Biewald, co -founder and CEO of Weights and Biases. We trace this journey from programming games as a kid to building one of the most essential tools in AI development today.

Lucas's career demonstrates that conviction often matters more than consensus. From surviving the AI winter in the mid -2000s to the AlphaGo moment that changed everything. Lucas reminds us that we're still early in humanity's most transformative project. He challenges conventional leadership wisdom and shares his unwavering belief that executives must stay technical.

He explains bluntly, if you're going to work for me, you better be able to do the IC job. Most importantly, Lucas's philosophy that feedback loops are your unit of work transforms how we approach both machine learning and life. His advice to his younger self cuts through common doubts. You think you're late, but you're early.

In a world racing towards progress by any means necessary, this reminder couldn't be more relevant. Thank you for listening. Be sure to subscribe and share with a friend or colleague. Until next time, keep on learning.