The World’s Safest Driver Isn’t Human. Can Waymo Stop Traffic Deaths? | Chinmay Jain, Dir. Product

Speaker A: Foreign. Mei, welcome to the show, man. Thanks for coming on.

Speaker B: Hey, Jeff, thank you for having me. And happy Friday.

Speaker A: Happy Friday the 13th. Actually, it won't be Friday or the Friday 13th when it's published, but hopefully we have a good show. It is an auspicious time to have you on because one piece of good luck is Waymo is making its entrance to Boston. There's been a few cars spotted around the city, and our Reddit boards are lighting up with spottings and all sorts of stuff. So welcome to Boston, even if you're not here.

Speaker B: Thank you. I mean, we would love to serve more and more people across the US across the world, and Boston is, of course, an extremely important city for us to be in.

Speaker A: I do want to talk about Waymo, obviously, but you've done a lot more than that. You were YouTube for quite a while during a very key part of that McKinsey consultant. Before that, could you just like, give us the tldr on, like, how did you get here and what were the key things that have led to where you are now at? Ah, oemo.

Speaker B: Yeah, absolutely. I've been at waymo for almost seven years now. More than seven years. Before that I spent some time at YouTube. I was working with the creator team, and before that I worked on my own startup and spent some time at, uh, McKinsey as the management consultant. And I think all of that did come together to get me to waymo, like 2018. When I joined Waymo, it was still much smaller and to be honest, we were still in a phase where it was not clear if this could actually become a new product. But as I was thinking about it, it was something that still cutting edge. And I mean, again, if we could make it successful, then it's definitely civilizational changing, right? Yeah. And so while I was at YouTube, I got to know about Waymo and got extremely excited about it. And now I'm here.

Speaker A: It seems like you were at YouTube at a time of huge growth for the company and a really major important thing, and it was just going really strong. How did you think through? Let's go to this startup, you know, very early strap, that point that, like you said, wasn't even sure if it would work. Sometimes you have tech companies like, oh, it might be hard, but it's going to happen. This is like, fundamentally don't know if the thing will happen ever or not. Yeah, absolutely.

Speaker B: I mean, maybe it was a, uh, mix of being extremely excited about working on this advanced piece of technology that could save lives. And then second I think I always have tried to go back and work on something going from zero to one and way more presented with that great opportunity to take it from 0 to 1. And maybe there was some navity on my part to say, you know, you don't fully understand everything, so sometimes you have to take a chance.

Speaker A: Yeah.

Speaker B: And I think I took a chance.

Speaker A: I feel like the world would be a very boring place if there wasn't a proper spot for a little bit of naivety and maybe hope every once in a while. Right? Yeah. You gotta kind of have that positive this thing could work and it's more likely to if I can help make it happen kind of thing.

Speaker B: Absolutely. I feel, to be honest, maybe Valley, Silicon Valley is one place where it is a lot more beneficial to be optimistic about the future.

Speaker A: Exactly.

Speaker B: And I think that's a great mindset to have here in the valley.

Speaker A: Yeah. That's what power so much has been like. Yeah. Is this possible? Who knows? But we're going to make it happen.

Speaker B: Yes.

Speaker A: So I'm glad you owned it, because looking at where we are now, you guys are in a bunch of cities. But also beyond that, the impact of driverless cars is potentially, like you said, huge. Because I think I talked to a lot of people who haven't really kept up with this, and to them it's like, oh, great, I get to take an Uber without a driver in it. Oh, uh, maybe I don't have to talk to a driver if I'm feeling, like, ornery that day. But it's so much more than that. It's. Humans are fallible by definition. They don't pay attention. Sometimes they get distracted. I mean, it's a huge, like, safety thing. Right. And more than that.

Speaker B: Absolutely. I mean, there are so many aspects. Of course, people like it for the privacy. Yeah. But there are also aspects of it being 90% safer than human drivers. Right? Yeah, there are, I think, if I remember correctly, 40,000 fatalities every year in the US because of traffic accidents. Right. And that's something that Waymo could really help improve. And not just that, I think also think about the improvement in accessibility. It provides people who have other accessibility challenges. Waymo is a great way for them to now be free, uh, get their own freedom. And so I think there are a lot more benefits to Waymo. And I do believe as we grow, as people use us, more and more people will find even more use cases.

Speaker A: Digging in. One thing that's interesting for you guys is across AI, there's this idea of evaluation models. Right. And probably, you know, let's be honest, if chatgpt, their model regresses a little bit, what's the worst that's going to happen? I'm going to have like an EM M dash where I don't want it, or a sentence might not be as exacting and compelling as I hope, but here, any shortfall in anything can be pretty serious. What was it like going from YouTube, which is very test heavy, you know, very digital. The stakes aren't that high from a life or death standpoint over to Waymo, where there's probably a very different process for evaluation. How do you guys kind of look at that and think that through and how do you still run tests when downsides can be bad?

Speaker B: I mean, that's probably one of the most important things I learned for coming to WEMO. So as you mentioned, at YouTube also, you launch something, you do an A B testing, you understand how users are responding to a change and based on that learning, you launch or you don't launch a particular feature. So in that sense you have testing across all the digital tools, all software. One difference of course at BMO is we are a, uh, physical AI company. Our products are not digital products. They are out there in the world. So it's not as easy or safe or cost effective to do all the testing on ground. So you have to have a system which is really well tested, uh, before it goes out. The second component of course is, and this is, I think becoming a lot more obvious to people post the LLM revolution. We understood it even before LLMs was that if you largely depend on machine learning systems, then you need to have extremely strong evaluation. And in some sense evaluation is your way to define what needs to happen to test if that is exactly what is happening. Because machine learning is not as logical or as deterministic as old software. So I think those two things are extremely important to keep in perspective. And that's why evaluation at Waymo is extremely important in making sure we can successfully deploy things in the physical, uh, world.

Speaker A: Thinking about that a little bit more deeper, how are you actually running evaluation? How are you running tests or quantifying does a new model is probably not the right word, but a new version, does it improve, does it not? Because you can't just throw it on the street. Are you guys running a simulation? Is there some giant Waymo video game? Basically it's really high quality or there's

Speaker B: not one thing that we do that's the other thing important to understand, right? So like in the whole development cycle, the testing is basically part of the development cycle. And it happens as, let's say, a researcher or an engineer are trying to improve something, they keep evaluating it. Then we have this big software that's ready to go out. We have to run huge simulations and testing to test that out. And then even in the physical world, before going full driverless, we'll have testing done with the, uh, drivers in the driving seat. And so there are multiple layers of testing, and you can think of it as like multiple layers of Swiss cheese, which is like a problem. Maybe was not caught at the first level, but will be caught second or, uh, third. And so if you have a lot of evaluation coming together, it makes it very easy to catch. As you're saying, maybe there is a regression that's possibly there or some issue that should not, absolutely not go in the physical world. We end up catching it long before. Right.

Speaker A: Are you saying there's kind of multiple potential evaluation paths going on and maybe something dangerous or risky gets caught on two of them, not one. But that's exactly how it's meant to be done is why you kind of have.

Speaker B: Yeah, of course, multiple, uh, you know, evaluation techniques might catch the same thing. Absolutely. Yeah. We do have, of course, an overall framework. But as I'm saying, you'll do the evaluation that helps you understand, you know, the behavior with respect to a framework. Also, you will do evaluation at various steps of the, uh, development lifecycle. And every step, the key point is to understand if I'm making a change, do I understand what it does? And so you need to have those feedback loops at various levels to help people understand, you know, developers understand, product managers understand, our external stakeholders understand how the behavior is changing.

Speaker A: What do those outcomes, uh, look like when you're saying we're trying to change this behavior or trying to improve X factor? Is it as blatant as we want to be better about stopping when someone's in a crosswalk, or is it more fundamental than that?

Speaker B: It could be at various levels, to be honest.

Speaker A: Yeah.

Speaker B: Again, the system is extremely complex.

Speaker A: That makes me feel better, though, than it is. It was like, no, it's like three decisions and that's it. I'd be really worried.

Speaker B: Yeah, exactly. We take decisions at various levels.

Speaker A: Yeah.

Speaker B: I'll give you an example of how do we go about, uh, evaluating. So let's say we have to make an improvement on how do you get closer to the curve while doing the pullover?

Speaker A: Yeah.

Speaker B: And so how do you do that? I think the simplest thing, of course, is we'll have Some old cases where we were not getting as close to the curb and so you can simulate over them. You say, I have this new change now. If I see how the software now performs on those old cases, is it getting closer to the curve? That's the simplest form of evaluation. And then you can have huge test sets. If try and do that, that can really give you confidence on how good the performance is now. And then you can do it at various levels. Again, the size of the test set could change. Maybe it's very small initially when the engineer is working on it, but maybe it's much bigger, uh, when you are actually doing the whole release evaluation because you need a lot more confidence on the capability. So I think that's how you go, you keep going. Right? And then some of this. As I said, we could use our own driven logs to test, but we could also create fully simulated environments to test it. We'll have various tools to create these systems for validation. They have their own use cases. It's like a matrix of using all of them at various stages, but based on the use case, we'll end up using the right tool.

Speaker A: It seems like almost everything now is an exercise. And pick the right tool for what you're trying to do, whether it be something, any kind of application you're writing that's leveraging AI. Uh, usually it's probably not just the biggest, best model. There's usually probably something specific that's going to be better. Has the growth and improvement in capability of AI helped advancement here? Because I have to assume at some point in history that parking to the curb was a very deterministic. Okay, as you pull up, calculate the exact angle of approach. And as you hit this and this, you probably had to be pretty strict about certain computation things. To approach the curb now is a little bit more approach the curb and kind of help a couple parameters, but it can be done a little bit more fluidly.

Speaker B: I'll answer two passages. First, like the close to the curve problem may sound like very simple thing.

Speaker A: Oh no, I parallel parked a lot. I think it's pretty hard.

Speaker B: I think again the real world is very complex. The scene could change a lot. Maybe sometimes you don't want to get close to the curve. You want to be away from the curve because there's something there.

Speaker A: Right?

Speaker B: Right. So like the complexity is what makes it harder for a simple logic to help. And that's where we need more machine learned systems which are more scalable, more generalizable. Even a situation changed a bit. A machine learned model will have a better answer to how to respond to something like that. Going back to your first question of did machine learning, all this evolution, machine, um, learning, help us? Absolutely. I think all these LLMs became a big thing, you know, a couple of years back, maybe. Was it around like 20, late 21, 22, I think VEM of course, has been using the same transformer technology for much longer. And we discovered that as we were using more of machine learned systems, we were seeing a lot more improvement in the behavior of the car. And so absolutely, as machine learning is evolving, as it's becoming more powerful, we do hope that we can leverage all of that to make even better drivers, even safer drivers, and more generalizable drivers.

Speaker A: If you drive a lot, you just kind of go on automatic when you drive. Right. Because I live in downtown Boston, which is notoriously hard. We just had a giant blizzard about two weeks ago and the snow has stuck around until m. I mean, I think it's still there. So there's still areas you can approach the curb. And if you do what would have normally been a good curb approach, basically the rider has to step out into two and a half feet of snow. But in other spots it might be easy. Those are all the little nuances that I think we as humans take for granted that we just naturally process.

Speaker B: Exactly. Human mind, again has learned so much. Some things that are very obvious to us may not be as obvious to machines.

Speaker A: Right.

Speaker B: But that's where machine learning comes into play. Hopefully learning from humans, like things that are easy for humans, kind of become easy for machines too.

Speaker A: Thinking about how you actually train these and get like, I mean, at some point you just need miles, right. To train the models. I mean the problem, most of driving, it's really boring. You don't have people crossing the street and cows and curbs with snow on it and people changing and falling and cars stopping last minute. 99 point something percent of the time. Is it just like this huge amount of data that just kind of isn't really that important? And you have to key in on certain. Like how do you get enough of the really important data to do this?

Speaker B: That's a great question. And so I was talking about, you know, building evaluation. There is also the evaluation aspect of all the driverless driving we are doing out. Um, there, as you're saying, can we figure out what are the more interesting aspects of driving? Because 99, maybe even 99.9% of the driving is pretty boring. Right. And so, I mean this is going back to something similar to evaluation to say, do we know when a situation is interesting? Are there systems that can figure that out? Right. And that's why evaluation needs to be important because you also don't want any problems on ground that you're not catching. Right. So you need very robust system evaluation that's not just catching problems before, but also is great at catching any things that happen on ground, which then helps us, uh, improve the system, helps us learn more. And then as you're saying, filtering out the most interesting aspects of driver. And I mean in some sense this is like you would have heard of this term, um, like the data flywheel approach, which is to say, okay, now if I have a way to find all the cases that I need to improve in an automated manner, then I can just find them, um, and then feed that back to my machine learned system and my machine learned system can improve on those. And so I think if you can create this whole flywheel to say machine learning system did something, I evaluate it, I think it's great, let's go, let's go on ground. And then I evaluate the on ground driving. And if there is something that we need to improve, I bring it back to my machine learned system in the right manner, then we'll improve. And then this is a flywheel that can keep moving and the faster it moves, the faster the improvement. And again, I mean this is more of an ideal picture. There's a lot of effort that goes into making this flywheel work. Uh, but ideally in the future, if you can automate large parts of this flywheel, then we are all talking about how fast machine learning is improving, right? The speed of improvement could be even faster and faster.

Speaker A: Well, that's what I was wondering is like is all the miles on the road, while a huge amount of it is boring, is it uh, fundamentally important? Because that's how you find the cases that you really need to work on. And so you need some level of miles to go find that. But then I guess I hadn't quite thought how important, like the simulation aspect of being able to just run those kind of things in a much more dense environment.

Speaker B: And both things are absolutely, both are important. Like think about it this way, we just like again, what we learn more and more is that the physical world is chaotic. Most of the time it's normal. But then there are these edge cases that only happen 0.1% of the time. So to even know about them, you have to drive a lot on ground. And if you keep driving, then you see all those situations and then you understand how your system is performing in them and you can improve on them. And at the same time you can take all those models and then use a simulation system to 10x the learning, 100x the learning. And so both are important. You cannot just have very few miles and then just try to learn from it because you have not even seen a lot of edge cases. But once you see them, then of course you can dramatically increase the learning from the same set. Through simulation.

Speaker A: You'd assume at some point you get to a level where it's no longer a ability to process and understand the road standpoint of what holds you back from going city to city, but more just a capacity of cars or something like that.

Speaker B: Yes.

Speaker A: But I do have to say I appreciate the leaning towards safety and be a little bit more conservative on that piece there because, you know, everyone has varying levels of how excited they are or you know, how trepidatious they are around just general, like driverless cars. But the more I have talked with you about it both today and previously, it does make me feel a lot better about just how much goes into all this and how much you guys are really, really evaluating and running through that before anything ever really happens that could be dangerous.

Speaker B: I mean again, safety is top priority. And so we would make sure we are investing in the right manner, going in the right manner to make sure our uh, product out there in the physical world is extremely safe. But the reality is that the evaluation and the understanding of the behavioral software requires us to look at many more aspects of driving.

Speaker A: When you say there's a lot of other things you're looking at from a multi layered evaluation there, what does that mean? Like how do you think about it from a more broad stance as you

Speaker B: look at something like that at a high level? In some sense it's very uh, easy and obvious of what we want to measure. Right. Like one, is the software working then? Is it working safely and within the constraints of environment? Then is the software working in the most efficient manner, in the fast manner? Right. M Then finally, is it working in a cost effective manner? So you'll have to look at, in some sense evaluation hopefully covers all of it. I keep cost effective on the side for now because like really the performance of the software is largely captured in the first three. Right. Which is, is it working? Is it working safely and within constraints? And is it working efficiently and effectively? And maybe you have a comparison there to say is it working better than humans?

Speaker A: Right.

Speaker B: So I think those are the areas we would want to focus on and This I think is true for I would say any AI agent out there of the day. You're trying to test the same thing. And so you have to test all these things and under each of them. Let's take safety as an example. Let's say we are doing a simulation, figuring out whether a new software is safe. An easy way is we simulate millions and billions of miles and we can actually find if there were any collisions and see how do you improve them. Uh, but of course if the software is great, you'll find very few of these. So then the second layer would be maybe we don't look for collisions, but you look for what you can call a denser signal, which is, let's say a close call, just vehicles getting close. They didn't have collision, but they got close. And that's uh, a very good metric to understand to say what could have led to collision. Right. And now what will happen is you'll find uh, hopefully more such cases in the simulation. And what that helps with is that helps the development team, um, the engineering team to find out what they could improve, which will then eventually lead to improvement in collision rate. And this is true for most of the things we work on, which is you'll have a very clear sense of what you want to get to. And then hopefully you can find denser, uh, metrics directionally represent the same thing. But then you'll have, because it's a denser metric, it's much easier to hill climb on.

Speaker A: When you look at overall, like beyond the concept of model evaluation for me, more like higher level, outwards facing stance. What is the goal of ride? Is it a, uh, safe ride? Is it a fulfilling ride? I don't know, fulfilling would be a weird word. Fulfilling ride.

Speaker B: Of course vimo's goal is to be the most trusted driver.

Speaker A: Yeah.

Speaker B: And of course to be the most trusted driver, we have to be extremely safe. That's always the key thing. But we also have to be very good at driving, which would mean multiple things to say. Driving should be smooth, we should have the right set of routing, good ETAs, good pullovers.

Speaker A: Mhm.

Speaker B: So I think a lot of it comes together and that's where I was saying, I think my evaluation becomes important is because it forces you to really concretely say what is it that matters to the users and can you measure it and can you improve it?

Speaker A: Does it get difficult at times? Because you know, if you think about how you build a normal SaaS company or something like that, where you have users, so much of what we Key in on, you know, here and multiple other companies have been at is we got this really strong feedback about how amazed they were about something or someone's mind was blown about this thing we were able to do. And it feels like great. Driving by a Waymo car might be similar to like in a sporting event when the referee is great. The best version of a referee in a sporting event is you don't notice them. They just do it right. And it goes smoothly and it's easy and everything is just good. Usually when you notice the ref, it's because they haven't done something great.

Speaker B: Yes, absolutely, I agree with you. I mean the best way more is probably that you will forget about. Yeah, maybe the first ride you would never forget about. But once you become used to it, it should just be part of your day to day life. There's no reason to remember using a Waymo. You have been driving a car, sitting in cars for forever.

Speaker A: Yeah.

Speaker B: So yes, I agree with you. I think the best outcome is it just becomes part of day to day life that uh, you don't even notice.

Speaker A: I can tell you actually factually, I still remember the first time I was ever in a Waymo. It was in Austin and I was taking video of the whole thing and sending it to my kids and be like, look at this, this is insane. And their, you know, minds were blown. But even the third time I got in one, it's still impressive, you know, on my phone talking about something else and just kind of. It's amazing how quickly you settle into it.

Speaker B: Absolutely. This is a good thing, right? At some point they were like, again, long back. It was like, well, would humans trust if there's no driver in the driver's seat? I mean, at basically, of course, you know, computers versus humans. Why computers would be better at driving. They're always alert, they're not distracted. They are doing one job and they're trying to do their, you know, do best at it. Just that basic reason is why it's pretty clear that Waymo driver, because it will keep improving and becoming better is the right way to go.

Speaker A: All right, I gotta ask you a couple quick questions because I got you from Waymo here and there's been things I've been dying to know. What has been the hardest thing to actually teach the cars to do that maybe you didn't expect? Like, what's the hardest maneuver?

Speaker B: Actually, I'll give you two examples. One, something that looks simple but is hard and one that I think even humans would find hard. I think one that Even humans find hard would be an unprotected turn because there is so much going on, there is a lot of oncoming traffic. And so you have to make sure, you have to be extremely safe. Also you have to make sure you're giving the right indication. There's like a lot of negotiation happening in that. And so it is extremely tough. And it's great that VIMO does really well at those. And that's where I think the machine and machine learning systems come into play. Because we can really understand everyone's intent really well. But it is a hard one. The one that let's say looks simple is. Again, I'll go back to the pullover one, which is. No, no, it's just, you know, you have to just go and pull over. But there are aspects that are very obvious to humans that machine learned systems are still learning. And this is, I mean, going back to like, why can't a, uh, robot just fold a shirt? Like, it's so easy for humans.

Speaker A: Yeah.

Speaker B: But there are some aspects which are very easy for humans that machine learned systems have to really learn.

Speaker A: Well, you think about pickups, it's not just you pull over. I have talked to so many Uber drivers and Lyft drivers and stuff. People get really, really picky about where they get picked up, where they get left off.

Speaker B: Exactly. And then Uber drivers learned that from years of experience. Right. Of how the riders behave.

Speaker A: Right.

Speaker B: Maybe someone at the corner just said, hi, I'm here.

Speaker A: Yeah.

Speaker B: And an Uber driver from the corner of the eye could see them and say, oh, I guess they are my rider kind of thing. Right. There are a lot of nuances about these simple things that we humans are very good at.

Speaker A: It's not even logical always. It's probably just emotional. It's just someone's going to be kind of jerky about where they get picked up and someone else is just happy to have the ride. So it's fine. Okay, last one. When you look at the human side of driving, what's one thing that you've seen that humans do when they drive that if they didn't do it would make training these cars so much easier?

Speaker B: Good question. I do think, I think a lot of our effort also goes into predicting what other humans would do on road. I mean, some simple things of, well, humans running traffic lights. And so our systems are pretty good at predicting that and then avoiding an incident. But that's of course takes us a lot of effort to understand that intent. But again, if humans were like robots where they were always following rules and driving really well, then our jobs would absolutely get easier. That's not the reality.

Speaker A: I'll be honest. I have, like, a hundred more questions I've written out, but I also realize that you have to go back to helping to build the future of driverless cars, and that's much more important than just answering a bunch of my questions. So, Jime, it's been great having you on, man. Thank you so much for taking the time. If people are curious or just want to get in touch and say thanks OR hi, is LinkedIn a good spot or is there a better place to reach out?

Speaker B: Yeah, LinkedIn is a great spot, and they can absolutely reach out with any questions. And yeah, I mean, it was great fun talking to you, Jeff. Always happy to talk more about Vimo. It's, of course, something that I deeply love. Great chatting with you.

Speaker A: Awesome. Well, thank you so much for coming on. It's great having you on. Have a good rest of your day and yeah, hopefully talk soon, man. Good to see you.

Speaker B: Good to see you, Sam.