Paige Bailey: Google Deepmind, LLMs, Power of ML to improve code | Learning from Machine Learning #5

When I had entered into my graduate program, I was told that it was a waste of time to do like computer science and programming, that I would never be respected if all I did was like build tools or build libraries or write code to solve science problems. How did the best machine learning practitioners get involved in the field? What challenges have they faced? What has helped them flourish?

Let's ask them. Welcome to Learning from Machine Learning. I'm your host, Seth Levine. Welcome to Learning from Machine Learning.

On this episode, we have a very special guest, Paige Bailey, the lead product manager for generative models at Google DeepMind. And at such an exciting time, this week was Google I.O. where they got to announce new advancements for Google Bard, talk about Gemini, and how they're incorporating many aspects of generative models into all of their products, really.

Paige, welcome to the show. Awesome. Thank you for having me. It's such a pleasure to have you.

You have such an interesting background. You want to just give a little bit of background, introduce yourself, how you got interested in computers and machine learning? Sure, that sounds good. So my background, I did geophysics and applied math when I was in school.

That was kind of the focus of my career. I think when I was younger, I wanted to be like some sort of lady, Carl Sagan, like focused on planetary science and sort of being able to explain these complicated technical topics to the world. I got into computers very early. I grew up in quite a small town.

And my family kind of rescued an apple, too, from being thrown away. And that was how I first learned how to program. And then really, I think this time in particular is kind of what I've been waiting my entire career for. I've been doing machine learning for a bit over a decade.

And previously, you would have to go through all of this pain to get the data, to build a model, to try to choose an algorithm, and then to do the really hard work of trying to get those models into production. And even then, those models were just kind of single task models. And today, we have these highly capable, very general purpose models that are doing an overwhelming number of things. And really, we keep discovering new ways that they're useful based on the way that people are sort of testing them out and stress testing their boundaries.

So it's been really exhilarating, honestly. I would never have imagined that this would happen back when I started doing machine learning in 2009, 2010. So back in 2009, 2010, what were the types of machine learning problems that you were working on? Oh, lord.

So back then, I was fortunate enough to do, I got some NSF kind of grants to do planetary sciences research both at the Laboratory for Atmospheric and Space Physics in Boulder, as well as at Southwest Research Institute. And most of the machine learning there was just kind of fancy statistics, right? So you were doing linear regression, logistic regression. You might be doing decision trees or support vector machines.

But really, it was just taking this very tabular data and attempting to make sense out of it. And it was useful, very, very useful, and capable of driving many interesting scientific advances, but nothing like what we have today. And also using sometimes very, I would say, very niche tools. So there was, in the space sciences, there's something called IDL.

There's also, I remember, many, many sleepless nights attempting to wrangle MATLAB into doing what I needed it to do, whereas Python always seemed to make much more sense. So just, again, like a sort of an explosion and a revolution in terms of the kinds of models that we can build now and then also the tools that are available to help build those models. Yeah. I love MATLAB, by the way.

Oh, wow. Yeah. A lot of my undergrad work, I use MATLAB. And one of my first big machine learning projects, I inherited the project.

And a lot of it was in MATLAB. So I started out doing a lot of work there. I think it's great. People don't talk about it enough.

Python has kind of eaten it up like a lot of other things. But I love MATLAB. Yeah. One of my professors always used to say that it doesn't really matter what tool you're using.

It doesn't matter if it's Python, a spreadsheet, MATLAB, SPSS data, or R, or whatever it happens to be. The important thing is that you're asking an interesting question. And you're being thoughtful about analyzing data. And the tool is just kind of something that gets you to the answer.

It shouldn't be the answer itself. Yeah. Yeah, definitely. That's for sure.

Yeah, I remember I did some very interesting work with SPSS as well. But yeah, it's not the tool, really. It's understanding the problem, understanding that, do you even have the right data to start approaching this problem? So yeah, so I guess at face value, people might not understand.

So you're in geophysics. How did you become the lead product for generative models at Google DeepMind? But I mean, it makes sense. There's so much dealing with so much data.

And I guess a lot of the lessons that you learn with how to handle all of that kind of data apply. But if you want to speak to that. Absolutely. So one of the cool things about geophysics is that geophysicists have been using GPUs for a long time.

And they've also been kind of the flavor of Earth scientists that are more likely to be delighted by computers as opposed to running away from computers to the mountains. So seismic data is very massive, as well as well-logged data, kind of the interpretations of subsurface data. And all of this Earth sciences data really needs heavy kind of computational horsepower in order to analyze it. So even before deep learning was a thing, Earth scientists were already building models and sort of attempting to analyze these patterns in the subsurface and in seismic using GPUs.

And that's something that I don't know if many people know. But I actually had started experimenting with CUDA, not just for deep learning, but because of Earth sciences problems and quantitative hydrogeology, and also understanding fluid dynamics problems. Yeah, that's fascinating. I guess having that experience with CUDA and dealing with that type of data, understanding how to process it set you up probably pretty well for your role with TensorFlow.

I know that you played a pivotal role in the development of that. Can you speak to some of your work with TensorFlow? I don't know if it was a pivotal role in the development of TensorFlow, but I certainly was delighted. So TensorFlow was, I've been experimenting with kind of open source machine learning tools for a long time.

I think I mentioned before, Python always seemed to make more sense than MATLAB. And there's kind of a really nice Python data science ecosystem, SciPy, NumPy, Matplotlib, Scikit-learn, like all of those great things. But then TensorFlow came out and was open sourced in late 2015. And it was kind of revolutionary in a few ways.

It had some documentation. It had some examples that you could use. I think all of those deep dream images went viral over the internet. But it was also in Python.

And even though it was a really kind of janky, weird sort of Python that you had to construct graphs to use it, it was still something that felt a little bit more approachable than maybe some of the other deep learning frameworks that were implemented in C++ or Lua or whatever they might have been. So I got very excited about it, started learning how to use it, trying it for my projects. My first deep learning project was applied to the Earth sciences, so understanding how to categorize different shapes of its holes and reefs and just being overwhelmed at how something would have taken a poor grad student six months to do.

And then suddenly, this was able to blast through it in just five minutes with a really modest amount of training data. But I started contributing. Eventually, the TensorFlow team kind of took notice. And I got to join them and to work at Google Brain.

And in addition to getting to work with the TensorFlow team, the JAX team, which is also an open source numerical library for doing deep learning in addition to many other things, especially in the sciences, I got to work with them quite closely. And then all of our machine learning frameworks teams at Alphabet. So like I said, it's very exhilarating. And it's been really interesting to see how the space has evolved over the course of the last many years.

Yeah. So fast forwarding to today and some of your more recent projects with all the things that are happening with these large language models, can you speak to the work that you've been doing with applying large language models to different software development tasks? Absolutely. So I guess this is also a nice segue.

So I'm boomerang back to Alphabet. During the pandemic, I spent a little bit over a year at Microsoft, specifically GitHub, helping with introducing machine learning features into VS code, GPUs in code spaces, and then also, of course, copilot. And so I think the potential for generative models in the software development space is huge. Historically, single task models for things like doc string generation or single task models for code generation or for code completions, single task models for build repair or for helping resolve or identify errors.

And now we're seeing models do all of these things and even more with just a singular model. One of the coolest things about the announcements that we had at I.O. this past week is that they're all using kind of our latest large language model, the technical paper is out.

I encourage everyone to read it. We also have a website at g.co. slash Palm2.

But this model, based on kind of the way that it was trained in it and the input data, it's capable of doing a broad variety of software development tasks. And it's supporting code generation, code explanation, error explanation, error fixing across so many alphabet products. So both are tools within Google Cloud. The product is called Duet, as well as all of those features within Google Colab, which is, if folks aren't familiar, it's kind of a data science notebook environment that's ready to use and that you can kind of have handy in your Google Drive instance, as well as Android Studio, as well as code, features, and BARD.

But it's all powered by the same model. And it's been really energizing to see how people have been testing it out and using it. And then also the features that we have coming down the pipes, so things like self-healing code, and then also things like tool use. So being able to access a Python interpreter with code.

Yeah, so I'm familiar with a lot of those tools. I think I was an early adopter for Google Colab. Yeah, I've loved it for so long, the ability to have free access to GPUs. In the past, it was a little bit longer access.

Now it's a little bit less, but that's OK. Just being able to, any practitioner getting access to a GPU is just like, yeah, it just changes your iteration speed. And you can kind of work so much faster. For the other things, I don't know why, but I was more of a late adopter for Copilot.

I got it within the last month, it's embarrassing to say. But that's when I finally got around to trying it. I don't know, I was hesitant. I just thought, oh, maybe it's going to introduce bugs or something like that, or maybe it was my pride or something.

I wanted to just continue to be coding on my own. But I started to use ChatGPT, I started to use Bard to help me with certain things. I mean, it's not like I wasn't finding myself on Stack Overflow like every other programmer looking at things. But yeah, it's incredible.

I think that this paradigm shift that's taking place, I mean, with machine learning in general, but for generating code, it's like what it used to be is you used to write a function, and then you would struggle to write your documentation or your comments and things. Now with this new technology to do code generation, you're writing what you want the code to do, and then it's making these suggestions for what the code should be. And multiple lines also, which is really cool. And it's not just trying to autocomplete, it's very smart.

At least it appears to be very smart. One of the questions that I have is, so yeah, there's a lot of code out there, just like there's a lot of information out there. GitHub is filled with so many libraries, but not all of it is battle tested. Not all of it is peer reviewed.

Code goes through code review, so it might not all be the highest quality. Or it's from, I mean, the way that things are moving these days, it's just from a year ago or two years ago, and it's using a different version of a library that has a different dependency or whatever the actual specific is. But I guess, how do you mitigate those sorts of problems? I know this is like a loaded question, but how do you mitigate those sorts of problems where you might be training on data that's either not the highest quality or out of date?

Yeah, that's a great question. So we have a collection of fine tuned versions of our large models for internal use only. So kind of supporting the software engineers within Google who are building the software that powers all of the Alphabet products. And these fine tuned models are fine tuned on Google 3 internal source code, which is many, many tokens of super high quality peer reviewed data that is the equivalent of an L5 suite or more.

And that moves the needle quite a bit in terms of making sure that the recommendations that we're giving, the explanations that we're giving are pretty solid. For the GitHub code, I can certainly attest. There's a lot of very low quality code on GitHub. A lot of it doesn't run.

A lot of it is using dated APIs or maybe have this very low process of somebody just push committed it to the repo without going through any evaluation from a peer. And so I think that if you are building these kinds of models externally, there's a lot of work that needs to go into kind of carefully curating and cleaning the GitHub data sets in order to make it solid for use. And if people are curious about this topic, I strongly, strongly recommend taking a look. There's a recent paper from the Hugging Face and Service Now team called Star Coder.

And they go through kind of with their big code data set like all of the things that they needed to do in order to get a higher quality data set to train. And it includes things like deduplication. I think they also were considering preferentially waiting code that is newer or code that comes from repos that follow software engineering best practices as opposed to code that might be from like a Python 101 student. And all of those kind of careful bits of attention to the pre-training data set move the needle significantly in terms of model performance down the line.

But those things, there are also other tricks that you can do like retrieval techniques or being able to do these kind of self-healing operations where you recursively apply the model to the output code to see if it would actually run and then like fixing anything that might be wrong or spotting any security vulnerabilities if there are any in the output code. But it's certainly like an ongoing field of research in order to understand what the best options might be. Right. So yeah, so I guess designing, I don't know what to call it.

I guess systems that help software engineers or machine learning practitioners sort of do their job, generate code, test code, help you write documentation, do all these things. I guess other than dealing with maybe outdated code are the things that we were just talking about. Are there any other challenges that you face when trying to design those systems? Oh, absolutely.

And I'm sure I'm going to list a few, but there are many more. And I think people are discovering even more every day. So one is likelihood of reciting code. There's something that we've implemented for BARD where if you generate code and a portion of it is verbatim identical to something that might be within a GitHub repo, we point you towards the GitHub repo and then also tell you what license it was under.

So whether it was Apache 2 or MIT, which are very permissive, versus GPL, which is not permissive at all and is something that you would probably, you would not certainly want to use for your business. I think there are also questions about security vulnerabilities and performance. So if you generate code, ideally you would want it to be efficient. You wouldn't want it to be something that would take 10 or 100 times longer, more compute in order to execute.

And you would also hopefully want the code that you generate to be consistent in terms of syntax and conventions with the code in your existing code bases. So all of these things are considerations that folks have to think about when implementing tools for their own users. And it's something that we certainly think very deeply about when orchestrating ML applied to software systems internally at Alphabet. Right.

So for Bard, this week you guys announced that the underlying model, I guess, was upgraded from POM1 to POM2. What is it you think about POM2 that makes it like a better base model? Is it more data? Is it whatever you can speak to about it?

That's a great question. So the model that we upgraded from, it wasn't actually POMv1. And POMv1 was a very, very large model that there's a paper about it so folks can go read if they're curious. But it was not ever used in production for Alphabet, I don't believe.

But the first model for Bard was a version of our Lambda model. The model that we have upgraded to is a version of POMv2, which was announced on Wednesday. And some of the capabilities of it are that it's much, much better at code, at math, at reasoning. It's also much better at multilingual tasks.

So POMv2 was trained on over 100 spoken word languages, dozens and dozens of computer programming languages. And as a result of that really robust and very diverse pre-training data mixture, it can do things like translate from one language to another. It can explain idioms and riddles. Even in different languages, it can translate from one programming language to another.

It can tell you if code might be vulnerable or if it needs performance fixes. It can generate code. It can explain code. It can write mathematical proofs.

It's just lots and lots of different things. We're discovering new uses for it every day. And one of the other most compelling features about this POMv2 family of models is that it comes in a broad variety of sizes. So everything from our smallest version, which can fit directly on a mobile device, to more modest sized versions that are still, despite being an order of magnitude or more tinier than the largest version, still preserving all of the capabilities and doing it just faster, more efficient, and cheaper.

So it's definitely, from a business perspective, the POMv2 family makes a lot of sense. Yeah, that makes sense. Just to dig into the idea, because this is something that you see across a lot of large language models, you see Llama that has four maybe different sizes and obviously POM as well. So the reason for the different sizes, the different number of parameters, is it just to deal with the different trade-offs of where you're running the model, what your trade-off is for latency and performance?

Is there anything else that goes into it? That's a great synopsis. So the smaller versions of the models make it easier to serve in a broad variety of locations. It also makes them much, much quicker at inference.

And then the kinds of capabilities that we've been seeing from these smaller models that were open sourced is you can have very, very modest sized models. And as long as you fine tune or instruction tune on very high quality data sets, you're capable of getting the exact same performance that you would from a much larger model, despite it being faster, cheaper, easier to deploy. And I personally, I think that's the most interesting field of research right now is trying to take these highly capable models and make them more accessible for people all over the world, even if the only device that they have to work with is a mobile device.

Yeah, that's something that I'm also really interested in. Training like teacher-student models and distilling information from these large language models to get them to be stored into smaller systems. Because yeah, it really depends on your use case. Sometimes there's just something really nice about being able to run it on your laptop.

And a lot, I mean, almost all of these models, it's just like it's not even possible. You have to access it through an API or you have to access it through the interface that the company offers it. But yeah, I think that there's something really nice. Like even, I mean, I'm doing some of my work.

Like I'm trying to get things that are like 400 megs down to 40 megs, just because it gives you the ability to like run maybe five or 10 times the amount of things in a much faster iteration time. So there's been some like dropping of some ideas about the new release, the new model that I believe it's DeepMind with Gemini. What can you tell us about Gemini? So many of the people from the Palm B2 team are on the Gemini project, including myself.

And we're still actively training the model. It's intended to be Alphabet's most compelling model, which kind of tracks with, every time we build a model, we hope that it is a super set of the capabilities of the models that preceded it. But Gemini is very special in that it was built from kind of the ground up to be multimodal. So we're already seeing multimodal features in the first versions of the model that we've trained.

And we're anticipating that it will be in production very quickly, or at least the smaller versions of it will be in production quite quickly. But I guess the only thing that I can say specifically is stay tuned. We're very excited. It's the first model that Google DeepMind has trained kind of jointly together.

And it should be particularly compelling for not just the text and the code use cases, but also the multimodal use cases. Yeah, that's very exciting. I'm looking forward to it. In regards to multimodal, what multimodal use cases are you most excited for?

Yeah, so multimodal, I love this idea that you can have audio, video, images, text, code, as inputs, including many of them kind of being interspersed together. And then kind of define what your output should be, either by, you might get some text as output from your original model, and then you stack a diffusion model on top such that you can generate an image back out, or you can generate audio or video. But some of the use cases that I'm most excited about for multimodal models is that you can imagine, say you're taking a physics course, and you just can't nail a concept.

Like you just, for whatever reason, like angular velocity just isn't clicking. And you could easily just ask, like, hey, I really, what I would love to have is, like a video that explains angular velocity to me. And I really want it to just have like cats. And then I also want you to have like a quiz to check my comprehension every like one minute of the video.

I don't want this video to be longer than four minutes. And then I also want to have like an outline at the end that explains what the concepts were in the video. Also with images. And that's something that a multimodal model would be able to support.

So just being able to do something like that is huge. And it also brings about this world of, super, super tailored custom materials for folks. Like you could imagine each kid gets a new bedtime story every night. That's complete with, you know, a new story, a new adventure and new images.

It's like the, I'm not sure if you've ever read The Diamond Age by Neil Stevenson, but it's like having the primer just kind of like available for every person. And that's, it's really energizing. That's awesome. Yeah, I think that that's such a cool use case because, yeah, everyone's unique.

Everyone learns in their own, in their own ways. People respond to different mediums, you know, differently. So being able to tailor the learning approach, that, yeah, that could change, that can really change the way that we learn, the way that we interact, the way that we interact with the different, you know, material that we're trying to learn. That's really cool.

Yeah, I saw a tweet the other day that someone with the, with the, that someone with the, with the learning disability, they didn't share what their learning disability was, but they mentioned that interacting with these generative models has sort of, you know, made it that their learning disability isn't hindering their life as much anymore. You know, they're able to, you know, instead of having to attempt to digest all of this content, even though it's not architected in a way that is, that's optimal for their learning style, they're capable of working with generative models to consume the information in a different way.

And it clicks, and it's the first time in their life that they had said that it felt like that. And that's huge, right? Like being able to, being able to, you know, unlock the joy of learning for people who had previously been frustrating. That's opening up the world to so many more creators and so many more potential engineers and folks that, you know, can contribute.

Yeah, absolutely. That's so rewarding. It's so nice to hear, you know, especially now with all of the people that are sort of bringing up all of these negative potential future use cases for AI, but to know that there are all of these use cases where it can help level the playing field you know, in some ways and open up opportunities for people. You know, things are changing, things are changing rapidly.

And machine learning does have this tendency to perpetuate a lot of the things from the past, but it's nice to know that there are times when it can actually increase accessibility. So that's a nice use case. One of my favorite stories, one of my favorite stories from the copilot days were somebody, usually on GitHub, when somebody files an issue, it's like not a fun thing. You know, it's like, man, this is broken, or like, I'm confused or whatever.

But there was one person whenever we had first released copilot who wrote and said, you know, like, I have tremors, like, and they've gotten so bad recently that, you know, I was forced to, you know, stop doing my job and this person had been a software engineer. And they said, you know, you've introduced this kind of like speech to code, basically, generation features within the IDE. And, you know, it's made it so that I can actually build software again. And I never thought that I would be able to do that.

And that is also, you know, not late. That's the kind of thing that makes you delighted to come to work every day, I think, is, you know, the potential of making it so that people can do the things that they love, even though they might be physically limited in some way. Right. Yeah, knowing that you're having a positive impact on that person, I mean, who knows?

There's probably many other people that are also, you know, getting that positive impact. So yeah, that it's definitely delightful. It's rewarding to hear, you know, working in this field. Thanks for geeking out with me, talking about large language models and all of that.

But I wanna switch and zoom out into just the machine learning field in general. So what do you believe is an important question that remains unanswered in machine learning? There's so many, right? And I think one of the questions that I'm most interested in is how do kind of the pre-training data mixtures for large language models, how do they impact performance on downstream tasks?

And this is an unsolved problem. There's this notion of like, oh, well, I wanna do code stuff. So like, perhaps I should have more code or I wanna do multilingual stuff. So perhaps I should have more multilingual data, but nobody knows how much, nobody knows how much quality impacts that performance for data.

Nobody knows like how the data should be structured or formatted or if it should be included in a broad variety of ways. Like if you want to predict edits, perhaps you should have like code dips, right? Like instead of just the source code itself. So all of this experimentation ends up being pretty expensive, right?

Like, and people end up taking like really expensive risks. So for Palm V2 as an example, the input pre-training data mixture included an awful lot of multilingual tokens, which meant that the number of English tokens was much lower. And there was a risk from the team that like, perhaps we'll train this model, this like super expensive, like millions of dollars a model, and then we'll get something and it won't be as good at English tasks as it is on all of these other things.

Right. And that was a risk that the team took. And it ended up paying off because the performance ended up being better actually across all of the task ranges. But that could have been something that turned out very differently.

And I think for people who are interested in the large language model space and the deep learning space, understanding how the data choices that you make, and then also the quality of the data that you make impacts how your model performs is really, really compelling and important. And then also how does that, like the pre-training phase and the attention that you pay in pre-training, how is that compared to like downstream fine tuning and instruction tuning? Because we're increasingly seeing that the fine tuning instruction tuning RLHF portions are much more impactful and move the needle much, much more than everything else.

Right. What's your intuition on that? Why do you think it has such a big effect? So I think it's because it helps the model focus, which is not a technical way to describe it at all, but I think it's intuitively maybe a little bit easier to understand, right?

Like the model, you initially train it on a lot of data, like the entire internet, right? Or a lot of data. And so it's capable of doing a broad variety of things. Whenever you instruction tune it or fine tune it or RLHF it, you help it kind of focus on the kinds of questions that you're really interested in answering or the kind of format that you would really, really like to see.

Like say, your model is giving relatively short outputs and you want them to be a little bit more long form. You can kind of tune that with RLHF such that your model is kind of rewarded whenever it outputs longer context things as opposed to shorter things. Right, so you can kind of tune it to how you want those outputs to be. Yeah.

One of the things that you mentioned before, it made me think about like, well, two things. I guess how there are these different waves that have happened, I view in machine learning. Like I think there was a time a couple of years ago where it was like, you need to have a fine tune model on this task, right? And now with all of the advances in generative models, it's like, oh, maybe there's one model that can kind of answer all of these questions.

But there still is this question about, can we use that generative model to then point you in the right direction of the best tool to use to solve your problem? I guess, what's your take on that and how have you viewed that sort of transition between having a generative model that can kind of solve many problems versus fine tune models that are for specific tasks? Yeah. So we found just anecdotally, so initially for some of our large models that perhaps weren't pre-trained on as much source code, we did have to have fine tuned versions of those models using source code.

And then for future iterations of the larger models, we included all of the tokens that we had used during the fine tuning process within the pre-training data mixture. And the resulting model exceeded the performance of the fine tuned model based on those kinds of inclusions and then also using a tokenizer that was a little bit friendlier for code. So practically we've seen that the generative models provided you keep adding in high quality pre-training data, they can sort of absorb the tasks from the fine tuned models.

But I also will say that for businesses, whatever solves your problem, like that's the thing that you should use. And Lord knows the world is like running on Excel spreadsheets for much of the finance sector. And the government is running on COBOL. And like the cost of migrating off of either of those things is just overwhelming.

And I don't wanna be the one causing like the financial downturn of like telling the finance community that they shouldn't be using Excel spreadsheets or basic or whatever it is. So I would say that whatever modeling meets your needs is what you should use. And certainly experiment with generative models, see if it makes sense. And then also see if it makes financial sense.

Cause if you can get by with a smaller model that perhaps is open source and easier to maintain, then why should you be paying for API calls? But that's personal opinion. Like new technology is always going to be very cool and push boundaries. But at the end of the day, what matters is your business, your users and their problems.

Yeah, absolutely. I agree 100%. Identifying the problem, the underlying need, knowing that there's many approaches. I view that generative models are one tool in your toolset.

You don't necessarily need to use it for everything. It has a lot of amazing use cases, but yeah. Certain business practices like you were mentioning are, I'll say stuck in there, stuck in there certain ways. And we're moving towards this new future, but everyone's not ready for the leap over the canyon.

I don't know. There's a place, there's a space and we have to get people over. And some people will take smaller steps than others. Yeah, and there's also like some appetite for risk.

I think you have to have if you're preferring generative models, just given that we're still exploring factuality and being able to verify that models are returning correct responses. Cause a lot of the time they don't. Like there are always examples towards like counterfactuals. But the one thing that I saw on Twitter earlier this week, cause all of these generative models things seem to be percolating to Twitter.

But someone instead of like consulting with a financial analyst, they were like, I have some money and I wanna invest it in some stocks, like recommend some stocks that I should buy. And they got Chad GBT to do that. And then they purchased the stock. They kind of mapped it out over time as to what would have happened if they followed the portfolio recommendation versus the buy some stock recommendation.

And the Chad GBT recommended stock purchases or the generative model recommended stock purchases actually ended up doing quite, quite well compared to the kind of investment portfolio recommendations. But all I could think was just like, I don't want my 401k to be at the mercy of a generative model, at least not now. Like it's, and it might be that perhaps these investment strategists are recommending certain things because they know that it's more stable than perhaps these stocks that might be in the near term, high performers, but long-term, like much more variable in terms of return.

So it's like choose your own adventure and be very careful for betting all of your money on generative models. Yeah, I mean, yeah, in terms of financial decisions, I think, yeah, everyone's sort of like, how much risk are you willing to take? And then there should be like another step, like, are you willing to take the risk of using a generative model for your financial portfolio? Maybe some people will say yes, who knows?

Okay, so switching into like learning from machine learning standpoint and just sort of general advice for people in the industry. Well, I'll start with this one. Who are some people in the machine learning field that influence you? So there have been so many, I think, and I'm fortunate to work with many of them every day.

So I've been coming from kind of the open source tools, developer tool space. I've really loved working with the JAX team. So JAX is for folks who might not know, it's an open source library for doing, kind of building deep learning models, but also doing kind of scientific experimentation and all of the models that we build at AlphaBet are built using JAX. Matt Johnson, Peter Hawkins, James Bradbury, like they are all, Sky Wonderman-Milm, they're all very, very close collaborators, Roy Frosting as well, Yash Kataria, and they've been delightful to work with and to learn from.

I've also really, really loved working with Jeff Dean, as well as the collaborators at DeepMind. So as part of this Gemini effort, I've gotten to work more closely with Oriol Benyals, who's kind of driving much of the efforts. And then also, of course, the learning for code folks, who care very deeply about machine learning as applied to software systems and think very carefully and thoughtfully about it. And if you're on Twitter, I would also highly recommend following the Hugging Face team, because they have a lot of passion around open source machine learning and deep learning and sharing their knowledge, as well as Andre Karpathy, who is a dear, sweet human just generally, but also cares a lot about education and advocacy and making sure that these concepts are understandable to everyday humans.

So there are a lot of people to admire in this community. I feel very fortunate. Yeah, absolutely. Yeah, there's so many great people in the field and people are pretty generous with their time also.

And there's this sense of wanting to help other people out. And it's a really nice community to be in the machine learning community. On your website, which is nicely named Dynamic Webpage, right? Is that?

Oh, so my website, I have a tendency to purchase URLs that are puns. And I think probably most of us in this industry are like URL hoarders, but the one that might have, I think the URL might have even expired, but page views is the one. Okay, so under a section that I saw on one of your websites, it was under page views. You had some really interesting tenants.

I'm gonna read them if you don't mind. Yep. Yep. So bring data to opinion fights, relentlessly ask questions, communication is everything, especially in open source, choose growth opportunities, nurture and build communities.

If you don't have documentation, you don't have an MVP, give without expecting a return and believe in people, not acronyms. I love these. I love them so much. Thank you.

Are there any that you, is there anything that you would add to any of them or anything that you wanna talk about with any of those? So those, I think I put down on paper about half a decade ago and they still seem like pretty good life roles. I think now, especially the believe in people, not acronyms, one is going to become even more compelling and important. So COVID kind of turned everything on its head, right?

Like I personally, like it helped prioritize things for me. It was part of the reason why I left Alphabet. Like I wanted to move back to Texas to take care of my parents. They're like 70 plus, 80 plus.

And I think a lot of other people also, it helped them prioritize what they should be spending their time on. So we saw a lot of kids choose not to go to college and to perhaps enter into the workforce or to perhaps like start building software or to build their own businesses, right? And these kids are incredible. Like they're doing such amazing things.

Like the most killer use cases for generative models are all these kids that are like entrepreneurs and somehow like 16 years old, right? And so I feel like, and then also in the deep learning community, we're increasingly seeing that folks aren't really publishing papers anymore, right? Like whereas previously there was this academic ivory tower of like, oh, you do some research, you produce a paper, you present to the conference, and then you add it to like this extensive scholarly pedigree.

Whereas now it's just kinda like, well, I did some work. I produced something that people can test out. I will perhaps write a blog post, but for the most part, it's just like something cool that's out in the world that people can like see and touch and experience. So I think that if you don't have a college degree, if you are debating whether or not college is right for you, if you're concerned that like perhaps not having a degree will limit you in some way, whether it's an undergrad degree or a graduate degree, or if you have like a journalism degree and you're like, but can I do these work?

You can, there are no rules in this space. Like, you can do anything that you would like to. And the important part is just kind of focus on building something great that delights you and that delights other people. And that's all that really matters.

That's really good advice for people starting out in the field. A little other, another variation of that, when you were just kind of getting your feet under you, or I mean, I don't know if that is even the case, but what advice would you give yourself when you were starting your career in data science? So I would say, I would just kind of reinforce, do what you think is right and what interests you, and then don't feel, give yourself permission to do that and don't feel guilty or like you aren't, like you aren't meeting up to the expectations of your peers or your academic advisors.

I was told when I had entered into my graduate program, I was told that it was a waste of time to do like computer science and programming, that I would never be respected if all I did was like build tools or build libraries or write code to solve science problems. And I was also told that machine learning would never have a place in the earth sciences, which was in hindsight, kind of silly. But at the time I was just like, am I making the worst career choice of my entire life?

And I had a lot of internal emotional angst about it back then, but now it's definitely the right choice. If you feel like something is important, sometimes you can see the future, even if other people are standing, facing a different direction. That's great. Yeah, it's so interesting, right?

When you're in the moment, things can seem so chaotic and there could be so much uncertainty. And then, I stop myself, like the things that you're saying, they're like looking at things now, it's like some of that's like laughable. Like, you look now, I mean, hindsight's 2020, of course, but it's just, I mean, machine learning has become so pervasive and using all of these sorts of techniques and technologies. But it's amazing how, yeah, in the moment, it can be so uncertain, but then when you look back, it can become so clear.

And that was just like eight or nine years ago, by the way, that people were thinking that machine learning wouldn't have a place in the earth sciences, wouldn't have a place in the physical sciences. All of this is super new. And so I think that given the trajectory and kind of the exponential increase and how these things are progressing, it's really tricky to predict the future. And there's, I forget who said it, but like, and it's kind of cliche, I guess now, but the best way to predict the future is to be the one building it.

Like, it is very clear that the future is going to be built using the generative technologies and assistive technologies and productivity enhancing technologies. So if you're doing that, you're probably gonna be heading in towards the right direction. Absolutely. So speaking of how fast things are changing in the last decade, in the last five years, even in the last year or week, how do you stay up on all of the newest techniques?

I mean, I know in a way you're part of it, you're building the future, but there's so many people that are releasing stuff. Do you have any techniques to sort of stay up on top of everything? It's a really tricky problem. I would, and again, this is gonna sound silly.

I would be on Twitter just because the machine learning community seems to hang out on Twitter and they talk about things that they find interesting. It's like getting to sneakily listen in to all of the hallway conversations in all of these AI research labs. And increasingly, we had mentioned it earlier, but people aren't writing papers anymore. And so the best insights that you can give into kind of pedagogy and kind of how these models were built is by hearing what people are currently focused on.

Like, is it long context? Is it this new tokenizer that seems interesting? Is it some sort of algorithmic advance that people are really paying attention to? Is it the data mixtures?

Is it some other model architecture? But the kind of honing in on where the conversation seems to be happening is really, really important. And right now, if you're just looking or attempting to look at every deep learning paper that's posted on archive, your life's gonna be insane. Like there was a website a while back, I think it was created by Andre Carpathy, the archive sanity website, and it would post the paper and then have like a really nice kind of images with PDFs.

But in the last three or four years, it's gotten overwhelming to keep up even with that. So I would strongly encourage you, pick people that work at the AI research labs that you care about, Anthropic, Google DeepMind, OpenAI, and perhaps some of the others, like Hugging Face that I had mentioned, and follow them, see what they're talking about, see who else they're following. And then also don't be afraid to ask questions if something isn't clear, because a lot of time, if you ask a question to these people on Twitter, they will respond, and that's huge, right?

Like being able to have access to some of the folks that are building the future is massive. Yeah, absolutely. One thing that I didn't get to touch on, so yeah, obviously machine learning, AI, generative models has really made it into the mainstream, and it's created this like frenzied hype. I'm wondering from your perspective, how do you view the gap between the hype and the reality of AI?

Yeah, and this industry in particular, there are all these hype cycles can be overwhelming. I personally cannot wait for the AI hype cycle to be overt so that we can all have a little bit more peace to do the work. I miss when AI was not cool, because like, and NeurIPS was a lot more fun than two. Like all of the academic conferences were much more chill.

Now it feels like there's more VCs than researchers. But I do think whenever you're trying to distinguish between who is like an AI influencer versus who is someone who does this for their day job, like definitely look and see where they work, look and see what they've built. And if it's somebody that decided to get excited about AI, just about the time that web three went downhill, because like somehow a lot of these web three influencers have turned into like generative AI influencers, and I don't know how that happened.

But just kind of like look at the backlog, see what the person has accomplished and what they've been interested in. And if they haven't been doing AI for longer than the last year or two, then I would take what they say with a grain of salt. Yeah, absolutely. I think it's really good advice.

That's why it's really nice talking to somebody like you, who has experience at Microsoft and GitHub and Google and Alphabet and DeepMind and everything like that. The last juicy meaty question, what has a career in machine learning taught you about life? That is a wonderful question. I think what it has taught me the most is what to appreciate about being human.

Like the more you see what these models can do, the more you can understand what they're doing. And it's very exciting and it's very cool. And it's like discovering new capabilities every single day, new ways in which I can automate myself and sort of remove the tedious parts of the day that I like creating meeting transcripts or emails or drafting a paragraph for a doc or something. But also seeing the ways in which it falls short.

Like it's not going to be able to give you a hug. It's not going to sort of understand, at least not yet. It's not going to understand that you might have a bad day and that it should ask you, how are you doing? It's not going to do those things that are uniquely nice experiences of interacting with humans in real life every day.

And I think that's important. And I think one of the nicest possible outcomes for generative AI becoming ingrained in every person's life is that, it helps us appreciate more what it is to be a human and to interact with humans. That's really nice. Yeah, that's beautiful.

So by exploring the capabilities of machine learning and AI, you can really appreciate the human connection. Yeah. So for folks that want to learn more about you or the work that you're doing, what would be a good resource for them? Cool.

So I strongly recommend taking a look at both the Google AI and the DeepMind websites, though they will probably be merging at some point in the near future. I'm also chronically available on Twitter. So twitter.com slash Dynamic Web Page, and I'm Dynamic Web Page pretty much everywhere else on the internet.

And then I also strongly, strongly encourage everyone to just kind of get involved. There are certainly ways to begin learning more about the AI community without necessarily having to have a career in it. And then if you do want to have a career in it, I think it's going to become increasingly possible as businesses seem to be adopting AI at a much quicker rate these days. So don't be afraid to pitch to your boss if you have an idea and you want to create a prototype.

Absolutely. Paige, it has been such a pleasure. I really appreciate your time. Thank you for letting me pick your brain for this time.

Thank you so much. No, thank you. This has been delightful. I am so glad to have gotten the chance to chat and thank you for the awesome questions.

Really appreciate it. Thanks so much. Cool, cool. Thank you for listening to this episode of Learning from Machine Learning with the remarkable Paige Bailey, the lead product manager for generative models at Google DeepMind.

Her work is pushing the boundaries of innovation with Bard and the soon to be released Gemini. Don't miss out on the valuable resources in the show notes. Please leave a review, share it with your friends and let's create a community of continuous learning. Until next time, keep on learning.