Some Changes at The Gradient

Speaker A: Hi friends. If you're listening to this, this is Daniel, uh, your usual host. I'm here with Hugh Zhang and Andrei Korenkov, two of our gradient founders. And we are gathered together because we have a couple of updates for you about the gradient that uh, we wanted to convey. Um, in summary, a couple of things are going to be changing and we wanted to take the opportunity just to talk a little bit about how we all got involved in the gradient, what this whole thing is, because a lot of you have joined us recently and, and then talk about what's going to be next for us. So maybe Andre, if you want to get started, um, you and Hugh are probably the best people to talk about the history of the gradient. So if you want to tell us a little bit about that, maybe we can get started there.

Speaker B: That's right. Yeah, I guess I can throw, uh, in some things in a way I'm part of history partially as a person who created a podcast. So maybe some listeners from long ago remember me from that. But yeah, me and Hugh have been around basically since the inception of a gradient hue slightly before me and that was what, like 2018? So it's been over six years. Yeah, since it started, uh, as any listeners or readers of a gradient may have noticed, it's been a bit of a slower phase for the article side of the gradient. We really have not published that much the past year or so. And that is, has to do with a lot of factors. Partially all of us graduated from grad school and are now working full time jobs, which makes things a little bit challenging. So the announcements will have to do with that. Uh, and I guess we can get into the background and stuff a little later in detail, but that's the preview. And uh, maybe Hugh, you can frame how the gradient got going in the first place.

Speaker C: Sounds good. I mean, um, the grading was started by six people at Stanford who were just uh, at the time in 2018, Andrej Karpathy had been blogging and blogging had been really starting to take off in ML. And we were like, wow, it would be really nice if we could just create a collection of the highest quality posts and sort of give publicity to people who had really good things to say but maybe didn't have network or Twitter following at the time. Uh, to just have their own reach. Um, we would just collect everybody's posts and then sort of edit them and share them with the community. And that was just the idea and we just kicked it off from there. We were very lucky to get a bunch of really, really cool articles, um, at the very beginning, and then just sort of things just went from there. Um, and over the years, it's sort of evolved a ton. You know, we also started the newsletter and then the podcast. Um, but over the last six years, it's been, like, phenomenal. Just working with everybody and just helping share people's machine learning posts with the world.

Speaker A: Yeah. So if you've been following us along any of our avenues, the newsletter, the podcast, the articles, um, we all really want to thank you for doing that. This has been a very meaningful part of all our lives, I think, in many different ways. And so, um, it's great to see that people just get something out of this. I think that really for all of us, this is kind of about making something that people can learn from, that means something to people. Um, and so we want to talk a little bit about the changes that are kind of happening to the gradient. And this is primarily on the side of articles. As Andre mentioned, things have been slowing down. Um, and that is in part because we are all having busy lives. I'll make a slight amendment. We didn't all graduate grad school. Actually, we all have, um, slightly different relationships to grad school. Maybe we can elaborate on this later. Uh, Hugh is laughing really hard right now because he has a very interesting story which he may want to relate later. I won't force you to do that now, Hugh. Um, but the way we're going with this right now is, for every article you see on the gradient, this requires quite a bit of effort from either one, maybe two or three editors. We have a pretty high bar for the stuff we put out on our site. And so this often takes the form of going through drafts and quite a few back and forths between an editor and the person who wants to write the article. And up till this point, we've been accepting pitches for unwritten articles. So you can pitch us with just an idea for an article, maybe a couple of thoughts. And we really try to work with you to flesh that out into something that can go up on the site. This takes a lot of work, and at, uh, this point in time, we are pretty lone editors. We are also not an organization that really has any serious form of funding. And so it's a little bit hard for us to put the time we want into this kind of thing. So what we decided to do at the moment is we're going to go for a period of kind of limited activity right now. And what that M means is we won't be accepting pitches for unwritten articles anymore. But we will consider posting full length fenished drafts if you have things that you want to submit to us. If you have something on your own blog, we would love to amplify it if it seems like it's a good fit. Um, really the way all of us have been seeing this is we considered fully sunsetting the magazine, but we really think that the gradient still can be this place where, as Andrej said, if you're somebody without a big Twitter following of your own and a lot of people read your blog, we want to be able to amplify your work to the ML community. And so this is the best way. We think we know how to do that.

Speaker B: Right, Exactly. And, uh, I guess just aside from editing, we have a problem. And this is going out to anyone who submitted a pitch. Probably haven't gotten back to you since, uh, you did that, or we might have gotten back to you and then never followed up. So unfortunately that is another factor. Just having people who are kind of responsible for all the communications, all the back and forths has been challenging. I tried to do that for a little while and didn't do a very good job. So we made the call that, uh, it probably makes more sense to limit activity, to accept fewer pitches, but pitches with more finished, uh, drafts. And part of the reason why it's not just that we don't have kind, uh, of a capacity to do this with that much kind of consistency. It's also because things have changed to some extent since 2018. There's been, uh, much more avenues for people to do their own writing on things like substack. Things have grown. You know, it used to be people had their personal, uh, blogs and they did post their own articles, but it was somewhat niche and not that common. I think it has become more common and many people have started their substacks and started, uh, putting out really nice pieces. So the gradient is this sort of, uh, centralized place to publish makes a little bit less sense, especially if we don't provide that editing service. So the other thing that's worth mentioning is the newsletter will continue as before, pretty much in the same format, but we will be adding a new section that we are going to call something like Best from a Community or Highlights from a Community, where you can also pitch us or just tell us of things you have published on your own blog, on your own website and your own substack. We will go through those submissions, see the ones that seem like the sort of stuff that would be on the gradient, like really good, thoughtful content that would be nice to amplify to the, uh, community. And then we will amplify it through the newsletter and I guess probably also through our social channels, rather than, uh, posting it on the magazine, which I think serves the same basic purpose as the magazine did in the first place. Uh, and that's part of the reason we do want to keep it going, but just slightly less active than before on the magazine front and on the

Speaker A: podcast front, where you're, uh, hearing this now. I will be continuing to do this as normal. As you've noticed, I've slowed down quite a bit recently, and this has just been because of things going on in my own life that have made it harder to do this every single week. Um, the plan right now is to continue doing this roughly through the end of the year, a little bit into 2025 as normal. Um, and you should all look forward to this. You'll be seeing my usual end of year episode with Nathan Benesh from Air Street Capital. We'll be going over the state of AI report and everything that happened this year in AI and after that, a little bit. The plan is for me to transition that a bit away from the gradient into something that, uh, still has its roots in AI, but isn't going to be entirely AI focused. And if you've been following for a while, you've probably noticed some of these shifts already. I will be announcing more about what that looks like in the coming months.

Speaker B: And one last thing, I guess maybe many people don't know of this, but the gradient does actually have a Mastodon server that has been around since, uh, late 2022, I think. So it's been almost two years now since we started that thing, and there's still a fair number of active people on that. Mastodon Sigmoid Social is what we call it. And we will, you know, keep that going. We are still moderating it, still kind of keeping an eye on it. There's a fair number of active people on there. So, you know, as long as there is money to keep it alive, we will. No changes planned on that front. So I think that's pretty much all the announcements on the front of a gradient. Not too much is going to change. It's just kind of a structure of how this works. And what you can expect from us is becoming a bit clearer. And, uh, we'll probably implement the changes of how you can pitch us your published blog and so on, uh, pretty soon. So you'll be seeing that go out.

Speaker A: And with that, we wanted to use the rest of this episode just to tell you a little bit more about what the gradient has been over time. A lot of you as I've said, have joined us pretty recently and so you might not be familiar with the full history of what this thing is. As Andre and Hugh mentioned, they have both really been there from the beginning. Uh, I joined in around 2020 when we were just getting the newsletter and podcast started and so it's been quite the time since then. A lot of interesting things have happened in the time that we've all been working on this. Um, I guess Hugh, do you want to go into a little bit more detail about how the gradient got started, where things went, what this all looked like?

Speaker C: Sure. Uh, I mean at the gradient at the beginning it was just some sort of random chat. Uh, uh, we just sort of. There were five undergrads and one grad student, Andre. Um, and we just sort of uh, discussed the ideas possibly at like a party or something. I don't know exactly how the idea evolved but at some point we were like we should start a blog. Seems like the kind of thing that you kind of come to at a late night while in college. We were like we're going to start

Speaker A: a blog,

Speaker B: parties at Stanford I guess

Speaker C: we're going to write the first few articles some of ourselves. We're going to go email people that we really like to just collect them. And then we got um, some really, really exciting blogs. I think one of the first blogs was Sebastian Ruder who essentially predicted this uh, entire LLM scaling pre training, um, style paradigm, um, and he called it the imagenet moment for NLP is here. And uh, it seems kind of dated in the sense that nowadays you don't compare things to imagenet, you compare things to LLMs. At the time imagenet was the thing that everybody compared to the thesis was something like you did a bunch of Pre training on ImageNet and, and then when you fine tuned on other image classification or other sort of image data sets it would be much more effective than training from scratch on those data sets. His claim was that something similar was going to happen in the process of happening for NLP where you would pre train on some large text corpus such as the Internet, um, and then you would fine tune on some other uh, things that you cared about and then things would do really well. Um, and uh, they had some preliminary results of that. Of course uh, things have clearly progressed tremendously since then to the point where I think this is the dominant phenomenon and people don't even Think about training from scratch on the downstream tasks. They just obviously say you should train an LLM and then fine tune and everybody has forgotten everything about it. That's how dominant this paradigm was. But back in 2018 it was really, really new and exciting. Um, and then we had a bunch of other things. I think one, my favorite personal one was, uh, Horace, he, who's now at Pytorch. Um, he had a very spicy, very spicy article, um, which essentially said Pytorch, uh, would outrun TensorFlow. And at the time TensorFlow was completely dominant. Um, Pytorch was sort of this new thing which was up and coming but wasn't anywhere close to the dominant framework that it was today. Um, and basically he was like, yeah, Pytorch is going to beat TensorFlow. And I remember because I was interning at Google at the time, uh, people would just, you know, people had long, long discussions about this article. It was, it was just a huge thing. Uh, because this man who clearly knew what he was talking about, knew all the stats and all this stuff was like, yes, pytorch is coming. TensorFlow is, you know, not going to be the dominant, uh, framework. And people were just like, had lots and lots of opinions, just talked about this all the time. I lit a fire in the community and uh, you know, in many senses of the word, I think Horace was correct. In hindsight it did seem like at least in this period of time, Pytorch, uh, has become the dominant machine learning framework. Um, and yeah, like this was, uh, we just sort of evolved from there. Um, we published more and more articles, we got more excited about other things such as the podcast and the newsletter, which I think Andre and Daniel can tell you more about.

Speaker A: Yeah, I think the big takeaway from Hugh's history there is that everything important that has happened in AI in the past couple of years, you heard it all here first.

Speaker B: I don't know if we'll go that far, but, uh, you could say we called LLMs maybe, or not us, but Sebastian, uh, Ruder, I do think that NLP article holds up. Uh, it was also still one of my favorites going back. And it was kind of an interesting example because I remember editing it. That was. I offered a lot of edits to Sebastian, so it was nice that he was, uh, kind of up for that process. So I guess I can go next since I'm kind of the second oldest gradient person. Hugh and his friends, I suppose at the time were the real originators of it. And I sort of joined on Pre launch. Uh, so the way this happened is there was another one of the people involved, Nancy, um, I guess it seems like you were looking for people to actually write for the site. And I had already started doing some blog post writing and I was involved in things like that at the time. So Nancy was just like, do you want to contribute something to this new thing? This was before it even launched. This was kind of in the planning phase. So I wrote some absurdly long, uh, post about reinforcement learning, because I had a bunch of thoughts at the time. Probably still also holds up, you could argue. And then I had this 10,000 long article, uh, and, uh, was kind of impatient for the site to launch and then just wound up sort of joining and, uh, trying to push it along and get it to go public and take off the ground. So after that I sort of stuck around as a person involved in the editing process and the sort of running, uh, process of it. And then later I forget this was maybe 2021 or even 2022. I had by that point already started podcasting with last week in AI, which I still do. And one of the great things about working with the Gradients has been, uh, obviously we don't mostly we publish articles from outside our team. We partner and help people to push their own stuff out. And so that's been one of the rewarding things, is getting to collaborate with a lot of really talented, interesting people from different backgrounds, um, release information on different topics like AI art, AI for medicine, uh, all sorts of stuff, finance, etc. So that pretty much brought the idea of the Gradient podcast is at the time, there are already AI podcasts out there, but none that were sort of going deep on topics in quite the same way, as you could see on Gradient articles. And so I like this idea of instead of doing sort of expansive interviews in the style of Lex Friedman or other podcasts out there, we could do something a little more focused and a little more technical where we could get people from AI community and really dive deep into AI research, into their background, uh, things like that. So I got that going at some point. I ran it for a few dozen episodes. Then, you know, life got in the way. And it made a lot of sense to hand it off to Daniel, uh, who has of course been doing it since and has really made his own kind of thing. So, yeah, it's been a ride. And I guess, uh, part of the reason that we do have to make the changes, um, myself and Hugh and to some extent Daniel, do just have less time to be doing it between doing a startup and a podcast, I have personally not been very involved over the last couple of years, just kind of jumping in here and there. So even though I do really feel uh, very fond of a gradient and wish I could commit more time to keep it as active as possible, it seems like a very practical thing to change things a little bit, but still keep it uh, with the central role being uh, a place to amplify other people's voices and to help people read and learn about AI, uh, as it keeps moving faster and faster.

Speaker A: Still a question for you Andre. Among the few dozen interviews you did, do you have a favorite?

Speaker B: Yeah, that's a good question. I mean uh, I guess it's easy to say Yann Lecun was a highlight because uh, that was just pretty exciting. And uh, I think that one turned out pretty good. And then uh, aside from kind of that obvious one, I guess one I think back on fondly is an interview with an AI artist, uh, Helena Saren, who goes back a long while. She was active in AI art before you had all these fancy text to image models when gans were the main game in town. And yeah, she's I would say one of a major artist and someone whose art I really do like a lot. So I found that to be really interesting talking with much more of an artist who is still pretty technical and learning about that process. Uh, and aside from that, I mean there are quite a few episodes with professors or researchers and in those episodes I got to kind of read through their research work, try and go through their almost career in a way, really going from early days to later on and seeing how that progressed. So I believe I talked with Chelsea Finn, um, probably Sergey Levine or I don't know, a couple of people in robotics. And I was a, ah, grad student in robotics. So that was pretty fun. So yeah, uh, it's hard to highlight. I think there were quite a few that I liked a lot, but those are a couple that come to mind.

Speaker A: It's always hard to narrow down. Yeah. Since we were talking about earlier articles, another one I want to call out was the. I guess the title is Benderrule. Emily Bender wrote something for us at the end of 2019, um, and she references this 2011 paper she wrote about do's and don'ts for language independent nlp. The whole point was about the vast majority of NLP is done in basically two languages and I think it was English and Spanish. And the Bender rule basically just says do state the name of the language you're studying even if that language is English. Um, and the point is that acknowledging this foregrounds the possibility that techniques you're using might be language specific. Languages do work differently. They have different grammars. Um, things are going to work differently for them. So the idea that work that is done primarily in English is language independent is according to Bender, just false. Um, and I think that was a pretty, pretty important, interesting early article. Um, also call out a recent one that I really liked. Um, well, there's so many. Um, the one I thought he was going to call out, which maybe I'll call out since he didn't, was this incredibly long detailed article by Arjun Rahmani and Zheng Dong Wang. I see Hugh thinking that he probably should have called that one out, called why Artificial Intelligence is really, really Hard to Achieve. That's with two reallys, uh, for emphasis there. Um, and it's a pretty detailed argument collecting a lot of reasons, economic and different from different areas, about why the idea that you're going to have super economically transformative artificial intelligence the way that a lot of prognosticators might imagine is actually a little bit harder than you might think. And I think they provide some great structure to how you should think through some of the economic impacts of AI. So I really appreciated that.

Speaker C: Yeah, that article was great. Um, Arjun and Zhen Dong are super insightful. Um, probably the best article that we've published in recent years.

Speaker A: Uh, way to make all our other authors feel bad, Hugh.

Speaker B: Well, yeah, we haven't published too many. That's not too hard to do. And uh, I guess if we are calling out some more articles and maybe we'll also highlight them in the uh, substack post just to help people go back to some of the ones that are still probably worth reading to this day and are not kind of aged. One I really like is the Past, Present and uh, future of AI art by Fabian Offert. So this goes back to 2019 and you know, is I think still relevant. Like AI art has changed in some sense. Now AI art usually refers to text to image. But I think the basic questions around it of is it art, how is it different from human art, etc. Are still pretty relevant and still largely maybe unresolved. So I quite, uh, like that one and think it's worth reading. And I'll call one extra one. We had a fair number of kind of overview articles that introduce you to a general topic, general, uh, area of research, or something along those lines. And one that I think is quite cool and maybe something many people are not as aware of is AI story generation. So Professor Mark Riedel, uh, published this piece, an introduction to AI story generation that has a very nice, pretty in depth overview of this area of research. And of course now with LLMs, that's a whole thing. But there's as with many things, a long history of people working on it. And uh, it's definitely, you know, if you're a nerd and like to learn about AI history, which I am to some extent, that's a really fun read.

Speaker A: I mean if you're listening to this, you're probably a nerd. So you know, you'll, I predict, a pretty high chance you'll like it. Before we begin closing out, do you two have anything else you want to talk about, tell the listeners?

Speaker B: Well, I guess it would be irresponsible of me not to plug my current more active project. So actually this is a funny point. So maybe it's worth also telling this little story. Uh, somehow I wound up doing sort of two AI magazines in parallel for a while. So when I joined the Gradient team I was already, uh, already working. I hadn't even launched this yet, but I was already doing this thing called Sky Today, which was started with intent to kind uh, of combat AI misinformation and AI hype at the time. So there was a lot of, even at that time, you know, AI was already getting kind of big and there was quite a lot of articles, uh, being posted, some of them being pretty inaccurate. So as a grad student at the time, I was like, well this is kind of silly. Maybe we should start something where people who do work in AI can provide more context and a better explanation of certain news stories. Uh, things like alphago. And some people were saying alphago means that we'll have AGI, which is pretty silly, especially now in hindsight, uh, but also other things like OpenAI's Dota research, things uh, like that. So anyway, I wound up doing vat and VAT also launched and was like a parallel project that uh, had a pretty different focus. It was less for the AI community. Uh, but either way I wound up publishing and editing on two fronts and then that also kind of morphed. So VAT magazines kind of today is now sort of frozen. But what did come out of it is last week in AI, which is a newsletter where we have news from the past weekend AI and also a podcast where we summarize every week's AI news. And not being just technical, we focus on kind of across the board tools and uh, applications. You can use. We cover business, we cover research, we COVID policy. So we try to get it all. And that's why the episodes for that podcast tend to be like two hours long. So yeah, I'm gonna go ahead and plug that and say if you aren't yet a listener or a subscriber to Last Week in AI, maybe you should check it out. You can go to lastweekin AI for that substack and find all the links and so on there.

Speaker A: Cool. Um, well, if we're going to do shameless blogging first. Also I'll mention that, ah, the way I met Andrej is during college. I started working on Skynet today with him because I pitched an article on AI policy while I was taking a US foreign policy class or something. And very excited about all of this. Um, and continued writing articles and working on the newsletter and things like this. So that was really, um, I'm glad we did it. It was really fun. It was a great time. Um, and I've guest hosted a couple of Last Week in AI episodes with Andre and can confirm it's a great overview if you're looking to keep up to date with things that are going on. Um, and to close out this episode, I guess I'll finish with Hugh. Do you should have things to plug? You have some cool stuff going on,

Speaker B: some good research, cool papers.

Speaker A: Yeah. Tell us about your papers, Hugh.

Speaker C: I don't have too much to plug. I guess there's been some research that I've been excited about. Um, we recently released GSM1K, um, which was a replication of a common math, uh, data set called GSM8K. Um, and we basically did that to check how much data contamination was affecting the benchmarks. Um, and uh, the conclusion was actually that most of the frontier models actually showed no sign of data contamination. All the benchmark performance was real. Um, and there were a few cases where there was more data contamination than others. But as a whole most models were actually quite good at generalizing to novel math data sets that I clearly never seen before because we just made it. Um, more recently I've been working on, uh, I guess we recently released Plan Search, which was a sort of uh, a method of doing inference time compute. Obviously there's more popular methods of doing inference time compute now. Um, most notably. Okay, um, but it's a very exciting direction with reasoning and infrastructure compute that I'm super excited about. Um, and finally something that's not out yet but is uh, publicly announced. We're making humanity's last exam, which is supposed to be our goal. Um, the hardest uh, exam type question out there and uh, uh, maybe even if we do it right, the last exam type eval out there where there's a single correct answer, um, all the other evals after that, if we do our job right, will sort of be interactive or other non exam type evals. Um, it's not the last evaluation, but maybe it's the last sort of quiz type evaluation um, that's been publicly announced but hasn't been completed yet. But hopefully will be completed in the coming short period. I don't want to make any promises and then we'll have it out very soon. Um, but yeah, other than that I think I just wanted to thank um, all the listeners and readers of the Gradient. Um, it's been really an honor to, to sort of have you folks for the last six years and um, we really appreciate it as screening editors.

Speaker A: Yeah. Um, also if all of that overview excited you and you at some point in the future want to hear an extended Hu Zhong interview, let me know and we'll try to make this happen.

Speaker B: Um, real quick actually before we close out since uh, we are talking so much about the Gradient, uh, probably worth calling out to the other people currently involved. You've had many people involved over the years. Years. Many years. And you know, all of us have done this as a side project. You know, we do make some money for Substack, but we're very terrible about paying ourselves. So uh, this is really a passion project for everyone involved, uh, with most people having full time jobs and so on. So it's been really great to have everyone uh, who has worked and contributed and currently there is still a number of people who actively work on the newsletter and really make that very uh, I guess successful, consistent project, unlike a magazine. So Daniel, you're the most involved with him right now. Maybe you can give him a shout out.

Speaker A: Yeah, let me call you out all by name. Cole Frank, Jamie Seong, Shahrukh Gupta and Justin Lande. I've been working with the four of you for a while now for various lengths. Justin and Sharuk have been with us for quite a while, especially Justin. He's been there for years. Um, and you all have just been so wonderful to work with, so thoughtful. You have very different writing styles, different interests and it's really wonderful to get to put that together in the newsletter we pull out. So um, I want to thank you all for doing that. It's great to have you all. Um, and with that I think I should also probably Mention that, uh, we've had a number of editors who have been with us and since left. Um, and I don't know if any of you are listening, but if you are, I'm also very, very thankful to have worked with you. I think that. But, um, we've all learned a lot from each other, so I'll stop being sentimental and end the enclosure for some finishing plugs. Uh, if you want to follow us in various ways, we'll drop links in the substack post that this comes out with. I have other places where I write, so if you're interested in hearing weird thoughts, that's another place where you can hear more from me. Um, and if you are listening to this, you should subscribe to the podcast rss. Get in touch if you want to help us out in some fact fashion. While we are slowing things down a bit, there is still a potential future world where we have more editors with more time, and we're able to invest more into that. That just doesn't happen to be this moment in time. But we do want to keep this effort going. So if you want to help that out in any way, we'd love to hear from you. That's the editor at, uh, not the editor, that is editoregradient. Uh, Pub.

Speaker B: I think we have contact. We should really start using my contact email at some point. But yes, anyway, uh, and yeah, to make a fine point of it, if you are currently listening on the web version of Substack to this, then there might be a feature. In fact, it is somewhat likely that this podcast will move off of the substack. Uh, so you would want to subscribe on Apple Podcasts or whatever app you use for podcasts to make sure you don't lose track of it. Uh, and one last sentimental note, since I don't think we've done it, if there are any contributors to the gradient from outside, any people who have written with the gradient or even just pitched articles to us, and somehow we failed to follow up. Uh, we do also of course, appreciate that it's been really fantastic to have a fair number of articles, a fair number of people contribute, and hopefully we have provided some beneficial editing and uh, I guess signal boosting to your work. Uh, obviously at the end of the day, the reason the gradient even became a thing that stuck around is that people did submit some really cool stuff and we just got to help people share, uh, um, their thoughts and their work. So it's been great.

Speaker A: Yeah, I think that's a good place to close. Um, we love and appreciate all of you. Thank you for listening.