Beyond Experimentation: Prof. Yoav Shoham's A121’s CEO, talks on Gen AI's Future and AI21 Journey

Welcome to Future of Tech, hosted by Avishai Shalin, Division President of Amdocs Technology. In this podcast, Avishai sits down with some of the most innovative minds in technology to learn how they are disrupting the present and what kind of impact they hope to have in the future. From the machine learning programs that are solving some of the world's biggest problems to what AI can do to help fight biological bottlenecks in human thinking, no topic is off-limits. In this special season, Future of Tech tackles generative AI with interviews with some of the industry's leading thinkers.

So sit back, relax, and maybe take some notes. Because what you hear on this show might just be a glimpse into the future. In this episode of Future of Tech, Avishai sits down with Yoav Shoham. Uav was a professor of computer science at Stanford for 28 years, where he was the director of the AI Lab.

He is also a serial entrepreneur and has founded various companies across industries. Most recently, he co-founded AI21 Labs. AI21 aims to take AI to the next level and builds LLMs for enterprises that make machines thought partners. You have also initiated the AI Index in 2017, which tracks activity and progress across AI.

Join Avi Shai as he sits down with an AI pioneer to uncover his thoughts about the future. Future of Tech is brought to you by Amdox Tech. Amdox Tech is Amdox's R&D and technology center, paving the way to a better connected future by creating open, innovative, best-in-class products and continuously evolving the way we work, learn, and live. To learn more about Amdocs, visit the Amdocs technology page on LinkedIn.

Hello and welcome to a special episode of Future of Tech. The subject of today is Gen.AI, part of our special season. About Gen.

AI, I'm very pleased and honored to have with me Yoav Shoham, which is a professor from Stanford University. Worked there for many years. After leading there the AI lab, went into becoming a serial entrepreneur, founded many companies, sold many, and today is one of the co-founders of AI21, which is one of the pivotal companies in the industry, leading Gen.AI and bringing many new innovative solutions.

I'm very happy to have you with me, Yoav, and Shalom. Hi, Shalom. Thank you very much for having me. Let me start with taking you many years back.

When was the first time you started your technology career? When you say many, I don't think you realize how many it is. I studied computer science by mistake just because I needed to study something. And I did computer science because it seemed like it would be useful, not because I actually knew or cared.

Then I stumbled on AI. I like to say I went. I started to read in the Internet. It's a joke, of course.

But I did go to the libraries and read and decided to do a PhD in AI because it seemed like you could do philosophy and psychology. Oh, and by the way, also technology. So this is how I started, went to do a PhD in AI abroad because Israel didn't have it yet. And what happened then?

I did my PhD and became a professor. When was the first time that you became an entrepreneur and went out to form a company? That's interesting. So my area in AI was more on the theoretical side, like the quote-unquote non-useful stuff.

I worked in logic and philosophy and economics and game theory. But when I worked on economics and game theory, I got close to the people working on auctions. eBay was starting, but also the industrial auction, the Spectrum auction, the FCC auction in the States, and so various kind of industrial auctions. And I realized that the people were contracting with system integrators to create one-off solutions, the very much legacy system that you couldn't change at all.

And I remember one bid was for $100 million. It ended up being $200 million, straight jacket kind of outcome. And I knew we could do it much faster, much cheaper, and more flexibly. So we started the first company that I was involved with called Trading Dynamics.

That's how I started. I knew nothing about really entrepreneurship or business, but somebody paid the tuition. So I learned. Good.

And when was the first time that you've decided to go back into Gen AI? I think I'm going to probably disappoint you. There's no such thing called generative AI. There are various terms that I dislike.

This is one of them. But the thing that usually comes under the umbrella, namely large language models in particular, is a recent phenomenon. When we started our current company, AI21 Labs, we didn't think of large language models or generative AI. Neither were terms, really, at the time.

But we started the company because we believed that deep learning, which, of course, is the technology underlying large language models, that deep learning was necessary in modern-day AI but not sufficient. You would never be able to do the robust, reliable reasoning that you expect, certainly in the enterprise setting. and the sort of thing AI used to do in the 80s, kind of a sore detour. About seven or so years ago, I started something called the AI Index out of Stanford, where I was teaching, that puts out annual reports about the state of AI, and both technological advance, but also just mere quantitative volume of activity.

And all the volumetric data, number of students, papers, VC money, they're all U-shaped. back in the 80s, everybody's doing AI. Then you weren't allowed to admit you're doing AI, what we call the AI winter. And now my plumber is doing AI, but it a very different kind of AI Today it all about statistics under the header of machine learning or even more generally deep learning And back then machine learning was a thing but didn play a key role in intellectual activity But it was about knowledge representation, inference engines, and so on, expert systems.

It wasn't as terrible as people sometimes think, but it certainly overpromised. So we've pivoted now to statistics, which does amazing things, primarily because we have infinite data and infinite compute. but it still doesn't do the robust kind of reasoning that we could do in the 80s. We started the company, and we very quickly fell into natural language because machine vision is interesting.

We sometimes say that machine vision is a lens into the human eye, and natural language is the lens into the human mind because there's no thought as complicated as you want. You can't somehow express the language. It makes it harder. And so we felt this was where the action was, and as we delved into it, We quickly realized that the end-to-end training of large language models were dominating everything.

And so we became excellent at that. And that's how I fell into what some people like to call generative AI. So I'll try to avoid the term. It's okay.

You got license. Okay. But I still want to pick your brain when it comes into the history. So why now versus 10 years ago?

Why now versus what happened when you started your career? The answer is data and compute. I remember my colleague Andrew Ng, who's a very famous, brilliant machine learning guy. At some point, we were discussing, and he was weighing various things to spend time on.

And he said, I'm gambling on deep learning. If you go back to history, you know, heyday in the 80s, knowledge, present, reasoning, kind of one phase. Then a kind of trough of disillusionment, the winter of AI. And then the 2000-ish kind of time period, suddenly deep learning began to emerge out of the ashes of neural nets and machine vision being dramatically improved.

You didn't see the big improvements in language. Like I said, it's harder. But when we got transformers, this is going back now really six years ago, suddenly the needle started to move. And this new version of neural nets really ushered in the new era that I think we're all experiencing now.

Interesting. Now, if you need to provide me some basics to understand the difference between LLMs in the world, because everybody speaks LLM now. This one creates, this is a small LLM, a big LLM, a verticalized LLM. Give me like a glossary to start and find my way in this ecosystem of LLM.

Sometimes people use the term foundation models. Okay. So my colleague, Percy Liang, coined this term, and I think they did it for two reasons. One is because they viewed this technology as a foundation for other stuff you'll build on it, but also not calling it language models because it's not only language.

It can be multimodal. You can have images and video and audio and what have you. People realized already a couple of years ago that size matters, but only to a point. If you don't have enough training data, for example, then you can't use that size for anything useful.

So there's got to be some kind of balance between the amount of data you train on. And at some point, you know, there's just that much data. By now, our models train on everything that's ever been written. Now, it's not really true, but it's a close proclamation.

Somebody calculated that if you were to take the amount of data that we train on and a person actually had to read it 24 by 7, it would take them 10,000 euros, something like that. So it's an insane amount. So that's another dimension of how much you train on. In the past, we trained on 300 billion tokens.

Token is a word or part of a word in a textual kind of domain. And that seemed like a ton. Today, we don't get out of bed for that. It's multi-trillion tokens.

And so that's another dimension to think about. Then not all data is created equal. Some data is much more useful to train on the others. You have this natural data that you get from the Internet, from Wikipedia, from books, and then you can generate data.

So you have this synthetic data. Then maybe the last dimension I can think of right now is how specialized the model is. What we've pioneered is what we call task-specific models. These are models that, by the way, they also tend to be small.

but they're optimized for certain tasks like summarizing a text, answering questions in a rag style, retrieval augmented style. And they tend to be excellent at those, much better than even the best general purpose large models. They're also small, so the latency and the unit economics kind of make sense. The well-known secret is that you've got to augment language models with stuff they're not good at.

and whether it's pre-processing the data or verification afterwards or various optimizations that you do around it. But this is kind of a little bit of the lay of the land of language models. Which is great. Now I'm coming back to my previous question about AI21 lab.

So what exactly did you target as the mission or what you are trying to solve? It's two-pronged. One is we really want to take AI to the next level So leveraging neural nets, language models, what they're good for, and augmenting them with the element or not, the reasoning elements. But we're not just a research lab.

We're an actual business. What business are we on? It was clear to us from the beginning that we want to change how enterprises interact with text. When we started out, enterprises weren't there yet.

We spent three years just building technology. But then we went to say, OK, let's do business now. The enterprise market wasn't there. So we created our own market.

We built our own application called WordTune, which really set out to change how people read and write. One thing we didn't do deliberately at the beginning, we didn't do spell checking and grammar correction. We viewed that and still do the table stakes. Anybody can do that.

We do it too, but we deliberately suppressed that. So we did things that were much more ambitious. You write something down, and we're not here to tell you, oh, you made a mistake. We're here to help you articulate what you really wanted to say.

The application did really well. we well beyond 10 million users Two examples of use cases that you see recurring One is summarization And so we have a task model summarization And we had a very large financial institution. Everybody's experimenting and trying multiple solutions, as they should. So this large company did a bake-off on their internal document to do summarization.

and two of our task-specific models just dominated the field against the well-known competitors because they're just general purpose. So that would be one example, how to take large financial documents and summarize them. Another would be question answering, educational company, billions of documents. We are powering their contextual answers so you can go and query the repository, do basically a semantic search of that repository.

That's another example. Zooming out for a second, trying to see this industry as a whole, spoke about the phenomena that data is almost infinite and we have a lot of processing power. So now are we in a situation that we can resolve all the problems or we still face things that this word that I cannot use needs to address? We're not there yet now on robust application of AI.

In general, you see in the industry, while the giant has awoken, we've moved from sporadic experimentation to mass experimentation, everybody experimenting, not mass deployment. And you can speak to how much deployment you're doing here at Amdocs, and maybe you're an exception, but I can tell you in general, deployments, everybody experimenting, very few actual deployments. And I think there are a few reasons for that. One is people are unclear on the use cases where you actually get ROI.

Another thing is really the technology risk. And people are concerned, and rightly so, because there's a big difference between a flashy demo and a robust application. And just on the algorithmic side. And this is something that was so obvious to us from the beginning.

And so reliable AI is something that I think is still an art. and out of science and i think that's slowing things down the other thing are just a little more bread and butter but just legal issues that people aren't you know we're new grounds on having to do with legal exposure on intellectual property or bias and toxicity and so on that i think everybody's still wrapping their arms around you mentioned the giant awaken so what's your thesis about AI residing on all clouds? Is there a preference where it lives and where it goes?

What's your concept about that? I think AI in one form or another will live anywhere, everywhere. If you're speaking about the hyperscaler, about clouds, of course they all want to be AI. We have a very close collaboration with both GCP and AWS.

NVIDIA is an investor in us. Everybody would like to be a major player here. It's not just the clouds. You know, the big ISVs are embracing AI in a deep way.

And again, you know, you can probably tell us more about that than I could. And you're going to see AI push down to the device level. And so it has to be. So you need distillation, you know, so things actually can run.

So right now, certain kinds of LLMs, like the tiny new Gemini and maybe a version of Pi can run on your mobile phone. How good is it? Not quite as good as a real LLM, but we'll get there. There's no question.

Yeah, I agree. We'll get there. Now, tell me a few words about precision. How important is it?

Or why is it important? I'm going to answer a slightly different question. I'm going to speak about predictability, reliability, explainability. Because precision...

So, for example, if you go to any of the best language models today and you give it a problem in arithmetic, you'll get an answer. And it's actually remarkable that the system can do arithmetic. Somehow it emerged, and you'll give, you know, whether it's our Jurassic 2 or GPT-4, I don't care, a two-digit number to add. they'll do it quite reliably.

You'll give it a, you know, three-digit number to multiply. You'll get confident garbage. You'll get a very confident answer. That's one thing about these language models.

They don't know when they don't know. It's like a good MBA. They'll give you a confident answer. We know how to do arithmetic, right?

HP told us in the 1970s with the calculator how to do arithmetic. So that's an example where reliability, and we asked, by the way, when we asked the language model, why do you give me this answer? It's also very good to give me verbal explanations. And of course, in the case of arithmetic, if the answer is wrong, no amount of explanation will convince you.

But if you try to give it, you ask it a question to which you don't know the answer, and to be a little more confident, you ask to tell me why, and it'll give you a plausible sound explanation, that can be very misleading. So reliability, explainability, predictability predictability is something that I think is key in mission-critical applications. And what about your ability to trust the answer? The word trust, I think, encompasses everything I said.

You need to trust the system. And to trust them, they need to be reliable. If you don't know what you're going to get, then you can't trust it. Even, you know, it's like having this idiot savant that sometimes gives you amazing insights and sometimes it's total garbage.

You can't rely on that. And if it can't explain it in a way that you understand the explanation, you can't trust that either. Very much so. But with a younger generation that consumes everything, more or less, from the online interaction, do you feel we might be in a situation that we will consume garbage in a way without even knowing that it's garbage and then internalize it and be confident that this is the truth?

That you now pointing to what I think is one of the most fundamental challenges we facing as a modern society It didn start with AI certainly not with gen AI but application of untruths that are presented as truth Of course it existed before The flattening of the technological world and social networks made it easier to disseminate things. And we've all seen Cambridge Analytica and so on. And the argument is that AI could amplify that. It's interesting.

I'm not sure that AI is the main culprit here. Certainly not in disseminating, creating false text. I think amplification will be non-textual content generated by AI. And Yuval Noah Harari, our compatriot, spoke about the danger of false intimacy.

That's actually a very interesting observation that it's not just a matter of And often we trust information because of the source from which it came. We feel a certain intimacy, whether it's my colleague, my friend, or a public figure we trust. And we viscerally, when you see a person speak, hear them speak or watch them speak, we viscerally trust that. And now that you can fake that, that's very interesting.

So, you know, when I grow up, it's a problem I really would like to understand and maybe contribute to solving a little bit. Can you maybe share, maybe not necessarily just in the learning domain, but how or what are the next steps that we can foresee to this industry? So first of all, I'll make a prediction you can check, then I'll make a prediction you'll be harder to check. I'll make a prediction that you can check, one that you may be able to check, and one that's no way you'll check.

The prediction that is crisp and checkable is that in 2024, we'll see the transition from mass experimentation to substantial deployment. Enterprises will be sufficiently comfortable with the technology to include it in actual massive deployments. Now tell me maybe a practical question before we kind of wrap things up. You worked for many years in the lab and then you went into the enterprise domain and the consumer domain.

How do you, from a technology perspective, make sure that it's scalable? It's interesting. The world has changed this way. When I was starting out, you would do the theory, you'd do maybe a small prototype, you'd prove theorems, you'd write papers.

and then if you want to do something real, you'd go to industry and maybe start your own company or go to a large company and do it there. Dynamics have changed. First of all, innovation is happening in industry as much as it's happening in academia. It's not exactly the same.

There's certain you can take risks and go on tangents in academia you don't tend to do in industry. So I think academia will always have a key role, but you can't ignore the innovation happening in the industry. The other, especially in AI, the big is the resources available. Money, amount of compute.

It'll be a question to the extent to which government steps in and makes up for that and makes sure that academia has at least a fighting chance to do things at scale because a lot of what's needed right now is not a matter of just being very smart and putting pencil to paper, but really running experiments at scale. I say now we in industry will always have an edge in terms of money and compute, but I still think that academia has a key role. My colleagues in academia are running experiments at scale.

I know that here in Israel we're trying to put together a cloud for the use, among other things by academia who could run experience of scale. So interesting dynamics. Yeah. Also cost you one.

So I think that not every small company can afford itself to build this mega scale and experiment with it. So endeavors like the one that you mentioned, building something on maybe a country level that can assist might be very, very beneficial. Yeah, I don't think there will be many entities, private, public, for-profit, not-for-profit who will build mantra systems the way we do. Just not only expensive, it's also a major engineering undertaking.

As you know, when you're running, you know, several thousand GPUs or TPUs simultaneously, things happen that don't happen otherwise. And it takes also just technical skill to do that. And it takes attention over time, which often in academia you don't have. So I don't think there'll be many such entities.

There will be, you know, a handful. I think at the national scale, I don't think you'll be competitive with that, but you could fine-tune. For example, we don't have a Hebrew, a very substantial Hebrew language model. It's part of the National AI Project that I'm involved with.

There's an effort to create such a thing in partnership with one of the large companies. And it's not a commercially attractive proposition, I think, for a company. it's more a matter of demonstrating prowess, a little bit of goodwill, and maybe a way of recruiting people. But I think the most of the innovation at scale will happen, not under government.

I have one personal question. You being an entrepreneur for many years, how do you balance your personal life with being someone that needs to be there constantly and monitor the market and meet investors and travel to customers. It's very easy, actually. You spend 50% of the time on science and technology, 50% on business, and the other 50% with the family and for yourself.

So, you have. It was a pleasure meeting you. Likewise. Really enjoyed the time.

I hope our audience as well. Thank you very much and best of luck. Thanks for listening to The Future of Tech. If you like what you heard and want more, make sure to subscribe on Apple Podcasts or your favorite podcast app.

And if you have any comments or questions, feel free to write our host, Avisa Ishalin, directly on LinkedIn.