The B2B Podcast Index
Index
All categories
MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
MethodologySubmit
Best of:MarketingSalesSaaSFinanceHROpsLeadershipCustomer SuccessAI & DataProductStartups & FoundersRevOpsEngineering & DevTools
An independent project byFame
SearchBest episodesGuestsInsightsMethodologySubmit a podcast
Index/HR/Hidden Layers
Hidden Layers artwork

Why Agentic AI Isn’t Ready for Prime Time - Yet | EP. 47

Hidden Layers · 2025-11-13 · 37 min

0:00--:--

Key moments - from our scoring

Substance score

60 / 100

Five dimensions, 20 points each

Insight Density13 / 20
Originality11 / 20
Guest Caliber14 / 20
Specificity & Evidence10 / 20
Conversational Craft12 / 20

The episode dissects the gap between agentic AI hype and reality through the lens of two practitioners actively building these systems. Zizi Psi reports that 80-90% of new code is now written by coding agents, though quality varies and manual verification remains essential - particularly for production systems. Michael Warden emphasizes the "slop" problem: even best-in-class agents generate bloated code with redundant error handling that masks root causes rather than solving them. Both speakers ground their pessimism in concrete constraints. They note that chaining 20 agentic steps at 95% success per step degrades to near-zero reliability (0.95^20), which is why autonomous browser agents and travel planning remain unsolved after three years. The hosts reference Andrej Karpathy's "decade of agents" framing (not year) and discuss how narrow, domain-specific AI systems federated together - rather than general-purpose LLM agents - will likely drive real enterprise value. They also surface a sobering observation: in a room of 50 aerospace students, only one used LLMs daily, suggesting massive underadoption outside tech bubbles, while over-hype drives misguided deployment elsewhere.

Key takeaways

  • →Coding agents are the only genuinely mature killer app today; most other enterprise agentic use cases are still unreliable because error compounds exponentially when chaining 20+ steps together.
  • →Reliability requirements differ by domain - 99.9% accuracy is fine for brainstorming but unachievable for sensitive tasks like form-filling or financial transactions without new breakthroughs.
  • →Narrow, domain-specific AI systems (X-ray detection, fraud detection, object recognition) need to be built and federated as external skills for agents to call, rather than relying on general LLMs for everything.
  • →True AGI with embodied capabilities (robotics, 3D manipulation) is likely 30-50 years away due to hardware constraints and precision requirements, independent of software breakthroughs.
  • →Current transformer architectures may be hitting limits; new approaches including linear attention variants and biological inspiration will likely be needed to move beyond scaling alone.

In this episode

  1. 1Defining Agentic AI and What It Means in Practice
  2. 2Current Applications: Coding Agents as the Killer App
  3. 3Challenges and Limitations of Coding Agents
  4. 4Beyond Coding: Document Understanding and Other Use Cases
  5. 5Enterprise Adoption Concerns and Error Compounding
  6. 6The Reality Check: Agentic AI Hype vs. Timeline
  7. 7Narrow AI Systems and Domain-Specific Approaches
  8. 8AGI Timeline, Architecture, and Embodiment Requirements

Mentioned

OpenAIZizi PsiMichael WardenHoastron GreenGreg BrockmanAndrej KarpathyGary MarcusGPT-4 AtlasComet

Guests

Zizi PsiMichael Warden

Topics in this episode

Agentic AILarge Language Models (LLMs)coding agentsTransformersGartner Hype Cycleautonomous AI systemsLinear attention architecturesDomain-specific AIEmbodied AI / roboticsError compounding in multi-step workflows

Questions this episode answers

What are the actual current use cases for agentic AI that work well today?

Coding agents are the proven killer app - Zizi reports 80-90% of new code is now generated by agents, though outputs require verification and frequent nudging. Document understanding and tedious workflow automation show promise but remain less mature.

Why does chaining multiple agentic actions together fail so often in enterprise?

If each step succeeds 95% of the time, 20 chained steps compound to 0.95^20 ≈ 36% overall success, making autonomous multi-step workflows unreliable for high-stakes business operations.

How should enterprises actually adopt AI rather than just adopting it for its own sake?

Build narrow, domain-specific AI systems with superhuman performance in specific domains (e.g., X-ray detection, fraud detection), then federate them as callable skills that general agents can invoke, rather than expecting LLMs to solve everything.

What timeline do these practitioners expect for real AGI?

Zizi is more pessimistic than Andrej Karpathy, estimating 30-50 years for true AGI with embodied capabilities because hardware is "very, very hard" and requires precision that scales up error requirements exponentially.

What's the biggest problem with code generated by AI agents today?

Agents generate bloat including redundant try-catch blocks that silence exceptions rather than handle them properly, duplicate functions, and architectural choices that obscure intent rather than solve underlying issues.

What our scoring noted

Our reviewer’s read on each dimension, with quotes from the episode.

Insight Density

13 / 20

The episode contains several substantive technical discussions about agentic AI limitations, coding agent trade-offs, and AGI timelines, but is undermined by repetitive tangents and vague philosophical meanderings that pad runtime. Strong moments (e.g., the 0.95^20 error compounding, narrow AI systems as building blocks) are offset by lengthy, meandering explorations of neuroscience parallels that generate heat without illuminating practice.

if it's, you know, 95% successful at taking an action, and then you chain together 20 actions, which you want to do, that's kind of the point for these agentic systems, is your probability of success really, really diminish. Yeah, point nine, five times, point nine, five times, point nine, five... it gets small pretty quick.
agent systems that can reach out to these narrow AI systems. So for example, if you are, if you're dealing with an agent system right now and it needs to calculate some numbers, we know LLMs are terrible at that inherently, you want them to have the ability to call out to a calculator

Originality

11 / 20

The episode rehashes familiar frameworks (Gartner hype cycle, pre-training as evolution, narrow AI systems) without sharp contrarian takes. While the narrow-AI-as-building-blocks framing is useful, the broader arguments about embodied AGI timelines and neuroscience gaps echo existing debates. Few genuinely novel claims emerge; most positions are reasonable but well-trodden.

I feel like the term is almost too overloaded where I'd prefer not to use it and use something more descriptive if possible.
I don't know if you include the ability to move in 3D world and do any physical task a human could do... We're probably decades away from that.

Guest Caliber

14 / 20

Zizi and Michael are credible practitioners - co-founder and VP of engineering at an AI firm - with hands-on experience using coding agents and building agentic systems. However, they are internal company voices rather than operators proven at scale outside their own firm, limiting their external credibility. The framing as company co-founders discussing company work raises questions about independence.

My co-founder and distinguished machine learning engineer, Zizi Psi, and our vice president of engineering, Michael Warden.
I already use it for multiple places. Coding agent is a obvious one. I think right now 80% or 90% of my new code is written by Coding agent

Specificity & Evidence

10 / 20

The episode relies heavily on personal anecdotes (using coding agents, trying agentic browsers, observing aerospace students) and vague timelines (30-50 years, 10 years, "the 30s") rather than concrete data. Few named companies, metrics, or case studies ground claims. References to Andrew Karpathy and Gary Marcus papers are cited but not detailed; broader assertions lack supporting numbers.

I think right now 80% or 90% of my new code is written by Coding agent
I gave this talk this week up at the group of aerospace students in DC... one person in the room raised their hand... it was like 50 people in the room

Conversational Craft

12 / 20

The host asks reasonable opening questions and permits genuine back-and-forth, but rarely pushes back on claims or demands specifics. Follow-ups tend toward agreement-building rather than productive friction. Notably absent: challenge when guests make vague timeline predictions, probing on contradictions (e.g., why narrow AI hasn't proliferated if the ROI is obvious), or skepticism toward neuroscience analogies. The tone is collegial but lacks intellectual rigor.

Thanks for being here. Great to be here. Thanks for having us.
Michael, what are you using? We're all doing agentic coding. There's no question there. Are you using agentic AI in any other way?

Conversation analysis

Computed from the transcript - who did the talking, and the words that came up most.

Most-used words

agent23feel18code18coding17agents16agentic15learning13systems12point12agree11different10seems10learn10cases9human9almost8

Episode notes

Artificial intelligence is shifting from prediction to autonomy - and “agentic AI” is leading the charge. In this episode of Hidden Layers , KUNGFU.AI’s Ron Green, Dr. ZZ Si, and Michael Wharton unpack what it really means for machines to act on their own, what’s hype versus real progress, and how far we are from true artificial general intelligence (AGI). They discuss how coding agents are transforming development workflows, why agentic AI is both overhyped and underutilized, the challenges of scaling reliable autonomy, the connection between AGI, biology, and lifelong learning, and whether new architectures or cognitive inspiration will take us the rest of the way. 00:00 - Intro: From prediction to autonomy 01:30 - What is agentic AI? 05:00 - Coding agents and creative workflows 08:00 - Reliability, risk, and real-world use 12:30 - The agentic hype cycle 16:00 - Why businesses underuse (and overuse) AI 19:00 - Narrow AI and domain-specific intelligence 22:00 - The AGI timeline debate 26:00 - Learning from biology and cognition 33:00 - Lifelong learning and what’s missing today

Full transcript

37 min

Transcribed and scored by The B2B Podcast Index.

Welcome to InLayers, where we explore the people and technology behind artificial intelligence. I'm Hoastron Green, so here's the question. What happens when AI stops just predicting the next word and starts taking action on its own? That's the question at the hardest-day discussion about AI agents and the road to artificial general intelligence.

Some see agents as the next leap beyond large language models, a shift from prediction to autonomy. Others argue we're still decades away in that real reasoning and reliability demand new architectures and breakthroughs we just don't have yet. So the question is, what will it really take to build agent to AI and when will we finally do it? Will it transform how machine thinks or just remind us how much further we have to go?

To understand where this technology is really heading and what's standing the way I'm joined by two of the sharpest minds in the business. My co-founder and distinguished machine learning engineer, Zizi Psi, and our vice president of engineering, Michael Warden. All right, Zizi, Michael. Thanks for being here.

Great to be here. Thanks for having us. All right, so let's start off by defining agent to AI and what it means to you. I think this is a term that's getting a lot of traction in 2025, but I think different people mean different things when they say it, like Michael, what does it mean to you?

I have a very similar opinion on it. I feel like the term is almost too overloaded where I'd prefer not to use it and use something more descriptive if possible. And to me, it just is these systems that can take action now. I think in a lot of ways, people always compare them to interns, but in a certain light, I could almost see it as like a marketing hat that we put on top of automation that's got years and years of history of development.

Yeah. Yeah. I don't know. Yeah, what do you think, Zizi?

Yeah, I like that a lot. In practical terms, I think, you know, we use coding agents a lot in number firm. And to me, agent really means, you know, and AI, that's more autonomous that can, you know, execute multiple steps towards a goal in the process, you know, there may be mistakes. It will self-correct.

It can use tools and it will actually affect your workspace. It can delete your files and it can make your code much better. Right. That's what the agent means.

Yeah. And I'm right there with both of you. The way I always think about it is, it's something that can handle, it can bring reason, like common sense to a task and handle ambiguity in a way we never could before. Right.

So you can give it a high-level objective. You can give an agent a high-level objective and it can break that objective down and just maybe discreet the tasks, figure out reason through executing those and counter roadblocks, work through that in a way that you couldn't do with, you know, more programmatic approaches. Right. Okay.

So, all right. So, I feel like agent AI has to be the most hyped thing going on right now within artificial intelligence. Everybody, everybody's pushing agent to AI, you know, Greg Brockman at OpenAI called 2025, the year of agents, Krapathy on Dark Cash podcast recently pushed back on that heart. He said, no.

Yeah, it's really going to be one of the decade, the decade of agents because he doesn't think that we're really there yet. I've got some strong thoughts on this. Let me ask you this, where do you see value in agent to AI right now? Is there anywhere that you're using agent to AI?

It's a great question. Yeah. So, I think there are a lot of places that agent to AI can be used. And for me personally, I already use it for multiple places.

Coding agent is a obvious one. I think right now 80% or 90% of my new code is written by Coding agent rather than me. All right. And you're not vibe coding, quote unquote.

You're letting the agent run, you make it, I know this, you make it right test as well. And when they pass, then you go and examine the code, right? You don't just accept it. Yeah.

You need to not do the law. That's what I find. The Coding agents are very fast, very powerful. They can turn out a huge amount of code, sometimes high quality, sometimes it's in a different direction.

If you don't nudge it, it could go in circles, but if you nudge it frequently, it can actually go pretty far. I know Andrew Cabratti in his talk, he probably said that he didn't use the Coding agent for the nano chat, which is really a great piece of art. I don't even consider that as code. I mean, I've seen different opinions from very good engineers towards Coding agents, whether they adopted or not, some adopted a lot, but some really do not like it.

I think it really depends on how you treat your code, how do you view your code. If you view it as a piece of art, and you want to express yourself, if you want your design choices to be so densely represented in the code, like in an art, then you probably won't use Coding agents a lot. On that, because I've noticed that as well, there are some people who view code as a means to an end, right? They're like, I'm just writing this code.

Yeah, just to work. Just to work. I'm trying to solve a problem, building an application, and the other people, and I probably admittedly lean a little bit more in this camp where I view the code as something that I want to be beautiful, that I do care about. You know, I don't care about that, you know, just in purely abstract terms, but I do actually care about code quality and stuff like that.

Right. Yeah. Okay. Yeah.

I completely agree. On the coding, or at least on your side of the argument run, I think it really depends on how far you are into like the individual research, doing things from yourself, and you are the end user of the code that you're writing versus if you're building something that a whole team is going to have to inherit, if it's eventually going to go to production. And I think if the code inherently is throw away and you are doing some sort of research, that's great. Yeah.

If it works or not, and then, yeah, see you later. It's fine. You saved a day, maybe in some cases. Right.

Right. But yeah, if it's art that you're building, I still say that you need to treat code like art in order to build something that's a production quality subcomponent of a much larger system. I totally agree. I totally agree.

I think, you know, coding agent is probably the top one kind of killer app for agent right now. Completely agree. But I'm curious, what do you think are the next kind of killer apps for agent? One thing I can think of is document understanding, because it's so mundane, so boring, enterprisey stuff that is everywhere people need it.

Also personally, we also have a lot of digital thought documents that probably can benefit from some kind of agentic document understanding. That's where I started to use it some of the documents with the personal, like a locally deployed AOMs, but, you know, I'm curious what you think are other kind of applications for agentic. Well, Michael, what are you using? We're all doing agentic coding.

There's no question there. Are you using agentic AI in any other way? I would say not to any reasonable extent, you know, I think on the, I mean, I obviously use LMS on a day-to-day basis, just almost as like a, like a thought partner. And that has been a huge game changer.

And I think that anyone that's not doing that is probably going to be left behind. It's like not using Google, you know, 10 years ago, or even today, you're just walking around with an ankle, or a weight on your ankle. But I would say I'm not even close to as seasoned as you are as easy in terms of using coding agent specifically. And I'm finding with that use case in particular, there is a lot of slop in it.

Like at a certain point, you got to either trust in, like you can quickly verify a system and you just need to trust that the parts you can't see are doing what you expect. Or, you know, in some cases, I find that it does add some work because, you know, if you feel like you need to trust every single line of code that you commit, which I think you should, just starts getting into this weird, like, scary territory where you're either risk averse or you're not, you know, that's a great point.

I share your view that there's a lot of slop generated by even the best coding agents. There's a lot of try catches. There's a, you know, just a lot of bloat, duplicate functions that you, you know, later on when you view is, oh, why does it do that? And, you know, I especially dislike the try catches because I think maybe it's because they're trend to minimize errors that are reported by the code.

And one way to set down the errors is just to silence it. That's not the right way to do that. Oh, that's funny. You mean like try catches where they just eat the exception.

Oh, that's funny because in that, in that Andre Karpathy podcast, you're referencing, I remember him talking about how, you know, you can try to incentivize or you can use LLMs as judges in the learning process, but they'll always find a way to game things. They'll find these degenerate cases that make it really easy to just blow past, you know, the whole intent behind whatever learning cycle or loop you put on. Yeah, yeah. Have you, have you used GPT Atlas yet with its agented capabilities?

Oh, you mean the browser? Yeah, the browser. The browser? Yeah.

A little bit. I use Comet a little bit more. Yeah, I'm, I'm trying a lot of agentic browsers for, because I really wanted to automate part of my life. I'm right there with you.

Yeah. Yeah. It seems to me that for some reason, the, the go to example for agentic AI, I, I think for like the average person is like traveling, travel plans, restaurant reservations. Yeah.

And it's been that way for like three years, which I view as a bad sign, right? Like we're not, we're still, we're still trying to get relatively simple things like that working. So my take, my take on agentic AI, and I want to know what you guys think on the timeline. I've been saying for a couple of years, you guys probably tired of hearing this, which is agentic AI is going to be a big, big, big deal, but not yet, like settle down on a personal usage basis, that's fine.

It's great for a agentic coding, really, really, really great. It's the, it's the dumb area, but if you're a business, if you're an enterprise and you're looking at embracing agentic, you really, really need to be careful, because you know, we've got, we've got the problem, if it's, you know, 95% successful at taking an action, and then you chain together 20 actions, which you want to do, that's kind of the point for these agentic systems, is your probability of success really, really diminish.

Yeah, point nine, five times, point nine, five times, point five, yeah, that gets small pretty quick. It gets small pretty quick. I mean, I find myself in this weird place, and I feel this with several different AI technologies. I am as excited and, you know, bearish as anybody out there, but I also feel like sometimes people want everything now.

And if you look at, again, going back to the, the, the, the, the episode with Torqueche, you know, he was saying everybody settle down, it's going to be 10 whole years till we have AGI and agentic systems, and if you stop and think that's crazy, I mean, that's just 10 years, that's kind of nothing in the grand scheme of things, right? Yeah. And he, he's considered to be quite pessimistic by a lot of people just saying it could be a decade. Yeah.

Where are you on, I want to know what you both think about the timeline on real agentic AI, where you could, you can interact with the system and give it a meaningfully difficult goal. Yeah. I feel like it really depends on the use case. So only one end, there's the autonomous driving where the penalty or the cost of a mistake is so high, it's human life, to somewhere like, hey, brainstorming some ideas from my next story to tell my kid, you know, so I think it really depends on the use case.

So I like your example around about the agentic browser, it has a hobby, I'm also trying to build an agentic browser myself with, of course, you are with code and it's hard. And the reason I think it's, it's hard is that there's, there's, there are so many kind of edge cases and long tails. If you wanted to work on one side, maybe it's feasible, but if you wanted to work on, you know, all the different sides, form feeling and especially if it's handling your like sensitive information, feeling out the credit card information and you know, you don't want it to accidentally just hit the buy button, you know, so I'm just making sure that all the edge cases that the 999 that's mentioned by Andrew Karpazi, like, you know, if you're expected, your, your required accuracy is 99.

9999, 99% each nine is going to be, you know, a lot harder than the previous day. Yeah, it may take as, may take as long as that previous one, you know, each, you know, each reduction error. I think that's actually true. And I think that's the reason I'm, I'm excited, but I want to just temper, temper the, the hype a little bit.

Yeah. Yeah. Where are you on the timeline? I tend to be a little more pessimistic too.

You know, I think the, a, a, a, a, gintic AI this year feels almost like a solution in search of a problem. Yeah. In a lot of cases. Yeah.

And I think, I know the Gartner hype cycle isn't like the most rigorous thing in the world, but it's a good mental model of how these hype waves kind of play out. And I think something fuzzy and subjective happened in the last three months to six months, maybe where I feel like we crossed the top of that peak of inflated expectations. I think there are some real sobering realities that people are starting to get, you know, become aware of, but at the same time, there are so many like many technological revolutions that keep happening, like, you know, computer vision was huge in 2012 and beyond fusion models under LMS, you got agents, you got, you know, personal assistants, whatever.

There's still a lot of innovation that's happening. I just think two things I think are interesting. One is the importance of experts seems to have gone the opposite direction that people thought. Most people were like, oh my God, we don't need experts anymore.

We have LMS. They can do all this stuff. And now it's almost like people are coming to the reality that it's a tool that can maximize whatever capabilities you already have. So, you know, given that, I think we're still limited by whatever the speed limit of humans being involved is.

So I think the rate of progressions is probably still going to be smooth until we just totally hand over the keys. You know, the other thing is, I guess it's kind of related, but we are a speed limit. Like humanity, we have to refactor society and do all this stuff to respond. Even if we had artificial superintelligence today, we need to meet that technology and capability somewhere in the middle in order for it to actually transform society.

And we're going to, I don't think we're built for it right now, you know, that's a really good point. And this actually reminded me of another point that Andre mentioned in the podcast. So he's consulting for other companies to adopt AI. I think, you know, he mentioned that oftentimes he would advise against using AI, because, you know, there are other solutions that are that is better.

I think, you know, for us, it's recommending and, you know, advising companies to adopt or not adopt AI. I feel like, you know, maybe a few years later, there's a lot of under adoption of AI. People use AI too late, exactly. In recent years, maybe people get too excited and there's a little bit of over adoption in some use cases.

Do I see under adoption in other use cases? But you know, it's funny. Okay. So we, I think us three and most of the people we listen to, we live in a little bit of a bubble.

It's like a digital bubble. But I gave this talk this week up at the group of aerospace students in DC. And I was trying to get kind of teed up with getting a sense of the audience. I was asking a couple questions like, okay, raise your hand if you use LLMs on a day-to-day basis.

And one person in the room raised their hand. Oh, wow. And this was like, it was like 50 people in the room or something. Wow.

I was like, are you kidding me? Because if I would have talked to people that look like us in a similar room, it would have been the entire room just like, oh yeah. I thought it was just a no brainer, but that shocked me. And I think, you know, to your point and ZZ, there's this rapid aggressive adoption with people that kind of look like us.

The rest of the world catching up is going to be a really long journey. Oh, thank you. Right. It's easy.

I want to dig in a little bit more on what you said, because I think I really agree with your sentiment. I find our student. When you say companies are maybe underutilizing AI right now, can you give me an example of that? I feel like there are a couple of different use cases, a couple of different scenarios, but I'm very curious to hear your experiences too.

So I think part of this may not be purely technical reasons. So like there are maybe cultural reasons or like, you know, just inertia, like people just do not feel they want to try a more risky thing. Or that there is a fear of this new thing that may take over the workflows that may take over jobs while in reality, if you adopt that your efficiency gets better and you're your capacity to serve your clients become better. So I feel like there is sometimes irrational fear or resistance that kind of contributes to adoption of AI where I feel like internal workflows, especially like the mundane that so sucking part of the job that can be automated, especially I would say, you know, document understanding and automating very tedious kind of routing of the workflow, coordination of people's schedules, you know, yeah, I just wish that we, you know, collectively we can spend more time on creating useful things and helping other people to have a better life rather than fighting my calendar, you know, always being meetings, you know, stuff like that.

So I can't believe I'm going to say this, but Gary Marcus wrote something recently that I agree with, which I never thought I would ever say that, but, you know, he's a long, long-term critic of LLMs and AI and very much a pessimist and dissenter, but he had an article, I think it was in New York Times a couple weeks ago where he was talking about too many companies are rushing forward and trying to leverage generative AI for every problem, and they need to pause and think about domain-specific AI, right?

Meaning, you know what I mean by that is AI that is really good, maybe at one or two or a handful of tasks, but it's not a general AI. So an AI system that might be incredible at reading X-rays or detecting fraud or doing loan factoring or for an object detection, all these types of things that are examples where it's narrow, but these systems have superhuman capabilities. And to me, that's probably the thing that I'm most disappointed with right now in AI adoption is that everything you said is easy, I completely agree with, and then I would add on two few companies are taking advantage of their proprietary data and building narrow systems, and here's how it ties in with agent AI, and this is what I think will be the future for, you know, the next few years, which is agent systems that can reach out to these narrow AI systems.

So for example, if you are, if you're dealing with an agent system right now and it needs to calculate some numbers, we know LLMs are terrible at that inherently, you want them to have the ability to call out to a calculator, right? To some external service, some external skill. Well, I think once we start embracing these narrow domains a little bit more readily, we're going to have these AI systems that will be superhuman, but they'll be aggregated out, they'll be federated.

And I think that's going to be one of the real keys to getting agent AI into the enterprise. Totally. And that's a long uphill battle, too. That's not something you do overnight.

That's right. And I feel like there's this universal temptation right now to just kind of prompt engineer and ask an LLM what an answer is, but I think most people that have worked with them enough to, in serious context, to found, and you can't just upload an image of a, or like a mammogram or something. That's right. Because their cancer here.

That's right. That's right. That's right. That's right.

Just it's much more complicated than that. Yeah, I think I shared some, some of the scenes here. When I use coding agents, I don't remember how many times I type LS, just trying to list the files, but you know, I type it to the coding agents. Right.

Right. Okay. So I want to, all right. I want to, I want to kind of do a segue now.

This is the other big topic I want to talk about. And this is the AGI timeline, but I really do think it's deeply tied in with some of the topics we've already discussed. I don't know, oh, I have so many thoughts here. I don't know if we will have true agent AI until we have AGI in the sense of agents that are capable of doing all the different things you need to do to have them do are, of course, going to have to have general intelligence.

Now, that said, you both know that I feel like the goalposts keep getting moved on AGI, right? Meaning, I think for really any reasonable definition of how it would have been to find 20 years ago, we've achieved AGI more and more. It's no, no, you need to be, you need to be capable of any human skill, including now people are, you know, assuming physical capabilities like embodied capabilities. Right.

So this is what I want to ask you about. And there's a component of sort of biology here, right? How much do we need to borrow from biology as we improve these thinking machines? Are we blocked?

Do we think we can get to AGI with the current architectures? Or are we going to need wholly new ideas? And independent of that, what's your timeline for achieving AGI? Do you guys agree with capacity that it's a decade away?

Do you agree with some of the really excited people and say the value thing? It's next year. ZZ, I want your thoughts on that first. Okay.

That's a really good, but tough question. I'm actually more pessimistic than Andre in terms of the timeline for AGI. That's really because my definition of AGI involves physics in most, the body man, right? Yeah, I feel like more than 30 or 50 years because hardware is very, very, very hard.

Yeah. And because of the kind of the 999, you really need to get to high precision for certain tasks, you know, that are sensitive for, for, for error. But what's your other question again? Well, I'm just kind of curious there.

So on that AGI timeline, do you think, and again, you're including a body man, which I think that's fair. I think if you, if you include the ability to move in 3D world and do any physical task a human could do, which, you know, for example, would include playing sports, you know, at a professional level, right? We're probably decades away from that. But outside the embodiment issue, do you think we need new architectures, new approaches?

Are we too focused on scale? Are LLM sucking all of the research and money away from things that might be more biologically inspired? I feel like both model architecture as well as the training algorithm, there's, there seems to be still a lot of room to improvement. In terms of a model architecture, if you look at the recent kind of chemi, you know, KK2 thinking model and it's and it's pre previous models, they made some interesting architecture change to make the transformer layer more like a linear linear attention layer.

So it's also kind of related to Mumbai architecture. And that is more efficient. And you know, given the recent success, it seems like, you know, there are still a lot of upside to investment in the model architecture change. So I'm pretty optimistic, you know, in the next few years, maybe we'll see some actually new architecture that transformer we have been known for 10 years.

Yeah, 10 years now. And then the other side, I feel even more optimistic is, you know, the data that we use to train the models or the simulators, the world models that we train the agents within, as well as how we teach the model. And that's where I feel like we can learn a lot from human biology. Well, not really biology, but you know, learn from the cognitive science, where I think right now, the way that we pre train and post train LM is already pretty similar to how human learn.

Like the pre trainings is like, I just think so. All right. You think it's it's a more than how we learn. Love love to discuss about that.

I think I know where he's going. Yeah, yeah. And this actually relates to some earlier tweets from Andrew Kapati. Yeah.

Well, he really releases a lot of super valuable insights. So like the pre trends, like just reading a lot of book without fully understanding it, you know, just read, you know, in in China, you know, there's this saying, like, you know, if you if you read the book a thousand times, the meaning emerges to you. So it's like, you know, just just reading a lot. And then, you know, it becomes kind of implicit knowledge and you can apply it just by muscle memory.

So that's the first stage. And then supervised fine tune is like the teacher doing problem, doing math problem on the on on the blackboard. And then the students follow just the step by step. And then there's the RIO where you're just, you know, you're just given exercises where you would try and have the have the chance to fail and learn from your mistakes.

Right. So that to me is like already kind of a good analogy to how human learn. And I think there, there is a lot more that we can explore. For example, I think the recent thinking thinking machines blog post about the on policy distillation, where it's like you have a strong teacher model that critique the students generation step by step, not like just providing the reward at the end.

But just when the student model does it, the teacher give feedback. That is dense so that you don't have to suck this. Which is from the straw. Yeah, exactly.

You have so many things I want to jump on here. I agree with everything you said, ZZ, the, you know, that that that comment at the end about reinforcement learning and, and, you know, sucking signal through a straw. And that's the problem you, you know, you only get feedback at the end. That is one of the biggest problems with reinforcement learning and these new approaches where you had just much, much interest, denser signal, I think is huge.

And the other thing I've heard, and tell me if I'm wrong about this, but I've heard pre training, I think again, even by capacity, you know, sort of analogized to evolution, this idea that, you know, and LLM has all this knowledge sort of baked into their DNA into their parameters, right. And that is a starting point. It's the reason like a baby zebra can be running within hours, you know, that's not enough time to learn that, right. And there's an element of that with LLM's, but then you have to, the biggest flaw is the ability to learn or improve over your lifetime, right.

Which we have very few things, even remotely capable of that, which is why I think skills and as an external mechanism for agents is so important. So I really, really like that approach. And like, I want you, I want to hear what you think. Yeah, actually hearing you frame it that way is it makes more sense to me, because I mean, when I think about pre-train, I think about like the data inefficiency, probably compared to how humans learn, like the like neural correlates of consciousness, like all these things that happen in your brain that seems so fundamentally different to how we've engineered these artificial intelligence systems in a lab, like the fact that we train these systems with gradient descent, whereas in the real world seems like there's like heavy and learning or something like the neurons that fire together, wire together kind of thing.

It seems to me like there's this huge gulf between what the neurobiology folks and the neuroscientists are looking at interested in and what the computer scientists are interested in. And I don't know this for a fact, I'm really really curious what you all think. But I get a sense that one hand is not really talking to the other as much as they should be. Occasionally, I'll look at a like a comments thread on a hacker news post from a paper or something, and you'll see these people that chime in from the neuroscience world.

And they're like, we figured this out 20 years ago. Like, what are you guys talking about? I've seen that on multiple occasions, and it almost seems like this adversarial relationship, whereas, you know, the, I mean, I think this transition to AGI, regardless of how you describe it is going to be a pretty smooth process. And you think about how our brains evolved.

There are all of these sub components and subsystems that you can see evidence of in other animals. These are things that have evolved for a long period of time, and they've kind of been honed and tweaked. I don't know if what we're doing right now. Is it just sensitive to initial conditions?

Is it sensitive to the weird relative configurations of these sub components? I mean, something I think it's really remarkable is how little we use genetic algorithms in our modern, you know, way of doing AGI. I mean, I'm curious what you think, because that's your bread and butter. Mm-hmm.

Obviously, it's computationally expensive, but are we under utilizing genetic algorithms? I definitely think we are. In fact, I think I just saw it was yesterday, or last night that Alpha evolved came out with another paper. Have you seen this yet?

And they, you know, I only read the abstract really briefly, but they ran it on like 60 known solve problems in mathematics, and it was able to, if I recall correctly, improve on 20 of those. Right now, they do say, yeah, there was a human in a loop, and it was guiding and things like that. I do fundamentally think that there is probably an over-investment into, you know, gradient-based learning systems, like deep learning of things associated with that, but it's natural. We definitely should be looking at other things.

It works. I'm not looking to get worse than I'm out there. Exactly. It works.

We should be spending more time thinking about reinforcement learning while the industry is doing that now, right? In the last few years, that's really had a renaissance. I think evolutionary techniques are in a way sort of the final frontier in that you can bring them to, you can bring them in any party. There's not a situation in which evolutionary techniques fail because they almost need nothing, right?

All they need is the signal on fitness at some point. There is no gradient. They can handle incredibly sparse rewards. Of course, the downside is, you know, evolutionary systems can be highly inefficient.

They can be paralyzed pretty well, but they can be really, really inefficient because they're operating without a signal. But generally, I think we'll probably see my guesses in the 30s. It's kind of wild to say this in the 30s. I think we're going to see a complete renaissance on the evolutionary side.

Interesting. That actually brings up another point I'm going to ask you about, which is that it seems like humans use the world as a really crucial source of entropy in the process of learning. There's just a lot that we get from the environment that we operate in that allows us to generalize. It's like a forbidden experiment, but if you drop a human into a featureless sort of world and just watch what happened, I don't think intelligence would just naturally emerge.

It seems like the environment's important. We don't have that with just like pre-training on 15 trillion tokens as much as we would. I think that's true. There are experiments, one of the most famous being, it's a little sad, but they would take kittens and you may know this one where they would sew their eyes closed when they were little, when they were just right born.

I believe these researchers won the Nobel Prize for this because they showed that if you don't let the visual cortex receive signal for some period of time, those animals can never learn to see. It's literally baked into your biology, and there are other experiments like that as well. I think we're the product of billions of years of evolution. Our bodies know the environment.

We thrive on that entropy from learning. If I was going to critique the current architectures, that's the thing we're missing the most. These are models up for the most part are frozen in time, and we've got to get better about them learning over their lifetimes. It's so inefficient right now from a computational perspective, and they're also just sample inefficient.

I'm a big fan of lifelong learning. This is a human ability that is not replicated in any of the LAMs yet. Even the strongest coding agents, the next day when I ask, okay, what do you think is the next for the project? Sometimes it can read the previous conversational history and the recollect some, but it's never feeling like a call-heak.

It's almost like I've been coming to think of LLAMs as sort of like, it's like you have this amazing library that you have access to, a lot of its retrieval, and then you have a moderately smart librarian that's at the front that can go synthesize all of it for you. But for the most part, it's an exercise in retrieval. There's some cognition, but if you strip it away to that cognitive core, there's relatively little cognitive residue left over, and a lot of it's just kind of memorized some patterns.

I totally agree. I think that is a perfect place to stop the podcast guys. Thank you so much. That was an awesome conversation.

It's so fun. Thanks. Thank you. Thank you for listening to Hidden Layers.

This series is hosted by Kung Fu AI, a management consulting and engineering firm focused exclusively on artificial intelligence. If you have any questions or thoughts about today's episode, or if you know someone we should feature, please visit us at Kung Fu dot AI.

Related episodes across the Index

Other episodes covering the same guests and topics, from across The B2B Podcast Index.

  • Utilizing AI internally to iterate faster and empower smaller teams to upskill w/ Vivek Raghunathan #263The Engineering Leadership Podcast · on coding agents96 / 100
  • Why your research needs a “thinking cave” with Sarah KlingThe Curiosity Current: A Market Research Podcast · on Agentic AI89 / 100
  • Ep 92: xAI Co-Founder Unpacks the Future of Model DevelopmentUnsupervised Learning with Jacob Effron · on coding agents87 / 100
  • Decision Logic: The Difference Between an Answer and a DecisionThe AI Forecast · on Agentic AI87 / 100
  • KYA Won't Always Protect You. The Real Risk Is the Swarm!Fintech Conversations & Insights with Efi Pylarinou · on Agentic AI86 / 100
  • Why a $1.2B exit felt like his biggest failure, and the customer-obsession thesis behind AgencyThe GTMnow Podcast · on coding agents86 / 100

More from Hidden Layers

All episodes →
  • AI Is Designing the Next Cancer Fighter | EP.5383 / 100
  • Anthropic Code Leak: A Rare Look Inside Frontier AI | EP.5282 / 100
  • The "AI Bubble" Bubble | EP.5174 / 100
  • Did AI Kill Programming? | EP. 5072 / 100
  • Your AI Is Too Big, Too Expensive, and Probably Wrong | EP. 4978 / 100
Explore the best B2B HR podcasts →
All Hidden Layers episodes →