90% of AI prototypes never reach production (w/ Temporal's Samar Abbas) | AI Basics

Speaker A: All right everybody, welcome back to our Basic series. What is our Basic series? Well, on this week in startups we get asked the same basic questions over and over again. Sometimes it's a legal question for founders, sometimes it's finance, sometimes it's about product market fit or getting customers. And increasingly and almost all the time it's about basic things in AI. Every startup is not just building AI products, but they're using AI to build those products. Right. It's uh, you can only be competitive today if you're using AI inside your startup. So today on AI Basics, our guest is Samar Abbas. He is the C E O of Temporal T E M M P O R A L I O if you want to go take a look at the website. And he's a 20 year engineering veteran, worked at AWS, Microsoft and Uber. Today we're going to talk about why agents desperately need harnesses and how enterprises can safely dip their, their toes into the agentic era and why he's building temporal on Google Cloud. Welcome Samar.

Speaker B: Thanks Jason for having me here. Excited for.

Speaker A: And so you've been at this for a while, you've been an engineer for a while. How have things just as a warm up here, how have things changed from when we were building products and services 10 years ago to how products and services have been built in the last 10 months or even the uh, last 10 weeks? Because it feels like it's changed even between those two time frames.

Speaker B: Yeah, yeah. Uh, so Jason, like I get this question a lot. Like a lot everything looks different. Especially in this world of uh, there's a clearly a new platform shift happening with the rise of AI. Essentially these models get smarter and smarter. One of the things that I see a lot is although the way people are thinking about applications is very different, especially with the rise of now coding agents coming in the mix, everyone suddenly is a software developer on the planet. And now because software is now more approachable to a wider class of people out there, not just like uh, people with computer science backgrounds, like everyone can build app if you have an idea. Coding agents are making those more accessible to everyone on the planet. So on one side it feels like the way application, the entire new application platform is emerging on one end. But on the other side what I see is we are seeing a very similar set of problems is that's why you hear a lot that oh like I have an awesome idea, I got started and built an application through a coding agent. But like 90% of those ideas die after a POC. Essentially they never see light of the day. And why is that?

Speaker A: And they die. At that proof of concept stage, people get super excited, but then all of a sudden there's like this despair. It's not stable, it's brittle, it's not replicable. What's the series of problems here? Because I see that all the time. People get excited inside my organization, they show me something and then I'm like, okay, is it ready for production? And the answer is inevitably, I don't know, it doesn't feel like it. But maybe, um, and that's really the difference between production level software and a proof of concept. There's a big gap there, isn't there?

Speaker B: There is a big gap there. And I think this is where the worlds are starting to look more and more similar to what we are seeing in this platform shift essentially is as more and more of AI applications, and especially these agents are starting to hit production, what we are seeing is they get longer and longer durations in time. The more scale, the more asynchronous they get. The m, uh, longer they run, the more actions they take in real world, the more value they create. It's so clear for uh, every organization that that's the pattern. But the moment they go there, you start running into the same set of reliability, scalability, durability challenges, infrastructure level, which feels very similar to when the cloud platform shift happening because uh, suddenly all of these agents are now starting to uh, graduate from running from your laptops into a uh, real distributed environment. And these problems start to look more and more similar to like distributed in nature rather than a completely new AI problem essentially.

Speaker A: And one of the problems is the durability of the execution. Everybody's had the experience, no matter what AI product you're using, that it's working, it's working, it's working and that's, it peters out, it stops and you have to start over. And you might have to start over a very large job. This happens in generative AI, whether you're making videos or images or sounds. This happens when you're doing research reports or you're doing big web research. Whatever the product or service you're building is, durability keeps happening. And then it doesn't seem to know because this is not, you know, perfect software yet. And these large language models in some cases are making their best prediction. It doesn't know where it left off. Why is that? And how do you solve that problem? This, this durability when running?

Speaker B: I think, Jason, this is basically, I feel this is becoming table stakes as these AI systems are maturing from like BOCs to powering real production systems, imagine an agent which is kind of building, processing a refund. And literally, literally, uh, immediately after processing a refund, a failure happens, a machine restarts, our process crashes and it is before even notifying the customer, it just dies. And I think this is where a lot of these, uh, POCs are getting stuck, is solve that durability problem. Because now what are your options? Either you reprocess, either, uh, that give

Speaker A: another refund, a second refund.

Speaker B: Yeah, you issue, it could get costly. Exactly. And so handling those failures are exactly the place where all of those production, uh, systems are having challenges as this AI starts taking action in the real world. And this is where a platform like Temporal is kind of becoming super critical to just provide. Like, at the end of the day, what we give you is, uh, uh, we describe this thing as durable execution, where during an execution of a code, if a failure happens, we remember all of that state, uh, without you as a software developer writing a single line of code for it. It means that if you process a refund and a failure happens or uh, process crashes, you don't have to manage that state yourself. It's durably stored for you underneath the COVID by the platform. And we will guarantee your application can keep on making forward progress in the presence of all sorts of failures. So this is exactly the kind of thing which is becoming the foundation which is powering these AI systems out there

Speaker A: now, which makes a lot of sense. It speaks to the fact that people are starting to move AI into production. They're moving it from just little projects inside an organization to actually doing the thing. And if you were to just. When, uh, you were talking, I was just visualizing like going to the counter where you have to change something with your flight. And there's a young person who's new in the job and they're trying to figure out your baggage and what seat you're in. And then this, um, person comes with a lot of wisdom, like, let me show you. And then they walk through it step by step and they fix it. We have that in the human world. We have people who are trainees, we have people who are mentors. And they say, okay, walk me through what you did. And then they fix. Okay, yeah, we're going to get these bags checked into Paris, no problem. And I was on your website earlier and it all came together for me when I was looking at, you know, this beautiful, um, agentic AI example or the subscription example. And here you have I guess in the top, uh, left corner your workflow, your code and then you have this temporal event timeline that plays, uh, and you can kind of watch it work. I don't know what the console is doing on the bottom left. What is that part of the screen doing here?

Speaker B: Yeah, so I can quickly explain, uh, as I told you earlier, we are the platform which ensures durable execution of your code. So the top left window is actually showing your real application. This could be code written by a developer. This could be actually code written by one of the coding agents which we have a temporal skills which is actually you can just express this is. I want to build an application which helps me research a bunch of things on the Internet and then consolidate the response and then it can emit a code which looks similar to what you are seeing in the top left window here. Essentially the right window is showing you actual runtime when you actually run that application. And it's doing a query. Uh, it's showing you progress of how the application is making progress in real time. You can see it's executing tools, it's calling LLMs and then getting responses back. And then it's basically uh, showing you how it's making forward progress. I think the window in the bottom is the actual output it's generating. That application is generating.

Speaker A: Yes. This really gets to the core of it. I've never seen somebody show me what's going on behind the scenes. And most people who were just asking a question or where should I go on vacation? I guess I got vacation on my mind here as we end the summer. But it could be shopping application, could be a research application. We don't actually get to see too much of what's happening now. We're abstracting that away from the client, the consumer using AI or even the enterprise. And that's for good reason. I guess you want to make it simple, um, but it is more reassuring when I see it playing out like this. I wish I had this heads up display available to me in every consumer product. It would make me feel a bit more confident of where I left off.

Speaker B: And Jason, you are making a really great point and I think that's one of the bigger powers that you get out of temporal is first of all it gives you a transactional engine which guarantees not only durability and reliability, but at the same time it's actually giving you complete visibility into especially these AI agentic applications, which is mostly like uh, what is an agent these days? An agent is an LLM. You pick an LLM you take a prompt and give it a set of tools. And these LLMs are smart enough where they are deciding what they want to do and how to make forward progress. So exactly to your point is what temporal not only give you a transactional engine which give you durability guarantees, but it gives you complete visibility into what your agent is actually doing to execute your task. To get the right outcome.

Speaker A: I think would assume that the person could then go in and tweak some parts of that. I've seen some really interesting products emerging, you know, as I invest in companies where they say, oh, when we make you this short 10 second video or this movie poster you wanted to make, um, we're going to actually show you how it's being built and then hey, here's where you can make a change to something. And so that's the, I think the ultimate possibility here. Yeah. Is that it'll give us the ability to change certain steps in the process because we have that insight into where it's breaking down or where it might be hallucinating or not performing up to our expectation.

Speaker B: Yeah, it even goes beyond that tweaking. Uh, first of all, absolutely. It gives you visibility. It can see what are the things it's doing correctly, what are the things you need to adjust essentially. And it gives you a way to go and make those adjustments to your business logic to kind of be more closer to the real outcome you want. But I think imagine uh, the vast majority of these agents being written. They are. Now there is this thing called code mode where an agent on the fly solving a complex problem by building more code essentially. And it's actually step in the process where it creates, build, emits more code and then executes that code. And imagine a larger enterprise who is kind of now starting to uh, transition these large business processes to these code mode agents. It's a very unnerving world from a security lens because suddenly you are literally running code emitted at runtime from an LLM into a business uh, environment essentially. So what temporal gives you, it has two constructs called workflows and activities. Workflows is all about generating commands. So even those code mode actually allows you to generate the code which needs to execute in the right a next step. But then it gives you an ability where you can intercept all of those commands and provide the necessary guardrails to secure that environment before it gets executed in a real sandbox or a runtime.

Speaker A: Got it. Uh, so talk to me a little bit about the role of the harness versus the model. We've heard a lot of people saying these days, hey, models are great obviously, but you got to get the harness right. Explain to people in plain English what that means to get the harness right and that the power and this next phase of executing on the potential of the models is the harness. What does it mean for somebody who's listening, who maybe can imagine it but doesn't understand practically what it means.

Speaker B: So uh, at least the way I describe this world is I feel right now we are transitioning from an Ms. DOS era of agents to a real cloud environment essentially because imagine mass majority of the people today, how they are running agents. They install something on their laptops and like a coding agent and then they are running those coding agents which is actually solving complex problems and adding value to their day to day work essentially. But clearly the value we are getting from these LLMs is now people have started to build loops or even there's a thing loop engineering or graph engineering which is being talked about a lot out there, which means that people are building these agents which runs more independently for longer periods of time. Which means these uh, agents are going to now move away from your laptop to a distributed environment. This is where a hardness becomes a very core component of driving execution is it's the brain. Where people are separating out the brain is if you give an agent a very complex task which requires to talk to dozens of agents underneath the COVID It needs to coordinate and it runs to do multiple tool invocations which themselves can run for m, uh minutes, hours or even days. To get some complex tasks done. You need to separate out a brain. So when an agent goes down the wrong path, it doesn't uh, break the entire process. So harnesses are these brains which is separated outside of the agentic loop to drive these distributed architectures.

Speaker A: How do people know when it's time to take that agent they've been playing with on their desktop and then say okay, I want this thing to run in a loop. I am confident enough, I've worked with it, it's my researcher, it finds me, let's say it's the sales department finds me my next set of leads and then I want it to put them into the CRM and then I want it to draft me a custom message and then I want to get them on a zoom call or whatever. How do we know when to move it off of our laptops and move it into uh, this loop where it's been given? Hey, your goal is to find me the great leads based on our CRM that are going to result in more sales and everything's going to be hunky dory. How do they make that jump? What needs to get involved when you're in an enterprise in making that kind of jump? Where it goes from my desktop I'm just playing around trying to make myself more efficient. But now I want this to be a corporate process.

Speaker B: I think it's for enterprises. It's even, I don't think enterprise can even adopt these agent tech architectures until they can run with all of the guardrails which are needed to uh, execute these business processes in those environments. Which means like none of the enterprises will ever allow these agents to run on a laptop essentially. So for them it's core table stakes in my opinion. And I think for it is basically more of a question for I think as a software developer. I think what's happening is like in smaller organizations people, you know, like people started with using AI by M, giving their prompts to ChatGPT and they're getting intelligent uh, answers back. Now we are in the next phase where people are building these agentic loops which is giving it a bunch of tools also and let the LLM invoke those tools to get their right outcomes. I think at this point running agents on your laptop is no longer even an option. So that's why you are already seeing these architectures emerging with sandboxes, memory and like all sorts of uh, uh architectures evolving where how people are recommending how you should be running these agents in a safe and secure manner. Even security alone is pushing and uh, uh making it a requirement that people should be running these agents outside of your laptop. But I think the other indication is at some point you graduate from just a single individual working on a problem to team of people. And now these team of peoples are coordinating with an army of agents running to solve complex business problems. Once you are in a world where it's no longer a single individual working on a task, it's a team solving on complex business problems, the only viable path is through running those things through an uh, agent in a uh distributed environment.

Speaker A: And that's where yeah for a engineer has to get involved. They've got to embed themselves in that business unit and use the proper tools and build it properly so it is not brittle. And that's I think what a lot of organizations are experiencing now. Huh huh. We're going to have the paradigm shift from I'm just going to do things on my laptop and have fun to hey, how does this affect the entire organization? Or it's a really interesting moment in time. The value is amazing, the opportunities are unlimited, but you're going to have to be thoughtful folks and definitely check out Temporal IO. Uh, Samar, thank you so much for coming on the program and sharing these important perspectives.

Speaker B: Thanks a lot Jason. Really enjoyed the conversation.

Speaker A: Find out more about Temporal As I mentioned by visiting Temporal I.O. and all of our AI basic episodes are available at thisweekinstartups.com basics thank you so much to to our friends at Google Cloud for making great products for all of my startups. Uh, we really appreciate that. And for supporting independent media like this week in Startups.