Co-evolving with Physical AI

Welcome to the Design Mind Frog cast. Each episode, we go behind the scenes to meet the people designing what's next in the world of products, services, and experiences, both here at Frog and far, far outside the pond. I'm Elizabeth Wood. Today on our show, we're exploring what it really means when AI moves off the screen and into the physical world.

To do this, we're joined by John Robbins, head of physical AI at Frog. For John, physical AI isn't just about systems, sensors, or autonomy. It starts with how we experience the world, what we notice, what we're drawn to, and how we share meaningful moments. It's also about the changes that happen when AI can move through space, share our environments, and take action alongside us.

Here's John now. At a personal level, I'm actually enamored by beauty. I live in Marin. My apartment faces Mount Tam and Corte Madera Creek.

And so I'm just like drawn by beauty all around us. I enjoy exploring new countries, you know, just experiencing new cultures and new cuisines. I'm based in San Francisco. I see beauty when I go to a ballet or a musical.

And as I was thinking about it, I realized that every time I'm experiencing a beautiful moment, my first instinct is to pull out my phone and capture that moment. I want it to last as long as possible. So this whole idea of creating products and experiences that bring delight and bring joy and create memories for people. Of course, leveraging technology is more than a job for me.

Hi, I'm John Robbins. I'm heading physical AI for Frog. I have an engineering and business background. I was involved in the creation and launch of one of the world's first 3D map products, which is basically a digital representation of the world, in one way, a representation of physical AI.

I remember, you know, very fondly at the time, my company president handed me a copy of Don Norman's Design of Everyday Things. And he was very particular about identifying user needs. And one of the formative experiences I had was, you know, taking a paper map. These are the days when we still use paper maps and sitting down with end users and trying to understand what did they want to see in a digital map.

So early on, I had my thinking was shaped around human-centered thinking and thinking about how does technology, not just how does it work, but also how is it used for human beings and how does it make our lives more meaningful. You know, fast forward many years later, a few years ago, I took a course in AI because of my background in navigation and the advances in autonomous vehicles. That is when I first started getting fascinated by the idea of AI for the physical world. Following that, I worked in connected technologies, sensor technology and IoT as well or industrial and consumer applications.

So throughout my life, I have worked in technologies at the intersection of the physical and digital. And I find that very fascinating because as you know, humans are analog, but we seamlessly navigate between the physical and digital world through technology. And so to be able to enable that is a fascinating part of my work. Physical AI is showing up in more conversations, but as an emerging field, it's often approached with assumptions and sometimes misunderstandings.

Here John shares what physical AI actually is and why the correct definition matters. When I think of physical AI, I think of intelligence that understands the physics of the world. It means understands force, causality, consequences. We all are familiar with large language models.

You know, LLM-based models process text or information. If I type, pour me a glass of red wine, it's basically a string of words, right? But physical AI understands the weight of the glass. its contents, the friction that is needed to move it safely.

So it translates my text into action. Basically, physically, I seize the world through sensors, understands context, reasons about what is happening, and then translates that into action. Physically, it's very multidisciplinary. It is one of those few fields where you need software engineers, hardware engineers, designers, strategists, and it spans the entire innovation funnel all the way from opportunity definition, ROI framing, through product definition, early concepting, to down-selecting those concepts and prototyping it to validate whether it works and is the kind of product that we want to build, and then finally building it and scaling it for impact and scale.

So yeah, our teams are involved in every stage, and in my role, I'm really fortunate to be able to get like a front row seat talking to clients who are ambitious, who want to impact the life of their consumers and users, as well as create new business value. So understanding what their aspirations and ambitions are and translating that into technical requirements and design requirements and bringing it to reality is kind of what is part of my role. This is where AI stops being something we look at and starts becoming something we live with.

For John, a lot changes when AI leaves the screen and begins to operate in the physical world. It means understanding that there are constraints like gravity and as I mentioned things like friction and distance, time, human unpredictability. It also means understanding affordances. Now I'm using a design language here, but what objects allow, what spaces constrain, and what do humans expect?

The difference between, as I explained earlier, AI that knows what a door is versus physical AI is knowing how to open the door. So it translates intent and text into action. It really is like making sense of the 3D world, perceiving it, reasoning about it, and then taking autonomous action. Sensors is a very broad term for multimodal perception.

So it can be voice. It can be recognizing voice or gesture or biomarkers, like temperature and all of those things. So it's multimodal and, you know, computer vision, camera-based systems as well, but they also fall under the definition of sensors. It's not just about seeing something, but deriving meaning from it Physical air of course includes robotics You know it one of the major applications of physical AI but it not limited to hardware There are many digital applications of physical AI as well.

I think the key idea is that the substrate can be digital or virtual, but the logic has to be physical, as I mentioned, understanding the physics or the laws of the world. I can give an example to explain a virtual application or digital application. NVIDIA came out with this system called NVIDIA Earth 2, which is a digital twin of the planet for simulating climate. So it internalizes the physics of the atmosphere, pressure, heat transfer, fluid dynamics, and it can simulate a typhoon at like two kilometer resolution in seconds.

So there is no physical product here, no robot involved, but it's pure physics grounded intelligence in a digital world. I mentioned about Fei-Fei Li. She's one of the pioneers in the space. She says that LLMs understand the language of humans.

Physically, I must understand the language of nature. I think she puts it really well. So there are many other applications. You can think of, you know, game designers and filmmakers who can now conjure entire worlds, scenes and perspectives, which is less time-consuming and less expensive compared to like the traditional production pipeline.

Think about in every manufactured object or constructed space, they're usually designed, again, in virtual 3D before its physical creation. But with spatially intelligent models, we can now quickly visualize structures and walk through spaces and understand how we might want to live and work and gather. But again, the core is those physics-informed intelligence layer. What's taking shape is an intelligence that goes past prediction and perception.

It's AI that understands how the world behaves and why humans move within it the way we do. Here John shares some of the potential use cases for physical AI. I think it's a combination of different factors. I think the immediate opportunity is definitely in robotics, is embedding intelligence into physical products that can do things that were not possible in the past.

I think the hero moment is going to be when physical AI intersects with human needs. Think about an aging population, maybe a robotic dog that can alleviate loneliness. Or think about a drone system that can ship supplies to remote areas. Those are all applications where physical AI can make a huge difference.

Is there going to be one main application? I don't know the answer to that yet. I think it's about understanding context. Just to give an example, if I were to ask you what time is it, you might say 12.

30, and that's a correct answer. Or you might say it's time for lunch. So the nuance is understanding the context. You know, it's not just a string of words.

You're able to translate my question into the intent that I have embedded in my question. You know, our sister company, Synapse, built a contextually aware robot many years ago, which kind of solidifies the point that I want to make. Imagine an Alexa system that we have used in the past. If you were to ask it, turn that light off, Alexa doesn't know which light to turn off.

It just understands your words and your language, but doesn't understand context. So Synapse built this contextually aware robot, Gerard. If you were to look in a particular direction or point in a particular direction and say, turn that light off, the robot then senses, it understands gaze and gesture and can translate that into intent and take the right action of turning the right light off. So that's what we mean by understanding human intent.

It requires multimodal sensing of voice, gesture, gaze, movement, and sometimes even emotional state. Imagine if a human stops suddenly. Is it because of fear or did they drop something? The system has to be able to understand those kind of nuances.

And we're not there yet, but there's a lot of advances that are happening in that space. As autonomous systems leave controlled environments, they start to meet us where we are. And that changes the design challenge entirely. Here John shares why the human must always be at the center of physical AI innovation.

I think increasingly we will see autonomous systems sharing human spaces. And so we have to ask ourselves, you know, how do we want technology to show up in our space and share our space? What form it takes, what expression it takes is really very nuanced. My colleague Ina Lobel can speak more about this, but imagine if you're making a coffee making machine, a robot that can just dispense coffee.

Making coffee for us is a ritual. It's more than just an activity. There's emotion involved in it. There is a routine involved in it.

Maybe you drink coffee with someone special in your life. So considering all those contextual factors of emotion and context and social cues is very important thinking about the form and the expression of the system that you design. Robotics itself is not new. It has been around for many decades.

But there have been advances in technology as well as shifts in how people think about technology, which is accelerating the advance of physical AI. Thinking about it first from the perspective of AI, robotics in the past was mostly confined behind fences in constrained environments. And they were scripted to do like a specific task, like a pick and place robot, repetitive task that it keeps doing again and again. But today, foundation models are allowing robots to learn from data, to learn from experience.

So instead of programming it to do something, almost like human beings, they're learning from our experiences, which we show them through different kinds of training models, but also through their own experiences of doing something and then learning from it. And that means that today robots are more adaptive. They're not just doing one set task. They can adapt to new situations that they have not seen before.

So that's like a big change when you think about it from the AI perspective. The other angle is advances in hardware and costs of hardware, especially sensors and compute, have dropped significantly. What that means is that physical AI is becoming more accessible. Maybe what costed, let's say, $100,000 to build in the past, maybe costs a fraction of that today.

And so in addition to being confined to high industrial applications or automotive applications physically I will start entering service contracts or consumer spaces as well So that the second piece And the third piece is socioeconomic factors as well You know, when we think about, you know, the evolving consumer needs, this push for greater personalization, faster delivery, is really what is accelerating physical AI in industrial settings. But meanwhile, you know, aging populations are driving use cases in elder care and structural labor gaps in retail are accelerating poor automation robots as well.

So it's this combination of technological advances in AI drops in costs of hardware and socioeconomic factors that's driving physical AI. We're going to take a short break. When we return, John will discuss the human choices at the center of physical AI, what we hand over to machines, what we hold on to, and how those decisions shape the world we're building together. How might empathic AI enable a world beyond apps?

What happens when AI unlocks a new collective wisdom? And will the line between humans and machines disappear? If you're curious about how trust, care, business, and creativity will evolve as the new human-AI collaboration rewrites everyday life, take a look at Frog's new Futurescape report, Artificial Realities, featuring 12 predictions and provocations exploring the future of human-AI chemistry. Find a link to the full report in today's show notes.

Now back to John Robbins, head of physical AI at Frog. So I think the technology is not in its R&D stage. It's certainly like in early stages of deployment, maybe not fully ready to scale. Because of that, I think we will see earliest adoption of physical AI in industrial settings.

What I mean by that is that industrial settings are kind of semi-structured. The workflows are standardized. The space is more predictable. And the stakes are low as well.

If something fails, it is not catastrophic. versus let's say if you think about like a home setting, it's a little bit, or maybe not a little bit, it's the most unstructured environment, unpredictable space. You have wires hanging around, you have kids running around. And so building robotics for home spaces is going to be tricky.

Or think about healthcare. I think that's going to be one of the biggest spaces for adoption of physical AI. But it's a highly regulated industry. Not have a system that hallucinates or makes an error.

And so both from a technology standpoint as well as from a business standpoint, I think the earliest adoption is going to be in the back of the house automation. As I mentioned earlier, there are systemic structural gaps in labor force. And so there's a clear ROI in the industrial space. And as technology matures, as people's trust in autonomous system increases as well, we will see adoption of physical in more consumer facing spaces as well.

All these physical systems are also data collection engines. If you think about whether it's just sensor in equipment, industrial equipment, or whether it's a robot, it's actually collecting data about the real world, interaction with human beings, automating different activities and tasks. And so it is both an opportunity and also a reason for caution as well. I think the opportunity is in integrating this new layer of data with existing digital data as well.

Systems like an ERP or a WMS system. If you think about integrating data coming from different streams, then you're able to create new categories of new meaning and new categories of products, new revenue generation opportunities. And so I think data is going to be a key and data will be a differentiator and potentially a moat for the company as well, because this is your own proprietary data. It's not just internet scale data.

This is specific to the enterprise or the application or the context in which you're deploying the system. So data is going to be a key differentiator. When I talk to clients, one of the things that I want to address thinking about what kind of activity do you want to automate as this autonomous system starts sharing human spaces? What should remain deeply human and what is it that you want to offload to a machine?

And number two is also calculating or understanding the business value of automating a task. I was recently talking to a client. Their business goal was to double the output without increasing the number of employees. And, you know, to build a system like that, the upfront capex was really high.

So it didn't really make business sense for them just to create one robot for one retail space. So I think we have to ask ourselves, like, what activities are automatable? And is there a business value? Like, mapping it against those two axes will give us the answer of what are the things that we want to really automate.

The tasks that are automatable are the ones that are usually repetitive and where the stakes are low, meaning that if something goes wrong, it's not catastrophic. The upside of automating a task is high because you're automating a repetitive task and taking away that mundane task away from a human being. And if for some reason something fails, the downside is very low. So that for me is like the criteria of deciding what to automate and what not to automate.

Sharing physical space with AI is only the beginning. The deeper shift comes when machines start to share our tasks and our time. During our conversation, John shared his perspective on turning from the functional aspects of the technology to its real values. I think the other important consideration would also be, what is it that humans want to hold and what is it that they want to fold?

What are the things that humans like to continue doing and what are the things that they don't want to do? Maybe it's dangerous, maybe it's dull and not as meaningful for them. So again, thinking about machines as autonomous systems that will share not just human spaces, but human tasks and shared decision-making, we have to consider what are those things that will keep humans at the center of innovation and make life more richer for them than feeling like they've been replaced.

When you think about systems today, they have human-like characteristics, which means they have the intelligence to ingest information, reason about it, and take action. They have that agency to decide for themselves without needing a human intervention The human is not programming and telling it what to do It learning from what humans have taught it but then it can adapt itself to new situations This is a big shift because if you think about it humans have always directed the work of other non entities right But now machines are becoming intelligent like humans.

They can see, they can talk in natural language, they can perceive, break a problem into multiple steps, and then take actions. And this is both exciting but also demands real caution. In a digital system, if a system hallucinates, we all have had experiences where you ask a question and it makes up something and makes it sound like it's real. You can use human judgment to understand that there is an error and you can correct it.

But in the physical world, if the system hallucinates, so for example, if you ask it to pick a pipe and it pulls a live wire instead, it can be catastrophic. It can harm people or it can cause damage to property. And so as systems become more and more autonomous and machines are able to learn a range of new behaviors, it really becomes harder to predict and constrain their behavior. And so building systems with guardrails, the operational limits, and having humans in the loop as much as needed is, I believe, the right direction.

So physical AI has to be predictable. It has to have legible behavior where humans are able to understand what the system is doing and why is it doing that and whether they can override it. As the field matures, the question is no longer just can we build it, it's how and why we choose to. Because ultimately, these systems will shape the world we live in.

When it comes to business strategy, John shared why certain principles must apply to designing, building and scaling physical AI. I live in Silicon Valley where the guiding principle is, you know, move fast and break things. I believe that when it comes to physical air, we have to design with purpose and engineer responsibly. I believe that we are not just creating products, we are creating a new world, a new reality in which we are sharing our lives, our spaces, our tasks, our decisions with autonomous systems.

And so we have to carefully think about the unintended consequences and also what kind of world do we want to create for ourselves and our future generations. The opportunity is real, but I don't think that the answer is robots everywhere. You have to apply robotics and physical AI where they intersect with human needs. And as I mentioned earlier, that could mean a robotic dog that helps reduce loneliness for an aging parent.

It could mean robot-assisted disaster evacuation or drone delivery of critical supplies. Regardless of what that use case is, then we have to identify where the pain points are and where it makes sense to have robotic systems take on some of these tasks. But the core principle is to design with purpose and engineer responsibly. I would add that when you think about great physical experiences, I would say that technology should never be in our face.

It should be invisible. It's a system that fades into the background and quietly makes life easier for us. We see robots flipping and sometimes you want to ask, is it cool or is it really creepy? We don't know.

So I believe that, you know, great physical experiences are the ones that make humans feel respected, empowered, augmented, and are taking on the tasks and things that humans don't want to do or should not do instead of managing them or replacing them. Amidst all this complexity, John shares how something simple always comes back into view. Humans are social. We attach meaning.

And any intelligence that shares our space will eventually share our emotional landscape too. I think human beings are social by nature. We like to bond. We like to connect.

Even when you think of something like Chad GPT or Claude, any LLM-based system, humans are forming a kind of relationship with it, a parasocial relationship, so to speak. So when you think about physical AI, robotic systems, these are systems that will occupy our space. So we will, over time, maybe form some kind of connection with these systems. So having systems that share our space and take on our tasks while being relatable and systems that you feel comfortable around and systems that create delight for you is where the magic lies.

I believe that physically I need both designers and engineers. Designers imagine what could be without any constraints, new futures, experiences, systems that could advance human life. I believe engineers bring that to life, bring that to reality, working with the constraints of what technology can do. I often like to think of them as, you know, the left and right side of our brain, analytical and the experiential side of humanity that kind of, when it comes together, becomes one cohesive unit.

I do think that there's a lot that we can learn from each other. I believe that engineers have to understand and develop fluency in what I would call soft metrics, understanding delight, trust, meaningful experiences, memorable experiences. And I feel that designers have to respect hard metrics, latency, reliability, safety margins, meantime between failures. So I think we need a shared language between both designers and engineers because design decisions become system behavior and system behavior drives human experiences.

And so it's, you know, when we bring those two sides together of imagining what could be and bringing it to reality, that's when magic happens. That's our show. The Design Mind Frogcast was brought to you by Frog, a leading global creative consultancy that is part of Capgemini Invent. Check today's show notes for transcripts and a link to download Frog's new Futurescape report.

We really want to thank our guest, John Robbins, head of physical AI at Frog, for sharing his insights. We also want to thank you, dear listener. If you like what you heard, tell your friends. Rate and review to help others find us on Apple Podcasts and Spotify.

And be sure to follow us wherever you listen to podcasts. Find lots more to think about from our global frog team at frog.co slash design mind. That's frog.

co. Follow frog on X at at frog design and at frog underscore design on Instagram. And if you have any thoughts about the show, we'd love to hear from you. Reach out at frog.

co slash contact. Thanks for listening. Now go make your mark.