DOP 356: Warehouse Robots Are a Distributed System

Narrator: Everyone talks about summer like it's supposed to be carefree. But if this season brings up money, stress, body stress, family stress, or social stress, that's real too. Grow Therapy can help with that. Whether it's your first time in therapy or your 50th, grow makes it easier to find a therapist who fits you, not the other way around. They connect you with thousands of independent licensed therapists across the US offering both virtual and in person visits nights and weekends. You can search by what matters like insurance, specialty, identity or availability and get started in as little as two days. And if something comes up, you can Cancel up to 24 hours in advance at no cost. There are no subscriptions, no long term commitments, you just pay per session. Grow helps you find therapy on your time. Whatever challenges you're facing. Grow Therapy is here to help. Grow accepts over 100 insurance plans, including Medicaid in some states. Sessions average about $21 with insurance and some pay as little as $0 depending on their plan. Visit growththerapy.com booknow to get started. That's growththerapy.com booknow growththerapy.com booknow availability and coverage by state and insurance plan the

Tomash Kovachoski: optimal peak space is infinity, so that's a simple answer, but it depends on what kind of objects it is. So if you would want to pick X Infinity would probably destroy them pretty well. But it comes like, uh, it's grocery or it's, I don't know, rod ball that cannot be broken. But mostly I would say the speed is still limited by the dexterity of the Robot.

Darin Pope: This is DevOps Paradox, episode number 356. Warehouse robots are a distributed system. Welcome to DevOps Paradox. This is a podcast about random stuff in which we, Darren and Victor pretend we know what we're talking about. Most of the time we mask our ignorance by putting the word DevOps everywhere we can and mix it with random buzzwords like kubernetes, serverless, cicd, team productivity, islands of happiness, and other fancy expressions that make us sound like we know what we're doing. Occasionally we invite guests who do know something, but we do not do that often since they might make us look incompetent. The truth is out there and there is no way we are going to find it. Yes, it's Darren reading this text and feeling embarrassed that Victor made me do it. Here are your hosts, Darren Pope and Victor Farsick. Victor, when I say the phrase fleet management, what does that mean to you?

Viktor Farcic: It means that I need to deal with ETCD and I want to run

Darin Pope: away Meaning you're probably managing a bunch of EC2 or similar type instances and you're having to.

Viktor Farcic: Oh, in my case, I thought etcd in terms of fleet of kubernetes clusters.

Darin Pope: Oh, fleet, exactly. We're used to dealing with it from a software and a, uh, deployment perspective, but we don't deal with the physical side of things. On today's show, we have. Deep breath. Let's see if I even get it close. Tomash Kovachoski. Am I close?

Tomash Kovachoski: Yes, I think you're pretty close.

Darin Pope: But I'm wrong.

Tomash Kovachoski: I'm wrong just slightly here.

Viktor Farcic: Okay.

Darin Pope: Uh, he's with BrightPick. He's the CTO. And they do robotics. Like legitimate robotics? Like warehousing robotics, Right?

Tomash Kovachoski: Yes, that's right. Yeah.

Viktor Farcic: It's not those types of robots that you can put into your home and then somebody controls it telematically?

Tomash Kovachoski: No, no, no. These are proper, fully autodomous robots that really do the job, so they're really going around and do, uh, the job without someone really needing to control them.

Darin Pope: When I said fleet management to Victor, he said etcd. What does fleet management mean to you in the physical world?

Tomash Kovachoski: Yeah, exactly. For me, it means very different thing. So of course, I understand also for the software part, but for me, it's mostly really physically managing the fleet of robots. And, uh, in the physical space, like wherever each robot is, whatever it needs to do, and, uh, basically commanding them around. At the end of the day, it's basically same. Ah, so you have more orders, you need to have more robots, uh, doing the job, like moving them around. For me, it's more about the. I would say the algorithms and also the infrastructure that keeps the Warehouse going.

Darin Pope: What does an algorithm look like? I guess, to me, an obvious one not having worked in that space ever, is there's gotta be vision in there making sure I'm not running into each other, I guess vision, making sure I'm not running into the racks.

Tomash Kovachoski: So basically you're exactly right. So it depends, uh, what kind of automation it is. For example, for our auto pickers, uh, they are full aware of their surroundings, so they use LiDAR navigation. And we have, uh, algorithms for navigating around based on the surrounding, based on how the Warehouse actually looks. That's just one thing. The second thing, of course, is to understand whenever each robot is around, so we need to go around them. The third thing is it's a very different space from the software perspective. Like, you need to have several layers. How would robots react? So the first one is, of course, Everything works like it's supposed to. Then every of these robots knows its parts and when everyone else is. But you have different layers, and this is not always true. Sometimes you have network disconnections and so on. So it means that robots need to be totally independent. It needs to do decisions on its own. That is, I would say, a very, very important part of the story as well. So the robots are able to really function when connected, but also whenever they are working out just on their own. So they need to make a decision if something bad happens.

Viktor Farcic: You know, before we started recording, I had to leave and said, I'm back in two minutes. There were two reasons for that. One is to fetch my headphones. The other reason was to turn off my Roomba that keeps hitting the chair over and over and over again. So, uh, I cannot make one of them work without colliding with something all the time. I don't know how you do it for a fleet man.

Tomash Kovachoski: Yeah, it's pretty funny. And, uh, it goes to little things like network connection, which is something that was for me before we started doing robotics and fleet of robots. It was like, okay, so with WI fi does work. It's simply technology that was there forever and it's simply working. I was never having a problem with that. But, uh, when you have hundreds of robots inside a huge warehouse full of metallics, metallic stuff, and you're working around, then even things you take for granted are not that easy anymore. So we need to develop a lot of things, how to be sure that the robots stay connected, actually. So, yeah, it has its challenges. Yeah, sure. But, uh, at the other hand, I would say it's satisfying when you see something physical is, uh, moving around and is happening. So I would say it gives you a, uh, feeling that whatever you do, it just have really that kind of impact to the real world.

Darin Pope: So you said WI fi.

Viktor Farcic: Sure.

Darin Pope: Is that where you started? Can WI fi support up to five or ten robots okay. Without saturating the access points? Or is it just, ah, because of the infrastructure of the buildings themselves? It's like it doesn't really work out.

Tomash Kovachoski: I would say, um, it's a combination. You need to really ensure that you have dedicated channels for the robots and that they are not overlapping with. For example, of course we are always working with the customer and so the customer needs the WI fi too. So we need to agree how fluid channels and how we split them at the same time. Then you have roaming because like when you are sitting around, you're typically connected to just one AP and Uh, it's pretty okay. But as soon as you move around, you roam a lot. And whenever you roam you have just small disconnections. And we want to like, really wanted to minimize that and ideally get rid of disconnections at all. Because every little disconnection means that the system is not totally working to its potential because someone has delayed information. As I would say, that's something which is a little bit harder with robotics. It's not only about the, I would say the payload or the load balancing system as think of scale, but also the latency is important, uh, that things really happened at this time. It should, because things depend on each other, uh, and so on and every. But basically in the warehouse you have a lot of robots and basically everyone depends on each other. Who is the fastest to go to the crossroad and what is the exit order, they are going through it and if it's efficient enough and so on.

Viktor Farcic: You're mostly focused on warehouses. Right. How does the um, let's call it training work in terms that. Do you give them some information related to a specific warehouse where they're living or. It's all generic and kind of, I can put it anywhere type of situation.

Tomash Kovachoski: The robots themselves, they are pretty generic so you can actually work in almost any environment. But because they're interacting with the physical world. So you have toads, uh, they basically have an item for the customer. You uh, need to know where toads are in the customer system and then of course when these places are in the real world. So yeah, you need to have some kind of map that maps the digital space to the real space. And also like in that map, wherever is our. We call them storage position or toad positions. So where these are located and then we have of course a database where we uh, remember wherever and in what quantity of each items there is.

Viktor Farcic: You mentioned the database. Do robots themselves also update it? Kind of. Oh, I just picked up this package or whatever they're doing. Kind of like there is none left, things like that.

Tomash Kovachoski: That's a good question. Yes. So of course they do. So the whole system is basically connected. So they're internal, like individual layers. So the robots are really the end node. Then of course they are connected to a server system that is running on edge on the customer side. It basically keep tracks of everything it communicates with the customer. Wms, which is a warehouse management system. Whenever we pick an object, by the way, for that we are using trained neural networks, but they are pretty general, so they are able to recognize different kinds of objects. We of course, track the number of items that is left, stuff that really happens. Someone, for example, make a mistake and there is not enough of the items. So yeah, the AI can recognize that a bin is empty and then we need to take care of that.

Darin Pope: So my question here is, obviously there's a lot of telemetry going on, saying databases are being updated, revisit databases in a few minutes. You've got a basic training in there. You said it was general training, right? What happens when you need to update the robots? Like when you, let's say you need to push a vision model update to the robot itself. Is there downtime on that? Can you do it live? Visit Canary blue green. What's it like?

Tomash Kovachoski: That's a great question. Yes, we can do it live most of the time. So sometimes there are updates that need to stop everything. Like for example, when you're changing the layout, like adding something, uh, or something like that, these things are not backward compatible. So you need to stop the fleet. But for most of the things, we are able to do a rollout that is basically ongoing. And yes, typically we are doing the whole bunch of testing on our premise and our testing facilities. And after that the feature is, for example, solving some kind of problem, some configuration difference. Typically do a kind of retesting and slow rollout. So if anything happens we can act fast. And because like the fleet of robots, like there are multiple of them. So whenever there's no single point of failure in that. So if there is something wrong, for example, with the update, typically the rollback is not that big of a problem. On the robot side are, uh, decisions

Viktor Farcic: made inside machines or some kind of communication to some server that kind of like, okay, here's the input, I receive output and then I do this. How does that work? How much is decision making inside of the device as opposed to elsewhere?

Tomash Kovachoski: So the roads themselves, they are pretty smart. They have a fleet of normal cars with people in them. What would be perfect? If some central logic can control the whole traffic, which is almost impossible in the real world right now. That would be means that there would be a, uh, small amount of congestion and so on. So yes, this is the optimal part. If the system is able to do all the work, and this is what we typically want to do. But once in a while, things like, okay, someone's uh, slower than it should be, or something is on the ground and the robot needs to react and you basically cannot rely on the servers that they will act, so you need to basically override the action of the system. So the smallest part that is nearest to the real world have always have to have the highest levels of control. So we have actually in the Roboda, uh, even two layers. One is the software layer that is, let's say high level logic that is able to do things slowly and with good amount of time before. And then there is another PLC which is basically a safety gate. So if something gets really near, the robot needs to stop the hard way. There's actually really the several layers. So one, the master system, then the robot software part and then the safety at the end.

Viktor Farcic: What's the size of them? Is that kind of Roomba size? Small ones, big ones?

Tomash Kovachoski: I would say they're about like. Do you know what's these toads? Uh, in centimeters, like 60 times 40 centimeters toads like this is pretty standard in logistically it's used. So the robot have like two of them. So it says it's about half of the pallet typically. Uh, so that's the footprint. And of course they are tall. So we have three meters tall, as I would say a standard. And we have also some we call giraffes that can reach up to the 6 meters, but they are telescopic. So they're to basically extend the pillar.

Viktor Farcic: One thing I'm curious, can I walk that warehouse or is it kind of like no humans allowed or how does that work?

Tomash Kovachoski: The robots themselves, they do have the sensorics to work. Like I am totally okay working around the warehouse and uh, the robots will simply slow down and stop. So it is totally okay. But of course the warehouses are designed to be operated by machines because whenever someone is involved, it slows things down. And I would say the collaborative robotic is fine whenever there is a reason for it, but typically whenever it's not a reason, it typically slows things down. And it of course depends on what kind of application. For example, we do have a new product, uh, a grid picker and that's designed a little bit differently. For example, in that system, uh, they're absolutely separated from humans.

Darin Pope: What happens when you've tested everything out in your local environment? So you've got probably, I'm um, guessing three or four robots in a scaled down environment. Like you maybe have one aisle or an aisle to the landing spot modeled physically in your spot. And you test it out and it's great and you start rolling it out. Canary in the actual warehouse and it's so, so it's going okay, but then you cross over say 70% canary, and then things go completely sideways because scale. Right. Typically with normal Software, you don't see problems when it's small, but once you get it to scale, there's a problem. How do you roll back? Because it seems like, okay, I've already plowed out all this software, now I've got to roll it back. Do you roll forward? Do you roll back? What does that look like?

Tomash Kovachoski: We actually do have a process for that. So whenever something is wrong, we simply roll back to the last boot configuration. So it's not automatic, of course, it needs to be decided by the support. But it typically doesn't happen because in real world when you have the robots, they're distributed system mostly. But of course these things can happen. But it typically is not a pandemic or an epidemic that you just cross something and something happens too much. I would say sometimes there are subtle things that we did, for example, we did fix some problem and then we see on the statistics or long term statistics that things are a few percent worse than we expected, like, uh, I don't know, the cycle time and so on. This can be also the reason, for example, to go back, because that's exactly things we don't want to do. So it doesn't necessarily need to be like some epidemic. Typically it's hard to do because if you're really going one after another and of course when you have a small number of robots, you cannot be 100% sure that everything works great. But on the larger scale and you can see there's a lot of interactions whenever you start to see some patterns and problems, they're typically one of, I don't know, hundreds, thousands. So they are typically not that rapid, not that extreme or something like that happened. But of course when there is uh, really reason to roll back, we do rollback.

Darin Pope: How do you manage logs, metrics, traces? Do those even exist? Robotics?

Tomash Kovachoski: Yeah, actually quite a lot. Because whenever you don't measure something you don't understand, there is a Prometheus server that is basically uh, scraping metrics from individual robots. So that would say one thing. The other thing is the robots have something we call panics. So whenever there is something unnatural, some kind of anomaly, they are able to record kind of everything that they was able to see in the last couple of minutes or something like that. We have actually a system how these are automatically processed and uploaded and within our engineers are able to access them and replay really part of what was happening, how the robot interacted with other robots. Uh, what was happening with the sensor can go pretty low on the level of, okay, what is the individual angle of Some servos and um, uh, things like that so we can go really, really deep into understanding what is happening. So that's I would say more investigative tool and the Prometheus and then Grafana is more like okay, statistics. You have a fleet of robots which I would say a little bit different with the software probably is than when you have a different kind of, for example Kubernetes nodes. When you scale a system they are typically very similar and can do a similar job. Of course like if you have certain, let's uh, say CPUs maybe some of them are different a little bit. But on robotics if for example something wears off, you want to see an anomaly m between different uh, robots and that's I would say even more important maybe to look for them what's powering

Viktor Farcic: their brains is that kind of traditional machine learning is that the lamps visions, uh, what's behind it without revealing probably what you cannot.

Tomash Kovachoski: Yeah so it depends what part there. So for the picking, yes there are neural networks involved of course for the classical navigation and so on. These are more classical algorithms. So they can be like proven, can be tested. I would say we are now really trying to use uh, AI and LLMs for everything. But for the robots like once you solve, I don't know, a dijkstra and when you solve an A and things like that, it's basically an optimal solution. So for some of the problems there's no need for AI. But of course I would say that's the main part. So when we're picking, yeah, that's an AI when we're traveling around, that's I would say old school algorithm.

Viktor Farcic: Is that conceptually similar to what cars are ah, doing like Waymo and things like that? Uh, in terms of making decisions, move forward, go back, speed up, things like that?

Tomash Kovachoski: I would say it's similar from some points of view. Uh, we are interacting with other agents, basically other AMRs of our own fleet. So we have the liberty to know exactly what they want. So I'd say that's kind of the easier part. It's a very similar problem. But yes, we know whatever they want so they can agree. I honestly don't understand and I did see a lot of videos for example like say with Vamo and many times you were on the crossroad and you see the other guy in the other car and he's just doing something with his eyes and you say okay, I understand, I should go or he's going to go. The robots are simply not there and you don't want to Do a decision like that based on, I don't know, uh, some unnatural wing. Wing, yes, exactly. So from this point of view, robots can make, I would say, calculated decisions based on the real data.

Darin Pope: What is your tech stack? I mean based on what I'm hearing, it sounds like there's some sre on call type place that's probably co located with your test facility. Is that right from that perspective or am I off on that?

Tomash Kovachoski: The Nifto robots, they run basically some operating system and communicate with central node. We call it intuition, the application layer. But actually the server that's running kubernetes, pods and uh, that's on warehouse sites. That's on warehouse sites. It's always on edge. Like the main reason we can run on the cloud, that is not a problem. The problem is that we want to have very tight synchronization and latencies. As I told before, each individual robot and also the intuition, they're synced to the same clock. They really have like I would say almost one millisecond different drift between the timestamps. And that's really, really important for what they're doing, for the efficiency. But uh, we do have a centralized cloud that's running on the gcp. That one is basically orchestrating each individual part. So the individual nodes when they're running then they're using AGROCP to basically get GitOps. So we have git. Uh, this is basically a layer of truth. So this is the source of truth. Whenever we change something, the instance, then the individual on site servers day check for differences and updates themselves.

Darin Pope: Let's break that down. What's language of choice for the robots and what's language of choice for the software running in the local Kubernetes stuff? I'm assuming it'll be the same with the cloud kubernetes as well.

Tomash Kovachoski: It's probably analysis, it's 50, 50, maybe it's more on the Python side, but I would say that the backend is mostly in Python. In the robots themselves they're mostly in C because of the speed and I would say the momentum of uh, robotics is mostly probably still in C and things like that. And I would say when you build billing clouds then now we have more options. But uh, Python is still pretty popular so I would say that. And yeah, of course even uh, on the side of fleet management there are algorithms that are optimized and they're running on C, so compiled.

Darin Pope: Let me ask one question about maybe one of those algorithms. Let's say you have congestion, start building up. When I think congestion, I think about that Waymo video to where all the cars gathered in one place and couldn't get out.

Tomash Kovachoski: Yeah.

Darin Pope: Uh, what happens if you have a robot stall? Let's just say the rubber fell off the wheel. I'm assuming there was probably rubber on the wheels. Like just something happened to cause the robot to not be optimal. How do we keep that from cascading through the fleet and potentially causing a deadlock?

Tomash Kovachoski: It depends. Like there are two options if something is off but robot is still able to move. And also that's mostly the case. So in that case we're trying to get the robot out of the system. And uh, most of the time that's possible. And if the robot is not able to do that by himself, we have remote support that is able to basically like similar to Wymo. So it's able to connect the robot and is able to solve the problem remotely and take the robot to a maintenance position in case something would happen that then at all. As you said, the wheel will fall off, then the robot would be stuck there. And it then depends on the situation. One, if it's of course in a really bad place, then something needs to physically go there and help the situation. Or there are cases in which we can, for example leave the robot there and the uh, things can go around him. For example, in the case of the grid picker new product, the great thing is with something of that. I don't know if you are familiar with the products that we have, but Griffith Picker that we unveiled on the Mod X and the logimat, the robots are basically on top of a warehouse, on top of the shelves. And uh, whenever that robot will be for example broken, something like that, there is a lot of workarounds around him. So he will simply be there and can be there for ages basically until some maintenance will clear that position up. But in many cases it can just go around and system will just work with almost no change in the performance. But it of course depends on the situation.

Viktor Farcic: I was half hoping that you would describe something like a robot ambulance that you have a specialized robot that comes, picks it up, sends it to repair shop or something.

Tomash Kovachoski: Actually was thinking about that like, or awesome drones that can do something like that maybe in the future. But yes, it's still a little bit hard to prepare for everything. And yeah, sometimes it is good to have people around that can help.

Darin Pope: You were saying all the robots are able to talk to each other. They're also talking to that warehouse cluster. The warehouse cluster is Talking to the cloud cluster, just staying inside the warehouse for a minute. It feels like mesh networking at its finest. Am I incorrect in that thought?

Tomash Kovachoski: Although it would be a match if uh, the robots itself would say they would act as an AP or something like that. So the robots themselves, they are jumping between each ap. But of course because it's standard that the warehouse is physical infrastructure, of course it's better to run cables between the APs.

Darin Pope: Let me restate the question then. It's not so much um, the not mesh network, but it's a mesh network of robots because they are talking to each other. Forget about all the, okay, the worrying of it. Right. So because I wasn't being clear. So since they're all talking to each other and let's say we have that one robot go into one of those low cell areas of the warehouse, right? There's, there's that one dead spot in every warehouse, at least one dead spot. And we lose contact for five seconds for whatever reason. Once it comes back online, quote unquote back online, it's seen again. Is it dumping that five seconds of lost data back on the line so we know what was going on or is it just lost?

Tomash Kovachoski: So it did. The robot itself, it's half it's internal storage, so nothing is really lost for that purpose. But we don't want to dump all of that information because like most of the things like where I am and most of the time it's not that important. Whatever you was doing last five seconds, I want to know wherever you are now, that's the most important part. So actually at the beginning we had problems that we had the network stack designed in a way that after the reconnection everything was sent every message and then uh, the problem was that okay, whenever there was as you said, it was like five seconds off, then it have so much to say. Basically the silence was for several more seconds until he was saying everything he wants to say. So currently most of the sensoric data is typically trying to transmit the most recent part. Yeah, the rest is not important and it's recorded on the robot itself. So whenever. I don't know, you would need it at least some history is always there.

Darin Pope: What do you do in the situation to where a robot feels off as a human? Every once in a while we'll get up, just wake up one morning, it's like I just feel off. I'm fine, but I'm off. But there's no errors happening, right? Just off. We could sort of classify that as, I guess as A flaky test. How do you deal with things that become flakes?

Tomash Kovachoski: There's actually a good thing because like, this is where the statistics come in handy. So, uh, when you see each individual robot, you're not able to say something is off. But if you are seeing them comparing to each other and one is acting differently and is doing something wrong, and you already have a good statistical sample of the population, you can, okay, this guy, something is off with him. So for this purpose, I would say it's some kind of soft error. Depends whenever the maintenance is coming because of course is a robotic system. Every physical system needs to check up, needs maintenance. So then the robot is planned for maintenance. And if it's really bad, sometimes it's not crazy. So, okay, it works and we know that the robot can perform and it can wait and really do the job. Or sometimes it's better to get the robot out of the fleet, park it into a maintenance position and wait for service to come. And check, if you were to pull

Darin Pope: one robot out, you had 100 running, that means you have 99.

Tomash Kovachoski: Yes.

Darin Pope: Do you try to speed up the fleet to make up for that one missing?

Tomash Kovachoski: You typically don't lose one whole percent because like when you have more robots, of course, like every other robot, you have more traffic around. So typically, yeah, uh, you want to have some surprise of the performance around, if that's possible. But yeah, when it's off, it's really hard to really get the other part of the fleet to take the place. So yeah, uh, uh, other robots, like, they can maybe go a little bit faster, but if they would be able to do it, we would probably have them faster anyway.

Viktor Farcic: If you can have them go faster, why not go faster all the time?

Tomash Kovachoski: All the time, exactly.

Darin Pope: Well, because you may not have enough product in the warehouse for it to fill, then you're going to be down. So there's probably some management of what is optimal. What is the optimal pick speed, I guess. And how does it go from there?

Tomash Kovachoski: If you have like, if there are, there are like less, less work. Of course, the, the robots don't need to necessarily work all the time, or some of them do, some of them are not working, then I would say like you still want to go the full speed. Ideally, if you finish the work sooner than the better. Yeah, sometimes it, like even for the customer, like, they typically do have busy hours and busy parts of the year, uh, where for example, they're doing three shifts and some part of the year they are doing just two, one shift. So it's I uh, would say more managed in this way and typically better than like making the robots deliberately slower. Sometimes for example, when we do have two shifts, three shifts and there is a shift when we are not working, we do something we call a good night mode where the robots are moving around and basically waiting for the next shift because we don't have a uh, charger for every robot. So there would be like too many places even there they need to go around and charge and then they are going slower, at least for that. Yeah.

Darin Pope: What is your ratio of chargers to robots?

Tomash Kovachoski: Basically similar to what's the charging current. So I would say it depends what are the legislations. For example in some of the US places we have a little bit we'll say slower chargers sometimes depends what is the power grid can handle. But in some cases it's 8 to 1. In some cases a little less something around that.

Darin Pope: So people just get in line and wait. It's sort of like back in the seventies in the States when there was the fuel crisis and cars just lined up at the gas stations and slept overnight waiting to get the next tank of gas. It's sort of like that, right?

Tomash Kovachoski: A little bit different because like we know when the other guy will leave so we send the next robot whenever the slot is ready. So he knows even when he's at the other side of the warehouse. But basically similar but a little more organized in this way.

Darin Pope: Do you have to worry about Split brain at all? For people that aren't familiar with Split Brain, this is where you have two things working together and all of a sudden there's a network partition between the two and eventually they come back and there has to be some sort of merge conflict resolution or some variation. Is that something you have to deal with?

Tomash Kovachoski: Sometimes you have. Yeah. For example when you winnow picking. Yeah. So you, I would say the sacred part, what you do is when you move um, physical things around, when you do digital things you can roll them back but when you I uh, don't know, you pull something out of the tote, it's gone. So you want to have the digital world really copying the physical world. So sometimes I don't know when there is disconnection during these kind of actions, you need to get that information after. Okay A really find out what has happened. They'll say in that kind, this kind of thing. So okay, the robots know something, the server does know something. And yeah, in the worst case of scenarios when both systems are doing something wrong, then you need to decide even on the human Side. But that's happening basically almost never.

Darin Pope: You said warehouses that are doing this are designed for robots and not for robot human interaction, at least from an optimal perspective. Correct.

Tomash Kovachoski: There are places like we do human picking stations, so sometimes we have applications where the robots are doing something we call G2P or boots to robots. When we take basically what, say, unique in our machines that they are carrying around, I would say the basket, so the order of the customer and also the storage tote. So whenever there is an object, when we do not know how to pick, because the robotic arms station, the AI, they are able to pick most of the stuff. I would say the problem is not in AI, the problem is typically the fingers. Like the human dexterity is extremely, extremely good. And there are simply things that we're not able to pick and this is okay for them. We go to a human and he basically can interact and can pick. We have a tablet, a neutral mod, and he knows, okay, I want to pick three of these items. And it's very simple ergonomic operation because you just move things around from one bin to another so the robots can be lined up and he can work really, really efficiently in that way. So in this way, it's very collaborative. But of course, like from the way they move around the warehouse, it's separated. Like the warehouse are typically okay. This is an automated part and this is the part designed for humans. So they are typically separated places.

Darin Pope: How do we get to lights out? Like true full lights out fulfillment. Because to me, at this point, we can't do true lights out because we need humans today to be the distributors into the warehouse and then the robots could pick off from there.

Tomash Kovachoski: There's a lot of things happening and a lot of companies trying to solve also some individual parts of that equation. So you have obviously the picking. I would say we're very close to full automated picking. So that's one thing. Of course, we need to pack and then send things around. You have also the inbound. I'll say each individual part is solvable. Some I m would say are far away, but I would say still several years. Even with the automation, sometimes you need to pick the battles. So which problems are actually pretty efficient with people? Which problems are really tiresome and complicated and expensive based on duplicity? One pick, one item.

Darin Pope: What is the optimal pick speed? I guess it depends on size of product, robot, everything else. Is there an optimal. It always depends.

Tomash Kovachoski: The optimal pick speed is infinity. So this simple answer, but it depends on what kind of objects it is. So if you would Want to pick X infinity would probably destroy them pretty well. But it comes, it's grocery or it's, I don't know, a rock ball that cannot be broken. But mostly I would say the speed is still limited by the dexterity of the robot. So that's, I would say, still the most limiting factor. So it can be faster, but at the end of the day, I would say the traveling is still, uh, the biggest side of the problem. So actually this is something that we are solving in a little bit different way. We don't want to necessarily be fastest in every single thing we do, but we want to be smart about things that we do. With autopicker, grid picker, we don't want to move around because like typical automation in the warehouse logistics, what they do is they go from edge of the warehouse, where, for example, the human parking station is, they go into the warehouse, they pick some storage node and they go back all the way. And then one thing, two things are picked from there and they go back and et cetera. So it means that there are firehose places that create congestion. And what we try to do is we try to take the picking inside the warehouse. So it is distributed around that. So robots go and pick on one place, the second place. And, uh, then we can optimize things like traveling salesmen, things like that. So we don't need to travel a lot. And that's the key. So we travel smallest amount of distances possible. And then of course, we don't need to be a race car in terms of picking, because we can do it parallelly in the whole part of the warehouse that can be thousands of robots picking at the same time. And that's great about that. So I would say that the biggest bottleneck of the typical system is the congestion. It's not typically the action itself. It's the getting around.

Darin Pope: So it's the getting around is what really makes your slos make sense. Because if you were having to home run every run, like if everybody started from the same place and had to come back to roughly the same place, nowhere near as good.

Viktor Farcic: Is it always robots kind of doing the whole trip, or do you use also conveyor belts and things like that? Kind of, oh, I put it here, and then it travels the rest, uh, or something like that.

Tomash Kovachoski: We interact with the conveyors. So because like many customers have different kind of processes. So many customers do have conveyors. So sometimes the conveyors are the input to our systems. Uh, either if these are storage toads, or sometimes they are basically tolls With a customer order that already have a part of the order picked and it goes to our system when we pick other part of the order and then it goes out potentially to other part of the system. So they tend to be conveyors at the end and in the beginning of the system. But we don't use conveyors them much. So we are trying to be as flexible as possible. So I would say yeah, we are interacting with conveyors but not uh, typically using by ourselves.

Darin Pope: We know that AI is considered to be, at least from software development side, a great enhancer. Some people argue that point, but I'm going to stick with that for now. I will say that right now I am writing a program that and it's. I've been working on it for probably 8 clock hours. There is a 0% chance, if I could have a negative 0% chance it would be that, that I could have done that in that amount of time in a language that I don't know, on a platform that I don't understand. But it's doing what I want it to do. So I feel like as a software developer I feel like I'm empowered to do even more. I think in a warehouse it might be backwards because people are concerned about, from the blue collar, excuse me, from the white collar jobs losing their jobs to AI and we get all up in hands about that. But what about the people that have worked warehouse jobs their past 35, 40 years and now robots are coming in. What have you seen happen when you show up? It's like, hey, we're the robot guys. Are you fearful for your Life in about 20 minutes.

Tomash Kovachoski: The funny thing is that it typically do not happen because what you typically found out is that uh, most of the uh, most of the places there are missing picker jobs. So in us, in many cases I think was like hundreds of thousands of people missing in the position. And the job is not. It's pretty crazy if you think about you really do a lot of miles per day in a warehouse. It's not an easy job. So of course it's not easy. And of course you have a lot of it so you want to optimize it. So it's not the best paid job too. So I would say this combination makes like the job is not very attractive. So there was always a vacuum that we felt. So we helped the customer rise and solve the problem that he was not able to get enough people. So I would say that's more of the case to the future I would say and hope that automation will bring More opportunities and it's a little bit, I'll say that's for purpose for different discussion but at the end of the day the job is done. I would say the value is delivered. So I would say that the value should not vanish so the people should be able to get part of the value out of it in the end.

Darin Pope: What's one or two things that you wish the software community because you said you're more focused hardware, uh, what do you wish the software community would do to make your life easier?

Tomash Kovachoski: Sometimes I'm as me as a person, I'm more of a guy that have a lot of drive and I try to move really fast and sometimes I see a lot of people are ah, focusing more on um, the way they do things as the end result. So I for example am more motivated by okay, what I'm trying to achieve and what is the shortest way to achieve that of course like when things go in scale it means that things need to be clean and so on. I would say I would like most of people to think about not only how they do things like coding and so on, but what is the product perspective? Because what I found is the hardest and I would say this is the hardest thing for the people or the programmers now because like the craft is getting eaten by the AI as you said. But what is now what really matters is how well you're able to connect the dots, how think about a product, how think about the business value that it brings and how you are able to navigate in that space. This is really what matters because if you are just about the craft then it's going to be really hard.

Darin Pope: What's a prediction for this year for 2026 for physical AI? What's the equivalent of a production grade Kubernetes milestone for warehouse robotics?

Tomash Kovachoski: I would say I see a lot of that happening, a lot of great things. I'm looking forward for it. I would say it would still be hard when you for example, take your hand and close your eyes and put your hand, I don't know, in a bag and you're searching for the keys. You're typically able to find them um, pretty quickly even if you don't see anything is about the touch, about the tactile, like the dexterity of your hand moving around the back. And this is so natural for the people. I would say this will be probably a little bit harder than the intelligence. I would say the intelligence will be done sooner which was something that people, they think the other way around. I would say the intelligence would they'll be sooner than the ability of, I would say dexterity of the machines to do the general kind of job. Will see more and more of generalized machines. They are able to do 90% of the things. But I would say there will still be several years from. I would say full automation. But for, I would say for warehousing, I'm looking forward for humanoids working around everywhere doing cool stuff. But for something like when you want to buy a pasta and the pasta we will now never probably see a humanoid making pasta made maybe some homemade. Right. There will be factories because the factories they're doing it really good in scale. And I uh, would say in logistics, these are factories for shipping things out. There will be machines that are dedicated for that, not only because of the intelligence, but because of it does have sense because of the scale to have something that is specialized. And then we have a lot of intelligent robots that are generalists. We are humans are general creatures that can do a lot of things. And we'll see them, I uh, think on the places where they can do a lot of things that they will bring the biggest kind of value.

Darin Pope: What's one thing that your team has built that the wider software world should steal? Like, what's that one thing that you feel like, you know, if everybody would just do this one thing, then my life and everybody else's life would be easy?

Tomash Kovachoski: I would say it's a combination of things, but I would say that like the, maybe the drive that we try to have when we see that in the people around us, how motivated they are, how can they do things? And that's something that really everyone have people that can do things with that kind of passion. I uh, would say that will make a lot of things easier. But that's not a softer part. It's something I would say hardest. Steal something you want to grow.

Darin Pope: Well, if everybody could steal it, then it would be no fun, I guess. Well, you can find out more about BrightPick at uh, BrightPick AI. So you were early on in the AI. Did you used to be something else before you were AI?

Tomash Kovachoski: We started as a company doing three cameras and we're called Photo Neo. We sell the Photo Neo to Zebra. So we were basically photonea.com before that was the founding team and now they're proudly a new home in Zebra. So it was a really, really great achievement.

Darin Pope: Check it out at brightpick AI and all of. I'm going to try it again, Tomash. Am I closer this time?

Tomash Kovachoski: Yeah. Uh, Tomasz is good. Yes, that's very bright.

Darin Pope: Why don't you pronounce it one good time for everybody to hear, like Tomasz.

Tomash Kovachoski: But I always say, Thomas, that's a

Darin Pope: I was trying not to do that. I was trying to be cool, but obviously my cool was it was just crashing out every time. All of his contact information will be down in the episode description. Thanks for being with us today.

Tomash Kovachoski: Thank you for having me.

Darin Pope: We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are@DevOps paradox.com if you subscribe through Apple Podcasts, be sure to leave us a review there that helps other people discover this podcast. Go sign up right now@devopsparadox.com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.

Tomash Kovachoski: M. When I found out I was going to be a parent, I immediately felt a lot of anxiety and worry. So I went on to BetterHelp to try to look for a therapist to help me with that.

Darin Pope: Uh, my relationship with my family and with with my boyfriend and with myself were suffering. I really needed help.

Tomash Kovachoski: I was ruminating a lot. Really getting those thoughts out to a therapist and getting feedback was just life changing.

Darin Pope: Discover what BetterHelp Online Therapy can do for you. Visit betterhelp.

Tomash Kovachoski: Com today.