Judgment in the Age of AI with John Cutler

Speaker A: There are certain things where judgment only happens through repetition and picking up the signals. And one of my favorite stories there in Iraq, there were US soldiers who built an amazing instinct for understanding where IEDs were. And if you had asked them, how do you know where an IED is? They would say, well, I, I don't really know. I don't really know what I'm looking for. But it turned out when they did research, it found that they were noticing things. They were noticing the fact that a fence was bent a certain way, or there was a handkerchief left on a pole, or there were certain times of day where these things were more likely. And so that kind of tacit knowledge, judgment. But it was still single player mode. They were out there observing through lots of reps.

Speaker B: Here's the uncomfortable truth about AI. Uh, the gap isn't between companies. It's between leaders. Same tools, same access, same hype, but completely different outcomes. Why? Because AI exposes how you think your decision making. If it's unclear, AI will expose it. If your system is strong, AI compounds it. Cross the executives I've worked with, from Slack to HSBC to American Airlines, the ones who win don't use more AI. They use it deliberately. They build a personal system, capture their thinking pressure, test their decisions, turn every interaction into reusable asset. We call it the CSTA loop in action. And the results? They're pretty consistent. 3 to 4x more meaningful decisions, up to 60% less preparation time and faster alignment across their teams. This isn't about becoming an AI expert. It's about becoming a better decision maker. All captured in our book, Artificial Organizations. Out now, enjoying the podcast. Don't forget to subscribe and follow us on YouTube. This way you. You won't miss any episode. And it greatly supports our, uh, podcast. Welcome to the unlearn podcast, where we explore the opportunities and challenges shaping the future of work, leadership and decision making. In today's episode, I'm thrilled to welcome back John Kuttler, product thinker, systems explorer and head of product at dartwork, for a deep dive into one of the most timely topics in business today, judgment in the age of AI. Now, uh, as these AI tools weave their way into the corners of our organizations, they're not just accelerating how we work. They're fundamentally changing the way we exercise judgment, make decisions, and collaborate. Some of us are feeling energized by what's possible. Others are feeling uneasy about what gets lost. And, uh, when we lean in too heavily to the machines. In this candid chat, John and I unpack the shifting balance between individual and collective judgment, how AI is improving both and maybe impacting both. We talk about the hidden systems we all use to make decisions, often unconsciously, um, and why exposing and improving these systems is now more important than ever. John shares lots of stories from his own journey, building the platforms@dotwork for better decision making. It also illustrates how AI can extend our intuition and accelerate our mistakes, but also offers practical insights for leaders who want to raise the quality of their thinking and teams. Whether you're excited about AI, skeptical of the hype, or keen to just keep up, this conversation is packed with actionable insights to help you make better judgment. So let's dive in. John, we are back on the show again. My gosh, it's, uh, always good to have you. So thanks very much for, uh, joining us again. But this topic I think probably coming up more and more. Right. Like people, even since we did our first show together, these tools are really starting to permeate all aspects of business and they're challenging many people in ways probably they didn't imagine, uh, before. Some people are obviously a lot of scared about how these tools start to change the way that they work and behave. Other people are probably more excited and re energized about work than, ah, they've ever been. And then there's sort of everything in between. Right. And one of the fun things we've been exploring lately though, is this idea of judgment in the age of AI and how in some respects it's helping many people make better decisions. But at the same time, you know, people are also getting a little bit more lazier about how they make decisions and deferring to the machine, you know, for, uh. And in many ways it's how most people start to engage with LLMs specifically, they use it as a way to get answers rather than ask questions. So, you know, you're obviously deep in this. You're building a platform essentially designed around decision makings with dartwork. So what are some of the fun things you've been sort of just observing in yourself not only as you start to design systems in this space to help people, ah, as they make decisions, but also the human side from when you're doing these sort of conversations and talking to customers and people in the space about what they feel will be supports to help them make great decisions?

Speaker A: Yeah, I mean, a couple of things come to mind. One is that we build judgment in different ways depending on the context. Right. There are certain things that you can build judgment in single player mode where you can Sit there and you can read a book or understand something, and then you can test yourself and you can see, did I get the answer right? Or you know, am m I building a heuristic for this? There are certain things where judgment only happens through repetition and picking up the signals. And one of my favorite stories there in Iraq, there were US soldiers who built an amazing instinct for understanding where IEDs were in. And if you had asked them, how do you know where an IED is? They would say, well, I don't really know. I don't really know what I'm looking for. But it turned out when they did research, it found that they were noticing things. They were noticing the fact that a fence was bent a certain way or there was a handkerchief left on a pole, or there were certain times of day where these things were more likely. And so that kind of tacit knowledge, judgment. But it was still single player mode. They were out there observing through lots of reps. And then I think about my music days. You build a lot of judgment by comparing yourself to music that you liked, and you learned what great was looking like or sounded like, and then you practiced it and you tried to match things up. But then I also think of music examples where you built judgment because you were with other people and you were processing things together, you were working things together. There were more experienced people in the room coaching you. It wasn't just fill out the blanks for something like an okr. Uh, okrs are a great example. Okrs are simple, taught college students and high school students to create OKRs. Like, the actual intellectual capability to create a good OKR is within everyone's bounds. However, there's instinct that you build by talking about it with other people. So that's what fascinates me lately, thinking even how I see people use our platform. You see people fostering situations where they can build judgment together, supported by AI. And that's very different from a kind of box checking. Fill out a form, fill out a prd, check the boxes, submit it to some kind of repository. Those are the things that come to mind, those two things. One, that we create judgment differently in different situations. Expert judgment, experiential judgment, coaching, mentorship, et cetera. And then we also apply it in the workplace in different ways. Sometimes in single player mode, but often with other people.

Speaker B: So let's start unpacking, um, a couple of them. Right? Like building judgment. This is probably one of the questions we get probably most is that folks for many years probably aren't even aware of the judgment System that they use to make choices, Right. It's sort of this innate thing. Like people say it's a gut response.

Speaker A: Uh, it's like the Rick Rubin, I just kind of know. You know, I just sort of feel it. Right.

Speaker B: It's a great book. I really like his book. Right. But this sort of notion though, that no one's really self examined the system that they're using to, uh, make decisions. Right. Which is again is if you want to get better at something, you've got to sort of expose your system to yourself in a way, even if you're in this making decisions individually as you're describing. Right. Because if you can do it individually, you can start to model it at a team level as you're describing, like when multiple people have to, what's our decision making system? How do we test resiliency around that? So for people then just even to build this awareness in themselves, Right. Like these great examples you're describing that when people have to make high stakes decisions where there's consequences actually where they build experience. So this example of IEDs with soldiers, right. Like they're constantly in these high stakes situations, so they start building patterns and mental models, as you say, almost intuitively to themselves, that until they can expose that and make something that a system that other people can start to leverage, you know, that's how you start scaling. So think about for yourself then, even about how you make decisions. What are some of the things that have helped you be aware even if you're in your single player mode? What were some of the components for you that maybe you've noticed over time about how you're making decisions?

Speaker A: The first thing that comes to mind there, AI has been incredible at forcing me to think about how I was making decisions.

Speaker B: Yeah.

Speaker A: So I had an experience recently that stood out with a coworker where instead of giving them a prioritized list of things, I wrote down my internal scoring heuristic for how I thought about things. So think about something like DOT work. You know, I had one item which was, you know, does this demonstrate our unique ability to understand the history of items? And I described it in a story and I went down that list and I forced myself to do that. Now, if you've ever been in an environment where people created a spreadsheet heuristic for scoring or prioritization, you know, what typically happens to those? It's a game. People game the numbers up and down. And I'm actually reminded, I think it was two decades ago, I used a system called Pew Analysis, Pugh analysis. And one of the things about Pew analysis is it's meant for experts to sit together and then independently rank and rate and then have a discussion with each other about why they ranked things the way they did. That was the original intent. Now, if you've ever had someone in your company create a prioritization spreadsheet, the person dutifully fills it out. You get what I saw with a, uh, customer the other day, where they said 20 teams were saying they were all going to move the same metric. It's impossible, never going to happen, right? So going back to this thing is, uh, I'm dealing with this coworker. I thought, okay, what's the. What does the PM normally do? I've prioritized these things. This is the order that they're prioritized in. And. But now with AI, I thought, well, you know what? I want to be even more structured with my own thinking. Like, what am I thinking about when I decide that something's a good idea or not? Why is this idea to do, you know, a plain Jane kind of chat interface not interesting to me right now? I go into my thinking and I say, well, anyone can do a Plain Jane chat interface right now. That's not really highlighting what our product is great at. So I write that down, and it was amazing, because then I handed over to these folks and I said, well, I put a couple good ideas in there that I thought were reasonable. I've also put in a couple of my ideas that actually scored terribly against my own thinking. Here's the markdown file. Play with it as you like. Do this thing. So then it was about a couple days later, a coworker said, well, I've been obsessed with your list. I've been trying out these ideas all the time. I've been seeing now. Now I've gotten a very deep understanding of what you're thinking and why you're thinking things are prioritized the way they're. You know, originally I thought, let's just put a chat interface and let them just ask questions. And you're telling me that doesn't cut it for what we're doing? And so I think that's a great example of how AI, in the right context, can actually be helpful for extending intuition and extending what you're doing. Now, had I gone to that person and said, no, let's never chat again, I've decided, here's my heuristic, here's my things, just deal with it. That wouldn't have been effective if I hadn't evaluated it against what I thought. It wouldn't have been very effective if I hadn't challenged myself at every single point saying, is this really what the strategy is? You know, if I had just kind of closed my eyes and said, ship it, I wouldn't have done that. Now, I'm going to just contrast this with one situation I had today too where I, I had a problem of someone asked me, does this survey make sense? Now, I am a reasonable UX researcher, but I am not an expert in survey design. I know that people are experts in survey design. I know what LLMs can do. So I write saying, providing all the research that exists on what makes a good survey. I want you to create a heuristic for what makes a great survey. And I want you to give me all the research that backs these things, like why things should be phrased in a certain way, why you should have mutually exclusive options, why you should put behavioral questions, not just perceptual questions or whatever. And I make the rubric. And then what I did is I challenged myself. I went through the survey myself and said, with my not so expert knowledge of survey design, what would I have called out? And then I plugged the same thing through AI and I compare the two and then I try to write a thoughtful response back to someone. The reason why m I'm mentioning those two things is it's the awareness of what one is good at and where someone has done a lot of the thinking that's important. Like I know someone has spent their whole life deciding how to write great surveys. I'm not going to reinvent that. And so I'm using AI very differently in that sense. I'm, um, building intuition, knowing I'm still not going to be an expert, expert, expert survey writer. What I am really good at is product and prioritization. And that's where I'm spending a lot of time building a cognitive architecture that my coworkers that we can extend what my knowledge is to the team. And hopefully those two examples are a good contrast there where one known complicated problem with a lot of expertise, that's the survey design. The other one, an emergent strategy situation where intuition and tacit understanding is important, which is my prioritization example. I'm approaching them in different ways. And I think that that kind of awareness is the kind of awareness that I think leaders need to be good at. Uh, I don't profess to be great at all these things. I certainly mess up a lot with AI but it's all this process of building the meta skill to understand what you're dealing with.

Speaker B: Yeah. So I think one of the things that are just to unpack what you're sharing there, you know, most people are still in an experience where they're looking for answers from these tools, right. They're saying, here's what I think, tell me the right answer. And then maybe the next step is here's what I think, make my thinking better or uh, challenge it or improve it. And then I think one of the things that certainly I'm seeing more and more, and again, you're sort of a good example of this here too is starting to think how people's behavior is changing about how they collaborate with other folks like previously, you know, without these tools as sort of like your first round of, let's just call it like resiliency in whether it's thinking, whether it's a decision, whether it's a choice, whatever it is, whatever, ah, an idea that you're thinking could be a good product. Most people would in a world, maybe people are still in this world where they're sitting there asking themselves these questions and maybe the speed that they can get new information to pressure test their thinking would have been they'd have to go and search on the Internet and read something or try and find another resource or an expert or human that could sort of help them maybe Socratically improve their thinking to again build the resiliency into their thoughts or the action that they want to take or the choice. So that sort of again fascinating that the cycle behind this can shorten massively. Plus the quality potentially of the feedback that comes back as you're sort of like internally iterating and improving your approach, just, you know, it starts raising the bar. And then what's fascinating is again, as you're sort of alluding to here is that's an individual activity that people can invest their time, their energy to raise the quality of the thinking. And then when the quality is higher, you can start to engage teammates in, in ways that you're describing here, either by sharing that thinking with them to help them start bar raising their own understanding of your expertise, or if you're in a group of similar sort of experts or you're relying on someone else's expertise and I think, you know, you're coming to them with higher order, more thoughtful, more resilient, robust thinking. Right. And again, this is, uh, for me, when you start to think about how performance continues to rise by using these methods, the productivity of just being faster to get feedback or information, sure, that's table stakes. But the performance piece is that you're no longer, like, tapping someone on the shoulder to sort of pick their brain. You're actually coming to your colleagues with really robust thinking or really deep, thoughtful questions that if you need to ask the CFO a really detailed financial analysis question, you know, that they can see you've done the work and it's really leveraging their human expertise at the maximum level that you should versus the sort of. Hi, I've just got this rudimentary question. So for me, when you start thinking about the performance levels that this starts to drive and how either individuals and companies and teams especially start to separate themselves from what's happening in the market, is that these folks are making better decisions at higher velocity, but they have a greater decision advantage because they're pulling all of this information in to keep, like, bar raising the quality of their thinking and therefore, when they make these decisions. And this is probably a question now from the examples you're sharing, right? How do you think once. Let's take the survey example, you know, how do you think that surveys got better? It will have, like, you made decisions about what the survey was faster, you have decision advantage that you feel like the quality of the survey is higher. Or more importantly, you probably have less reversal, right? In terms of like, everyone, like, iterating and iterating and iterating and iterating. Like, it's actually our first go at this could even be in the 80 percentile sort of zone. That's good enough for us to ship to market and iterate. And that could be a product, that could be a survey. So, um, yeah, here, tell me your reaction to that.

Speaker A: Well, a couple of things. There is, I think that. Well, let's just use the survey example. What we're doing at that point is at least building our collective confidence that we've at least woven in some expert perspective on that. What happens normally with a survey, right? Someone tosses out a question, I don't think that's the best question. And someone else has another question, and before you know it, someone just says, oh, just ship it. Just ship it. And then you get this survey back. And then people are sitting there and you know, the difference between a, uh, 6 out of 10 survey and an 8 or 9 out of 10 survey can be massive in terms of the decision quality you take from it. So using the approach that I mentioned, a, we're also planting in our minds that there is a whole world of knowledge that many of the people weren't quite aware of. Someone has done a lot of thinking about that. So just knowing what you don't know can be a huge advantage. The next time you go and sit down to write a survey, you're like, wow, some people have really put some thought into this. Second, you might have that person on the team that says, wow, I didn't know that. There's this whole world out there. I'm going to go so deep into this world, I'm going to learn more about it than I ever have before. So there's an inspiration component to it. Thank you. Third, you have the person, uh, who normally just checks out. When they think that there's not really much validity in qualitative research, suddenly their ears perk up. Wow. People have done a lot of thinking on this. You know, I used to discount surveys because I read one article. I never thought about the behavioral perceptual divide. You know, that's, that's why mps is broken, because you're asking people to predict their behavior in the future, not ask them about their current behavior. I don't know. I'm just going through examples.

Speaker B: No, it is. The last examp is Reed Hastings didn't believe in, uh, doing customer surveys. I remember I had Gibb on the show, and he was talking that he had to drag Reed Hastings to, like, customer interviews because he thought they were. They were useless.

Speaker A: How funny.

Speaker B: Until he started going to them. So, yeah, it's good.

Speaker A: And that's a great example of you're going to get all sorts of people. I was chatting with an educator recently, and they had this great thing that stuck with me, and it's just the Pareto principle of education, they called it. Look, I used to teach college. They said, okay, 80% of the people in the class are there just to get a grade. And as an educator, you just have to accept that they derive pleasure outside your class. You're not going to sway them too much. You have a responsibility to try to do what you can for them to get the grade. But that's what I'm going to do. And then he starts breaking it down. Okay, now of the 20%, there's 80. 20. So 80% of the people there are impressionable and maybe curious, and you have an ability to sway them over. You have the ability to kind of sway their curiosity and keeps going down the thing. He's like, well, then there's the 80, 20 of. You've got these genius students who will straight A anything whether they process it or not. You know, they're like the expert students. And you sometimes invest a lot of time in them only to find out they weren't the pat and he's going down the things. But one thing I like about that is that, you know, I think that people imagine in the workplace that some of these things are going to spark everyone's imagination in the same way. And I think realistically, the AI stuff proves a point, right? When given the option to just get a grade or just pass the thing or check the box or do what they need to do, there's a fair number of people who will just take that option. And I will be the willing genie. You know, as Kent Beck puts it, like, they'll be the willing genie to be like, yeah, boss, you wanted to show that the numbers will go up. I'm going to give you the numbers to go up yet. Think of the opportunity there. It leaves the 20% of other people. So in that example with the survey, you know, they're probably one person in the room who's like, I hate these things, or one person doesn't really care what it says as long as they think it's going to support what they do. But there might be the curious person, There might be the person to sway over to the other side. There might be the genius student who's just waiting for a new topic to come up. So I kind of like that. I think what's bothering people now is that 80% problem where it deeply impacts people. You know, they think like they're just using it to get the answer they want. They're just checking out on this. They're just doing the bare minimum, and it really bothers them. And I tend to flip it around. We'll be like, well, okay, maybe that's true. But that leaves 20% of the people that are totally going to expand their knowledge and expand their horizons in this thing. So I don't know. I think that that's how I think about it in a lot of times. And when you observe people doing that, 80% just get a grade use of AI. An important thing this educator reminded me is, remember, they get their pleasure elsewhere. They've chosen not to bring their curiosity to this problem. But there's some other problem that they really like bringing their curiosity to. You just. You just have to kind of step back from that point. So I don't know if that helps point, though. Uh, that's how I've been rationalizing a lot of these efforts for myself.

Speaker B: Yeah, like again, the part that's always fascinating to me is that you know, there's a huge opportunity here, like, even with your example, with the survey, right? Like, how do we just keep getting better? That's, you know, the core question for me anyway, that I'm always interested in. And then if these tools as you described, like that example about where your team is, you know, uh, asking for support, for feedback, for input on something, um, how do you just keep making the quality of what you do better? Right? And so first of all, I'm like, you know, I'm curious how the survey went, right? Because again, all of this stuff, stuff, it's driving towards more quality decisions. You create a better survey, as you said, you get better quality of information from the people that you reach out to. Then you get better quality of information to feed into your product to make a better product that more people use to make ultimately better decisions. Right. And that sort of like craft of doing that work is exciting for many people, I would argue. Right. But there's also a craft of learning how to collaborate with these tools. Right. And I think, you know, one of the most interesting pieces of research that I discovered when I was writing Artificial organizations was there's this, uh, piece of research that shows that they did with Procter and Gamble. They studied 776 of their teams with Harvard. And what they found is that when you're an individual working in a way, there's a certain level of outcome that you can achieve. And if you pair up an individual, uh, with an AI, they can almost hit parity with a team versus what was really fascinating is if you pair up teams with AI, they can have three XD outcomes, um, than they would have had if they were just working on their own. Right. So this instantly was fascinating to me. And we actually ended up like re engineering how we do all of our ideation activities in our venture studio because of this. Because what we found is that you always wanted the human dynamic of that team sort of in a room, cross functionally challenging, pressure testing, coming up with the ideas, getting excited about the target customer and who you're trying to reach and how. And then when you hit that moment where you're like, if we're going to launch in Eastern Europe or Southern Europe, which is the best country to do, because the target we're aiming for and um, this populace and that's demographic and typically that would always result in someone having to disappear for maybe an hour or maybe six days to go find some information. And yet you lose the moment, you lose the momentum of that creative sort of excitement and energy. And yet what we were finding is that when we were, we would work as a human team to like take our thinking to a level and then we would, we would throw it to the machine to say right now let's ask a bunch of disc confirming questions and pressure test this and run five scenarios about launching in each of these countries with these sort of dynamics. And um, then interject, uh, you know, oil prices goes up, uh, to $200 from $100. What does that look like? Just the quality then again of the thinking, it just keeps getting higher and higher and higher and higher because you're able to sort of get the best of both human and machine intelligence to drive better outcomes. Right. And again, your example, whether it's a survey and the team are sort of like working like the survey is a product, we're working to get the best version of this product to the 80%, 90% that, that will ship it again, another dynamic to manage, like how long do you invest or time, money, whatever it is that you invest to get it to the point where you're like, let's ship it, whatever it is. And for me, this is again one of the exciting things to figure out about this stuff because the data is starting to show that when I think your language of single player mode, you can get to the level almost of a team, right? Because you've got a teammate, you've got this virtual teammate to sort of play around and pressure test ideas with. But teams almost have teams of teams that have access to this like deep, deep information. And um, if, if used well again, we're still figuring this out, can even push back in ways that help break the group. Think again. Another thing to manage when you have a group of humans together. So for me, like, that's super exciting, right? As you're describing it, even in the guise of building a survey or exposing your thinking to teams to one another, it just keeps pushing this sort of bar up and up and up and up. Right?

Speaker A: But you're down, I mean, or down. I think that that's. And I think that's. That, you know, I'll use and I would say a positive example versus imagining it going the wrong way. So we were at an offsite@dotwork and we did one of the most effective AI activities that I've done, which we were saying to ourselves, well, we're going to meet for an hour and talk about. Objection. Handling. People are talking about that. Something struck me at that point. I'm like, well, why don't I take my phone and we'll put it on record. And we're just going to start role playing objection handling right now. Let's have fun. So we were going and hitting each other with things, and then someone would go and I would respond objection handling to it. And I go to someone else and say, why don't you Objection handle. We did this for about 45 minutes and it was really fun. Then we took that and I go off into a corner and I come back with the DOT work v1 objection handling guide. It's our collective intelligence, our intuitive intelligence.

Speaker B: Yeah, yeah.

Speaker A: Organized to do that. Then what I did is I extended it because it turns out, as one would expect, that objection handling is a complicated problem. But it's a known thing. I mean, there's context that you need to know about your local context, but you can do it better or worse, and there's general knowledge about doing it better or worse. So then I went and I analyzed all of our. What we thought were great. So that's where imagine you had stopped right there actually. Well, imagine before people just had that meeting and then someone zoned out and then someone at the end said, you know, I'm tired of this, I'm writing the objection handling guide. So that's like the zero level activity. So then we were at one or two levels. Someone got the idea me to record us doing it and imagine we had stopped there. Well, then we would have had a lot of confirmation bias, like, well, I really liked how John did it there. And we would have told AI that we liked how John did it whether we liked that or not. And it would have biased to that. That's like two or three levels. So we tried to take it up a level to three or four level, which is why objection handling is a thing. Let's go and grade our own things. Right? So then we're kind of. Now we're humbled a little bit, right? We're like, well, the content was right, but we defied some sort of generally accessible rules about objection handling. Uh, so that's like four level thing. And then five level is. So we could have stopped right there. We could have written some cards and we could have said, you know, next time someone joins the company, send them to this notion page and they'll learn about that. Well, no, the next level is to write a skill that analyzes any fathom call, takes our prior objection handling experience matched with best practices, and then creates a like a guide or help based on a call about whether we did objection handling correctly. So we've created A repeatable system that drops it into slack and says, you know, hey, on the call, John didn't do a good job at objection handling, or maybe our concepts are wrong. Do you want to go and revise the heuristics based on this? And so I think that that's just a great example of, you know, it starts on one level where how. Uh, I would. I'm not gonna say. I'm saying like a lazy team would have approached it in an off site, like, oh, we did some activity, people were generally checked out and someone wrote the doc and we're done. And it goes up in a ladder of basically two, making it more. Um, there's four E's, which is the four Es of cognition, which says that your cognition should be embodied, which means we need to use our bodies and use our mouths and like use and interact with enacted. So enacted with other people and then extended. That's one of the ease too. So you extend it through the tools and agents that you have around it so you can. The five level option that I mentioned was all the ease, you know, it was embodied, it was enacted, is extended. Whatever the last E is, it was all the E's.

Speaker B: Awesome.

Speaker A: Yeah, awesome. Yeah. To do it. But. But you get the idea. And, and what I've been thinking a lot lately is that took three skills, one of which has to do with AI. One, it's individual metacognition, being generally aware about how you think yourself. Social metacognition is the understanding that people think differently or perceive things differently, and that a team might have different preferences for engaging, et cetera, and bring different things. Computational metacognition is understanding what the hell LLMs are doing and what they aren't doing, and therefore what they can be expected to be good at and what they should be expected not to be great at. Like, for me, I'm not surprised that LMS are really good at solving really hard math problems. I'm also not surprised when LM gives me weird advice because I'm generally understanding how they work. I understand what's going on beneath the hood. So if you only have one, you know, if you only know how you think and you don't know how LLMs think or how people think, you'll be in pure single player mode and you'll be bad at it because you won't really know how the LLM will work. But if you have all three, that's like the example I gave of the objection handling, which is the spark to know how to push that idea forward. To make it repeatable given the tools that are available. One person on the team didn't know you could just send off something to be transcribed in 30 seconds. That's where they were at. They knew transcription existed, but not that it was just baked into every tool now to be able to do it. So. And then the most advanced person in the room was already imagining it as a skill like I was imagining it as a skill ultimately where we were going to do it. So hopefully that example helps of like a, A tiering of capabilities related to it.

Speaker B: Yeah, like, and that, that's the point, right. It's like helping people understand how the game works and being self aware of their own game and then how that plays with teams and how it plays with tools. For me that's one of the most exciting parts of all this is that anything the behavior change I've certainly experienced myself, it's just becoming more and more self aware about how I do things and can I turn what might be intuitive to intentional or exposing that to others and that others can be teammates or teammates as machines. And that's a really powerful way to keep building systems that scale. Right. And for me that's the way leaders I think think. Right. They think in terms of systems, they think in terms of how we can sort of share the systems that we use to make choices so people can understand them, they can um, improve them, they can challenge them.

Speaker A: Some um, leaders, I mean no, I'm not like saying to dis on particular leaders, but I have noticed that certain leaders have certain strengths. Some of them which shine in this current age. Right. Like the, the leaders who are highly interpersonal, very like centered around m. Like they just. Some people are just self. Admittedly like I don't, I don't really get the systems thinking stuff. Like I don't think in terms of this lots of amazing traits but you notice that they're approaching this certain era a little differently at the moment. Whereas leaders who could imagine what they're doing is sort of who always imagined they were creating a context system in a sense that like they perceived it that way are approaching AI differently. So I'm just noting that one of the big things this is exposing is different mindsets and approaches to existing things can have a strong impact on how they're, how people are approaching the current moment, I think.

Speaker B: Well, here you go John. First line of the book, AI uh isn't replacing leaders, it's exposing them.

Speaker A: Yeah.

Speaker B: You know, and like this is sort of the moment really for me is that uh, when you start thinking about this, right, like the, as you're, as you're sort of describing like being aware or not aware, trying to suppress the systems that you have. And ultimately I think this is one of the places where judgment goes right, where it's challenging people, which is uncomfortable to sort of, they have to expose some of their um, systems. And when you don't have a system or you're not aware of your system, that is uncomfortable. It's actually extremely uncomfortable. Right. And, and the power then of the opposite, you know, to your good versus evil or uh, you know, the contrary option is if you can become aware of your systems and put them out there, you can improve them. That is the exciting thing for me anyway is that I don't know the best way to make a survey. I don't even know the best way to make a decision. I have a way I'm m making decisions but if I can put it out there, it's going to get better. My teammates, again how we've done it for years. My teammates would be kind enough to say, oh, have you thought about ever thinking about this aspect when you're making a decision right through to a machine that is a proxies for an expert or expertise as you've described. Right. And again the fun part, again this is why I get excited is like it just if you're willing to sort of put yourself out there, you can get better and your thinking can get better. And that for me is like the promise of this stuff is. That's why I get giddy even now I'm smiling here as we're talking about it because all I'm thinking about is more stuff that I am um, naive to that I would like to be better at that I might believe I'm better at, but could get even better, you know, like this. If you can come at it with this sort of viewpoint I think it's, it's an exciting opportunity. But as you say, it's uncomfortable for

Speaker A: many folks too and potentially draining. I was talking to someone the other day, they made a great point that sometimes AI can make it feel like you're on hard mode decisions the whole day because the easy stuff you're doing and one thing I loved about that description is uh, back to the 4Es of cognition and sort of how we think is that the downtime, the lazy time, the sort of pondering time is innately human right to do those things. And so I was chatting with the developer about this recently, not at dot org but at an event and they said, you know, I'm just so drained at the end of the day. You know, it was the little time I was spent twiddling my thumbs when like CICD was working, or it was the other things that were doing their thing when the good ideas came to me and I was kind of collecting these things. And so one thing that's funny is I love the appreciation that people are building for that. Even I think there's even a Claude command which is like sleep or something. And when you run sleep, it sort of processes the memories, it processes the day, which basically means it's sort of finding themes and giving you an index to search what you had. Which is kind of an interesting metaphor. But I think what's interesting, as I'm chatting with another leader, they said, I didn't realize how hard all this writing stuff was for my team. And, uh, what are you talking about? They're like, oh, the new system we have means that we write these briefs and I used to do these verbally and where I put the deck out there and I would explain the context. And I thought I was really good at this. But now, you know, my team's getting crummy result. They, they, they picked up one of these things on LinkedIn. You know, everyone's making, you know, like an AI chief of staff or something like that. And I think that the hidden blessing of all these particular things is people are realizing how hard some things were if you actually have to make it concrete. And that even goes to development. All these PMs saying, oh my God. I mean, I'm going to share a personal experience. The last day or two, I've been jumping in to do some coding@.org today. Someone wasn't looking at my pull request or something like that, or I didn't know. And I was juggling between things and, uh, I think I deployed everything. And I immediately had this terror in my eyes. So I get my co worker Daniel. I said, daniel, I think I just deployed everything to prod, like right now. I think I did that. And he's like, oh, don't worry about it. That's why I'm building out the checks. That's why we're doing what we're doing. If it relied on us all being at 100% all the time, we wouldn't have a very resilient system that we're doing. And it's like my instant respect in that mode because, uh, if it had been a normal John Pre AI, I wouldn't have even been in that situation to build that particular instinct or desire to deploy everything randomly to do it. But it just shows how it allows you to flex in ways that maybe you hadn't flexed before. Whether that's a writing culture thing or it's about getting your thoughts down on paper or doing various things. And I think that the, you know, my, the current thought I had about where it goes wrong is it can certainly accelerate any bad instinct so you can do something bad way faster. So I see these teams now saying, you know, popping up the same software development life cycle on the screen and basically taking the little AI Magic logo and just pasting it on top of every item and saying, you know, I'm done. And I asked them, like, does it, Is it really linear? Like, where's the feed? Like, where are people collaborating? They're like, I don't know, I guess this is our new SDLC leadership, uh, asked for the new AI infused sdlc. So I went to Claude and I said, what's the new, uh, AI sdlc? And Claude makes me a slide in HTML and pastes the AI logo over every single box that I had. And that's what I presented. And I thought to myself, like, oh my God, this is like, it's accelerating the crap, you know, to be able to do it. But the ability, if you've done great collaboration with people, to say, wow, how could I make that easier or more effective is just so wonderful. So the mix of this kind of best of times, worst of times at the moment, I think is just a feature of the moment. And it can be very hard to process for me, at least individually, but trying to balance the real optimism versus seeing things that I'm like, oh my God, it's just accelerating the race to the bottom of something. So, I don't know, just trying to mix the optimism with the reality in some cases.

Speaker B: Yeah, yeah, no, I hear you. You know, uh, you know, one of the things that strikes me as you're sharing before is there is this notion, I think, of productivity fatigue where folks are so, in many respects they see the. I've built 7,000 agents on the weekend and I've created all this great output, you know, or, yeah, I have this sort of pressure on myself that now that I have this support, I have to run multi threaded tasks and be just going from one thing to the other. Or there's a, there's a flex to your point around this, which to me, when I hear it, it's just like, well, hang on a sec, are you just accelerated? You're Running from meeting to meeting with an over overloaded head and no presence.

Speaker A: Yeah, the people who are like, I've got 15 agents working for them too. I mean some people, I mean I've known people personally who have gotten into almost like a really bad mental state. Right. Because it is intoxicating. It's sort of like an amphetamine driven IKEA effect. Right. You know, it's like, it's really about as, as exhilarating as it can get. You know, it's like, oh, you know, I mean certain industries have been hit really hard. Like the agile coaching industry is really difficult. Right. So you see a lot of these folks being like, you know, screw trying to help companies. I'm going to make my 20 agent productivity farm and I'm going to, I'm going to show those people, I'm going to build something to do it. And you see how deep down a rabbit hole they go because it's exhilarating and, and it feels, it's amazing to make stuff. It feels really good to make things. It feels really good to make progress. Progress like that. But then like as I was joking with a friend recently, they're like, yeah, I'm on the 20th of those projects right now. Right. So, so it's a, it's a, it's um, it's so enthralling and it can pull you in, which is, it can feel. I mean at dot work we see that too where you know, we have deep insights into how organizations work, which is really, really incredible. And so we have an intuition about where things are being made faster and where they're not. Uh, you know, generally looking at, I wouldn't say that it's like fully supported with all the quantitative data at the moment, but you get a sense of what things are going and you really do see that there's, you know, even in groups that are very sort of like AI assisted, you can tell that, that that's not the problem at the moment. Right. You can see that there's like a fair amount of overload. It's almost just ramped everything up to 11. And people aren't slowing down at the moment.

Speaker B: And I think to your point about slowing down and I think this is where intentionality in your systems becomes even more important. Right. Famously, Daniel Ketterman talked about these ideas of system one and system two thinking. And in many respects when you fall into productivity fatigue, you're probably over optimizing for system one behavior where you're just constantly making these snap decisions, snap decisions onto the Next activity onto the next activity. So one of the behaviors I've certainly recognized or tried to shift in myself is actually designing in time. Where I put her, where I ponder, where I block in my calendar, going for a walk around, uh, where I live for an hour, doing exercise during the day. Things that get me away from the hamster wheel of just running from one thought to the next activity to the next activity because you're not going to do your best thinking. And I think one of the fun things about this for everybody, I would encourage them to think out there, is that you have to start really exposing your systems for how you work and um, intentionally trying to design them. And part of design is rest, part of the time is thinking, hey, hey everybody. Thinking is an activity. You're allowed to do it. It's a good thing to do, you know. So this is again for me, some of the stuff I think we're figuring out both personally and as teams. As you say, you're starting to see data around it too as well. And your platforms is that, yeah, it's great, it feels good, the dopamine hit of launching another agent to do a task and so forth. But you also need to recognize, you need that time to think, to just marinate. Um, and that that is just as valuable as an activity is launching 7,000 more agents on Friday afternoon to do the task. And so it's great to hear you, you know, share that and think about that. Even the platforms that are being built

Speaker A: at this moment, intentionality is really. I, uh, mean if I were to pick out one thing, the intentionality is really where it's at. I have thought about this a bit in the sense that I've met highly intentional leaders who weren't necessarily. They equated the sort of systems angle to have more of a process angle. And that was just not their jam. And I'm referencing also what we discussed 20 minutes ago, which is that there's some people who are very good at many things, but they'd even self professed, especially in product management, product leaders said, I could care less what we do to create that outcome. What I want is that outcome. That's what I best. Right. Then there are those folks who, you know, swing way far in the other direction and it's almost intentionality for intentionality's sake. So they're just heavily process driven, not nearly as iterative as they think they are. They're kind of what, the extreme in the other direction. I think that there is this, uh, uh, I have just observed among Leaders is these tools are also. There's a folks in the middle who, you know, wouldn't see themselves as highly process centric. They also wouldn't see themselves as, you know, completely instinctive, intuitive kind of vibe leadership. And it's just a great place. And we see that, uh, we see that@dotwork too, because the platform basically lets you be intentional about the design. You know, the average product leader five years ago would say, look, this is going to be chaos no matter what. As long as we just get our slides to look right and have the conversations we have, that's it. And now the technology really does, uh, what attracts me to something like dotwork at the moment is there's a sacrifice people play in this, right, which is you either flatten reality to appease the lowest attention span person in the company, so you flatten it down to make the slide that looks the snappiest and then what that does are two things and both aren't great. One is all the frontline teams do whatever they need to do to do a good job and then have to translate that every day, or probably worse, the frontline teams just lockstep to that level of flattened reality. They just do it one way and they don't really adapt. Right. What I think is kind of amazing, um, from an org design standpoint now with these things is you could get both the ability to have the richness and variety, local variety of ways. And we see it when we see how people configure our platform. Team A likes to use insights and bets or whatever and is very intentional in one way and Team C is using capabilities and thinks in a different way. The ability to kind of translate that local variety into something that's legible by a leader has improved vastly. Like, it's an amazing time to be doing this kind of thing because you don't need to. I almost say it's like imagine if you had a team of a thousand chiefs of staff, your company, who dutifully sort of, sort of understood what was going on locally and then translated it to the global version and they did a great job of it without losing a lot of signal. Now you have that. And so it means that if people were on the fence about being intentional designers because they sort of threw up their hands and said, this is, this is a forever lost trade off. You either got to pick the simple decks or you got to pick the teams or it's. I just don't want to get into it now. It's empowering people to be more intentional. And actually like get the systems working. And so I think that's kind of an amazing phenomenon I think when I see how people are responding to it. So I mean that's positive I think.

Speaker B: Well listen John, it's been great to have uh, time with you again to dive into these topics. They're again fascinating. They're forming uh, all the good stuff that we're learning along the way. Thanks for sharing your thoughts with it and I'm sure we'll get you back on again to.

Speaker A: Yeah, I'd love that.

Speaker B: Some more fun ideas.

Speaker A: Yeah. There we go.

Speaker B: Thanks so much for listening. If you found this episode useful, make sure you subscribe on Apple Podcasts, Spotify or your favorite podcast app. Please consider leaving a rating or review as it helps others find our podcast. For more episode and details about the show, visit barryorilly.com and we'll see you on the next episode.