AI in the room, helping non-technical teams actually use it

Welcome And Conference Catch-Up Peter : Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and Dave Sharrock discuss the complexities of adopting new ways of working at scale. Hello, Dave. How are you today? Dave : Peter, fantastic to catch up with you.

It feels like it's been a while since we've really had a chance to have a one-on-one discussion here. Peter : It does. You've had a lot of travel, I've had a lot of travel. That's the way it goes.

Dave : Conference season does that to us. Peter : It does. Interesting conversations though, meeting new people, all sorts of fun stuff. Speaking of which, as I understand it, you've been running some workshops at conferences recently.

Dave : I've been to three conferences in the last few weeks. One that really stood out was the Global Scrum Gathering here in Vancouver. Very easy for me - I was able to walk out my front door and wander down to the conference. I ran a workshop on how to use AI when you're in a non-technical role: product manager, Scrum Master, agile coach, that kind of thing.

Specifically, how people in those roles are starting to change what they actually do day to day. Peter : That's interesting. I'm seeing it with my clients too. Everybody now has access to tools that let them create and generate things.

In some cases people are generating code, which raises its own questions. But the problems you were working through in the workshop sound really worth unpacking. Tell us a bit more. Why The AI Workshop Exists Dave : The starting point was a situation a lot of organizations are in right now.

They've told their people: you can use AI. Copilot is available, this tool is available, that tool is available. Go ahead and change what you're doing. What we see in many cases is that for the individual, it becomes a freezing point.

They're caught in the headlights. They don't know where to go. So the goal of the workshop was to gently guide people toward using prompts in a way that actually works for their specific role. We'd introduce an idea, they'd try it on their phones right there in the room, then we'd add a bit more structure, they'd try again, and they could see how each iteration produced a noticeably better result.

Peter : Coming from the technical side, we sometimes take that kind of thinking for granted. We're used to fast feedback loops. We've been doing it for years in agile contexts. Dave : Right.

And the people in the room - Scrum Masters, product owners, agile coaches - they absolutely understand iterative feedback and learning as you go. That's not the gap. The gap is that they don't have a clear picture of what they can actually change about their prompts. They get an okay result and stop there.

So part of what we were doing was giving them just enough knowledge to take the next step. You could almost see the lightbulbs going on as they realized that adding a bit of context, or describing what the output should look like, suddenly tightened up what they were getting back. It was a real shift for a lot of them. Prompt Structure That Improves Results Dave : The iteration is happening, but people aren't always sure what to change on the next pass.

One thing that got a big reaction was something pretty simple: going back to the LLM and saying, "Help me improve this prompt." That sounds obvious to anyone who's been doing this for a while. But for a lot of people in that room, it was a genuine revelation. Peter : There's something important in that.

You're getting more people actually using these tools to create real value. I've built quite a few of these systems now - experimenting with things like LangGraph, chaining agents together, seeing what different development toolkits can produce. But what you're describing is helping less technical people understand how to take an idea and iterate on it. How to help the system learn what you actually need.

That matters a lot, because I also run into plenty of people who tried AI once and walked away thinking it was all hype. "I got garbage out of it, I don't know what everyone's going on about." Dave : And in a lot of cases, they're already overloaded. They've got a backlog of work, they're being told AI will make them more productive, but they have blinkers on because of the stress.

That removes your creativity and your willingness to experiment. Which brings it back to that scaffolding of learning. How do you create an environment where people can see just a little further down the road, build on that, and then see a bit further again? Beating AI Overwhelm With Scaffolding Dave : Walking around the groups during the exercises was fascinating.

Some people in the room already knew all of this. But there were others who you could genuinely see thinking, "This is going to help me," and not just at work - personally too, because we all use these tools in our daily lives. The interesting thing was watching them realize just how much further the door could be pushed open. Peter : There's a real nugget in what you're describing.

When we have these conversations as humans, we create learning moments. That's not a new idea. But if we're trying to build organizations that can actually develop and grow with these tools, having people who don't understand something yet is actually valuable. They try to solve the problem in a way that nobody else thought of.

Dave : Exactly. And they ask the most unexpected questions because they don't have the context to filter themselves. There was a lot of that energy in the room. And then toward the end, we introduced something that generated a really interesting conversation.

The concept of agents talking to other agents. Orchestration. For those of us who've been doing this a while, nesting agents and feeding output from one into the next is pretty standard thinking. But for a lot of people in that room, it opened a completely new mental model.

Orchestrating Agents To Build Work Dave : The way we wrapped up the workshop was by explaining how the workshop itself was built. We've been running this for well over a year now, and the thinking around how to prompt LLMs keeps changing. So we actually created a group of agents that rebuild the workshop by looking at current thinking and evaluating whether the content is still relevant. It's a linear sequence of agents basically refreshing the material based on what's current.

Peter : I love that. I've used similar approaches for a variety of things. And as I was suggesting to you earlier, the next challenge is to have it run on autopilot, generate the workshop, and then walk in blind and see what comes out. Dave : You like putting me on the spot!

But that question of trust is real. Do I really trust it enough to walk into a major workshop with something my agents created, sight unseen? Probably not yet. But we're learning as we go.

Peter : Fair enough. At a conference presentation, the stakes are a bit lower than, say, an executive committee meeting. Dave : True. And one of the interesting things we're thinking about is how we evaluate quality in these multi-agent systems.

It's not just: what are the steps to build a deck? It's: who are the different roles involved in reviewing this thing, and what are they actually checking for? Target audience, workshop intent, design, speaking notes. How do we make sure all of that is being validated against real expectations?

Peter : There are two things I've built into every system I've worked on recently. One is a dummy LLM - something that lets me functionally test the system without burning through tokens every time I make a change. The other is a test harness with an LLM built in that validates the logic. At some point in every one of these systems, the only realistic way to move forward at speed is to build something that does the non-deterministic testing for you.

I create a persona of the customer, have it interact with the system, and then evaluate the outcome. Is it on topic? Does it fit within the constraints? That judgment is happening automatically, and you can even parallelize it across different personas.

Quality Control With Tests And Personas Dave : What I really like about that is the shift it creates. You move away from a linear optimization model - what are the steps to create this thing - toward something more like a team-based approach. What are the different roles that need to touch this artifact, and what are each of them actually validating? It stops being a checkout flow where you just go from A to B to done.

It becomes much richer than that. Peter : Exactly. So if we're going to wrap up for our audience, how would you summarize it? What are the three things you'd want people to take away?

Three Takeaways And How To Reach Us Dave : I'll give you two and you can add a third. First: context matters enormously when you're prompting. The tendency is to work with generic examples and generic prompts. But if you're a product manager, you have specific product management needs.

Generic exercises don't help you get there. Contextually relevant examples do. That was one of the clearest things we saw in the room. Second: the idea of agents as a team, and rethinking how we build things as a result.

Creating something and then using persona-driven perspectives to evaluate it - that quality control piece - really changes how you think about what you're doing. It stops being "build this using AI" and becomes "how do we manage this process in a way that actually produces something good?" Peter : For the third one, I'd say: don't assume everyone gets this. If you're close to this topic, you're probably surrounded by people who live and breathe it.

But a lot of people out there don't know how to apply these tools to their specific problems. That gap is much larger than we tend to assume. Getting people who understand this and people who don't into the same room - and helping them think through it together - is where a lot of the real learning happens. On both sides, honestly.

Alright, I'll wrap it up there. Thank you as always, Dave. Anyone who wants to reach out can find us at feedback@definitelymaybeagile.com.

We have some great guests coming up over the next few weeks. Don't forget to subscribe. Until next time. Dave : Until next time.

Thanks for the chat. Peter : You've been listening to Definitely Maybe Agile, the podcast where your hosts Peter Maddison and Dave Sharrock focus on the art and science of digital, agile, and DevOps at scale.