Who owns the product when AI decides? | Product Builders | AI-Native #3 | PRODUCT

Speaker A: M

Speaker B: Hi. Welcome to Product Builders AI Native, a community hosted by Boulder. This is a space for practitioners to share knowledge, exchange perspectives, and discuss real experiences in building products the AI Native way. Delivery strategy, discovery and growth, all in just 25 minutes. I'm Anna, uh, your host today. I hope you enjoy this episode. One more find Product Builders AI Native on substack.

Speaker C: And today's topic. Sounds simple, but, uh, turns out to be one of the hardest things in AI Native teams. And we see it almost every day, uh, in our company as well. Who owns the product when some of the decisions are already being made by a model. And, uh, it's the first, let's say, attempt to answer this question today from the designer's perspective. That's why with me is Kamil Marczak, a designer at Boulder. And Kamil doesn't work next to the development team. He works inside it, which is exactly what, um, and why he sees the design to code CMAP to close. Um, yeah. I'm glad that you are here, Camilla.

Speaker A: Thank you. Thank you for introducing me. You probably maybe had a chance to, uh, the viewers maybe had a chance to hear me out in the last episode from a month, uh, ago. Uh, if. No, I recommend you to try to go there and yeah, today we'll try to find answers. Uh, who is responsible for AI outputs in the company?

Speaker C: Yes. Uh, I want to start with some agreement up front, if it's okay for you. This conversation is not any defense of design, and it's not any complaining that design is taking something away from someone or us. It's a conversation about where accountability for decisions should sit in the teams like we built in the product teams. And what happens when it doesn't sit anywhere. It will not be any deep philosophical conversation about decision making, uh, as well, just the short discussion about our opinion about it. Is it okay for you?

Speaker A: Yeah.

Speaker C: Great. So let's start from, uh, like a real work and the real job. Let's imagine a feature, uh, where that design choice, whatever it means, wasn't made by a person person. The model made it and your team took it. The developers built it, QA passed it, it goes live. And there is a serious problem. Quite simple, but serious. And nobody made the call. Exactly. And nobody caught it. So who is responsible?

Speaker A: Some people would say that the last person, uh, that said, let's go with it. So, yeah, uh, I've even heard stories, uh, where, uh, issues led to trial, uh, in course in court. So people had to find out legally who is responsible. And they've tried to do that but uh, practically uh in a team like ours uh it would be hard to find out that person. Uh and I have great uh example of such uh issue from last year uh when I was experimenting with AI for the first time. I wanted to uh experiment with AI as much possible and for a meeting with a product manager I've designed uh a prototype. It was very much quicker than I prototype nowadays. And I came to the meeting and and I felt something like I'm presenting a uh design uh made by a colleague or someone else like uh, I mean an exchange of someone because uh uh I had a goal which uh. I presented uh to the ChatGPT or any other tool um and the AI has uh answered all the questions which uh could somehow be answered by me during a normal design process. And uh, in that meeting I felt that something is not right. I feel like I'm presenting somebody's else work that mentally I'm not the owner of the decisions uh which were uh, which were uh accepted, approved during that design process and I didn't like it. Since then I've tried to find solutions. How could I uh uh do uh something about it to use AI uh but together save that country role I'm supposed to have. Because without that who am I? Uh am m I somebody who just pushes prompts and moves output to another person? Uh not really. And yeah that problem with who is responsible uh in the companies in teams is scaling uh rapidly because more and more issues uh emerge and uh. I hope we'll find answer today because nowadays people push that responsibility from one person to another.

Speaker C: So taking it in the summary, do you have any idea who is responsible for today in the teams, you know or it's still a part of discussion or it depends on something because uh. As I understand right now it can be the last person as you say. What does it mean? Can be very different. But do you have any strict opinion who could be responsible? Like in average, in average or mostly or more frequently

Speaker A: it's hard to, it's hard to find one specific person. I try to believe that it's more a problem of a system uh actually because the way we use A.I. uh that process, that system doesn't acknowledge that there is somebody responsible for, for, for something. So I, I, I would say it's more of a system issue of, of a system problem. Uh if you uh. I've heard such story that if you need to, if the answer for your question is like it depends it means that you have a governance depth uh a term which uh which emerged nowadays, which is, which is kind of fresh, which many teams try to find solution for. But if you happen to be in such a situation that you have a problem finding answer for that it's a governance depth and uh, it could lead to big issues to be honest.

Speaker C: Okay, so of course uh, we can imagine that leadership says we are adopting AI, please do it. Um, and as you are trying to say right now, it should be not the person, maybe it should be systematically solved. Um but of course governance means many things. It can be on the C level like a C suit level before, like the first AI decision in the company. It can be decision at the level of the team as well. If you could say from the perspective of a team member what should be settled at the C suite level before the first AI assisted decision reaches users through the team, or what C level team can do or give you as a team member to have the clear governance.

Speaker A: Mhm. I think that between the output which is moved later somewhere else and approval there should be someone, one person responsible for that output. Because without that, I mean nowadays many people contribute to an output. For example something is uh, designed in cursor on something else. Product manager has its own opinion and does something with that output. Uh sometimes me, sometimes times developer, that line, that border, uh between those specialization, specialization, specializations somehow uh spreads because uh of those AI tools. So assigning one specific person, it could sound uh, uh a little bit like an effort. But uh I believe it, it's worth it. Assigning one specific person gives that person responsibility and it makes him uh or her stakes are higher. So if your name is assigned to something you are more careful about whether the sources the AI have used are uh, up to date because it's not always obvious uh, was the prompt uh proper to the goal you are trying to achieve. Uh, so there are many different questions which. Assigning responsibility opens the door to many questions which nowadays uh they are just passed by because nobody feels responsible for that. So one important thing for the C level is uh a system where people are assigned to yeah especially current AI features. Features which are already developed on production in the market somewhere you need to assign those peoples because without that uh, any issues, any incidents you will learn about will be when something will happen. And it could lead to not only um, issues regarding uh law, uh or finance, but also brand uh, and also trust uh users uh AI features which are left uh to itself can lead to issues uh, which could harm people, harm the users at the very end. So that could lead to lack of Trust in the end. And that's hard to rebuild, uh, in comparison to for example, technical debt, which is maybe not, uh, it's common, maybe not the easiest to solve, but it's much easier than working on the governance debt which we are currently talking about.

Speaker C: Yeah. And uh, I saw in some places that uh, it's even called accountability debt.

Speaker A: Yeah.

Speaker C: Because uh, the governance is one thing, but at the end accountability is the second one. But let's, let's do it maybe differently because I feel right now that I do not have uh, the answer from you. Maybe because the answer doesn't exist yet. So let's get back a little bit to the background of your work. And you don't design beside the team, you design inside it, your decision. And design decisions become code almost immediately right now. So as you said, uh, the line between designed and developed or built is very blurry. And this is the moment maybe where we can find the biggest blind spots in accountability. Maybe. You have any ideas how we can find these blind spots between developers and designers or the process between design and developer?

Speaker A: I've actually had a chance to find a few myself because uh, if you had a chance to attend any meeting recently, there's probably the same amount of atoms on the meeting as uh, attenders, as, as people. And I've tried to use that somehow because during meetings, uh, in a product trio with product manager developers, uh, we discuss things, we approve, uh, specific decisions. And I came up with an idea. What if I somehow, uh, if the Fatim is already in the meeting, maybe I could express what I would like to change, use its transcription from that meeting, put it in the AI tool and uh, I wouldn't have to make my work twice. Somehow. That was my, that, that was my thought at the beginning. But generally what happens, what decisions, uh, AI takes out of that transcription, that's the situation. That's the blind spot. That's the blind spot. And uh, taking that into consideration is one thing, but the other thing is that um, and I saw that myself, that AI sometimes loses uh, track of what was happening before because the context is, is getting too, too, too big. So apart from uh, understanding the, the feedback I gathered in the transcription, somehow it adds up different features, different things which weren't before. Uh, sometimes it loses track, sometimes the, the, the features which were supposed to be global in the whole product, sometimes AI loses it. And then I had a situation where I uh, didn't know that such occurrence happens. And the developer asked me why uh, that feature is here, why there's that button and that's the blind spot. It's a very small scale, but I can imagine how could it scale into something much bigger. So without a proper review, without proper uh, accountability uh assigned to for example me, uh, that design could go into, into production and lead to later on, lead to probably uh, some, some issues. So. Yeah.

Speaker C: And from the practical perspective, um, what does a decision log even look like right now in a team where there is a designer and developers and uh, you work together. I remember how it was built before AI and right now is a model or assistant or any AI, let's say companion, a separated option to take decisions in decision log. Or you combine your work together, you a developer and your assistants. How, how this decision look, um, is built today? Is it a real discussion who took this decision and when and so on? Or it means that your assistant is a part of you as a designer or as a part of developer, as a designer, as a, as a developer and who is in between? Because you build code, developers build code. Do you have any experience about this um, with this uh, in a teams?

Speaker A: Yeah, I've tried to experiment with that. And uh, to be fair I started when I started a ah conversation with the AI. I often tried to finish it with a uh design log. But uh, at the very beginning that decision log only had uh, uh decisions only made by me, uh individually. So not AI, not the whole team, just me. Later on I discovered that some of the decisions are made by AI and uh, that's not always uh an issue. It's great to be uh, conscious about it. Conscious what type of decisions are made by AI. But later on I've updated my uh, my assistant, uh you could say to collect decisions made by me and made by AI together. But it's still tricky which decisions AI will identify as its decisions because it's not, not so obvious uh, to be fair, uh, as teams we are still experimenting. It's still uh, something which um, might lead to conflicts uh between different decision logs. They don't always uh, acknowledge the same situations. They lack different decisions. Uh, so uh, it's hard to connect uh, all of them. But I imagine a situation where we for example iterate on the product, on the, on the design and all the decision logs are in the loop to get merged together. So I can imagine such situation. I hope that in the next half, for example in the next half year we will be able to, to do that. Uh, but now I think it requires further experiment, experimentation and I believe that that's the current state of the Market too.

Speaker C: Everywhere I would say not, not only in the development teams. Um, okay, so trying to um, to sum up what we said today, maybe we can turn this into something people can take away and apply. If you have or had to name one thing, it can be structural in the flow or cultural or any kind of thing that a CPO or a CTO should change this quarter or this month so that AI assisted design decisions don't land in the code without any owner. What it would be from you as a designer, what, what we should do?

Speaker A: I would say let's do all this. But it sounds uh, Islam. Uh, it sounds like a bit of work. But uh, uh, it will be quick. Uh, uh, it will be quick. I hope so. I have like three or five questions which uh, sea level should probably ask themselves. Ask themselves. Uh, the first one, apart from that, what, uh, the function of the AI is the obvious one, uh, to start with. But the second one, the one we've mentioned together at the webinar, is who is responsible for that AI feature? One specific person. Because thanks to that, the impact on that feature is trackable. Uh, it works in iterations. Somebody keeps control of uh, what feedback comes back to that feature. Um, and it's easier for the whole company. But you have to think about uh, edge case, maybe not even edge cases because that's actually situations which may happen. So what happens if something goes wrong? Who are we going to contact? Because if we uh, when there's an incident and then we are already thinking about hm, who will be responsible for that? We have a problem. So doing an audit and thinking who will we, who will we call when something bad happens? Who will be notified about such situations in the meantime? Well, let's say that it's quiet, nothing happens. But who will be uh, kept uh, on track? What is happening with that feature? That's another thing. And who is responsible? Uh, who has an authority to overwrite AI decision? It's also a great one. For example in public systems financial products. It's really important because such AI tools can decide whether somebody gets a loan, um, gets algorithms. Nobody decides whether somebody can uh, access uh, a social service or no. So somebody could be also responsible for uh, uh, overriding that decision. Uh, it's popular nowadays that uh, thinking about human in the loop. So that's a part of it. Without that it's hard to say about accountability and governance.

Speaker C: I would say that it's always about the same thing. Who is responsible at the end, um, depends on the culture I would say and the processes in your company as well and the service you provide. Um, I would go even with, uh, the thought that it should be, of course, done before the serious failure. But sometimes the first serious failure could be the best way to learn who should be responsible. Um, or maybe a group of people. Maybe one person. It depends. But sometimes overriding it up front, it's, uh, even worse. Let's wait for a serious failure. I hope not the big one. Um, okay, Kamil, I think that we can leave it on this line because it sums up the whole conversation today. Um, the model did it. It's not any accountability structure as we know. Someone has to own the decision. Um, and of course, the only question is when, um, we settle this person or the way of decision making or decision process.

Speaker A: Yeah.

Speaker C: Thank you very much, Kamel, for your time today again. And to all of you, um, if you want the recording or the notes or the next session information, it all comes together on our substack Product Builders AI native substack. There is a place where we announce upcoming topics and guests too. So, yeah, see you next time.

Speaker B: Thank you.

Speaker A: I wish you all a cold evening today. Uh, uh, in Poland. Uh, and, um, yeah, see you next time, too. Thank you for today.

Speaker B: Thanks for listening to this episode. I invite you to join the Product Builders AI Native community powered by Boulder. And if you would like to join future live sessions, follow us on social media. All the links are in the episode description.