Inverted Podcast #21: AI Regulations & Humans in the Loop

Dexter Aoyemi: Foreign.

Host: Hello to the Inverted podcast today. It's a bit of a special episode because we're missing Jeroen and Tom who have been held hostage by their day to day, uh, corporate jobs. And um, probably Dana is fighting with his coffee machine as it's six o' clock in the morning on his side. Maybe we'll see him. But we have a guest so that's great. Um, Dexter, thanks for being in our podcast and uh, maybe you introduce yourself and um, tell us what you do.

Dexter Aoyemi: Yeah, thanks for having me. Um, so my name is Dexter Aoyemi. I work as an independent AI systems architect, transformation partner, mainly working in regulated environments. Uh, my focus is on something that I uh, call clarity before code, which is um, mapping intent, risk, uh, obligations and behavior before actually building the uh, AI system. And uh, the recent work that uh, got us talking uh, was um, my turning the EU AI act into uh, an executable knowledge graph, uh, so that teams can use it to reason about their obligations, uh, in a structured manner rather than trying to read through a PDF. So it's all for cross team alignment. Well cool.

Host: Yeah. And that's exactly what like um, what caught my, caught my eye because as we talk about inverted problems in here, like where problems are where success is when nothing happens and uh, we have compliance uh, to regulatory frameworks. And like in this case the AI regulatory frameworks are coming up in Europe are certainly one of them. Like if, if you're not compliant, if you're compliant you will have fewer signals. Um, and then I found it really interesting your approach of helping people understand how do regulatory requirements even translate into what people have to do and help them understand. I think one thing you wrote in the LinkedIn post, which will add to the show notes is um, that people don't have time to read the 500 page documents. And if I look at myself, like sometimes I get myself to read long documents but then it's really hard to translate them into, um, translate them to actual work. How does it impact my daily work? How do I make day to day decisions based on those, on something that is so extensive as a regulatory thing. So I wonder how do you see AI regulatory changes affect the way product teams actually build products? And maybe in general, and yeah, in general, how do they build product? How does it affect them on a day to day basis?

Dexter Aoyemi: So I think for AI in particular, uh, it's, it's quite interesting because teams tend to default to wanting to use AI almost to do the work for them, which is, I mean fair. Uh, in Some sense, because AI, um, it allows you to do work a lot more efficiently. But I find that a lot of the ways that people typically implement AI into their workflows is often um, like the most kind of naive way of using it which is to try and um, just offload all of the work. Like you've kind of dump all the information in. And I think for needle in the haystack type work where um, a needle in the haystack being a task where you're trying to find some specific information within a corpus of uh, text um I think AI is pretty good at that and scores I think almost 100% on um, benchmarks for that kind of task. But if you're trying to understand uh the narrative within uh a uh document there's an interesting um, interesting benchmark actually. I think it's fiction, live bench, uh that measures kind of like AI understanding of uh, the narrative, um, of novels. And AI isn't always good uh at that. So really I think the approach to using AI to understand uh, things like regulations, uh needs some thoughtful consideration because otherwise I um, think treating all tasks uh the same with AI is kind of like a foot gun uh for a lot of people. Um, when it comes to AI regulation and building products, I think the ideal kind of outcome is for the classification to start at the beginning of the design uh rather than later. So a lot of conversations around building products, especially products that leverage AI end up being conversations about what model to use, what agent frameworks to implement, uh, rather than starting with like the classification of risk for um, that particular product, um and all the considerations that need to be made around um, like human in the loop is often kind of an afterthought or isn't really considered as part of the system design. But the way that I see it is that Human in the loop, uh to date, and I see it for at least like the next few years. I would bets, but I'm not entirely sure the AI keeps getting better and better at like uh, being able to monitor and um, plan and manage its own work uh recursively. But I think Human in the Loop is a uh, non negotiable and has been a non negotiable. I think it's always been part of the system. Uh, There was a US Department of Defense study from 2012 where they were investigating levels of autonomy within their systems and their conclusion was something like um, all systems uh, have different levels of autonomy. It's a spectrum. But there's always some kind of human control and it's just about how much human control, where the human control gets added that kind of determines um, how humans interact with these autonomous systems. But ultimately there's, there's always some kind of um, human aspect to it. And it was an interesting finding I think or an interesting part of the EU act is, is the fact that they, they have human oversight as a requirement within the regulations. Uh yet there isn't really much kind of like consideration about uh the fact that like it's a requirement, it's something that you potentially have to have to prove or show. Uh and maybe it goes the way of uh, like GDPR and complying with gdpr. I think the most thoughtful implementations of GDPR compliance have very easy to access like data controls like within settings, uh you can export your data, you can request uh, for your data to be deleted. It's built into the technology so it's not like a pain for somebody to have to go and implement later. That then becomes expensive for the company. So uh, the short sightedness can definitely uh, be a uh, hindrance for companies down the line. And I think it uh, potentially matches with the philosophy of inverted uh products where um, it's the kind of thing that you don't really consider uh because it doesn't have the clearest upside but you'll end up paying for it later down the line in a, in quite an asymmetric way because um, a lot of the GDPR requests I remember quite infamously used um to be expensive for a lot of companies. Especially if you don't have it built in. Uh then it can become quite tricky to then retrieve that data and prove that you've actually uh, removed somebody's information um from your system or um. Yeah, or I forget what the other one is now. So it's a removal and uh, also giving a dump to the user. Yeah, retrieval the data that you have. Subject access requests.

Host: Yeah, yeah. That's a critical one. I think that's one that people often forget when it comes to GDPR is the importance of giving people full access to the information. One has you mentioned the human in the loop. I find that a fascinating conversation these days days because when I look at like we're still early phase of even generative AI, we're still not in early phase of AI as a subject overall but like a generative AI, uh automation of workflows. We're still in um, in in in relatively early days and I think people start to, or have been in the beginning starting to think that human loop is the one that just Checks for, for errors like generates, looks at the output and then looks at well is this correct or not? Do we see in regulatory requirements or in your best practices Also the place where the human in the loop is the one that generates the clarity on what the AI actually has to do and verifies that. Because you mentioned clarity before and I think clarity is something we've discussed also in this podcast previously. Like the clearer you are about what the problem is you want to solve, the fewer possible solutions there are. Um, and I think that's where AI tends to be weak is in implying clarity. How do you see clarity being given to AI by human in the loop? And um, yeah that's just my question.

Dexter Aoyemi: I'm a big fan of the topic of human in the loop. And um, and like what it

Host: all

Dexter Aoyemi: the things that intersect with uh, the human in the loop aspect of agentic systems. Especially because I think people often think about the errors and handling the errors in humans like dealing with the mess or mitigating um, risk of AI not being able to solve a thing. But I think the user experience actually um, the crux is within how humans uh interact with the system. Because when you map out how a system works, often you're delegating decisions um, and those decision nodes, um, the hardest decisions I think uh, or the most impactful decisions that you make in system design for uh, that type of system is which decisions you delegate to um, the model and which you delegate to a human and ultimately any outcome uh that the human user and you have internal um, or human in the loops, uh, patterns that inward looking that are internal to the people managing the system. Um, and that's kind of where all the error handling and compliance kind of lives. And then um, the user experience um, adjusts depending on how you implement the human in the loop within an agentic system for both um, the users of the system and the people ah managing the system. Uh, I think you have to very intentionally design those um human in the loop patterns because it impacts uh, who it is that made a decision uh, and the auditability of that um it also. So there are like a bunch of different uh, custom kind of human patterns I've built. Uh, one of them uh, was for a customer support agent, um, and it was uh entirely custom built. And um, the human in the loop within a chatbot experience is uh, just us as the user. So for example I'm using ChatGPT, um, I'm the human in the loop that's prompting the model, um, even if the model is working Potentially one overarching goal. Um, and I'm injecting uh, the context that I think is relevant. Um, I'm giving uh, steering um, to align it with what I believe the goal is. And um, as my understanding um, improves I get more clarity. And then um, that's how you get this uh, kind of collaborative back and forth with the AI. Uh, but those human and loop patterns, they often uh, end up looking. There are so many different ways that we interact with the AI and uh, a lot of the kind of user experience that we're now seeing with developer tooling and also now with Cowork, which extends it out of uh, the world of developers and also into the hands of regular users or at least power users. Because I think you need to be on the max plan, uh at least um, is that the AI will also um, ask you questions and then you'll have to um, give it the context that it's requesting from you. And so um, yeah you have human in loop that enables clarity, enables um, auditability, liability, um, and also just general kind of like user experience and how a product feels. Um, yeah. So there's a lot to human loop and I think it's one of the most important parts of building an effective agency AI system.

Host: Mhm. When we're looking at like human in the loop in the context of meeting regulatory requirements like inward facing, like if I build a product or the people who are listening to this and want to build product, maybe either a new one or build AI on top of their existing product. Um, are there any checkboxes they need to think about? Are they process that they need to establish in order to implement the human in the loop in the right way? What are the responsibilities in regards to when that human has to be involved? How fast do they need to respond to potential issues? Um, to look at not just to improve quality over the long time, but also to look at potential incidents that could have been caused um, by AI agents. Um, are there best practices that have already been established there?

Dexter Aoyemi: I find it hard to point to best practices given uh, how new um all of this uh is and how much is changing. But I've definitely uh, tackled it in a number of different ways. Um, and I think this keeps evolving. Uh, so on the. So you're more interested in like the inward human in the loop for um, like compliance?

Host: Well I think in this question, yes, if you look at the inwards part, but we'll also be interesting then in the second part on the outwards part and maybe around um, reporting responsibilities but if we're looking at the, in one part it's probably the ones that people might have to think about first while they're building the product.

Dexter Aoyemi: Yeah, yeah, I think you definitely want to have the, the insights of what it is that the, the models are generating. Like observability has to be built in from the beginning in order for you to be able to track how the system is responding uh to usage. And I'm not sure if it comes second uh to other human in the loop patterns. I think it's all complementary just because the system itself um, might need um, the, the human using the, the AI to. Well it will need the human using the AI to, to drive like the, the outcome. Um, but for, for somebody managing an AI system trying to think of useful examples because often if you deploy a system you're monitoring logs or you have it hooked up to uh, some provider of um, a platform that gives you traceability of what's happening within the system. But I think how you end up designing a workflow for people to improve the system um, will also depend a lot on the internal kind of uh, set of tooling and uh, the objective of the system itself. Because ultimately all of these systems are designed to drive some particular kind of outcome, um, and behavioral change in some downstream system. And I think that's going to be the most informative um thing once you have that clearly defined. Like what behavior is it that we're trying to change? Um, what does failure look like? Um, what are the failure modes? And it's a lesson, it's an annoying answer but it kind of is one of those, it depends kind of situations I think.

Host: Yeah, yeah and that absolutely makes sense. Um, currently with a lot, I mean a lot of the conversations around problems with generative AI. I mean last weeks or so a lot of people have been talking about uh, image generation like how uh, like X as a platform doesn't take responsibility for the outcomes of the image generation. Um, do you see human in the loop or um, do you see there opportunities or maybe missed opportunities that um, companies that offer services like this have to take in order to be clearer about what type of outcome they want but also maybe what type of outcome they don't want to have? Um, because I think that's where obviously the discourse is focusing on um.

Dexter Aoyemi: Yeah, so you wouldn't want to have human in the loop on every image uh generation because then uh, you'd have to have a whole bunch of people just constantly monitoring the images and I guess it could work but it would defeat the purpose in some sense. Uh, but having oversight over the types of images that are generated. I think it all just um, points to conventional software at that point because you need good uh, monitoring. You need uh, to um, have a good view of the, of the uh, images that are being generated and uh, the way that people are using the system in general. I think what AI enables is uh, for the process of building like bespoke interfaces for looking at what is being done within a system. Uh, it just supercharges that and makes it like really really uh quick and easy to build. Very bespoke, um, yeah, experiences for uh, internal staff to be able to keep track of things that uh, the company is responsible for.

Host: Maybe going back a bit more to the regulatory part like the AI regulatory part that you've been looking in and when you work with customers and clients where do you see the biggest challenges in people understanding the, the details of those regulatory frameworks? Um,

Dexter Aoyemi: I think there hasn't been much, There hasn't been much focus I think from early uh stage startups on regulations and I don't think that's ever been the. Sorry. I think that's always the case with startups is uh, regulations are kind of like either an enabler for the business itself uh or a secondary consideration uh, that is going to come to bite them at some point when they need to uh, be compliant to prove to the people they're selling to. And then uh, on the other extreme, uh you have these large enterprises who have uh, what one of my clients uh referred to as these AI centers of excellence. Uh but they're effectively these committees, um, I've had these AI committees um turn up um in the middle of some of the projects that I've worked on before. And they are often driven by uh, this need, this very legitimate need within enterprises to be compliant and to also have a well understood um, I guess security um uh design that applies across uh AI systems. Um, and they always kind um of converge on very similar or overlapping kind of designs where you have guard railing on the, on the inputs and the outputs. Um, you have this um, interfacing with whatever uh if they're. Actually I can. I guess it depends, sorry. It's the protection of people's uh personal data, um, it's uh, authentication, access control, uh guardrailing on the inputs and the outputs. Um so there is a need within uh large companies to be compliant um and to be secure. Um so you end up with this blend of needs that come from the compliance teams and the and the security teams uh that come into whichever department within a company is um, who takes responsibility for these things. And that's often where it, where it appears. But unfortunately um, those, those conversations don't always happen at the right time and in my experience have always happened uh later than I think um, you'd ideally want them to happen um because the, the implementation has usually started well before anybody from compliance uh or security has had some kind of input and often the engineering and product teams don't have a clear understanding of uh, the company's compliance needs. Yeah.

Host: So how, how can product teams, engineering teams engage earlier on? Like what are the best ways to do that? Music like when is too early or what are the ways to engage in uh the right stage with these teams, with, with the compliance teams to get the most out of them and as you say like clarity before code. How do you get the clarity on something that you haven't built yet?

Dexter Aoyemi: Uh, I think that is a very good question and I am not entirely sure that I know the answer. I think it's something like uh, we have shift left or this concept of shift left in software development where um, often we're talking about security shifting left. And I think maybe if teams truly embraced uh a shift left culture then perhaps that converges on taking all of the compliance considerations in early as well. Because often we see it uh, in software development at least I think there's a tendency to see compliance as documentation uh and a box checking exercise. And this is part of what motivates me to turn the EU AI act into a knowledge graph is because um, actually it's better to see these uh regulations um as ontologies or as uh process maps or something that uh, can be described in terms of um, actors, triggers, obligations uh rather than having to try and pass through some text. Um so it was the question, um, I got distracted thinking about the EU AI Act.

Host: I'm just curious what can we give people on the way that are in that situation where they're starting to build something but they don't know yet enough about what they're building to have a clear questionnaire. But they want to involve these competence um lacking a word like um these teams like the compliance team to really have them involved early on. I mean we see them in product development, we see them in so many ah areas like involve product designers early in order to build better products, involve um sales earlier in order to build products that can actually sell. Like so why should it be, it shouldn't be different with compliance but often in Large companies compliance teams even in the process they see themselves as being the guard keeper from, from flipping the launch, the launch button. Uh how do you get these compliance team away from being the ones that decide if something can be launched or not to be uh, an integral partner in early stage of the process of building something new.

Dexter Aoyemi: I think there's like a psychological challenge or something to overcome there because thinking um about the EUCT in particular they have specific clauses or phrases around human oversight and the need for human oversight uh within AI systems as an example. And people don't think about the uh, human oversight but it's actually written within the regulations that you need to have human oversight in some way, shape or form. And that maps very cleanly onto this idea of a human in the loop uh being built into the product. And I think it's a constraint but it's not uh. It's not like this limitation on how good the product can be. If anything it just gives you the direction uh for you to stay uh compliant and to build a good product. Uh because these regulations are designed in a way that uh actually protects people um and protects uh systems and organizations and society. And so it's uh, kind of like a broken thinking to see uh. Them as a pain. And there is something in the way that I think um. That people interact within organizations or have a tendency often to interact within organizations that makes it feel like um. I've definitely felt this or gone this impression myself and I don't know how much of it is just like a biased thought that I had in my mind but um. Of kind of like this person with a clipboard um Having uh like a sense of um. Authority to use and wield um rather than uh. Uh. But I don't necessarily blame uh the person. I kind of think that like the way that incentives um are oriented within organizations creates these environments that um. Aren't necessarily conducive to building uh good thoughtful products. Um and I think that these regulations, you know they're. They're there for a reason, they serve a good purpose and the, the more accessible um. I guess we can make them for people. Uh hopefully um, that that gets to uh. Inform people how to. How to make good products. Um. I think that uh, are actually yeah beneficial not just to uh, uh society as a whole but uh, to them as an organization.

Host: I really like that approach of looking at that from. I mean the problem is certainly multi level, multi layered. I mean I remember in my time spending time in large organizations and you build and build and like the security team in the end needs to give their thumbs up or thumbs down if you can publish an app on the App Store or, or if you can launch uh, the website, um, or if you can launch an update. And one element that I also found was the staffing often doesn't allow them to be engaged earlier in an earlier stage, like security or also compliance. Like they're often so thinly staffed that they can only join product. They can only focus on um, fixing the fires, like preventing bad things from happening. And usually that's when you launch. Like nothing bad usually happens or the risk of something bad happens, happening before you launch is usually smaller, is significantly smaller than after you launch. So um, I think that's where also people are being forced into just focus on where they have the most immediate impact. And um, that's one of the things that fascinated me about that work you brought about with creating a graph based on the AI regulations that helps more people interact with it and get knowledge out of it. And that might also reduce the workload, um, for the specialists and being able to deliver value where it's most effective rather than just be always the ones that has to prevent things from launching. Um. Do you see that with more and more regulations, companies that you work with are increasing the staff, are increasing the capacity, um, of these uh, specialists. Or do they just. Are they still massively underfunded and understaffed and have to chase the most urgent fires all the time?

Dexter Aoyemi: Definitely feels like the latter. Uh, I don't feel like I see an increase in the number of people working within compliance. It feels, feels like companies um, hire them when they feel that they need to. Um, sometimes that can be late. But I don't, I don't think people necessarily, um, to your earlier point, like, I don't think they necessarily, um, avoiding responsibility or trying to circumvent compliance or anything malicious. I think it really is just a matter of like, what is the, uh, the priority for a lot of companies. And it really is often just to survive, uh, to um, the next fundraising, um, or whatever the next milestone is. Um, yeah, it is hard, it is hard to put a value on uh, compliance. But I am hopeful that, um, because AI enables the development of uh, these tools and interacting with information, um, in the way that it does that it becomes easier to reason about these things. And for whatever it's worth, um, maybe the conversation becomes uh, less frictioned, uh, when the work to understand um, how these regulations affect, uh, the systems that we design. Um, uh, yeah, understanding that impact, if it Becomes easier than I hope, or at least I think the best I can do is hope that AI is actually going to make that easier for teams.

Host: Interesting. So maybe to close down. What are the things, what are things you recommend? Product teams who want to understand better how they can learn more about compliance. Especially now thinking about the AI compliance, um, um, requirements that are coming up, like where do they start, what should they look at and also um, what are the, maybe the most important points they need to understand about AI regulation, um, here in Europe.

Dexter Aoyemi: Yeah. So I think where to start is which classification the system that you're working on, uh, falls under. Uh, so is it high, um, risk, limited risk, um, and uh, for European companies, I mean it affects everyone. So I think it's whether or not you're in Europe, I think this regulation and these types of regulations are going to set the uh, the trend, um, or become the default for uh, for governance, similar uh, to how GDPR did. And so by being compliant uh, today with the EU AI Act, I think you're potentially, um, saving yourself a lot of hassle down the line. Um, similar to if you were to design your system to have good data controls, um, as we were talking about earlier in the conversation.

Host: Cool. Well thanks. So I hear like taking away, uh, I mean for myself, but also probably for people who listen, understand where you stand with those within those regulations, like assessment, like risk classifications, um, we talk about that often. Security in privacy. There's a lot of. I think you always need to know what could go wrong and where do you stand in that. And then also maybe a bit of a reassurance or some peace of mind that what you're doing now for the EU AI act is not, is unlikely to be throwaway work because following regulatory requirements or bodies will, uh, go and use that as a reference. I think that's something that we certainly have learned from uh, gdpr. Cool. Well, thank you so much for being here. Sorry. Yeah, go ahead.

Dexter Aoyemi: I should caveat that I'm not a lawyer. So, um, all of this is, none of this is legal advice.

Host: I think that's always a good. Yeah, I don't know how many conversations I start with I'm not a lawyer. Um, yeah, I mean obviously this is not legal advice. This is just our view on how do we think, um, people will have to navigate. And I think, um, more and more, I mean we see that more and more companies that are building things that used to be, quote, unquote, non regulated, just that doesn't exist anymore. Like nowadays almost everything you build, um, you're going to be regulated and maybe we can do a follow up, uh, as the Cyber Resilience act is coming up. I'm sure you're thinking about that in some areas that is going to affect even more people. Like not even just companies that make AI, but, um, I mean, even a USB stick will somehow be part of the Cyber Resilience Act. And so this mindset of building products that are not just cheap or not just good, whatever good means, but also need to be, need to prevent harm to society, to their customers and employees, I think that mindset has to establish much more with all types of organizations. While in the past it was banks and insurances and health companies that were forced to think about that. Great. Um, well, thank you so much, Dexter, for being here. Thank you also for, um, just handling the unusual situation. It's just the two of us this time. Um, but yeah, well, thank, um, you so much and hopefully happy to have you. Another episode where we talk about some other topics that are dear to your heart and you're working on. And um, everybody else, Visit us on invertedPM.com um, where you see all the episodes that have been published so far. Um, we also sometimes write blog posts or translate the conversations we have in these posts, um, into posts into written form for people who. That's easier. Thank you so much. And, um, well, yeah, now I'll close the podcast and um, see you next time.