KYA Won't Always Protect You. The Real Risk Is the Swarm!

Speaker A: Foreign. Risks around AI have been a hot topic for more than one year, probably the last two years. And there's a fairly standard list of risks that we've all been discussing. We've talked about hallucinations. Once, uh, generative AI model started taking actions and not just generating text, then we started discussing about authentication and authorization risks, about cyber risks. Many regulators, especially in financial services, have started addressing the vendor concentration risk. And now with the latest restriction on the models for anthropic failure, Fable 5 and Mythos 5 coming from the US government, we have realized that there's a new risk category that looks like sovereign risk with uh, a geopolitical flavor. My guest today has spent at least the last two years looking deeply into agentic finance and the large scale problems that, that are not discussed enough. Amna, welcome. Let me introduce you and thank you for joining us today.

Speaker B: Hi Effy, great to be here.

Speaker A: Thank you for having me for my audience. Amna Usman Chaudhary is a financial economist and the Frontier technology strategist. You teach from what I understand at the graduate level on frontier finance and emerging technologists. You advise your own boards across fintech. Uh, you are very much focused on responsible innovation. You've spoken at the United nations on algorithmic accountability. You are an expert at uh, UNESCO for women, for ethical AI. So that says a lot. And thank you really for joining us. We've been trying to get together for a while and what a moment, I must say, as developments are accelerating. When you talk about autonomous finance, I understand that you're referring to multi agent workflows with no human in the loop. That is really where your research and your strategic focus is. Let's start with why isn't it too early? What are you seeing in the market that leads you to say that it's not too early to address this issue?

Speaker B: Yeah, thank you Effy for the kind introduction and glad the stars finally aligned at a very interesting point in time when there's a lot happening on this autonomous finance front. So just stepping back. Autonomous finance, according to my formal definition is when, as you mentioned, multi agents, multiple AI agents, work together using swarm intelligence. And this is done over digital infrastructure. And it's also with inclusion and accountability as core requirements. So there's three layers to it. One is the multiple AI agents. Then the digital infrastructure there can be blockchain and beyond, not just restricted to blockchain. And then the importance of accountability and inclusion from the very start. So when you mention that no human in the loop, does that mean that we don't require any humans at all. That is not necessarily what autonomous finance is. So a human may set intent, it may initiate or design the permissions, the guardrails and all of that. So that is covered. So we've had AI in finance for decades. And what's changing now with autonomous finance in particular, is the agentic AI front, which is exponentially expanding over the past two, three years, particularly the past two years. So with autonomous AI, finance is not just human initiated, but it is increasingly AI initiated. So AI agents are not just being prompted by humans to do something, but they're autonomously taking decisions, they're accessing data points, they're hiring other AI agents and even hiring humans through platforms like Rent a Human. So we're at a very important conjunction in time, and it's very important that we address these points. So with finance moving to machine speed, can we have a human in the loop at every step of the way? That is not possible. And that really makes the whole point of having these machine economies and autonomous economies, it's not practical. So instead of having human in the.

Speaker A: Yeah. Uh, recently I think the FSB issued a report, if, if I'm not wrong, exactly pointing to what you said, that this, this can't scale. You can't have a human at every decision point checking and approving. It's just not scalable for the agentic economy. The question is where, what is the role? Where and how do you design the automation of the decision making to make it auditable, transparent, and in alignment with either the ethics or the values of the company?

Speaker B: Right, yeah, that's an important point. And I totally agree with the, uh, statement that it's just not scalable now with the humans in every point of the loop. So what I discuss in my framework, we have one of the pillars as human override. So that means at the critical conjunctions where there is a very important decision being made, or when you get a notification, that intent drive, if any agent has been given these permissions and this intent, but it has drifted from them, then you get a human notification and the human intervenes at critical nodes and intersections. Those are the points where human oversight is necessary, not necessarily at every point of the loop. Yes. So as your point with. Is it too early? I think in the past week alone we have seen that there's been so much momentum on this front of agent tech finance, agentic payments, which are important pillars of the autonomous finance system. Last week we had MasterCard launch the AP4M. Um, agent payment for machines and coinbase also last week had their introduced their AI agents. Visa has been working on it.

Speaker A: AWS is introducing Robinhood, launched an agentic credit card with Visa. So many examples. Right. Stripe has it the wallet. What's it called? I think it's called Link. Uh, uh, yeah, um, I think so.

Speaker B: I think at this point it's easier to find the companies or the players who are not working on this instead of the ones who are actually on this front because there's so much momentum happening at this front front. So that's really exciting. Yes.

Speaker A: I would say that like with every innovation in the early stages, there's a lot of scattered pieces that are being built and eventually there's going to be integrations and rebundling to make whole processes work, whether they are you mentioned, um, on blockchain rails or on um, traditional rails. But a lot of discussion has been around KYC not being suitable anymore for the agentic world. Again, whether it's the delegate economy or it's the machine to machine economy. And the solution that everybody's talking about is kya, know your agent. And you have taken this a step further and you claim that KYA is good, it's necessary but not sufficient. So I want to stay at this point because it's a very unique perspective and not discussed and understood. So explain to us why KYA is not important and why you've introduced this new framework called Swarm.

Speaker B: Know your Swarm. Yes, yes. So great point. So one thing I want to. I like your point about the delegation economy versus the machine to machine economy. When previously humans used to delegate machines and now machines are delegating between themselves so becoming increasingly more autonomous. And for both of these I believe KYC is still important because kyc, it's part of the know your customer human aspect of it. You have different various components. Even in the machine delegation economy where humans are there, and even in the machine to machine economy with guardrails pillars such as human override. You do need kyc. That saying KYC is still there and important, KYB is still there, know your business, etc. And kya, the emerging know your agent frameworks are also very important because you cannot govern what you don't know. So I'm really happy that the focus of the conversation has shifted to KYA frameworks with IMF itself recently in April publishing about emerging KY frameworks and how we need to look into that more. So my, my introduction is a, uh, government governance framework called Know youw Swarm. And what Know youw Swarm does is that it moves Moves a layer above, know your agent. So it says that in the future it will not just be one AI agent working for in the financial systems, it will be multiple AI agents interacting between each other, which you mentioned is the machine to machine economy. So what happens when these multiple AI agents work together and then they collectively they're, each of them is, is KYA compliant. Each of them has passed the KY stage. But multiply, they produce an outcome that was unintended or not intended. So a bad outcome for example, or an outcome that was just unintended. So who is responsible for that outcome? And that is why KYS is important, because these are all at the very basic level, they are software. So it's not like when these multiple softwares interact with each other, they can introduce unforeseen circumstances. So I think the best way to explain this is via an example. So um, I think one good example would be like if you have a, uh, credit decision making pipeline and then you have one AI agent who checks the applicant's details and verification id, et cetera, then the other AI agent checks the data of the applicant, what is the credit history, payment income, all of this. Then the other third AI agent checks the affordability, One looks and then calculates how much credit should be given or the loan should be given, and one checks the compliance and regulations. So these are just different KY compliant agents doing what they have been authorized to do and working uh, according to their permissions. So the interesting thing with biases is it is not necessarily protected characteristics that you know that are there. The protected, uh, characteristics as a variable can be introduced into the swarm through proxies. So going back, suppose one of the agents finds that the applicant has an employment history gap, work gap, right? And that can be for various reasons. For example, it can be that the person was caretaking. Or maybe another example is the uh, credit history doesn't date back as long as it should have. But that could be because they were excluded from the financial system historically. So there could be very valid reasons. And this AI agent knows that. That's why this is just like some information in that AI agent. But then when the pricing agent comes and sees this, then it finds out, oh, what is this? And that it takes what is noise in the agent one as input into agent two. And then the risk agent comes and then it amplifies that. And so at the end the outcome that is intended might be that someone who is very deserving of credit is totally denied just because of the collective outcome of the swarm. So this is the aspect that we need to look into that is why it's very important to not just check each KYA agents, individual AI agents, but also the collective swarm.

Speaker A: So I hope that that is an interesting aspect because it seems to me that it connects deeper to explainability and understanding how these, the flow of these decisions may create uh, gaps that are amplified and result in a suboptimal decision. And isn't that solved by feedback and learning loops? What is the KYS really measuring? How does it work in implementation? I understand that you have designed a framework with five pillars that really compose this know your swarm. So one can evaluate at scale when you have multi agents, any process and see where it is. Is there a benchmark? How does this actually work?

Speaker B: Yeah, great question, Effie. So basically, know your swarm KYS framework has five pillars as you correctly identified and it starts with kya, because you can't govern what you do not know. So agent identity is very important. Whether you call it KYA or whatever the emerging agent identity frameworks come into, like what is this agent, what is the name, purpose, etc. All of this. So then you have authorized intent. So what is this agent intended to do? And intent drift you have to monitor, which is if the agent is intended to do this behavior, how far is its behavior drifting from that behavior? And after a certain amount then the risk is triggered and then the human override aspect comes. Human override is one pillar as well, which is basically once they, there are certain important critical nodes that the human has to review and can these decisions be modified or reversed, et cetera. So there's human override as one of the pillars. Then you also have interaction mapping, which is basically when swarms, which is basically what swarm is like when the different AI agents work together and then in swarm governance and then you have accountable redress as well. So when a swarm produces an outcome that is discriminatory or unfavorable or customer protests against that, is there a viable mechanism and a redress, um, option that this institution, this institution, this is where the accountable redress happens. So it is these five pillars. And I think an important thing to overline when we talk about swarms is that obviously you have these five pillars and you have, we are talking about multi agent frameworks. So every swarm is multi agent. So taking the credit example, right, Every swarm is multi agent in the credit workflow, but not necessarily. If you have multiple agents working together, that is a swarm. So this is, yeah, this is a very interesting thing. So it's not like all multi Agents working together are swarms, but swarms are multi agents. So what happens here is that in multi agents it can be if it's a very strict centralized workflow and accountability is very directed and linear and centrally controlled. So that is just multi agent orchestration. So it's. The swarm happens when these. It's not very centrally controlled and when there's outcomes that are not accountable and pinpointable to.

Speaker A: Does it have to do Amna, um, with the conditionality of decisions if you have five agents working together, is that the difference? I still don't. I get your point that a multi agent workflow it can be very controlled

Speaker B: if it's very controlled, very linear, centrally governed. So if uh, it's all very controlled and centrally governed, it is just a multi agent orchestration workflow. But the swarm happens when you see unforeseen outlook of the circumstances. And every. A swarm has to have multiple agents, but not every multiple agents.

Speaker A: That is clear that we're talking about this issue in a multi agent framework and not that's necessary not with one agent. Is there. Do you have an example that shows this difference of a situation of multi agent framework workflows, centralized workflow on my

Speaker B: computer, but on my computer between a certain program very controlled, very linear that this agent can only do this agent can only do this very controlled and very linear, very permissioned centralized workflow. Yeah, but in the future obviously it will be in the financial ecosystem. It will not be very centralized. There will be different MasterCard AI agent working with like all of these different parties and swarms. So that is where.

Speaker A: So you're talking also about swarms that are not confined within an organization, but they can be swarms across networks or across.

Speaker B: Yes, across networks also. And also within organizations as well.

Speaker A: Yeah, yeah. The big. There's a bigger risk especially in ecosystems where you go across organizations. Is that correct to say that?

Speaker B: Oh yes. The risk amplifies when you go across organizations because you notice that a lot of organizations have different architectures and flow and as well. So then it becomes increasingly with different architectures increasingly difficult to govern the swarm. That is why it's in regulated finance financial services. Swarms is still an emerging framework. So it doesn't exist in regulated finance or tradfi yet. That is why the governance question is very important to ask now because the best governance questions are asked before the output or the outcome happens. Once the outcome happens then it becomes a very expensive, difficult governance situation to map it into the rails that are already built. So that is an important point. As well.

Speaker A: Yeah, but what you're really saying again is that know your agent is necessary, but there are still risks because it's not a sufficient condition when you have is it the risk of delegation or it goes beyond the risk of delegation? Uh, in a multi agent framework.

Speaker B: Yeah. So delegation, speaking of delegation failure is something I address in my research as well, particularly with relations to know your swarm framework. And the uh, delegation failure happens when you have multiple AI agents executing instructions, carrying out the workflow and yet when they're collectively they reach an outcome. There's no single person or does the clarity of the outcome, the collective outcome, who is responsible for that becomes unclear or it's not very clear. So that is a delegation failure. So if in the credit decisioning um, example we use something went wrong, who is to blame? Is it the AI, is it someone else? Is it the institution? So it has to be very clear where the accountability lies, which is basically what I call the swarm accountability gap. And once that is the framework missing issue and then the failure that occurs is the delegation failure.

Speaker A: Yeah, yeah. What we're seeing now, whether it's a uh, multi agent or an ah, agentic workflow is we have some regulations that like the UAI act that this classifying certain areas like credit decisioning as high risk. And then you see market players really designing agentic workflows and saying who takes the responsibility? It's market driven and there's no consensus, there's no standard out there as to who is accountable when things go wrong. As if things will not go wrong. Right.

Speaker B: Go wrong.

Speaker A: Yeah.

Speaker B: EU AI act itself mentions that with advanced AI systems the systemic risk changes and then it really needs to be taken into account. So because the technology is advancing so fast that it's moving faster than regulation can keep up. And that's usually what happens with technology. But I think with the way AI is moving, it's almost exponential.

Speaker A: You do a lot of work around financial inclusion, around um, algorithmic bias, around gender bias. And all these are overlapping and can be um, propagated through these frontier technologies. And I'm thinking about financing whether it's on a personal level credit or at business and country levels. We have huge financing gaps and biases that already exist in the system. And I'd like to hear your thoughts about the risks from autonomous finance in terms of amplifying these risks or covering them up in making things worse. What are your thoughts here?

Speaker B: Yeah, I think bias is almost something people take as cliche when it comes to AI that I not uh, like When I meet people, they're like, oh, bias is an afterthought. It's just something people bring up. But you and I, people who are working in this space, know how much these biases can amplify existing systems. One thing I often like to quote about and mention is that uh, UNDP did a study, the Social Gender Norms Index and they found that 90% of men and women hold some sort of bias against women. Then they repeated the study after 10 years and found that 90% of men and women still hold some sort of bias against women. Bias exists on a very societal level and it is then translated into the AI and digital financial systems. And then we have to be very careful that as these autonomous financial system is taking place is being built, that these do not translate into that. So for example, these are very real as well. In the, if you look at the African continent, we found that there was a study done, which I quoted in my research as well, that around 37% underfunding penalty exists, exists across 10 common crediting algorithms, um, for women funding. And there's a 42 billion financing gap for women led businesses as well. So these are very real and I'm also passionate about it because I've seen how it has through various parts of my work and how it translates into the outcome that is not intended potentially. So this also exists in financially historical data sets. Historically finance, digital finance has these limitations because maybe the data is not complete from the global south, maybe there are certain missing, so there's weaker interpretation of that. And we need to make sure that this does not get translated. And that's where the swarm accountability gaps gets even more important. Because traditionally we've, if we are checking for bias, we look at a certain point whether it's in uh, traditional finance or digital finance or AI. You can start from for example in AI, from a LLM or uh, AI model, a data set, et cetera. But when you have these underlying biases, even though an agent may be KYA compliant, but as we discussed in the credit financing pipeline example, when these interact together in a swarm structure, that bias can get amplified and it is very difficult to capture. So we need to make sure that there is a swarm level bias auditing, for example on gender. So each AI agent is not just checked on its whether it has a gender bias, taking gender, singling out gender, whether there is a bias in just one agent, but whether the collective, when these collective AI uh agents work together, does that translate into an unforeseen outcome? So swarm level bias testing as A collective as well. So this is one of the starting points. And again, because discrimination is often introduced, not directly, but through proxies, so it's very important that we take these things into account. And yeah, I think the more these, we have these conversations, the more awareness spreads.

Speaker A: And I find it positively ironic that these frontiers technologies are pushing us to look into issues that exist everywhere and we've maybe become numb to them. And now we are pushed to look at into them because of our fear of them getting amplified and getting results that are at scale, um, very discriminatory or not desirable outcomes. And it's pushing us to look at these issues, which is a positive thing. The more we discuss about it, the more we have ways to measure it, then it's a great thing because it will make us improve. Right. Uh, if we can measure it, then we can track it, then we can learn and adapt from it. And I would say that in closing your line of thinking that we need to, like everybody says, shift from KYC to kya, but then also shift to kys, from agents that looked at in a silo to swarms. And what are the risks there that we need to identify?

Speaker B: Right, yeah, no, a great point. I was going like, I'm a huge believer in the power of emerging technologies to create positive change. So like you mentioned, there is a positive aspect that we are aware of these biases and that is why most of the biases that I mentioned in the UNDP study, they're subconscious. The more conversations we have, the more awareness and the more we can ensure these don't get translated. So which is why it's so important. And even the IMF itself says that agent take autonomous finance can actually have a positive outcome, specifically with regards to financial inclusion. So it's. There's a lot of positive outcome as well. I'm not like there should be, we're going the path of doom. But there's a lot. The 1.4 billion unbanked can be brought into the financial system. There can be new ways to enhance inclusion, remove biases and all of these different things, which is why it's so important to address these governance questions, as you mentioned. So yes, going back to your question, I believe that KYC remains important. Keep kyc kyb. KYA is also important, but it is not enough when it comes to these AI agents. You need a layer above kya which maps and governs the individual AI agents interacting on a swarm level.

Speaker A: Great.

Speaker B: Yes.

Speaker A: We'll be sharing with our audience your paper for those that want to go deeper and understand this concept in more detail. And in closing, I want to thank you and ask you maybe a more personal question. What are you reading now? What are you excited whether it's within finance or not? Yeah.

Speaker B: So I'm usually reading two books, two or three books at a time. So one interesting book, whether finance or not is the Venture Law. So I'm reading that it's about venture capital. That's quite interesting. I recently started and another very interesting book that I picked up is Dopamine Nation and it's, it's mind boggling. I picked it up out of interest and when you start reading it, it's just shocking how technology, we're talking about autonomous finance, but the irony is that it can have very negative technology as a whole. Doom scrolling all of that. It can have a very negative connotation to it as well. So we have to make sure that we balance the good with the bad. So those are very interesting books, I definitely recommend them.

Speaker A: Uh, interesting books. I think that we have to keep. We have to make a huge effort these days to keep a balance between our digital life, which consumes our life, and being real and grounded, being in nature, being in human relationships and not forgetting that we are not only machines, we are much more than that because we forget about it by spending too much time on screens and with technology.