How AI Agents are transforming internal audit

Hello, I'm Richard Chambers, the Senior Advisor of Risk and Audit for Audit Board, and welcome to another in my continuing podcast series, Speaking of Risk and Audit, your go-to resource for actionable ideas, expert perspectives, and the real-world experiences that help you strengthen your organization's risk posture, and add tangible value across the three lines. Today, I have a special guest, another Richard, Richard Penfill, who is a data science manager in AIOps with PayPal.

Welcome to the show, Richard. Thank you for having me. Looking forward to it. Richard, you have a deep background in data science, and you know, that's not something I say very often because not a lot of people have a deep background in data science.

It's a relatively new field or at least a new field around internal audit. And I guess I would note that you've been with PayPal for almost four years. And before that, you were at Wells Fargo. How did you get into the data science field?

And how vital do you think that data analytics are to internal audit? When I was eight, I decided I wanted to be an internal auditor and thought data science was a good way to pursue that dream. When you were eight? That's a, that was a joke.

Oh, okay. I had a doctor. All right. See, I, you know, I'm an internal auditor.

I have no sense of humor, so I, I, I get it. Okay. You'd be the first one who ever talked about wanting to be an internal auditor at that age. That's for sure.

Were you, were you 10? 10. No, no. I was thinking, what about being a fireman or a policeman or all those things most eight-year-olds wanted to be?

But anyway, so how did you get into it? I have a background in math and actuarial science and computer science, so I just lended itself well to data science. And I was fortunate enough to experience all three lines of defense as a data scientist at Wells Fargo. And as I progressed from first to second and second to third, each time was like a new awakening, being able to see a different perspective.

And I believe you've discussed it before, but having that view from the crow's nest, that's the most exciting for me. And I'll continue to argue that the third line is where it's at. And I think data analytics plays a crucial part of any modern internal audit department. That said, I believe the skills to do data analytics is becoming more accessible via AI.

So the mindset, the thought to ask how can data help here is becoming equally or more important than the code itself, which could be both scary or exciting. You've got a real passion for AI, too. And my sense is you were dabbling in AI before everybody else was. Is that fair to say?

Yeah. Remember when GPT-2 came out prior to ChatGPT. I tried using it for a simple summarization of a paragraph, and it returned a jumble of words. But you could see, like a toddler learning to speak, Like, that it had potential.

And a few models later, when ChatGPT did finally arrive, some of that potential had been realized. For someone like me, who is not the best writer, or it was already a game changer at that point. At the same time, there was major issues. Hallucinations being a prime example.

Nevertheless, I think the last few years have been the most exciting time in my career and only anticipate changes and advancements to accelerate. A lot of companies, and we've done some recent research on this, but a lot of companies get stuck at the prompt library or co-pilot phase, right? What limitations did you face and how did you push beyond that? Yeah, so one of the first things we did was build this gamified prompt library in which anyone could contribute to or run prompts directly from the library.

It helped get everyone's feet wet in a fun and collaborative way. The biggest limitation with prompt libraries is that the prompts themselves are useless without the correct context. And getting that correct context usually means copying and pasting data or documents, agents what I call paying the bridge tax. But copying and pasting doesn't scale.

It's like having a Ferrari but asking the driver to push it to turn it on. So the next logical step was incorporating AI agents. Would you call it paying the bridge tax? I mean, it sounds clever, but what does it mean?

The bridge tax is just the act of having to move data from one system to another, usually done manually via copying and pasting. So it defeats a lot of the efficiency you get out of using AI, right? Let me ask you, why do you think internal auditors, you've been around them for a while, why do you think they're sort of nervous or skeptical about using AI? I think there's a couple reasons.

The first being that... People are scared for their jobs, which to some extent is a valid concern, but at the same time, not a reason to not pursue AI because whether you like it or not, AI is here and you have to learn to adapt. The second reason is the risks that it has embedded in it, including the fact that, But if you're using AI, you don't want to become completely dependent on it and have it make mistakes that you would have caught previously. But as we all know, humans make mistakes too.

So just becoming too dependent on it is another concern that I've seen a lot of managers have. We did a survey in the fall, and it was part of our annual Focus on the Future report. And only 25% of internal audit departments were actively using AI. 50% were in that piloting or experimenting phase.

And then another 25% said they're not even planning to try to use it in the next year. And do those numbers surprise you, or is that kind of along the lines of what we've been talking about? That maybe we're just a profession that's a little bit slow to adapt to new technologies? Some of those numbers definitely surprise me because I think the question isn't whether AI will change audit, it's whether audit will lead that change or chase it.

And I'm surprised that there's still people that haven't realized that. Well, that's a profound statement. Will we lead the change or chase it? Great clip there.

How are you exploring agentic use and what exactly is swarm auditing? What really intrigued me when I first invited you here is that I know you're working with your colleagues there in PayPal to experiment with something or you guys are using something called swarm auditing. Sounds like a bunch of bees coming in. So tell me what it is, and how is it related to a Gentic audit?

I mean, to a Gentic use. I'm sorry, a Gentic use. So to clarify what AI agents are, because that term is just being thrown around loosely these days. Yeah.

Prior to agents, we had assistants, co-pilots, chatbots, whatever you want to call them. But these, you fed data to them and ask it a question, and then it would give you a response. The biggest difference between agents and those are that agents now have the ability to act or do things within systems. One of the earliest and most commonly used agents is the research agent, which has the ability to use the internet and search for live data and go and retrieve that live data.

But beyond that, agents can now connect to a multitude of different systems, whether that be your ERP system, whether that be your data warehouse. So that's something that we've started to explore and create some solutions for is having those agents connect to those data sources and pulling our data to help streamline our data analytics. But to answer your question about swarm auditing, and we can coin the term here, we see it as a multi-tiered framework where tier one is orchestrating teams of AI agents, what are referred to as swarms.

And using techniques from software development and adapting that for audit to have these swarms complete audit tasks. Tier two is how do you integrate those swarms, how do you integrate those teams of AI agents with your team of auditors in a safe and reliable way. And then lastly, tier three is the collaboration across audit departments, moving beyond benchmarking to sharing best practices with this nascent technology. So those three layers together are what we refer to as swarm auditing.

So what would you use it for? I mean, do you have use cases where you say, oh, this is really something great to use swarm auditing if you've got, I don't know, let's say it's an engagement where you're going to be looking at some sort of procurement activity or something. I mean, are there certain types of audits where this swarm auditing approach makes more sense? Holistically, all three tiers, if all combined, should be applied, I believe, in a mature department across all audits.

That said, the individual swarms of agents, the tier one, like you said, do have more, could be more targeted in their use cases. So, again, for us, helping and streamlining that data analytics task was a key use case for us for that orchestration of the agents. So this whole concept, I guess, of integrating the AI agents and the human auditors, it's kind of futuristic sounding, right? I mean, we wouldn't have been talking this way three or four or five years ago.

So how do you keep the internal auditors in the loop while you're allowing your AI agents to operate at scale? Great question. I am under belief that it has to be built into your processes and architectures. So for anyone that's audited AI use, it's a key distinction between there's AI governance, which has the policies and the monitoring and the dashboards.

And then there's governed AI, where access and authorization and controls are embedded into the system itself. And that's where the real accountability lives. I mean, how long have you guys been using this approach? Is it relatively new or have you been experimenting with it for a while?

It's still growing and still being implemented. we've taken pieces of it and have like I said implemented a couple of the tiers and we're looking forward to growing it as a whole, Now, I understand that you and your boss are going to come in and talk about this at the iGAM conference in March. Yes, and we can get more into details about specific implementations then. So anyone listening that wants to come to that.

So this is kind of a plug, right, to have people come and, you know, this is kind of like film at 11, right? You know, when the news guys are teasing something. So I'm intrigued. I'm going to be a gam.

So I'm going to definitely come in and sit in the room and try to learn more about this because, you know, I mean, it just sounds cool, right? Swarm auditing. Anything that makes auditors sound cool is always going to get my attention. So I'm intrigued.

Right. So how is building AI specifically for internal audit use helped your department. Think about the broader companies AI use? I mean, so we talk about AI in so many dimensions.

I mean, one is how's my company using AI? And then the other is how are we going to use AI? And then we got to think about how we're going to use AI to audit how they use AI. I mean, And how is that going?

Building AI for audit means we got to experience the full process, the AI governance firsthand, not just reviewing it. So we got that full understanding of all the challenges and everything. I like to say that helped us earn a seat at the table when the company makes broader AI decisions. As AI automates more of the objective testing, which is, I consider the science of auditing, the future lies in the art, the human judgment, the ethical oversight, the strategic storytelling.

And that's what keeps auditing, from my perspective, as a partner to the C-suite rather than just a cost center. So I'd like to ask you, in the context of accelerated AI capabilities, how would you define the art of auditing? Oh, that's a great question because I think we're going to be experiencing a fundamental transformation. I mean, you know, I've been around the profession of...

For 50 years, probably since your parents were kids. And I've seen so much change. I mean, I've seen everything from introduction of mainframes to desktops to laptops to internet to email. I mean, I've been through all of those transformations and they were all impactful for internal audit.

But I'm not sure anything, I mean, my imagination gets away from me when I think about internal audit, because I could see that it could essentially replicate most of what we've traditionally done. And then I think what we have to do, and I wrote a piece last year called, Will Internal Audit Win the Race for Relevance? And I think what we have to do is we have to reinvent ourselves, right? We have to say, okay, if AI can do the basic blocking and tackling, or it can do the fundamental assurance work, sort of the backward-looking work, what is there for us?

And I think that's where I talk about the AI superpowers or the internal audit superpowers. That's where we have to really leverage the skills that we can cultivate in ourselves that maybe AI doesn't have the ability to or maybe we wouldn't ultimately trust it to. Things like intellectual curiosity, things like, you know, a non-biased ethical compass, all of those kinds of things. So.

If you start thinking about those are the skills we're going to need, you could easily envision that internal audit just will be fundamentally different in the future. And I'm not enough of a futurist to say exactly how it will be different, but I do believe that a lot of what we've traditionally done is destined to be done by AI, to be done by the agents, whatever terms we want to apply to it. So let me ask you then, I'm going to turn it back to you. I mean, you've asked me how i see the future i just shared it with you i mean do you see the future any differently or am i am i perhaps being too dramatic and what i think the future of internal audit could be i although i was evasive maybe i was still kind of dramatic do you think it's going to change that much uh in the future.

Absolutely. I think it's definitely going to change in the future, so much so that who knows if three lines will even follow the same format going forward. But when that happens and what comes after that is still debatable and I think shapeable to some extent, like you said. And how we approach it and how we define ourselves is up to us at this point.

You've got some great insights. And you're clearly out there in front of a lot of us and a lot of people in the profession. I'm just curious, how do your colleagues kind of react to and relate to you? I mean, do they see you as, hey, we got to go ask Richard what to do because we don't really know this stuff.

Or are you guys all on this journey together and everybody is equally confident about how to use AI and where it's going? All of our success at PayPal has come from collaboration from the entire department. No matter what level you're at, AI has made it a level playing field for everyone. It was just a brand new technology just a couple of years ago.

And in practice, I know AI algorithms have been along much longer than that. It's safe to say now you can't spell internal audit at PayPal without AI, right? I'd like to say so. And I do want to give a big shout out to my team, to our leaders who have been pushing this from the top down and also from the bottom up.

I think it's been a great collaboration, and I think there's still a lot of work to do, but I think we're making great progress on it. It sounds like you guys are my favorite line out of all of the Back to the Future movies is where we're going, there ain't no roads. It sounds like you guys are going where there ain't no roads, right? So, my congratulations to you.

I'm excited for you, and I'm looking forward to sitting in the back of the room and trying to take in more from you and your leader there when you guys come to GAM to talk about swarm auditing. Richard, thank you for joining me and to our audience. Thank you for joining us another episode of Speaking of Risk and Audit. For Audit Board, I'm Richard Chambers.

Thank you.