Agentic AI Rewrites SOX Testing

Transcription by CastingWords, Hello, I'm Richard Chambers, and welcome to another episode in my podcast series, Speaking of Risk and Audit. Joining me today is Kieran Taylor, the co-founder and CEO of Midship, a company that just made real waves across the industry with its recent acquisition by Optro. Kieran's team developed a Genic AI that can take a 16-hour SOX testing process and finish it up in under 15 minutes. It sounds pretty cool.

And now he's at the forefront of the industry's shift toward autonomous controls. Kieran, it's great to have you on the show. Thanks so much, Richard. Likewise, it's great to be here.

Kieran, can you tell our listeners a little about your background and how did you get into creating agentic AI solutions for internal auditors? Yeah, absolutely. So I actually spent the first couple of years of my career at Deloitte in London, which makes everyone think I was in audit from the start. I was actually an engineer there working alongside some of my peers going through the audit track.

After I moved across to the US, I was working at Instacart for two years prior to their IPO. So we were doing a lot of SOX readiness programs for the data platform we were working on. At that point, I saw firsthand some of the pain which my colleagues and peers were experiencing. And at the same time as that, OpenAI just started releasing access to their APIs to start building these freeform AI products.

We were asked by the management to work on some hackathons to say, what's the most painful problem you think you can solve right now? To me, the most obvious thing was SOX control testing. So we started building some internal things to try and build that out. Eventually realized this was a big enough problem to take it to the market.

And so I decided to start Midship to actually go and solve that problem. Well, that sounds pretty cool. Pretty exciting. Well, you know, Karen, Midship was rapidly gaining traction as an independent AI native SOX solution.

Beyond tech, what did you see in the Optro leadership and vision that convinced you that they were the right stewards for the agentic AI technology you and your team had built? Yeah, absolutely. So it was the overarching vision of actually becoming the system of action for every point in time piece of technology that was being built out right now. So right now, ourselves, all of our competitors are taking one particular cycle, one particular bit of work, and trying to automate that completely with AI agents, these point in time utilities.

The next step is how to unify that and synthesize it across a full audit management program. And so we thought to ourselves, we either have to take on the really big swing of trying to build it ourselves, or we get to go join the market leader who's already doing this and already has this vision for building it together. So I think a couple of those conversations with the team just made it really clear that it's not just a short-term, hey, how can we automate this one thing? It's to build that synthesized platform, which is really going to solve that whole problem.

It sounds like you guys zeroed in on one particular aspect of internal audits mission and really got that right. So that's pretty exciting. How does Midship's autonomous testing, in your opinion, how does the autonomous testing engine plug into Optro's existing control management solution? Yeah, so one thing we get for free out of the box by just plugging straight into the Audit Board Optro API is we get all the control definitions and all that management of exactly what does the program look like.

On top of that, we will take past work papers from previous testing cycles and essentially reverse engineer them to enrich those controls with a detailed description of how you actually go about testing them in practice. So this is one place where teams will manage this in multiple different ways. Sometimes it lives inside of an Excel work paper, Word documents, freeform PDFs. There's various different ways.

So we build additional AI agents which can do that reverse engineering to build out the test procedures that our AI agents can then follow when they're actually testing itself. So all of that part of seamlessly connecting into Optro to actually get the base control definition is only going to get better over time. And that's really what we're building towards in the next few weeks here. You know, Karen, for the internal auditor who lives in Excel, which is a primary focus for Midship, how will this acquisition change or the acquisition of the agents that you guys built?

How will that change their daily workflow? Will native Excel experience remain a priority within the larger Optro platform, do you think? Yeah, so I've been involved in various projects over my career where it's an obvious kind of tension where engineering teams will see people using Excel and say, hey, I'm going to build a system, which means that you don't have to do that anymore. Very, very frequently fails.

And the reason why it does that is because people enjoy working in Excel because they get to actually express the mental model of the work they're performing however they like to. It's flexible enough to accommodate that. The problem and the tension it runs up against is when we're then trying to define a well-defined task for an AI agent to perform, all of that complexity of Excel can actually make it more difficult to standardize the work. So all of that background saying is that we really want to make it clear where the handoff is between tasks that are purely being done by an AI agent, tasks where you have to collaborate with the agent synchronously, and then tasks where it's still a human completely managing that themselves.

That third bucket, where after we've done the work inside of the midship web app with the AI agents iterating on that together, and we've ejected out and now it's a human purely working on that, Excel is still the most obvious vehicle and still the most flexible model for doing that. So really, we're being very intentional where we hand off between doing work inside of the web app versus then doing work in Excel downstream. You know, Kieran, as I talk to internal auditors, a lot of the internal auditors, particularly in smaller shops who don't really have a lot of resources and don't really have the reach or knowledge around AI that others do, what I'm often struck by is that they're just sort of overwhelmed by all the terminology around AI.

They, you know, they first started hearing about AI, then they hear a lot about generative AI. And now more recently, we're all talking about agentic AI. Maybe you could help them understand a little bit. So how do you define an AI agent in terms of audit or internal audit?

And what about it could be considered a missing ingredient in maybe a lot of the legacy GRC tools. Yeah, absolutely. So very broadly, an AI agent can navigate an unstructured environment. So in the context of audit, the most obvious one is PBC evidence.

It can come in various different forms. You have no idea what that structure is going to look like up front. It can also take a goal and work independently to actually go and achieve that by itself. by, again, reading from documentation, writing back into it, and actually interacting with that environment.

For Audit, this is such a game changer because it's been previously impossible to write software that accounts for every possible variation of data which is seen in an enterprise testing cycle. So because agents can account for these variances, they can finally achieve the goal, which everyone has been talking about for the last 20, 25 years, of actually automating SOX testing end-to-end. You guys certainly, I think, cracked the code on that, and people are pretty excited about what you've achieved.

You know, you've mentioned that every step that an agent takes must be re-performable, right? So this is terminology I think that's going to ring true to the ears of internal auditors. Can you walk us through how the technology you guys develop provides an audit trail for the AI to satisfy skeptical internal auditors? I guess some might say that's being redundant, skeptical internal auditors.

Because almost all of us are skeptical. But how does your technology provide that audit trail for the AI? Yeah, absolutely. So there's really two primitives that the agent relies on to show what work it is doing at any given time.

It's going to do one of two things. It's either going to point back to some documentary evidence and say, you can plainly look at this yourself and understand it, or it's going to point back to some intermediate work that it has performed. So you can imagine inside of an Excel workbook, if you have to sum together five different values, the agent can write back into that workbook, sum the values itself, and then point back to it at the end of that procedure. Say, here, this is the work which I did to prove out the conclusion I came to is correct.

So every single conclusion that the agent comes to a midship is going to have one or two or a combination of both of those pieces of evidence to point back to. And this is really important because I think people are familiar with working with free form AI tools like ChatGPT, where it can go away for 10 minutes, do a bunch of really complex things and give you an answer. And then when it comes to the task of how do I actually trust that this is correct or not, it's impossible to deconstruct and actually look at each of those layers that it took.

So that is one of the invariants that we would call it, that we will always say every single conclusion we come to is supported by one of those two plainly evidenceable artifacts. You know, Kieran, when I first heard the news that we were in discussions for the acquisition of Midship, I started doing my own kind of informal homework. And I was blown away to learn that Midship has the ability to automate over 85% of SOX controls. That just really captured my attention right away.

And I would think it would almost anyone out there who's got that SOX responsibility in internal audit. Which specific control areas, IGTC, financial, et cetera, are seeing the most immediate agentic lift? And which are still best left, in your opinion, to human judgments? Yeah, absolutely.

So, ITGCs are definitely easiest to get started with. I think a lot of this is due to the nature of the evidence itself and the control procedures. Because the systems themselves being audited are structured software components, it gives some level of standardization that the agent can take advantage of. BPC is often more difficult because the test steps being performed are either poorly documented or it just takes some time for midship users to actually offload the knowledge living inside of their heads into the platform.

And then there is a third bucket of controls where it's saying, hey, long-term, we don't think this is ever going to be a candidate for automation. And that's exactly what you're talking about. There's high levels of judgment which have to be applied to actually evaluate the control. If you can't describe in natural language what test steps to take, it just relies on your experiences and all of a sudden that skepticism you're talking about.

That is something which long-term humans are still going to handle. So Kieran, when I heard about the acquisition, I was, as I said, I was really excited, really blown away by the fact that we were in a conversation with such a highly reputable organization, highly reputable company, Midship, in a space that a lot of people are still pioneering. But I was as excited as I was about the fact that you guys had already developed the agentic AI solutions for SOX work. I was even more pumped by the idea that we were bringing in a team that had that experience in creating AI agent solutions for internal auditors.

How do you and your team feel about that and going from here in terms of helping to create AI agentic solutions for a whole range of internal audit responsibilities? I think one of the things as we're going through the decision around the acquisition, the classic story for any startup is you've been pouring blood, sweat and tears into something for a number of years. You don't want to give up on that vision that you had moving forward as well on some of that autonomy. And I think harking back to some of those conversations we had with leadership, the thing which made it much easier from the decision on our side is that we don't have to stop working on that vision itself.

There is so much still to be done where this field looks like two years from now versus today is going to be very, very vastly different. And we have that mandate to go forwards and deliver on that together. Really, it just puts us into this much nicer protective bubble with more resources to move faster and do more things and take bigger swings as well. So I think in terms of the roadmap we were planning on, none of that changes.

And it means that the team still has that internal motivation to go and deliver on the future, which we all know is coming. Yeah. Speaking of the future, you know, as AI agents take over a lot of, well, I don't want to use the term grunt work, but it seems maybe descriptive, that internal auditors have traditionally done. What do you think the internal auditor of 2030 is going to actually do?

What's going to define the role of internal auditors in the future? And what skills should our podcast listeners be developing right now to stay relevant? All of the recent AI adoption in different industries is causing some anxiety. And so I think that the broad message for myself is that the next few years are going to be really exciting for the internal audit profession.

So I spoke at GAM about the idea of auditors being uniquely positioned with a skillset that is complementary to AI agents. I could probably talk about this one for an hour, but really the biggest change I think is as the marginal cost of testing goes to zero and we move away from structured annually driven audit cycles towards ad hoc work that is like the most obvious progression for how broadly the topology of work is going to change and really it is any of the tasks which can be tightly well defined and repeatable where you're doing it multiple times every year that is obvious candidates for AI agents go take over and take some of that work that then frees up auditors to actually do the core skill set which they're meant to be doing like you said yourself thinking critically applying skepticism and thinking about those really broad, difficult, constantly changing problems and enterprise, where it's difficult to even write down in language exactly what it is that you're focusing on as well.

So definitely the move away from that structured annual cycle to the ad hoc work is the obvious progression. In terms of the skill sets themselves, I think that there is an argument that just learning managerial skills that you would apply to humans is going to be useful in the AI audit context as well. And so figuring out how everyone becomes their own kind of mini manager of a fleet of AI agents. This concept is coming up in various different professions and industries right now, but I do think it's somewhat of a truism at this point that people should accommodate themselves for.

Yeah, I smile a little bit when you say that. I know as an audit manager, I would always have my teams go out and do work. And it wasn't as simple as it will probably be, or it wasn't as easy sometimes because human internal auditors tend to complain. They don't tend to always be as highly motivated, but I've not had any complaints from AI tools when I've used them.

So that's pretty exciting as far as what the future holds. Kieran, listen, thank you so much for joining me on this podcast. I don't want to repeat myself, but I truly am excited that you and your team have joined forces with us. I think there is a lot of very exciting news to come in terms of what we work together to create.

And so thank you again and congratulations and welcome to Optro. Of course. Yeah. Thank you so much for having me.

The feeling is very much mutual on our side. We're extremely excited to get going and revisit this conversation in a couple of years' time and say, look at all the amazing things that we've achieved together. Sounds great. And to our listeners, thank you for joining us.

I'm Richard Chambers, the Senior Advisor for Risk and Audit at Optro.