Phil Dawson, Armilla AI: Insurance for AI Risks

Speaker A: Foreign.

Speaker B: Hi, I'm Kevin Werbeck, professor of Legal Studies and Business Ethics at the Wharton School of the University of Pennsylvania. For decades, I've studied emerging technologies from broadband to blockchain. Today, AI is promising to transform our world. But AI needs accountability mechanisms to ensure it's developed and deployed in responsible, safe and trustworthy ways. On this podcast, I speak with the experts leading the charge for accountable AI. Phil Dawson is head of AI Policy and Partnerships at Armilla AI. We talk about the need and demand for AI specific insurance, what it takes to develop effective AI policies, how AI evaluation and assurance mechanisms feed into insurance underwriting, what policymakers can do to facilitate a healthy AI, uh, insurance market, what we can learn from cyber insurance to apply to the AI context, as well as how AI insurance could create pressure for companies to invest effectively and efficiently in AI governance. Phil, welcome to the road to accountable AI.

Speaker C: Thanks for having me.

Speaker B: So, uh, let's start by talking about what exactly AI insurance means in specific terms. What exactly is that that companies are insuring against? Who's the one getting the insurance policies? How does that process work?

Speaker C: So AI insurance today, it's a new category of insurance, but it is still fairly heterogeneous. So people think of AI, ah, insurance when they hear the word as potentially being a standalone product such as our Milla's. Um, but there is an expectation in the market that other policies that make no reference whatsoever to AI are also covering AI risk. Right. So we have, that has kind of been the historical status quo. And now when we talk about AI insurance, typically we're meeting one of these, uh, standalone products like Armilla has standalone insurance product that deals with AI risk and related claims. Firstly, that would mean for. Let's talk about the insured asset as you, as you mentioned for us at Armilla, how we think about the insured asset, that is the AI system or model or system of models. So effectively the AI powered application, we're not talking about necessarily the foundation model itself out of the box chatgpt, uh, off the shelf. We're talking about a model, could be an LLM, could be a classical machine learning model or a system of them that, that have been orchestrated and maybe fine tuned and with associated guardrails used for a specific purpose or series of purposes and if the surface area for those uses is expanding all the time. But we're talking about the AI system that is fundamentally driving or powering an application, a software application. So the insured asset is an AI product, an AI powered product that's kind of where we start, that is what we're ensuring. And in terms of covered losses, there's two types of losses. First party losses that someone, for instance, using an AI, uh, application might suffer if it hallucinates or leads to model error or some other type of unintended behavior or harmful output that causes property damage, for example, to the insured, say it's using AI in an industrial plant or manufacturing plant for predictive maintenance of equipment, and some failure leads to property damage. That would be a first party harm or loss. There are also third party harms. I think that's where we've seen the greatest interest so far in protection against third party losses. And that would be where AI used by a company leads to some third party claim against them from a customer, a partner, an individual who has suffered themselves some type of harm on account of the AI underperforming or some error or systemic discrimination. And maybe some of those class actions that you see in both the health insurance space or in AI and hiring and some of the HR tech platforms that are being sued with alleged bias or systemic bias. That would be a great and prominent example of third party harms.

Speaker B: All right, well, let's get into that question. As you say, there are lots of existing policies that companies have, whether it's general liability or directors and officers or something else. Why do we need a company like Armilla to specialize in AI insurance?

Speaker C: There's a really good and also nuanced answer to that question, I think. Why do we need AI insurance policies today? Why do we need a company like Armilla, uh, to help to launch a new category of insurance? Well, the most basic way to answer that is to say that traditional insurance policies, like you mentioned, today's directors and office officers, uh, insurance policies, errors and admissions policies, tech E and O professional liability, cyber insurance, they were not designed in their wording or even in their underwriting intent to cover AI risk. Today there was two things there. There's not the explicit wording and as part of the policy and the coverage. So there's not that clarity in the market that the market is actually really craving right now, given the extent of AI risk and liability exposures. So there's not that clarity in the wording. But there's also has not, as of yet, been the intent on the underwriting side. So many of the policies today that make silently cover or address AI risk and do not explicitly exclude it were not underwritten to the risks as part of a insurance underwriting process. When an applicant, a large company, for instance, applies for One of these insurance products, they will typically undergo some type of an assessment, let's say, on cyber insurance, to understand their risk posture. This is changing. But up until recently, mostly these policies and the underwriting questions did not make any mention of AI. AI, uh, the applications that are being used, the inventory of applications that are being used, the specific risks of each of these systems, that policies or governance processes that companies have in place to manage and govern AI, uh risk, the testing of these systems, the validation, whether there's third party audits or certifications that are obtained. This has not been part of the process of underwriting in cyber insurance or tech uh, eno. This is changing and we're starting to see questionnaires where insurers are asking more questions about AI. But historically this has not been the case. When an insured knows that the policy has no wording on AI, and in discussion with the broker and potentially also the carrier, they have an understanding that the underwriting process doesn't really address AI risk. And there is the assumption then that the pricing has not reflected a quantification of AI risk and that at some point the policy could change or certain exclusions, for instance, might be relied upon to deny claims if they're in some way applied to a certain scenario, or that more practically speaking, there could already be gaps, there could already be partial coverage or areas that insurers have a desire to move away from covering. So given, uh, the importance of AI in the economy today, given how important it is to some companies growth, particularly if they're software developers or if they're investing a lot in AI as a large enterprise, large deployer, with dozens, potentially hundreds, sometimes of AI uses in production, in material, uh, risk and sensitive uses of highly regulated industries and so on. You have very low tolerance for uncertainty in your insurance coverage that is supposed to be covering AI. You want to know that it covers AI risk. You want to see how it covers AI risk. You want to understand what limits will be available to you if there is an AI related loss that leads to a claim, a sizable claim you would let, you would prefer today potentially to pay an additional insurance premium to have that security around the dedicated AI insurance limits that are, that will be available to you. And there's an interesting survey from the Geneva association, which is a think tank that is, uh, led by 40 of the largest insurers in the world. They put out a report this past fall and a survey of 600, um, corporate insurance buyers at top Fortune 1000 and over 90% of them indicated they were now Seeking dedicated coverage for AI, generative AI. The market is asking for this and the policy language and the underwriting is struggling to keep up.

Speaker B: What then exactly does Armilla do? Presumably the writing language to be explicit about AI is the easy part. Is it that underwriting that risk asset help us understand where you fit into this landscape?

Speaker C: Yeah. So Armilla is what you would call a uh, managing general agent. And we are also a Lloyd's cover cover holder. So what that means is, is we as a startup insurer, we, we sit between the insurance capital providers or capacity providers as we call them. And for us that means large reinsurers and some insurers who have syndicates at Lloyds who are also interested in backing specialty insurers who are often MGAs, managing general agents who can help provide specific expertise, technical expertise in areas that are more difficult to underwrite or new products. Um, so yes, we're involved across the board in the drafting of the policy, wording and our products. We've done that with great collaboration from our insurance partners, our capacity providers. We're also technical underwriters. We perform the underwriting at Armilla, uh, and we bind the policy as an MGA and cover holder. When an insured or a broker brings us to one of their customers, they deal exclusively with Armilla and they get an Armilla insurance policy. They go through an Armilla led underwriting process and Armilla binds the policy and access the insurer.

Speaker B: The company didn't start with this business model. So tell us a little bit about the history about how Armilla got to AI insurance.

Speaker C: Our mill was founded almost five years ago now. I joined the company four years ago, uh, when the company was founded. I think the mission is stayed the same, but the business has evolved in search and support of that mission really to build justified or evidence based trust in artificial intelligence and ultimately help with adoption. So the company when it was founded was a SaaS company that was providing testing and validation tools to enterprise data science teams as a SaaS platform. And this is five years ago. The uh, goal being to provide as companies, particularly larger companies who are already expanding in the number of development projects that they were embarking upon. The idea was that they would need a standardized testing toolkit to be able to for efficiency reasons and then also to more comprehensively have access to the most comprehensive toolkit for testing. So that was the original company business model. But very quickly after launching the company, we saw an opportunity in the insurance market. Having discussions with reinsurers such as Swiss Re, who were already looking into the potential challenges emanating from silent AI coverage, or at least the uh, AI coverage that was there in insurance policies but silently addressed. They were looking at this already and trying to understand, and they've written some very helpful reports on this over the last several years trying to understand what the impacts could be for the industry itself and potentially for insurer to the extent that there were gaps in coverage or the need for new products to address new market requirements for AI. We thought that this would be uh, a potentially enormous opportunity and had uh, established some excellent partnerships including with Swiss Re and other partners, Greenlight Rental, others who we're still working with today. And the first product that we launched was a performance guarantee, addressing the problem of not only a, uh, need for third party validation of AI model performance metrics, but more than that, some coverage in the event that they underperformed with respect to the KPIs that are often stipulated, um, in the service level agreement, the sla, that was our first product. And even all the while we were working towards launching this liability insurance product that came out last May. As I said, the mission to build trust in AI has remained. The business itself has evolved. We performed dozens of audits and assessments of applications and in highly regulated contexts in HR and financial services and insurance customer service chatbots. All of this helped us build up the scalable evaluation capabilities that we need for underwriting as well as our own proprietary AI risk data on the understanding we developed and the data on how models fail, how robust, and the robustness requirements of AI systems, the probability of failure in different scenarios. And yet we notice that still companies, after we had provided them reports on how well models performed or potentially underperformed, they became fundamentally more concerned about the risk and exposure of these systems, understood the limitations and uh, the bounds of the performance. So we just saw insurance increasingly as we embarked on this journey. We saw insurance as a catalyst for trust, as a financial incentive for continuous improvement in safety and better testing practices. It became more and more clear that insurance would provide uh, a new incentive for companies to invest in the practices and the tooling and really the requirements to build and deploy safe, responsible AI.

Speaker B: But for insurance to be that catalyst for trust, there has to be enough clarity about causation. There has to be enough information out there both for the insureds and for a company like Armilla, um, and there needs to be some understanding about what are the kinds of practices that a firm can and should be doing and how they might affect the risk. Are you able to address all of those issues? In the current environment?

Speaker C: Yes and no. And I think most many of them are definitely works in process. You mentioned the idea of causation. And in some ways that also brings in lack of clarity or poor clarity on liability regimes related to AI. Yes, I think the area that with more clarification of the liability regime, for instance, or if there was a clear standard by which responsible, UH, conduct could be measured against that would bring a lot of clarity to us as an insurer. But for the time being, it's something that we're having to look at on both sides is both parties are concerned about liability, unclear about necessarily where it lies, and interested in exploring insurance coverage.

Speaker B: Your job specifically at Armilla is head of policy. What exactly does that entail? And why does an insurance company need someone focused on policy issues?

Speaker C: Yeah, my title is Head of AI, UH Policy Partnerships. In some ways, ahead of policy title is a little bit of a legacy title from Armilla's origins. And when we were launching our SaaS platform and providing third party assurance, there was always a need for, for someone to continue to follow, shape, participate in the public policy conversation on AI governance and ultimately nurture expertise on AI regulations, AI UH standards, AI risk management frameworks. And that has been my career for almost 10 years. My first roles in AI policy were 2017, 2018 at UH companies that were helping to draft the OECD AI principles. I've been part of the AI policy world for a number of years and it became directly relevant to Armilla from a commercial standpoint as we were providing the tools to operationalize these policies and these risk management and governance frameworks. That was still true when we became a provider of AI assurance services and assessments and evaluations companies. Our clients were very interested in how we had constructed our assessments based on what frameworks, what standards. And we're often looking for guidance in addition to our service on how they might adjust or develop their own responsible AI programs or AI governance programs. So there has been kind of a need for Amelia to showcase leadership and maintain expertise in AI policy governance, responsible AI, AI safety. And that has actually been true in insurance too.

Speaker B: Are there things governments can be doing to facilitate the development of a healthy AI UH insurance market?

Speaker C: Yes, and we get. This is a question. We get, I think more and more frequently, including from state legislatures and lawmakers who are interested in better understanding all the ways that public policy or legislation could incentivize better practice beyond command and control regulations of prescriptive obligations. They want to understand how to activate the market and tap into market Incentives for self incentivizing behaviors, uh, if we can put it that way. So some of the things I think we typically recommend are to explore policy and even legislative measures that could help incentivize transparency, disclosure as well as the emerging AI assurance industry. So things that we've already been talking about that help generate the standardized data that insurance underwriters ultimately need to help assess and quantify AI risk. And that's because today one of the challenges of covering AI as an assure is the absence of claims data, even proxies for claims data, to build actuarial uh, models and underwrite as underwriters would traditionally underwrite. And in the absence of that, what you do need is to have access to credible proxy signals for understanding AI risk. And for us, as I mentioned, that is rooted in companies overall governance and risk posture and critically the AI system itself, how performant it is, how robust it is, how safe, how guardrailed, how performant the guardrails are and to have data and all of these attributes of a company. So if policymakers as they explore different uh, policy initiatives or even legislation, regulation, if they can help incentivize companies to self assess, to undergo third party assessment, to provide, explore things like safe harbors, to incentivize disclosures and standardize what types of disclosures both in terms of the substance and the manner in which they're provided, then we are uh, beginning to have information that is readily available for underwriting, that is standard practice in the industry and not a special request from a specific underwriter as an example. So if it becomes standard practice to both generate this data and to share it commonly with protections and incentives in place, uh, in law, then that can significantly enhance underwriters ability to access the data they require to, to underwrite.

Speaker B: Does that work given how fast AI is moving? You mentioned when you started it was predictive. Machine learning was the primary thing companies are doing. Then generative AI and foundation models came along. Now so many firms are deploying agentic systems which have a different risk profile. So even if companies have to disclose, given the rapid speed of change, is it really possible to keep up and underwrite what they're actually doing today?

Speaker C: That's one of the core challenges, one of the many challenges of both assessing, assuring, evaluating AI systems but also underwriting them today is just the pace of change, the speed at which companies are experimenting or relying upon multiple models as part of systems and updating them as models are updated. And it's extremely dynamic. One of the things that Armilla does As part of its underwriting and conditions of binding policies. We have certain monitoring obligations that an insured would have. And other insurers who are looking at AI insurance as well are exploring this. It's a bit of a new construct in insurance to have continuing duties for policyholders. But uh, I think we think it is critical and progressively. I think what we would love to see, and we know it's not possible for every company today, would be continuous monitoring at greater debts. And that's not possible today and certainly not in place for most companies. But there are technologies that enable this. There are NLO ops platforms and observability platforms with evolving capabilities and ability to both detect drift and even attacks, uh, creatively. Just provide model output visibility into model outputs on a real time basis. I think they're in the fullness of time, which is maybe a bit of a strange thing to say in the AI, uh industry. But let's just say looking ahead, I think we definitely see the potential to have progressive monitoring in place. And for the time being, depending on the solution for us, it would be requesting updates when there was material change. That has been defined. And many companies do define material change in the system and we can also help them do that. But we would require updated information about the system that has undergone a material change and certainly annually. So that's how we're addressing it today. And depending on the use of, there may be more stringent monitoring requirements. But the idea is to be proportionate to really the business reality of the customer today, while taking into account our own needs as an underwriter to have sufficient information to underwrite and maintain a policy.

Speaker B: Mhm. What can we learn from cyber insurance, which seems like the most analogous scenario or product? There's lots of policies out there, but my understanding is the perception is that the state of practice in cybersecurity has not really improved dramatically. There hasn't been that virtuous cycle of insurance getting companies to improve in the same way. Are you able to take anything at Armilla from what has happened in that sector to learn how to do better for AI?

Speaker C: Yeah. And it's something that we hear many different types of analogies from the cyber insurance space, both in terms of how that product emerged and the time it took and some of the exclusions that catalyze, uh, or confirmed the category and turned it into the product, product that is today, as well as the evolution of the underwriting and some of the pitfalls, some of the challenges of, of keeping staying ahead of different threat vectors and attacks, particularly With AI now part of that threat landscape. So I think there are some good lessons there for AI, uh, insurance as well if it ever is to play that type of incentive for better governance. But I think first I'd say is that looking at AI insurance, risk based pricing will and should reward organizations with strong evaluation and governance practices. As if insurance premiums increase in proportion to the risk and if risk is measured in part based on how mature companies are from a uh, governance risk management standpoint, from a uh, testing, from a control standpoint, then the incentives should be aligned. That is not always the case with underwriting. We do see today in the AI insurance market different companies that are exploring alternative means of underwriting based on some of these proxy data sets. For instance AI litigation data, maybe incident data. But I think the consensus is that those data sets are very limited today when it comes to underwriting AI risk. They can be used as, they're not sufficient as proxies. Mostly it's kind of wild to think of them uh, providing the foundations for underwriting. If you think of litigation data is looking at systems that are being litigated. Many times those systems are. And now we're trying to predict the risks of agentic systems or multi agent, multi agent systems that are just being developed now versus you know, the copyright and copyright and training data lawsuits that are filing its chatgpt one or two years after it was launched. So I think there, there are cases today where we see underwriters whose approaches are not aligned with safety or governance practices or based on them and therefore providing incentives for continuous improvement. They're, they're looking at risk, macro level risk signals that do not necessarily translate into system specific incentives for improving safety or risk posture. So uh, it can go both ways. I think we definitely view our approach as helping to incentivize evaluations and incentivize uh, improvements in safety, reduction of risk. In part that's explained by Armila's journey towards underwriting. The second would be that insurers can also require coverage conditions. We mentioned duty to monitor and potentially duty to have a third party assessment performed. There are different controls that an insurer can place or subjectivities an insurer can require of an insurer to, as part of binding the policy. So there's um, a quasi regulatory role that the insurer can play. We're ready to bind but we're going to need to see these adjustments over the course of the policy period or a renewal for instance based on the new feature material change in the application or new deployment I think another thing to take into account is that premiums should scale the deployment size at risk and risk exposure. Some traditional underwriting process insurance products might look at some things like company revenue and that would play a significant role as part of the pricing and amongst other factors. It's m a little bit more challenging for artificial intelligence where you can have very small providers uh, uh, who become one successful very very quickly or land their first client which is a Fortune 500 company that uses their HR AI hiring system at scale. See the exposure, the profile a little bit different and the premiums can and should take account of that so that as a company grows or depending on the type of deployment, the type of customer it has, the exposure profile is very, very different and pricing should reflect that. So one thing we've done at Armilla, um, is we've launched a partnership program that well known AI, GRC platforms or auditing companies, certification bodies, assessment providers into services that we can offer to our insured. So now we're investing time and support and helping to create for companies who don't have already this whole marketplace of uh, technologies and services that can help them improve their posture and of course that makes them more ready to be underwritten by Armilla. But there's a role there for insurers in helping to improve the overall maturity of the industry. Uh, as a begins to explore insurance and insurability. And beyond that, think of some of the things that we mentioned before investing time in shaping industry standards and some of the research into areas around liability. For instance, these are things that we're doing at our Milla and that other insurers would naturally have an incentive to do as well to help build the market.

Speaker B: Mhm. It's a really exciting opportunity and fascinating to watch as you and others are trying to build this. We have to wrap up but Phil, thanks so much for being part of this podcast.

Speaker C: Thank you. Thank you so much for asking me m to be on the show today.

Speaker B: This has been the road to accountable AI. If you like what you're hearing, please give us a good review and check out my substack for more insights on AI accountability. Thank you for listening.

Speaker A: If you're listening to the road to accountable AI, you're probably thinking about how organizations can build AI systems responsibly. But another important question is how are companies actually putting AI to work in the real world? That's what we explore on Where AI Works, Conversations at the Intersection of AI and Industry, a podcast from the Wharton School in collaboration with Accenture. Each episode brings together Wharton research and real world case studies to examine how organizations are using AI to upskill their workforce, improve customer experiences, and transform how they operate. If you want practical insights into how AI is shaping business today, listen to where AI works wherever you get your podcasts.

Speaker B: If you want to go deeper on AI, uh, Governance, Trust and Responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of Wharton's Strategies for Accountable AI online executive education program featuring live interaction with faculty, expert interviews, and custom designed asynchronous content. Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit Execed Wharton UPenn. Edu acai for full details. I hope to see you there.