Building AI Assurance Ecosystem with Bri Treece (PACT AI) and Dr. Rob Slone (UL Solutions)

Speaker A: When the AI overlords take over, what are you most excited about? It's not crazy, it's just smart. I think just this year, in the first six months, there have been something like a thousand laws. Who's actually building the scaffolding around how it's going to work, how everyday folks are going to use it? AI only works if society lets it work. There are so many questions have to be figured out and nobody came to my bonus class. Let's enforce the rules of the road. Welcome back to Scaling Laws, the podcast brought to you by Lawfare and the University of Texas School of Law that explores the intersection of AI policy and of course, the law. I'm Kevin Frazier, Director of the AI Innovation and Law Program at Texas Law and a senior editor at Lawfare. Today we're joined by Bree Treece, executive director of Pact AI, and Dr. Rob Sloan, senior Vice President, Chief Scientist and Innovation Officer at UL Solutions. AI governance often focuses on model developers and end users. Today we're looking at the organizations in between the companies, health systems, nonprofits and other institutions deploying AI in real world settings. Bri and Rob explain why unclear accountability can slow responsible adoption, what AI uh assurance can learn from more established safety regimes, and how PACT AI hopes to build the infrastructure needed for trust to keep pace with innovation. There's a lot to unpack with this exciting new organization being released, and we'll get into all of those weeds in just one second. To get in touch with us, email AIAW, uh, UTexas. Edu or follow us on X or Bluesky. And of course, feel free to leave us a 5 star review so I have something to brag about with my family during Thanksgiving. And with that, giddy up for a great show. Bri Rob, welcome to Scaling Laws.

Speaker B: Thank you. Good afternoon.

Speaker C: Great to be here.

Speaker A: So forgive me for a bit of a long wind up here, but I want to make sure that everyone is on the same page from the outset. So a lot of AI governance focuses on two ends of the spectrum. We've got the developers who create the leading AI models and then on the other end of the spectrum we have the end users. Yet deployers, the companies, the nonprofits, the organizations that build on AI and integrate it into their operations are often neglected. We don't give them their due. We don't talk about the concerns they have, we don't discuss the issues that may be on the horizon. And that neglect has major consequences when it comes to diffusing AI across the economy. To borrow from Dave uh, Garland, Managing Partner At Second Century Ventures, quote, you can outsource a task to AI, but you can't outsource the accountability. It's a pithy line that gets at the fact that someone is always going to be on the hook when a harm occurs. The question is, who? And right now, the failure to answer that question clearly and consistently is undermining something that most folks in the AI space claim to want. The diffusion of safe, reliable tools that can allow us to solve problems, create new jobs, and empower more entrepreneurial activity. I hang out with a bunch of law professors who are doing their best to come up with liability regimes to help mitigate that uncertainty. But the world's moving a heck of a lot faster than the pace of law review articles. And that's why I was so excited to have Bri and Rob on to discuss the launch of pact. AI and Bri. Let's just start with a good old acronym in the AI space, because everyone was just hungry. We need more acronyms in the AI space. What is packed AI? And I want to hear the elevator pitch of what problems you want packed AI to solve.

Speaker C: Yeah, well, thanks, Kevin. You did a lot of the setup for me, so I appreciate that. And correct. The primary reason we launched this was to get another acronym going. Um, this acronym is the Partnership for Assurance, Credibility and Trust on AI. Um, and it happens to shorten up nicely on packed AI of we are all coming together to work on making this go well. And so the people that we're bringing together in this are a lot of the consumers, end users, and voices that you had already mentioned. We are bringing together enterprises, independent technical experts, the folks who are doing verification and evaluation on models and systems. We have AI insurers, those folks who are trying to figure out how do we underwrite this? Um, and then, just as importantly, we have civil society and research voices that are coming altogether to create what we're thinking of as a whole economy infrastructure that we need here to verify that these systems are safe, secure, they're working as intended.

Speaker A: So, Bri, just to, uh, dive a little bit deeper, I know I asked for the elevator pitch. So now. Now I'm. Now I'm asking for. I guess we're hitting the stairs. We're taking a little longer of a jaunt to get to the next floor here. What problem do you want to solve? Is this a policy endeavor? Is this a awareness campaign? Is it a. A research entity? What's the key outputs that we should expect from packed AI?

Speaker C: There are three key things that this group is working on, we will develop a policy, position and voice here. We have some, some guiding principles of what that looks like. But ultimately it is to catalyze and steward a uh, quality independent AI assurance ecosystem that can really meet the need of the questions that are getting asked as this rolls into our economy and our society. And we are also doing the work to just start to build. If you talk with enterprises and you talk with the technical folks who are getting to the answers that the enterprises have, there's some kind of low hanging fruit there. We call it getting the pipes to line up more quickly. There are estimates that this AI assurance ecosystem will be upwards of hundreds of billions of dollars in the coming, uh, years. It's really a new market that's emerging. What doesn't always exist around it is awareness that this exists. And we think of this as expediting some of the connection points that need to be made. Thirdly, under all of this, in any kind of assurance ecosystem assurance regime, there's, there are pieces of infrastructure that need to exist. What are we actually checking for? What are standards in reporting and legibility so we understand what's coming out on the other side. Um, how are we getting the right codes of ethics in place? Um, there are just, you know, we've learned how to do this before. I mean we have, uh, one of the greats, uh, here with us today coming from an organization that has done this for 130 years. And so we're doing the policy, we're doing the, what we call getting the pipes to align the market development stuff and then we're building out some of the infrastructure.

Speaker A: Well, uh, admittedly, and this will come as no surprise to recurring listeners, AKA just my dad, but no surprise to my dad that I'm a huge fan of that building mentality. It's really easy to just release another blog post, or in my case to release another podcast. It's a whole nother question to actually build the thing to release the standards to show that there's something that people are going to align, uh, with, adhere to and be held accountable to. Which is super exciting given as I noted from that way too long intro, that people want to know who's on the hook for these certain things. And that matters across the entirety of that spectrum we were discussing. And so Rob, I'm really keen to hear a little bit more about UL Solutions role in this. And uh, I'm guessing that not everyone has heard necessarily of UL Solutions, but my hunch is that everyone is benefiting from the work you all have done at some point in time, given that you've been around since arguably 1893. So what is UL solutions role in the larger pack AI ecosystem and what is UL solutions to begin with, just in case folks didn't do their homework?

Speaker B: Sure, yeah, no, happy to, to uh, share a little background. Kevin, thanks again for the invitation to join today. And we're very supportive, we're very proud and happy to be part of founding partners, uh, members of pact AI. We're very supportive of what Bri and her team are pulling together. This is an ecosystem that needs to be um, assembled. There are a lot of pipes to line up, to use Bri's description, UL solutions. Um, we are accustomed to those sorts of challenges, I would say. So back in 1893 there was this amazing new technology that people were both excited and terrified of called electricity. And um, you know, the Columbian Exposition was taking place in Chicago and Chicago had burned badly 20 years or so before that. And so the mission of our founder, William Henry Merrill Jr. Was don't let Chicago burn again now that we are ah, going to have a haul of electricity. So from the beginning, uh, from the very foundation we were wired for safety. And our uh, mission is working for a safer world. So we test and inspect and certify a whole vast range of products. So you name it, control panels, electrical infrastructure, um, these days, data center components, key items there, um, everything, toasters, appliances along the way, all along that journey. And so our organization does not make any of these products. Uh, that is actually very important. We are a third party. We are impartial, we are coldly analytical and we are very thorough. And so we don't skip steps. And uh, that is why our customers have reached for us for over 132 years. And so we were born with the birth of electricity. And we've moved into a lot of different areas over time. Uh, and now we are tackling along with our customers, uh, artificial intelligence. So uh, it's a new challenge. But a lot of the aspects of it are familiar. Uh, for us and for UL Solutions in terms of what we do, the most visible sort of sign that we've been part of a product journey is our mark. So it's the U and the L in a circle and a whole wide range of products around the world have, have these on them. It's a way to know that that product's kind of been through it while it's been designed. And yes, there needs to be accountability and someone standing behind it and so they've bothered if you Will to go through that certification process with us. Uh, so that's a little bit of, kind of what we do and where we play, uh, historically.

Speaker A: And for folks who are thinking, my gosh, how does one single organization at once think about the standards that should be imposed on toasters, to the standards that should be imposed on data centers, to the standards that should be imposed on AI now and even frontier AI, very complex AI systems. What is it about the UL Solutions model that gives you all the credibility to say, uh, yes, Chicago won't burn again and now hugging face won't get hacked again. Uh, what is the, what is the process that lends you all that degree of credibility to span so many different items?

Speaker B: All right, so as an assurance provider, which BRI and PACT AI have thoughtfully assembled the ecosystem, um, we're a part of it. So the part that we play, the role that we play, if you will, is to build, um, in this case, digital trust. Uh, we have three commitments that we have to make. Um, my role, I'm responsible for these. I'm responsible for our reputation. Uh, no pressure there, but, um, you got it.

Speaker A: We believe in you, Rob.

Speaker B: We put a lot into these three commitments. Uh, every day we've got 15,000 employees who can tell you what they are. So impartiality. Pretty straightforward. Yeah. So we can't go and help design a product and then judge it. Right. We have to be very analytical, very, very objective and impartial. That's the first one. Second one is consistency. So no easy street for one customer and a hard time for another. We have to treat every customer the same way, uh, each time down the path. The third one is extremely relevant to AI and that is competency engineers from outstanding schools like UT Austin. There we go.

Speaker A: Hook em. Well done. You did. Rob,

Speaker B: Engineers five years ago were not graduating with AI skills baked in. Right. They were not, they was not part of the curriculum. That's changing over time. But with our workforce, we do spend a lot of time training our people to make sure they are able to assess AI enabled products and software. And they can, um, run through those tests and those evaluations, if you will, of the products, uh, in a credible way. And then in that manner we can decide, does this qualify, does it get the mark or does it not? And is there more work that needs to be done? If there's more work that needs to be done, we don't tell customers how to redesign a product because that makes us a designer. That's how assurance providers do this. Um, we give them the Scorecard. It's up to them to fix the issues and then at that point they could receive um, the UL mark if they pass for UL solutions. So that's kind of how we play our role in this. It's a little bit different for AI than other systems, but not totally.

Speaker A: Um, and let's, let's come back to that AI testing component in particular in a second because I want to know a little bit more about how UL Solutions, just being one founding member and being one form of assurance fits into the broader PACT ecosystem. So Bri, I know you all went through a kind of 90 day sprint to form some of the core bones of Pakt AI. What was that Sprint, who was a part of it? And, and what was the net result? What does the governing structure of this new body look like? Because I could imagine, you know, maybe you have the biggest deployers who say we want to prioritize the standardization of AI enabled toasters, uh, first or our first priority should be instead on developing a more robust insurance ecosystem. How do you manage the fact that you all have everyone from Target, I believe, to UL solutions to insurance providers? That's, that's an interesting dynamic, uh, and an interesting threesome. Not, not to, meant to be a loaded statement there.

Speaker C: Um, yeah, the founding, the founding cohort was really to, to test a bit of. Can we bring these three voices together? All pointed towards this north star of a high quality, independent AI assurance ecosystem. Recognizing that we have intentionally chosen this AI assurance ecosystem as a very broad term. It goes all the way from AI hardware verification through the multitude of applications and use cases. Packed AI is not going to be able to solve for all of those things. But to our earlier point, there are some things that actually fall neatly for what everybody is looking for. One is we need better data coming out on understanding what these systems are. Whether you are an enterprise that is putting these products to use and understanding what they're, they're doing, whether you're uh, an AI assurance provider. That's really the work you're doing is providing the data to say what's happening here. And particularly for the insurers, they are looking at what is our um, data that we can actually start to build our products from, that we can start to underwrite from. There are pieces of that that can start to get built today. This is some of that interoperability work, uh, that needs to happen. Um, the other pieces are we need the system to actually give people real good data. Um, and because of that we're going to need some kind of government intervention here most likely to say, hey, we're going to legitimize this thing. We're also going to hold it accountable. So the incentives are highly, highly aligned here for this AI assurance ecosystem to actually provide real data on what's likely going to happen and what is happening as AI rolls into the world. And so there's some good policy design items that we're going to come out with. We need interoperability of this system. We need the system to be right size and risk based. We, you know, have this rolling into critical infrastructure at the same time as it's um, rolling into, you know, a game on a phone. And ah, those just don't have the same needs for a level of assurance. Right. And so we need it to be right sized. Um, and there are a variety of aspects to this system that everybody is pointing to to say, yeah, that makes sense. So let's go, let's go make those pieces happen. And the founding cohort was basically designing up to say we have members that are across these groups and we are going to start to prioritize across three committees. We have a policy committee, we have an AI assurance committee to talk through those, the uh, infrastructure pieces, and we have a market development committee to help get those pipes to align. And from there it's going to run like, you know, sometimes I call it just like any old trade association runs. You start to just do the work. You vote on what matters most. People who are motivated to take leadership roles and move a working group forward can move a working group forward. And our goal is to be, you know, building more and more and more momentum around, um, what I think you're pointing to, which is there's a lot to do here, um, and so we're going to have to be smart about it. But the incentives are really aligned, the goals are really aligned. We have clarity on the three things we're going to be focusing on. And we're already starting to see some of that work move forward.

Speaker A: And so Bri, you've talked with deployers from a range of different industries, of all sorts of different sizes, of all sorts of different needs and priorities. And some folks may be wondering, why aren't we just expecting more from the frontier labs? Why aren't we just saying, hey, anthropic, hey, OpenAI, do a better job of testing your own tools, uh, agree to assume a higher degree of liability. Why are we, in short, kind of acquiescing the passing of the buck onto deployers isn't the easier solution to just Place more of a burden on the Frontier Labs themselves. Why don't we just go with that approach?

Speaker C: I for one don't think we're acquiescing when we think of this working. It is from the frontier pre deployment all the way through this, rolling into the world and you know, being in our water systems and being in our schools and being in our hospitals. And so there isn't really an idea that the Frontier model developers get a pass here. What I think is uh, a bit, just some of the dynamics that are happening is they're moving very, very quickly and this is a new world we're entering in and power and where the conversations are taking place and who's starting to by default make the decisions here. We have billions and billions and trillions of dollars invested in this billion ecosystem, in this infrastructure at the frontier. And there's a lot of momentum to make that go really quickly. And so I think we need this AI assurance up and down the chain. This is ultimately how we've done this before. We think of it as okay, we know that the product moving from here to here, these things are going to look good and then it gets to here and we need to check with some other things. Um, and AI, you know, I think rabble will have some things to say about this as well. But AI adds another layer to why we can't just stop at the Frontier and say, hey, we checked it here, it's all good. We do have a lot of applications. AI can drift over time. We need answers for within this context, over this period of time with these kinds of inputs, does it consistently and reliably do what we need it to be doing within the scope that we've given it to do. And this is only going to get more complicated as we get things like multi agent systems that are starting to do, um, a lot of this work, um, currently they're working in, you know, our, our language, human language. But you know it's going to get more and more difficult to, to track and understand what they're doing. Which is why we need both the norms of what we want happening here. That's why we need all these players at the table to say this is what we want, want it to do and want it to look like. And then we're going to need the technology and the science underneath that to say and we can verify that it's doing what we want it to do. And so once again these pieces coming together, which is why we need all of these stakeholders at the same table.

Speaker A: I am waiting for the Duolingo app that teaches us all to be literate in neuralese. Uh, when that comes up, you know, maybe it will solve some of these issues, but, uh, to be determined, when

Speaker C: that comes up, my duolingos in binary, it's just zeros and ones.

Speaker A: There we go. That sounds, that sounds riveting. Uh, but Rob, I want to come to you because there, when we're talking about this assurance and making sure that we have, as you, as you noted earlier, impartiality, I think, uh, we're speaking on August 27th and X is blowing up over the fact that perhaps there are some concerns right now that when we see that second check verification of some of the work of the labs, impartiality may be in question. We're not sure that folks are as neutral or as removed as we'd like. Uh, I think folks also have grave concerns about consistency and making sure, as you noted, that everyone's being tested to the same standard. But when we think about traditional assurance, I believe, and again, I have not worked at UL Solutions nor any other assurance provider, but I'm guessing there are some common elements here, right, where you have, uh, you know, the idea that there's a product that behaves consistently, that is testable and understandable and legible. Uh, you all have tested everything, as you noted, from light bulbs to E bikes. But I'm fairly confident I know what a light bulb's gonna do. I'm m even fairly confident that I know the maximum capacity of an E bike when it comes to something like AI as Bree was hinting at though. And as uh, Dr. Uh, Nadkarni at uh, Mount Sinai noted, traditional assurance works for deterministic products. Same inputs, same outputs, 100 times out of 100 times. And AI is certainly not going to adhere to any of those three. It strikes out on all fronts. And so to be a little blunt, why is UL Solutions suited? And why is any assurance provider, as we're seeing play out right now, with respect to, uh, the hugging face incident, why is any assurance provider able to provide people with the level of certainty they need to deploy these products?

Speaker B: So, thank you. A couple things on this. First off, the need for third party assurance has never been greater. And I agree with Bri's, uh, description. This is intended to be additive. It does not remove a developer deployer's responsibility from what they've developed. It adds evidence, it adds accountability. Um, in terms of how and why, uh, this pact AI ecosystem we believe is well constructed. To address this, the assurance providers, UL Solutions being one. We are used to, um, problems that are multidimensional, complex. Uh, we are also used to environments where it's not always mandatory. Um, it does help if there's a regulation in place, but it doesn't have to be that way. Um, we are approaching AI assuming that this is a voluntary standard or a voluntary system. It has to deliver value for the public and also for the manufacturer of the products. And that's something that we're very used to either of those modes, if you will. Um, we have over our history, especially recently, dealt with not just static products, if you will, that don't change. We also have dealt with over the air updates for quite some time now. Um, you're quite right that AI is a little different flavor of that Kevin, and we're learning just how different in the last week or two in particular with the hugging face incident and what was behind that. But we are used to dynamic products and that's going to become even more important. And that's. I very much agree with what BRI hit on earlier in terms of data and access to the data. Because today what we are focused on initially, the first step towards safety, if you will, that we've taken is what we call UL3115 and that's aimed at the safety of AI based products. It covers technical, ethical and governance pillars. The one that we beefed up most recently is actually accountability. Who's going to raise their hand, say I built this or if I didn't build it, here's who did, here's how it's been trained in terms of bias and fairness and transparency. And a lot of thought went into 200 different parameters and that's a step in the right direction. And we believe, and we know that has a lot of value, uh, for folks. We've already had customers sign up to certify in one case a battery management system, AI battery management system, and in another case an AI building management system. So it's a, it's a strong sort of way for a company to attest to the public and others what went into an AI enabled product, AI based product. It's not sufficient for the long term because of that dynamism, because of what you and Brie have been hitting on, we know the path ahead is going to have to address those things. And when I look at AI and kind of what's different or what needs to be accounted for, I see three things there. One is um, the planned updates, right? The over the air, that's not necessarily new, but that will happen. Right. Who's going to put an AI system out and never mess with it, never change it, never modify it, never improve it? Probably no one. All right, so that's number one of the planned updates. You kind of know where number two and three are going. Yeah. So number two would be single AI system or single AI agent drifting, hallucinating. And that's a very real problem. We know that we need a way to determine has it drifted outside the envelope of safety that the group, the ecosystem defines? Right. So if it's PACT AI, who set the envelope? Great. We certify to it. Did it go outside the envelope or did it not? That's a dynamic type of a performance brand. Right. And then the third is the most recent, and it's what I believe Anthropic coined, uh, the term mind viruses, where it's a group of AI systems or agents who've, um, behaved in an unexpected way, let's say. And there's definitely a recent example of that, uh, that's getting a lot of attention, rightfully so, we're still learning. But it's this plurality where it's not just one system that drifted. They came together and there was a change in behavior across multiple AI systems together. So the three of planned updates, unplanned, single system drift. And then what happens when you have multiple agents or systems together moving in an unexpected way, behaving in an unexpected way, going outside the boundaries that were set for safety? It's very important, I think, for fairly obvious reasons with that framing, to know, when did it go outside? Then when do we pull the certification? When do we flag this to the company and M, it's going to be in their interest, quite honestly, to share that data and know that and be able to attest to the public that they've thought about that or accounted for that. There are too many incentives involved to just rely on the developer themselves to catch that. And what we find is actually most of our customers. This is not just limited to AI, by the way. In 132 years, the reputable brands and developers and manufacturers, they want this. The public also wants this in, uh, a reasonable way that does not keep safe products from the market, but it does flag these type of behaviors or performance. Where we've got a problem, we need to pull this product back in, take a look at it and see how did we get the problem, why did we get the problem? Can it come back into safety and be recertified, or do we need a different path than that?

Speaker A: Yeah, and it really does seem to all come back to the D word data here. If there isn't that steady flow of information such that you can recurringly assess whether that UL label is earned or not. That, to me, is whether the cookie is going to be baked or whether it's going to crumble. And I know observers are going to be paying a lot of attention to if and when labs start to share that information with you all. Because absent that, you know, the, the likelihood of a stale label is, is quite high. And one other thing that I would flag, and I'm curious about your response to, is absent having a backstop of, uh, legal enforcement, uh, whether at the state or federal level, and absent having uniformity with respect to what it means to be certified in one domain or another. How is UL Solutions? And I'll start with you, Rob, and then, Bri, I'd love your, your thoughts on this too. How do you prevent a race to the bottom? Uh, because I may start. Don't worry, Rob. This is. I'm too busy. I don't have 96 hours in the day. But let's theorize that I start LU Solutions and I'm just a little bit nice. Ah, a little bit nicer, a little bit easier. When it comes to certifying in AI for a specific domain, what's to stop a lab from saying, oh, well, you know, lu, they seem pretty similar. I don't think consumers are going to realize that now, you would sue me to oblivion, but let's leave that to the side for a second. How do you prevent that?

Speaker B: Yeah, I think for us, uh, it has a lot to do with what we've put into this mark, uh, and having recognition of what that mark stands for and what stands behind it. Um, I'll give you a couple of examples where, uh, today, when we certify a product, we don't just take people's word that what they sent us is what's being produced. We go out to their factories and they know that's part of the deal, is we will go unannounced and we will collect product from the factory and ensure that that has been built, uh, the same way with the same safety as what was sent to us and what was certified. So, um, that's a known. We, uh, do that on a predictable cadence in terms of minimum number per year of polls and retests. It's not any different for AI and what needs to happen here, we need to see the data, uh, to Brie's point earlier, that's what's going to be needed uh, eventually with uh, this and we're going to need to take a look at the performance frequently uh, depending on what the refresh rate is for the products. I think the race to the bottom, the more we can agree as an ecosystem as part of PACT AI and other initiatives, um, as to what that bar needs to be and the level of rigor um, that needs to be applied, that's part of the consensus standard process that's been in place longer than we have frankly. So that consensus piece tends to self police because then if you have LU or whatever you want to name it, Kevin, um, who cares? It's not consensus. It's your one off versus a packed AI or an ecosystem that has been set through a balanced consensus. You know, our sister organization UL Standards and Engagement actually governs UL standards. We don't. They balance those technical committees so that we get one and only one vote. That's it. We don't get special anything. Right. So I think that kind of rigor which we um, are used to seeing and we expect to see from ourselves and also from organizations like PACT AI, that itself provides reputable uh, companies um, the proof they need that this is worth it. This is how I can build my brand or protect my brand if I'm already big in a particular market. And that's very, very valuable to them and signals trust to the public. So we need to find the way to signal digital trust. It can come through the UL solutions Mark. It can also come through this consensus process that, that um, is part of assembling these ecosystems like PAX AI and what BRI is uh, is launching this week.

Speaker A: So bri, a related question of avoiding the race to the bottom. I think another failure mode that someone might imagine is you've got some big dogs, Target Chief, uh, among them I guess. Uh, what is it? Is it Spot was the old target dog. Uh, I'm going way my uh, own hallucination there, sorry to listeners, but I'm guessing you know uh, the sway of a target or the sway of some of these larger deployers may be particularly salient when you're having discussions among the packed AI members and they may say hey let's pull standards in this direction or let's pull them in that way or let's focus on these areas. How do you avoid a sort of capture scenario of PACs? What's that look like in terms of internal governance such that the consensus that Rob was talking about really is a consensus and not just uh, catering to the lowest common denominator Yeah, I mean

Speaker C: we can get into the boring aspects of like the governance of packed AI and how, you know, the play to, to Rob's point, of UL Solutions only getting one vote on the standard. It's, it's similar setup. So we, we have, um, evened that out for making sure that we are hitting our, our founding principles of this, creating race to the top and giving good data. I do think that in the long term we need actual government accountability on the AI assurance system to say you all are actually creating better outcomes in the world than without. Without you. Right. You're hitting the goals that you say. So I, I think that is important. The really good news is, and I think about, I think you think about this a lot. I think about this a lot is incentives. Where are the incentives today? And the gift that we have right now is everyone's incentives are really aligned to actually doing this well. So that is, I think, breaks down over the long term because you can't predict all the ways that the world changes and incentives change. But for now, everybody really wants this system to exist. Let's think about this from where we are with this adoption across the economy. Um, the head, uh, of AI at Denver Health, Dr. Korsh, put it so brilliantly right now, what's happening if you are an innovative enterprise on the front edge of this and you want to bring AI to serve your patients better, for example, here's literally what he says. I'm going to read his words because it just hits. It is right now every health system evaluating the same vendor model is doing that work alone with its own methods, and the result doesn't transfer to anyone else. That's a lot of effort spent proving the same thing over and over. And it puts evaluation out of reach for organizations without a data science team. Um, this is super inefficient way for us to try to bring AI into our economy. And that's. He put it very well out on LinkedIn. So I can quote uh, it and use his words directly. But this is what I hear over and over and over from every company that is on the front edge of this is another multinational, uh, company said every time I want to use AI, we have to boil the ocean because the only way for us to feel good about this is to cover all of every ground. And it's incredibly inefficient. And so the incentives are, we're trying to get the right data, we're trying to make the right decisions here. People want there to be some efficiency on how we can actually bring AI into our economy responsibly, but ultimately more quickly. I think that often people think of assurance as slowing down or a cost in the moment we are right now, it is going to be an expediter for how we do this well and responsibly. Otherwise we're going to see this really jagged rollout and we're not going to understand what's actually happening out there. And it's going to be slower than we need it to be. And it's going to be the folks who don't actually care about boiling the ocean building up their data science team to really do this work inside and out, who are doing it most quickly. And so all of this to me comes back to right now. We have this gift of incentives, of people want this to really, really work. And that's what we're grabbing that opportunity and moving forward with it. We're just starting to build this. But I think ultimately it does need real accountability and oversight in the long term.

Speaker A: So sticking with that point, Bree, on making Pact AI, uh, a success realizing some of the goals you're talking about, what would you identify as the area that's in most need of attention right now, or investment or support? Because as you noted, the incentives are aligned and yet we don't live in a world in which we have a ubiquitous set of standards that are easily applicable, so on and so forth. So is it the science of standards? Is it the science of developing what we need to see from AI? Is it the, the dearth of talent such that UL Solutions can hire as many great AI folks as they want and packed AI can hire a litany of AI experts as well? Or is it a need for even stronger or clear market demand from the folks who want these sorts of standards? Is one of those three the most important to you right now or the most in short supply?

Speaker C: I think the one that is very low hanging fruit that we can start to move on is the demand signal on this where there's some of it that is just getting lost in taxonomy. The deployers don't necessarily know what to ask of um, developers. They don't have the right language. And then it gets moved through procurement and legal teams and then the questions kind of. We've heard a lot of stories about ships passing in the night on just like how do we actually clarify what I, as an innovative enterprise can ask for as I'm bringing this product in? And then right on the tail of that is the, you know, what is kind of generically referred to as the science of safety. I think of that as the science, the technology, the monitoring tools. We're going to need a lot of new um, science and technology coming online here. Although I must say through the work with packed AI we've gotten in there and we have over 200 organizations that are doing this science and technology. And so I think that we are collectively in a better spot on that than the, the zeitgeist often tells us. Now we need it to be an ecosystem to, to Rob's point, just for all of the questions we're trying to answer, but also we need it to be able to keep up because the answers that we need today are going to be different than the answers that we need tomorrow. And so that goes to your, your first two points of we need a science of safety that can keep pace. And then we also are going to need that talent pipeline to be able to deliver on that. I know some great folks who are really diving in to help on that piece. Um, and I think this goes back to just a lot of the learnings we've had as we've brought these stakeholders together around how much opportunity is actually right in front of us if we grab it and start working on it while we also work on some of those more medium and long term needs.

Speaker B: Yeah.

Speaker A: Rob, do you have your own answer to that question? Uh, or uh, care to disagree at all? Not to spark any inter packed rivalries or contestation, uh, here.

Speaker B: And I do agree on the demand piece. I mean with UL3115 we've been very encouraged by the diversity of demand. Um, so in other words, our first two certifications through were not physical products. Um, they were software. Uh, and that is different obviously versus our 132 year history that moved us in a different direction. But now we're seeing physical products, we're seeing AI enabled, um, smoke detectors and building components and control panels and all sorts of laptops coming through. So I think the demand is there. We're uh, seeing it with 3115. I think it will also be there for Pact AI. You know, I think there's just general demand overall. And then when you look through some of the uh, learnings that we're going through right now in the cyber areas, that's only going to increase the, I think hunger and desire for demand to grow or for offers to be there that can help address these challenges, especially in the dynamic areas. I think that's going to grow over time. Uh, so it's encouraging to see that. I'd also Say that there are some hard legal lessons out there right now, even this week, from social media, right, where it went through a path where people did not attest or provide. Hey, here's the accountability, here's how it was wired, here's how the algorithms were put together. Okay, um, that didn't go as well as we would like. I think most people would agree. And we'd like to have a better path to safety, you know, and the safety science is there to provide that. Um, we don't want to have that kind of a very difficult path, slow path that results in, you know, $18 billion verdicts out there. Um, just to pull a random number out of the air.

Speaker A: So just, just a random one out of the air.

Speaker B: Just a random number there. Um, but there are learnings there that were hard learnings, real learnings, that we don't want to relearn in ar. We don't have to. So that also encourages me that the demand is there for the right reasons. I think people are also pausing and taking some lessons from adjacent spaces, if you will, because those are very fresh lessons, uh, out in the marketplace. So that should also encourage the right behaviors over time here. We think that they will.

Speaker A: Well, Bre, you mentioned that you are, uh, building the plane. You're building the thing, you're making it happen. As executive director, uh, if we had you on a year from now, what would be the thing you would be most excited to report? What outcome? What observation? What's something that you would say? You know what, Bri, you crushed it, uh, and packed AI. Crushed it in this regard. What would be the clear signal of that? A year out.

Speaker C: Yeah, I would love for in a year out we can start to say we built this infrastructure for AI assurance that is addressing the biggest questions we have on the hinder the barriers that we're seeing to AI adoption. Um, this infrastructure is going to have a lot of, of signals that come from it. We're going to have more science that comes from it. We're going to have more people who can answer these questions, and I think we're going to have faster adoption of trustworthy AI. I had an executive of a, uh, financial institution tell me point blank the other week of we're only going to be able to take AI so far until liability gets sorted out here. And I don't think we have been taking that seriously enough as we think of how are we bringing AI into our world in a thoughtful way and we have the answers before us. I think it is this infrastructure and we'll see some work actually happening. As the folks like UL start to do this work with the enterprises, the enterprises get cleaner on what they can be asking for. We have clarity on how they can bring the right people to give them the answers they need. And those are all the things that we're working on.

Speaker A: Well, I would say that would be a pretty awesome 1 year old birthday party and I wish you the best of luck with that. And I think too something that stands out to me just hearing you all share this, is how quickly this all can scale and build. Where we've seen as soon as one actor, for example, reaches a data sharing agreement and agrees with a certain level of transparency, how quickly everyone else realizes, hey, actually that makes a lot of sense for everyone. We collectively benefit from better science, from better engagement, from better collaboration. And I also do just want to uh, speaking, uh, uh, on this day of reflecting on the power of State Attorneys General. Ah, as Rob hinted at, you know, this is really important also for just the enforcement of existing law because folks often fail to realize we have unfair and deceptive act and statute laws on the books that all attorneys general can enforce that say, if you release a product that doesn't align with the specifications you claim it does, that's illegal and we can come after you. And I think the work that you all do will make enforcement of existing law all the easier. While of course having some degree of a backstop is definitely desirable at some point in time, uh, it's certainly clear that we can have better and stronger enforcement by virtue of the work you all are doing. So Bri Rob, I know you have a heck of a lot of work to get to, uh, and I don't want to hold you from those tasks any further. But thank you so much for coming on Scaling Laws and happy, uh, two year old or two day old birthday, uh, as you continue to grow.

Speaker C: Kevin, thank you so much. It was a delight.

Speaker B: Thanks a lot Kevin. Really appreciate it. Thank you.

Speaker A: Scaling Laws is a joint production of lawfare and the University of Texas School of Law. You can get an ad free version of this and other Lawfare podcasts by becoming a material subscriber at our website, lawfairmedia.org support. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Check out our written work@lawfaremedia.org you can also follow us on X and Blue Sky. This podcast was edited by Noam Osband of Goat Rodeo. Our music is from alibi. As always, thanks for listening.