Can responsible AI beat hallucinations?

Speaker A: Businesses are more eager than ever to implement AI in their workforce. But ambition doesn't always translate into success.

Speaker B: Hallucinations are an eternal problem in generative AI in particular. And while it's true that large language models require vast amounts of data, the quality of that information will affect the quality of the output.

Speaker A: What can businesses do to ensure they're using AI both effectively and responsibly?

Speaker B: Hi, I'm Ross Kelly.

Speaker A: And I'm Jade McCallion.

Speaker B: And this is the IT Pro Podcast.

Speaker A: Joining us today to discuss all of this and more is Amanda Stent, Head of AI Strategy and Research in the office of the CTO at Bloomberg. Amanda, welcome to the show.

Speaker C: Thank you. Looking forward to talking with both of you.

Speaker A: So, first of all, could you tell me what do we mean when we talk about AI? Because it's a term that's been around for quite a long time, but is starting to pick up a bit more or maybe even change definition.

Speaker C: That is such a great question. We call AI a suitcase term. You can pack any amount of meaning into it. Traditionally, AI referred to the use of computers to do things that we consider intelligent in humans. Perceiving the world, taking action in the world, using language, understanding language, reasoning and learning. However, there's another definition that I like better, which is computers doing things that uh, you cannot solve with an algorithm. So computers solving problems that you can't solve with an algorithm, which is a step by step procedure for completing a problem that is guaranteed to finish in a finite amount of time and give you the right answer. And a common problem that we don't have an algorithmic solution for is shopping. You have a shopping list, you have a budget, you have an amount of time that turns out to be a problem that we don't have an algorithmic solution for. Which is why it's so hard for all of us.

Speaker B: Amanda, we've heard a lot of discussions about responsible AI in recent years. Recent research from Gartner shows that three in four organizations will include some form of built in tools, some form specifically focused on responsible AI. Can you give us an insight into, you know, how this works in practice at an organization? You know how our organizations deploying AI in a safe manner?

Speaker C: Sure. You've used three words there that have slightly different meanings, responsible, uh, and safe. And then we would use the word trustworthy. So our approach to responsible AI involves developing AI responsibly. That includes choosing the right problems, choosing the right technology, choosing the right content, making sure our solutions are accurate, and providing what we call transparency or transparent attribution we always want to drive the user to the source information or data or analytic. So responsible AI is really choosing the right problems and then doing it responsibly is the other parts of it. But we also talk about implementation of responsible AI, and that includes, uh, our guardrails that we put around our AI solutions, the testing that we do on our AI solutions, and really, again, ensuring that humans are in charge by providing transparent attribution for AI solutions.

Speaker A: Is this because I don't think there can be anybody at all listening to this podcast, whether they're an IT decision maker or have stumbled upon IT by accident, who doesn't know about the issue of hallucinations? Is responsible AI trustworthy AI a way to get around these false but confident answers that generative, uh, AI in particular can put out?

Speaker C: Sure. So let me say a little bit about Bloomberg first, because a lot of people hear the word Bloomberg and they think, oh, uh, a television station or a radio station in the US or around the world, or website maybe, and that's true. We have Bloomberg News and Bloomberg Media. We make original content and we distribute it to decision makers around the world. But what we also do is sell data and analytics and this special thing we call the terminal, which is the original Bloomberg product and still a big part of what people in financial services think of when they hear Bloomberg. The terminal gives users access to more than 17,000 news providers, not just Bloomberg News, more than a thousand research brokers, more than 400 million documents from companies themselves, and billions and billions and billions of ticks. That's prices every day for equities, bonds, commodities, derivatives, any kind of financial instrument you can think of. So in that context, accuracy is paramount. If we hallucinate or do something that's otherwise incorrect, markets may move, and that might be bad. So hallucination is a potential effect of using generative AI. And, um, you use the word confident, which I find interesting. It's so easy for us to anthropomorphize these models, but they don't have feelings, they don't have confidence. The probabilities may be high. The probabilities may be high, leading to an output. So what we do to protect against hallucinations is to check that every statement that's being produced by one of our generative AI systems can be traced back to either a statement in a source document or to a, um, data element or set of data elements in a database or the result of a calculation. And we use agentic AI to do all of that. So we have tools and agents that specialize in what it means to look at research content or company content or news content or our structured data or calculators that we have or users data uh, like their list of securities that they're interested in. And solutions that are out in the world for addressing this include RAG retrieval, augmented generation which you might have heard of. And the idea there is you give the model access to snippets from the content and in this way it will uh, be more likely to write things from the snippets uh, than from what's in the model because the snippets are right there in the context window. And that works great until you overshoot the context window. So we have a paper from last year showing that RAG systems are not necessarily safer in the sense of less likely to hallucinate than non RAG systems. Now today there's a lot of advanced techniques that go beyond RAG for managing the context and there are equivalent solutions for accessing structured data. That doesn't mean that a generative AI system will never hallucinate. But by using techniques like RAG and uh, similar techniques for structured data, by doing checks for attribution, by providing transparency with, we minimize the chances overall. The reason I say it's never guaranteed, uh, think about the word not the word not teeny tiny little word completely changes the meaning of a sentence. It's not that it's impossible, but we work really hard to minimize the chances and the utility of these solutions. If you think about balancing risk and reward, the utility of these solutions is so great that it really is important to provide them to our users to improve their efficiency and effectiveness.

Speaker A: It's funny you mentioned the word not because it does kind of. You've put me in the mind of for example if it was warm outside and somebody said is it hot? And I'll be like well it's not hot, but it's not as hot as we have had. And a human being would understand exactly what I mean considering it's been kind of 38 degrees. But I guess a machine might struggle with that context because it's non standard grammar as well. Mhm. I want to roll back a little bit on what you said, so I was going to ask how Bloomberg is using AI. I think you've done a really good job of not just giving us an overview but actually some in depth. You mentioned cross checking that the references are right and the data ah, or the output is accurate based on the data that you know to be solid. Is it that you have AI cross checking other AIs work or how much do you have a human involved in all of this?

Speaker C: So to answer that, let me go back in time a little bit. Bloomberg has been using AI since about 2009. The first areas where we used AI were to enrich content. So you can think of adding topic tags or sentiment or entities like uh, the companies to news stories and research so you can connect unstructured data really to a rich ontology of what financial world looks like and also to extract information from content. So we have a long running tradition of extracting company financials from company documents and populating them into databases very quickly and very accurately so the clients don't have to read the document at all, can just go straight to the database. So beyond extraction and enrichment, then with the early days of Gen AI, we started to look at summarization. So using AI to summarize content. And now with interactive gen AI and agentic AI, we do have AI systems that check the output of other AI systems. So we have uh, AI systems that check that every sentence it can be attributed not just to a document but to a place in a document. We have AI systems that check when we are extracting from structured data that the securities that we are referring to and then we call them fields, things like price or the last price of yesterday, that those are accurate before we go and fetch from a database. And then we have guardrails that we run on every input to an output from a gen AI system. And the guardrails, we have a white paper about that too from last year. The guardrails are specific to financial services. For example, we don't want users to be injecting code into our systems. That's a generic guardrail. And we also are in the business of offering financial information but not financial advice. So if you say what's a buy case for IBM? We should give it to you. If you say, should I buy IBM? We should say, nope, uh, I can't answer that question because that's not something we're in the business of doing. And that's a finance specific guardrail. So we run these guardrails which are also AI systems on the input to our AI systems and the output from our AI systems to make sure that the inputs and the outputs and in some cases the agentic conversation in the middle are being done responsibly.

Speaker B: Now Amanda, just to touch on what you mentioned there around having agents monitoring other AI activities there Gartner discussed the concept of guardian agents last year pretty much exactly how you described it. Do you envisage us being, you know, a larger use case going forward where you have sprawling estates of agents, humans can't keep tabs on all of them, all of their activities. Do you anticipate this? You know, working at Bloomberg and in the financial services sector, you know, know, at large?

Speaker C: Uh, I think it's already the case that if you look at your computer, there are many algorithms running that you can't keep track of and, and can't trace. And a lot of those algorithms are there to ensure that your system is secure. So they're there to guardrail your system. Uh, there are systems, uh, algorithmic and AI that run on your Internet traffic to ensure that it's secure, to ensure that you're getting things that are similar to things you've clicked on before. Uh, that's an example of an AI system, pregen AI AI system. So yes, I think we will have algorithms and agents working together. We already have algorithms and agents working together to help humans work with each other and with computer systems effectively and efficiently. That's very much the case.

Speaker A: A lot of the focus of this conversation has been the fact that um, Bloomberg provides intelligence to the financial services industry. Do you have any insight on how this sector itself is using AI?

Speaker C: Traditional machine learning has been used in finance for a very long time. So you can think about machine learning models for uh, trading, portfolio back testing, event studies, a lot of quantity type things, and then a lot of rpa, robotic process automation. So using machine learning and AI techniques for extraction and enrichment, really to convert unstructured content into structured content, which is a signal that you can trade off. And then AI has also been used for compliance and to monitor communications, for example, in financial services, as it has been used in many other industries. What's different today is the ways in which people are interacting with generative AI beyond the trader. So the traditional, traditionally I would say machine learning, you would think of it as applied to trading workflows and compliance workflows. But today you have your portfolio managers and your analysts who are also really using AI. And that's because the thing they operate over is often text. And with generative AI it makes it easier. So we have analysts using our AI systems to uh, one of them said days and days and days, save days and days and days of time to help them write their research reports more quickly, more easily, to cover more companies, to, to understand the context of a company with its sector and its industry, to write on demand reports for clients. Instead of uh, a monthly or a quarterly report. We have Portfolio managers doing the same thing, writing on demand reports for clients using AI instead of quarterly reports. So these are some of the new ways in which people are using AI. But to me, traditionally it was about efficiency and signal generation, and today I think it's about effectiveness. So it's helping people become more effective in how they use AI. We've done some recent surveys of our clients that show that they are really making that shift from efficiency to effectiveness.

Speaker A: I wonder, do you think that possibly financial services has been more open to AI or being perhaps a little bit more AI forward? When we're talking about this new generation of AI, Given that, it makes me think of, um, the rapid automated transactions which happen much faster than a human could ever do them. And so has that groundwork been laid to make it a more accepting sector or. It's also a very regulated sector. So has there been, I guess, any balancing there?

Speaker C: You're talking about algorithmic trading.

Speaker A: Thank you.

Speaker C: Yes, I would say. Yeah, no, absolutely. For me, of course, because I work at Bloomberg, the history of technology and finance is very intertwined with the history of Bloomberg. So this might be a somewhat Bloomberg oriented perspective. But I will tell you both a story. My granddad worked for Barclays his whole career. He ended his career on this site, bringing the early computers to Barclays Bank. They called it mechanization. I think anytime you're dealing with a information, uh, rich space, which financial services definitely is, there is a, uh, compelling need and motivation for people to want to bring more structure, whether that's early computers or telecommunications. In the early days of Bloomberg, when it was literally going in and wiring up people's offices. So instead of having someone manually carry things around, information could just go down the telephone wires to. Now with AI in finance, people are driven and grounded and this is a good thing, in my opinion. Driven and grounded by making markets more efficient and making profits. So it's kind of a uniquely obvious and uniquely targeted set of, uh, trade offs and decision points to make.

Speaker B: Amanda, uh, you touched there on the long standing innovation history within financial services and how this sort of ties into the current generation of AI, you know, in terms of integration and implementation of the technology on the front lines, so to speak. You know, are there any areas that are challenging, proving troublesome for IT leaders? Jane mentioned, you know, regulation and compliance. I imagine that's a consideration. But in, uh, terms of the scale of infrastructure that a lot of these organizations have as well, what are the big challenges here?

Speaker C: Yes, so accuracy is definitely a predominant challenge, but so is speed. When you think about generative, uh, AI versus previous generations of AI, it's good, but it's slow. And um, speeding it up to the speed of the financial markets while ensuring that it stays good is a challenge that is being faced across the industry. Thinking about how to ensure that the middle front and back office can all continue to communicate when they're all using AI is another area of focus. So making sure that information that not just analysts can get the information they need, but trades can then happen accordingly and trades can get booked and that we can satisfy international uh, regulation about how accurately those trades get booked and the timeliness with which they get booked. It really requires multiple generations of AI technology to make that work. And then of course what has happened over the last six years really along with, but starting before generative AI, a lot of financial services companies are getting more comfortable with information in the cloud as opposed to information in their data centers. Because today financial services around the world is connected in a way that it wasn't historically, just incredibly connected. Milliseconds, sub millisecond connected.

Speaker B: Now in terms of regulation, you mentioned cross border considerations there. For organizations in the financial services industry, regulation seems to be almost like a dirty word when it comes to AI in some circles. You know, is this a barrier to innovation when it comes to AI development? Or you know, can it actually be helpful and help spur on progress?

Speaker C: So one of the things that I really like about the approach that the UK has taken is that it's a principles based, technology agnostic approach. What that means is instead of trying to regulate the implementation in the uk, we have decided that there are certain principles that apply across industries and regardless of how you do it, and then leave it to the regulators in each industry to decide what matters, what are the consequences of those principles for that industry. So in finance we have organizations like the fca, we also have organizations like the bank of England, we have other non governmental and quasi, uh, governmental organizations that are really thinking through how those principles apply in financial services. And what it comes down to is less about whether you're doing it with generative AI or traditional AI or algorithms or paper books, and more about the data and the output, which is, I think, right, you want to make sure that data is protected and you want to make sure that outputs are being done in a way that's fair and that there is transparency and that's regardless of how it's implemented. Also, it's great that in the UK there are things like the FCA testing framework, so private enterprises can get together with regulators and really experiment and validate things in a kind of low risk way. So the way that it's happening here is actually really positive and I would argue a model when we think about

Speaker A: the future of responsible AI. Amanda, uh, whether that's within Bloomberg or financial services or even beyond, we. What do you foresee?

Speaker C: I think we're going to have a lot more exploration with agentic AI this year going into next year. At the same time, we're starting to see conversations about the economics of AI, including conversations about in some cases, is the AI more expensive than having a human do it? Interesting. Um, so I think we're going to see an increase focused on cost efficiencies. At the same time, as we see increased exploration and really productionization of agentic AI across the space, across financial services, there is a lot of conversation about what this means for people's jobs. We are actually seeing people become more effective and more productive. And what that means is that they're scaling. So a company that used to, or a part of a company that used to cover just one sector can cover multiple sectors. Uh, it just means that people can really focus at a higher level and, and do more. And that's also a, a positive and exciting thing. We're seeing a lot of organizations, including Bloomberg in some areas think about how do you train and educate people so that they are effective and productive users of AI? How do you help them think through what their work looks like now and how they can really take advantage of the power that's there, but take it, take that advantage responsibly. And that can include things like an AI can take your input to an AI and rewrite it so that it's better. So coming back to something that Ross asked about earlier, right? You don't have to just sit there and think today, what is the best prompt for this AI? No, the AI can help you really turn it into something that's good. At the same time, we're only today really focused on a very small area of, uh, you asked about what AI means. We're really focused on understanding language, producing language, and a little bit of what they call reasoning, not a whole lot of what they call reasoning. When you think about all of our five senses, that's a teeny tiny fraction of the signal that we take in and the ways that we act on the world. So video is another vastly underexplored area. All those video archives that uh, media organizations like Bloomberg are sitting on, all the video that people generate as they go around in the world and then some other areas. What does it mean when you're reasoning, when you're really reasoning over multimodal output? That's video language, whether spoken or written, and potentially other senses that we might have or that other species might have. So I think there's going to be a lot to do there, plus linking all that back to what we just actually know about the world. So at Bloomberg, we link a lot of our AI back to our ontology, our understanding of companies and how they're priced in medicine. They're doing the same. It can make the AI more efficient. It can make the AI more accurate. It can make the AI more cost effective. It sometimes is called neurosymbolic AI. You might have heard that term. So I think all of these things are coming and it's just a really exciting time to be maybe in the middle to two thirds of the way through this transition.

Speaker B: Unfortunately, that's all we have time for this episode. But Manda, uh, thank you so much for joining us.

Speaker C: Thank you. It's been a really interesting conversation.

Speaker A: You can find links to everything we've spoken about today in the show Notes and even more on our website@itpro.com you

Speaker B: can also follow us on LinkedIn and YouTube as well as subscribe to our daily newsletter.

Speaker A: Don't forget to subscribe to the IT Pro podcast wherever you listen to podcasts. And if you like what you hear, leave us a rating and a positive review.

Speaker B: We'll be back next week with more from the world of it. Until then, goodbye.

Speaker C: Goodbye, Sam.