Agentic AI meets Suptech at the National Bank of Georgia

Speaker A: Welcome to the new series of Tech Talk with me, Yasha Popovic, Associate Editor at Central Banking. The national bank of Georgia has for a long time pursued an ambitious fintech strategy and digital transformation program. Now it is experimenting with AI internally with applications ranging from improving operational efficiency to mapping complex entity relationships and ownership networks at the same time as managing the cybersecurity and data confidentiality risks of using third party large language models at the central bank AI. Uh engineer Sandro Gogolasse and Velam Abanoze who is head of the fintech and subtech development department explain how the team audits AI models being deployed in the financial system. Welcome both to the podcast.

Speaker B: Thank you very much. Pleasure to be here. Hi.

Speaker A: Hello Valim. To set the scene could you give us an overview of how the national bank of Georgia is approaching AI adoption both internally and externally?

Speaker B: Basically we focus on three things when it comes to AI adoption, competition and inclusion, explainability and uh AI in subtech. When it comes to competition and inclusion we focus on external market developments in a quite concentrated market where 80% of market share belongs to two major quite agile Georgian banks. Those have basically most of R and D budget uh dedicated to AI. We want to make sure that uh AI adoption and the benefits of AI are inclusive and distributed equally in the financial sector. Key objective of fintech development strategy is uh to make sure we do everything for entry barriers, facilitation for new entrants and uh to the concentrating basically market AI sandbox is a place where we connect on one side uh the startups, AI startups those build their value proposition on uh democratization trend of OpenAI and on the other side the smaller players from financial markets those don't yet have uh enough R and D resources. Also like to mention about digital public infrastructure and making sure that digital public infrastructure is ready for AI agentic commerce future in particular I mean by that uh open banking payments infrastructure. Open banking currently is uh one of the most promising alternative infrastructure used by newcomers uh of the market in the Georgian market uh because uh every day you see news that uh big payment players are uh announcing new initiatives regarding agent E commerce and uh AI payments. There is a risk that uh if only uh few players are ready to absorb this innovation they actually would widen the concentration risks. Second um initiative we are um focusing on uh external is related to explainability. For few years now we are uh quite actively exploring models uh coming uh from financial sector uh alternative data based um models uh related to income estimation and credit scoring and more and more those models are Coming um generative AI or AI uh input uh in those models and we want to make sure that uh those models are explained and uh we don't have uh much of black boxes here. We are currently exploring the framework of the risk appetite and thresholds for explainability. Uh but uh in general I think uh um with it will be quite individual and um use case driven. Third uh our internal experimentation regarding um AI uh adoption is uh mainly focused AI in Subtek but sometimes we go beyond Subtek taking care of overall operational efficiency within the central bank. So far uh I would characterize that uh for past two years we had quite bottom to up approach when it comes to AI uh usage in uh central bank environment. Basically what we did uh we interacted with all 26 departments within the central bank and um asked uh the most relevant use cases for them to bring generative AI uh as an enabler for greater automation. We are in process of uh transitioning from bottom to up approach towards more strategic uh top down roadmap when we have uh AI platform in a uh place which is uh reliable which is uh under our control and uh secure enough to enrich with the sensitive uh MVG data. So the success of uh your AI model and uh added value pretty much depends how uh free you are with the enrichment of the model with your own data. Huh. We have ah similar uh challenges I would say which uh most uh central bank experience in this scaling uh exercise. I would say those challenges are related to cloud adoption uh cloud adoption which might be located outside of the country third party dependency which is similar to other central banks. So we have our own context as well. Georgian language is um pretty low resource language. Um not many people around the world unfortunately use Georgian language and uh you got uh less data overall in Georgian Uh so these are limitations I would say general and uh specific limitations. We are currently trying to respond with some creative solutions now in short term uh possibly it could be enterprise account solutions in uh midterm. We are also exploring private cloud Solutions and uh OpenAI uh uh Solutions uh in the private cloud and final destination I would say it's uh sovereign LLM model which would be more long term

Speaker A: with regard to third party dependency. I just wanted to ask with everything that we've seen with Anthropic for example recently where foreign users have been locked out how are you addressing those kind of geopolitical risks associated with using AI models?

Speaker B: Uh we try to diversify as much as possible not to be dependent on one provider only. We are exploring in everyday uh manner several uh providers and solutions. So in long term uh, uh our solution would uh be uh defining our own LLM model. But so far we try to navigate between distributing um and uh having connection with uh several OpenAI services rather than to stick to only one.

Speaker A: I also wanted to ask about the costs. How are you factoring in the costs of using tokens which seem to be growing exponentially.

Speaker B: Our um costs are um quite um limited uh with the minimum resources we try to lend as much as possible. But we have some calculations in place already for midterm perspective. When I mentioned this uh enterprise account solution or uh private cloud providers, OpenAI solutions they don't seem to be quite astronomic. So far we have limited budget but uh, I guess uh this is uh quite strategic priority for the country and from state institutions I would say we are the, we are the one of the pioneers if not only which is experimenting uh AI usage uh for uh state institution. So ah, I guess uh this budget cost benefits goes beyond even central bank mandates and we try our best to have a life um in government that this should be sufficiently supported because of its importance.

Speaker C: Also you have to be clever in usage, right? So you don't need very big models and thinking models for simple tasks, right? So in our AI tools we manage it great. If we need a task that needs a lot of thinking we give a higher thinking uh model which spends a lot of cost uh tokens. And if it's just a simple for example sentiment classification you don't need such a clever model. So it's manageable. And what we have reached so so far is we are very proud that what we built so far is under limited budget.

Speaker A: And it's interesting that as the central bank essentially you're setting a blueprint for how AI might be adopted at uh other state organizations. Sandro, I wanted to turn to you and ask you a little bit more about the agenda AI being used at the central bank. As I understand it, three key use cases that uh, the MBG started out with was experimenting with a new sentiment analyzer, web data extractor and AI agent that maps business ownership. So could you tell us how that works um, and helps to uncover ultimate beneficial owners?

Speaker C: Yes, absolutely. The three use cases that you mentioned were among the first AI solutions that we developed. But since then our work has evolved significantly. We have developed much more agentic AI workflows and importantly we have created an in house AI platform with the national bank of Georgia which is basically a web based interface that centralizes all of the AI Tools and AI capabilities that we have developed internally. The platform was in the testing phase for several months and now it has went live, uh, it's been live for several months now and it's being actively used by our employees on a daily basis. So one of the tools that you mentioned is what we call the Registry Agent which is an agent that identifies borrower related parties. So to explain the idea, traditionally uh, additionally supervisors had to manually navigate uh and search business registry websites. They had to download documents, extract relevant information and construct uh ownership structures themselves. So the Registry agents basically automates this process completely. A supervisor can simply ask uh, what they want in natural language and the agent does all the work on its own. It navigates the Georgian business registry websites, uh, it extracts information, it identifies the links between borrowers and shows a nice visual on the sheet map. So instead of, instead of spending time, instead of a supervisor spending time collecting information and uh, on repetitive task and manual task they can focus more on analyzing risks and making decisions. So Registry Agent is uh only one tool that we have on the AI platform. But um, we have 10 more tools available currently and 10 more tools are in the development currently. So another example just to give um uh, the scope and ideas is uh, another agentic tool is our Text to SQL Agent which basically allows non technical employees to uh, retrieve information from databases using natural language instead of writing SQL queries. So this uh tool, this Text to SQL Agent significantly lowers the uh technical barrier for our employees to access data. We have also developed several uh, specialized chatbots for different domains such as a regulation chatbot which knows all of the regulations, NPG related regulations, also a chatbot for IML team, for fintech team and etc. So these are not simple work based chatbots, they act more in agentic manner, they retrieve information from multiple sources, combine it, ask for clarification if it's needed, et cetera. So overall our vision is not to uh build individual AI tools for different departments. The idea is to uh, create an institutional AI ecosystem where employees can access a wide range of tools. And that's the idea of the AI platform.

Speaker A: So Varnam also mentioned that you're working towards enabling staff to use confidential and sensitive data in the AI tools that you're developing. So could you speak a little bit to the cybersecurity and data risks that you've identified along this journey? Um, and also um, I'm interested in how that's come up when you're using AI to write code. Um, and what solutions are you currently working on to um, enable staff to

Speaker C: use confidential data and it uh, thank you. Yes. We quickly recognize that Dao AI brings a lot of productivity benefits. It introduces new risks particularly around uh, data confidentiality and cybersecurity. So we are a central bank, so we are naturally a conservative institution. So uh, the data confidentiality and cybersecurity methods were the most challenging for us to address. So because of this at the beginning we started with use cases involving only public information. So we didn't want our AI journey and AI experimentation to be delayed because of confidential matters. So that's why we started with um, use cases involving only public information. This allowed us to build internal expertise to create an AI ready culture in the organization and demonstrate uh, clear benefits for our employees and show initial success stories in the organization. So uh, now um, you also mentioned using AI for coding. So yes, uh, it's an important um, matter that is not often talked about. Um, we do use AI assistance for coding but you have to be careful in that process. So AI generated code contains vulnerabilities, uh, often a lot of errors and it might cause data leakage, you might leak some confidential testing for information to the AI providers. For that reason we treat AI generated code more like a starting point, more like a template rather than a final product. So we always review uh, the code, we always fill in the missing gaps, we always test the code before we put it in production. That's how we address the uh, AI used in software development. But um, uh, how we are moving from public information use cases to confidential. Like if we uh, the, this constraint, the confidential constraint limits a lot of use cases. Right, right. So we are actively exploring how to enable the use of AI with confidential information. One possible alternative that Mr. Wallam also mentioned is using open source models.

Speaker B: Right.

Speaker C: But we have uh, but there's also a challenge for us uh, which is unique for our central bank. So we are Georgian central bank, we operate in Georgian. So uh, using open source models directly is not a solution for us because we ah face a trade off. So either uh, data confidentiality, data safety if you use open source models or uh, data quality generated quality. So if you use open source model they generally do not work in Georgian because Georgian is a low resource, uh language. So we have a phase of trade off that we currently face and we are exploring ways how to find the golden uh, balance with minimal resources to get maximum benefits. Mr. Wallow mentioned few solutions but we are currently uh, exploring it and hopefully we'll have significant progress in near future.

Speaker A: The risk and supervision departments were Most open to using AI at the beginning. So how um, is that being combined with Soup Tech?

Speaker C: The connection between AI and souptech is very natural because supervisors deal with um, enormous volumes of information, including documents, including regulations, including databases, so both structured and unstructured data. So when we first um, were exploring how we could make their life easier, AI was a natural solution. Right? AI can handle both of these, um, kinds of information. Uh, however, um, one thing to notice is that successful adoption was not only because, uh, the technology that AI brought, it was also about people and culture. So we have invested heavily in AI literacy initiatives. Internal, including internal workshops, presentations, practical training sessions to help employees, um, understand both the opportunities and limitations of AI. And this has caused a significant cultural transformation within the organization. Employees started using AI in their daily work, they started seeing real benefits and adoption became organic. So our objective again is not to replace supervisors, but it is to help them and equip them with better tools so they can perform their jobs more effectively and focus on higher value activities.

Speaker A: So I wanted to um, delve a little bit deeper into the external AI use cases that Varlam sort of broadly set out for us at the beginning. Um, the national bank of Georgia runs both a regulatory sandbox where it has a role in validating all of the machine learning, AI and statistical models of commercial bank. So could you tell us a little bit more about that?

Speaker C: Yes. We often describe our AI activities, uh, as having two dimensions, our internal dimension and our external dimension. The internal dimension focuses on improving operational efficiency. So all of the AI tools, the AI platform that we talked so far was our internal dimension. Now our external dimension relates to our role as a supervisor, uh, in the sector. So we have a model risk management framework, uh, and under our model, um, risk management framework, certain statistical machine learning and AI models used by commercial banks must receive approval from the national bank before being deployed into production. These are typically models that are considered material, high risk or innovative. In this context, we act as independent model validators. We have a, uh, dedicated model risk management team. Team with specialists coming from different domains such as finance, economics, statistics, econometrics, machine learning, AI, etc. And we validate the models before they are being used in production. So what does this validation process mean? We challenge and assess every important aspect of the model, including its assumptions, methodology, algorithm, purpose of use, performance, governance, explainability and etc. We focus on explainability a lot of uh, because most of the models that are being used and are being developed are mostly black box. So there is a neural networks or gen AI or et cetera. So especially these models are being used for financial purposes, for credit acquisition purposes or et cetera. So it's important to understand how a model um, came to a, came to an output. So it's important to understand the explainability part of the model. And that's what we focus.

Speaker A: How common is it for central banks to order AI in the financial system in this way? Because the level of expertise is incredibly high to be able to do that.

Speaker C: Uh, yes, correct. It is not very common. Many central banks and regulators around the world are currently exploring AI governance and supervisory approaches. But very few have established hands on model validation frameworks. And the national bank of Georgia takes a relatively active approach in this area. Uh, so that said, we do not validate all of the models used by financial institutions. We focus primarily on models that are material and potentially high risks. And examples include credit scoring models, probability of default models, income estimation models, and other systems that can significantly influence lending decisions or risk management practices. We take this approach because these models can have significant and meaningful consequences. Right. If a model systematically overestimates income or underestimates risk or unintentionally discriminates against certain, uh, customer group, it can create risk not only for the commercial bank but for the financial system as a whole. So this is why we believe that model governance and validation should be essential as AI adoption continues to grow.

Speaker A: Got it. So it sounds like um, the sort of, the micro decisions, how those models can um, eventually impact sort of, yeah, the macro environment and financial stability, um, comes down to looking at code essentially line by line. So that's fascinating. Then you also have an AI sandbox. So I just wanted to um, ask you, sort of like explain I guess, the difference um, between the regulatory sandbox and how is that being used?

Speaker C: Uh, yes, um, before we move to that we need to uh, say that m. We need to understand that our fintech and software development department is not a uh, traditional central banking department. We act more exchange facilitators in the bank and in the industry. And this is the philosophy that shapes our approach towards, towards AI. So different jurisdictions and different central banks have uh, different approaches to AI. Some focus on restrictions and risk mitigation, but our approach is more encouraging and supporting. And this is the, and the AI sandbox is a tool for that. So um, rather than uh, limiting the use of AI, we actively try to support and accelerate responsible use of AI. The AI sandbox basically does this. It connects AI technology providers with financial institutions and uh, create a supervised environment where they can experiment uh, collaborate and build solutions together. So the AI sandbox allows new ideas to be tested before deployment. Our goal is not to, um, limit the innovation, but it's to create the conditions for innovation to flourish. So we encourage innovation, we try to increase competition, um, but do it in a responsible and safe way.

Speaker A: Great. Thank you so much. Are there any other sort of, uh, final thoughts, um, you'd like to share with other central bankers on the AI journey? Maybe how you think AI is evolving? It's moving so quickly. So, um, just. Yeah. With looking forward, what are your thoughts with regard to the pace at which AI is developing and how central banks can keep up with that?

Speaker B: For me, it's a marathon, not a sprint. It's a journey. And, uh, if you consider marathon like 26 miles, I would say somewhere in the fifth mile. And we have a very, very interesting journey ahead, but we have strategy.

Speaker A: Thank you so much both for participating in the podcast, and I hope to welcome you both back soon.

Speaker B: Thank you very much.

Speaker C: Thank you, Asia.