Agentic AI in finance: automating P2P inquiries (case study)

Speaker A: Foreign.

Speaker B: Welcome to the AI automation Dojo. My guest today is Mateus Schlamp Mart to anyone who's had to spell it twice, a senior manager for AI and Intelligent Process Automation at amersports. If the name amerspots doesn't immediately ring a bell, the brands, um, underneath it absolutely will. Salomon, Arctorex, Wilson, Atomic, Peak Performance, Suunto. So the gear in your ski bag, on your tennis court and quietly judging you from the back of your closet. It's a globally sprawling sports company and somewhere inside it there's a team in Krakow making sure the invoices get paid, the robots behave, and now the AI agents stay on script. So whether you're an accounts payable employee sick of answering, hey, did you get my invoice questions? Or a security officer who breaks into cold sweat every time someone says, but what if we just plugged AI agents into SAP? You're in the right place. Now grab your favorite katana or that squash rocket you haven't used since the pandemic hit and let's get to it. Today our guest is Matt, who has been with Amazpot since 2013. He started in accounts payable, moved into continuous improvement, uh, ended up running an automation team which produced more than 200 automations so far. His team has also been working with AI agents for some time and together we have just shipped the first global AI agent into production, a chatbot called My Invoice Assistant. That will get into. We've worked with Matt and his team for like seven years now, which in our industry is somewhere between long term partner and common law marriage. Uh, Matt, welcome to the dojo.

Speaker A: Hello. Uh, welcome and uh, it's a pleasure to be here. Thank you for the invitation.

Speaker B: Thanks. M. Tell me, your official title is AI and Intelligent Process Automation Senior Manager. That's a very long and fancy title. Um, if you could explain to us what is it that you actually do.

Speaker A: Yeah, indeed, it's a long one. But uh, actually it's explaining uh, in a nutshell what I'm responsible for. Because on one hand, uh, I'm responsible for the AI and finance. So I'm um, managing small, uh, AI specialist team, uh, focusing indeed on the scaling up AI, delivering AI solutions, uh, around finance and also promoting AI and finance. On the other hand, uh, uh, I'm responsible also for the team of uh, Intelligent Process Automation, which is like uh, if you recall, old rebranded, uh, let's say Robotic Process Automation team, which is uh, at Armor Sports, managed uh, by Eva, uh, Sonchek, uh, who's quite brilliant in making sure that our bots are working properly and are delivered uh, according to best compliance practices.

Speaker B: Mhm. So you have those two legs of kind of RPA and AI. Um, are those technologies collaborators or do you see them uh, kind of eating at each other?

Speaker A: Actually that's a good question because uh, on one hand it is very much collaborators and when I've been actually starting with AI I was really sure that it's a proper way that from robots we should add them a bit more possibilities so they will comply uh, each other. Uh, that bot will be a bit more intelligent beasts which we call now agent and even more agent AI that can use uh, bots on a way. But on the other hand uh, AI seems to be much wider thing because we are really talking here about building uh, a bit standalone things without the robot itself. Uh so uh, this is like uh, extra possibilities that AI gives us uh, and maybe worth to mention looking in the future. Uh, when you think about that agents will be eating robot let's say. I see maybe two extra things worth to mention here. On one hand it's a ah, kind of naming convention that I guess business will be using. I guess it was something similar with the robot if you recall like so some years ago that what is actually the definition of the robot in the business? Yeah. So that would there would be a question what is actually the definition of the agent in the business? I have a feeling that at some point we will not have robots. It will be robots for us and for the audience that are listening here. Because those are the pros uh who understand this in the detail. But I guess from the business perspective they would like to have an agent, you know and uh, more and more bots that will be kind of combined maybe with AI here or there will be I guess a bit more understand that or named as an agent or like this or something similar. On the other hand when we're talking about eating, I'm very much looking into the solution like from Claude, which is called Claude Cowork, uh, which is already basically live. And of course Microsoft is announcing that copilot. Uh, there will be, will also give this possibility to use cowork in the end which means that your personal processes can be by the end user uh automated with basically a cowork. So on one hand like of course for the business is still a risky, risky area. Yes, we will need to really get mature and and uh, um, and understand it a bit more. On the other hand it is a big, big opportunity and we should closely look uh, into it, into this. And that might be the part where I could be actually eating robots that you know it's kind of using like automating your desktop things. But on the other hand it's like maybe we will not have to build some, some of the robots here there if we'll get to enterprise in this, in this area.

Speaker B: Yeah, yeah, sure it will take some time and a lot of consideration around the costs and risks. But um, I have kind of similar feeling that in kind of high volume, center of excellence kind of automations the AI is more an addition to what you currently have. Right. Just expanding the possibilities of what you can do. But when it comes to personal ah, productivity, what we always used to talk about as a ah, citizen development kind of a thing, it's, it has become even too easy. Right. Um, so, so um, uh we used to do this with VBA which required learning and certain level of um, technical prowess. Then we had um, uh those um, citizen development robotic tools. They didn't really catch on as much as we thought they will. Right. But they were easier to use than vba. And now we kind of go into this AI territory where uh, like we discussed this before the recording, you basically don't need to have any technical skills to actually produce something that you can use. It's a different thing whether it will be tested enough and secure and so on. Right. But it is possible to do those things very very easily.

Speaker A: That's a super valid point. Uh and first of all it's a great opportunity and already some of the key users are catching that. They see that potential and they are trying to translate that things into their business to be safe. I'm saying that let's say for the time being it's a very good tool for prototyping solution that could be delivered. On the other hand we are really talking about delivering the software with all of the processes that have to be around it. Um, when I'm thinking about the future of the enterprise then it will be super important to have a governance over it and it will be super important not to block it. I guess this approach with having um, the M opportunity to have a prototype from business and then having a team that actually will deliver that as fast as possible. Because we cannot talk here about delivering the software in half or one year because it doesn't make sense for the business. We are talking really if you think about how on the other hand how good uh, AI becomes into the coding, there should be a team I guess and the businesses and enterprises have to really rethink uh, and have the internal teams or either external, internal, whatever, but a capability to use AI and AI agents that will be building in a safe manner those ideas from the business, not to block it. But on the other hand it will be really tough to control those sometimes bottom up ideas. It's no problem at the moment to build a macro. It's no problem to build an Office script and put it to Excel so it works. HTML files that are giving you an opportunity to build your local application here, uh, or there. So um, it's opening so much doors for the business and the key users and uh, indeed like the governance is uh, the most important part here.

Speaker B: Yeah, for sure. But um, I mean it, it has a huge potential to speed a lot of things up. I really like the idea of the business kind of already developing a prototype that they come to you with. Instead of filling in a form they come to you and say hey, this is what I came up with. Can you make it uh, proper and uh, give it to us? And of course the central team will also use AI to build it to some extent. Uh, but they will have the knowledge about the security and all the best practices and so on. I think that would be um, great to set up. When you do, make sure to tell me, come back to the podcast and tell everyone how to do it properly. Um, but um, I would like to come back to the AI agent that has just been uh, put into production. Um, my invoice assistant agent. Um, I'm wondering, your team has delivered more than 200 automations over the years. Um, and most of them, I presume probably 190 something were kind of classical kind of automations. Um, what made you decide that this particular problem that you had to solve with the invoice status that let's build an AI agent around it and not try to go into the classical things that we usually do.

Speaker A: Yeah, uh, the story is not that super short. So uh, we started really actually with looking on the accounts payable queries that our uh, accounts payable team uh, is getting because I guess most of us uh, already know and recall uh, from different uh, companies that there are agents that are making, actually solving those queries, like dispatching them, like maybe solving a few of them or trying to prepare a draft and getting some insight. So our idea was like, hey, let's discuss, let's uh, deep dive into the P2P queries and let's see what is the potential there at Amar Sports. So we did that maybe uh, you Recall actually it was also with Office Samurai that we actually did this analysis uh, with your support. Um, and in the end we actually have designed how agentic world could support this in the end. Uh, and then we started to think how to implement it. But at the same time we started to think also yeah, we need to really be careful and think like long term. Uh, and in short we decided that we need to make some cleanup and stabilization first like uh, from system to process also to take that into account. So that a bit stopped us from moving further in this direction. But at the same time we're thinking like one of the biggest uh, questions and the most simple questions that queries that we are getting or P2P is getting is actually about like is this invoice? Has this invoice been paid or not? Do you have it actually in workflow? Because my vendor is actually approaching me and you know, I don't know what to answer and you know end user like always sending the ticket, waiting for some time and the uh, response comes sometimes ping pong email, you know, like hey, you gave me only a uh, vendor name but you didn't answer anything more that we need like reference of the invoice maybe at least amount or something and then to respond the ticket again sometimes. So it's uh, really this throughput time that uh, we saw that could be improved. At the same time uh, we were considering like yeah, we could actually build power bi report with all of those data that will be available like report for all of the users. But at this time we thought yeah, it would be faster, it would be easy and satisfactory. There would be a satisfaction I guess for, for the end user. But at the same time we are causing some risk, you know, because we are giving to everyone in our sports. Because the idea was to give it to everyone in amber sports. Like not only finance, we are talking about like sellers or whoever is doing business with external guys. Uh, so we are causing a risk to that someone will extract all of those data or whatever happens like from security perspective. So we started to think like what are options we have? And it was quite natural. Like yeah, we could try Chatbot actually that could give that answer. We can have a guardrail in there like um, don't share like all of the invoice, let's say maximum 10 invoices per vendor with one response. So we are a bit more safe. At the same time you're getting an invoice really fast answer, like immediate answer. If there's a like ping pong like you you didn't provide enough information to get the outcome. You would get it uh, extra, you would get extra question from that, from the chatbot, like hey, please provide me a company code because I have no idea from which company like business unit you are from. So it seems that it's, it's uh, helping extra for the business. Um, and in the end uh, it's something that will not replace in the future because we still are considering to have of course this agent over the queries, but this is something that we still would save and keep this MIA chatbot which we actually delivered. So in the end, uh, uh, the idea was that we will limit the number of queries very fast because it was really kind of standalone with really low risk and basically quick win that we could, that we could deliver.

Speaker B: Yeah, this is, I think this is um, um, great thinking from the perspective of. Because when companies try to automate their mailboxes and ticketing queues, um, they think about, you know, the first few, how do we automate it? But they keep on thinking inside the ticketing system or inside the mailbox. But here you have actually you were able to move from tickets to live chat. And this not only automates the whole thing so that people don't have to just go to SAP, check something and answer uh, a ticket, which is not a very interesting thing to do.

Speaker A: True.

Speaker B: But that time to get the response is like seconds instead of possibly days. Especially with the ping pong, I presume it could take days for someone to actually get the final answer. Right. As you said, even when you get to automating the actual uh, ticketing queue, this chat can still be there and uh, to offload some of the work uh, from the queues. Right?

Speaker A: Yeah, indeed. And you know, um, the biggest benefit of this chat of course because we are looking on the time saving as usual within RPA on the time saving that uh, this solution uh, gives to the accounts payable. Uh, but on the other hand the more important part is related here to the uh, satisfaction of the client, let's say satisfaction of the business. That is uh, asking a question if you can think about, if you think about it, if the business has a question like query like what's happening with the invoice? I'm sure he's not already, already not happy with something. Yeah, because like why even they must have screwed up something. And so of course we know like that different things are happening and uh, and so on. Um, but so, so here was the main, our main benefits from our perspective was actually to to speed up the response for the business, to make this bigger change, to make this available, to have this immediate, immediate answer, to have this data available for the end user. So that's our big game changer. Also this like Amber Sports wide tool, like it's not limited uh to some financial people let's say as I mentioned already, uh, so it's a uh, really scale up tool.

Speaker B: Yeah. So um, yeah, I mean you have been working in uh P2P as have I and um, I know how it is when people get a vendor screaming at them and then they scream at us and then we do our best but then something is missing and if we can remove this friction then this can be a process. I mean they can scream at an AI agent all they want, it won't care. So it's also kind of uh, uh, added bonus here. But so, so from the um, end user perspective and you said that the end users would be a lot of different people all throughout the amersports organization. So this is not a chatbot for the viewers. It's not available to people outside of the amesports. It's not a self service tool for the vendors. It is available for the people uh, within amersports. From their perspective, how does it look like when they get a query from a vendor about an invoice? What do they do?

Speaker A: Well basically this chatbot is available for everyone via teams which is the tool that our Amber Sports and most of the companies I guess at the moment are using. So it's super easy access and then you basically ask a query about the invoice and uh, you don't have to know all of the numbers like vendor number or company code or specifically like the invoice number. Uh AI is trying to understand you. If you would say like name of the vendor, it will first of all try to catch what kind of vendor it is. If someone will explain like what's my business unit, what's my entity? Again not really a company code. You can explain like it's Poland. So it will give you a proposal if it's not sure and then if you don't know really the invoice reference, uh, which is one of the key things that we have there or invoice date. It will try to give you like a proposal, maybe it's one of those few, uh, you can check it out. So basically it's a conversation for, for like natural language conversation for, for the end user as an outcome. In the end you get information like this is the invoice that is in A system or we don't have this invoice in the system. And if this invoice is in a system, you're getting what's the status? Is it paid, is it not paid? When most probably it will be paid, uh, if it is in the system. So those are really the outcomes from the chatbot.

Speaker B: But those things that you talked about, they are not like, if we look under the hood, they are way more complicated than the end user will probably, um, imagine. Because say they, they tell you that they need, uh, um, there's an invoice from whatever, Lufthansa. Right. But you will. You probably have more than a few Lufthansas in your systems. Right. Because there's different, um, group companies of Lufthansa. And then, um, to find the right, um, uh, the right vendor, and then as you said, to find the right company code that the invoice should have sent to business. Users don't have to know those codes. So the tool needs to help them and kind of keep narrowing it down until it is able to find it. Um, uh, the same thing with references. Those are really funky strings. They can be quite long, they can be complicated. Right. Uh, sometimes they need to be shortened when entering into the system. So behind all of this, there's a lot of business logic that the agent needs to go through to be able to talk to the person like a human would.

Speaker A: Yeah, indeed. Uh, and we had to really go into the really deeply in those scenarios that might actually happen, as you say. Let's take an example of the selection of the vendor and Lufthansa. Ah. And the different groups that are there. So indeed, the chatbot is kind of querying the data, uh, for potential vendors that could be there in use. Could be what actually business is asking for or users asking for. So in the end, we are giving the proposal and then the end user is deciding like, yeah, I know that this is the vendor. And then we can move on for, uh, further selections in the end, uh, to give the proper answer.

Speaker B: Yeah. And when those things happen, um, the agent isn't querying the SAP directly. Right. What's the data source that the agent is working with?

Speaker A: Actually, the data source directly, uh, from the agent is actually Power bi, like basically data model from Power bi, uh, that we have used. We didn't go directly to SAP, even though maybe one step back, Power BI is refreshed once in a half hour. Uh, so, uh, it's a short delay for invoice processing. Come on. It doesn't make really the difference Maybe in some really narrow cases. Um, but uh, here we went with Power bi, not directly with SAP from different reasons. First of all it was security reason. So ah, within Power bi we actually have the data that are only required for us. We don't allow agent to go into the core system. We don't want to overload SAP, uh, with different queries. We're thinking actually that maybe we should give access to some kind of database. What we are thinking like yeah, this database that we are actually using for different reporting purposes. We also don't want to overload. We also don't want to open um, too much tunnels, let's say for the agent to um, that particular database. So we thought yeah, we could go with Power bi. There's a technical opportunity to get there. Uh, we also had uh, within our team, uh, Bhavo, who's actually a great Power BI experts who could really help us very, very quickly within the project, which is also super important for this case. We had the data there so you know, we didn't have to.

Speaker B: You're one of the lucky few who have the data exported so.

Speaker A: Exactly. So we didn't have to you know, go through the different requests to get like uh, some, some extra, some extra data. And why I'm saying we had the data like not all we wanted maybe, but it was good enough, it was really good enough at this stage to get there. Uh, so uh, that was the main reason where we went through.

Speaker B: Yeah, I mean getting it directly from SAP would pose a lot of different challenges. Uh, like you would have to do it kind of low level because through a GUI that would take too long for a, for um, a chatbot. And um, I think that all the stuff about not overloading SAP, I mean we all know extremely important if you want your people to be able to work on the core system, um, comfortably. Um, but um, I think it's worth noting for the people who are thinking about such a project that uh, this data security aspect is extremely important and in this case rightfully as you said, in the data that the agent had access to there was only data that it can give to an employee. So you don't have guardrails within the agent and hope that it's really not going to tell someone something they should know. You just trim the data to the data set that is safe to share. And this is one of the things I wanted to ask you about because this was um, quite an interesting discussion during the project about the amounts. In the end you decided to uh, remove amounts from the Data set. And also this means that the agent isn't even able to discuss the amount. What was behind that decision.

Speaker A: Yeah, of course, as you can imagine as those projects, AI uh projects. Those are something in general new for the business, for armor sports in general. So and even though if that would not be new, like uh, uh. We are very careful with uh, with what we are delivering because we have to be at this stage very careful to understand how it works. So within our processes, we are consulting whoever we have in a company that could help us like legal team from the data, uh, privacy perspective, like our auditors, to make sure that we are not uh, uh, overlapping somewhere uh, with our proper processing of either data or in general performing the process cybersecurity, of course, as you know very well. So uh, having that in mind, like thinking about really carefully what kind of data we can share, should share like uh, well and which one is a risky one. We decided that really amounts is something that we cannot have. Which is a bit tricky as you say, because we had this much discussion because for the business is one of the key element to understand which invoice is that because emote is usually kind of uh. Unique to uh. Recognize the invoices, at least for the business user, not for P2P.

Speaker B: They will, they will often have a question. I should receive an Invoice for like €10,000. Did we get it right?

Speaker A: Exactly, exactly. But yeah, from the security perspective we had a few fields uh that we decided that either like even though we have them in a system and we could put uh it to power bi and then to the agent we decided that no, no, we are sticking uh, taking this data. So we're talking really about amounts uh from the perspective of like giving access to someone uh from business. Like really data that should be secured. Like you know, raids between the vendors. Uh that we have. Uh, we have also different brands which uh. Of course we are one company. But of course like in some cases they are I wouldn't say competitive but uh. It would be a bit risky to show like in details what are the prices here or there. So this was really, really important to take that out to be secured. Other data that we excluded are uh, like uh, were like who's actually the coder slash approver for the process. So the personal data, we didn't want to touch any personal data, uh, in that and of course the coding. So uh, if potentially um. There would be a situation where um, we have information about Cost Center. So you know, Cost center very often is you know, confidential Data. What kind of, kind of cost center is here or there? Like I'm talking a bit about narrow cases. And as you can imagine, like we sometimes have like B2B guys like that are working for us and.

Speaker B: Yeah, exactly.

Speaker A: It could be like one employee checking what's the rate for other employee. You know, it would be really, really tricky. So you have to be super secure there.

Speaker B: This is something that requires a lot of attention. A lot of companies doing this kind of projects, they don't think about those small corner cases, those um, combinations that each piece of the data by itself is fine. But when you combine a few pieces, you can get information you shouldn't get. Right. So when I was at ubs, we used to um, call this uh, I think toxic combinations of data. Um, at the ubs it was kind of a different story because it was like if somebody, even if you uh, mask the name and the surname, if somebody's profession is a prince and they are from this country, this country only has one prince. So it's kind of right. Um, but in this case, rightfully, as you said, if someone's working B2B Ah, and you can check any amount of any vendor, then you can find how much the company is paying them. So um, yeah, I think it makes all the sense. Um, and in terms of um, things that the agent doesn't do that have been decided to be out of scope because it doesn't do um, payment confirmations or ARIBA stuff or PO queries and TND and so on. Um, so how did you get to decide what's going to be in the scope for version 1? And also do you plan to expand this, do you plan to go in the next iterations with more things that uh, MIA can, can do for the people?

Speaker A: Yeah, well, the story behind the scoping for this particular project was that the idea that we would like to deliver something valuable for the business, uh, so we wanted to balance it. So maybe not go with really minimum M value product, but something that would be bringing some additional value for the P2P and for the business. So uh, we decided this at some point that this scope is enough. Some of the extra things that we have in mind, uh, we could deliver later because it will be already helpful. At the same time, at the moment we are really looking into how this bot is actually in use, like how much it is helpful for the business. So we are indeed planning to send over some questionnaires to some of the users to get their feedback. And of course we are planning the further expansion and Iterations uh, uh, of growth for this particular agent because there's a big potential in there from the one that we clearly see is extra information related to po, ah numbers because some of the business we notice and we already had the signals at the very beginning that they would rather start using PO number and vendor versus invoice reference. So well unfortunately we didn't have at this stage enough data like not, not fully, not, not, not full like set of data that we could uh, go and go, go further, live with that. It would just take time and you know how it works with AI if you struggle too long or stop too much, AR is changing quite fast and then you have to reset the project because there's always something better on the market that you could use. So, so those projects cannot uh last too long. So from this perspective uh, we wanted to be rather faster uh from the further development as I mentioned we have in mind a few uh, interesting features or things that we could do in there. Uh but we decided that okay let's give some time to this agent and for the business that try to promote usage and let's see what will be the feedback from the business and basing on the customer needs, let's call it lean, like uh, what will be needed. We will mostly need it then we will focus on this one. But at the same time we have a sneaky plan coming back also to our uh, improvement of the queries or making the queries uh clear and agentic. So I guess the idea is behind, we hope that actually MIA would be playing a key kind of background role. So you know we don't need to build an extra, extra database uh, uh to get like invoice statuses and as you can imagine within those um, queries we have to do the queries like hey, I need a copy of an invoice. Uh hey I would like to I don't know check the statement from the vendor. And so like almost like I was I think checking uh like 50, 60% of the queries requires in the first step to check what's the invoice status. So we already have MIA in the back that could answer not to human but actually to other agents uh, uh, that could use it other than this for Mia, um, uh we can add some knowledges like uh policies from P2P um that would be helpful for the business users. We could expand maybe to travel and expenses team which might uh also make quite of the sense. So a lot of ideas obviously.

Speaker B: I can see that, I think it's quite interesting and again something that um, a lot of companies forget or don't really think about is that as you said it is a tool built with humans in mind but the logic and the data behind it could be serve other automations too. Right? So it doesn't have to be through a team's interface, it can call an API saying hey, this is the data on the invoice, give me the statuses and it's going to get uh, a well formatted um, and checked response.

Speaker A: And maybe just to mention as amersport is not unique, we have enough of the tools so uh, we are struggling as everyone, we really are trying to be a bit more smart with the number of the tools that we are at amersports delivering and have to maintain in the end. So that's also important.

Speaker B: Yeah, yeah, yeah. It would be better to um, put more things into a fewer number of tools for sure, yeah. Ah. And uh, when it comes to the response, because on one hand we have um, your P2P teams, right? That potentially I presume the reaction could be anywhere between oh my God, oh my God, the agent is taking my job and finally something is going to be answering those burdensome queries. Right. Um, and then we have the end users that also can go from um, I really like speaking to humans and I'm used to this particular person checking things for me to wow, now I have an answer in seconds instead of hours or days. Right. So what was the reaction on the P2P team side and the business side

Speaker A: so from P2P, uh, like our P2P first of all was really heavily involved in that project and they were really, really supportive and uh, I'm quite happy to have like the P2P team as a specialist that we have there because they have really good ideas and also which is super important really uh, like wide knowledge uh, around the processes. Thanks to this our project actually was successful because we've been getting the proper narrow cases like hey, we have unique here, unique process there, unique vendor here, there. So we had to take into account a lot of different stuff. So based on this, what I can say that they were rather in general quite happy to have that tool because you know, uh, on one hand you know that people do not like changes but on the other hand this one was really a burden. Like uh, I guess maybe you've been working on queries. Like I also work on the queries, I actually like that part but not the part that I had to just answer was the status because that was boring. I liked solving cases yeah, when you

Speaker B: need to do an investigation and check all the mess on the accounts. That's fun.

Speaker A: That was really interesting. But that was really one of those boring, annoying things that nobody's uh, missing. In the end we are still in a promotion phase of it. So we are slowly looking like uh, closely looking on if the number of tickets is uh, lowering and so on. So we will see how it will end uh, and where we will get coming back to the business I guess as you say due to the fact that ah, this is a really uh, wide tool. It's big role of our P2P actually to educate. So we are getting those queries from Steel of course. But in response there can be always added information like hey, here's the link. Please use it like next time maybe it will be more, more, more helpful for you. Yeah. But as you can imagine there will be always someone that I know that I know the guy in P2P. I will make a call and uh, because I need it kind of. And I guess we will need that like it's a kind of customer satisfaction. And this is something that I guess will remind for some of the, for some of the users and uh, looking in the world for example privately when you think about uh, let's say being a customer of a bank in Poland. Do you have a number to call to the bank? It's really hard to get actually the phone number to call them. Everything is.

Speaker B: Nobody wants to take your calls anymore.

Speaker A: I don't know. How about you? I'm super annoyed. Uh, so it's not really a customer satisfaction. But I guess this is our internal employees and we care about them as gbs. So from our perspective uh, like this is the service that we provide and I guess for some exceptional cases there will be still that and just too much.

Speaker B: Yeah, there needs to be kind of the second line that um, when the tool isn't able to do something that you will that for sure. I mean that model that like Facebook or YouTube are using where it's virtually impossible to speak to a human, um, it just doesn't work because there's always a corner case that a chatbot won't um, won't solve. But I think um, the bank example is really good because I used to have, when I had my mortgage I used to have a particular person that, that knew everything about my case. Right. Um, I would rather do things for, through a form or a chatbot. If it worked right. If it worked, it was great. But very. It wouldn't work. And then I would have to get them uh, sometimes even go to physically to the bank. But then because it was this one person I would have to wait till they serve other people who came to them. So it um, like, I mean in the end probably we're not going to miss those human interactions in service as much as we think we will.

Speaker A: Yeah, and that's a good point also because m. I think it's a big job of function like myself uh, within organizations also in general to promote of usage of AI. So maybe at the beginning really the business will still keep calling because they're used to do that. And um, when everyone maybe a bit more wider will discover that actually AI chatbots and I'm talking about playing copilot or cloud GPT, whatever you would use is actually giving a good answer uh, a bit myself also I never liked the chatbots on the bank side. It was like this old times when there was a scenario not really AI behind. So I always avoided it. Now I'm actually really curious how it works and I very often are you're probing it, trying to check, maybe even to check if that could bring me some Python code to check if they made the proper. But uh, on the other hand I know that I'm sure that within uh, together with the growth um of the mindset that AI can be used that kind of tools will be uh, more and more in use. And as you can imagine in organizations those chatbots uh, like in ours with the knowledge instead of Q and A's, instead of reading their big long policy, those stuff are growing. So I guess it will be more and more available. So it will be change of our mindset in the end that uh, we will start more and more use it.

Speaker B: Yeah and the tools need to be good enough for us not to get frustrated and switch to a human. Right. But did you on this um, kind of uh, breaking or jailbreaking, uh, did you catch anyone trying to jailbreak your meal?

Speaker A: Well they were trying. Okay, well lucky as mentioned, maybe not lucky. It was thought before that we need to really limit the number of the data, the risk that we've had in mind that might happen that it will actually hallucinate and give some amount. Uh but yeah, uh, it is something that we had to, to take into consideration and you know very well because you've been developing that uh Sophie Samurai for us that particular agent. So you know very well that developing in AI era is quite a uh, smaller thing. Uh as it comes to timeline and time spent actually on the solution is the test phase that is much much more crucial. So we did spend a lot of time on the testing and uh, tried to narrow the cases involved like quite wide team. Uh so um, in the end, in the end thanks to this we've been quite successful and maybe besides this like internal test phase like P2P test phase we also had like the test phase with the business.

Speaker B: Yeah.

Speaker A: And as you can imagine there was one person from the business that caught one narrow cases that we didn't think about. So actually it uh, was good idea from our end to to have actually this extra phase that uh P2P was not considering because they're not business in the end uh, was not considering um, the scenario that happened there. So we managed to catch it still

Speaker B: before go live like yes, no, I agree completely. This is what um our experience is with AI projects. There is a lot and a lot of testing and uh, uh it takes a lot of work to do it well uh, to do it properly because there's also a lot of things that can go wrong. Right. Um. So um, preparing a good set of testing scenarios and then testing and testing and then as you said doing it in phases so you give it to a smaller group of people piloting um. Um absolutely makes a lot of sense. I mean otherwise you're just going to give people something half baked and they're going to take a look at it once and they're going to never use it again.

Speaker A: And I guess it's a worst ah case scenario because you're burned. Basically you put this tool it's like ah. For me we had this discussion just before about Copilot in Excel at some point I've been doing quite a lot of testing how this could work. It. It really didn't honestly it was not really good tool and I wanted to test like uh, how what's the potential from cloud perspective. I heard a good stories and I started to uh, uh work to compare. I started to work and see how Copilot works and now Copilot is a different story within Excel. So it's doing great but you needed

Speaker B: an impulse to come back to it again and check it again because otherwise

Speaker A: that is a uh big lessons like that uh everything is moving that fast. So fast. As we discussed with Microsoft, it's even a bit more tricky uh due to the fact that it's not that clearly announced sometimes. Um and uh, as you say uh, I stopped working with Copilot in Excel because I tried it once, it didn't work. Now I know that it works but it was kind uh, of luck.

Speaker B: Yeah, yeah, absolutely. Um, I think, uh, also from the perspective of how the tool works, it's worth mentioning that um, when we were discussing the architecture, um, it was decided, uh, and it probably should be for 99% of the projects. This should be a rule of thumb, um, that every logic that we can we're going to put into the classical layer of uh, the tool. And we will only use AI where classical programming doesn't cut it.

Speaker A: Right.

Speaker B: So the AI is needed to understand the user's intent and to ask them additional questions and so on. But then um, getting the data, filtering it, presenting it, this is all classical logic. And this way we also get rid of possibilities to like hallucinate an additional invoice because it's just the data from the database presented by AI, but in a nice way, in a nice template basically.

Speaker A: Right, yeah, correct. And you know, as I told you, I'm in the end in finance and looking from finance perspective where we have to be as accurate as possible. Uh also having in mind that uh, we have a lot of controls around, we have to hallucinate as less as possible.

Speaker B: Yeah, I mean you're a SOX company, you have to be.

Speaker A: And uh, even from the like we have a big discussion with uh, internally, uh, with audit, like uh, what does it mean to have an agent in the regulated environment because it's not standard controls that we have to take or it is standard controls but something extra. Uh, so uh, it's another round of discussion that would need to be. That we will have and we are, we are tackling at the moment. Uh, I guess not. It's not only us of course, uh, because it's a bigger problem. But as you say, if we are, if there is a potential or possibility to have a rule based things, let's go with that. Because it's easier to control. Because it's a bit more accurate. Yeah, we have a much better uh, overlook of the, of the whole.

Speaker B: Yeah, absolutely. And when you think about like, I know like we discussed that you already have a lot of ideas of how, where it's gonna go. But like in the perfect world, in three years, what is Mia doing? Like is it still doing what it does? Has it been promoted to something else? Do you have some sort of an, a vision of, of where it could be?

Speaker A: Yeah, definitely promote it. Yeah. After three years.

Speaker B: Yeah, I mean it makes sense. We don't want it to go somewhere else.

Speaker A: Right, exactly. Hard to predict honestly. Uh, on one hand those are those things that I mentioned. It's, it's rather clear for me that we should give a bit more data there that still will be safe to, to. To. To be. To be available. We taking into account that um, uh there are some from the end user perspective there's a big, I wouldn't say need but uh. They would really love to have the amounts. I think we'll need to tackle the topic at least to analyze what does it mean having in mind that we have 14k employees. We would not like to have role level security by the persons or the department and so on but maybe for some group we could reconsider some key users or something like that. Um at the same time I'm seeing Mia as ah a key chatbot or key agent for accounts payable from one hand like one stop shop for the end user. So whatever I have I would like to check very quickly. Uh I can do that uh via this chatbot. It could create a ticket if needed or forward the case to the human if needed. Um maybe it could trigger some actions like I don't know, request a copy of an invoice for some reason or maybe copy of invoice still amount related. So tricky one but uh, uh something similar uh so actions actually that it could do um at the same time as I told you, some kind of back engine for other type of automations that that could be there. So I guess that's a high level vision that, that I would see me in some time. Uh, maybe there's my invoice assistant would be not only my invoice. Yeah, yeah yeah but, but with promotion you know it's a might be identity change. Why not? Um so we will see and we are really uh looking on it with really big hopes to uh, improve. At the same time when you think about the future of AI in three years I already will not risk at all to say where we could be and what will be possible because it's so much constantly changing and there's a lot of surprises that are in front of us which is interesting times for us.

Speaker B: Yeah, I share the sentiment absolutely. It's really hard to make a prediction when it comes to what the models can do. It's really hard to make a prediction for like six months let uh, alone three years. Um and for the people who are listening and maybe they are about to kick off their AI agent project, um, do you have some piece of advice that you would give them? Looking back at um what you were thinking last year when the project was starting, looking back at everything that's happened, what would you Tell them um, they really should think about or know or consider

Speaker A: do basically try to do something with AI. I mean uh, of course the idea is important to have a good idea, not, not too big one because too big scope might kill. Uh, especially like with the big longer projects for AI as we discussed. It's a uh, risky area because the AI is just changing and improving constantly. So something with a bit more limited scope with some nice impact. Maybe before that try to do something smaller. This is what we tried also last year we did small, smaller cases, smaller chatbots with some chatbots with knowledge to understand how that works. Maybe publish here or there some small, small stuff to check what's the like uh, end user's feelings of that and promote also like uh, that AI is helpful. Um, and um, also I think uh, like trying out in the area of possible. I know that there's a small hate on the copilot but you know, it's solving some of the stuff you can try out if uh, it's available. And of course if you want to be faster because of course for me personally it would be the best to have internal skills and knowledge how to deliver stuff. I think if you want to be a bit more fast, it would be good to have someone with experience like we've had you guys that uh, you've already had experience how to deliver those stuff. You knew what possible, what's not possible, what to take into account. It's like not, not, not inventing the wheel from the, from the beginning.

Speaker B: So it's not making all the mistakes that other people have already made. Yeah, yeah, yeah, okay, okay, that makes sense. And, and with your experience because you're um, not only in those projects but you personally, I know you use AI a lot, um, what's the dumbest thing that people think AI can do but it cannot.

Speaker A: That's a good question. Uh, and uh, my answer varies because uh, it's constantly doing better and better. And I have experiences from the like half year ago that I cannot say anymore because it's actually doing quite good stuff. But I guess you know, sometimes AI still gets stuck with thinking that it's right. And you're telling this like no, you're not really right. Uh, and it's constantly so sure self confident that it's correct in some areas it's hard to convince AI that it's uh, actually making a mistake. So uh, for me this one is a bit frustrating. On the other hand it's also a matter of uh, experience, uh, how to properly prompt how to Talk to AI. So I guess it's a, uh, learning curve also on my end, for example, like how to properly query, how to properly prompt. Uh, but on the other hand, still sometimes like, getting stuck. Getting stuck.

Speaker B: I had this experience recently. I, I needed to write some simple Python code for an API, just an internal project that nobody else had time to do. Right. And it worked, but it wasn't showing exactly what I needed. So I asked it to do a change and it stopped working. Right. And I've been working with Gemini and I told it it's not working, this is the error code I get. And then it gave me another solution, solution B, and it didn't work either. So I tell it, this is the error I get, it's not working, it switches back to solution A and then I tell it doesn't work, it goes to B and it goes into this loop that I stopped it only by moving the conversation to Claude. Uh, I gave it. So this is the code that I currently have helped me fix it and it was fine. So sometimes you just need to kind of try a different approach. But it is frustrating because we think like those models at coding, they are amazing. Like, that's like probably the best thing they can do is to write code. But even with this, even the good models, sometimes something just doesn't work exactly.

Speaker A: Like, my advice is like, reset, go to other model, start new conversation. Like, not, do not frustrate. Like we have enough of it.

Speaker B: Absolutely. Even though the model can take you screaming at it all day long, it doesn't really help. Right. Okay. Um, Matt, thank you so much for, uh, being here and for sharing your experience with introducing the um, AI agent for, um, invoice status. I am looking forward to seeing where Mia is in some time. Um, where it goes. Um, and for sure, once you figure out the citizen developer part, I will love and I know the listeners would love to hear how it should be properly done. Um, so thanks again and see you soon.

Speaker A: Thank you all so very much and thanks, uh, for having me. Again.

Speaker B: Thanks. And that's a wrap. The invoices have been located. The P2P inbox has gone quiet for the first time in recovery recorded history. And somewhere a finance manager is using their lunch break for actual lunch. Don't worry, Gato for listening. We know your time is valuable, almost as valuable as the invoice you swore you submitted three weeks ago, which is definitely, absolutely in the workflow. A massive thank you to Matt for proving that self service doesn't have to be corporate code for Good luck. You're on your own. And to our producer Anna Tsubal, who has been quietly approving my scripts for all the episodes and has never once asked what the company code is. And to Vodsubis Studio, whose invoices, I should mention, are always paid on time. If this episode convinced you that asking a chatbot is less embarrassing than emailing Karen in Finance for the fifth time hit like subscribe and check out the Office Samurai AI Automation Playbook from the description until next time posted Paid Peace of Mind Matane.