Benjamin van der Maas: Using AI to Scale FinOps

Speaker A: Hi, I'm benjamin van der maas, and this is the infamous finops pod.

Speaker B: Witty comment here.

Speaker C: Witty comment, witty comment. Hi, I'm Stacy Case.

Speaker B: And I'm, uh, Joe Daly.

Speaker C: And this is Finops Pod.

Speaker B: This is Finops Pod.

Speaker C: Ooh, Joe. This is our fourth and final podcast of the year.

Speaker B: I'm so excited we got four episodes in this year.

Speaker C: Listen, we need to celebrate every win, and sometimes the wins are really big and sometimes there's just four podcasts. But those are four really good podcasts.

Speaker B: They are.

Speaker C: And I don't think this one is going to disappoint at all. If anything, we're going to put this podcast out and then we're like, yes, this was worth waiting for, for sure. Yeah, right?

Speaker B: Yeah. That's why I told our guest today as he waited months for me to edit it. But it's a great episode featuring Benjamin Vandermas. He presented at Finops X Europe last year on how they use AI to help scale out their Finops team. And it was great. And I asked him some more questions about, like, hey, why did you do that? How did you do that? How would anyone else go about doing that? And it was a really interesting conversation.

Speaker C: Yeah, I'm excited to listen to what he has to say. I mean, from my own perspective. You know, we launched the FinOps for AI series this year, and the reception that we've had for it has been amazing. People are so hungry to learn more about FinOps for AI, how they can use AI for FinOps and how can they apply FinOps to. To AI. So I think the more that we can talk about it, the more that we can share real stories. It's just going, uh, to benefit everybody all around. So, yeah, I've been really excited to listen to see what Benjamin has to say.

Speaker B: Absolutely. And in a nutshell, just give a preview. AI doesn't necessarily make Finops easier. In order for it to work, you still have to be a master of the domain and know exactly what you want the AI to do. Otherwise, AI will just agree with whatever information is put in front of it.

Speaker C: Absolutely. And I think that is that one thing. People get a little bit nervous when we have new technology of how it's going to impact their role. Listen, you're in finops, baby. You've got a job. Keep doing it.

Speaker B: Finops, baby.

Speaker C: This is Finops, baby. There's still a place for you.

Speaker B: New catchphrase. That's awesome. And then at the end, um, he had mentioned this to me before I started recording. I was like, oh, man, I don't know if this is if you're right on this or not. And we had this conversation about how they use AI for sustainability. So I'll just throw that in there. It's really interesting. And he won me over.

Speaker C: Uh, seems like quite the contradiction.

Speaker B: It does. But he did win me over. And really compelling. And a different way to think about sustainability.

Speaker C: Hm. Well, I'm excited to hear it and listen to what Benjamin has to say. And I'm really proud of you for getting out this podcast before the end of the years.

Speaker B: It's not released yet.

Speaker C: I think you'll do it, but.

Speaker B: No, I think I'll do it.

Speaker C: It'll be good.

Speaker B: It'll be good.

Speaker C: All right, let's stop talking and listen to Benjamin.

Speaker B: At finops X Europe in Barcelona a year ago now, you and Jan Peters, your colleague at JJ, presented on how you were using AI to help explain expand your finOps reach. I thought it was a really great presentation. It was really intriguing, and it really made me feel like everyone was doing cool things with AI but me. You've had really good success with this so far, so give us a quick update of what's happened since you gave that presentation. You've gotten a promotion, right?

Speaker A: Yeah, I got a promotion. So I'm, um, super happy. I'm stoked. It's also about how finops is growing not only in our organization, but SCOPES has grown dramatically changed the landscape for us, and I think for finops as a whole. We can get into that later. Whenever I think back to that talk, there was someone at the end, and I don't know if it's in the Q and A that is in the YouTube recording, or if it was slightly after someone asked a question like, what did it take for you to convince your management to use AI for this? And my reply very candidly is, well, they had to convince us we weren't really looking to use AI. We were fine with what we could do with Python and traditional automation, but our leadership was really looking to see what value AI, uh, could bring to us, and they wanted us to explore that. So we started doing that. And I started to talk saying, it's not a magic wand. And this is maybe, uh, ten months after. It's still not a magic wand. Definitely not. But what I do see where we went very right with the, uh, approach we display in the talk is that we slice and dice data to an extremely selective format and amount, and then we add AI on top. We have never believed and still do not believe in here's the data points AI added and have fun. Good luck. That doesn't work. Even with the new reasoning models that can sort of look at it, their own calculations and reasons so to speak, that they are wrong, you still need that human in the loop. There's actively news stories coming out of agents making mistakes. When we look back at the presentation we gave 10 months ago, that for us was actually the seed of, hey, this way of working, of looking at the data, putting an engine over the data, doing your basic Python polar's work, to data frame your way through to something that makes sense is the way to implement AI in a safe, productive way across various workloads. There's outliers, but this way we can use it not just in finops, but also in observability and in security and in other things.

Speaker B: So you've got an approach with AI, something you were telling me a little bit before and I think it's a great answer that you gave me. So let me ask you again, what is AI good at?

Speaker A: Yeah, I mentioned it in the talk. AI is still typically a large language model that is really good at taking data and then presenting it in a text or story format. It is not good at creating something from scratch. It is good at summarizing data, showing data that might be inaccessible in a JSON string to you in a way that makes sense with the different parameters in that JSON string. If I give you 100 data points in a JSON, it will take you at least a few hours to make sense of it and connect the dots. If I give it to an AI within eight seconds, it has a nice report. It can probably also create a graph out of the data to show you this is what's going on. These are the resources that are spiking or causing a cost anomaly and how you can take action if it pulls in other data formats. That is the true power of AI. It's not uh, going to create that JSON because it has no function to create context or to really understand something. But it can do so much with what you give it. You still have to give it the data you select. Basically, yeah.

Speaker B: And something you told me that really resonated with me, LLMs will always agree with you.

Speaker A: Yeah, absolutely.

Speaker B: So that's a danger.

Speaker A: And they will also agree with you. They will say among everybody that has used AI in their daily lives, recognize this feeling where you ask a question, they give you an answer, you ask an additional question and they just go ahead and say, oh yes, this is a great question. This, this, and this. And it adjusts its answer, whereas it should have been there in the beginning. Everybody with a normal frame of reference will have put your additional question in the initial answer, but they will just keep saying, oh, great, how great that you add this or of course or this or that. You want anything that does an analysis to have the potential to disagree with you and to find things that you cannot, um, find, because those would be useless analysis. And also it agrees with you always. But then also it ignores complete basic instructions. Like, we have it set up. So large language models love markdown format, and so they will provide their answers in markdown. We ingest those and handle those, and it tends to put double stars across the title of the report it sends us, because that's how it puts it in bold and it thinks that titles should be bold. Problem is how we display that doesn't support markdown. And so we'll get these titles in. Even though we tell it, don't use markdown in the title, we keep this whole context of how we will then ingest that answer into HTML. So it knows where it needs to go. It ignores that because it's programmed to use markdown for a report. It's somehow trained so much on markdown reports that that is how it should be.

Speaker B: Yeah, you mentioned you're leveraging, uh, with all that context in mind, like you have to set it up correctly and set up AI and LLMs and ways for it to be successful. You are leveraging it to help expand the scope that your team is managing. Tell us more about that.

Speaker A: Yeah, so we realized that the first POC in our team or POC or project was what we did with customization. That's where the talk in Barcelona was about. Definitely go watch that if you have the time. But the same methodology, basically what it does is about a specific topic. Cost anomalies. We data mine, expose data, and then create a report that we expose to our, um, owners. This is something that we used to do manually, where I or a colleague would go in and see anomalies. The ones that were most concerning, we'd investigate, look for the resources that caused it, et cetera. Took us time and, uh, we automated that away, so to speak. We figured out there are teams of account owners or application owners or product groups that might want insights into the accounts in the cloud that make sense to them. And they sometimes come to us and say, hey, we need your help with a maturity assessment. We do these monthly reviews of our landscape for uptime delivery other KPIs. But we want to include costs. So they would pull us in how we would sit with them, um, as a dedicated sparring partner to give them those reports. But we realized that for an organization of our size that is absolutely not scalable, the exact same approach we're doing for investigations, we could do for a whole other host of dimensions. We could do it for your accounts. So how are your accounts doing data quality wise in terms of ownership and views? We even go as far as to data mine how they are viewing the cost tooling and if all owners are viewing the cost tooling, yes or no, one might go on holiday and nobody then cares. We also go in and then data mine. Okay, how are they trending against budget and how is that historically going? Are they frequently trending against budget or not? Have they taken action, et cetera? Same for anomalies, same for sustainability. This is something that we're working on very new. We found that all these dimensions that we think of when we assess scope of accounts to assess our maturity, we can data mine everything we look at, sort of uh, create that blind analysis like just grab the raw data that we need to say yes, they're mature or not, and then go ahead and create reports from that. So now we created a tool where any account owner within our organization can select a scope of products, applications, cloud, accounts, folios. We have a bunch of filters. They can just group and mix and match and then run these reports within. I think the longest I've seen it take. So for the largest section was like a minute and they'll get at this point it's like seven maturity assessments for the scope they selected with actionable items, um, relevant information, anything they might get from us. This first thing that we built, that we presented on Barcelona, went on, um, to change the way we function as a consulting party internally, so to speak. M so now we really try and give the teams the power to do it themselves. They can come to us, uh, sure, but this way we don't need to be there every single step of the way. In my opinion, there is a possibility where we become a bottleneck of people trying to make that cultural change. We can only have the conversations we can have in a day that's not that many. And so now they can get that data and those reports themselves in a very accurate way. That is where AI really shines for us.

Speaker B: So you're using AI to help scale out, get teams started and doing the basics on their own, at their own pace, and then they can Reach out to you and your team when they need that value of human concentration and human thinking and additional context there. You've put an impressive amount of scopes into this tool. Public cloud, private cloud, data cloud, Snowflake and databricks. Networking. You put networking data in there too, right?

Speaker A: Uh, that's the goal. Networking is a scope, we think, that is not getting the love that it deserves. One thing that I'm still chasing is.

Speaker B: Mhm.

Speaker A: Getting people that cost per IP to IP traffic and getting them that analysis of like, hey, if we give them a network report, it should say your traffic for example, from DCC2 to S3 is going overnight. That's the recommendation we give. It's a typical inefficiency where you push your traffic over a net gateway instead of an endpoint. So it gives you a high cost. We want to give people visibility in the data flow of their account, visualize it so they know, okay, this is for networks. We're trying to get deeper on that also, like if you have an enterprise environment where you have transit gateway set up in a sort of hub and spoke model, you want to be able to really show, okay, you don't want your networking team to eat the cost of the networking traffic of the entire organization, so you want to be able to distribute that. And we realize that yes, we have full showback and chargeback, but if we look at networking, that's absolutely not true. We might have an environment where there's multiple applications using shared resources where we can accurately split that uh, networking traffic between IPs, so between applications and get even better there. But we're not doing that yet. And so I think that's a growth area.

Speaker B: That's the path to that is getting unlocked through this LLM tool. Because you're not having to be there every step of the way. Being a bottleneck.

Speaker A: Exactly. I mean it's a team effort, right? I'm one of the lucky people to have a large finls team around me. The fact that we now don't have to spend so much time creating these individual analyses, these reports, these deep dives, because we can within eight seconds have a root cause analysis or uh, an anomaly or an analysis of a scope of accounts on how they can improve based on our data means that we can spend time developing more ingestion pipelines, onboarding new scopes, looking at, okay, how can we efficiently retrieve flow logs for literally thousands of AWS accounts, treat them in a way that keeps our database sane, and put a front end that is interpretable for the user understand that they what the numbers mean. Right. And what resource they can click to. Then spread the cost side over IPs and understand what that means and where that data comes from and have that perform as you would expect from a modern tool. So all of that development work and creation work, uh, is now more possible because we don't have to spend time answering it sounds disrespective, but answering basic questions. The way I really like to think of it is finops is all about ownership. They can own it far more than if they had to email us or we would have to teach them. And so I think that is where FinOps is going to go as well. This sort of analysis work should be far more accessible to the masses within your organization, at least for our size.

Speaker B: Mhm.

Speaker A: So that we can focus on getting more data dimensions into those reports, researching more inefficiencies.

Speaker B: Let me ask you, circling back to what we were just talking about. So what you're doing is you're taking all this information from all these different sources. There's even more sources I didn't even go over. But once you have the pattern you can figure it out. You're taking the data, you're getting the data into a efficient format and that's not simple, that's work in and of itself. But then you're feeding it through the LLM to analyze. How do you get the LLM to be good at that?

Speaker A: So if I would show you any of our LLM ingestions or uh, integration. Sorry, the script we keep saying we built this with an LLM but at the end of the day the LLM is a tiny fraction. If it was a Python script with 20 functions it would be like one of the 20 functions. And that's it. The way we do this is for any of these dimensions or scopes. If I go look at the data of a scope, I know what to look for. After research and study to determine what information is valuable, I might be wrong. But I then go talk to a kind owner, say is this the way you want to see your data slice and dice? Does this make sense to you, etc. And based on that I get a list of data points that will tell the story. So if someone would come to me this. So basically I create the query I would run if someone asked me that question and m I get exactly the data I would need to then go back and write the email saying hey, sure, this is what makes sense to you. That never changes. Even if we have a language model or Another AI model, I think that work should never change. You should always define what data makes sense and pull it out in a specific as possible kind of way. Then you start thinking of your LLM, I mentioned JSON before. You can then expose those data points in a JSON format or in a CSV or whatever. LLMs are trading on language. And if you really want your LLM to understand what the numbers mean, because they are not good with numbers, you take those numbers that JSON parse it out into variables and they put basic sentences around your variables, like the accounts or the workspace id. If we're talking databricks, the workspace ID that I analyzed is. And then you add your F string with your variable and then we saw a growth in numbers of this and that, which is a growth percentage. It's all things you can calculate in Python and you can filter out like, hey, this is the largest growing resource, this is the cluster or the job that drove the largest. If we keep talking databricks, this is the largest contributor to that spike or to the account or this user is driving the most jobs in a month. We can do all of that with basic sentences in Python. Still no LLM involved. But when we put that in front of an LLM, we have a text we can give to an LLM and that's when we can have a prompt where we say, these are my basic research nodes, help me make sense of those, analyze those. As AI models get better with access to public documentation, use documentation found here and here, for example, I'll point them to databricks documentation and give me a report in these sections where I expect to be able to then do this and that, where I expect actual next steps. Uh, concerns, risks and your conclusion, for example, and by the time the AI or the model picks up the text, all it has to do is process language. And not only language, but language that we have created. So we'll have created the data that's in there, but we'll also have created the wording that the sentences are in. We determine if a 50% growth is severe or not. In a databricks environment where there's spikes happening all the time, 50% might not be severe, we might need 150% growth, but we can determine all of that with basic scripting.

Speaker B: Mhm.

Speaker A: And so if I had, this is how we work for every single scope implementation that we build within finops. If you want to build something with Genai or large language models, just build it and have your MVP be completely without a large Language model. If you can do it without, you're good to go. Then you can add an LLM on top because the value there is that it can access public documentation and it can structure the answer and it can maybe provide it to you in HTML format that you can send out as an email. I mean, there is value there.

Speaker B: Well, uh, that's super interesting and I'm glad you framed it up that way. You're still doing all the work. What you've done is you are defining what is good and what is not good. And actually I really like that. And when you're developing it, don't develop it with an LLM, then you'll be good to go. You're making me think of the most recent episode we did with Kim Weir and Ron Tetro from Target, where they are defining good and efficient based off their agreements and interactions with all the development teams. You're doing something very similar. You have all these different scopes. You're still deciding the data that's getting included. You're working on an agreed upon model of what's efficient, inefficient, what's good, what's bad. And you have that framework already set, developed and engineered and coded. Then you put the LLM on top. So the LLM is not LLM go wild. It is LLM report on the processes and agreements we have already set up and then be something that can interact in and communicate in plain text with users.

Speaker A: Yeah. And I'd like to add to that. If I had to write the reports I get in eight seconds from an LLM, would take me maybe two hours if I stay focused, mar focus, eight hours straight. So I can maybe generate three in a day if I'm not bored out of my mind after five days of that work. That is generating one report. If I had to look at just one account, if I didn't have any meetings, if I didn't have to do anything or develop anything else. And the value is that AI can expose that data in an efficient way, quick standardized way, where we also expose like, uh, the history of reports that we have created for an account. And if they go back and look at the history of things I wrote or that we as a team wrote, they might see five different styles, different pieces of information. Someone might forget to add a, uh, certain data point once and then people expect that next time and it's, uh, not there and you have this back and forth, et cetera. AI can, if you feed it the way I described it, can be consistent and fast. And we can focus on other things. So we have to do the research. Once we define the standards, we write the scripts, we write the prompts and then the LLM can go do the light work. And the way I, I, I, I try and describe it to people is it's like you have a uh, a junior resource fresh out of school that you help see what they need to see in the data and then have them write a report on it. Except they're really fast at writing the report.

Speaker B: College interns with really good typing skills.

Speaker A: That's true.

Speaker B: I want to jump to something and you hinted to this before your tooling users go in. They're interacting with this LLM. They're choosing what data they want. They're able to choose what information they see and some of the information that you provide them. When you told me this the first time I stopped, I said I don't know if we could talk about this. But you said it and I love, I just, it expanded my perspective on this. So I appreciate the heck out of it. You give them sustainability information. So people are leveraging LLMs in order to get sustainability information. There are going to be people who hear me say that and they're going to be like, oh my God, it's like he's holding a lighter to the polar ice caps. But I really like what you said about it.

Speaker A: Yeah. There's two things I want to say. One, we are completely aware of the irony of the whole situation. We have slides. I literally did the calculations what the emissions m would be using the method that I've been describing so far with slice and dicing the data and then asking about an LLM what the uh, emissions would be. Just saying here's the table. LLM based on your knowledge of public cloud, make me an analysis of what sustainability footprint looks like and then what it would roughly be if I had to sit in an office, had air conditioning the office power up my laptop, power up the light bulbs, power up. I have to drive to the office to do the work. Like what the emissions of that would be for the amount of reports I can create with that. If we want to give people the data there is always going to be a uh, sustainability cost to it that is no way around it. So any argument there have to make but I think the way we do it is in a very efficient way, low impact way because if we don't do it, if we have to go and talk to the teams, believe me, I also have a carbon footprint attached to me. It's naive if I can be so bold to say that the LLM is the wrong way, because I can prove to you that that is, I mean, the scripts to give you an idea, the scripts to get the data out of the database, structure it, feed it to LLM and get it back, is seven to eight seconds. And the token count is extremely low, probably less than if you go to LLM and ask, hey, write me an email. We do everything we possibly can. Like, we buffer it, we store it, we limit, like, so if you refresh the page, it won't keep sending the same requests. So we try and manage that. So we're aware that's the first part. And the second part is I went to Barcelona, I went to a talk, and for the life of me, I can't remember which one it was. But basically he came back and said, there's really no excuse not to do anything about sustainability. There is no excuse saying, oh, the data is not good, so we're not going to do anything with it. So right there we decided that we need to start working on this. And so we decided we have all this cloud billing and usage data, literally hundreds of millions of lines per day. We have some sort of sustainability carbon emission data from the cloud providers, only scope one and two for all of them. So not even scope three, which is 40 to 50 to even more percent of your emissions. But we have something. And we also know that in the backend, anything is actually cloud. Like Databricks is a new scope, but it's in the backend, it's still public cloud. Same goes for any resource that we consume via a public cloud provider. It's still running on a vm. Even your Elastic IP addresses are virtualized via a server in a rack in the data center. If we look at something that we have for our cloud providers to compute emissions of their EC2 VM, the equivalent of a VM we have for scope one and two for each provider. And so we can say for every dollar spent on one of these providers based on the data that we have, this is the carbon footprint for scope one and two of running, uh, that resource. And because we assume we'll always be under reporting, because we don't have scope three for all of them, so we're safe. And second of all, if we assume that everything in the backend is running in a data center on a rack as a vm, however it's exposed in terms of service, we can extrapolate that to not just EC2 or VMs, but also to databases and clusters, because it's still virtualization and something that is more resource intensive will also be more expensive. We can assume that if it has a higher dollar cost, it also has a higher carbon emission. That way we can create a report where we actively, without very much, very little effort, say this is your dollar cost and based on the data we have from the providers, and we're happy to improve that as we go along or as other data becomes available, this is the emissions of, of you being in the cloud. Are you aware of this? These are the recommendations that we have open that might save your team a thousand, two thousand, three thousand or more dollars. But they also make it so that you have a few kilos less of carbon emissions so that you don't need to plant extra trees to offset that emission.

Speaker B: Let me jump on that too, because you're using the LLM to do it. Got me to buy in a bit to this because when if you told me, joe, your app is costing or it incurs so many thousand carbon kilos, I generally don't know what that means. I know larger carbon is bad. Call me ignorant, that's great, I am in this space. But your LLM converts that into an understandable situation.

Speaker A: Yes, it's great that you mentioned that we take those almost mathematical mystical numbers of MtCO2e, so metric ton carbon emission or carbon dioxide emissions, we take that and basically use or leverage an LLM to make it tangible for people. Now I want to preface that with saying that even there we don't trust an LLM because maybe one day it will say a kilo of carbon emissions is five light bulbs and the next day it'll say seven. The numbers mean nothing to an LLM. So what we do is we've done the research based on studies that we could find where we have a table that we equate that kilo to different real life scenarios. So let's say one tree, five light bulbs, ten cell phones charged, half a mile, travel by car, whatever. And that table, we again as I described of how we slice and dice and prep the data for an LLM, similar to how we get those data points into a text, we also get that sustainability data via that table into a selected scenario that we then leverage, um, AI to make more sense of to describe behavior. So that we hope that if teams go home and drive around and look around them, um, and see. Okay, yeah, I ignored a recommendation today. Yeah, that's open. I have tickets, I have features, I have things I need to develop. I'm going to ignore that that behavior Might go away. They then drive home and look around and say, huh, um, these trees are working to fix what I'm leaving on. So we try to make that super tangible. And we do notice that for a lot of people that was an eye opener. We had a, ah, hackathon. Um, one of the great things about J and J is that they really promote innovation and pushing the needle, so to speak. And we participated in a hackathon where we. One of the criteria for the hackathon was you had to use disruptive technology. And so you had to use something like data fabric or Genai or. That got our, uh, blockchain. And we wanted to do something about sustainability, but we were very aware of, hey, it's not really matching our message if we put Genai into this. And so we were going to go into this hackathon with a presentation saying our use of disruptive technology is not using it at all because it doesn't fit sustainability. We ended up not doing that because of how we implemented it and how we were able to prove, like, if I had to do this manually, that was a certain emission. But our slide deck was just like five slides, with the first slide being last month. This was our cloud spend, which equates to this emission, which equates to this amount of trees that we'd have to plant now that we then have to grow for 10 years in order to offset last month's carbon emission. And we're already halfway this month, so good luck. And it was like a lot of trees. I want to give you a number, but like a significant enough amount that people were like, oh, yeah, we were not aware. And if you then are able to show, because every organization has these statistics, this is the identified open recommendations and you're able to say, but wait, this is on the shelf right now for you to take action. Yes, sure, it's dollars, but are you aware of what it could do for our carbon emissions? That's when you get people intrigued because suddenly they have skin in the game.

Speaker B: Yeah. Not only do you need to plant X number of trees, you need to keep that number of trees alive for just for that last month. Yeah, that's pretty cool. All right, we're going to leave it at that. Thank you so much, Benjamin. This is a great conversation. I always enjoy talking with you and thank you for sharing today.

Speaker A: No problem. You're welcome.

Speaker B: That's where we're going to wrap it up today. What I really appreciated about this conversation with Benjamin is how grounded it was in reality. This wasn't an uh, AI will Magically Fix finops Discussion it was an honest look at what it actually takes to use A.I. uh, responsibly and effectively. Requires deep domain knowledge, intentional data modeling, and a clear definition of what good looks like before you ever put an LLM on top. I think the big takeaway here is that AI doesn't replace finop's judgment, it scales it. When you do the hard work up front to define ownership standards and meaningful signals, AI becomes a way to give more teams access to better insights faster without turning finops into a bottleneck. If you're thinking about how to scale finops in your organization, or how AI fits into the journey without losing rigor or trust, there's a lot here to reflect on. A huge thank you to Benjamin for sharing how they're approaching this, and for being so open about what's working and what still isn't finished. As always, thank you to Stacy Case for helping make FinOps pod possible and for keeping us moving forward. We'll leave it there for now, but until next time, keep on finopsing.