Decision Logic: The Difference Between an Answer and a Decision

Speaker A: One of the risks of being a podcast host is that a topic will come up from time to time where you're pretty certain everyone else understands it and that you should too. And the temptation, for me anyway, is just to sort of nod along and hope no one notices. Well, today's topic is one of those where we're exploring the topic of decision logic. And the reason I opened up the way I did is because the term seems seductively self descriptive to the point where again, you'd kind of nod along thinking, uh, I probably understand what we're talking about. But does that mean it's then an overlooked discipline that might actually be one of the determinants of whether or not your AI program works for you? Well, according to today's guest, without it, AI is likely to generate insights that will go unused or misapplied. With her experience leading large scale transformations, I'm hoping today's guest might help unpack how organizations can move from abstract to operational outcomes. Welcome to another episode of the AI Forecast, proudly sponsored by the amazing folks at Cloudera. I'm your host, Paul Muller. Join me along with leading companies and industry experts as we explore the past, the present and the future of data and AI in the enterprise every week. So today we're joined by Darlene Newman. She is the managing partner at Ivy Captech Advisors, um, where she advises enterprise leaders on AI from strategy all the way through to execution, getting it done. She specializes and has a background in helping businesses by, you know, putting structure around operations, knowledge and decision logic. Welcome to the podcast, Arlene.

Speaker B: Thank you for having me. Appreciate it.

Speaker A: Now, having completely expressed my ignorance, um, you know, there's a little bit of shared vulnerability here. You're going to have to be vulnerable for a minute as we do the fast four, the lightning round that we used to start, start every podcast. Four questions. Are you ready to do it?

Speaker B: Yeah, let's go for it.

Speaker A: All right, let's make it happen. First question, what would people who know you best? They could be friends, family, work colleagues. What would people say is your superpower?

Speaker B: I get people through the messy middle. I work a lot. I work a lot.

Speaker A: Sounds like you're describing my waistline there, darling.

Speaker B: I work a lot in emerging tech or startups, and there's always moments of I don't want to fail or I want to go so fast that like, this is just going to break and you kind of have to get people through that. Where do I even go? How do I even start? How do you handle the challenges when you have to pivot and things like that. So I tend to be that, that right hand of just guiding people through very unknown territory, which I like to call the messy middle, because you always hit a part where it gets really messy and you just either abandon ship or you start doubting yourself.

Speaker A: It's where the work happens.

Speaker B: Exactly.

Speaker A: The most impactful technology of all time. It can be whatever you want.

Speaker B: I'd like to say it's an airplane because I like to travel a lot, but I'd have to say it's in my mind. It's not a car, it's a plane. But it's the ability to basically create, store and share information. Right. Um, think about what we did with the printing press, the Internet, and whether you love it or hate it, social media, the mobile phone. You think of what AI is doing right now. It's just taking it up a notch. But like being able to really, like think about using Google Maps. I mean, I used to pull out a piece of paper and look at an atlas and try to get around. And we've been able to kind of restructure that information and share it in a very different way. So, you know, it's not specific, but it definitely is to me, kind of the most intriguing aspect of new technology.

Speaker A: Somewhere there is a millennial listening going, what's an atlas? Moment.

Speaker B: I probably still have one under my car seat. Somewhere there's someone.

Speaker A: So I've told this story before. I remember, uh, jumping into a Volkswagen Golf in Germany many, many years ago as I was traveling around the world and, uh, at one of the car rental places, jumped in, start driving. And I go to look down at the dashboard and there's a picture of a three and a half, three and a quarter inch floppy disk. Was a half a quarter. I can never remember.

Speaker B: I don't even.

Speaker A: As an icon. Right. Which is where you put the USB stick. Before people thought of using the USB stick, weirdly, the weird USB symbol as a, as the icon. They had a picture of a, of a disc. And you sat there thinking to yourself, someone's going to be jumping in this car going, what is that?

Speaker B: And they probably already do because, like, nobody knows. Yeah.

Speaker A: What would you like to see AI automate for you in the future? For you, not for everyone else, just for you. If you want be Grady.

Speaker B: I'm going to actually use it for my work because I'd really like to automate tribal knowledge because, like I sit daily trying to figure out, document how people make decisions and it's so painful. Like I'd love to be able to just tap into the brain and just automate it because it is, you know, nothing's ever a straight line. So I'd like to tap into people's brains and extract some of that information.

Speaker A: I love it because that's actually, in many respects, if you think about two companies who are doing the same thing in the same industry, to a certain degree, what differentiates them is part of that. The culture, the tribality of how they work. Um, uh, yeah, very fascinating idea. All right, last but not least, you work with data a lot, um, and have in the past as well. What are some of the best practices? Hacks maybe, or even just tips or tricks you've got for folks. As someone who's worked a lot with

Speaker B: data, you have to start thinking of data very differently. It's not just what sits in your databases, in your relational tools, like the data you extract and report on and kind of what you put in your SharePoint as your operating procedures. It's also how you make decisions off of that. That is the new data. It comes back to a lot of what your decision logic is. It's really like, how do you now how you make that decision and the why of it becomes data.

Speaker A: So let's talk about the term decision logic and we'll, we'll. I'd love to talk a bit about your career as well, but it's, um, a, uh, it was one of those topics where literally I looked at it and I thought to myself, do I just roll into this podcast and kind of effectively do what most blokes would do, I imagine, and just sort of go, yeah, I know what's going on. And again, an executive, you kind of get to the age you do because your inference engine is usually pretty good, right? It's fairly well dialed in. And even at a cursory glance I thought, I have no idea what we're talking about and, and, and what the discipline versus the, um, sort of the pastiche version or interpretation of it is. So hit me. Decision logic. What are we talking about? Why should I care?

Speaker B: So you have an LLM, right? And it generates really awesome stuff, right? But it's pattern matching on text, text that it's trained on, on the Internet, which is probably all not great anyway, but it really is just pattern, pattern matching text. Like, I don't know why anybody would think it's intelligence and it isn't right, but it is going to change how we do our work day to day. Like, think about what it will change and just how you Interact, Right. Keyboards, um, my verbal, like you know, a chatbot will be doing stuff that you're asking it to do, so to think about it. But in order for it to know and have any sense of reasoning, you have to help it understand that. So I spent a lot of time in the contract management space because I think it's phenomenal use case for LLMs. Right? But you think about it, it's like, okay, well I want to know whether this contract has auto renewal. Well, does the term may auto renew, shall auto renew, will auto renew, tacit renewal, do they all mean the same thing? So when someone says, oh yeah, this is an auto renewal contract, it's like, well how do you know that? So I look at decision logging, uh, it's not kind of like you're operating your procedures and your flows, but it's kind of like the why of what you do. Right? How do you know something when you put it in? Let's say I'm entering data into a change ticket order. How do I make that decision of what that category is or that drop down value is? It's that tribal knowledge that people just know instinctively to do. Well, in some ways you have to tell that to the LLM, otherwise it's going to make its own decision. Right? It's going to do and it may not decide. So you think of that and a lot of people have a lot of definitions of the semantic layer. M They're like, oh, put context into it. Well, that's what they're telling you to do. They're like, put context into that prompt to guide that LLM on what to do. So I'm a contract reviewer, I'm looking at this, but think of pulling that out as a semantic layer of defining that decision logic in a way that an LLM can access it and any agent could access it. And they're understanding it in the same way. When people talk about decision law, they're usually talking about the semantic layer. And some people will also think a spreadsheet and rules in a spreadsheet is a semantic layer. No, I'm talking about true knowledge design where you've got controlled vocabulary, you've got ontology, and you're pulling it all together probably in some kind of a knowledge graph. There's a lot going on right now about context and understanding context. The LLM, um, we all know needs to have context. I'm saying pull it out from being inside a prompt. Put it in something that is truly designed for an LLM to traverse and look at, um, to Give it that understanding of what it means and it can be auditable. That's the other thing is how do you make it so when it's answering your question, it's not inferring from a text, it's actually saying, here's the exact statement. And it's a yes, because it says this. How do you make that not only scalable and audible, but you know, something that you know you can reuse.

Speaker A: And again, this is a lay person's understanding of things. It sounds, some of that language sounds vaguely reminiscent of what we would have been talking about with expert systems if I was to go back 25, 30 years.

Speaker B: Yes. We're just kind of. It's now coming into more focus. Like you think of the Semantic Web. It's been around forever. Wikipedia has been around forever. It's kind of like getting its second look life right. Because of LLM, it's getting a second life right. Like I didn't, I wasn't, you know, educated into this area of ontology and whatnot, but I did go and learn and understand because like I was seeing the problem. So went out there and really educated myself, certified in it, have been building it. Right, because you realize that there's just fundamental problems with LLM and what's going to solve it. Right. And everybody's trying to solve the problem, but you kind of go back into how knowledge design has been around forever and so now it's just getting a second life and a purpose. So I think there'll be a whole lot more coming out in terms of the semantic layer and ontologies and controlled vocabularies and knowledge graphs, you're going to start hearing a lot more of those things because we've kind of realized that you could put a lot into a prompt, give it all that context. But why would a. Why would you silo it? I equate it to. Have you ever heard of key person risk? So in the financial industry, you have Cobalt developers, there's probably 20,000 of them in the world left. Like 90% of our transactions go through Cobalt code. So there's a key person risk that you have someone who's about to retire and they're your last Cobalt developer. Well, by putting your context in a prompt and it sits within that prompt, you now have key agent risk. You have to know that that context of how they define what an auto renewal term is is inside that prompt versus inside, uh, a, uh, manageable layer, foundational layer, if that makes any sense. So your definition of what that clause means, how you Identify it in a contract. What are alt names that we might give it synonyms? What's the scope of that term? So it's kind of like building all those concepts around what a contract has, making it available to anybody who wants to build an agent to do things, whether it's assessing for negotiation, whether it's doing, putting the data in your contract management system. It's kind of like taking that out from a uh, prompt because every M vendor right now will say, well we sort contacts, here's a text box, or we pull it from your SharePoint and SOP and it's like, well, hold on here. This becomes another kind of foundational layer that you should have beyond just your data that you store in your systems of record. It becomes your decision layer that is very datish. Right. Where it's like it taps into it.

Speaker A: Yeah, More logis than um, pathos, for want of a better term. So we're talking about being more um, well, logic driven. The hints in the name. Paul One of the things that I'm uh, sort of casting my mind back to is we had first, very first podcast guest, I think first episode was Ray Wong from Constellation Research and he talked about the idea of precision AI and his definition of precision, or maybe, sorry, let me rephrase his framing of the term precision, was the notion that precision is the sort of precision you want from a surgeon who's operating on you or from an aircraft that is making a turn from its autopilot. And it again sounds to me like what you're extolling the virtues of is taking the very messy fuzziness of the statistical side of LLMs and AI as we think about, well, as the layperson would think about it, and the Boolean rigidity of uh, well, Boolean logic and getting us somewhere in between that says we can't afford in high stakes situation. Obviously you've got. You're in New York at the moment, is that right?

Speaker B: Yeah, Yep.

Speaker A: Yeah. So you're in the center of like one of the centers of the financial universe. You can't get it sort of right with a contract. Right. You've got to get it right or it's wrong. Like at some point in time it'll be up in front of a judge and they're going to go, I don't know.

Speaker B: And then how did the LLM, uh, make the decision? Because right now someone's putting it on saying, well, how risky is this contract? Well here it gives you a bunch of lists of what makes it. Well, how did you define it? It's inferring it, right? No, I want it to give me the exact sentence and say, was it a yes or a no for this criteria? Right? Like, did it. Does it have, like, a, uh, termination clause? Yes or no? Does it. You need to know. And so it should pull that out. And I wrote an article about this recently that I'm also taking Italian. And one of the things the Italian teacher is having me do is, okay, well, this direct object, tell me what it should be, but then tell me in this sentence exactly what makes that true. So I had to go write what I thought, and then I had to go find in the sentence what that direct object was related to. And I thought, well, that's a great way of thinking about an LLM and how you want it to be able to say, okay, I. I'm saying it's this. In this bucket of hundreds of pages. And I'm saying it's just because of this, right? And it pulls that exact. And it said it met your criteria. So that's why I chose it. And you can clearly see in that auditable output that, yes, it met that criteria. And here's the exact statement that made it true. And that's kind of how I look at what you need to do from decision logic is what do you. What does it have to be able to understand to make that true and give me the exact output of it?

Speaker A: And so going from the discussion, going again to those two sort of extremes of Boolean rigidity and the fluidity or ambiguity, if we want to use a better term, of LLMs, the risk associated with it, uh, of misunderstanding the go to phrase at the moment is human in the loop, right? I've just put a human in the loop. But human in the loop doesn't really solve the scaling problem. You know, to put my Eli goal rat hat on, we've just created another constraint. And so what I'm getting the vibe on is that decision logic, again, is a mechanism by which we can take some of that tribality, some of that human in the loop, not all of it, but for a defined set of cases, start to build that in. What sort of talent do I need to help me with that process of building that out? Because I'm presuming that person indexes more to business process understanding than they do to technology. Would that be a fair statement?

Speaker B: More of your, like, you know, I equate it to, like a librarian, right? Someone who's organizing information. So I always say there's two really good skills that you need in Terms of putting an agent together, one is business process and the other one is understanding the data enough and you blend those together and someone can actually do the knowledge design of it, right? They're not pure engineers, technical, like I'm building the agent, but it's like someone who can understand structure of the data, what drives different data output, input, what kind of causes things. But then someone who understands the business knowledge and it really is more of a process librarian type person, more so than it is an engineer, right, who understands the tech, but you still need a tech person, kind of do some implementation. So I feel like it's kind of a bit of two people. I think what a lot of times what will happen is someone who's designing it, if they're too down the path of technology, they're just going to structure their vocabulary around what's in a system. So a lot of people are like, well, we've got Collibra. We have data definitions on all of our fields and we map all of those and have the lineage. I'm like, that's not a controlled vocabulary. Because if you look at a majority of information that's decided on, let's say a contract doesn't even sit in a contract management tool, right? It may just be the financial stuff. It wouldn't necessarily be every term, right? So it's kind of like, okay, well that's great that you have the definitions in here, but it's the concept. What does the concept mean? It doesn't. Just because I'm a definition doesn't tell me what it means and how it should be interpreted and how an AI would identify it. So you kind of have to take it even a little step further that you don't want your, your concepts and your understanding to look exactly like what you have in a relational data model, right? And that's the risk that you have if you go too technical on defining this semantic layer.

Speaker A: I think I understand again, the words with the minutes I hear the words ontology and semantic layer, I immediately think I need to go back to university for a little bit. I'm certain those words are designed to scare me off. When we think about being able to package work up for the consumption of another, you know, some of that rigidity and that logic becomes really important, right? Because I need the work to arrive on my plate in how do I put this with a degree of certainty that it's passed through certain gates and that it's in a certain form that it's worth me doing work on it. Now if that Makes sense. And that's as true of traditional human workers. It is of sort of agentic concept. And I bring up the term agentic because I imagine that decision logic and agentic. Well, everyone's super excited about this idea of agentic at the moment. But I have a feeling without a structure like decision logic in it, a lot of people's worst fears about agentic are likely to come true. Thoughts?

Speaker B: It gives it its guardrails, right? Because what will happen with an LLM is if it doesn't have those guardrails and it doesn't have an answer, it'll just make one up. And that's kind of where hallucinations come from. Right? And so you could say, okay, if you don't know, just say you don't know. But it's like, well, no, just go down this path, right, and have it follow specific instructions. Right. And it gives that guardrails. And as long as then your output and some of those guardrails make it obvious as to why it made that decision, you get closer to automation. Like true autonomy. Do I think we're all going to be running on like, is there going to be a single kind of company out there running autonomous? No, because it's really easy when you start out to have simple processes, but as you get bigger, more edge cases. I can tell you right now, I don't know any large organization that is going fully autonomous anytime soon. Right. Like, uh, you know, sure. Age. No, like, yeah, I. I'm just in it to know that I. You'd have to define. I think there was a. An ontologist that I was talking to. He's like, the ontologist will be the last man standing. Because once they've defined every process that people do, then you can do full autonomy. In some ways. It's true that you'll be able to get rid of certain jobs, but you'd have to just define what that AI does and then it could then start building on its own. But if it doesn't know, it doesn't know. I equate it very much to. If I was to hire an intern, I have to give it. They're going to have questions like, I have people cleaning data. They're like every day. I have a question for you. I have a question for you. Well, an LLM is just going to not. It's going to answer those questions itself, right? And you don't want it with confidence, you know. And I always tell developers whenever I build them, like, you know, when you offshore and you hand them the Requirements and they're like, come back m. Like, if you have a question, please ask. Because how many times you think you're right on making that decision and you are wrong because a client will always have the opposite decision. Right? No matter what you think. So it's like ask the question. An LLM is just not going to ask the question. It's just going to keep going, right? And then it's going to go so far down the path that you don't even know where it broke.

Speaker A: Let's maybe a quick call to action around decision logic for people who uh, like me, who again would probably just nod along when they hear the term or may have passing familiarity with it. What should we be doing about it? Where does it sit in the pantheon of my AI programs? You know, whether I'm at a board level, at an executive level, how should I be thinking about decision logic? Is it. It's not, I'm presuming it's not like a board level thing. We need to have a conversation about where we're at with decision logic. Who owns it, what should I be doing about it and when in my process should I start to think about introducing it into the vocabulary of my project?

Speaker B: When you're starting the project, that's when you start, right? In my mind you are when you are designing that agent, right? Because, uh, what I'm finding with agents, it used to be that you could kind of get away with like minimal requirements and kind of agile build, but I feel like you're having to go to far much more specificity in terms of what needs to be done because you're really having this agent do every single task. And every task well, there could be an edge case. So how do you handle that edge case? Does it go to a human or can it make its own decision and move forward? That's where you starting, right? What is it? So I always tell people, start with your process flow, but you're not going to build your process flow with an agent, right? You're going to define the activities that are critical in that process because everybody's going to redesign it. You don't even know how this is going to work with an agent. You really can't sit there and say, okay, well this is how we're going to work with agents you don't know. So don't try not to pretend you do. Right? But what you can do is say, here's all the things we do in this flow. Here are the core activities. Which ones make the most sense that you can Do. And at that point, what are the tests, what's the input, what's the output? And how do you make that decision for that output? And you start documenting how you do that. So it starts at the beginning and when you're going to the board semantically or isn't something people are like, oh, yeah, let's build it. They're like, oh, yeah, we want a data lake. Okay, that makes sense. Awesome, right? Oh, I need the front end. Awesome. We get that. Okay. We're going to do. We need APIs and an API marketplace. Okay. People get that over time, they're going to start learning that they need a semantic layer, right? They will. The way m I've done with some clients is we just kind of had to prove it, right? So we started off with one agent. The output was terrible. When we started putting a very simple, controlled vocabulary together, and in just like defining what these things meant and then having the agent go to that spreadsheet, we could then prove that it was a little bit better. Then the rules had to be put. We had to describe in order for it to get better. Well, essentially we were just doing a very, very basic semantic layer, which is what people are doing now. But now we're formalizing in true knowledge design and open standards and things like that. But it was something we had to kind of do behind the scenes because this isn't something people are jumping on board. Right. It's hard to understand. Like you're saying, you kind of question it. This isn't an easy topic, right? You can kind of get a database and a relational database because you play with spreadsheets, right. And if I want to join things, I have a unique identifier and a primary key. Secondary key, Right. When you're talking, this is just, this doesn't just come naturally to people to go, okay, let me take my business and turn it into concepts. And then what is that? What are the rules, the object of the subject, and what causes those things to connect in those relationships? Like, people just don't think this way typically. And then, you know, when you get down to, well, do I need a knowledge graph that pulls, pulls all of this together? Right. You know, I've had to actually show what that looks like more than explain it, because it's, it's. They're just not easy concepts. But right now it isn't. People aren't just funding this. Right.

Speaker A: But it also sounds like as a result or as a consequence, instinctively, but based on no, uh, experience, that it would be one of those motherless children in a company as well. It's like who's, who's got to take response. Is this the person initiating the project? Is this one? Is this a technical thing? Because it sounds pretty technical. You've just used two words I've never heard before. Yeah, um, you know, I'm thinking like a board person. Right. They're like I'll send you to the person. You sound like you've just used two Scrabble words. Um, yeah, I'm going to send you to the CTO and they can work it out. Right. Like where does this thing go? Um, and as a result I can imagine it being neglected somewhat.

Speaker B: Yeah, you're very advanced. Um, firms do. We'll have ontologists in, in the firm. Right. That are defining um, kind of the structure. But those are your very like salt like your Netflixes that have really good, you know, your Google's, your Amazon. Right where they, that so they understand it but your, your, your standard like enterprise doesn't. Right. And so it'll be interesting to see where all this is because I think AI itself is having a hard time finding a home, you see. Is it with it. Is it with a business? Is it a separate AI enablement team M Now you've got this added layer. Well, it's data, it's knowledge data and that actually should sit with the business. But then the governance around it should sit with it because it's going to sit in a platform and it should have. Because there's code involved. There should be standards around the code and you don't want to like concepts shouldn't overlap. So legal should have the same concepts we do because now you've got conflicting conflicts and now your AI is going to have more problems. So I think there's a whole lot of uncertainty right now because we're moving so fast that I can't even tell you where it's all going to sit. I do know that there's got to be a huge partnership between the business and tech and data without. There's got to, you know, there's been the term for deployed engineer. I do feel like there's going to be some movement where the platform and the governance and the controls around the platform sit centrally somewhere within probably it. Right. The actual build out of the knowledge has to sit with the business but it needs to follow standards and it has to be. If you're going to, let's say do your operations, it's going to have to be somewhat controlled from a, uh, you don't want it to Overlap and you want standards and some of the agents still have to be built. So I feel like there's going to be a shift of kind of a, um, not fully federated, but like a hybrid model of like certain skills moving out towards a business to help build on a platform that they've been given that's controlled by the center. But I feel that that's kind of just where things are headed because you also have low code. You have things that you can do that the business can now do, but you don't want them going rogue. Right. And shadow AI everywhere. But I do feel a sense of much of this has to sit with the domain expert. Even from prompt building. How many prompt engineers are in their itu? I'm like, yeah, like then you need to sit with the SME, because they're gonna be the ones that really should be typing it. Building it.

Speaker A: I was gonna say it'd be pointless sending it to it because unless, you know, let's just.

Speaker B: They have to know it.

Speaker A: You're a. Yeah. I mean a legal firm, for example. Well, why would. Unless that IT person's also a lawyer.

Speaker B: Yeah.

Speaker A: They have. It would be like giving them an intern and going like a legal intern, a paralegal and going here, this person reports to you for the day. Uh, I don't know what to do.

Speaker B: I mean, even projects have to change. I was looking at a presentation the other day from an implementation firm and in it, it was like requirements, unit testing and then for two months and then uh, a couple weeks of UAT and deployment. And I'm sitting here going, you're in a different world. Like that used to be the case maybe in, in like the digital era. I mean even agile, like still look at waterfall. But I'm like, if that agent is. If that unit test doesn't have the SME, you're wasting your time because only. If only that, uh, that business owner, domain expert knows that output. Unless you really have solid grounding on the rules and you're just verifying that the yes or no of whatever IT output. But again, you still like solid. So I look at it going, your unit tests are no longer the developer. Your unit testers are your business. Because how else it's not the same thing as putting in a rule in Salesforce and having it kick off that rule. Right.

Speaker A: Hm.

Speaker B: It's very different. So I feel like there's a whole lot of uncertainty and everybody's navigating it and as they should. I mean, emerging tech usually takes years to mature. We're adopting it at fast, rapid pace and it's not even, I mean, agents, what, a year and a half ago that word came out and LLM is three years. Like, we've got to. We're moving really fast.

Speaker A: Uh, I've been thinking about what we do with this podcast as we've been talking. Um, because it's fascinating. I would love to get you back on if possible and have just a podcast because we talked about decision logic and I feel like keeping the topic, um, constrained is worthwhile because it's been a really valuable conversation for me and I hope for our audience. I'd love to get you back on to talk about. We'll do a special episode on the messy middle. Because I think right now where I think a lot of folks are struggling is am I alone with everyone else? If I pick up, if I pick up the Wall Street Journal, Harvard Business Review, whatever. Everyone seems to be having a great time with this AI except me. What am I doing wrong? Um, maybe I need to fire my team and start again. I suspect there's a lot of people stuck in the messy middle and be great to have a conversation about.

Speaker B: I think there's a lot of, um, AI washing. Is that the term they're calling it? I ignore startups that are sitting there going, yeah, I've automated everything because I'm like, a startup's very different and my personal attacks are very different. But going into an enterprise, I mean, I know of firms that are still like, should I even start? Right? So I think there's a lot of people along the spectrum and I do feel like there's a lot of just hype that is not factual. And I think there's a lot of AI washing of people. Just.

Speaker A: I definitely want you to come back on because it sounds like I'm talking to myself here. Not that we want to have confirmation bias, but. Yeah, no, I'm, I'm extremely skeptical of it, especially. Just a quick side note, I, um, I was talking to somebody about this other day who was, again, they were creative and they were fearing for their job. And I said, look, I just, I was having a session with uh, one of the LLMs and it was, it was a procurement related thing. Help me find a list of suppliers who are good at this. And it happened to be, I have to be specific here. It was for avionics, right, For a plane. And um, it came back with this list of companies that I could talk to. I'm like, I've never heard of any of these. And I had My little spidey senses started tingling and I'm like, let me just Google this company. And um, the company was called say, um, Podunk Air Services. And I had a quick look, Googled them. They're an air conditioning provider. Um, and then I, you know, and the second and the third and I went back to ChatGPT. This happen, happened to be. No, no offense. Sam James representing said, it appears that all of these suggestions you've given have nothing to do with avionics whatsoever. Could you fact check it? And it comes back contrite as it is going, oh, yeah, it turns out I got that all completely wrong. And you just sit there thinking to yourself, that was a low stake situation

Speaker B: and it doesn't actually know it got it wrong. It's just agreeing. It's just agreeing with you. Right? Like it doesn't actually know. Like it doesn't even know it's wrong. Like it's just. Okay. Like an LLM, um, should be in my mind, just the natural language request and the natural language return. Unless you're truly generating a summary or things like that. I'll ask it in natural language. Then it needs to follow some guardrails and, uh, follow what it means. If I want to know what an aviation technology is, how do you define it? Right. What's your definition? Could have been in the prompt, like, here's my definition of an aviation. Then it's like, okay, and then I'm looking for firms that are this size to this size or what is it that I'm looking to get out of it? Right. So you start building in that context and it might get a little bit better depending on kind of what it's trained on. But again, like LLMs are trained on the world Wide web of whatever people want to say. Right.

Speaker A: Junk. Yeah, no, absolutely. If you would come back, it would be an absolute pleasure having you here. I've had an absolute ball today. I've learned a lot. Um, I feel smarter. So thank you for that. Um, uh, if people are interested in learning more about what you do, if they have a messy middle problem that they're looking to deal with, where can they go to learn more?

Speaker B: The best place to go because I don't really have a great website is I'm on LinkedIn. So darling M is my LinkedIn, um, handle. So, uh, I do post pretty regularly. I've been doing a lot of frameworks trying to answer people's questions that are challenging. So if I have a problem in a week, I tend to put it out there. Um, I try to be as pragmatic as possible and not just jump on the hype train, right? But, like, here's really what you need to do, right? Here's kind of what you're looking at. If you're looking at kind of the operating model or if you're looking at prioritization, here's what you need to do. So I would say my LinkedIn is by far the best place to get me because that's where I spend a lot of my off time.

Speaker A: I love it. The signal to noise ratio, they were still pretty good. I can understand that. Excellent. All right, well, that is a wrap. If you have enjoyed today's episode, and I hope you have, make sure to, like, share and subscribe. You know the drill. Uh, check out Darlene's LinkedIn. Uh, if you've enjoyed, uh, some of that content as well, you can also hit up. Speaking of LinkedIn, um, uh, sponsor Cloudera using the handle at cloudera on LinkedIn. We'd, um, love to hear your thoughts. Um, the team's got a great array of guests coming up, including, I hope, having Darlene back.

Speaker B: Uh, of course.

Speaker A: So, again, make sure you subscribe so you don't miss another drop. Thank you again to the amazing team at Cloudera for hosting another episode of the AI Pod podcast. Thanks for tuning in. Thank you, Darlene.

Speaker B: Thank you for having me. I really enjoyed it.