Navigating Build vs. Buy Decisions in Emerging AI Technologies - ML 180

Welcome back to another episode of Adventures in machine Learning. I'm one of your hosts, Michael Burke, and I do data engineering and machine learning at data Bricks, and I'm joined by my amazing co host Ben Wilson. I write LinkedIn posts about open source releases at data Bricks. So Ben, if people want to contact you, should they do it through LinkedIn?

And what should they say? Funny? What we were talking about before the recording. Yeah, I'm a friendly guy reach out.

I do respond. If you're trying to offer me a position in another company, I probably won't respond to you. But if you just want to talk ask some questions, I always respond. Cool.

So you audience have just been directly lied to. But if you ask the right questions and have it be funny, he might respond. Well, see, it's all about time. Like if it's if it's something that's clever, then I'll make time immediately to respond, or if it's something about something that I work on that's actually fundamentally broken, I will respond immediately and.

Try to fix it. But if it's a it's just general chit chat, man, I don't It doesn't spark joy and does not give me a grin I might wait a couple of weeks to respond hard cool. So today's topic is something that I have been seeing a lot in the field, and I think if you work at a company, you probably have seen this sort of antics in line of questioning as well. So there's this thing called jen Ai.

It apparently is super cool. It does your job, it takes your job, and it also talks and processes image and all these amazing feats of sci fi. And so what we're going to talk about today is you're the CEO of a company. You heard about this magical thing called Jenai, and you want to see how you can integrate it within your business and specifically, how do you scale this decision making process so that you actually have usable, production ready features that are not maintained to are not a nightmare to build and maintain.

And you also want to do this in a finite spike based research process so that your employees are not just spending one hundreds of thousands of dollars and tons of time in prototyping and exploration. So Ben, your CEO of a hot dog stand. You have a bunch of hot dogs and you have franchised throughout the New York City, area, and you have a machine learning team to predict demand, you have a product team to handle branding, you have hot dog suppliers and all this amazing infrastructure, and you want to use Jennai.

How would you for hot dogs? How would you go about starting to answer this question and specifically delegating key components of decision making to your leadership. I mean I would always start off. I mean, if I was CEO of this company, I would probably talk to the leaders in tech and in the business side to strategize about.

Does this make sense for our business? And if I'm CEO, I probably wouldn't know that much about the technical details, so I would defer to somebody who does know, make sure they're in the room, and give them fully way to tell me if I'm being an idiot or not, and start the conversation like, Hey, explain this thing to me. What does it do, what doesn't it do? What or its capabilities?

And how does this impact our business? Go think about this, research it and come up with a plan for me about what are some options. We could do. And I would wait for their responses and then discuss it with them.

But I'm not in that role. I don't ever think I will be in that role. I'm more on the nerd side, so it would be more of getting assigned a task like that, say like hey, go build something with Jenai, and that process would be on the receiving end of that previous statement of what could we build and does it make sense? So it's called you CTO.

Then CEO says, hey, let's use Jenai. I want to be advanced and cutting edge. What do you do? I go talk to whoever the people are that know most about that.

Even as a CTO, you're not You're not cronking out code all the time. You might our CTO does. But you go and build a working group to go and discuss this, say, like, what are our options from a technical perspective of what theoretically could be done and make sure that everybody's on the same page with understanding what the limitations are and what do some scenarios like some some working group scenarios like Okay, we're going to like theoretically we're going to try to tackle this project within like this type of scope.

What it like, what would we need to build? And what are our unknowns? Like do we know how to do? Like out of these thirty seven things that we need to do to do this hypothetical project.

How many of these have we done before and how long did it. Take to do these? So we need to figure out, like deployment, where would this thing run, whether we're deploying our own custom model. That would be a large infrastructure discussion and you would need a lot of people in the room.

To comment on the technical feasibility of doing that. Well, I think that we skipped an important step like finding the business need and sort of understanding what Jenai actually is. So let's assume that our company, it's a hot dog stand we don't have that much technical capability. We like, know what Jenai is, how would you go about researching the different applications, the different use cases and how could actually benefit our business?

So that would be within the technical discussion. But I would always for something that's new, you always discuss that in tech first. You don't involve the business first, because the danger of that is you get some really fantastic ideas that the business comes up with of like, hey, we could solve these problems that are really hard for us to solve right now. But if you don't know what the capabilities are of what you're going to be doing or how you would build it or how you would actually deploy this thing, and what does the maintenance life cycle look like?

What are the performance considerations, like, what are the solas for this thing? What is the availability that we need to have? Is it capable of doing this? So having the technical understanding first in sort of a simulated environment to discuss this stuff that prepares you for that next phase of Okay, now let's go meet with business and see what problems they want us to solve, and we can tell them uniformly as a group what we think is possible and what is impossible.

That's super fascinating. I've actually legitimately never heard that before and it makes a lot of sense. So starting with a deep understanding of what building, maintaining and using that tech would look like, and then you come with that knowledge into a business meeting and say, hey, what problems do you guys have? Can we apply this tech to those problems?

Yeah, because otherwise you're going to be holding the bag as a tech organization of like, hey, the business wants to do this, CEO, board of directors, whatever, they say, we got to build this JENNYI thing and then the tech team is like, Okay, sign this to us, and then they spend weeks like trying to learn this thing, like what. Could we use here? Think it's a it's a cool problem that we should really solve. But you're pre.

Loading the tech team with the assumption that they need to build this thing and get it working. At all costs, and they'll do that. But if you don't know whether it's possible or not going forward. Or what the the possibilities are of the tech you're now potentially setting yourself up for massive amounts of scope creep or just blazing forward and solving a real business problem with so many unknowns that your initial you know, best effort guess at the complexity of like, oh, it's probably.

Take us like four months to build. We're going to hire some consultants to help out for like staff augmentation, to get this going. And then you find out at month three that like, WHOA, we didn't know that all of this was going to turn out like this. We didn't know that it couldn't do X, Y and Z.

We didn't know that we needed to build a, B and C. We had no clue that like this is how much the. System is going to cost to run. And when you're at that point, you now have to make a really difficult conversation with you know, C level folks like you have to tell the CEO, like, yeah, we we just burned three months of our dev team's time into solving this problem, and now we realize that it's going to take another five months and maybe we have to ingest.

Or purchase additional data to make this work. We have to build all these ETL pipelines to figure out how to deploy our solution in a more scalable manner. We've never done that before and for this type of thing. So everything just becomes like a research project and your deadlines.

You just blow through your deadlines. You never have a product that makes it. And at some point the executive staff is going to say, why have we spent all this money and we don't see anything. You told us that this is going to take four months.

We're at month seven, like what is why are you all so incompetent? And the tech teams like, we didn't know that it was this hard, or we didn't know what we didn't know. At the time. So if the tech team is allowed to at least.

Evaluate what it is that they could be building so that they know, yeah, we can build this, and here's a bunch of stuff that we know it's just not possible to do. Going into that business meeting for that ideation session. Of what could we do? You can head that.

Off the pass and focus on let's just talk about the stuff that we know. Like, we've done some research, we know that this is possible, it's proven. As you were talking, I was thinking that there'd be a ton of scope creep when seeing the art of the possible from just a tech perspective. So if you're not looking to solve a specific problem, you can say, oh, we can try this, or we can do this, or we can serve it on our grandma's basement, or we can have it write all of our code and see what happens there.

But as I was thinking through that, I think, as long as you understand the atomic components of the technology and the basics of as you said, how do you serve, how do you query? How do you interact with? How do you build? With a lot of those fundamentals can then anchor these decisions.

Do you agree or do you think that lacking a set of initial business problems would lead to too much ambiguity. For a useful exercise. You have to think about who's in the during that discussion. So it's not a data science team or an mL team.

Or AI whatever you want to call it. It's not just a bunch of ICs that are sitting around the table, you know, brainstorming and think about all the art of possibility that's out there. You have a tech lead, you have a manager, a director, and the CTO that's in that room. That CTO is not interested in, well, what's the tech stack that we should use for actual serving this thing?

Should we use like Kubernetes, on like managed aws, or should we use like fargate for V That CTO is going to tell that person very politely or very not politely, please stop talking. We're not here to design. We're here to talk about what is the bigger picture of do we know how to do X? Do we know how to take some code that's written in some language and deploy it in such a way that I can send Jason to it and if I get burst traffic, it can horizontally scale, or if I need to deploy a more complex version like an agent that needs to call you know, Python tools, where are those Python tools executing?

Do have we done something like that before? Do we know are we going to use like aws Lambda to do that execution? Are we going to use what Ben and Michael just released yesterday Unity catalogs, or do Python function tooling execution? You need to understand that tech.

And if you're and you're just going through a list of what is the application development life cycle and then the deployment life cycle, and say like, do we know how to do this? Have we done it before? Has somebody else done it before? Is there tech out there that we can use that makes this easy?

And what do we do when this goes wrong? How do we redeploy? How do we. Change the characteristics of this?

What's the devolop for building one of these things? How do we do monitoring? There's a lot of questions that come up with that sort of analysis, But what you're really trying to figure out is do we know how to do this? And are we comfortable of providing general scoping of this problem?

And it's not a oh, if you have a no there, then we can't do this project. It's more like, okay, out of these thirty seven steps. We have ninety percent that we're golden on. We know we've done all of this stuff before.

It's trivial, but hey, maybe there's a couple things in here that we don't know how that works. So right after that meeting, you assign people sprint tasks, be like, hey, go build a demo, like just try this out do for us. We call them customer user journeys, like hey, go and try to build this thing, and the other places you might call it a hackathon, like just go and try to build this. You're not writing tests, you're not making sure that you can actually deploy it.

You're doing it in like staging environments or dev and you're just trying to see if you can make it work. And in the process of that you might be like, yeah, I didn't make it work, but I know what I need to do to make it work. So it's a discovery process which gives. You information about what that scoping will be, Like, well, I couldn't figure out how to get this like proxy to work right, but I know how to do that.

It just takes a week to do. So okay, that's one person week to do that part. The rest of it Okay, it's four person weeks to do all the rest of these components for this one task. That's your scope.

Cool. As CEO of said hot dog company, I am very worried about my team sinking too much time into this. How should I think about the pareto frontier of knowledge gathering and the amount of completeness of the information versus amount of time and money spent gad that information. If you're the CEO and you're worrying about that, you should not be the CEO.

You should have your CTO be in charge of that. Your CTO should not be worrying about that either. They should have hired people that are now in positions of management that are making sure that their teams are time boxing all of their stuff, like there's deadlines associated with a I'm going to give you a day to figure this out, and that professional engineer knows that, like I just need to. I know what I need to do here in order to get the scope, and they should know what to do to learn that really quickly.

All right, let me escalate this a bit. Let's say that we have a bunch of I, as CEO, started this company a while back, and I hired a bunch of idiots. They have no idea how to write software. They are generally incompetent and don't know anything about Jenai, serving models, none of it.

So we don't know what we don't know. How do you operate and you just joined the company, You're a pro, you know everything? How do you operate in a space where you don't know what you don't know? Like?

How would you guide that team? I would if I just got hired in, and I'm sure everybody was so risk averse to learning anything new, my first task. Would be how do I actually replace the staff here? But like, I guess what I'm trying.

The motivation for this question is a lot of times I work with relatively non technical teams. They've been handed this addendum of you're supposed to use Jenai to solve a bunch of problems. Great, they don't know everything, and they also, more importantly, don't know what they don't know. So in that scenario, let's say you're a software engineer.

You've been incorrectly managed and given this task. How do you navigate in a space where you don't know what you don't know? How do you do that efficiently? You do research?

Spikes yourself in order to provide the evidence to go back up the chain to say we should shift gears or downscope this and release something that is that is doable, but you have to provide the evidence for that. So if you're good and you've done this before and you know how all this stuff is supposed to work, and everybody around you just doesn't know, or they're really eager to Yeah, let's dive in. Let's go as deep as we can and try to build this, and they end up building super complex, unmaintainable abominations.

You can be the person who's like, yeah, it's cool, like that's our our end state goal. But let's approach this from a position of let's just build the minimum required features at first, Like, let's build something simple and see how this works and we can all learn from this together and then when we're ready to increment to the next layer of complexity. So if we're talking about Jennai, maybe the first thing you do is just a very simple rag like, hey, we need additional context for question answering based on our business data.

Go build that agent, Like Okay, we're gonna you know, encode all of these vectors from texts and store it in a vector you know, search database, and then we'll connect it to an LM and we'll see how that works. We'll evaluate it, We'll do testing with the business, say like is this good? Does this answer questions about our business? Then let's deploy it and see how it performs.

Are we going to see stuff in staging on our own testing that we're like, oh, yeah, we need to fix that, or that's that's kind of broken. And then when you release it a production you start slowly ramping up users and see how people are using it, analyze their patterns and say are they asking questions that align to what that end goal everybody wanted to build it first is or are they asking things that are completely different? Do we need to adjust what our end goal is? So if you do, incremental complexity increases over time, your team is learning how to do that.

You're taking on. Way less risk because you're just pushing something out that's somewhat, like some a little bit simpler than that big project that you wanted to do. But you're also able to analyze user behavior, say like, do people have the right impression of what the capabilities of this are? And are they using it as we intended.

If not, then let's go talk to them and say what do you actually want to use this for? And then collect your user feedback and adjust your roadmap accordingly to try to solve that problem if it's possible. And if not, and this is this project going nowhere, then pop smoke and get out, like, go do something else, or build something limited in scope that's just for this one thing. Got it?

So it sounds like you just have to learn it. There's no shortcuts, like if you need okay, cool. But the only way to learning these these like processes of how to build something at minimal risk. The only people I've ever met myself included, who have learned this lesson usually learn it the hard way.

And if you're like a team that's a. Startup straight out of college, like you're all like buddies in a PhD program, you go and build something, you're gonna make mistakes, and you're gonna make these exact mistakes. You're gonna go and build something because you think everybody wants it. The number one goal that you should have in a team like that where you're young, ambitious and just want to build is get an advisor who's made those mistakes before.

Somebody who has experienced it can be like, listen, I made this massive mistake seven times before, or here's how I learned from it, and I worked with other people who also made mistakes and I learned from them. And this is why we have this formulate process of going forward, of not just trying to invent the end goal state of something that's super complex, because you don't know if people want that or if it's even worthwhile to build that. You could change directions midway through.

You could find out that, yeah, this is a total waste of time. Let's cut our losses. We learned a lot, so there's a benefit there. But should we pursue this to the end state because somebody told us to do it?

No, because what's going to happen is you release it and it's not really what people want. People are gonna they might not tell you to your face, even the C suite might not say that, but those conversations will happen about like, nobody's using this, why did your team build this? Why is this such garbage? Turn it off?

But then you've sunk a year of your life into this project that nobody cares about. Got it? Okay? Another question, if you were given this scenario, you read a blog, you heard about Jenai.

It's super cool, and you want to use Jenai. What do you think are the best applications currently as of December twentieth, twenty twenty four. The best bets right now that are proven that I've seen running in production. I've used.

Like a gentic rag with a couple of tools that are doing deterministic execution. So you define a function that does this one like this purposeful thing, and you're instructing the LLM like if you need to do this task, then this is the function you're going to use. This stuff works, it's been proven out and it's awesome. Some of the more sci fi stuff that isn't quite mature yet but it will be six months from now, is stuff like I ask a very obscure question to something to an agent.

This agent calls another agent to go and do some task, and then there's another agent that's reviewing that result and comparing it with the initial question, another agent that's going and fetching some data, and then another agent that's doing something with that data. And then you know, it's. Kind of like the edgentic frameworks that you can have the multi agent architecture in, but having that so that it's not so complex to. Build and can actually.

Like I just think the tooling isn't quite there yet, Like we're all working on this stuff right now to make it easier. Can you go and build all of like a very sophisticated, extremely complex agent yourself right now? Yeah you can? But how much code is that?

Like it's a lot? Yeah, this stuff gets super complicated. So until tooling comes up to the point where it's like, yeah, you can kind of do this and it's not such a burden on the user. Is the difference between Jenny I stuff and traditional mL or deep learning is we know that process.

We know we're not going to be retraining and redoing feature engineering once a week on a model that we deployed. We could be retraining and on new data, but we're not going to be selecting a new like. Library to do optimizations. And I can be like, oh, well this week, like we're using psyche learned last week.

This week, let's just switch that to x you boost and we'll deploy that. Nobody does that, I mean, if people do that, my condolence is good luck in your endeavors. I wish you all the best, and I hope that one day you learn from your errors. So it's a slower.

Process and you're not gonna be like, hey, yeah, the model's doing okay today, but hey, let's deploy a new version tomorrow that has like forty extra features in it. We can do that, right, can you? Yeah? How well is that model going to perform?

I don't know. I wouldn't try it. I would do a dev loop of like I need to validate this and I need to test it, and I need to do a b testing and I need to see how this performs against the existing one. It's weeks of work to do all that.

For the super complex stuff, that could be an entire quarter of work to release a new architecture of a solution that's already in production. With Jennai, you could make that code change and redeploy in an hour. So it's very fast, and lots of things can go wrong, like very quickly, right, and making changes in a complex code base that doesn't have good tooling built yet because it's still kind of in development. In industry in general, it becomes a big maintenance burden.

What happens if we have to rewrite our system prompt for each agent. How many places in the code do we have to do that and do we need. To customize that? Is it a sentence that we have to add or is an entire page of text that we have to add and then validate?

So that becomes a lot of work that you have to do to do this. Hard cool. So we're going to keep this episode short and sweet. But before I summarize, one other thing is that I've been feeling very strongly and I'm curious been your taken like super fast the buy versus built, I think, and also the weight versus build it Now, I think we're still in the infancy of a lot of these agentic logic based frameworks, and if you don't need it now, don't spend money on it until there's a good managed solution.

There will be. There's seventy startups all working on it. There's going to be convergence in the next year of something that's pretty good for logic based LLM based things. What are your thoughts on just waiting?

I'd say, build simple stuff that can solve an exact business need so that you gain an experience and understanding of how this stuff works. I would not recommend going to get hub and seeing a library that was released last week that seems like and it's read me, it's going to do all the things that you want it to do, and then go all in on that and be like, we're going to learn this library and we're going to build the craziest thing that's going to be amazing. Yeah, do that in the hackathon.

Learn it. It's cool, but don't. I wouldn't push something like that that fraud without thorough evaluation and analysis of it. And who knows if that library is going to be maintained a month from now.

You don't know what whoever's maintaining it right now. Are they working for a startup that is going to get bought by somebody and that repo is going away, or it's going to be abandoned because it's now going to be rolled into some cloud provider service. You just don't know. I would be careful about.

Building super complex stuff for purposes of production deployment, but agreed for purposes of the team doing akathons, the team learning this tech, you know, go nuts like learn it, play with it, break it. Fix it. It's going to benefit everybody who's doing it is whether the Luddites believe it or not. This stuff is here to stay.

It's not going away. It's not a hype. It's not like, you know, people are just like getting excited about this, but it's actually garbage. No.

No, people are deploying stuff. I've seen lang graft deployments at for customers, open source and data ricks customers that you look at what they're doing, you're like, yeah, that's that's legit, and yeah, it's it's complicated like some of the stuff I've seen, but it actually works because they have a team of like thirty people working on it and they're serious about it, and they did their homework and they proved that it would work, and they're building like a production version of it.

Yep, I don't think lang graph is going anywhere. Agreed. Cool, So the shortest episode in the history of adventures of machine learning. But i'll quickly summarize.

So, if you're given this prompt of use Jenai in the business and you are a technical person, start with the tech figure out what will take to build, maintain and use this technology, and then from there you can go into business focused conversations with an anchor in reality a great way to figure out what it would be like to build, use, and maintain. Create cuj's or customer user journey to identify what are the typical user behaviors and try to hack solutions. See if they're.

Scalable, see if they're cheap, see if they're easy to build and maintain. If you haven't done this before, you got to learn it. There's just no way around it. So just do research, read books, look at articles that type of thing.

And then finally, with the current state of Jenai, we're reaching production stability for really sci fi use cases, but right now some of it is not as robust as it maybe could be and should be, so just wait a little bit for those more logic based agentic based systems. But RAG, i agree with Ben, is very proven and is super super high ROI. If you're going to do anything with Jenai, focus on RAG anything else. Cool.

Well, until next time, it's been Michael Burke and my co host Ben. Wilson, and have a good day. Everyone will catch you next time.