Integrating Business Needs and Technical Skills in Effective Model Serving Deployments - ML 184

Welcome back to another episode of Adventures and Machine Learning. I'm one of your hosts, Michael Burke, and I do data engineering at data Bricks, and I'm joined by my amazing co host Ben Wilson. I do quarterly planning at Data Ricks for the whole organization. Oh most certainly not for a one team.

That's just what we're working on this right now in the sprints and extprints. Do we like quarterly planning? Do I like quarterly planning? I like the idea of being able to collaborate with a team of super smart people who have amazing ideas, and then working through the process of distilling that down to what are we going to work on that's going to have the most impact in the least amount of time, And how can we sneak in all of the stuff that I worry about link stability and robustness of the service and making sure that we get bug fixes and stuff done.

Yeah, I heard signatures can break a lot of stuff, so I probably want to test. That a little bit. I think that was the biggest regression. It's definitely the biggest regression I ever caused in my entire career.

Was it was like almost three years ago with one of my first big prs and mlflow was adding support for additional signatures, and then realizing after release, like less than six hours after I released, that we had no test coverage for any of that stuff, and then realized that anytime you deployed something to model serving it would just detonate. So a quick scramble to fix what I screwed up and a re release of mlflow and all of that done within twelve hours. Who reported it? Uh?

Is it a customer? No, that was somebody in the field who has just happened to luckily be doing testing and they didn't pin their install of mlflow that afternoon, and they're like, uh, something's weird here. And I was like, what do you mean? And got the repro code whipped something up and I was like, oh, no, everything is broken.

This is not good. Yeah, panic, nice, extensive root cause analysis. It's the reason we do integration testing now. It's the reason that we have bug bashes, it's the reason we do release candidates.

Yeah. Yeah, we'll get to the official topic in just a second. But I was wondering this myself as I'm learning to write code. If all of this has just been learn the hard way by someone before us, and or learned the hard way by us.

Do you think that they're like axioms of design that you can take from other fields or is it just sort of an iterative iterative process where someone has created the exact same issue, learned from it, and now there are these practices like GitHub tests things like that. I think everything it's human nature too. We kind of open the box, poke it and see what happens with everything that we do. As a species.

We're very curious and don't always think about repercussions of things until after the fact, that reactionary response and also what that extends into with sort of thinking through like post talk, an event that happened, how could this apply to other things things that we do. That's definitely something that happens, and it's pretty common. But the first time something really nasty happens, it's usually yeah, we poked it with the stick and found out, oh no, this is bad, so let's make sure we don't ever do that again.

Yeah, exactly, Okay, cool. So today we are talking about model serving, and model serving has a bunch of different aspects to it, but we're gonna be using an interesting case study where we have a search engine and this search engine is surfacing hot dog recipes, and we have a variety of ways to make a hot dog. We can add mustard, we can add ketchup, we can get crazy and add chocolate, and we basically want to have the most robust, stable, low latency, and quote unquote best feature possible.

And this is inspired by a case study that I had to work on a few months back. It was super successful, but basically the problem statement that we were given is exactly that we have an existing search engine, we want you to make it better. Go So, Ben, how do you start? I mean I think I start the same way that you did.

And we talked about this project a number of times, well like, well we need design talks, we need to know like you know, you can go like the hile Meyer approach, where you're like, what is it that we're trying to build here? Leave all the technical jargon out of this, leave all the solutioning out of this. What is the product that we're building, what exists today? What do we need to do to make it better?

And that can help formulate our process of design later on. So I always start there it's like, why are we doing this? What is this going to build for us or gain for us? Cool?

So this is a really interesting and challenging and fun question for our hot dog recipe look up, we have a very popular website. It serves let's say one hundred thousand users, and we want to actually let's just have that be the only constraint, and we want to make this better. So our line of business is selling hot dogs. And let's say we make money on our website via purchases of our hot dog products or kits or recipes or something like that.

We just sell stuff. So how would you go about defining what makes this better? And again, as a data scientist, this is more of a product question, but sometimes data scientists need to be involved in this design. And if you're given this make it better request, you often are doing a lot of the design.

So how would you approach designing this to improve the search functionality? Yeah, you always got to start. I mean to second what you said. I don't think as a data scientists working on a project that is product related, I don't think you can avoid putting on the product hat, and I don't think you ever should because it's super critical about knowing what it is that you're working on, what you're trying to improve.

You have to understand the product. So the first thing that I would do is use the product, Like, Okay, somebody said our search engine sucks, I'm going to go and search some stuff that is very adjacent to things that our customers probably search for. Because I probably am not in the market for looking for hot dog recipes. I don't eat them very often.

I would probably pull the logs of the search and say, like, what are people using the search engine for right now? And what are what actually returns results? And you could do that, you know, analytically by pulling the raw data and seeing like titles of things that return from the search. Hopefully you are collecting those logs and they're in a way that you can access them.

If it's not that, and like the logs are in a place that you just don't have access to, then I would just use the tool and make a bunch of educated guesses of like, Okay, I'm gonna search for hot dogs with pickles and hot dogs, you know, best New York hot dog recipe, you know, just a bunch of random stuff, and then see what's returned and record that, so that I have a mental model and a physical model like data that shows whether this is good or not. And how would you then that you go and talk to the person who asked you to build it and say, what do you want improved?

What's so bad about this? Cool? So let's narrow it down a bit. Let's say they think that these search is too narrow.

They have a very simple elastic search that just does exact matching on prefixes of search terms. So if I type in hot, it will find all things that mention hot. So with that, how would you go about This is jumping the gun a bit, but I'm very curious your take. How would you go about.

Evaluating the different components of this piece of functionality? Would you do a B testing? Would you do offline testing? Would you do integration testing?

How would you approach the testing and evaluation aspect. For comparing what exists to whatever it is that we're going to be building. I wouldn't do anything fancy. I would I would probably route force it.

So if we have two endpoints that are available, one is the existing search API, and then we're we're going to be building something that might replace that. I would write like a simple script that just sends requests of a static data set that I would curate and be like, see the result difference between these, And then, depending on what the nature of the data returned, do I need to get like relevancy scores, maybe I'd use Jenny I to do that. If it's a large corpus of data. If it's like in the early stages, I'd probably start with like twenty or thirty questions or search terms and just use my own brain to determine like is this good or not?

Or how relevant is this and just human curated annotation. Really, yep, this the original search query term. This sucks like the results are not, like not helpful, And then use that as a baseline and be like, Okay, our new stuff is better, right. Okay cool.

Yeah. For this project, we did something a little bit similar. We created a very simple front end, and we had side by side results of the prior system and the system we were working on, and so that really helped business stakeholders understand the improvements to search. So if you type in what's a semantic word for hot dogs delicious, Let's say you don't want just a prefix match on the word delicious, because there aren't that many products that we have that say the word delicious in the title.

Instead, we want to look at semantic reviews, let's say, and see where people are calling tasty, fun to eat, whatever it might be, any synonyms for delicious, and then return those results in those products. So by having a side by side app and the exact same user interface, it really highlighted the improvements to the search quality. Yep. Cool.

Yeah, I mean I would do that later on after I built something if it warranted it. Yeah, if the customers were people that are like, oh, it's a bunch of software engineers that are just trying to build this thing, make it better because there's there's issues with the implementation that exists. Maybe raw data is fine, like sade by side, Jason, don't need to build a web UI. But if it's it's like, oh, this is the business, Like there's some C suite person that's like, I'm interested in this becoming better because this frustrates me.

I want something where I can see this side by side. Then yeah, build a simple React app that will show that sort of left and right comparison, maybe with like contextual highlights of what it what it actually found as a significant term as part of that search query. There's all sorts of clever things you can do and modules you can use. Right, yeah, exactly, Okay, cool.

So so far we've decided what to build, and again hopefully product will inform this to some degree, but you should also put on the product he at yourself to get intuition about it. And then we've also generally defined our success criteria, so latency is somewhere in their search quality is somewhere in there, and we've, let's say, define how we're going to measure this, whether it be Jason or offline testing or you need it. What is your first prototype. As dirty as possible.

You want to like fail hard and fast as early as possible. When you're doing a proof of concept, not worrying about code quality, You're not worried about readability. You are going into full hacker mode and just making something work. You want to do stuff like have a bunch of conjectures of what the actual systems architecture will look like for this, So you look at what is existing originally, Like what is that stack they're actually serving responses here?

What is the back end for the front end? Okay, you said before, it's a lastic search. Awesome, great solution. If we're going to make this better, do we go into using managed yes, that has some of the more advanced like similarity characteristics.

Do we take the entire search term and do like Levenstein distance to find similar things? Are we? Like? Does that work?

So? I would play around with that to see do we have a quick win here that makes it significantly better than what is existing other than you know as per typed basically a completion search engine, which sounds like that's what it was before. So can we get it done with the existing tech and make it work? I don't think that's the case for this because of you know, the sort of project that we're talking about, So we probably want to use something a little bit more advanced and more modern.

So if we're going into the jen Ai world and like, okay, doing similarity on embeddings, we're going to have that sort of comparison. We need some sort of you know, rag system that's like I need to find a document that most resembles this and search rank order that based on that relevancy. Right, So I would just do that as quick and dirty as possible I have all of my data somewhere stored in es that should be backed up somewhere on object store. Let's just embed like five thousand articles that are all somewhat similar but have enough of samples that are very very different in context.

Load that up, like, calculate all those embeddings storm in a vector store, and then just start querying them like does this make sense? Right? So one probably very deep question for you. That could be six episodes.

I've been working with some really fancy engineers at Data Bricks and looking at their prototyping code. It's a private preview feature and I have the privilege to go and read the repository and looking at the code, there's a bunch of discrepancies in style and overall it seems like in quality, like there are better ways to do some of the things that they're doing. But it's clear that they're just hacking stuff out in a sort of a way that works, and they don't really care about the intricacies of style.

So I thought that that was super important. And after seeing this, I was like, huh, senior staff engineers writing a like fifty if statements in a row noted Yep, what when you said quick and dirty. What should not be skipped in quality? If I'm so, there's a difference between me filing a PR against the code base, where I market as draft and tell people don't review this, like this is a proof of concept.

I'm just trying something out, but I want CI to run on it. That lets everybody else know that I have my my dirty hacker hat on and nobody's going to look at it, or if they look at it, there it's not a judgment type thing of like what the hell did Ben do here? Why would he file this PR? You see that as a direct indicator for everybody else, or just like yeah, he's just trying to make this work cool.

They'll make Maybe they'll look at the implementation and suggests like oh, you've like we should probably add these like seven other if statements here for this other like this edge case. But nobody's like why are you doing it this way? Why aren't we adhering to proper software design? Like how is this even testable?

Where you unit test? Nobody's doing that because we know what that phase of development is. Most of the time, we're doing that locally. Will fork a branch, we'll test it locally.

If CI can run on our local machine, we'll do it that way, and then once we have something that kind of works, then you have that open on one screen, and then you have a fresh branch on another, and you're converting that garbage code into something that is proper Yeah. I still don't get it though. For our use case, what are the things where you would really think about and design properly and what are the things where you'd just be like, eh, fifty of statements is fine. I don't worry about implementation details when I'm getting something to work because I know how to do that.

What I want is a service that does this thing so I can determine does this even work? Like based on the data that I have. And I don't know if this semantic search, if this approach with using RAG is the right thing to do. I'm not going to sit there and write a whole bunch of code that properly interfaces with a vector store.

When I can test that out in a command prompt, I could just hit it with a Curl request and be like, yep, that seems legit. Because the Curl request is going to take me thirty seconds writing all that code to make it look good. Is going to take me six hours and file that PR that's another ten minutes, and then ping people for review. If I don't even know if that's going to work, I just wasted six and a half hours, and I wasted other people's time too without even knowing it.

This works because if it doesn't work, I now delete that PR and wonder why I spent the whole day of my life working on something that doesn't work. Got it? So? I think it clicked in my brain.

The thing that you were optimizing when building a component is the amount of time it takes for you to write stuff that will give you signal on whether that component is a valid design choice. I think that perfectly encapsulates what a hack is cool. All right, So. Let's say there are in your brain fifteen different design points.

Do we do this or do we do this? If we go down route X, then we'll have these other six things that we need to answer. If we go down route why we have these four other things we need to answer. So let's say you've.

Built out a prototype that generally maps this search space of design, and you have an optimal design, or at least a seemingly good design. What are your next steps. There's a step that you're missing there. What tools do I have available to me?

Not what tools exist, it's what can I use? So when we're doing when you're working on a project that is self contained, that you control the environment, and you can use any tool you want, any open source tool, any tech doesn't matter. Startups are like this right, Like you can just choose whatever works the best. That's a very different story than you're doing.

You're deploying the service at a company that already has a tech stack, already has services that have been vetted that you know how to, Like, there are other people that know how to run those. You might be interfacing with other teams. There could be somebody else who's on call for this thing that you're using. You need to involve people in that decision process of the design and very clearly articulate why you can't use something that's already in existence at your company.

I know a lot of people that get into this this field from another field like data scientists, to move into like software engineering or MLE. They're like, why can't I use this like super cool open source project that does all this this amazing stuff at my company? For this product an appointment, and the people that would be downvoting that and saying no, we don't need to do this. They've been there, done that, and like introduce some new tech that doesn't check all the boxes or is like super insecure or is not stable or has some maintenance burden on it that nobody understands, or they don't have capacity to bring this on board.

So you have to be very careful about that stuff. What do you have that's already been built that you can use? So sometimes that means you have to get clever with your implementation to be like, well, yeah, we want to use for the data serving layer of this thing we want to use like Mango dB. We don't have a Mango account, we don't have that service at our company, but we have aws.

Can we do this with Dynamo because that's vetted and approved? Okay, change to use that instead. M Yeah, that's a good point. And also if we're iterating on an existing system, I'm sure a lot of the existing system can be repurposed.

Exactly, and that's the key goal. Like we do that at data Ricks, right, We've spin up a new service that does this new cool thing. It's a lot of it is reusing components. It's hilarious.

Yeah, I mean it's critical. If you saw what it was like to onboard a new service, you would realize why people are like that because that can take like a year of validating stuff and then building the infrastructure that allows that service to actually run. There's so many back end components too, something that's like, well, I need to deploy this to every data center in the cloud. How what's the mechanism for that.

Do we have all of that set up? How do we do change management with that? When we need to bump the version of this and do a migration, what does that look like? Oh, somebody needs to build all that code that does a migration without a service disruption?

How do we do that? You know, it's what seems like a really small and significant thing for your little project all of a sudden balloons out of control where you're like, I need to involve seventeen different engineering teams to just build this, Like how much capacity do we need? That's two hundred people's time for six months not gonna happen. But you have something that's already there that you can just be like, oh, I can just call this API and I can spend up a new service doesn't do exactly what I want, but it's pretty close.

Okay, let's do that. Exactly. Yeah, it's been funny to watch new data bricks features get released and it's just like half the time it's just like a packaging around an old service and it does something different. So it's super valuable, but it's the back end is the exact same thing.

So it's like nice to be able to just transfer your knowledge over to that new feature. Yes, and that is intelligent product design. Mm hmm. Look at all the cloud providers, they do the exact same thing.

Like a lot of the AWS services that are out there, they're building on the foundation layer of things that have come before, and they're building new products that are just when you look at what it's doing, you're like, hang on, isn't this just using fargate like Kubernetes pod management and then OKTCL stuff is like orchestrating this. You look at it and you're like, yeah, that is what it's doing, and it's really clever that they built a new product that leverages this. So really smart engineering organizations do that extensively.

Yeah, what is it just out of curiosity. The AWUS backbone is Dynamo EC two. What else. S three isn't S three on Dynamo though.

No. S three uses Dynamo for its location. Lookup like where exactly hash table But it's a separate service, separate technology than object store, got it? But you look at like ADWS lambda right, that's their servileist architecture for executing arbitrary functions.

So much stuff runs on that because it's the intelligent design. They even open source of the thing too, which is crazy m hm. But they're like eks like Kubernetes backbone for AWS. So much stuff runs on that.

Fargate is an abstraction layer on top of that, like a service abstraction and it's it works at stupid scale. And Azure has the same thing, right, same sorts of infrastructure, and they're doing the same sort of build on these these building. Blocks, right yeah, okay, cool uh fun little aside, So back to the question at hand. We've sort of iterated, well, first all, we selected some tool stacks, and then we've iterated throughout the design space and have a general idea of what and how we want to build.

I guess the next question is how do we build it? Yeah, how what does the team makeup? Who's getting who's putting hands on keyboard? Right, So let's say it's two engineers.

And you're bringing in an interesting caveat to this because the two people working on it don't work for the company. So this is consultancy work, which is a different dynamic than building it in house. Right, let's say yes for this use case. Yeah, So with consultants, you're you're on the hook for building something and delivering something that you're getting paid for or your company's getting paid for.

So the two extremes of that is one you're doing team embedding, so you're helping out a development team and you're contributing code with them, and they're doing the product product management side of the house. They're giving statements a work to you, like hey, we need help with these five elements and you own that end to end. But here's the contract. The input is going to be this, and this is what we want as an output of the service.

Go build that thing and here's the specs. That's more pure consultancy work, or you're a coder for hire. The other extreme of that scenario is you get hired as a team of one to ten people, and the companies like, we want this magical thing as a service, go build it for us, And they have no direction other than here's the out, like the end result that we want. Go figure that out.

Smart people, mhm. And you usually have no interaction with the team at the company. You might be talking to somebody, some executive or somebody in upper management who's just handed you the statement of work of what they expect and they don't care how you build it. They just want that thing to work.

Right. My question to you, which which where on that? On those two extremes do you like to operate in? Uh, They're actually fun for different purposes.

I've been really enjoying design recently, but sometimes just banging out code is what you need emotionally. So I think it really depends. I don't know that I have a strong preference on either right now, slightlyly needs design, But what about you. I mean, I don't do any of that anymore.

Everything we do is like immediately right in the middle of that. We have a team. We know what our team's capacity is, what our charter is, and we own the design and the implementation and the maintenance of that. So there are a handful of situations where sometimes we'll get help from a consultant, not a consultant, a contractor.

It's like, Hey, we just need somebody who's really good front end for helping us out with because we just don't have capacity. But we do have a deadline of this date, so we just need a gun for hire to come in and bang out some code. We're doing the design for that and then handing over a spec like here's what we need, and then we're reviewing every pr that's involved with that. Right.

But when I was doing stuff like what you do now, I didn't enjoy the latter one that much, where it's more like here's a vague list of requirements that we want go figure this out. I preferred on that side. But like engagement with the team or engagement with subject matter experts at the company where it's like, oh, this is like what I used to do when I was a data scientists and companies or EML engineering companies. I have to go and talk to people and be like what do you think they should do?

And what do you like about this? What don't you like? What are your expectations? That's fun and that's more like the product engineer sort of design side of the house, but the full extreme is very frustrating.

Or it's like, yeah, I have access to your systems, but I don't understand your business completely and I can't read your mind and you go in to try to talk to people and they just don't have anything to say. Yeah, anything is better than this, like well okay, and that brings you. That leads you to the point of needing to learn very quickly what's the difference between good enough and perfect? All right?

Yeah, before we define those That was what I had to do for this prior project. They were basically like, our current system sucks. Everybody knows it. Can you guys make it better?

And we hammered them for SLAS and it's US criteria and we finally got a latency requirement under a certain load and I think there was one other request, but beyond that, they were. Just like make it better. And the backbone of the reason why this worked is we built that front end app that allowed side by side comparison and allowed a bunch of users to go and explore it themselves submit feedback. We had a little like feedback bar in that front end that would submit to a data bricks table so we could actually look at the feedback.

And then also it clicked in people's brains. They're like, oh, the old chearch didn't do this, and now it does this. That's so great, what about this feature? And it was pretty inefficient, but at the same time built a lot of rapport and the business stakeholders were really happy that we were really taking what they said seriously because honestly, there wasn't a product person in the room and so they didn't know what's There was no products back, so it was just intuition by business stakeholders and so we iterated together and it worked, but it would have been nice if there is a products back somewhere along the road.

I mean, I mean that can be exciting where you're just given free rein to make something amazing exactly. The scary part for me historically in those situations is, I know the limitations of my own creativity and the fewer number of human minds that are involved in that, the worse it's going to be. Or you have a probability of overbuilding something because if you're not in that ecosystem and you're not like a user of that platform, or somebody who actually works there and understands the customers, or whatever use case is.

You could start going off the rails of like, well I need to I tried this really crazy thing and this totally broke this, so I need to engineer something around this so that this doesn't happen. And then you look at the like you launched a production, you pull the logs, you start looking at what people were doing, and you're like, why did I even build that? Like nobody uses this. We don't have a single instance after a million queries of anybody doing this stupid thing that came into my head of like how could I break this system?

And you realize that you just built something that didn't you ain't gonna need it, like it's. Worthless funny you say that, Yeah, that's actually hilarious. You bring that up. We load tested pretty stringently and like after it launched into production.

My counterpart that I on the data breaks side when I created the logs and it was like a hundredth of what. We tested for. So it was it's it's a very seasonal query amount, but still like we could have half asked it a lot more. I wouldn't, So that's one exception, like service stability is something that you don't typically half asked.

You set a spec and you guarantee performance within a certain probability range for that performance, and that's more for infrastructure needs. Then that's budgeting. So you build a rest API that can serve one hundred requests an hour, and you don't ever expect that to go above, like say a thousand requests an hour based on the history of this search tool for all time. You don't go out and set a spec for, you know, ten thousand quaries per second, because the infrastructure for that is fundamentally different, the expense of that is fundamentally different, the complexity is fundamental different.

So you don't build that if you don't need it. But the half ass comment that can come and bite you if it's like, yeah, we search the logs, there's like one thousand requests an hour. Why didn't we just suspect this for like ten thousand requests an hour. Well, ten thousand requests an hour, you can get by with a single VM running a flask server.

Just expose that behind an API gateway. It'll cost you five bucks a month to run that service. It's nothing. What happens when your psyche goes viral and you get one hundred thousand people hitting it in ten minutes, that search functionality just broke, like broke hard.

That server bright crash. And did you build all the infrastructure for like instant resumption of that. Probably not if you're thinking that it was going to be really low and you're like whatever. So for stuff like that, you would want to have a minimum bar of quality and performance that you've validated to say, like even at peak times, like I'm going to test ten x peak of all time just to make sure, and then have some sort of failover mechanism where I can protect that server from crashing.

Interesting. Yeah, that's a good point. And ideally some sort of auto scaling so you're not spending money when you don't have to, but when traffic comes in it'll scale up. Yeah, like simple cloud managed Kubernetes set up to can run a container that has your stuff running in it.

Yeah, let somebody else worry about it for you. Yeah, but also set limits on that, like don't just say yeah, defaults are good because the default, depending on what service you're using, it could auto scale to something where you're like, well we just blew the yearly budget in one hour. Yeah, because a it horizontally scaled to support this burst of traffic because somebody's being a jerk with seeing if they can take your site down. Yeah heard, okay cool.

So rounding out this list, we've designed it, we've prototyped it, we've prototyped within the design space to select tools and finalize what we think is a good approach. And then we've also started building and assembling this team. And then a team has started actually developing this feature. So what's next?

Heads down, crank out some code, No, set up a cadence for a team to discuss not everybody likes stand ups and stuff like that, figure out something that works for the team. You need somebody who's leading all of it and setting expectations for what is the project delivery timeline. That's also another thing that I think a lot of people don't really enjoy building or adhering to. But it's super critical to know what is the scope of what we're doing, what are the components that we need to build, who's going to do what, when are they going to get it done by?

And at what particular cadence in that building of that system. Can we all take a break for a day and test what we've built so far, because if you're not doing that, you're all just heads down, like, hey, let's go build this. Michael, you build the rag thing. I'll do the front end to back end like translation layer.

We're good. Meet again in a month. You don't know whether it works like those components work together. The scope could be completely way off.

You could estimate like, oh, yeah, I built web servers before. It's no big deal. It could be way more complex than you thought. What if you it?

And then when you when you merge these two systems together, the latency is a thousand x what you expected. Oops. Yeah, I should have figured that out six weeks ago. Yeah, no, one hundred percent.

That was one of the We did a great job building it and collaborating throughout. We had not stand ups, but I think like check ins every few weeks. I reviewed all of my counterparts PRS. He reviewed all of mine.

So there was great collaboration on that front. But the scariest moment was the Big Boss came in the room and there wasn't a ton of collaboration on the stakeholder side between Big Boss and the people we were directly interacting with and we were just like here, we didn't get any design specs, we didn't get any details beyond the simple slas you provided. Do you like it? And uh, yeah, it went super well, but it was a bit unnerving at first.

And hopefully you can have some of that back and forth and feedback not just from within the team but from the stakeholders and the leaders that you're looking to cater to. Yeah, and making sure that's part of the project plan. So what you don't want to do is involve an outside user into something like an internal bug bash. Yeah, that can go very poorly and they can lose faith in you.

We don't do that. Most serious engineering organizations never do that. You always test internally because we all know things go wrong, stuff is totally broken. You list all out all the things that are just kind of messed up, and you get it to a state where you're like, Okay, everything we've tested works pretty well.

Now let's bring in people who can test how they're used to using it and see what they come up with, and then you record everything that's broken that they find or is it unintuitive, does it not behave the way that they expect Okay, how can we adject so that it's more intuitive, And then right after that and you're doing those fixes, you start your next phase of development work. Okay, maybe we need to work on the performance of this, so now let's focus on that together, or maybe it's the accuracy of what's coming back.

What can we do there? Do we go back and fix, Like the mL side of this thing, is that something with the data we need to change and iteratively work on that and show incremental improvements within the team and then have everybody say like, yeah, this is way better. Let's get the like the SMEs in here to have them see. Sometimes that you don't do that solely for the fact of giving them a good feel of yeah, the project, the project's on track and it's looking good.

Sometimes it's purely political so that they can come in and see like, oh, yeah, they made huge improvements since last time, and you can do tricks with that as well, like you can show the results of what it was in the past, even if it's just total garbage, and then have them look at it what it is right now, and then their perception is like, whoa, this is so much better? Is it perfect? No? Is it ever going to be perfect, probably shouldn't be, but you need to get it to a point where they can have an emotional response where they're like, this is awesome, We're good, we can ship this.

Yeah, And that's a one way to kind of force that by showing how bad it was before versus what the improvements have been. Yeah. Yeah, it's interesting. I've been learning how to do this properly, and it's perception of deliverables is so malleable.

Yeah, it's a very human thing. So the engineering team knows all the holes typically, and when you're working internally, you're like, that's going to break. That's going to break. If you do that, these seven things will break.

But within a thirty minute to one hour call, there's not enough time for them to completely vet your system, and they also typically don't have the technical capabilities to do that anyway. So it's important not to lie, but it's also important to frame and pitch so that the right things happen. Whatever the right things are yep. Yeah.

You don't want to hide something that's serious and it's probable to call it like to happen. You want to disclose that and say, like, how urgently should we fix this. But if it's something like our process on doing stuff like that is, if it's a bug that and we haven't released it yet, it's a sev zero like that needs to be fixed. If it breaks the workflow of what somebody's accustomed to, that's a bug, it's a sev zero fix it.

But if it's something like we think people are going to want this feature and we think this is important or critical, that almost always comes down to like a P two like we're we're not building that unless we get feedback. We know what that feedback will probably be, but we don't know if we should build that yet, And the smart thing to do is don't build it unless somebody asks for it. Yes, because when you do go and build that, like, oh, I got this this awesome idea to make this super like super great, and then you go spend a week building that thing, there's no worse feeling that you can have aside from shipping like total shit broken code, other than you had this great idea of this amazing thing, and then you pull the data or talk to people and they're like, yeah, that sucks or I don't use that, I don't need that.

Yeah, man, why did I build that? Yeah, Yeah, it's funny. Ten days before the project ended, we got a new requirement that was like a legitate like they called it a requirement. It wasn't defined at all ever, and so we had to sprint to complete it.

But yeah, it Thinking through this, I'm wondering if we framed it different, if we did like a different framing of the results, if that requirement would not have come up. And I don't know. But the key thing is these stakeholder meetings are very human oriented, so don't ignore the human aspects. Put on your charisma hat, put on your sales hat, and really like pitch what you've built, because that can be night and day for the reception.

Yeah, could not agree more. It's also a different it's a different ecosystem if you're building something that's internal to a non technical team but is approachable and that anybody can use it. And I think the emotional aspect of the perception of quality is more important to sell to like highlight all of the highlights as much as possible, and maybe not dwell too much on things that has a couple of rough edges in this one part. Wait for them to discover that if they never see it or they never complain about it, then it might not be that important, right.

But what you don't want to do is walk into that room hey live demo time people and you're just sitting there. You're like, man, I really hope they don't find this thing, and you didn't disclose that, and it's like the first thing that somebody finds you now look like a complete idiot. Yes, that is always test your stuff yourself and have somebody who doesn't have context but is technical to come and test your thing. Yes.

Cool, So we're about at time. We brought it all the way up through implementation and presenting the stakeholders. The last step would be productionization, and that's a whole other can of worms. But if you have this initial prototype that is working, you can typically push that into Proud via whatever your deployment strategy is.

And maybe that's a future episode, Yeah. A part two. Yeah, revisit this from this stopping point sounds good? Cool, So I will summarize.

We revisited sort of how to deploy model serving capabilities, specifically with a search case study. At a high level, the steps are, first, figure out what you're trying to build, put on your product at understand the use case. Understand how the business makes money or succeeds. Next, figure out how we're going to measure success.

So is there going to be increased engagement? Are there slas, et cetera. Third, build a quick and dirty prototype, and the thing you should be optimizing for here is the minimum time to signal, and signal means a indicator on whether this decision on a given design is good or bad. Once you have that sort of initial prototype of working, then we want to look at the tools that you can leverage as the underlying structure for this overall functionality.

Fifth, iterate with a bunch of prototypes, gets more signal on design, including these tools. And then finally, once you are pretty confident that this is how it should be built, be heads down, develop it and customize the working style to the team that's involved. So maybe have stand ups, maybe don't, but a lot of feedback is important. Once you have a V zero point one or even less, it's important to have an internal bug bash and try to break it.

Be really creative, use your engineering skills, use your product skills, combine them to break the product, and then finally bring in stakeholders for feedback and see what they have to say. Anything else that covers that devlu cool I do. Have one more note, oh for service stability, don't have asset and sure you can meet the slas. Yes, all right, Well until next time, it's been Michael Burke and my co host Ben Wilson, and have a good day everyone.

We'll catch you next time.