Navigating Common Pitfalls in Data Science: Lessons from Pierpaolo Hipolito - ML 183

Hey everybody, and welcome back to another episode of Adventures in Machine Learning. This week on our panel we have Ben Wilson. Hello, I'm Charles Maxwood from top End Devs and this week we have a special guest, and that's Pierre Paolo. Do you want to introduce yourself real quick?

Let us know who you are and what you do, why you're famous? So, yes, I'm here and I'm currently working as a scientist at them Too in the UK and part time my spare time. I also contributed to Todata Science in publication online and so them after also the published on applicational sas p dot com or kicking on It and so on. Awesome, so you're a data scientist.

I'm sorry where did you say you were working? I didn't quite catch that. Well, okay, that's very cool. The company headquarters is about six miles from my house.

Yes, yes, awesome. Well you wrote an article talking about the paradoxes of data science, and in particular, yeah, you outlined four different paradoxes or four different things that you can run into with your data set. And as Ben is often fond of pointing out, your data kind of determines how good your machine learning models are, and so it was I thought, oh, this would be interesting to dive into and talk about. Okay, you know, how can we get a mysterad on our data?

And yeah, how does that happen? So do you want to first before we dive into these, give us a little bit of background on why you wrote this article or you know, was there something that prompted this. Yes, so I think it came from the sort of study I edit for my master degree. So I did the master in artibitional intelligence theniversy and for my master I decided to over a topic reasoning learning, and part of the topical covered during the projects about three for months was researching about how society can be embedded in machine leary models and what kind of problems you can have between getting wrong relationship within society and correlations in DA says Pia Boats and for example, how can you should be applied also in the case of COVID or things that when you for example, like as you might remember at the early of beginning of twenty twenty, like very generally twenty twenty, there were lots of people trying to, for example, think oh, let's see how I can help with the pandemic development and try to stop or prelift and under locases in these kind of things.

But at the beginning, and now they can do it also pretty well, but at the beginning was quite impossible because they didn't have any actual data to use the trainer model or to CUID model in the first space, and that was sort of a paradox. And therefore in to solve this problemly then not to work around the other different center to augment the data or models real for the environment as a synthetic environment in which are people moving around and spreading the virus, or mathematical models to see how people can chain states and so on, And that's what prompted them the regrating of the article and so on.

I think also the book by the Book of Buy was also a really good starting point where we also explained this in some paradox, which is one of the fourth paradox, as I thought. The very cool. So do you find then, and I'm just curious as we get into this, do you find then that people tend to they get more or better data, Are they able to overcome some of these issues with understanding the data or is this something that you fall into even with a good data set or more mature data set.

Obviously, yes, if you have some process of data quality in the first place, that makes the work of the data scientist or this is you make by itself easier, but you can still go into this kind of pitfall. So you need to have some fort of basics. So knowing how the aniline system that's trying to model up really works and then modilate so that we make predictions makes sense. It's a very interesting set of topics that you're brought up with simulation modeling when you have a sparsity of data coming into a problem.

And this is something that as data scientists we deal with quite a bit, where you might have a new project that your boss is saying, or somebody at the company says, here's a new initiative we need to solve. For go figure it out, data scientists, And even if it's a very well defined problem, which most of the time it's not, but sometimes it is, you go back to look in your data warehouse or in your data lake and you suddenly have that sinking feeling of like we don't even have this data.

Talk to the front end devs or back end devs like, hey, we're trying to look at this this signal here, and they're like, oh, yeah, we were going to capture that or put that in the logs, but we figured nobody would ever, so they turn it on and you might have a month worth of work where you're trying to get everything started doing your research, and then you just have a month worth of data to play with, and to make an effective algorithm, you actually need years of data. Maybe it's solved this problem.

So what are some of the techniques that you found for doing that simulation of saying, like you said, people have been doing it with COVID, and how do they do that or how have you seen that done? So, yes, with regards to COVIDE, like you mentioned, and I know, there are two main approaches you can take. So one is with like sort of eggemology models, which are based on using differential equations. So in this case, for example, you can start by creating a set of ordinary differential equations, and each equation represents a compartment into the model.

And then as you simulated and run these equations against time, and you can see how the different people are in the number of people in your populations, the move within a compartment another into the simulation, and in this case most simple model that you think office could be an sir model, which is a move with three compartments people that are succeptable to disease, people who are infective from diceas, and people who are recovered from the disease, and therefore your districting compartments with these three three equations representing different compartments and depending on the number of people of the other compartments and the probability of one compartment to another, they can then run as simulations for what which even get the sort of curves that you well have seen quite a lot around with the people the pandemic and stalls been around with this program by different governments.

For example, I know I know that at the early stage of the pandemic the governments using at the team of from the imperior culture government's using a a sort of an environmental model with something like twenty twenty four viables it and artist being one with tasball. That's true. Also how complex this thing can go in order to make it more actually resemble the actual real world balings. But you can take into account many as many reductions you want, so instead of having just susceptible infacted.

The covert a compartment for people who actually die because of it. Is a compartment of people who have been vaccinated in the for their immim form. It is a compartment of people who have the time limited immunity to take up for with the they are immune for just certain lots of time and then they can get it again and so on. So you can have as many compartments as you want in order to try to make it to look as as close as possible as a real good.

And the other approach is that you can take is by using agent based modeling. And this is used not just in videomology but also like in form for training, military applications or for also a big play consumulation. So I can form like they're using the formula one are raising or to simulate tracks and so on, and how are informing the track and in this case yes, basically the main approach here one of the most one could be like to create a class or not jitter in the class and which are tent for resemble a person in the population, and as all of these categoristics which are person enough such as an age accept and that type of community that it might might do it within a city, and other factors such as are not being all the younger and female or male and so on, so that to make sure that for each of this fact then it is associately the stable of video of aving getting its more emotional a disease or not, and so on, And then you get instantially as many people as possible in the class and obviously run an algorithm to see what it how they interact each other within artificial environment, so that you can get the exactly saporse that you can get from aprilog more normally based.

So this is really in game theory and basian probability statistics, where you're saying, hey, for a given population, I have an number of, say if we're talking about COVID, we start with a loki of say ten infected in individuals, and we say, map out how many people you're going to come in contact with, and then what is that multiplicate effect or that geometric increase in the number of people that might be infected if people don't change their behavior based on the current state of reality.

And you can kind of approach a bunch of different problems with doing this as well. I knowther. I mean, I've personally worked with companies that have done this sort of approach with tracking cell phone location data of customers, where they're saying, hey, we know where these people are. In general, we don't know who they are.

We just know a cell phone had an IP address ping that was triangulated to this street corner in Manhattan, and we know where they're kind of going, and we can simulate where they're going to go and where similar people to them are going to go next week or next month, and then sell that data effectively. So it's interesting to hear about these approaches for something that's really important, like a pandemic. What would you say is an effective strategy if somebody is thinking about applying these principles to their business.

From a data scientist perspective, it's it isn't traditional supervised learning, this isn't traditional deep learning, where everybody seems to be focusing on those two things. In the data science world, how would you recommend people get started in this sphere of data science work. Yeah, I think that's much more a niche type of application, since that for many cases you can sort of gather a data icily, but from other cases such as for the beginning of the pandemic New York.

And I think you also for other applications with us for example, I'm predicting escaping goals for more green eruptions or things like that. When you want one to know model how many people can escape from cd avot kind of wraps. That's because you don't have many much data about anything or fast and edgepeak of datata. You can just use this kind of model instormrations and and that's where it become useful.

But when you update, obviously it's better to drive use the real world data to the public in some form of artificial data, which then is making models and in order to do it also if anyone here is interested. In my case, for my project, I MELI created everything from from scratch and that was acustom application. But there are also some open source packages for liby views such as MESA which is Titan or ash which is also Java split in Ciposplasiachube videos, and they also provide you with some form of for for some common place common common problems.

I could be for example, some example, the ad was modeling the logistics, so optimizing logistics for a score for example, or modeling the wildfire spreads in the forests, and for this kind of thing also other epoxis education even so that you can actually focus them in the best model instead of having also to create a fancy to show base power actually model level sort of that. Yeah, there's a customer of my companies that has been working on tracking the population density of both predators and prey in a national park.

And it is interesting talking to them and asking them a couple of questions like hey, how do you do this and stuff, and they walk through it and it's almost identical to what you're saying. It's like agent based modeling. We're creating a set of conditions based on mathematical models, and they're basically just passing partial functions around inside this class, saying, hey, here's the probability that this happens based on what we know of the interaction between these species on a very limited observer basis.

But they want to forecast, like what should we recommend to the federal government of the number of prey that humans can hunt versus how many are available for the predators, and how do we get an ecosystem balance. And it's one of those things that you can't actually run those tests in the real world because that has a huge impact, and you don't you can't just run experiments do like a design of experience. But like, well this year we're going to allow hunters to call forty percent of these these creatures.

You destroy an ecosystem doing that, So that the way that they get around that is the simulation of saying, let's just run a million simulations and see what happens. Yeah, that's that's that's also application because I don't even but there are a set of two three equations which represent this kind of situation that commonly use in the equations. Yeah, it's really really interesting for anybody who's listening who is really curious about some of these mathematical models. Some of these equations are very old that predict where stability will be for a particular population or a particular like natural for domena, but it applies to a lot of other things as well.

Trying to remember that one magical equation that gets a periodicity repeating. But there's an equation out there that if you feed in a certain value into it, and I think it's in the denominator, it'll based on what that value is what that K value is. It will generate either a repeating binomial series or multinomial series that actually maps to observe data in populations. And it also translates to stuff like routing of vehicles, like how many paths can we actually successfully do in a steady state a situation.

So it's pretty fascinating stuff. So what are some of the other applications that you've seen for handling early stage serious problems that we need to solve now, such as the beginning of a pandemic and saying what should we be focusing on or how important is this? Yeah, I think the most common lines. So then another see for example, could be that for some pieces where you don't have any NEEDA to start with, but you know that you can get data faster, you can also in order to try to tune these models, you can also start with the model insinuations and then for some many pause or COVID after six months when you actually get the data, then compare the curves that were predicted with the actual ones so that you can find some alloys like fiber en parameters almost and then get better results.

So that's definitely another one. And then for example, Apart from that I see now there is also propose also for supervisor models or unsupervised models, who creates actual synthetic data for example for a maturing more problems, which could be like over something techniques understanding or even for example I also different type of some likniques in financially so usual creating the synthetic data or financial as in the data naturally setting the mass of product. So I think there is interesting in this topic also, especially conceiding that data and yourself what expensive to buy and creating your own data can bee gigr sanage basically on other people.

It's a synthetic data creation. Where we're talking about supervised learning, it can be a technique to sort of reduce the probability that your model is going to receive completely unexpected data and not know how to handle it. And we're talking about linear models. Your your space is effectively contained by what your training set feature vector is, and if you get something that is such an outlier on enough of those vector positions, if they are dominant in that linear equation, you're going to get a prediction that is not going to make much sense.

So and tree based approaches are even worse where you're confined exactly to that that sort of label space, as well as the future space outside of that. It's not going to extrapolate what could potentially happen, So simulations can can sort of curtail the possibility that you're going to not respond to a black swan event such as a pandemic crippling the financial industry of many companies. Is are there any downsides that you've seen for people that are using your company software, which is I've only ever seen really SaaS users do this because it's pretty sophisticated stuff to intelligently create synthetic data.

But are there any downsides that you have of like warnings like hey, don't do this for this use case or yeah, definitely do this. I don't. I think it's quite use the specific, so you to be attention closely to what they're doing for or is it? And I think that we offer with the SASS so we use jumps to create acts and data and I don't know how spercid networks, but it's something that they've been working it up months for having for that.

But it's also quitely important you remember in this case, I quote from George Box which he said that all models are essentially wrong, but some of them are reform. So it is always because you're trying to make a prediction on a population from a sample, it is always going to be a certainty and errors. So just keep in mind that that in mind and be sure that you always check or bridged in this or anything after you deploy the model. Cause because the model is not a natural position about it, but just an approximation of it.

And for most use cases we're talking about, even something like modeling out pandemics, there's a lot of latent factors that go into that. I'd say the latent variables that are actually incredibly important for modeling that out are uncollectible data. And as we have some sort of weird surveillance state going on in countries around the world where everybody has a camera on them at all times, we know exactly what you're doing, who you're talking to with the proximity to physical interaction is But for a lot of use cases that people are doing an industry, it's similar.

You are trying to predict whether a customer is going to turn. You're trying to predict whether this activity is broad. You're trying to figure out if somebody is going to buy this thing, the actual underlying influences, those latent factors that really drive what somebody is going to do or that what the classification of that action is are unknowable. It's not going to be in your training data.

So how do you handle that? If you're talking to a lay person at a company and as a data scientist and they're like, well, why doesn't the model get this right? How do you communicate that to them? Yes, like, obviously you need to make sure that's they when you ask started asking a resiness question.

Once you set up your recent question and you know what you're trying to achieve, you need to make sure that you have all the enemies possible. Because if you have the data necessarily on the viables basically assign to achieve that, then all the other way you can try to go for working out a solution could be sort of to find approxy available on something that can communicate with the original variable that you are trying to do different but you don't that but it's connected with so that you can sort of corredorate with you the two.

That's the I think the main way you could try to focus on that, and the one way to do that is like using common filters and particle or particle filters as something that can be used especially it's usually like InCred robotics or enforcement learning type of applications, whether you are trying to for example, one actual bull example is like if you're trying to track the position of a car where it's moving around the map or in the world, and there's going to be points in which simple a card might go under a tunnel or my goal in a really red posting the forest or something and attle sweet signal.

And in order to try, for example, when you go under the tunnel to to to predict how where is the card, you just take measurements of before and after your information and try to work out what's going on the middle basically, And that's how this kind of particle features or cover filters can the more you know in order to do estimatings with approximble that you're on top. And but that's just a very specific artificial So like if you're working out with the standard supervised learning techniques, then the like feature engineering with the features so you currently have and try to see if you can work out how can they create most of your business spaces.

Probably the best. Solution so feature interactions, in particular of extrapolating the covariance associated with two separate features to provide further information. Is it is there a limit to that of how many elements or how many features in a vector that you would say, hey, don't go crazy, don't do something like, hey, I'm going to throw a thousand features into this model and I'm going to interact them all. What sort of.

Output would you expect from something like that? And supervised learning? No, I mean, the general advice I think would always be to try to solve a really completely problem in the most simple way possible. So especially as I benef you see a really good results attend and you have thousands of pictures then and you actually want to deploy it, you only have to maintain it and ensure that you can sort of understand how the model works.

And especially in some countries like for the Many, there is a the GDPR actor. There is an article for the right of experientire of a customer. So for example, that could be if you created this model for deciding is someone to get a loan or not, and they a person doesn't get their only go to the podca ask okay I didn't get to one, and you speaking why, and then the guy will see, okay, is this model two thousand matters besides things? I don't think that's that every good discrimination.

Yeah, one hundred percent agree. Even if somebody has ten thousand variables, it should be going through feature reduction and saying I'm going to do exploratory data analysis, I'm going to interact features. I'm going to iteratively go through and do covariance estimates and cross validation of different feature subsets and saying which ones actually solve the problem to keep it as the simplest artifact profitible. That's that's something that I don't see nascent teams doing very often unfortunately, and that's something that people assume, Well, I can throw, you know, ten thousand features into XG boost, and then I can calculate feature importances and then I can just take the top end, and then you have to kind of explain to them that's not how you do it.

You need to use other tools, statistical tools too. It sounds so smart, though. It sounds lazy, and as engineers, a lot of us are lazy, right. We want to get to the next thing pretty quickly, and it seems like it's a good idea or that it can save you a lot of time.

It actually just wastes a ton of time from every time I've seen somebody try to do that. You run into another interesting problem with supervised learning, where the cursive dimensionality as that feature space grows, you now need more rows of training data for a model to even detect the correlation signal correctly, and that becomes you know, exactly as you said, Pierre, it's it becomes expensive to start having to have that volume of training data and gets complicated. I think it's even more interesting now with how the whole spiel of transformats are evolving with the semi super advice learning so and and now they're going out to also vision data or sound data.

So I think that's also going to be interested next year, so that that is. Going to go on in the sound data. So can you talk a little bit more about that about what that research is like, No, what I. Just said, because I know that the hugging paste which can really launch anyficial of that.

So yeah, basically, yeah, you can use also like transform you know, you can transcribe any the data and then processing or any data. And I know for example, so also Unity, the Compule gaming engine, is doing quite a bit of research on that and also constantly data inta. Obviously they you have these sort of modeling environments like we mentioned before, like useful games, so they're trying to make a big advantage. Of Also interesting, so because you have hardware that's running that game engine that has generally free cycles of a pretty bp GPU sitting in a desktop, people are running that, you can just have a portion of those of threads in cuter running a simulation at any given time, so that you get dynamic gaming effects that respond to player input and how they're playing the game.

Yeah, I also know that Unity can use with Python and in order to create out for RL models, so they are explored on I don't know how precisely gedble I used to since I don't, I don't I never worked for them, or I don't do anyone more, but for sure again they have interesting I think I developed Desitate and everything that the app interesting in. That trea interesting metamorphosis of video games in general if you can have it with that, And I was like. Also, you know, a real engine with optimizing for example, great racing those things that use of say I put these countings and a social gaming.

So that's that's and they can see how for someone you can be great racing to machines that before couldn't be able to underten. So yeah, there is definitely what a lot of room of improvements for the kind of pation. Yeah. Currently rate tracing on GPUs that's a brute force algorithm.

It's going through and calculating every point of a light source to every like every surface that it's rendering. If you can have an AI solve those where then yeah, yeah, so it doesn't have to do that path and calculation. That's pretty interesting. It's almost like using an auto encoder to say, here's what my expectations should be from this point source in this location.

Now just generate to render the frame for me on the next cycle. That's pretty cool. So one thing that you know, kind of pulling it back towards this article a little bit, do you find that these paradoxes because you gave visualizations and so for me, I looked at them and I was thinking, well, I can see how I could come to that conclusion based on what I'm seeing on this graph. But do you find that these paradoxes affect the accuracy of machine learning models or is it just something that we fall into as we look at data and kind of try and find a pattern ourselves.

I think like if you don't leave the models, the all the virables, the misstructs resolve the paradox, and it also can get trapped to it, like in this case for what we were neglecting in the case of the example on the article, we were I think we're neglecting the age as a parable like and the paradom. And then if you do and then need to just leave the levels and the exercise and neglect age, then the model could get confused since that they it just looks at the how the different teatures are compared to each other or to each other and make prediction less of that.

It doesn't have any external knowledge like we human can. Naturally got a bunch of dead space that'll be headed it out. So I was just looking over your the article real quick because I could find the exact link to that one. But Simpson's paradox, that's one of my favorite ones to explain.

Yeah again right, it gets back to because yeah, I looked at it and I kind of did the lays the evaluation and was going, yeah, it looks like it says this, instead of actually applying a little bit of rigor and going, oh, this is what these numbers actually tell me. Yeah. So one of the things that I really like about that article is not just the explanation, which is great, and I find that I am fascinated by a reading style because I read a bunch of other things that you had posted over the past couple of days, but the plots, the visualizations that you used to actually tell that story are something that I don't see a lot of people doing.

I see not a lot of newer data scientists i'll call like people that have gotten into this field in the last five, six, six years. I see these explanations coming or I used to see them all the time, back when we used to be called analysis engineers or statistic statisticians. But this used to be the bread and butter of everybody's analysis. Before you even talk about modeling, you have to go through, and in this article you're in Simpsons paradox explanation.

You're going through and creating this visualization where you're displaying the Pearson's correlation coefficient of these these two variables and saying this is how they're related, here's my evidence of fitting a line to this and how correlated they are to one another, and extrapolating from that, what is the interaction between these these features? Like how closely related are they? How critical do you think it is for pretty much any mL to go through this this exercise that you did in explaining and teaching this on just general features, and when do you think that's inappropriate time to do it in an mL project.

I particularly like we're creating good leaders are just as one of the my favorite things when creating a more creating out analysis basically so, and that's also why, for example, I tell my fait to talk to your sentinels so that we want to go more where you see we are also article in which I I really took time also to create interactive regations so that for some of them, I also better than not three D plots you can actually rotate or play wait, explain how the private parameters works and how to find the best combination of private parameters, or working without encoders creating deploy for example, encoder is on an X and actually be able to draw on it on a web interface as the type of experience you can actually test the model and see how degreates we are put and then work out prospering, not because you make it more interactive and easily or to understand, especially because as I mentioned before, like, I think the obviously should obviously be to to try to create the simple solution to a complete problem.

And in order to let you need to lie deep into what you're trying to model first of all, and using the legation is one of the eastest way to do that, because otherwise you can also you know, create somebody solaces and things like that can help you, but without actually seeing how the things works out in pornograph, let's just do a part of the view that you might miss be missing out, especially like asaid before, us reducing the dimensionary of your data this preciated other coders or any develop also really helped, especially when you're working with previous part it or with texted that attract to convert into my format.

And I think that's also a good example likes or like, Yeah, so I think that the real preparation that of engaering your pictures and we actually be able to expect them, but also do what you are reading. The model, you can understand how how you would predict basically that really helps work out and that's always really like how I would really grate this thus on how the creator principal works, like how doing their feelings really well to give. You how do something like that? Yeah, totally, and creating those those charts and plots from my own personal experience.

I don't know if I'm just old school, but I look at the charts that you created for these articles, your correlation plots, I'm like, man, I do that too, so like, but I do it in that way, and I'll color code based on you know, label cuts that I do on the data, and I'm looking at continuous series, but I'll save them all off somewhere so that after I've gone through, you know, testing a bunch of different model approaches, trying to find the simplest, most performance, cheapest solution.

Generally, I always go back and look at those when I'm trying to do interpretability analysis where I'm running and saying shap on top of that that model and saying, hey, figure out, you know, I'm going to sample a thousand rows from the data set, and I want to see the force plots for shat for these, but when I'm looking at those, I have the correlation plots open right next to me with all of the color coding, and I'll look and I'll say, oh, yeah, that makes sense why the model has learned that, because the correlation relationship between these is this, And even creating synthetic rows to run it through shaft testing and then referencing these correlation plots that can help figure out where that space is.

And yeah, it's super critical, I think, and hopefully more people can read and look at the examples that you've done in these in these articles because it's an insight into how it's supposed to be done when you're going through a deep analysis of your data. Yeah, that's Simpson's paradox that is that's probably my favorite one. That can be the most interesting paradoxical discussion that you can have on a data set because it exists pretty consistently. I've found in real world data where if you just plot a continuous variable against another continuous variable, or if you're doing, you know, in a nova analysis of categorical data and you're visualizing that, it can be pretty misleading because there's some confounding variable that is actually explaining that noise that exists in that plot, And if you don't actually color code that or map that or split that and do that per sub sampled subgroup analysis, you might lose that signal in explainability use the age versus cholesterol level in your example.

But it's pretty rare that I've found an actual real world use case where I don't see interactions like that happening. So to the listeners out there, I highly recommend reading that article in particular from him, because it really does break down some of the ways that you can get burned in mL without really understanding your future data. We had another pause. Yeah, I'm just wondering, like, what other gotchas can you run into and is there is there a good way to make sure that you're not running up against that stuff?

Yeah, I think comport the overfitting. So I'll trying to, for example, focus so much from that most things can then straight away from the bigger picture make you lose points when trying to actually create from real war data. So and that's can be That is also another thing that it is quite common because at times, especially you might try to use when you try to use more more models, more complicated and necessary, then that's policially what happens, like you overfit to the actual drinking data, and then when you try to make creditions or out of sample and data points, you you get worse performances.

So and basically that's also wise a part of because you when you treat, when you try to create a model when we actually drink it, you try to improve the accuracy as much as possible, but when you tested, then after from the test set or validation sector, we just notice that trying to improve the curacy in the first place and let to work accuracy when you actually need it later on in the production and and and that's why, Yeah, it's really often to to not focus much on the like if you make up your own targets the only thing that matters, then it stops being a good target, like because you that's your way to neutural protects, which you've been in between creating product.

Yeah, I mean it's I'd say that it's endemic, and I have a name for it that I use. I call it the Cagle paradox, even though it is the accuracy paradox. But it's almost the fetistization that data scientists have these days of competing of like sort of winning this the accuracy game at the expense of maintainability and explainability. I've seen people spend months on projects that they could have solved something at an accuracy value for classification, for instance, like a binary classifier.

They could have had an area in our under RC of zero point ninety four. But and they could have done that in a week's worth of work of building a very simple model, just doing their their future engineering to get it to be somewhat accurate, and on a hold up validation they would have a result like that. Instead, they spend seven eight nine months working with an ensemble of you know, an implementation of basically somebody's white paper that hasn't been proven or reference, and they install pietorch and they build this graph embedding solution that I think is going to you know, work really well.

And after that almost year of work, their result on the same hold up validation is zero point nine five seven. It's like, what was that really worth it? I mean, it's cool, I guess to build something like that, but how expensive is that going to be to maintain? And can you explain that like how it arrived at that prediction.

Is probably not the best. Yeah. I saw about it a few actually a few weeks ago a tweets or something online or maybe what's LinkedIn of someone saying like, yes, if you are add the scientists now and you love to solve a problem of a model, you have to tick in productly and you have to solve a problem. Two things you need to do.

First, create a linear model which can go into production, and then spend months feting a really weird and so obviosic model so that you can just put it on your CP. Yeah. I mean, resume driven development is a thing. I've seen entire groups at companies just dedicate themselves to that.

They just want to do the most cutting edge thing, and it does at the service to that company when you have an entire team with that mentality, when that baseline that first attempt, I don't even use plentear models to do my baseline. By the way, I'll do a simple fit like a simple regressor if I'm doing a regression style problem. If I'm doing categorical I'll just use an IFL statement based on an EDA. I kind of have a breakdown in my head of like, okay, here's the relationship.

If the values are between this and this, it kind of correlates to this for a prediction and write a very simple decision tree and SQL. But that's my baseline to compare everything else against while I'm doing experimentation. But I time box everything too. I'm like, hey, if I'm testing out an algorithm, it gets forty eight hours of attention, that's it.

If I can make it work and it seems promising in that forty eight hours, it's a candidate. If it's too complex to even or the API just sucks so bad that it's almost unusable, it's out of contention because it's going to take too much time and money. And as you said, carbon footprint, that's a real thing. Electrons are precious.

They come from fossil fuels to generally, so don't waste them. But I like electrons. Everybody does, yep. But the overfitting underfitting stuff with that effect in the pursuit of accuracy at all costs, it also comes with it a process that you need to adhere to while you're developing stuff, where you know, you see people doing stuff like hyperperameter doing with grid search and cross validation where they're running through and they're doing random splits.

Maybe they're stratified, maybe they're not. They're doing undersampling over sampling if they have a massive class imbalance. And when you're doing that random iteration of cutting for train and tests and over and over and stuff, what do you recommend for that final evaluation When you're talking to people about avoiding the accuracy trap of this situation where you're saying, hey, you're pursuing the ultimate you know, cross validated accuracy measurement above all else, how do you make sure to say, hey, at least you have confidence in production.

This is going to work pretty well in this case. Probably, Yeah, if it's a pastilvation problem, make sure that there is a good balance within different classes. First of all. Then then you said, by the precision and Sensitivity guide you not not not securacy.

And then we look at the art seekers and also the gift of the kind of metrics you know, to to the track of how they varie from training to by addition to cultural testing, and also probe your model with with the testing and also come up with some good out of sample measurements that but you can actually tect it on you see if the model can get really from some specifically edge bases, like like you will do when an Apple piece of software that you think of edge bases that might three get your model prong and so that you can go also where actually is a band that is actually modeling and where you can can go wrong or not how to change your marchings.

So these penaltings and that's can be also important for them for our security point of view, because indeedly when you talk about every SEP, the attacks with the men that are actually not fyware of people are trying to prove these models to deductions then where they throw a line at the prediction and then go past them in this way. So that's actually something that's also a sort of can be authlementation, texting portment, like making sure that you understand how your model can attripate wrong so you can actually monitor in production in some way using for that.

That's funny that you mentioned that. I think we talked about that a couple of weeks ago, Charles uh one of one of the guests, was we were talking about fraud detection and how how hackers actually have criminals. Criminal hackers actually try to detect what they can get away with of you know, running brute force simulation attacks. And that's what they're actually doing.

They're building this sort of table of saying I'm going to do these actions in this order. Try that burner account, see what happens. Now I'm going to try another thing where do I actually not get detected for doing something illegal? And that's where they'll operate.

We're like, hey, your your system doesn't detect this, and I'm gonna I'm gonna steal a couple hundred thousand dollars from you. You're not gonna even know, and then it actually does happen. And so you brought up a great point. That's something that a lot of financial institutions do with that simulation.

Uh, they have white hat hackers effectively that are trying to defeat that that algorithm and say, hey, this is definitely some suspicious behavior, and see how the model behaves with it, and if it doesn't detect it, then they know, okay, we got to retrain or continuously retrain and monitor for those those edge cases. Yeah, and that's true. Also, how creating overly complete the multiple LEAs so sort of a law and intended consequences when some rug triggering input to them because let's structors, which you wouldn't have expected in the every post.

Yeah, with some of the interactions that happen with with super complex architectures. Yeah, there's no way that a human can reason about how the internals of that work. And that's why my advice to people is always, if it's something that is so important that there's a lot of money on the line, or your team's existence is on the line as a data science stayment a bigger company, it has to be something that you actually understand how it's reasoning through the correlations that you're providing it.

Because if you can't think about the worst case scenario where people rely on the output of this implementation to make a decision that influences say ten percent of your company's revenue, which could be fifty times or one hundred times your entire team's budget, think about what an executive is going to think about if you cause that amount of damage to the company because of your model. So understanding how it works, it's far more important than as you said, that resume candy like, oh he built this cool thing you I want to put it on your CV so people respect the fact that you put out ten linear models that actually worked and provided value far more than that super cool reinforcement learning system that was so overkill for predicting churn a customer.

Yeah, but I want to prove out cool I am. I mean, I tell people that's what the weekends are for. Nights and weekends do some cool stuff. And that's actually what Cagle's for.

Like, use that, use that tool for what it is. It's so your way of showing off and building a portfolio if you want to be that person. But when you're working for a company, you got to take it seriously and you need to produce something that is maintainable, solves the problem, and is explainable. Yeah.

Cool. That seems to be a common theme in almost all of your articles, is like talking about that foundation aspect of data science work. And I personally find it fascinating seeing somebody who comes from academia sort of rigor of how to do stuff and you've converted it into something and that I don't see a lot of people do, which is real world practical advice on here's how this should be done. But your posts aren't filled with pages and pages of mathematical proofs.

They're more like, here's some charts and visualizations, here's me explaining why this is a thing, and here's a link to the proofs and stuff. Or you know, you want to read about it, here's where you go read about it. But here's something that you know, everybody can consume. I think it's fascinating.

I hope you read a book, man, because I like your style. Yeah, when you're when you're a famous world renowned data scientists, remember us. I don't know. I think that's my name.

All right, Well, we're kind of getting toward the end of our time. I just want to, you know, call out the article again because I thought it was really well done. Ben pointed that out as well a couple of times. If people want to find more stuff from you, or you know, maybe connect with you and ask you questions.

Usually we're looking at like LinkedIn or GitHub or Twitter or something like that, and then it looks like you're also on Medium. Do you want to just let people know where they can find you? Sure, so they can message anyone to make message, or they can also go on my website. But I have a contact page sending an email that you can do it to them and also news letter so you can do subscribe to to get a TATO when you.

Have people come awesome, nice. All right, well let's go ahead and do some pics now, picture just shout out stuff we like, stuff we want to let people know about. Ben. Do you want to start us out?

Oh, I've got a nerd pick this week, and it's what's consumed a lot of my time this week, which is working on libraries and Python, and I hadn't used it before that much because a lot of the stuff that I've been doing of the last several years were sort of applications engineering, like using somebody else's a and now working on stuff that are building from scratch for other people to use, getting into the Piedantic library and doing something that is run time type checking for Python as a fantastic library.

Massive kudos to the engineers that built that and open source it. It powers a lot of the stuff that's in fast Api, which is also amazing package that's out there. But type checking is good. It helps you write far easier and more more knowledgeable and readable unit tests and catches a lot of errors that you might otherwise miss with a non compiled language.

Yep, very cool. I'm going to jump in with a few picks on my own. I think last week I picked the three hundred and sixty degree Leader, but I'm going to pick it again just because I've wound up doing a bunch of coaching and I answered some questions on our weekly Q and A call, which you can find at topenddevs dot com. Scroll down, it's under events.

But anyway, people have been asking how do I manage this? Or how do I deal with this? Or my boss is doing this? And this book really does kind of outline, hey, here's how you deal with people who are you know, above you, here are the people, Here's how you deal with the people that are around you, and here how you deal with the people who work under you.

And it really is just a terrific book that outlines a lot of that and a lot of the approaches and how to be a leader. Even if you don't have like lead DEV or manager or CTO or something for a title, you can still lead on your team. And I think it's an important skill to have to be able to just let people know, hey, look, you can make a difference wherever you're put. So I'm going to shout out about that also top End Devs.

So as this goes out, because I think this will go out next week, see the next week or the week after. As we record this, Bend Devs is getting a new website. And I've really just gotten a lot of clarity on what I would like top End Devs to be and you know, what our mission is for the dev community and things like that. And so the couple of things that are going on there.

One is that I am working with people to put together courses on various things related to software development. But I'm also tapping into I have a lot of friends who work not necessarily in tech or dev, but are experts in some of the soft skills that I believe technical people need. And so we're probably going to have some masterclasses and or courses on things like networking and not like it networking, but like networking with people or speaking at conferences, or some of the leadership stuff that I've talked about with three h ERD sixty degree leaders or how to find a job or how to collaborate with other people and pairing and stuff like that.

And then I do intend also have the technical content. So if you're interested in authoring any of these courses, go to top endevs dot com slash author. You know, we're looking at video content and some audio content on the premium end. The podcasts aren't going anywhere.

But this is kind of the next stage of where I see things going for us and for ways that we can make a difference for all of you. So anyway, I'm going to pick that, and then if you are looking for sort of one on one walk through coaching, typically people are getting on because they want to talk to me about creating a media channel like a YouTube video or a podcast. So I've been coaching people on that. I've also been coaching people on going freeinance, and I've been coaching people on just kind of taking the next stage of their career.

So if you're interested in any of that, you can also go to top endevs dot com slash coaching and see what's offered there. So I'm going to shout out about all that, and then I am going to do a board game pick. I need to start keeping track of what I'm doing so that I don't repeat. But last Saturday I was teaching people how to play games at a video game convention here in Utah.

It was a rather small one and I did it as part of a marketing push for a friend of mine who owns a board game store here in Utah. And the game we played, or we taught like six games, but the one that I learned on Saturday before I had to take my turn teaching was Planet X Search for Planet X. So if you like logic puzzles and you kind of like the dynamic of trying to figure out what the answer is like Clue. They kept comparing it to Clue, But this game is way more fun.

Go check it out the Search for Planet X. You do need at least one smartphone in order to play it, but essentially what it is is it's just the it's the piece of the game that gives you the clues, right, So it's, hey, here are the clues you're going to get, and then when you do research or you survey part of the night sky for items, right, it'll it'll fill you in and say, okay, you know, there's so many asteroids here or whatever. But it was way fun and so I really I enjoyed it and I'm gonna pick it.

So go check out the Search for Planet X. We'll have links to all that in the show notes, Pierre, what are your picks for me? The Book of Why, like I mentioned in case you're interested in to leave you more about causality or the sort of paradoxis and why it's important to make models explainable and that's I kind of events, some form of external knowledge of the world. So yeah, that probably is my pick for in this case.

All right, very cool, Well, thanks for coming. This was a lot of fun. I almost asked you where people could connect with you again. We'll make sure all that winds up in the show notes, but yeah, I just love thinking through some of this process and going, okay, how do I make sure that the foundational data for my models is correct in giving me the right information?

And then from there, yeah, now how do I build this so that's useful to people? So this was really helpful. Yeah, it's great. Thank you very much for inviting me today.

Yeah, all right, folks, we're going to wrap it up here until next time, Max out.