A/B Testing with ML ft. Michael Berk - ML 181

Hey everybody, and welcome back to another episode of Adventures in Machine Learning. This week, on our panel, we have Ben We'll SELLO. I'm Charles max Wood from chat dot TV. That rebrand is going to kick my button for a while.

Top end devs, top end devs, folks, come check us out for our pre launch special. We're here with our special guest, and that is Michael Burke. Michael, do you want to introduce yourself? Let people know who you are, why you're important?

Sure, I'd love to. As Charles said, my name is Michael Burke. I have been working as a data scientist full time for about a year and a half now and then part time for a couple of years before that. Currently, I work at a company called tv It's similar to Hulu, and we provide movies and TV shows that are AD supported and the like Quick Hitters.

It's like a four five hundred person org and we have the largest content library out of it any movie and TV streaming service. Oh cool, And what do you do for data? I work on two pods mainly. The first is sort of ab testing and infrastructure, and the second is ads configuration.

So on the AD side it's more decision science and data driven ad configuration. Experimentation is optimizing the experimentation pipeline and then building out infrastructure. Gotcha, And before the show we were talking about some stuff around AB testing and how you're using machine learning there. Do you want to just kind of explain how that's all set up because it sounds really fascinating to me, and just give a little bit of background.

I've done A B testing, but I usually like cook it up to analytics engine and say that number is bigger than this number, which doesn't require machine learning by the way, and then I picked the one with the bigger number, So yeah, what are you doing different from that? Yeah, So we subscribe to Frequentist Experimentation. The other main area is Beayjian experimentation, and they have their pros and cons, but we personally subscribe to Frequentist. And so roughly, when an A B test is it's a randomized control trial with a control group and end number of treatments, and then what you look to do is compare whether the treatment has a statistically significant lift relative to the control and if that's the case, then we know that there's an actual change, and if it's a positive lift, then it's a good change depending on the metric.

If it's negative lift, it's usually a bad change. And then from there we can decide whether to graduate the experiment or not. So when you're looking at stuff roll like evaluating changes, when you have your control group, how do you determine how many users would have to be in that in order to extract a signal? Right, so there are two ways you can go about this.

The first is pull a number out of your hat and call it that, and that is sometimes implemented. The second is, actually, there's. What everybody does. Yeah, yeah, for sure.

The second involves something called power calculations, and we to be have something called the sample size calculator that just uses a like front end dashboard that allows you to put in some parameters, and then on the back end it uses a Python library that solves power calculations. So you essentially put in your type one error, your type two error rate, and a couple of other things like minimum detectable effect, and that will output a number of unique units that you would need in this case users, devices, whatever it is, and then from there you can set in experimentation depending upon how many users you get per week, or per month, per day.

So the other the aspect of the power factor calculation is the detectable quantity of difference between your subpopulations. But that's predicated on what your sensitivity is. So a lot of times in data science where we deal with alphas of zero point zero, five ninety certain that this is going to happen, and exactly that when you do a power factor calculation, one of your variables in there is setting alpha is saying how certain do I want to be? Can you explain to people with the relationship is to your sample variance to that that certainty.

If we're measuring something that's super noisy, how big of a sample do we actually need? Right? So I'm so the most simple calculation for sample size and I found this derived in some YouTube video somewhere and it changed the game for me. But it's roughly sixteen times the variance over something I forget exactly what, but in the numerator is the variance.

And so the larger the variance, the larger the sample size required. And so that actually is a pretty good segue into variance reduction techniques. But long story short, larger the variants, the larger the sample size. By a good amount, and it's really powerful.

Well if you can thereby reduce the variants prior to the using information prior to the experiment, and that can really cut down sample size. So we do something like that as well. That was one of the initial projects that I worked on. Every major tech company that I know of does it, like the Googles, the facebooks, the netflixes, and we've stolen some of their ideas and they work great.

So yeah, So variance reduction, you're talking about subsampling of your populations to eliminate latent factors that contribute to variants. So that is an interesting point and that is an option. But well, another route is essentially forecasting variants using pre experiment data, and because it's pre experiment data, there is no confounding within the experiment. A very simple method is called q ped cuped, which stands for controlling experiment data using pre experiment or controlling something using pre experiment data, and that uses simple unit very linear regression.

But as you can imagine, if you have the tech stack for it, you can throw all sorts of crazy mL add it and just forecast out variants because it's pre experiment data. So when we randomize those confounders are evenly split between control and treatment, and we're good to go. So you could use more sophisticated models, not just a simple linear regression. You could say, I want to use Whole Winter's exponential smoothing, or I want to use a REMUD or ceramax, or if you don't want to go through the trouble of dealing with stats models package, you could use profit or great Kite and you can get that predicted value, forecast it out with seasonality factors and corporateg and into it.

That's pretty fascinating exactly. So I'm going to chime in and ask my new questions here. So when you're talking about variants, what you're saying is is that the numbers that you're trying to predict could be all over the place, And so what you're trying to do is you're trying to reduce whatever effects are making it go all over the place that don't actually matter to your experiment, so that you can get a result that you can can't rely on with some level of certainty. Is that what you're talking about?

Or did I completely miss the boat? You got on the boat and are sailing pretty yep. Okay. So at this point, then what you're saying is you look at your data from before the experiment because you haven't changed anything yet, and you figure out, okay, well we can ignore some of these effects because they're consistent through the entire thing.

And you forecast what a future state would be. So you're like, hey, have two years of data, We're going to run our experiment for three weeks or four weeks. Let's predict out what the next three or three weeks of variances, and then we can strip track that out from or manipulate the actual test data to reduce that variance. Right, And that way we can get a clear answer because we know that this much of the all over the place is stuff that we can reasonably predict is in there anyway exactly?

Oh kah, we're just isolating signal and a bunch of noise. Right, all right. I'm just making sure I get it, And I know that there's some people that listen to the show that are probably also trying to follow along. All right, go ahead, sorry, And the other aspect of what Michael was talking about with the variance component.

Two people that are familiar with time series forecasting One of the first things that you do when you're looking at a time series data set is you decompose that trend to your trend, your seasonality, and your residuals. And those residuals is what drives that variance different so much, and it's that noise. And if you can't reduce that noise when you're doing your test, that translates to, oh, geez, I need to collect more data and that means money. Right, Yeah, I guess that was the other thing that you were saying, right, is that your your variance is in the numerator, right, and so you have to collect that much more data based on Yeah, how noisy your data is, and so if you can account for it and not have to include it in that number, then you can collect less data.

That's what you're saying. Then definitely right, Yeah, And by the way, I remember the numerator is sixteen times the variance and it's over the effects size. So whatever our minimum detectable effective interest is, that would be in the denominator. Okay, So when you talk about the minimum detectable impact, who usually sets that?

That's a great question. That's a trick question. You get a yeah, A thousand different answers if you talk to a thousand different data scientists, but really there's one answer at the end of the day. Well, if there's one answer and.

Lightness, it's always the business right exactly, So it's always whoever is controlling that product. They're the ones that are saying, hey, we're only going to pull the trigger on this new feature if we get a three percent left. Yeah, exactly, And there are a couple just thinking about it from an entire business perspective. One interesting angle to take is let's say you're the machine learning team and you're looking to maximize a single metric over a year time.

The smaller your minimum detectable effect, the larger your sample size is required, and the shorter or the longer your iteration cycle would take. But if you have let's say a much larger minimum detectable effect, you can iterate a lot more quickly, spit out a bunch of different mL models, try them really frequently, and then see if that leads to a greater lift over time. So something that is actually really. Useful is if you look at the empirical lifts over let's say a year period for your team and see if you simulate just like random draws from that what would be the optimum minimum detectable effect, and that can be a good way to empirically choose that number.

That's a pretty cool tech tip for people out there is using analytics on your historical data to inform the business to make a decision that makes sense. So a lot of times when you're talking about a new feature where you know the company that you work for, your direct B two C. You're a company that interacts with individual account users and of course your ad you know your AD revenue groups that are putting ads on your platform. And if if you're not demonstrating some new feature that is that is definitely going to raise raise the bar on lift like, hey, we want to increase viewership, we want to increase the amount of time that people are spent on our platform looking at ads and hopefully buying products from them so that we get more ads ad revenue.

If you're not really moving the needle fast there or iterating quickly, then it's going to be a tough sell to the business to justify what your experiment is. Yeah, for sure, Yeah, I think I think of this in terms of because you know, I've been a freelancer, I've I've worked with companies where yeah, I've had to go and basically pitch ideas to the business and what they want to see is they want to see how much money can this potentially make us and how risky it is. And so that's where these numbers come in is, Yeah, we have a very high probability, very low risk way of increasing our whatever which increase uses our income.

Yeah for sure. So if you were to talk to people in different industries that aren't in where you work right now, like not in the streaming video service, somebody that works for like an engineering firm or manufacturing or something, and they've never heard of this concept, how would you break this down to them of convincing them how important it is to use these techniques on everything that they do. A B testing or vary introduction and. Those things A B testing and attribution got it.

So the main selling point of a B testing or randomized control trials is that they allow us to determine causality, and there are very few other methods that you can prove causality. So theoretically you could build a model that has all relevant variables in it, and then from that you can isolate specific impacts of excerpt by or whatever it is. But with randomized control trials, the key thing is that with a large enough sample size leveraging the central limit theorem, we know that confounders will be equally distributed on average between the treatment and the control.

So the only difference between treatment and control is the treatment effect. And so that allows you to essentially determine slash prove causality. And I think it's it's definitely the most robust causal framework. And there are other ones that you can argue also prove causality, but.

I mean that's a whole can of worms. I mean causality modeling in general is something that not a lot of traditional data science people actually deal with. That often about creating simulation DAGs and running it through millions of iterations to say, hey, I'm going to change this thing. How does this impact the Yeah, but result of this, and it's a highly esoteric realm of data science.

There's companies and groups within companies that focus on that entirely. That's all they do. They don't do supervised or un supervised learning. But for the other ninety eight percent of mL practitioners in the world.

Nobody ever touches that stuff, and the way too exactly, you said, if you want to prove that your model is doing something, the only way to prove it is run the test, collect the data, and evaluate it. But what would you say if somebody were to run an A B test and get that causal impact of hey, we had a ten percent lift on our test versus control, and then they go back and look at feature importances of the model and say, oh, it's this feature that caused this. So I personally don't have a problem with it it.

I mean, there's a big difference between determining graduation criteria and the why. So graduation criteria needs to be set prior to the experiment, and so say if we see a lift of greater than two percent of this metric, then we graduate, else we don't. And then after that if you want to go digging and figure out why this happened. I'm very supportive because a it tells a pretty story to whoever's listening, and then be it also allows for ideation for for subsequent experiments.

But the key thing is that those stats, statsig will only apply to that initial metric and everything will be directional and not necessarily statistically significant. Exactly that directionality is important for you can use that analysis to inform further iterations to prove further relationships of causality. But you can't infer causality from your features because of interaction effects and other confounding things that could could happen. The only exactly and actually prove is not running the model versus running the result.

And here's what we observed. That's a great exactly to put that. So I'm kind of curious, I mean, I, oh, is there an instance where you've put something together, you know, you ran your experiment and it like really paid off. Yeah.

So one of the things that I was hinting at earlier about looking at the empirical distribution of your experiment lifts is that more often than not a lot of the experiments tend to follow the eighty twenty rules. So we get eighty percent of the lift from twenty percent of the results. That's a like. A pareto something number, But that tends to be the case and a lot of things in life, in stocks and experiments as well.

And so there's definitely cases where we have big winners that drive the vast majority of the lift. I personally am starting to run some ADS experiments, so we're looking to figure out how to structure ads. So, for instance, I'm sure you guys have seen plenty of YouTube or Hulu or you name it. We can structure ads so that they are shorter breaks but more frequent or longer breaks, and there's more space in between each break.

So that's something that we're currently playing with. And obviously, if you add one ad per break, you increase revenue by like twenty percent or thirty percent, and just with that, so there's a lot of movement that we can do in terms of revenue versus engagement, and that's something that we've been playing around with for the past couple of years and we're starting to really make strides in. Have you ever gotten an idea for an implementation from somewhere in management and they say, hey, we really want to run this test and this is definitely going to change things, and people are adamant and about this is going to be a game changer.

You run the test, you evaluate the results based on your power factor calculation, saying we should have significance within say two weeks, based on what people are projecting and you have you see no real difference. For sure too. These are very data driven company, so we're licensed to pushback when execs say, implement this, and we can say, well, at least let's run an experiment. And the nice thing about experiments is they democratize truth and you can have the greatest opinion in the world, you can be the highest paid person in the room.

But if the experiment shows no lift or it's neutral, doesn't matter. I don't know, I'm pretty smart. Yeah, well other than Charles. Well that's what my mother told me.

Anyway, I believe it. So if you're going to sell this, this concept of data driven decisions, and it is something that certain successful companies in the tech space have been doing since their inception, Like a lot of account companies are founded on these principles. So we welcome all ideas from whoever they are. Whether it's it's the janitor cleaning the bastrooms or it's the CEO, everybody's ideas are weighed the same.

We let the data decide. But not all companies are run that way, and even if they should. So, yeah, what would you say to a company that's like, we know our customers we know our business, we can just turn this thing on. How would you defend the inverse of that?

Yeah, I'm sure you guys are more qualified to answer this than me. But I take a sort of different approach in that A B testing is the gold standard and it is not necessary. It is definitely helpful and important, and if you have a larger company with the infrastructure to do it, it is completely necessary. But oftentimes in smaller companies.

For example, I'm working with a tutoring nonprofit helping them with their data tech stack. It's called Learn to Be. They provide free online tutoring to underserved kids, so homeless kids, foster kids. They're just too small to run any type of experiment.

They don't have enough sample size, and so the highest paid of person's opinion does win. But theoretically, as you get to a larger state in your company, you do need to run experiments because results tend to be pretty counterintuitive, and even the best pms don't have one hundred percent future right. So yeah, but it does occur to me that sure, you don't have enough data to get to statistical significance, or you don't have enough data to build a model and run some robust machine learning against something.

But you do have enough data generally to say, hey, we're going to try this, and we're we're going to be because most folks can set up analytics and measure stuff right and just say, hey, we're going to see over the next two weeks if the number goes up. Yeah, exactly. So those are called pre and post methods, and we definitely so creating an ab testing infrastructure is a good amount of work. You have to randomize at usually the user level.

But there are other alternatives, such as pre and post methods that just look at pre intervention versus post intervention to. See if there's a lift. Yeah, those are those can be effective in smaller companies. Right and then and then as you grow, then you can start saying, hey, now we're reaching statistical significance or now we're getting enough data to where yeah, we can throw some more robust model at this and say, all right, what are we actually seeing here?

Yeah, And during that journey, you're graduating along the path. I mean, even if you're a single person startup, you should be collecting the data that like the results of what you're doing for sure, and as you grow there's a there's a long journey between just collecting data and manually analyzing it and then going all the way to Hey, we have an automated AB framework that is part of our CICD where we set the conditions of this experiment saying we need ten percent control and four ten percent you know, slices of these different test groups.

Set up my experiment for me, do dynamic traffic allocation and maintain state, and then collect all the data automatically, you know, analyze it through window to aggregations over period. There's a huge journey there in that end state. There's not a lot of companies that have that set up to the point where it's sort of autonomous and it just runs as a service. I have seen them and they're pretty impressive, but it's years of work to build one of those.

Yep. Yeah. But even during that journey, you can make steps. And one of those early steps is exactly what Michael was just saying, is describing that like, Hey, figure out your sample size and do your first simple control to tests on this, collect it for the amount of time that you need, and what you said with like hey, checking for these big lifts or no lift, you can get that pretty quickly.

If your expectation is, hey, we expect this to be fifteen percent lift differential if after and that means we just have to run this test for ten days. At the end of ten days if you see If you don't see any difference, move on to the next idea. Yeah. And also in smaller companies, you tend to be able to move lift a lot easier than at bigger companies.

I know, the larger companies you basically operate by working with submetrics. So let's say you want to improve retention. You need to improve submetrics or retention because you just cannot move retention with it with a new feature. But in a.

Startup, you can make a new website in a day and that could completely change everything. So yeah, it's really hard to move the needle at certain companies. When you look you realize that the potential user base is a significant fraction of the population of a country, Like user bases, there's two hundred and fifty million users, Like, yeah, good luck changing anything with that churn rate. Well, you're going to tell people to just have more children to get user numbers higher.

Yeah. So aside from that one other thing though, go ahead, Oh, go ahead, No, I was just going to say, I mean, yeah, if you're talking about like raw numbers of hey, we're growing our user base by such so much, but you know, maybe it's hey, we want to increase our watch time on to be your Netflix, right, how many hours? How many ads they see or you know, so there, it may not make sense to go for number of people anymore. It may make sense to optimize for something else, and so you wind up doing that.

And sometimes that's the trick that you have to play, is Okay, well we can't Yeah, we're not going to get significant lift in this category anymore, But can we get significant lift in some other category that moves the needle in ways that matter to us? Yeah? For sure. So moving on from ab testing, you do it.

You're working this project that we talked about before recording started, which I find incredibly fascinating. The blog posts that you've been doing, which from what I've seen, you're looking at very interesting white papers that get published and then distilling down what the contents of that are into a highly consumable, pleasurable to read blog post. Where'd you get the idea for this? And how do you pick your source material.

Yeah, thank you for the nice words. First of all, regarding the idea, I think it was mainly twofold. The first was I wanted to learn. There's something called the finement technique, which is explaining something to a five year old.

Every time you find gaps in your knowledge, you need to learn more about that topic. So you can explain string theory to a five year old. And if you can't do it well enough, then you don't know the material well enough. So it's essentially learning through teaching, And so that was the first reason for doing it.

And the second reason was sort of more of a fuzzy warm reason, which is I think there's a lot of really cool tech out there and really inaccessible tech, and this stuff isn't that crazier that complicated, and if you just have someone explain it to you, it's usually pretty accessible. Now, if you want to understand the back end perfectly, there's a lot going on, but getting eighty or ninety percent of it and then knowing enough to implement it effectively isn't super super hard.

So a while back I. Thought it would be a good idea to write a blog post a week for a year. It's going I'm like halfway through and haven't missed a week. Don't know how, but way to go.

That's harder than people think it is. By the way, oh I know. I mean effectively, what you're doing is writing a manuscript, is in the process of writing a tech book. It's effectively what you're doing every week is like a new chapter and you got to go do your research.

So you're prepping yourself for your first publication. Yeah, there we go. That's also a lot of work, harder than people. I will talk to my editor and let them know that you're interested.

Perfect. So yeah, the one liner is looking to bring academic slash white paper research to the data science industry. And there's a lot of stuff that's cool. A lot of stuff gets published.

How do you how do you sort through a li of all of the flood of papers coming out of MIT, Berkeley, Stanford, you know, reading through all this stuff and saying, Okay, this one seems relevant, or this one's potentially going to be important, somebody's going to build something based on this. Yeah, that's a great point. There is way too much out there for anyone to master or even roughly learn. I have, so I use a website called aar xiv dot com or dot org excuse me, and is Cornell's aggregation of just tons of papers that you can sort by keyword.

They have a nice advanced. Search feature and they have a pretty good statistics area. So that's where I get a lot of my papers. And I just do what I'm either interested in or working on at work.

So if it's related to a project, it's really helpful to have in depth knowledge, and then if it's just cool, then it's a fun read. I'm interested in, like simulations, and marine energy is another topic that I'm looking into, and like clean energy in general, So just anything that catches my eye. But the key is to do stuff that you like or else you won't get through it. And you know, it's funny you say that, because that's the same advice I give people about podcasting, about having a side project at all, is you know, they're like, what, I want to do this podcast and I want to tell people about this stuff, and I'm like, well, do you enjoy talking about it?

I don't know. I said, you're going to be done by episode five because you're going to be bored, you know. And it's the same thing with the side project. Well, everybody builds a Twitter clone in in rails, and so I'm going to build a Twitter clone in rails.

Well, are you interested in Twitter clones? No? Well, what do you do with your time off? Oh?

Well, I paint minifigures for D and D. I said, do you like talking about that? Oh? I can talk about it all day?

Do that? Build a catalog for minifigures for D and D. Right, it is your side project. So I love this right where it's it's Hey, this is stuff I'm really interested in.

This is stuff that you know, I could talk about forever. This is stuff that I'm really fascinated by. That. Yeah, do that, folks, exactly.

And it's tough to take a leap of faith and know that in the future it'll all work out, because I think a lot of these projects are very goal oriented and outcome oriented. But doing stuff you like almost always leads to better results than just what the rest of the people are doing. That's true. But the other thing is is I think people also have a tendency to they underestimate the work that it takes to like write a post or build a side project, and then they also underestimate how long it's going to take for people to really find it.

I mean you actually have to go do work to help people find it. And it's like, well, okay, so if I start the podcast now, or if I start blogging now, then am I going to have a new job in two months? No? No, you're not right.

I mean, unless you have one of your things go viral and people are going, who the heck is this guy, and then when they look into you, they go, he's he's amazing. It's just not going to happen. The odds that happening are really low. Instead, what generally happens is you you're consistent like you've been, and then as people move along, it's, oh, this guy's terrific.

You know, he's writing great blog posts, and so you get more and more and more people showing up. You start appearing on more Reddit threads and newsletters and people just talking about you at the conferences, and eventually you build into that exactly. Yeah. Yeah, it was very flattering to be asked on this podcast.

I'm pretty early in my career, so it's kind of wild stuff. Blog post. Yeah, yeah, I mean I thoroughly enjoyed the few that I've read I'll break up back and read all of them. The one in particular that it was really interesting I read yesterday was the one on IBM's one of the research papers that they released recently about explaining model deficiencies a free AI.

It seems like it's proprietary, which is interesting to see somebody write about a white paper that is a proprietary tool. Most people are writing about like, oh, you know, there's this new open source tool that people are using, or it just got released, everybody check it out. But I find it almost refreshing to see somebody reporting on something that is proprietary, because it's opening people's eyes up to the possibility of, oh, this is a new technique that I didn't think of before.

There's no code in it, and that makes sense from IBM's perspective because it is proprietary. But I wouldn't be surprised if within a year or so somebody will have an open source package that does that. They'll figure it out because you've helped translate something that is I've read the white paper as well. It's not one of the most dense and complex ones out there.

Some of them are almost illegible, but it's still not a lot of people I will actually go and read the original source paper of something, particularly they've never heard of it. So it's interesting seeing that. Yeah, and I definitely will not be able to take any credit for future open source packages, but yeah, it's really nice to have a digestible piece that explains the high level And I took a page out of my manager's book and basically tried to in one paragraphic explain the entire topic and see whether someone's interested in and then if they actually do want to implement it.

Blog post is not the resource they should go to the paper, They should go to other things. Well. The other thing is is with proprietary technologies like that one thing that I found just over the years, because you know, I'm a programmer, but i also run a business that runs the podcasts, and I've built custom software for pretty much every part of the podcasting stack. And what I found is that in a lot of cases, it's easier, cheaper and better just use a proprietary solution for some piece of the puzzle, right, and then I can focus on the parts of it that really matter, that are different from whatever else is out there.

And so, but people tend to shy away from it in a lot of cases because there's free, open source solutions, and so I'm also gratified to see, hey, look, you know, if you want to shortcut this process, save yourself time on these areas, then go pick up this thing and then move on to the next piece. Crazy thing in the mL space is a lot of the cool tooling that comes out that's proprietary. You got to buy an entire tech stack for that. You can't just say like oh I want this service from IBM, and then I want this service from SaaS and then I want this other service from h two.

Oh none of them do that. It's like, hey, you want the IBM, like all these things that IBM has to offer, here's the contract sign and you're in it for millions of dollars per year. So it's a big sell to use that stuff. But illustrating and putting it into layman's terms about like, hey, this is what this company is releasing of the research they're doing in Israel, which seems to be IBM's new like massive research headquarters of all their mL stuff.

It's all coming out of Haifa, and it gets people thinking of like, huh, I've been struggling with this, and I've been wondering because some people's implementations, to refer to that paper in particular that you talked about in the blog post, some models are out there that are running in production that you have a performance blind spot in a subset of your data. How do you find that? It's not trivial to find that. You have some massive tree based algorithm.

You know, you run extra boost and you know like, hey, yeah, we're at eighty eight point nine percent accuracy score on this, and then somebody does analytics post talk after you know, the real world interacts with this algorithm, you find that on a cohort analysis, you have ninety nine percent accuracy in eighty percent of your use cases, and you have one percent accuracy in your most valuable customer based segment. People are like, what went wrong here? And the tool, like what you wrote about in that paper is designed to illustrate that, like, hey, there's this issue here.

It can't predict this correctly exactly. Yeah. Anytime you simplify accuracy down to a mean or some you're throwing out lots of potentially useful information. So yeah, all right, well I'm going to pick pushes towards the end of the show because I actually have a work meeting in eight minutes, so I.

Have a question for you guys before we end up ahead. Cool. So this is sort of like a career advice E type question, and I think a bunch of your listeners could probably relate. I have more of a generalist skill set.

I've definitely picked some areas of interest, but I think because I'm new in my career and wasn't the coding math NERD PhD like like, I don't have all the educational credentials that many of the really technical people have. So I think I'm gonna make my like Mark quote unquotes in data science by bringing data science to new fields. And I hinted at them before. But I'm working in two areas for side projects.

The first is education and then the second is marine energy, so converting waves, ties, and currents to electricity. So do you guys have advice specifically for generalists on how to bring data science and machine learning to new industries that aren't ready for the next giant deep learning model. I mean, i'd say generalists are the only ones that are going to be capable of making that happen. And I've heard that from a number of people at companies about like, oh, well, we have, you know, two hats being worn by people in our in our data science group, we have we have some generalists and we have some specialists that are really deep in this one area.

And it always confuses me unless you're working in an industry that is hyper focused on one particular use case, that your industry just doesn't have any any need to embrace anything else ever in the future, unless some revolutionary thing comes out. I think everybody should be embracing some aspects of generalism in their data science career. And that also means that there are some specializations that I think every data scientist should focus on, which is learning how to write good code like testable code, knowing.

How to do the first time I'm hearing this from you, Ben, Yeah. Knowing how to simplify a problem down to something that you can communicate with lay people at a company, and gaining wisdom over time of how to most simply solve a particular problem. And I think people that see themselves or others see them as generalists are more equipped to do what you were asking, like, Hey, I'm going into this industry this company or this nonprofit that I'm working with has never done anything with machine learning or data science or analytics.

Somebody with a broad skill set of all the aspects is going to be able to make that happen a lot easier than somebody who is a specialist on GANS or or you know, is an NLP specialist, because those industry groups might have no need for that exactly. Yeah, yeah, I tell people I'm an adversarial network all on my own. Anyway, my advice on this is mostly from the other end, not necessarily building your mL career just because yeah, I don't. I'm not deep enough into it to tell you, hey, these skill sets, right, But I can tell you that if you're trying to break into a new industry that does not adopt technologies quickly, then the trick is is you have to show them the value before they're going to move right.

In other words, you have to prove out, hey, look, this is going to give you such such a competitive advantage that you would be insane not to adopt it. That's when they move, right. If you come in with the idea, they're going to be like, that's nice. But it's worked this way for eighty years, and they tend to not want to take the risk, even recognizing that if one of their competitors tries it and it works, that they're kind of in a bad spot just because they are risking capital and time and effort to get it.

And so if you can, like I said, if you can prove out hey, look, if you go adopt this thing or that thing, you know you'll increase efficiency or you can basically print money, or you're going to be getting some advantage over your competitors by reducing your costs in a significant way. Those are the kinds of things they're going to pay attention to. It's not going to be the idea. It's going to be some aspect of the implementation that they think they can drastically increase their profits and competitive nature on.

So that's what I would focus on, is I would focus on a proving that it works, and then be how do you communicate that to them and speak in their language as far as what the advantage is offered. Are amazing, Yeah, that both of those points make a lot of sense. Cool, Thank you guys. All Right, we're going to do picks really really fast.

So let's go in like thirty seconds a piece, so I can make it to this meeting. Ben, you want to go first. The only thing that I've been focusing on the last week, aside from writing and testing code, is reading the next book. David Thorn one of my favorite authors, humorists, sort of a very dark humor of an Australian variety.

Check out his latest book and his entire back catalog on Amazon. The guy is a riot. I've yet to meet a developer or a data scientists he doesn't find the guy incredibly funny cool. I'll go next.

I have two things. One I've been doing board game picks. My pick is Shadow Hunters. It's out of print, which means you either have to find somebody selling it used on Amazon or go get it on eBay.

But it's Shadows versus Hunters. There are neutrals. They have their own win conditions. If they get it first, they win.

Otherwise, the Shadows win by eliminating the Hunters. The hunters will win by eliminating the Shadows. And you play with cards and you move around the board and you use equipment to kill each other. It's way fun.

I'm moving fast. I would sell it more, but yeah, the other pick that I have really quickly is top end Devs, and I'm moving fast and trying to get this content together. I'm focused on putting out content on building your career and building leadership. But I am looking for technologists that want to help either build out an ongoing series of videos that demonstrate in fifteen minutes or less some concept.

Machine learning is one of the series I want to put together. I'm also looking for people if they want to put together like a one hour tutorial, so this could be here's how you do this aspect of mL ops, or here's how you do this aspect of building a model, this algorithm explained or whatever. And then finally, the longer form courses that I think everybody thinks of when they think of online online courses. I'd like to have some of those as well.

And then I have some other stuff going on that I would love to talk. To you about. If you want to be an author topendevs dot com slash author, if you're trying to get blog or podcast or YouTube channel or freelancing launch, I'll also coach you, and that's topendevs dot Com slash coaching. All right, Michael, what are your picks.

I had a longer list, so I'll condense. But basically, there's some interesting research out there. For the mL specific research, there's some research by Stuart Russell at UC Berkeley. He's reframing loss functions for general AI.

Super cool liam but two at Facebook's mL team, he was working on causal inprints for modeling and just cause the modeling in general, then conformal prediction, and then optimal transport for fairness. Those are all super cool topics and highly recommend awesome. All right, one more thing. If people want to contact you online, so that's Twitter, getthub LinkedIn, where do they find you?

LinkedIn is Michael Burke, Medium account is Michael Burke, website is Michael Dburke dot com. And those are all my things basically, all right, reach out. The note from my boss does don't worry about making it so we go a little longer if you want, if you have a longer list, go for it. I mean it was more like going into each of those, but just one quick point that I would be curious for your guys opinion.

Stuart Russell at UC Berkeley, it works on general AI, and he focuses on his hypothesis is basically that with a fixed loss function, you can't have smart general AI because people don't operate with a fixed loss bum correct. So you're getting from your Okay, cool, I just wondering if you guys are on board with that. I think that's a. Super cool new paradigm where you not only learn the data, but you learn the loss function.

Yeah, I mean, I think it's a natural evolution of not just applied to deep learning, although it would more aptly apply to something that has back propagation to adjust for and compensate for that. But I think in general in our profession, that is kind of where where tech will eventually go is more closely emulating human decision processes, which are not only relatively inconsistent, but are adaptable to new information as they as it comes in, and being able to understand interactive elements in a way that I don't think we even understand how human perception works and how the brain works very well.

Yeah, but however it happens to work more closely emulating that behavioral pattern is makes sense like fixed loss functions exactly as you said, it's not a it's not a finite, definitive element about how we decide on things. It's fungible, mutable. Yeah yeah, I feel like it has less application for industry, but for the big mL questions like eye robot style, it's a really cool thought experiment. Yep, I'm trying to explain why I do stupid stuff all the time exactly.

I mean, what's funny about that statement? Not applicable to industry. People said that about neural networks back in the early nineteen eighties when they first sort of came out. People are like, nobody will ever use this computationally.

It takes forever, you know, specialists required to actually write this code in whatever it was at the time, cobol or Fortran. And now look at industry with the advent of cloud computing and much cheaper gpuh salt change, everybody's doing deep learning. It's ubiquitous now, so never seen never. All right, Well, now that we live in a star Trek world, we're gonna wrap up.

Thanks for coming, Michael. This was fun. Yeah, this was great. Ye thanks for having me.

All Right, Well, we'll wrap it up here and until next time, folks max out.