Adventures in Machine Learning

The episode delivers substantive technical content on A/B testing, variance reduction techniques (CUPED), power calculations, and frequentist vs. Bayesian experimentation. However, it mixes this with considerable filler - extended casual banter, off-topic discussions about blogging and career advice, and lengthy tangents on board games and book recommendations that dilute the core insights. The A/B testing content itself is solid but not densely packed; there are long stretches of explanation and socializaton.

“So you essentially put in your type one error, your type two error rate, and a couple of other things like minimum detectable effect, and that will output a number of unique units”

“the larger the variance, the larger the sample size. By a good amount, and it's really powerful. Well if you can thereby reduce the variants prior to the using information prior to the experiment, and that can really cut down sample size”

The episode covers industry-standard A/B testing practices (frequentist methods, power analysis, CUPED variance reduction) that are well-documented and widely implemented at major tech companies. The guest explicitly states these are not novel - 'every major tech company that I know of does it.' While the application to ad-break structuring is specific to their platform, the underlying methodologies are mainstream. The discussion lacks contrarian arguments or first-principles challenges to established practices.

“We subscribe to Frequentist Experimentation. The other main area is Beayjian experimentation, and they have their pros and cons”

“Every major tech company that I know of does it, like the Googles, the facebooks, the netflixes, and we've stolen some of their ideas”

Michael Berk is a data scientist at a 400-500 person streaming company with ~2 years full-time experience, working on A/B testing infrastructure and ads configuration. He has relevant operational experience and has shipped production systems, which is valuable. However, he is relatively junior in his career and works at a mid-size company, not a tier-1 tech leader like Google or Netflix. His blog project and side work show thoughtfulness, but he hasn't achieved the seniority or scale of impact of someone from a FAANG company or a founding operator.

“I have been working as a data scientist full time for about a year and a half now and then part time for a couple of years before that. Currently, I work at a company called tv”

“I'm pretty early in my career, so it's kind of wild stuff”

The episode includes some concrete examples (ad-break frequency/duration experiments, revenue lift from adding one ad per break at 20-30%, the CUPED technique with linear regression), but lacks deep specificity overall. Most claims remain at the framework level without specific metrics, timelines, or company results. The ad revenue example is mentioned briefly but not explored with numbers. Blog post discussions reference research papers but not with enough detail to verify claims. Missing are concrete case studies with before/after metrics or specific business impact figures.

“if you add one ad per break, you increase revenue by like twenty percent or thirty percent, and just with that”

“we get eighty percent of the lift from twenty percent of the results. That's a like. A pareto something number”

The hosts ask reasonable follow-up questions and show genuine engagement, particularly Ben's clarifying questions about variance and confounders ('So you're trying to reduce whatever effects are making it go all over the place'). However, the conversation frequently drifts off-topic into career advice, blogging, board games, and book recommendations. There is limited productive pushback or challenge to Michael's claims; when disagreements arise (e.g., about feature importance vs. causal claims), they're conceded quickly rather than explored. The hosts prioritize moving through content to meet a deadline rather than deep inquiry.

“So when you're looking at stuff roll like evaluating changes, when you have your control group, how do you determine how many users would have to be in that in order to extract a signal?”

“So you could use more sophisticated models, not just a simple linear regression. You could say, I want to use Whole Winter's exponential smoothing, or I want to use a REMUD or ceramax”