Measure Up

The episode contains genuinely non-obvious technical ideas - deterministic callbacks as guardrails independent of agent instruction, the 'forking world of choices' problem in causal inference, and foundational models as a coming paradigm shift for MMM - but is diluted by substantial opening chit-chat, circular 'will AI replace us' discussion, and some vague hand-waving around value and demand.

“instead of telling the agents to verify this property, you can just add it to a callback to every model that it creates by adding it as a deterministic script on top of the code base. And so this is a, uh, check that needs to always happen. Don't tell the agent to do it. You tell deterministic script to do it as a hook and a callback.”

“you can have 10 models, they fit perfectly your data but they have different, they correspond to different uh, ways of understanding the world and reality depending on their configurations.”

The 'intent and taste' framing for residual human value is a memorable and reasonably fresh articulation, and the idea of biasing agents via skills files rather than teaching them to reason is a useful non-obvious distinction; however, the broader 'will agents replace data scientists' arc and open-source-vs-SaaS debate are well-worn territory in 2024-25 AI discourse.

“I do think that there are going to be only two things that distinguish us humans from AI ah in the future. And these two things are intent and taste.”

“the skill doesn't really teach the agent how to reason, but it tells the agents hey, if there are five different ways of writing these models, well the most appropriate one um, for these classes of problems is the number fourth and this is the reason why.”

Luca Fiaschi is a genuine senior practitioner - PhD in ML, VP-level roles at Stitch Fix and HelloFresh, CDAO at Mistplay - who is now actively building and shipping the systems under discussion, lending real credibility; he loses points only for being in a vendor-advocacy role that occasionally colors his answers.

“Before PYMC Labs, he was a Chief Data and AI Officer at Mistplay, VP of Data Science at Stitch Fix, VP CP of Data and Machine Learning at HelloFresh”

“we had measured um, one way of measuring this is to give uh, the LLMs, um, the agent um, coding the task of Coding specific well known m, um Bayesian models, uh, for example for stochastic um, price volatility.”

There are some concrete anchors - eMarketer's 62%/90% MMM adoption figures, a specific 0%-to-70% pass-rate improvement with skills files, named tools like Kronos and TimeGPT as analogs - but several key claims (token cost risk, demand growth, value unlock) are stated as belief without data, and the paper references are left unnamed and vague.

“I read that these surveys from emarketeers, very recent, a couple of months ago came out and says that 62% of all advertisers are using mmms”

“the pass rate tremendously increase from something that's close to 0% to something that's close to 70% uh, on something of the hardest problems”

The hosts ask a few genuinely useful follow-up questions - particularly on whether agents always follow skills files and on token cost economics - but the conversation frequently meanders into long host monologues, vague paper references ('the next generation neural nets'), and the closing section dissolves into community-promotion rather than pressing on unresolved tensions.

“Does it always adhere to those skills or does it sometimes ignore them?”

“at what point do we no longer need a data scientist? Are the agents going to be good enough where the cmo, the head of marketing is just working with the agents directly”