Beyond The Pilot: Enterprise AI in Action

The episode delivers a genuine cluster of non-obvious operational insights - circuit breakers over token limits, River's mandatory-public-channel design creating unintended collaborative learning, Sim Gym for small merchants who lack AB testing traffic, and the Universal Distillation Platform pipeline - but dilutes them with extended agentic-commerce vision passages and standard platitudes about AI removing toil.

“we have circuit breakers in place which basically allow someone to get a message if something they're doing is long running and spending a lot of tokens”

“River only works in public, which means you can't go into a private channel and say, hey, river, like, help me build”

A few genuinely fresh frames - moving from 'AI reflexivity' to 'AI leverage,' the ideal AI Centaur being two humans plus an LLM rather than one, and the aspiration to let the distillation pipeline auto-select its own target model - but the episode also leans heavily on common Shopify infrastructure-first branding and the ubiquitous 'AI replaces tasks not jobs' line.

“we also moved in 2026 away from AI reflexivity to AI leverage”

“the ideal, like AI Centaur is not just human and LLM. It's like it could be two humans pairing, um, and the LLM helping you kind of remove the toil”

Farhan is a senior practitioner actually running AI engineering at Shopify's scale, and his command of named internal systems (UDP, River, Tangle, Qwik, Sim Gym), specific toolchain decisions, and real infrastructure trade-offs confirms genuine operational depth rather than thought-leader abstraction.

“we call it UDP Universal Distillation Platform. And what it allows us to do is you give it the teacher model, you give it data, you give it the evals and you give it the target model”

“we also have, uh, our own agentic platform like river, which again, switches between models”

Named internal platforms, specific model references (Qwen 3.5, Opus, GPT 5.2), a concrete 2x - 30x cost-reduction range, the 2021 GitHub Copilot deployment date, Toloka as a named data vendor, and the eval threshold illustration (70%) give this episode real evidential texture - though hard merchant-scale numbers and accuracy deltas are mostly absent.

“We, uh, deployed GitHub Copilot in 2021, which is a year before ChatGPT”

“we see savings, uh, in size from like 2x, which is huge, um, down sometimes down to like 30x”

The host asks a good set of follow-up probes - pushing on the frontier-vs-distilled production split, on whether traces could seed a proprietary coding model, and on who owns runaway token spend - but rarely challenges claims or creates productive friction, and several questions are open-ended scene-setters rather than sharp interrogations.

“What sort of split do you see at the moment between in production use of the Frontier LLMs versus distilled models?”

“Are you guys keeping the traces from your, you know, your developers with the aim of eventually being able to train your own coding model”