How to Engineer AI Inference Systems with Philip Kiely - #766

Episode notes

In this episode, Philip Kiely, head of AI education at Baseten, joins us to unpack the fast-evolving discipline of inference engineering. We explore why inference has become the stickiest and most critical workload in AI, how it blends GPU programming, applied research, and large-scale distributed systems, and where the line sits between inference and model serving. Philip shares how research-to-production can move in hours, not months, and why understanding “the knobs” of inference - batching, quantization, speculation, and KV cache reuse - lets teams design better products and SLAs. We trace the inference maturity journey from closed APIs to dedicated deployments and in-house platforms, discuss GPU lifecycles, and survey today’s runtime landscape, including vLLM, SGLang, and TensorRT LLM. Finally, we look ahead to agents and multimodality, making the case for specialized, workload-specific runtimes when performance and efficiency matter most. The complete show notes for this episode can be found at .

• Why AI Agents Break the GenAI Security Model with Devvret Rishi - #77077 / 100
• Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut - #76982 / 100
• Relational Foundation Models for Enterprise Data with Jure Leskovec - #76892 / 100
• How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
• How Capital One Delivers Multi-Agent Systems with Rashmi Shetty - #765