Telstra’s Time Sync Outage, DoorDash’s 1.5M RPS Cache, Stateless MCP, GitHub and PyPI Supply Chain Delays, and Why the Quietest Infrastructure Often Has the Biggest Blast Radius

DoorDash turned a cache into a reliability layer handling more than one and a half million requests per second. MCP is removing protocol-level sessions so servers can scale more like ordinary HTTP services. And GitHub and PyPI are deliberately slowing package automation, because sometimes waiting is the safer default. This week is about the quiet systems underneath the system: time, cache behavior, state, and dependency policy.

Most people never notice them when they work. But when the assumptions around them are wrong, they can decide how the whole platform behaves. I’m Brian Teller from Teller’s Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at 3 a.

m. For the weekly story list and source links, check out OnCallBrief.com. For show notes and past episodes, head over to ShipItWeekly.

fm. This week: Telstra’s time synchronization outage, DoorDash’s transparent Envoy and Valkey cache, the stateless MCP specification release candidate, and new package-supply-chain controls from GitHub and PyPI. In the lightning round: ECS Action Logs, Network Load Balancer listener rules, larger Managed Prometheus workspaces, and PixelSmash in FFmpeg. And we close with Minus Two Minutes, about reliability work that prevents an incident before customers ever notice it.

Let’s get into it. First up, Telstra experienced a nationwide mobile outage after part of its network timing infrastructure reset its date to November 2006. The failure happened during maintenance on a network timing device. When it restarted, an undocumented design change caused it to return the wrong date.

A software update that should have prevented the problem had also not been applied. That bad time then propagated into the network. At the peak, Telstra said about 45 percent of calls and data sessions were affected. Six hundred and four calls to Australia’s Triple Zero emergency service also failed.

The interesting part is that the time source did not look unavailable. It looked authoritative. It returned a timestamp, and downstream systems trusted it. Accurate time sits underneath authentication, certificates, distributed systems, logs, databases, and telecom infrastructure.

A device can be powered on and reachable while still returning information that makes everything around it fail. And this was not an exotic attack chain. It was maintenance, a missed update, incomplete documentation, and a dependency that had worked quietly for years. That combination should feel familiar.

Critical infrastructure keeps working until teams stop thinking about it. Documentation drifts, experts move on, and routine maintenance eventually exercises an assumption nobody remembered was there. The lesson is not only patch your time server. Time should be treated as a tier-zero dependency.

Use genuinely independent sources. Alert on clock movement that is operationally impossible. Have sanity limits and holdover behavior for a source that is available but obviously wrong. And validate the service after maintenance, not just the hardware.

A green light on the chassis is not the same as a correct timestamp. A tiny piece of infrastructure became the whole production system because everything else depended on it telling the truth. Next up, DoorDash published a deep dive into Entity Cache, a transparent caching proxy built with Envoy and Valkey. A single request may trigger repeated calls for store information, menus, pricing, availability, and fulfillment data.

Much of that changes slowly, but services were still repeatedly calculating it, fetching it, and sending it across the network. Local application caches helped, but every team had to build and operate its own version. So DoorDash moved caching into the platform. Entity Cache sits inside its Envoy-based service mesh and intercepts HTTP and gRPC traffic before requests reach the upstream service.

Applications keep making the same calls. Teams onboard endpoints through mesh configuration instead of changing application code. DoorDash says it now handles more than one and a half million requests per second, with hit rates above 90 percent on many endpoints. But the more interesting part is what happens when the upstream is unhealthy.

Entity Cache uses a soft time-to-live and a hard time-to-live. Under normal conditions, the soft limit determines when data refreshes. But if the upstream is slow or unavailable, the cache can continue serving data that is stale but still inside the hard limit. During one multi-hour outage, DoorDash says this kept valid responses flowing instead of turning one dependency failure into a broader incident.

That is where a cache stops being only a performance optimization and becomes part of the resilience model. But stale data is not automatically safe. A five-minute-old restaurant description is different from a five-minute-old account balance, permission decision, or inventory count. Someone still has to define which data may be stale, for how long, and what happens when the hard limit is reached.

DoorDash also uses Kafka events for invalidation and samples cached responses against live responses, because a successful cache hit does not prove the answer is correct. The strongest part of the design is the failure policy. If the cache is unhealthy, Envoy can route directly upstream. If the upstream is unhealthy, the cache may serve acceptable stale data.

Early refresh and single-flight behavior help prevent stampedes. A cache is not automatically resilience. A cache with ownership, correctness checks, stale-data policy, and fallback behavior can be. Third, the Model Context Protocol published the release candidate for its 2026-07-28 specification, and the headline change is that MCP is becoming stateless at the protocol layer.

Earlier versions used an initialization handshake and protocol-level sessions. Later requests needed to reach an instance that understood the session or use a shared session store. The release candidate removes that requirement. Each request carries the information needed to process it, including the protocol version and client capabilities.

The practical result is that any request can land on any healthy server behind a normal load balancer. You do not need sticky routing simply to satisfy the protocol. That sounds like a protocol detail, but it is really an operations change. As teams move from local MCP demos to hosted services, they inherit normal distributed-systems questions: traffic balancing, replacement, retries, tracing, compatibility, and where state actually lives.

The stateless core moves MCP closer to ordinary HTTP infrastructure. That is good. Boring infrastructure is usually easier to operate. But stateless protocol does not mean stateless application.

Tools may still need browser IDs, basket IDs, or task handles. The difference is that state becomes explicit instead of hiding inside a connection or affinity rule. The release candidate also improves routing metadata, trace propagation, and support for long-running tasks. The operator takeaway is that MCP is growing up quickly.

Once agents depend on an MCP server, it becomes a platform service. It needs authentication, load balancing, rate limits, tracing, safe retries, deployment compatibility, and a clear answer for where the application state lives. Removing protocol-level sessions eliminates one burden. It does not eliminate the need to operate the application.

Fourth story. GitHub and PyPI both introduced package-supply-chain controls that add intentional friction. Dependabot now waits until a normal package release has existed for at least three days before opening a version-update pull request. Security updates remain immediate, and teams can configure or disable the cooldown.

When a compromised or badly broken package appears, automation may open and merge an update before maintainers or researchers identify the problem. A short delay gives the ecosystem time to produce a signal. PyPI is addressing a different version of the same risk. It now rejects new files added to releases more than 14 days old.

A compromised publishing token could previously add a new wheel to an old, trusted version. PyPI says it is not aware of that path being exploited, but there was no technical control preventing it. These controls share an important idea. Speed is not always safety.

Dependency automation is optimized for freshness, but the fastest update path also gives the ecosystem the least time to discover a poisoned release. Routine updates and emergency security fixes do not have to use the same lane. Security fixes can move quickly with focused review. Routine changes can have an observation period.

High-risk dependencies may need provenance checks, internal mirrors, canaries, or manual approval. Automation should remove mechanical work. It should not remove every opportunity for judgment. Sometimes a delay is not inefficiency.

Sometimes it is a security control. Quick lightning round. First, Amazon ECS Action Logs. ECS can now produce timestamped records of actions during deployments and Managed Daemon updates.

Application logs explain the container; Action Logs help explain the orchestrator. Second, Network Load Balancer listener rules. A dual-stack NLB can now route IPv4 and IPv6 clients into different target groups while preserving the original client address. It is a small feature, but it is useful for teams moving toward IPv6 without running two separate ingress stacks.

Third, Amazon Managed Service for Prometheus gets much larger. A single workspace can now support up to one and a half billion active metric series and 200,000 recording and alerting rules. Managed scaling is useful. It does not make a billion poorly labeled time series cheap, understandable, or operationally valuable.

The platform can store the data. Someone still needs to decide whether the data should exist. And fourth, PixelSmash in FFmpeg. PixelSmash, tracked as CVE-2026-8461, is an out-of-bounds write in FFmpeg’s MagicYUV decoder.

A specially crafted media file can crash a vulnerable application and, in some environments, may allow code execution. FFmpeg is embedded in media servers, thumbnail generators, upload pipelines, desktop tools, and containers. Inventory your bundled copies, not only the system package reported by your operating system. The human closer this week comes from an article highlighted in SRE Weekly called Minus Two Minutes.

The idea is negative time to detection: the moment when an operator sees an incident developing and intervenes before customer impact begins. That should count as a reliability success. But many incident systems do not know how to represent it. If customers were never affected, there may be no incident record or recovery time to improve.

The person who prevented the incident can accidentally erase the evidence that they were right. That connects to every story this week. A sanity check could reject an impossible timestamp before it spreads. A cache can absorb an upstream failure before customers notice.

A stateless protocol can remove routing problems before teams build around them. A dependency cooldown can stop a poisoned release before it reaches production. A lot of reliability work looks like nothing happened. Our metrics count outages, pages, failed deployments, and impact.

They are worse at recording the engineer who noticed the signal, paused the rollout, or fixed the hidden system before it became an incident. Prevention is quiet. And when prevention works, it can look exactly like overreaction. So maybe the question is not only, How quickly did we recover?

Maybe it is also, How often did someone recognize the shape of failure early enough that recovery was never required? That is the work that nobody sees. Until it stops happening. That is it for this week’s Ship It Weekly.

We covered Telstra’s time synchronization outage, DoorDash’s Entity Cache, the stateless MCP specification release candidate, and intentional package delays from GitHub and PyPI. Plus, ECS Action Logs, Network Load Balancer listener rules, larger Managed Prometheus workspaces, and PixelSmash in FFmpeg. Follow or subscribe wherever you are watching or listening. And send this to someone who is responsible for one of those quiet systems everyone else assumes will always be right.

You can find the weekly story list at OnCallBrief.com, and the show notes and past episodes at ShipItWeekly.fm. I’m Brian Teller from Teller’s Tech.

Thanks for listening. And remember: the systems nobody notices are often the systems making the real decisions.