AWS GWLB TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub Runner Enforcement, Docker Root Risk, Lambda IAM Updates, PostgreSQL Upgrade Traps, SonicWall Zero-Days & Better Incident Reviews

AWS can now reset failed Gateway Load Balancer connections instead of letting them hang for minutes. GitHub is getting ready to stop running jobs on stale self-hosted Actions runners. And a Linux desktop setup accidentally gave basically every user-space process a path to root through Docker. This week is mostly about failure modes hiding inside things we already trust.

I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show about the DevOps, SRE, cloud, platform, and security stories that matter when you are the person keeping the thing running at three in the morning. For the weekly story list and source links, check out OnCallBrief.com.

For past episodes and show notes, head over to ShipItWeekly.fm. This week, AWS Gateway Load Balancer gets TCP Reset to shorten recovery when a firewall appliance dies. Azure DevOps gets live migration tooling for moving enterprises to GitHub.

GitHub is starting enforcement against stale self-hosted actions runners. And Omarchy shipped with a Docker configuration that effectively made normal desktop processes root-capable. Then we have a quick lightning round and a human closer about whether incident reviews should happen asynchronously first, live later. Let's get into it.

First up, AWS Gateway Load Balancer now supports TCP Reset. This is one of those small-looking features that solves a pretty ugly failure mode. Gateway Load Balancer is commonly used to put firewalls, intrusion detection systems, and other network appliances inline with application traffic. When one of those appliances fail, new connections can usually move to a healthy target.

Existing TCP connections are harder. They may keep trying to use the failed path until the connection times out. Depending on the application and TCP settings, that can mean 30 seconds. It can also mean several minutes.

From the application's point of view, the network is not obviously broken. It is just hanging. With TCP Reset enabled, Gateway Load Balancer can actively terminate those affected connections. The client gets a reset immediately and can reconnect through a healthy appliance.

That turns an ambiguous failure into an explicit one. And I think that is the useful lesson here. Fast failure is often better than slow uncertainty. A failed request gives the application something that it can react to.

A connection sitting there doing nothing is much harder. Retries do not happen yet. Timeout budgets get consumed. Users wait.

Threads stay blocked. And everything upstream looks degraded instead of clearly failed. We talk a lot about graceful failover, but sometimes graceful means being very direct. This path is dead.

Start over somewhere healthy. This is especially important around stateful middleboxes because the backend application may be perfectly fine while the network path in front of it is not. The shorter you can make that ambiguity window, the faster the rest of the system can recover. Next, Microsoft has put Enterprise Live Migrations from Azure DevOps to GitHub into public preview.

This is aimed at large migrations where shutting down engineering for a weekend is not really an option. The idea is that repositories remain writable in Azure DevOps while changes continuously synchronize into GitHub Enterprise Cloud. Teams keep working during most of the migration. Then, when it is time for the final cutover, you stop writes, sync the remaining changes, and switch over.

Microsoft says the final downtime can typically stay under about 30 minutes. The migration tooling can also convert Azure DevOps branch policies into GitHub rulesets. And importantly, Azure pipelines can continue running while repositories move. That matters because repository migration is rarely just copying Git objects.

The hard part is everything attached to the repository. Permissions, branch protections, CI/CD. webhooks, secrets, bots, service connections, release workflows, and all of the little assumptions around URLs and identities that accumulate over 10 years. A migration can be technically successful and still be operationally terrible if engineers lose half a day every time you move another batch of repositories.

Live synchronization changes that equation. It lets the migration behave more like a controlled transition instead of a hard stop. But I would still treat this like any other production migration. Inventory what depends on Azure DevOps before you move anything.

Test policy conversions. Validate pipelines. Know what happens to automation that references old URLs. And have a rollback path for the cutover itself.

The goal is not just moving the code. The goal is moving the development system around the code without everybody discovering the missing pieces on Monday morning. Third, GitHub is getting serious about stale self-hosted Actions runners. This one is worth checking if you manage your own runner fleet.

GitHub is rolling out minimum version enforcement. Self -hosted runners need to be at least version 2.329.0 to register.

And runners also need to stay within 30 days of current releases to continue executing jobs. GitHub is starting runtime brownouts on September 9th. Full enforcement is planned for September 25th. The obvious action is to update your runners.

The more interesting question is why they are stale in the first place. A lot of teams treat self-hosted runners like infrastructure appliances. Build the image. Deploy the runner.

Forget about it until something breaks. But the runner is executing arbitrary CI workloads with access to repositories, credentials, networks, artifact stores, and often cloud permissions. It is part of your security boundary. It is also a client of GitHub's service.

And that client has to evolve with the platform. If your upgrade process requires somebody remembering to rebuild an AMI every few months, enforcement like this is eventually going to catch you. This is where ephemeral runners are really attractive. Start from a maintained image.

Run the job. Destroy the environment. Regularly rebuild the base image and roll forward automatically. That does not eliminate maintenance.

It makes maintenance part of the lifecycle instead of an exception to it. And if you cannot automatically replace a runner fleet today, this brownout is probably a good excuse to figure out why. Because September 25th is a much nicer time to discover that problem than during a production deployment. Fourth, Omarchy fixed a configuration that effectively gave user-space applications a path to root through Docker.

The default user was part of the Docker group. That is common on Linux systems because it lets you run Docker commands without sudo. It is also basically root access. The Docker daemon runs as root.

If your user can talk to the Docker socket, you can generally start a privileged container, mount the host filesystem, and modify the machine. So anything running as that user inherits a very powerful capability, a browser extension, an IDE plugin, a compromised development tool, a coding agent, malware running in the session. If it can reach the Docker socket, the jump from normal user to root may be trivial. Omarchy fixed this in version 4.

0.1. But the bigger point is not really about Omarchy. The Docker group has been a known privilege boundary for a long time.

What has changed is how much software we now run inside developer sessions. IDE extensions, package managers, AI coding tools, local agents, browser automation, dev containers, CLI plugins. A modern workstation is a pretty busy environment. Giving every one of those processes indirect root access because Docker is more convenient without sudo is worth thinking about.

Convenience is fine. Just name the privilege correctly. If membership in a group means the user can become root, treat that group like root access. Because an attacker absolutely will.

Quick lightning round. First, AWS Lambda now supports fuller IAM resource-based policies. You can define multiple principals and actions and use normal IAM condition keys for cross-account access. That should make larger Lambda authorization models cleaner than some of the older one permission at a time workflows.

Second, AWS is warning about PostgreSQL upgrades that can fail because of circular role memberships. RDS and Aurora upgrades from PostgreSQL 14 or earlier to 15 and newer can hit problems if roles like rds_ superuser and the built-in read-all or write- all roles end up referencing each other. AWS has a pre-upgrade query to find the problem. Not exciting.

Very useful before your maintenance window. Third, a researcher released a proof of concept called FalconFlank, claiming a CrowdStrike privilege escalation path. Right now, I would treat this one carefully. There is public research and a PoC, but no CVE, affected version matrix, or CrowdStrike advisory yet.

Interesting enough to watch though. Not enough evidence to panic. And fourth, SonicWall SMA1000 appliances have zero-day vulnerabilities being actively exploited. If those appliances are in your environment, this is firmly in the patch-now category.

Internet- facing remote access infrastructure is not where I want to wait and see how exploitation develops. The human closer this week is about incident reviews, specifically whether distributed teams should start them asynchronously instead of immediately scheduling another meeting. One SRE team wrote about reconstructing a 37-minute outage almost entirely async. People added logs, timelines, screenshots, metrics, code references, and corrections directly into the review document.

That part worked really well. It gave people time to check evidence instead of relying on memory in a meeting. Someone could make a claim, and another engineer could verify it. The timeline became much more precise, and people in different time zones could contribute without everybody needing to be online at once.

But there was a downside. Disagreements became long comment threads. Questions about causality got buried. And action-item ownership became fuzzy.

That sounds pretty familiar. Async communication is really good at collecting information. It is not always good at resolving ambiguity. A document can tell you what happened.

A comment thread can show that three people disagree about why it happened. At some point, putting those three people in the same conversation is probably faster. The model they landed on was basically async first, live later. Use the document to gather evidence and build the timeline.

Then hold a short live review only for the things that actually need discussion. Disputed causes, tradeoffs, decisions, and ownership. I like that a lot because the goal of a postmortem is not having a postmortem meeting. The goal is understanding the incident well enough that the organization learns something from it.

Sometimes meetings help with that, but sometimes they just force 12 people to watch one person scroll through Datadog. Async evidence gathering also changes who gets heard. In a live review, the people who remember the incident fastest or speak the most confidently can shape the narrative early. In a shared document, somebody can go back, check the logs and say, actually, that happened four minutes later.

Or this alert fired before the deployment. Or the database was already degraded before the API started failing. That is useful. But the async part still needs structure.

Someone owns the timeline. Someone identifies unresolved questions. Someone turns the discussion into actions. And someone actually owns those actions afterward.

Otherwise, you just traded a bad meeting for a very detailed document nobody closes. The thing I like most here is that it treats communication style as part of incident response design. Distributed teams do not have to recreate the same meeting- heavy process companies used when everybody sat in one office. Use async for the things async does well.

Use live conversation for the things humans resolve better together. And keep both focused on learning, not blame. Because the best incident review is not the one with the most complete document. It is the one that actually changes how the next incident goes.

That's it for this week's Ship It Weekly. We covered Gateway Load Balancer TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub's self-hosted Actions runner enforcement, and the Omarchy Docker privilege issue, plus Lambda IAM policy improvements, Postgres SQL upgrade traps, the FalconFlank research, and SonicWall zero-days. Follow or subscribe wherever you are watching or listening. You can find the weekly story list and source links at OnCallBrief.

com. and past episodes and show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech.

Thanks for listening. And remember, recovery gets easier when systems fail clearly, migrations happen deliberately, and the humans reviewing incidents have enough space to figure out what actually happened.