The Developer Tools Podcast with Fexingo · 2026-07-01 · 10 min
Key moments - from our scoring
Substance score
78 / 100
Five dimensions, 20 points each
Payment idempotency is far more subtle than treating it as a simple API deduplication header. Lucas and Luna examine a concrete scenario where a $50M-annual SaaS platform's checkout service generated idempotency keys from session ID plus timestamp, causing two requests within the same second to share a key. While Stripe's API correctly returned the cached result, the merchant's checkout layer misinterpreted this as a new charge, leaving customers with duplicate billing. The hosts unpack why deriving keys from transient state (sessions, timestamps) breaks idempotency, the state-machine trap where 'first write wins' can lock payments in transitional states, and why generic Redis-based idempotency middleware fails for multi-step payment flows. They also cover key expiration pitfalls - 24-hour TTLs are insufficient for wire transfers and bank delays - and discuss the real-world fix: using invoice ID plus attempt number as the key, paired with a reconciliation job that reverses duplicates at the business logic layer. The episode emphasizes that perfect API-level idempotency is often impossible in distributed systems, making layered idempotency checks and targeted testing more practical than claiming absolute guarantees.
Their checkout service generated idempotency keys from session ID plus second-level timestamp. When the server retried within the same second after a timeout, both requests got the same key. Stripe correctly returned the first charge as a cached result, but the checkout layer misinterpreted the response as a new charge and billed the customer twice.
Derive it from immutable business identifiers that won't change, such as invoice ID plus a sequence number for retries (e.g., 'inv_123-attempt-1'). Avoid transient state like session IDs or low-granularity timestamps, and include a suffix if retrying to avoid collision.
Stripe recommends a minimum of 24 hours, but payment flows with bank delays or wire transfers may need 48 - 72 hours. The TTL must exceed your maximum expected processing time plus your retry window, or late retries will create new charges.
Not reliably for multi-step payment flows. Generic middleware caches responses blindly, but payment state machines need context-aware logic - once a payment succeeds, the same key should return the result without allowing further state changes, while transitional states allow retries. Your idempotency layer must understand your business logic.
Layer idempotency at every boundary (client, middleware, payment provider, database), use business-level deduplication on stable identifiers, and run a reconciliation job that detects and reverses duplicate charges. Perfect API-level idempotency is fragile; 'good enough' with a sidecar is more robust.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode is densely packed with specific, non-obvious technical insights about idempotency-key design failures and their business consequences. Nearly every exchange introduces a novel problem (session/timestamp granularity failure, state machine locking, TTL expiration during in-flight requests, reconciliation as pragmatic alternative) that would be genuinely valuable to a developer building payment systems. Minimal filler or throat-clearing.
The idempotency key didn't prevent the duplicate; it masked it. The user got two charges, but the API said it was one.
An idempotency key can lock a resource into a state that prevents legitimate retries.
The framing of idempotency as a business-level invariant rather than just an API header is fresh and contrarian. The specific case of state-machine locking preventing legitimate retries, the discussion of key expiration during in-flight requests, and the pragmatic reconciliation-sidecar approach as 'good enough' idempotency are not standard frameworks found in typical payment-API discussions. The episode avoids recycled platitudes.
Stripe's documentation is actually pretty clear: the idempotency key should be derived from the immutable properties of the operation you're performing, not from transient state like session or timestamp.
Perfect idempotency at the API layer is incredibly hard, especially across distributed services. Sometimes 'good enough' with a reconciliation sidecar is better than a fragile system that claims to be fully idempotent.
Lucas demonstrates deep practical expertise in payment systems and distributed systems design. While not a household name, he speaks with the authority of someone who has debugged and solved real idempotency failures at scale (discussing what he's 'seen teams implement' and referencing Stripe's internal patterns). Luna serves as a knowledgeable foil asking sharp follow-ups rather than a true guest, reducing the overall guest-caliber score slightly.
A lot of developers treat them as a simple deduplication header and don't think about what 'same intent' means.
I've seen teams implement a generic Redis-based idempotency service that stores the response for each key for 24 hours. That works fine for simple CRUD, but for multi-step payments it's a disaster.
The episode opens with a concrete real-world failure ($50M SaaS company, Stripe dashboard, timestamp-to-the-second granularity, session ID reuse). It names Stripe multiple times, references a specific GitHub tool ('idempotency-tester'), and discusses concrete retry patterns ('inv_123-attempt-1'). The TTL recommendations (24 - 72 hours) are specific. Some discussion remains slightly abstract (e.g., 'business identifier'), but the specificity is strong throughout.
a mid-sized SaaS company processing about $50 million annually in subscription payments starts seeing a weird pattern
The timestamp was only to the second, and their server sometimes retried within the same second after a timeout.
Luna asks sharp, clarifying follow-ups that push the discussion deeper ('And I can already guess what went wrong', 'Doesn't that break idempotency for the retry?', 'What's the most effective way to test this?'). However, the conversation lacks genuine disagreement or productive tension; Luna mostly validates Lucas's points rather than challenging them. A few moments feel slightly scripted (the transition to the donation appeal). Host follow-ups could probe more ruthlessly on trade-offs.
But doesn't that break idempotency for the retry? If attempt 2 fails, a retry of attempt 2 with the same key works, but a retry of attempt 1 later would be a different key.
Perfect idempotency at the API layer is incredibly hard, especially across distributed services. Sometimes 'good enough' with a reconciliation sidecar is better than a fragile system that claims to be fully idempotent.
Computed from the transcript - who did the talking, and the words that came up most.
Every developer building payment systems has seen a duplicate charge disaster. The fix is idempotency keys, but most implementations leak logic errors that make them unreliable. Lucas and Luna break down exactly where idempotency-key design goes wrong - using Stripe's approach as the gold standard, with specific failure modes like the 'first write wins' vs 'last write wins' debate, key expiration edge cases, and state-machine collisions. They walk through a real-world example: a SaaS company that lost six-figure revenue in 2025 because their idempotency layer allowed partial retries on a three-leg payment flow. Listeners will learn why idempotency isn't just a 'dedup' header, why you need a deterministic key strategy, and how to test for the four most common idempotency bugs before they hit production. #Idempotency #PaymentSystems #Stripe #API #DeveloperExperience #BackendEngineering #SaaS #Fintech #DistributedSystems #RetryLogic #StateMachines #IdempotencyKeys #PaymentDisasters #DuplicateTransactions #Business #Technology #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo
Transcribed and scored by The B2B Podcast Index.
Lucas: So it's mid-2024, and a mid-sized SaaS company processing about $50 million annually in subscription payments starts seeing a weird pattern in their Stripe dashboard: a small but growing number of customers are getting charged twice for the same invoice. Luna: The classic double-charge nightmare. And it's almost never the payment processor's fault, right? Lucas: Exactly.
Stripe's API is idempotent by design - if you send the same request with the same idempotency key, it returns the same result without charging again. The problem was on the merchant side. Their checkout service was using a naive idempotency-key strategy: they generated a key from the user's session ID plus a timestamp. Luna: And I can already guess what went wrong.
The session ID might be reused, or the timestamp had insufficient granularity. Lucas: Both, actually. The timestamp was only to the second, and their server sometimes retried within the same second after a timeout. So two different payment attempts got the same idempotency key.
Stripe saw the second request as a retry and returned the first charge's result - which was a success - but the checkout service interpreted the response as a new charge and told the user 'payment succeeded' twice. Luna: So the idempotency key didn't prevent the duplicate; it masked it. The user got two charges, but the API said it was one. Lucas: Right.
This is the fundamental trap: idempotency keys only work if they're actually unique per intent. A lot of developers treat them as a simple deduplication header and don't think about what 'same intent' means. Stripe's documentation is actually pretty clear: the idempotency key should be derived from the immutable properties of the operation you're performing, not from transient state like session or timestamp. Luna: So what's the right approach?
I've seen people use a hash of the request body plus a user ID, but that breaks if you need to retry with different metadata. Lucas: That's exactly where it gets tricky. The cleanest pattern I've seen is to generate a deterministic key from a business identifier that won't change. For a subscription payment, that could be the invoice ID plus a sequence number for retries.
But even that has edge cases. Let's talk about the state-machine problem. Luna: Oh, this is where it gets fun. An idempotency key can lock a resource into a state that prevents legitimate retries.
Lucas: Yeah. Imagine a payment that transitions through 'pending', 'processing', 'succeeded'. If the first request sets status to 'pending' and then fails before moving to 'processing', your retry with the same key - if your server uses a 'first write wins' strategy - will see 'pending' and skip the retry entirely. The payment is stuck.
Luna: So you need a 'last write wins' for certain states, but that opens the door to overwriting a successful charge if you're not careful. Lucas: Precisely. The solution is a state machine that allows idempotency keys to be reused only for transitions that are idempotent. Stripe actually does this under the hood: once a payment is 'succeeded', retrying with the same key returns the existing result without allowing further state changes.
But if the payment is still in a transitional state, the key allows the operation to continue. Luna: Which means your idempotency layer needs to understand your business logic, not just blindly cache responses. Lucas: Exactly. And that's where most implementations fall short.
They treat idempotency as a generic middleware concern, but it's deeply tied to your specific state machine. I've seen teams implement a generic Redis-based idempotency service that stores the response for each key for 24 hours. That works fine for simple CRUD, but for multi-step payments it's a disaster. Luna: What about key expiration?
Twenty-four hours seems long, but too short and you risk a late retry hitting a different charge. Lucas: That's another common pitfall. Payment intents can take hours to complete if there's a bank delay. If your idempotency key expires after one hour, a retry after that window creates a new charge.
Stripe recommends a minimum key retention of 24 hours, but for some payment flows - like wire transfers - you might need 48 or 72 hours. Luna: And you have to handle the case where the key expires but the original request is still in flight. That's practically impossible to detect without distributed tracing. Lucas: Yeah.
That's the worst-case scenario: a request is slow, the client times out and retries with a new key, and both eventually succeed. Now you have two charges. The only real defense is to keep your idempotency key TTL longer than your maximum expected processing time plus your retry window. And monitor for keys that get reused unexpectedly.
Luna: So let's circle back to the SaaS company from the beginning. How did they fix it? Lucas: They switched to using the invoice ID as the idempotency key, with a suffix for the attempt number. So the first attempt gets key 'inv_123-attempt-1', and if it fails, the retry uses 'inv_123-attempt-2'.
That way each attempt is unique, but the invoice ID ties them together so they can deduplicate at the business level. Luna: But doesn't that break idempotency for the retry? If attempt 2 fails, a retry of attempt 2 with the same key works, but a retry of attempt 1 later would be a different key. Lucas: That's the trade-off.
They decided that retries should always use the latest attempt key, and they built a reconciliation job that checks for any invoices with multiple successful payments and reverses the duplicates. It's not perfectly idempotent at the API level, but it's idempotent at the business level - which is ultimately what matters. Luna: I think that's a pragmatic approach. Perfect idempotency at the API layer is incredibly hard, especially across distributed services.
Sometimes 'good enough' with a reconciliation sidecar is better than a fragile system that claims to be fully idempotent. Lucas: Right. And that brings up the testing dimension. Most teams don't test idempotency failure modes at all.
They test the happy path - send the same request twice, get the same response - but they never test what happens when the key expires, when the state machine is in an unexpected state, or when two different operations accidentally produce the same key. Luna: What's the most effective way to test this? Chaos engineering? Lucas: Partly.
I'd start with property-based testing: generate random idempotency keys, simulate network timeouts, and verify that no duplicate charges occur. Then add fault injection - kill the database mid-request, restart the service, and see if the idempotency layer survives. Stripe actually publishes a test suite for this called 'idempotency-tester' on GitHub, though it's specific to their API. Luna: Wait, there's an open-source tool for testing idempotency?
I didn't know that. Lucas: Yeah, it's not widely advertised. It replays requests with different timing and checks that the side effects are idempotent. But the key insight is: your tests should model the failure scenarios your system will actually encounter, not just the ideal retry case.
Luna: If today was actually useful to you, the way these stay ad-free is listener support - buy me a coffee dot com slash fexingo. Lucas: Absolutely. That kind of support is what lets us dig into these deep technical topics without worrying about sponsors. So, back to idempotency - one more edge case I want to cover.
Luna: Please, I've got more questions. Lucas: What about non-deterministic idempotency keys? Like, what if you generate a UUID for each payment attempt instead of deriving it from business data? Luna: That's actually what a lot of developers default to.
It's simple, always unique, but then you lose the ability to meaningfully retry if the first attempt fails and you don't persist the UUID. Lucas: Exactly. If the client crashes and loses the UUID, you can't retry safely. You'd have to generate a new one, which would create a new payment.
So UUIDs only work if you have a persistent store for the mapping between the business operation and the key. That adds complexity, but for some systems it's the right trade-off. Luna: So the takeaway for developers building payment integrations: think about idempotency as a business-level invariant, not just an API header. Derive keys from stable identifiers, understand your state machine, and test the failure modes.
Lucas: And don't assume that because Stripe handles idempotency on their end, your application doesn't need to. The most robust systems layer idempotency at every boundary: the client, the middleware, the payment provider, and the database. Luna: All right, I'm going to go audit my own idempotency key logic now. This was a great wake-up call.
Lucas: Glad it helped. Next time we'll talk about webhook idempotency - which is a whole other can of worms.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
How Kubernetes Topology Spread Constraints Create Scheduling HotspotsDevOps Daily with Fexingo · features Luna95 / 100
Why Pipeline Velocity Trumps Deal Size Every TimeThe Growth Operator with Fexingo · features Luna95 / 100
Why Enterprise Software Deals Now Include a Vendor AI Model Explainability MandateB2B SaaS Talks with Fexingo · features Luna94 / 100
How B2B Brands Wreck Pipeline with Unsyncroned CRM DataThe Marketing Operator Podcast with Fexingo · features Luna92 / 100
Why Marketing Attribution Misses the Seasonality PatternMarketing Analytics with Fexingo · features Luna91 / 100
How to Sell Against a Competitor Already in the BuildingSales Leadership with Fexingo · features Luna85 / 100
Why API Webhook Payloads Should Be Signed Not Verified90 / 100Why API Rate Limit Headers Confuse Every Developer90 / 100Why API Response Envelopes Waste Bandwidth82 / 100Why API Error Budgets Should Be Debugging Budgets92 / 100How API Rate Limit Headers Mislead Developers85 / 100