Why API Webhook Payloads Should Be Signed Not Verified

Lucas: You set up a webhook endpoint, you validate the signature, you sleep easy. Except that sleep is a little too easy for a lot of teams. Luna: Because validating a signature isn't the same as verifying the payload. Lucas: Right.

That distinction is the whole episode. Most webhook implementations I see - especially in startups that built their integration layer in a sprint - treat signature checking as a boolean gate. Did the HMAC match? Good.

Forward the event to the handler. But that HMAC only proves the request came from someone who knows your shared secret. It doesn't prove the payload hasn't been replayed, truncated, or swapped between endpoints. Luna: And the shared secret itself is often the weakest link.

Hardcoded in an env file, rotated never. Lucas: Exactly. So let's start with the Twilio breach from 2024, because it's a perfect illustration. Twilio's webhook signature system used HMAC-SHA256 with a shared Auth Token.

In theory, fine. In practice, attackers who got read access to Twilio's internal systems could harvest those tokens and forge webhook requests that looked perfectly legitimate to any downstream service. Luna: The signature verified, but the payload was fake. The receiving system had no way to know the event didn't originate from Twilio's actual event pipeline.

Lucas: So the fix isn't just 'use a different algorithm.' The fix is to change what you're signing. Most webhook providers sign the HTTP body plus a timestamp. Stripe does this - their signature header includes a timestamp, and they recommend you reject events older than a certain tolerance.

But even that can be circumvented if the attacker can replay an old valid request within the tolerance window. Luna: Which is where nonces come in. If the signature includes a unique event ID, and you track which IDs you've processed, replays become impossible. Lucas: GitHub does that.

Their webhook payload includes an X-GitHub-Delivery header with a UUID. If you store that UUID and check for duplicates, you're immune to replay. But here's the catch - most developers I've audited don't check that header. They validate the HMAC, then process the event.

The UUID is just another header they log. Luna: So the pattern emerges - verification is only as strong as the least-checked component. Lucas: And that component is often the payload itself. What I want to argue for is signed payloads, not just signed requests.

Meaning the event data - the JSON body - should contain a cryptographic digest that was produced by the source system's private key, not a shared secret. That way even if the shared secret leaks, an attacker can't forge a payload that passes payload-level verification. Luna: Isn't that what JWS - JSON Web Signature - does? Some providers use that.

Lucas: Some do. Slack uses a version of that approach. They sign the request body with their client secret, but they also include a versioned signature header so you can rotate secrets without breaking in-flight verifications. But the payload itself isn't signed end to end.

The signature is over the request, not over the event data independent of the transport. Luna: So you're saying the ideal is: the webhook event carries a signature that survives being replayed from a different channel. Lucas: Exactly. Imagine your webhook handler gets the same event from two different providers - or from a queue replay.

The payload-level signature should still verify. That means the source system signs the event at creation time, not at HTTP delivery time. That's a subtle but important architectural shift. Luna: Who actually does that today?

I know Stripe doesn't sign the event object itself. Their signature covers the HTTP body plus timestamp, but the event object is just JSON. Lucas: Stripe signs the entire body, which includes the event object - so in practice, if you capture the raw body and verify the signature, you're effectively verifying the payload integrity for that specific delivery. But if you later store the event and reprocess it from a database, you've lost the signature context.

That's where a payload-level signature would help: the event object itself contains a 'signature' field that you can verify even after the HTTP context is gone. Luna: That would let you safely re-process events from an offline queue without re-fetching from the source. Big deal for reliability. Lucas: Huge.

And it decouples verification from delivery. Right now, most webhook verification is tightly coupled to the HTTP request. If you ever need to retry an event from a dead-letter queue, you either have to re-fetch the original webhook or trust your storage layer. With payload-level signing, the event carries its own proof of origin.

Luna: What about key management? If each event is signed with a private key, you need to distribute the corresponding public key to all consumers. That's more complex than sharing a single secret. Lucas: It is more complex, but it's also more scalable.

You can rotate the signing key without coordinating with every consumer, because the public key can be fetched from a well-known endpoint. GitHub does something similar for their commit signing. For webhooks, you'd have a /.well-known/webhook public keys endpoint that serves the current and previous public keys.

Consumers fetch once, cache for a while, and verify. Luna: That also solves the 'secret in env file' problem. There's no shared secret to leak. Lucas: Right.

The private key lives only on the provider's side. Even if an attacker gets full access to your database, they can't forge webhook signatures because they don't have the private key. That's a massive improvement over the shared-secret model. Luna: But adoption is slow.

Most webhook providers still ship with shared secrets because it's simpler to document. 'Paste this token into your dashboard.' No one wants to explain asymmetric crypto in a getting-started guide. Lucas: That's the tension.

Developer experience versus security depth. And I get it - if the choice is between a shared secret that 80% of teams actually implement and a public-key system that 30% implement because it's too complicated, the shared secret wins for reach. But for critical webhooks - payment events, account deletions, security alerts - I'd argue payload-level signing should be mandatory. Luna: Let's talk about a concrete failure.

Say you're using a shared-secret webhook from a provider, and an attacker gains access to your secret. What's the worst they can do? Lucas: They can send arbitrary events that your system will treat as legitimate. If your webhook handler creates a user on receiving a 'user.

created' event, the attacker can create unlimited fake users. If it processes a payment, they can trigger refunds. The scope depends on what your webhook does. In the Twilio case, attackers used forged webhooks to intercept verification codes and take over accounts.

Luna: And because the signature matched, the logs showed nothing suspicious. Lucas: Exactly. The logs showed 'webhook received, signature valid, processed.' No red flags.

That's why signature verification alone is insufficient. You need to also verify the payload's integrity and provenance. Luna: So if you're building a webhook consumer today, what's the pragmatic takeaway? Should you implement payload-level signing now, or is there a middle ground?

Lucas: Middle ground: use the provider's signature verification, but also implement idempotency checking using the event ID, and enforce a strict timestamp tolerance. That covers the most common attacks. Then, for high-sensitivity events, add an additional layer: fetch the public key from the provider's documentation and verify a signature on a known field. Some providers include a 'signature' field in the event object itself - check for it.

Luna: And if the provider doesn't support payload-level signing, you can't add it on your side alone. Lucas: You can't, but you can pressure them. Or you can build a proxy that signs events on the way in, so your internal services can verify payload-level signatures even if the upstream doesn't provide them. That's what some large-scale integrators do - they ingest webhooks from multiple providers and re-sign them with their own key before dispatching to internal handlers.

Luna: That adds latency and complexity, but for a platform handling millions of events, it might be worth it. Lucas: It's a trade-off. But the core point is: don't treat webhook verification as a single gate. Treat it as a chain - signature, timestamp, event ID, payload integrity.

Each link adds cost, but each link closes an attack vector. Luna: I want to circle back to something you mentioned earlier - the idea that the signature should survive delivery. That's a design principle I don't hear often. Lucas: It's one of those ideas that's obvious in hindsight.

When you store a webhook event in a database for later processing, you lose the original HTTP context. If that event later gets replayed from the database, your system has no way to cryptographically verify it came from the provider. Payload-level signing fixes that. The event carries its own proof, independent of how it was delivered.

Luna: It also enables offline verification. You could air-gap the verification process entirely. Lucas: Exactly. And that's important for regulated industries where audit trails need to be cryptographically verifiable years later.

A webhook event from 2024 should still be verifiable in 2034 if you have the provider's historical public key. With shared secrets, that's impossible because the secret is usually rotated or lost. Luna: Alright, I'm convinced. But I also see the friction.

Most engineers I talk to barely have time to set up a webhook endpoint correctly, let alone design a multi-layered verification scheme. Lucas: That's fair. And that's where tooling comes in. If your webhook framework - like Express or FastAPI - automatically verified payload-level signatures if the provider supported it, adoption would skyrocket.

The problem is that the ecosystem is fragmented. Every provider invents their own signing scheme. Luna: Standardization would help. If there were a common header - say, Webhook-Signature - that all providers agreed on, libraries could handle verification transparently.

Lucas: There have been attempts. The Webhook Security Working Group proposed a standard called 'Webhook Signature' a couple years ago, but adoption has been slow. Stripe, GitHub, and Slack all do their own thing. Until developers demand interoperability, we're stuck with per-provider implementations.

Luna: And that's a shame, because the core idea is so simple: sign the payload, not just the request. Lucas: It is simple. And it's also one of those things that, once you start doing it, you can't imagine not doing it. I've seen teams retrofit payload-level signing after an incident, and they always say the same thing: 'Why didn't we do this from the start?'

Luna: Because it wasn't the default. Defaults matter enormously in security. Lucas: They do. And the default today is shared-secret HMAC over the raw body.

That's better than nothing, but it's not enough for the event-driven architectures we're building now - where events flow through queues, databases, stream processors, and serverless functions. Each hop is an opportunity for tampering. Luna: This kind of deep-dive into API security is exactly why this show exists. If you're building or operating systems that depend on webhooks, these details can save you a lot of pain.

Lucas: And keeping this show ad-free and focused on the technical substance is something we care about. If you find these conversations valuable, consider supporting the show at buy me a coffee dot com slash fexingo. It helps us keep digging into topics like payload signing, without chasing clicks or sponsors. Luna: Yeah, listener support is what makes this possible.

Head over to buy me a coffee dot com slash fexingo if you want to help. Lucas: So, back to the practical side - let's say you're starting a new project today and you need to receive webhooks from Stripe, GitHub, and a custom internal system. What's your verification strategy? Luna: I'd start with Stripe's signature verification - they provide libraries for that.

For GitHub, I'd also validate the delivery UUID. For the internal system, I'd design it from day one with payload-level signing using asymmetric keys. Lucas: That's a solid approach. And for the internal system, I'd add one more thing: the payload signature should be over a canonical form of the event - not the raw JSON, because JSON serialization is non-deterministic.

Use a stable serialization like RFC 8785 JSON Canonicalization Scheme. Luna: Otherwise, whitespace differences could break the signature. Lucas: Exactly. I've seen that exact bug: two different JSON libraries serialize the same object with different key ordering, and suddenly the signature doesn't verify.

So you need to canonicalize before signing. Luna: That's a level of detail most tutorials skip. Lucas: They skip it because it's not exciting. But it's the difference between a system that works in demos and a system that works in production for years.

Payload signing with canonical JSON, event ID deduplication, timestamp range checks, and key rotation - that's the gold standard. And it's achievable today with off-the-shelf crypto libraries. Luna: So the barrier isn't technical. It's awareness.

Lucas: I think it's awareness plus inertia. Once you've built your webhook handling around shared-secret HMAC, changing to payload-level signing is a breaking change. Providers have to update their SDKs, consumers have to update their handlers. It's a coordination problem.

But for greenfield systems, there's no excuse not to start right. Luna: And for brownfield, you can add it incrementally. Start by logging the payload signature if the provider sends one, then start verifying it in non-blocking mode, then enforce it. Lucas: That's a great migration path.

We'll probably do a follow-up episode on how to migrate a legacy webhook consumer to payload-level signing without downtime. But for now, the takeaway is: sign the payload, not just the request. Your future self - and your security auditor - will thank you. Luna: And maybe one day all providers will agree on a standard, and we'll look back at these workarounds the way we look at custom HTTP headers before CORS.

Lucas: From your keyboard to the API gods' ears. I'm Lucas. Luna: I'm Luna. Lucas: This is The Developer Tools Podcast with Fexingo.

We'll be back next week.