Webhook Setup

Receive and reliably process webhooks with signature verification, idempotency, and dead-letter queues.

Why

Webhooks are how the world tells your app that something happened off-platform — a payment cleared, a repo was pushed, a file was uploaded. They are delivered at-least-once, meaning you will receive duplicates, and they arrive in order you don't control. Without idempotency, a duplicate webhook double-processes the event. Without signature verification, an attacker can POST a fake webhook and grant themselves a subscription. Without a dead-letter queue, a transient DB error causes the webhook to fail permanently after the provider's retry window expires. These three mechanisms turn webhooks from a reliability liability into a reliable event source.

When

When you consume any event-driven external service. Takes 2-3 h for a robust handler. Skip for poll-only integrations.

How

1. Expose a POST endpoint that accepts the raw body (not JSON-parsed)
   - Signature verification needs the exact bytes; parse AFTER verifying
   - Return 200 immediately only after successful processing, not before

2. Verify the signature using the provider's secret and algorithm
   - Stripe: HMAC-SHA256 on the raw body, compare with timingSafeEqual
   - GitHub: HMAC-SHA256 on the body, sent in X-Hub-Signature-256 header

3. Check idempotency: look up the event ID in your processed_events table
   - If seen, return 200 (the provider considers 200 a success, no reprocessing)
   - If new, process then insert the event ID in the same DB transaction

4. On processing failure, return 5xx so the provider retries — but cap retries
   - After 5 failures, move the event to a dead-letter queue (DLQ) for manual review
   - Never return 4xx for a processing error — 4xx means "don't retry"

5. Process asynchronously for slow work: enqueue a job, return 200
   - The webhook handler's job is to acknowledge, not to do the work
   - A background worker picks up the job; if it fails, the DLQ catches it

Example

import { createHmac, timingSafeEqual } from "crypto";

export async function POST(req: Request) {
  const raw = Buffer.from(await req.arrayBuffer());
  const sig = req.headers.get("x-hub-signature-256")!;
  const expected = "sha256=" + createHmac("sha256", process.env.WEBHOOK_SECRET!).update(raw).digest("hex");
  if (!timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
    return new Response("invalid signature", { status: 401 });
  }
  const event = JSON.parse(raw.toString());
  if (await isProcessed(event.id)) return new Response("ok", { status: 200 });  // idempotent
  await enqueueJob(event);  // async processing
  await markProcessed(event.id);
  return new Response("ok", { status: 200 });
}

Common Problems

⚠Returning 200 before processing completes, so a crash mid-processing loses the event permanently.
⚠Using === to compare signatures instead of timingSafeEqual, enabling a timing attack on the secret.
⚠Processing synchronously in the webhook handler, so a slow DB call causes a timeout and the provider retries.
⚠Not handling out-of-order delivery, so event #2 processes before event #1 and the state is inconsistent.

Best Practices

✓Store the event ID + processed timestamp in a dedicated table with a 30-day TTL — old duplicates stop arriving.
✓Use a queue with visibility timeout (SQS, BullMQ) so the same event isn't processed by two workers concurrently.
✓Add a replay endpoint (admin-only) that re-fetches the last N events from the provider for manual reprocessing.
✓Alert when the DLQ has >0 items — a stuck webhook is a silent data loss bug.