Webhooks

Why it matters

When a payment settles, a build finishes or a form is submitted, the party that cares is often another system, not a browser. Webhooks let the provider push an HTTP request to a URL the consumer registered, instead of the consumer polling. Every platform with integrations offers them, and the design shows up on both sides: how do you deliver them reliably, and how do you receive them safely.

The idea

A webhook is an HTTP POST from provider to consumer, sent when an event occurs, to a URL the consumer registered in advance. The body describes the event; the consumer acknowledges with a 2xx quickly and does its work afterwards. It inverts the usual direction: the provider is now a client calling into someone else's server, over a network it does not control, to an endpoint that may be slow, down or malicious.

Delivering reliably (provider side)

1. 1

   Never call out from the request path

   Write the event to an outbox or queue in the same transaction as the state change, and let workers deliver. A slow consumer must not slow your checkout.
2. 2

   Retry with backoff and a cap

   Treat anything but a 2xx within a short timeout (5 to 10 seconds) as failure. Retry with exponential backoff over hours or days, then stop and surface the failure in a dashboard or dead-letter list the consumer can inspect and replay.
3. 3

   Deliver at least once, with an event id

   Retries mean duplicates. Give every event a unique id and a timestamp so receivers can deduplicate.
4. 4

   Sign every payload

   Compute an HMAC over the body plus a timestamp with a per-consumer secret and send it in a header. Receivers verify it to know the call came from you and was not replayed.
5. 5

   Protect yourself from the URL

   Consumers can register any URL, including your internal addresses. Resolve and block private IP ranges (SSRF defence), cap response size, follow no redirects, and rate-limit per consumer.
6. 6

   Order loosely

   Parallel delivery and retries break ordering. Send a sequence number or the full current state so receivers can ignore stale events.

Receiving safely (consumer side)

Verify the signature before parsing anything, and reject timestamps older than a few minutes to stop replays. Return 200 immediately after writing the event to your own queue; do the real work asynchronously, because the provider's timeout is short and it will retry if you are slow. Make processing idempotent keyed on the event id, because you will receive duplicates. Do not trust the payload as the source of truth for anything important: treat the webhook as a notification and fetch the current state from the provider's API when it matters (money, access). Offer a way to reconcile: a periodic poll of recent events catches anything a webhook outage missed.

Operating at scale

Fan-out is per subscriber: an event with a thousand subscribers is a thousand deliveries, each with its own retry state. Isolate consumers so one slow endpoint does not hold worker capacity hostage: per-consumer queues or concurrency limits, circuit breakers that pause delivery to an endpoint that keeps failing. Expose a delivery log and a replay button; consumers will ask for both within a week of launch.

Trade-offs

Webhooks deliver quickly and cheaply for the provider and shift operational burden to the consumer, who must run a reachable, verified, idempotent endpoint and still reconcile. They cannot be more reliable than the consumer's uptime, so pairing them with a pull-based reconciliation is the mature design. Signed, retried, dead-lettered webhooks are a real subsystem, not an afterthought.

In an interview

Cover both sides in one pass: "Events go to an outbox in the state-changing transaction; workers deliver with HMAC signatures over body and timestamp, retry with exponential backoff for 24 hours, then dead-letter with a replay UI. Receivers verify, ack fast, process from a queue idempotently on event id, and reconcile with a nightly poll." Mention SSRF protection on registered URLs if the provider is public; it is the detail most candidates miss.