Webhook debugging: find where a delivery broke
By the HookWatcher team · Updated
Start from the symptom
| Symptom | Look first at | Most likely cause |
|---|---|---|
| Provider shows no attempts | Provider's event/webhook settings | Event type not subscribed, endpoint disabled after failures, test vs live mode |
| Provider shows failures, your logs show nothing | Provider's delivery log: status and error | Wrong URL or path, DNS/TLS error, a proxy or firewall answering instead of your app |
| 401 / 403 | Middleware order | Auth or CSRF middleware runs before the webhook route |
| 400 / 422 | Body parsing | Unexpected Content-Type, strict schema validation |
| 500 | Your exception logs | A field you assumed is always present is null or missing |
| Signature invalid | What bytes you sign | Body parsed and re-serialized before verification, wrong secret |
| Timeout, then retries | Handler duration | Slow work done before responding |
| Processed twice | Deduplication | No idempotency check on the event ID |
For anything in the first two rows, point the provider at a capture URL for a minute. If requests arrive there, the provider side is fine and the problem is between the internet and your handler. If they don't, the problem is in the provider's configuration.
Nothing arrives
- The endpoint URL points at localhost or a private address. Providers send from the internet;
http://localhost:3000or10.x.x.xcan't be reached. Use a public capture URL plus local delivery, or a tunnel. - The event isn't subscribed. Most providers send only the event types you select. Check that list before anything else.
- Test mode vs live mode. Stripe, Razorpay and others keep separate webhook configurations and secrets per mode.
- The endpoint was disabled. Providers turn off endpoints that fail for a long time. Look for a disabled state in the dashboard.
- TLS. An expired or incomplete certificate chain shows up as a connection error in the provider's log, never in yours.
4xx responses
A 4xx means something answered, but refused. Compare the path in the provider's log with your route table first; a missing trailing slash or a /api prefix is a common cause of 404s.
401, 403 and 419: middleware runs first
Session auth and CSRF protection don't apply to webhooks: the provider has no cookie and no CSRF token. Exclude the webhook route explicitly and authenticate it with the signature instead. In Laravel, that means excluding the path from CSRF verification; in Rails, skip_forgery_protection on the webhook controller; in Django, @csrf_exempt on the view.
413: body too large
Reverse proxies and frameworks have body size limits (for example NGINX's client_max_body_size, which defaults to 1 MB). Large order or repository payloads can exceed them.
5xx responses
A 500 is your code throwing. The usual cause is a field that isn't present for every event: an optional nested object, a null where you expected a string, an event type you didn't plan for. Don't guess from the documentation; take the request that failed and run it again, unchanged, against your code with a debugger attached:
hwcli replay <event-id> --target :8080/webhooks # same headers, same raw body hwcli diff <failed-event> <working-event> # what differs from one that worked?
The diff ignores volatile fields such as IDs and timestamps, so the field that broke your handler stands out. See comparing failed and successful webhooks and debugging production failures locally.
Signature mismatches
Almost every "invalid signature" bug is one of these:
- The body was parsed before verification. The provider signed the exact bytes it sent. If a JSON parser runs first, re-serializing the object changes whitespace, key order or number formatting, and the HMAC no longer matches.
- The wrong secret. Test and live secrets differ; each endpoint usually has its own; some providers use the app secret, others a per-webhook secret.
- The wrong signed string. Stripe signs
{timestamp}.{body}, Slack signsv0:{timestamp}:{body}, GitHub and Shopify sign the body alone. - The wrong encoding. GitHub and Stripe send hex digests; Shopify sends Base64.
- Clock skew. If you reject old timestamps, a server clock that's minutes off rejects valid requests.
See the first cause happen on a real endpoint:
Try it: reproduce a signature mismatch
Live, on a real endpointVerify over the raw bytes. In Express, use express.raw() on the webhook route instead of express.json():
import crypto from "node:crypto";
import express from "express";
const app = express();
app.post("/webhooks/github", express.raw({ type: "*/*" }), (req, res) => {
if (!Buffer.isBuffer(req.body)) return res.status(400).send("no body"); // undefined without a body
const expected = "sha256=" + crypto
.createHmac("sha256", process.env.GITHUB_WEBHOOK_SECRET)
.update(req.body) // a Buffer: the exact bytes received
.digest("hex");
const given = Buffer.from(req.get("X-Hub-Signature-256") ?? "");
const wanted = Buffer.from(expected);
// Byte lengths first: timingSafeEqual throws when they differ.
const valid = given.length === wanted.length && crypto.timingSafeEqual(given, wanted);
if (!valid) return res.status(401).send("invalid signature");
const event = JSON.parse(req.body.toString("utf8")); // parse only after verifying
// enqueue work using event and req.get("X-GitHub-Delivery") …
res.sendStatus(204);
});To check what your code should produce, generate the expected signature for a body and secret with the signature generator, and use signature tests to confirm invalid and missing signatures are rejected.
Timeouts
Providers wait only a few seconds for a response. GitHub expects one within 10 seconds; Shopify's documented total timeout is 5 seconds. A handler that calls other APIs, sends email or generates a PDF before responding will time out under load, and the provider then retries, so the slow work runs again.
Verify the signature, store the event, enqueue the work and respond. Do everything else in a background job.
Duplicate events
Retries, network errors and provider-side incidents all cause the same event to arrive more than once. Record each event ID and skip ones you've seen, atomically, so two concurrent deliveries can't both pass the check:
CREATE TABLE processed_webhooks (
provider text NOT NULL,
event_id text NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (provider, event_id)
);
-- In the handler: process only if this insert returns a row.
INSERT INTO processed_webhooks (provider, event_id)
VALUES ('github', $1)
ON CONFLICT DO NOTHING
RETURNING event_id;Use the provider's own ID: X-GitHub-Delivery, Stripe's event.id, X-Shopify-Webhook-Id. Then prove it works with duplicate and concurrent replays.
Capture the request that's failing
Frequently asked questions
How do I see the raw body a provider sent?
Capture it before your framework touches it. A capture URL stores the body byte for byte with its SHA-256 fingerprint; in your own app, read the raw body (express.raw(), request.body in Django, $request->getContent() in Laravel) before parsing.
My handler works locally but fails in production. Why?
Common differences: a proxy in front of production (body size limits, stripped headers, a different path prefix), a different secret, middleware that only runs in production, and real payloads with fields your test data didn't have. Capture a production delivery and replay it locally to rule out the last one.
Should I return an error when processing fails?
Return 5xx only when a retry might succeed (a database briefly unavailable). If the event will never be processable, log it and return 2xx, or the provider will keep retrying.
How long do providers keep retrying?
It differs by provider, from a few attempts to several days. Check the provider's documentation and plan your deduplication window to be at least as long.
Sources
- GitHub Docs: Best practices for using webhooks (10-second response)
- GitHub Docs: Validating webhook deliveries
- Shopify Docs: Troubleshooting webhooks (timeouts)
- Stripe Docs: Resolve webhook signature verification errors
- Express: express.raw()
- NGINX: client_max_body_size
Laravel: excluding URIs from CSRF protection · Rails: request forgery protection · Django: CSRF