An idempotent webhook handler produces the same result no matter how many times an event is delivered. Most handlers check for a previous delivery first. That check only holds if two deliveries can't pass it at the same moment.
Sequential, then concurrent
hwcli replay evt_93df18ab --duplicate 5 # does the "already processed" check work? hwcli replay evt_93df18ab --concurrent 10 # does it survive a race?
Concurrent replays send every copy at the same moment, which is how retried and duplicate deliveries hit production servers with several workers. Typical failures:
- Two orders created: the check and the insert aren't atomic. Use a unique constraint on the event ID and treat a conflict as "already done".
- 500 errors from unique-constraint violations: the constraint works, but the handler doesn't catch the conflict. Return 2xx instead, or the provider keeps retrying.
- Deadlocks or timeouts: the handler holds locks too long while doing slow work. Record the event first and do the work in a background job.
Bursts
--burst 100 --duration 5s spreads deliveries over time to show how latency and errors grow under load, for example when a provider replays a backlog after an outage. Concurrent and burst replays need a paid plan.
Every delivery in a run is recorded with its status and time, so you can compare before and after a fix.