Your retry policy is a denial-of-service attack on yourself

The outage lasted 47 minutes. The database was healthy for 44 of them. What kept the service down was us — specifically, 11,400 requests per second of retries against an API that had been serving 900 requests per second before the incident began.

How one request became 81

Our stack had four layers between the browser and the database, and every one of them retried on failure, three times, with a fixed 200ms delay.

• Browser SDK: 3 attempts
• API gateway: 3 attempts per upstream call
• Service: 3 attempts to the database client
• Database client: 3 attempts to the connection pool

3 × 3 × 3 × 3 = 81. When the database stalled for eight seconds during a failover, every in-flight request fanned out into up to 81 attempts, all of them landing inside the same 600ms window because the delays were fixed.

Why does fixed backoff make it worse?

Because every client that failed at the same moment retries at the same moment. Fixed delays synchronise your clients into a wave. Exponential backoff spreads the wave; jitter breaks it up. Without both, a brief fault becomes a periodic pulse of traffic that hits the recovering system exactly when it is most fragile.

A retry is a bet that the second attempt will succeed. Eighty retries is a bet that the system has infinite capacity.

The AWS Architecture Blog piece on exponential backoff and jitter has the simulation that convinced us. Full jitter cut the total work in their model by roughly half compared with plain exponential backoff.

The policy we replaced it with

Three rules, applied across the stack.

1. Retry at one layer only. The edge — the API gateway — owns retries. Everything below returns errors immediately.
2. Retry budget, not retry count. Each service may spend at most 10% of its request volume on retries in any 10-second window. Above that, retries are dropped and the caller sees the error.
3. Full jitter, exponential base. Delay is a random value between 0 and min(cap, base × 2^attempt).

const delay = (attempt: number) =>
  Math.random() * Math.min(20_000, 250 * 2 ** attempt);

The retry budget was the change that mattered most. A count-based policy scales with load; a budget scales with what the system can afford. During the next database failover — 14 weeks later — retry traffic peaked at 1.1× baseline instead of 12×, and the incident lasted six minutes.

The idempotency problem you already have

Once you retry at the edge, you are replaying requests that may have partially succeeded. We found 19 endpoints that were not idempotent — mostly POSTs that created records without a client-supplied key. Two of them had been silently creating duplicate invoices whenever the old retry storm fired. Adding an Idempotency-Key header and a 24-hour dedupe table fixed both, and cleaned up 3,200 historical duplicates we had not known about.

What we still retry below the edge

Almost nothing. The database client retries once, immediately, on a connection-reset error only — the one case where the failure is known to be transient and the retry is known to be safe. Everything else fails up to the gateway, which decides whether the budget allows another attempt.

Latency at p99 dropped from 2.4 seconds to 840ms after the change, not because the happy path got faster but because failing requests stopped waiting through four layers of retries before admitting defeat.

If your stack has more than one layer that retries, you have a multiplier you have not measured. Work it out. Then decide whether you would ever deliberately send that many requests to yourself.