When Do Retries Stop Improving Reliability and Start Amplifying an Outage?

Retrying a failed request feels like one of the safest things a distributed system can do. A service times out, so try again. A database temporarily rejects a connection, so try again. An API returns 503 Service Unavailable, so wait briefly and make another attempt.

This works often enough that retries become nearly invisible infrastructure. HTTP clients implement them. Cloud SDKs implement them. Message consumers retry failed jobs. Application code wraps network calls in generic retry helpers. For genuinely transient failures, that behavior is exactly what we want: a second attempt can turn a short network interruption or process restart into something the user never notices.

The problem is that a retry is not free. Every additional attempt is another request, queue entry, database query, connection, unit of CPU work, or downstream operation sent toward a system that may already be saying:

I cannot keep up.

At that point, a mechanism designed to improve reliability can become the mechanism preventing recovery. Amazon describes retries as “selfish” because each retry consumes more server resources in the hope that the caller will eventually succeed. Google SRE describes the same dynamic as a positive-feedback mechanism capable of turning a small overload into a cascading failure.

That leads to a more useful question than:

“Should this request retry?”

The question is:

Under what conditions is another attempt more likely to help the system than hurt it?

Answering that requires more than choosing a retry count. It means reasoning about failure classification, ambiguous outcomes, idempotency, deadlines, backoff, jitter, retry ownership, system-wide retry budgets, overload, and the amount of additional work the system is willing to create while already unhealthy.

A Retry Is Additional Load

Start with a service capable of processing:

10,000 requests / second

Suppose incoming demand rises slightly:

10,100 requests / second

The backend still completes 10,000 requests, while 100 time out or are rejected. If all 100 retry one second later, the next interval receives:

10,100 normal requests
+   100 retries
------------------
10,200 requests

Now 200 requests cannot be processed. If those retry as well:

10,100 normal requests
+   200 retries
------------------
10,300 requests

The backend did not become slower, and real user demand did not increase. The clients manufactured the additional traffic themselves.

That feedback loop can continue:

This is the central idea behind safe retry design:

Retries are not merely error-handling logic. They are a load-generation mechanism.

Thinking about retries this way changes the design question. A retry policy is not only deciding how patient one request should be; it is deciding how much additional pressure clients are allowed to place on a dependency while that dependency is failing.

That is why retry behavior that looks harmless at the level of one request can become dangerous at fleet scale.

Classify the Failure Before Retrying It

A reasonable retry policy starts by asking what actually failed.

Consider these results:

400 Bad Request
401 Unauthorized
404 Not Found
409 Conflict
429 Too Many Requests
500 Internal Server Error
503 Service Unavailable
504 Gateway Timeout
TCP connection reset
Request timeout

Treating all of them as:

catch {
  retry();
}

is dangerous because they do not describe the same kind of problem.

Some failures are plausibly transient:

connection reset
503 Service Unavailable
504 Gateway Timeout
429 Too Many Requests
temporary leader election
short network interruption

Others are much more likely to be deterministic:

invalid request
invalid credentials
schema validation failure
resource does not exist
permission denied

Retrying this request ten times:

POST /users

{
  "email": "not-an-email"
}

will not eventually transform the payload into a valid email address.

A first-pass classifier might look like:

function isRetryable(error: HttpError): boolean {
  return (
    error.status === 429 ||
    error.status === 500 ||
    error.status === 502 ||
    error.status === 503 ||
    error.status === 504 ||
    error.code === "ETIMEDOUT" ||
    error.code === "ECONNRESET"
  );
}

Even that should not be treated as universal truth. A particular 500 may result from deterministic application behavior and fail forever. A 409 Conflict may represent a transient concurrency race for which retrying is entirely appropriate. Whether a failure is retryable ultimately depends on the semantics of the operation and the dependency.

This is why mature SDKs classify errors instead of blindly retrying every exception. Modern AWS retry behavior also limits retries through quota mechanisms so widespread failures eventually cause callers to stop generating more retry traffic and fail quickly instead.

The policy should be able to answer:

What failed?

Is that failure likely to change
if I repeat the same operation later?

If the answer to the second question is no, retrying is simply repeated waste.

Ambiguous Outcomes Make Idempotency Essential

The most dangerous failures are often not the ones where we know an operation failed. They are the ones where the caller cannot tell whether it succeeded.

Suppose a client sends:

POST /payments

The server:

1. receives the request,
2. charges the customer,
3. commits the result,
4. begins returning the response.

Then the connection disappears.

The caller observes:

TIMEOUT

but that observation is compatible with two very different realities:

Possibility A:
The server never processed the operation.

or:

Possibility B:
The server completed the operation,
but the response was lost.

The client does not know which occurred.

This is exactly where retries are both valuable and dangerous. Repeating the operation may recover from an attempt the server never executed, or it may charge the customer twice.

Stripe's API design work frames idempotency around this ambiguous-outcome problem. The client attaches a stable identifier to the logical operation, allowing repeated network attempts to resolve to the same result:

POST /payments
Idempotency-Key: checkout_8ca492_payment

The server can persist something conceptually like:

idempotency_key              result
------------------------------------------------
checkout_8ca492_payment      payment_7421

Now a lost response does not require recreating the business effect:

This is why retry behavior and idempotency belong together.

Backoff protects system stability. Idempotency protects business correctness.

You often need both.

A request that is safe to retry from the infrastructure's perspective may still be dangerous from the business application's perspective if repeating it can duplicate a payment, booking, message, or other side effect.

Backoff Needs Jitter

Immediate retry is rarely useful because the condition that caused the failure usually has not had time to change.

Suppose a dependency fails at:

10:00:00.000

and 10,000 clients retry at:

10:00:00.001

The recovering service has effectively had no recovery window. The clients simply replayed the load.

A fixed delay improves the situation slightly:

await sleep(1000);
retry();

but if the original requests failed together, they may also wake together:

10,000 failures
        ↓
wait 1 second
        ↓
10,000 retries

The spike has moved. It has not disappeared.

Exponential backoff

A better strategy increases the delay as failure continues:

attempt 1 → wait 100 ms
attempt 2 → wait 200 ms
attempt 3 → wait 400 ms
attempt 4 → wait 800 ms
attempt 5 → wait 1600 ms

A simple implementation might be:

function backoff(attempt: number, baseMs = 100) {
  return baseMs * 2 ** attempt;
}

The intent is useful: short transient failures receive another attempt relatively quickly, while sustained failures gradually receive less pressure.

But exponential backoff still has a fleet-level problem. Every client may be running exactly the same algorithm.

Synchronized backoff

Imagine 50,000 clients fail at approximately the same time. They all compute:

retry after 1 second

One second later, the recovering dependency receives:

50,000 retries

Those fail, so every client computes:

retry after 2 seconds

and the same synchronized wave arrives again.

The traffic rate decreases over time, but it remains concentrated.

Jitter breaks coordination

Adding randomness spreads clients across a retry window:

function retryDelay(attempt: number, baseMs = 100, maxMs = 30_000): number {
  const cap = Math.min(maxMs, baseMs * 2 ** attempt);

  return Math.random() * cap;
}

Instead of every client selecting:

retry at exactly 4 seconds

their attempts are distributed across:

0 ─────────────── 4 seconds

Conceptually:

Jitter does not necessarily reduce the number of attempts by itself. It reduces coordination between callers, which is often exactly what a recovering system needs.

Backoff creates breathing room.

Jitter prevents everyone from breathing at the same time.

Retries Multiply Across Service Layers

The amplification problem becomes much worse when several layers independently decide that retries are helpful.

Consider a request path:

Suppose every layer permits:

1 original attempt + 2 retries

Three attempts sounds conservative.

But if the database is unavailable, retries multiply:

Service D tries the DB 3 times.

Service C sees D fail
and tries D 3 times.

Service B sees C fail
and tries C 3 times.

Service A sees B fail
and tries B 3 times.

One top-level operation can generate:

3 × 3 × 3 × 3 = 81

downstream attempts.

With five independently retrying layers:

3^5 = 243 attempts

The architecture that appeared to be adding resilience has created a load multiplier.

Amazon's retry guidance warns about this exact pattern, and Google SRE similarly recommends avoiding retries at every layer of a service hierarchy.

The lesson is not that lower-level retry is always wrong. It is that retry ownership must be intentional.

Give Retry Ownership to a Deliberate Layer

One of the most useful architectural questions is:

Who owns the retry?

If a high-level operation can safely repeat the workflow and still has enough deadline remaining, it may make sense for that boundary to own retries while lower-level calls remain conservative.

For example:

HTTP request
   ↓
Order service       ← owns retry policy
   ↓
Payment service     ← no independent workflow retry
   ↓
Database

That does not mean a database driver can never reconnect or retry a transport-level operation. Lower layers often understand certain failure modes more precisely than application code does.

The important distinction is between purposeful local recovery and several unrelated retry mechanisms all multiplying one another.

You do not want this to happen accidentally:

SDK retries
+
HTTP client retries
+
service retries
+
queue retries
+
load balancer retries

with every layer independently reasoning:

“Retrying is safer.”

A reliable retry architecture should be able to explain where the retry policy lives, which lower-level retries still exist, and what the worst-case number of downstream attempts is for one logical operation.

If nobody can answer that last question, the retry behavior is probably not under control.

Use Deadlines, Not Just Retry Counts

A configuration like:

maxRetries = 5

says very little about the actual user experience.

If each attempt is allowed to block for 30 seconds, "five retries" can mean several minutes of waiting. What callers usually care about is something closer to:

I need the answer within 2 seconds.

That is a deadline.

deadline = now + 2000 ms

Each attempt, timeout, and backoff delay consumes part of the same finite budget.

For example:

async function fetchWithDeadline(
  operation: () => Promise<Response>,
  deadline: number,
) {
  let attempt = 0;

  while (Date.now() < deadline) {
    const remaining = deadline - Date.now();

    try {
      return await withTimeout(operation(), Math.min(remaining, 500));
    } catch (error) {
      if (!isRetryable(error)) {
        throw error;
      }

      const delay = retryDelay(attempt++);

      if (Date.now() + delay >= deadline) {
        throw error;
      }

      await sleep(delay);
    }
  }

  throw new Error("deadline exceeded");
}

Now the retry behavior is part of the caller's end-to-end latency contract instead of being an independent loop that can continue indefinitely.

Deadline propagation matters across service boundaries as well. Imagine a client gives up after:

2 seconds

while a downstream service keeps retrying for:

20 seconds

The work may complete successfully, but its result can no longer reach the original caller. The system is spending resources completing abandoned work.

A useful retry should still have enough time left for success to matter.

Bad Timeouts Can Manufacture Failure

Timeouts are meant to protect callers from dependencies that are too slow.

A timeout that is too aggressive can create failures the dependency would not otherwise have produced.

Suppose normal latency looks like:

p50   = 80 ms
p95   = 180 ms
p99   = 420 ms
p99.9 = 900 ms

Now configure:

timeout = 200 ms

A meaningful percentage of otherwise healthy requests will be terminated by the client.

If those requests are then retried:

healthy but slow request
        ↓
timeout
        ↓
retry
        ↓
additional traffic
        ↓
latency rises further

the client has created another positive-feedback loop.

AWS warns about this failure mode in its timeout and retry guidance. Amazon has also described a more subtle production case where a timeout unexpectedly covered connection establishment and TLS negotiation. New connections after deployments took slightly longer, which caused requests to exceed a timeout that otherwise appeared reasonable. Warming connections and choosing a timeout that reflected the real operation corrected the problem.

The broader lesson is that timeout values should come from latency distributions and business deadlines, not intuition.

A timeout determines when the caller converts slow into failed.

If failed calls automatically retry, it also determines when the caller begins generating additional load.

Overload Is Different From Random Failure

Consider two requests that both end in a timeout.

Scenario A: random failure

A packet disappears while the server is healthy:

Retry probability of success: high
Cost to the system: low

Scenario B: overload

The database is already at 100% CPU, queues are full, and every request is taking several seconds:

Retry probability of success: low
Cost to the system: high

The HTTP client may observe:

timeout

in both cases.

Operationally, they are completely different.

This is why an overloaded service should communicate its condition when possible:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

or:

HTTP/1.1 503 Service Unavailable
Retry-After: 10

The server often has more information about its own recovery state than a caller does. Respecting server-directed retry timing can therefore be better than having thousands of clients independently guess when they should return.

The important point is that failure type changes the expected value of retrying.

When the failure is random and isolated, another attempt may be cheap and useful.

When the failure is systemic overload, another attempt may simply compete with the work that could have succeeded.

Retry Budgets Put a Ceiling on Damage

Backoff determines when a retry occurs.

A retry budget constrains how much extra traffic retries are allowed to generate.

Suppose a client process handles:

10,000 successful requests/minute

and allows at most:

500 additional retry attempts/minute

Once that allowance is consumed, requests fail without another attempt instead of creating unlimited extra load.

Google SRE recommends retry budgets for this reason. AWS SDKs use a related retry-quota mechanism backed by tokens: failed retries consume capacity, successful calls replenish it, and widespread failure eventually empties the retry bucket so clients begin failing fast.

Conceptually:

This changes the retry question from:

“Has this individual request already retried enough?”

to:

“Has this process already generated too much retry traffic?”

That is a significant improvement because outages are usually system-level events. The retry policy should have at least one mechanism that reasons at the system level too.

A retry budget lets the client admit:

Another attempt might help this request,
but the dependency cannot safely afford
all of us making that decision right now.

That is the kind of restraint a recovering service needs.

Sometimes Reliability Means Rejecting Work

Failing requests intentionally can feel uncomfortable because reliability engineering is usually framed around making work succeed.

During overload, however, trying to process everything can reduce the amount of work that succeeds.

Suppose a service can handle:

10,000 QPS

and receives:

14,000 QPS

Attempting to accept all 14,000 may cause:

queue growth
memory pressure
timeouts
retry storms
thread starvation
eventual collapse

Rejecting the excess 4,000 quickly may allow the remaining 10,000 to complete successfully:

That is load shedding.

A fast failure can be healthier than a slow timeout because slow failures continue consuming resources while the caller waits. Fast rejection frees those resources and gives upstream systems a clearer signal that capacity is unavailable.

The decision becomes even more important when failed work automatically retries. A slow overloaded service can cause every layer above it to wait, time out, and generate more calls. A service that sheds excess traffic explicitly may prevent the incident from turning into a cascading failure.

Sometimes the most reliable thing a system can do for the majority of users is refuse some work immediately.

Circuit Breakers Solve a Different Problem

A circuit breaker watches failures and temporarily stops sending traffic toward an unhealthy dependency.

Its state model is usually described as:

CLOSED
normal traffic
   ↓ failures exceed threshold

OPEN
fail immediately
   ↓ cooldown

HALF_OPEN
probe dependency
   ↓ success

CLOSED

Circuit breakers can be extremely useful because they stop repeatedly invoking a dependency that appears unable to serve traffic.

They also introduce additional distributed state and tuning decisions:

What failure rate opens the circuit?

How long should it remain open?

How many probe requests may enter?

What if one application instance
sees failures and another does not?

What happens during partial recovery?

Amazon's retry guidance points out that circuit breakers introduce modal behavior that can be difficult to reason about and test. AWS SDKs instead use retry quotas and token-bucket mechanisms as one way to bound retry pressure without requiring a hard binary open/closed mode.

That does not make circuit breakers bad.

It means they solve a different problem and have different operational characteristics.

A circuit breaker can protect a dependency from calls.

A retry budget limits how much retry traffic is generated.

Backoff changes timing.

Jitter reduces synchronization.

Load shedding limits admitted work.

These mechanisms can complement one another, but they should not be treated as interchangeable names for “resilience.”

Queue Retries Can Amplify Failure Too

Retry storms are not limited to synchronous HTTP requests.

Consider a worker:

queue
  ↓
worker
  ↓
database

The database becomes unavailable. A job fails, so the queue retries it. The worker fails again, so the queue retries again.

With thousands of jobs and a short retry delay, the queue becomes a durable traffic generator aimed directly at the failed dependency.

Meanwhile, new work may continue arriving:

Now fresh work competes with old failing work for the same worker capacity.

A queue changes the timing and durability of retries. It does not make them harmless.

The same design principles apply:

bounded attempts
backoff
jitter
dead-letter handling
dependency-aware throttling
retry budgets

Eventually some work needs to leave the automatic retry path entirely rather than cycling forever.

For example:

retryable transient failure
        ↓
bounded delayed retries
        ↓
attempts exhausted
        ↓
dead-letter / investigation / reconciliation

A reliable queue consumer needs a mechanism for admitting that automatic recovery did not succeed.

Measure Operations Separately From Attempts

Retry behavior can also distort observability.

Suppose:

1,000 user operations

produce:

1,700 backend attempts

If the dashboard only shows backend QPS, it may appear that user demand increased by 70%.

It did not.

The system generated the additional 700 requests itself.

Useful metrics separate logical work from transport attempts:

operations_started_total
attempts_total
retries_total
retry_success_total
retry_exhausted_total
retry_budget_exhausted_total
timeout_total
requests_shed_total

That allows a metric such as:

retry amplification factor
=
total attempts / logical operations

For example:

17,000 attempts / 10,000 operations
= 1.7x amplification

During healthy operation, that number might stay around:

1.01x

During an incident it might jump to:

2.8x

Now the apparent traffic spike can be separated into:

real user demand
+
self-generated retry traffic

Google SRE specifically calls out retry-rate visibility as useful during cascading failures because retries can otherwise look like a passive symptom rather than an active contributor.

Observability should therefore answer not only:

How many requests did the service receive?

but also:

How much of that traffic represented new work?

How much was generated because earlier work failed?

That distinction becomes extremely valuable during an outage.

A Practical Retry Policy

For a typical service-to-service request, I want the retry decision to answer five questions.

Is the failure plausibly transient?

if (!isRetryable(error)) {
  throw error;
}

Do not spend more system capacity repeating something that is expected to fail identically.

Is the operation safe to repeat?

That may be true because the operation is naturally idempotent:

PUT desired state
GET resource
DELETE already-deleted resource

or because the system provides a safety mechanism:

idempotency key
deduplication record
transactionally safe retry

Automatic retries around non-idempotent business effects require much more care.

Is there enough time left?

if (Date.now() + delay >= deadline) {
  throw new DeadlineExceededError();
}

A retry whose result can no longer be consumed is wasted work.

Is retry capacity still available?

if (!retryBudget.tryConsume()) {
  throw error;
}

One caller should not have unlimited permission to increase load simply because its request keeps failing.

How should the next attempt be distributed?

Use capped exponential backoff with jitter:

const cap = Math.min(MAX_BACKOFF, BASE_DELAY * 2 ** attempt);

const delay = Math.random() * cap;

The complete decision process looks closer to this:

That is considerably more deliberate than:

try {
  await request();
} catch {
  await request();
}

The important design change is that failure does not automatically imply retry. Failure starts a decision process.

Retry Patterns I Would Avoid

Several patterns immediately make me uncomfortable because they remove one of the controls that keeps retry traffic bounded.

Infinite retry

while (true) {
  try {
    return await dependency.call();
  } catch {}
}

The system has no way to admit that automatic recovery failed.

Retrying every error

catch {
  retry();
}

Permanent failures become repeated load.

Constant immediate retries

fail
retry
fail
retry
fail
retry

The dependency receives no recovery window.

Retrying independently at every layer

controller
↓ retry

service
↓ retry

HTTP client
↓ retry

SDK
↓ retry

The number of attempts multiplies as the request travels down the stack.

Continuing long after the caller has left

client timeout: 2 seconds

backend retry workflow: 30 seconds

The system spends resources on a result nobody can use.

Retrying unsafe side effects without idempotency

POST transfer money
→ timeout
→ blindly repeat

Infrastructure reliability becomes duplicate business state.

None of these problems requires an unusually complicated distributed system. They can emerge from a handful of individually reasonable defaults that were never designed as one end-to-end retry policy.

Retries Need a Failure Budget, Not Optimism

A retry is effectively a bet.

The caller is betting that:

the problem is temporary
+
the next attempt has a reasonable chance of success
+
the extra load will not materially hurt recovery

Sometimes that is an excellent bet. A transient connection reset, short leader election, or isolated 503 may justify another attempt.

But consider a dependency where:

latency is exploding
queues are full
CPU is saturated
throttling is increasing
every replica is unhealthy

Another attempt may have almost no expected value.

At that point, the rational action can become:

fail fast
shed load
preserve capacity
allow recovery

That decision can feel less reliable from the perspective of one caller because the system deliberately stops trying to rescue that request.

At the system level, it may be dramatically more reliable.

This is why retry policy should be designed around a failure budget, not optimism. The question is not whether another attempt could theoretically succeed. The question is whether spending the resources required for that attempt is still a good trade for the system as a whole.

The Invariants I Want

When I evaluate retry behavior, I care more about system guarantees than about a library setting such as:

retries = 3

The invariants I want are closer to:

A single logical operation must have bounded attempts.

A permanent failure must not create repeated traffic.

Retry timing must not synchronize large client populations.

One user request must not multiply exponentially
through service layers.

The caller's deadline must bound downstream work.

Unsafe side effects must not repeat without
idempotency or equivalent protection.

Retry traffic must not consume unlimited capacity.

An overloaded dependency must be allowed to recover.

Retry amplification must be observable.

These are stronger than saying:

requests retry three times

because they describe the behavior of the system rather than the configuration of one client.

A library can correctly implement three retries and still violate nearly every invariant above if another layer is also retrying, the operation is unsafe, the deadlines are disconnected, or thousands of callers synchronize their attempts.

Reliability comes from the behavior of the complete system.

Final Thought

Retries are one of the most useful tools in distributed systems because many failures really are temporary. Networks drop packets, connections reset, processes restart, leaders move, and services briefly throttle. A second attempt can turn all of those into invisible recoveries.

The danger appears when we forget that every attempt consumes real resources.

When a dependency is overloaded, retries can generate additional work precisely when the system needs less. When every layer retries independently, one logical operation can become dozens or hundreds of downstream attempts. When every client uses the same backoff schedule, recovery arrives as a synchronized traffic wave. When timeouts are too aggressive, healthy requests become artificial failures. And when an operation is not idempotent, retrying can preserve transport availability while corrupting business state.

The mental model I find most useful is therefore not:

“Retries make failures go away.”

It is:

“A retry spends additional system capacity for another chance at success.”

Once retrying is treated as spending a resource, the architecture becomes clearer. Classify failures. Make operations safe to repeat. Use realistic deadlines. Give retry ownership to deliberate boundaries. Back off. Add jitter. Bound the number of attempts. Budget retry traffic. Shed load when necessary. Measure retry amplification separately from real demand.

And sometimes accept that the most reliable thing a system can do is stop trying.

The purpose of retrying is not to make one request succeed at any cost.

It is to improve the reliability of the system as a whole.

References

• Amazon Builders' Library — Timeouts, retries, and backoff with jitter. Production guidance on selecting timeouts, limiting retries, avoiding retry multiplication across service layers, and using exponential backoff, jitter, and local retry throttling.

• AWS Architecture Blog — Exponential Backoff and Jitter. Marc Brooker's discussion of why synchronized exponential backoff still creates unnecessary contention and how jitter spreads retry traffic.

• Google SRE — Addressing Cascading Failures. Describes overload as a positive-feedback problem and explains how retries can amplify incidents. It discusses randomized exponential backoff, bounded retries, retry budgets, and retry placement.

• Google SRE — Production Services Best Practices. Discusses shedding traffic when total offered load exceeds service capacity and accounting for retries as part of that load.

• Stripe Engineering — Designing robust and predictable APIs with idempotency. Connects retries to ambiguous network outcomes and explains why idempotency, backoff, and jitter complement one another.

• AWS — Retry behavior in AWS SDKs and Tools. Documents retry classification, exponential backoff, retry quotas, token-bucket behavior, and adaptive request throttling in AWS SDK clients.

• Amazon Builders' Library — Making retries safe with idempotent APIs. Explores why safe retry behavior depends on server-side semantics that prevent repeated attempts from creating duplicate business effects.