A user sends a request and the frontend is willing to wait:
800 ms
The API receives it and calls another service with:
timeout = 2 seconds
That service queries PostgreSQL with:
statement timeout = 5 seconds
and calls another API with:
timeout = 10 seconds
None of those settings looks unreasonable in isolation. Together, they describe a system whose internal work can continue for seconds after the user stopped waiting at 800 milliseconds.
That abandoned work still consumes:
threads
connections
CPU
memory
database capacity
network bandwidth
During normal traffic, the waste may be small enough to ignore. Under overload, it can become part of the mechanism keeping the system overloaded: callers give up, but servers continue spending scarce resources on results nobody can use.
Google's SRE guidance discusses this exact cascading-failure pattern. A backend that keeps executing requests after their callers' deadlines have expired consumes capacity without producing useful work.
So the answer is not simply:
“Put a timeout on every HTTP call.”
The stronger model is:
One user operation has one end-to-end time budget, and every downstream component should understand how much of that budget remains.
That is the difference between independently configured timeouts and a propagated deadline.
Timeouts and Deadlines Answer Different Questions
A timeout usually answers:
How long may this particular operation run?
For example:
You may spend at most 500 ms
on this HTTP call.
A deadline answers a broader question:
At what point does the entire result
stop being useful?
Suppose a request begins at:
10:00:00.000
and the caller is willing to wait:
800 ms
The deadline is:
10:00:00.800
If the API spends 150 milliseconds authenticating, validating input, and waiting for a worker, the next service does not inherit a fresh 800 milliseconds.
It has:
650 ms
left.
If that service spends another:
300 ms
the remaining budget is:
350 ms
Crossing a microservice boundary does not restart the user's clock.
That sounds obvious when written down, but many systems effectively restart the clock on every hop because each service owns its own independent client timeout.
The result is a collection of locally reasonable settings that compose into globally irrational behavior.
Independent Timeouts Can Outlive the User Request
Consider this call chain:
Client → API → Order Service → Inventory → Database
Each layer has its own timeout:
Client: 1 second
API: 3 seconds
Order: 5 seconds
Inventory: 5 seconds
Database: 10 seconds
The client gives up after one second.
Nothing about those downstream timeout values guarantees the rest of the system stops at that point.
Google uses essentially this failure pattern in its SRE material: one service consumes most of its allowed time and then gives the downstream service a completely new timeout instead of passing the remaining budget.
That downstream call may begin work that can no longer possibly finish soon enough for the original caller.
At that point, the work is not merely slow.
It is already useless.
Propagate the Remaining Budget
A better model starts with one deadline:
Client deadline = 800 ms
Every hop asks:
How much time remains?
For example:
T+0 ms
Client → API
Remaining: 800 ms
T+80 ms
API → Order Service
Remaining: 720 ms
T+210 ms
Order → Inventory
Remaining: 590 ms
T+350 ms
Inventory → Database
Remaining: 450 ms
Now every service sees the same temporal reality.
The request is not:
five unrelated RPCs
It is:
one user operation
with one shrinking budget
gRPC makes this model explicit
gRPC defines a deadline as the point beyond which the client is no longer willing to wait for an RPC.
A server can determine that the caller's deadline has expired and stop doing work associated with that request. More importantly, gRPC supports propagating the remaining deadline to downstream RPCs.
Conceptually:
User Service receives:
1.5 seconds remaining
User Service spends:
300 ms
User Service calls Billing:
~1.2 seconds remaining
gRPC's Java and Go implementations support deadline propagation directly. Its propagation behavior is also careful about clock skew: rather than blindly forwarding an absolute wall-clock timestamp between machines, the system can propagate the remaining duration after accounting for time already consumed.
That is the semantic we want regardless of transport.
The work should carry:
remaining budget
not:
a fresh timeout chosen by every hop
The Operation Owns the Deadline
So who owns the deadline?
Not inherently:
the database
or:
the payment service
or even:
the API gateway
The deadline belongs to:
the operation whose result someone is waiting for.
The outermost meaningful caller usually knows when that result stops being valuable.
A browser might say:
I need this response within 2 seconds.
A batch process might say:
I can wait 30 seconds.
A reconciliation job might say:
I can wait several minutes.
The same downstream service can therefore receive very different legitimate deadlines depending on the operation that invoked it.
That is one reason a hardcoded timeout cannot fully represent request intent.
The same dependency call may be acceptable with:
900 ms remaining
and pointless with:
12 ms remaining
even though the dependency itself is perfectly healthy in both cases.
Local Safety Caps Still Matter
Deadline propagation does not mean a downstream dependency should automatically consume every millisecond still available.
Suppose:
request budget remaining = 4 seconds
but the inventory service has decided that one database lookup should never consume more than:
500 ms
Then the effective timeout should be:
effective DB timeout
=
min(
remaining request budget,
local DB safety cap
)
For example:
const dbBudgetMs = Math.min(deadline.remainingMs(), 500);
The inherited deadline answers:
When does this result become useless
to my caller?
The local cap answers:
How long am I willing to spend
on this dependency?
Both are necessary.
A service should respect the caller's shrinking budget without surrendering its own protection against a pathological dependency.
This gives each layer freedom to be stricter.
It should not give each layer permission to silently extend the operation beyond the caller's deadline.
Queue Time Consumes the Same Budget
One of the easiest mistakes is counting only active processing time.
Suppose a request arrives with:
500 ms
remaining.
It spends:
450 ms
waiting for a worker thread.
When execution finally begins, only:
50 ms
remain.
Starting a:
300 ms
database operation at that point makes very little sense.
The service should be able to conclude:
There is not enough time left
to begin this work.
Google's cascading-failure guidance calls out this exact kind of queueing problem: requests can spend so long waiting for resources that their callers have already abandoned them before actual processing begins.
Queueing latency is therefore not separate from request latency.
It spends the same budget.
Rejecting doomed work can increase successful throughput
Suppose a service has:
100 worker threads
and 50 of them are executing requests whose callers have already timed out.
Operationally:
100 threads busy
but from the user's point of view:
50 threads producing no useful result
Stopping those requests gives the service capacity for requests whose results can still be delivered.
This is one of the most useful ways to think about deadlines:
A deadline is not only a latency control. It is also a capacity control.
Failing doomed work early can increase the number of requests that actually succeed.
Cancellation Completes the Deadline Model
A deadline expiring at the caller is only half the solution.
The cancellation needs to reach the code doing the expensive work.
Imagine:
Client
↓
API
↓
Recommendations
↓
Database
The user navigates away.
If only the browser knows:
request cancelled
then the API, recommendation service, and database may all continue working.
A better path looks like:
gRPC's cancellation guidance makes this explicit: when the caller no longer wants the result, ongoing computation should stop where practical, and cancellation should propagate through downstream work triggered by the original RPC.
That is work conservation.
Framework cancellation is not application cancellation
A framework can tell your handler:
request cancelled
but it cannot necessarily stop arbitrary work your application already launched.
Consider:
executor.submit(() -> generateLargeReport());
The request is cancelled.
If generateLargeReport() never checks cancellation state, it may continue running for another 30 seconds.
The transport did its part.
The application did not.
Long-running work needs a cooperative way to learn:
this operation no longer has a useful result
and stop at safe points.
Side Effects Change Cancellation Semantics
Not every operation should be abandoned the moment the caller disappears.
Consider:
calculate recommendations
If the caller no longer cares, stopping is generally fine.
Now consider:
charge credit card
The request has already reached the payment provider.
The client's deadline expires.
Can we assume nothing happened?
No.
The payment may already have succeeded.
Stopping our wait for the result does not undo the external side effect.
This creates the familiar ambiguous outcome:
client observed: timeout
server observed: success
gRPC explicitly notes this possibility: the server may complete successfully while the response reaches the caller only after its deadline, so the caller observes DEADLINE_EXCEEDED.
That gives us a critical rule:
Deadline exceeded does not necessarily mean the operation did not happen.
A useful distinction is:
| Work | Deadline expires | Typical behavior |
|---|---|---|
| Pure computation | Stop | Result is no longer useful |
| Read-only query | Cancel when practical | Preserve resources |
| External or irreversible side effect | Outcome may be ambiguous | Reconcile or retry idempotently |
A timeout around:
GET recommendations
and one around:
POST charge-card
cannot automatically share the same recovery policy.
Cancellation semantics depend on what the work can change.
Retries Must Fit Inside the Original Budget
Suppose the caller gives the operation:
1 second
The first downstream attempt consumes:
800 ms
and times out.
Should the application launch another request with a fresh:
800 ms
timeout?
No.
Only:
200 ms
remain in the original operation.
If the dependency cannot realistically succeed inside those 200 milliseconds, retrying is merely additional load.
Conceptually:
async function callWithRetry(deadline: Deadline) {
for (let attempt = 1; attempt <= 3; attempt++) {
const remaining = deadline.remainingMs();
if (remaining < MIN_USEFUL_ATTEMPT_MS) {
throw new DeadlineExceededError();
}
const timeout = Math.min(remaining, PER_ATTEMPT_CAP_MS);
try {
return await callDependency({
timeout,
});
} catch (error) {
if (!isRetryable(error)) {
throw error;
}
}
}
}
The retry policy operates inside the deadline.
It does not create a new one.
Retries spend both time and capacity
Suppose:
original request budget = 900 ms
Attempt 1 uses:
300 ms
Backoff uses:
100 ms
Attempt 2 uses:
300 ms
Now only:
200 ms
remain.
The retry-count policy might say:
one attempt left
while the time budget says:
do not bother
Both signals matter.
AWS reliability guidance warns about this general tension: overly long timeouts retain resources during failures, while overly aggressive timeouts can trigger additional retries and make an overloaded backend even busier.
Retries therefore need at least two budgets:
attempt budget
+
time budget
Backoff is part of user latency too
A policy such as:
attempt 1
↓
wait 100 ms
↓
attempt 2
↓
wait 200 ms
↓
attempt 3
does not get free time during the waits.
Backoff consumes the same deadline.
Before scheduling the next retry, the system should ask whether enough time remains for:
backoff
+
network
+
actual processing
Exponential backoff and jitter are valuable because they reduce pressure and prevent synchronized retry spikes.
But the best retry schedule in the world is useless if the successful result would arrive after the caller's deadline.
Choose Timeouts From Real Latency
Where should a value such as:
500 ms
come from?
Ideally from some combination of:
observed latency distribution
business latency objective
acceptable false-timeout rate
network conditions
dependency behavior
Suppose a dependency currently has:
p50 = 40 ms
p95 = 90 ms
p99 = 180 ms
p99.9 = 420 ms
A timeout of:
50 ms
will fail many requests that would otherwise have succeeded normally.
A timeout of:
30 seconds
may retain resources far too long during a real dependency failure.
The value should reflect the actual latency envelope and the amount of failure the caller is willing to tolerate.
gRPC similarly recommends selecting deadlines from real application characteristics and validating them through load testing rather than relying on arbitrary constants.
Network setup is part of the time too
Remote work can include:
DNS
TCP connection
TLS handshake
connection-pool wait
proxy/load balancer
network transit
server queue
application processing
response transit
Different libraries include different parts of that path in different timeout settings.
That is why connection timeouts and request timeouts need to be understood explicitly.
Consider:
warm connection:
request = 40 ms
versus:
cold connection:
DNS + TCP + TLS + request
= 130 ms
A timeout tuned only to the warm case can suddenly create false failures after a deployment, connection churn, certificate change, or network event.
Timeout design has to include the path the request actually takes.
Fan-Out Turns Deadlines Into Priorities
Suppose an API calls four services in parallel:
Profile
Inventory
Recommendations
Pricing
Each dependency might individually support:
500 ms
requests.
But if the entire API operation has only:
500 ms
remaining, those services do not each own a fresh 500-millisecond interval.
They all share the same deadline.
This becomes especially useful when not every dependency is equally important.
For example:
Inventory:
required
Pricing:
required
Recommendations:
optional
With very little time left, the application may choose:
return response without recommendations
instead of:
miss the entire deadline
Partial results can beat complete but late results
Imagine a dashboard assembled from five services.
Four finish within:
200 ms
One is struggling.
The total deadline is:
700 ms
At 650 milliseconds, the API can either wait and risk failing the complete response or return:
4 / 5 widgets
if product semantics allow graceful degradation.
That is not always acceptable.
Where it is acceptable, deadline awareness lets the system express a priority that a simple timeout configuration cannot:
this optional component is no longer worth
risking the entire user response for
Deadlines force the architecture to answer which pieces actually matter.
Queues Need Semantic Deadlines
Synchronous RPCs make deadline propagation relatively intuitive.
Queues complicate the model.
Suppose:
HTTP request
→ enqueue job
→ worker processes job
If the HTTP caller waits synchronously for the result, the original request deadline may still apply.
But suppose the service instead returns:
202 Accepted
and promises that the job will complete later.
Now the HTTP request and the durable job are different operations.
The business job might have a new contract:
Complete within 10 minutes.
Do not blindly copy a short HTTP deadline into durable asynchronous work whose purpose is specifically to outlive the original request.
A deadline belongs to a semantic operation.
When the operation changes, its deadline can change too.
Do not turn expired queued work into fresh work
For synchronous queue-backed execution, the message can carry the original deadline:
{
"requestId": "req_92",
"deadline": "2026-10-08T22:00:00.800Z",
"payload": {}
}
The worker can check it before starting expensive work:
if (Date.now() >= message.deadline) {
metrics.expiredBeforeProcessing.inc();
return;
}
That prevents this waste:
wait 2 seconds in queue
+
perform 1 second of work
+
discover caller stopped waiting
2.5 seconds ago
A queue should not make expired synchronous work fresh again.
Deadline Propagation Is Also Backpressure
Consider an overloaded service.
As load increases:
queue time ↑
which means:
remaining budget at execution ↓
Eventually, some requests reach the worker with too little time remaining to justify beginning expensive processing.
Those requests can be rejected earlier.
overload
→ queue delay
→ remaining budget shrinks
→ more doomed work rejected
→ wasted execution falls
That acts as a form of backpressure.
It does not replace explicit load shedding, rate limiting, or concurrency controls.
It complements them by preventing requests that are already temporally impossible from consuming their full execution cost.
This is why deadline propagation belongs in the same reliability conversation as queueing and overload.
Time remaining is itself a capacity signal.
A Deadline Is Not an SLO
Suppose the service-level objective is:
99% of checkout requests
complete within 800 ms
That does not necessarily mean:
every checkout request
must have a hard 800 ms deadline
An SLO describes:
desired behavior across
a population of requests
A deadline describes:
how long this specific operation
remains useful
The two should influence each other.
They are not identical.
A system may intentionally allow the slowest one percent of requests to exceed the SLO target while still having a somewhat larger hard deadline.
Likewise, a user or internal caller may have a much shorter patience window than the global service SLO.
Keeping those concepts separate prevents operational objectives from being mistaken for per-request cancellation policy.
Build the Budget From the Outside In
Suppose the product requirement is:
Checkout should normally finish
within 1 second.
A rough budget might be:
End-to-end budget 1000 ms
Gateway/network 80 ms
Application overhead 120 ms
Safety margin 100 ms
----------------------------------------
Dependency work budget 700 ms
Maybe the dependency path approximately needs:
Inventory ~150 ms
Payment ~350 ms
Database ~100 ms
with some operations executing in parallel.
These values will not be perfect.
That is not the point.
The useful property is that they were derived from one end-to-end requirement.
The weaker approach is letting every team independently choose:
Inventory timeout = 2 s
Payment timeout = 5 s
Database timeout = 10 s
and then expecting the composition to meet a one-second checkout objective.
It cannot.
A distributed latency budget has to compose from the outside in.
Make Remaining Time Explicit in Code
Even without gRPC, application code can carry deadline context explicitly.
For example:
class Deadline {
constructor(private readonly expiresAt: number) {}
remainingMs(): number {
return Math.max(0, this.expiresAt - Date.now());
}
expired(): boolean {
return this.remainingMs() === 0;
}
timeoutFor(localCapMs: number): number {
return Math.min(this.remainingMs(), localCapMs);
}
}
At the edge:
const deadline = new Deadline(Date.now() + 800);
Then downstream code can derive its local timeout from the remaining budget:
const timeout = deadline.timeoutFor(300);
if (timeout < 50) {
throw new DeadlineExceededError();
}
await inventoryClient.getItem(productId, { timeout });
The transport-specific implementation can change.
The invariant remains:
remaining budget
travels with the work
Java and gRPC can carry it through RPC context
In gRPC Java, a client can apply a deadline:
InventoryGrpc.InventoryBlockingStub stub =
inventoryStub.withDeadlineAfter(
remainingMillis,
TimeUnit.MILLISECONDS
);
When a server receives a propagated deadline, downstream calls can inherit the shrinking budget instead of inventing a new one.
For Spring applications using ordinary HTTP rather than gRPC, the same semantics can be implemented through request context or propagated metadata.
The transport is secondary.
The important part is preserving one operation clock.
Do Not Trust External Deadlines Blindly
Suppose an external caller asks for:
deadline = 6 hours
Should an API allow that request to reserve resources for six hours?
Probably not.
The service boundary still needs its own policy.
A reasonable model is:
effective deadline
=
min(
caller deadline,
endpoint maximum
)
If:
client asks for 30 seconds
endpoint maximum = 2 seconds
then:
effective budget = 2 seconds
The caller communicates when the result becomes useless from its perspective.
The service still protects itself.
The inverse case matters too. An absurdly tiny external deadline may be below the minimum useful execution time for an endpoint.
Deadline propagation is not the same thing as allowing arbitrary callers to dictate internal resource policy.
Different endpoints need different contracts
Compare:
GET /users/123
with:
POST /reports/yearly-export
A single global:
timeout = 5 seconds
may be too generous for the first and fundamentally wrong for the second.
The profile lookup might reasonably need:
300 ms
The yearly report may need to become asynchronous:
submit job
→ return job ID
→ poll or notify on completion
instead of holding one HTTP request open for minutes.
Deadline design often reveals API-design problems.
An enormous timeout is not automatically resilience.
Sometimes it is evidence that a synchronous interaction has outgrown the request/response model.
Observe Where the Budget Went
Traditional latency telemetry tells us:
request took 780 ms
Deadline-aware observability can tell us how the request arrived at that point.
A trace might record:
API ingress:
remaining = 790 ms
Order service:
remaining = 630 ms
Payment:
remaining = 410 ms
Payment retry:
remaining = 130 ms
Now it becomes obvious why another large retry would be pointless.
Useful metrics include:
deadline_exceeded_total
cancelled_total
expired_before_processing_total
remaining_budget_at_dependency_call
work_completed_after_caller_cancelled
retry_skipped_due_to_deadline
That last category is particularly interesting:
work completed after caller cancelled
directly measures wasted capacity.
Trace the time budget itself
A distributed trace could show:
Checkout [800ms deadline]
│
├── Auth 45ms
│
├── Inventory 120ms
│
└── Payment
├── Queue 180ms
├── Attempt 1 260ms
├── Backoff 100ms
└── Attempt 2 started
with 70ms remaining
The problem is no longer vaguely:
payment was slow
The trace exposes the real design problem:
the second attempt began
without enough budget to be useful
That is much more actionable.
Observability should show where the time budget was spent in the same way a financial ledger shows where money went.
Deadline errors need context
Suppose:
deadline errors ↑
That can mean:
backend got slower
caller shortened its deadline
network latency increased
queueing increased
retry policy consumed more budget
A deadline failure is not automatically evidence that the backend itself is broken.
The contract may have become impossible.
If a caller supplies:
deadline = 50 ms
while the service normally requires:
100 ms
every request can fail even though the implementation has not degraded at all.
Temporal context matters on both sides of the API.
Timeouts Create Ambiguous Side Effects
Suppose:
client deadline = 500 ms
At:
490 ms
the payment provider completes the charge.
The response would reach the caller at:
520 ms
but the caller stopped waiting at:
500 ms
Observed result:
client: timeout
server: success
The client may now retry.
Without idempotency:
charge 1
charge 2
The deadline did not create the underlying distributed ambiguity.
It exposed it.
AWS's guidance on idempotent APIs emphasizes exactly this problem: retries are much safer when multiple attempts can be recognized as the same logical operation instead of generating repeated side effects.
For a payment call, that might mean a stable operation identifier:
payment_request_42
survives:
original attempt
timeout
retry
status reconciliation
Deadlines and idempotency therefore belong in the same design discussion.
One limits how long the caller waits.
The other makes uncertainty after that wait safe to recover from.
The Request Can Be Doomed While Every Service Is Healthy
This is one of the stranger properties of deadline-aware systems.
Suppose:
API healthy
Order healthy
Inventory healthy
Payment healthy
Database healthy
The request has:
12 ms
remaining.
Payment normally takes:
100 ms
Every component is healthy.
The operation is already doomed.
No resource graph needs to be red.
No service needs to be unavailable.
The problem is that the request's remaining lifetime no longer matches the work required to complete it.
This is why health alone is not enough.
Distributed systems also need temporal context.
Shared deadlines prevent locally rational, globally absurd decisions
Imagine each service team independently decides:
Inventory timeout = 2 seconds
Payment timeout = 4 seconds
Database timeout = 10 seconds
Every choice may make perfect sense in isolation.
But distributed systems compose local policies.
If the user-facing operation is useful for only:
800 ms
the combined behavior is nonsensical.
Deadline propagation replaces:
independent local clocks
with:
one shared operation clock
Each service can still impose stricter local limits.
None silently grants the request more life than the operation actually has.
The Invariants I Want
For a synchronous microservice request path, I care about guarantees such as:
Every externally initiated request has a finite
end-to-end deadline.
Downstream calls never receive more time than the
original operation has remaining.
Local dependency timeouts may be stricter than the
remaining deadline, but may not silently extend it.
Queue time consumes the same budget as execution time.
Work that cannot realistically finish inside the
remaining budget should usually be rejected before
consuming expensive resources.
Caller cancellation propagates to cancellable
downstream work where practical.
Application code that starts long-running work must
cooperate with cancellation rather than assuming
the framework can stop it automatically.
Cancellation of irreversible side effects must not
be mistaken for rollback.
Retries and backoff must fit inside the original
deadline rather than creating a fresh time budget.
Timeout values should be grounded in observed latency
distributions and the actual network path.
Asynchronous jobs should receive deadlines based on
their own semantic lifetime rather than blindly
inheriting an HTTP request deadline.
Ambiguous timeout outcomes for side-effecting work
must be handled with idempotency or reconciliation.
Observability should show where the time budget was
spent and how much work continued after the caller
stopped waiting.
Those guarantees are much stronger than:
every HTTP client has timeout=5000
because a timeout configuration describes one call.
The invariants describe the lifetime of the operation.
Final Thought
Microservices make time deceptively local.
Each service sees something like:
one incoming request
one outgoing RPC
one database query
so it is natural for every team to assign its own timeout.
The caller does not experience those calls independently.
The caller experiences:
one operation
with one point beyond which the result stops being useful.
That budget should shrink as the operation moves through the system.
Queueing spends it.
Network setup spends it.
RPCs spend it.
Database work spends it.
Retries spend it.
Backoff spends it.
And once the budget reaches zero, continuing to work is often not resilience.
It is waste.
That waste is especially dangerous under overload because it steals capacity from requests that could still succeed.
So the most useful question is not:
“What timeout should this service use?”
It is:
“How much of the caller's original time budget is left when this service begins work?”
Then comes the second question:
“If the caller no longer cares about the result, why are we still computing it?”
Those two questions connect latency design to capacity design.
They also connect deadlines to several other reliability concerns:
timeouts
→ retries
→ idempotency
→ queueing
→ overload
→ backpressure
→ cancellation
→ observability
A deadline looks like a clock.
In a distributed system, it is more useful to think of it as a contract describing how long the system is allowed to keep spending resources on one operation's intent.
Every service in the chain should know when that contract expires.
References
-
Google SRE — Addressing Cascading Failures. Explains how requests that continue executing after their callers have exceeded their deadlines waste backend resources and can contribute to overload, and discusses propagating time budgets through RPC chains.
-
gRPC — Deadlines. Defines deadline semantics, recommends explicit client deadlines, explains expiration behavior, and documents deadline propagation using the remaining timeout rather than simply forwarding an absolute wall-clock value.
-
gRPC — Cancellation. Describes cancellation propagation through ongoing computation and downstream RPCs while emphasizing that application code remains responsible for stopping work it started.
-
gRPC — gRPC and Deadlines. Discusses deadline selection, server-side cancellation handling, and the important ambiguity that a server may complete successfully even when the caller observes
DEADLINE_EXCEEDED. -
AWS Well-Architected — Set client timeouts. Recommends explicit connection and request timeouts, warns about both excessively long and overly aggressive values, and cautions against relying blindly on client-library defaults.
-
AWS — Exponential Backoff and Jitter. Describes exponential backoff and jitter as mechanisms for reducing retry pressure and preventing synchronized retry bursts against recovering dependencies.
-
Amazon Builders' Library — Making retries safe with idempotent APIs. Explains why distributed operations that may be retried need stable request identity when duplicate side effects would be unsafe.
