← Blog

Microservices · Deadlines · Reliability

Who Owns the Deadline in a Chain of Microservice Calls?

Independent timeouts across microservices can outlive the user request they were meant to serve. This article explains end-to-end deadlines, remaining-budget propagation, cancellation, retries, queue time, local safety caps, gRPC deadline semantics, partial results, idempotency, and how deadline-aware systems avoid wasting capacity on work whose result is already useless.

Aashaya Aryal

October 8, 2026 · 21 min read

Analog stopwatch photographed against a black background.

A user sends a request and the frontend is willing to wait:

TEXT
800 ms

The API receives it and calls another service with:

TEXT
timeout = 2 seconds

That service queries PostgreSQL with:

TEXT
statement timeout = 5 seconds

and calls another API with:

TEXT
timeout = 10 seconds

None of those settings looks unreasonable in isolation. Together, they describe a system whose internal work can continue for seconds after the user stopped waiting at 800 milliseconds.

Diagram

Client

800 ms

API

2 s timeout

Order Service

5 s timeout

Payment Service

10 s timeout

Database

That abandoned work still consumes:

TEXT
threads
connections
CPU
memory
database capacity
network bandwidth

During normal traffic, the waste may be small enough to ignore. Under overload, it can become part of the mechanism keeping the system overloaded: callers give up, but servers continue spending scarce resources on results nobody can use.

Google's SRE guidance discusses this exact cascading-failure pattern. A backend that keeps executing requests after their callers' deadlines have expired consumes capacity without producing useful work.

So the answer is not simply:

“Put a timeout on every HTTP call.”

The stronger model is:

One user operation has one end-to-end time budget, and every downstream component should understand how much of that budget remains.

That is the difference between independently configured timeouts and a propagated deadline.

Timeouts and Deadlines Answer Different Questions

A timeout usually answers:

TEXT
How long may this particular operation run?

For example:

TEXT
You may spend at most 500 ms
on this HTTP call.

A deadline answers a broader question:

TEXT
At what point does the entire result
stop being useful?

Suppose a request begins at:

TEXT
10:00:00.000

and the caller is willing to wait:

TEXT
800 ms

The deadline is:

TEXT
10:00:00.800

If the API spends 150 milliseconds authenticating, validating input, and waiting for a worker, the next service does not inherit a fresh 800 milliseconds.

It has:

TEXT
650 ms

left.

If that service spends another:

TEXT
300 ms

the remaining budget is:

TEXT
350 ms

Crossing a microservice boundary does not restart the user's clock.

That sounds obvious when written down, but many systems effectively restart the clock on every hop because each service owns its own independent client timeout.

The result is a collection of locally reasonable settings that compose into globally irrational behavior.

Independent Timeouts Can Outlive the User Request

Consider this call chain:

TEXT
Client → API → Order Service → Inventory → Database

Each layer has its own timeout:

TEXT
Client:       1 second
API:          3 seconds
Order:        5 seconds
Inventory:    5 seconds
Database:    10 seconds

The client gives up after one second.

Nothing about those downstream timeout values guarantees the rest of the system stops at that point.

Diagram
DatabaseInventoryOrder ServiceAPIClientDatabaseInventoryOrder ServiceAPIClient1 second passesDownstream work may continueRequesttimeout=3stimeout=5stimeout=10sGive up

Google uses essentially this failure pattern in its SRE material: one service consumes most of its allowed time and then gives the downstream service a completely new timeout instead of passing the remaining budget.

That downstream call may begin work that can no longer possibly finish soon enough for the original caller.

At that point, the work is not merely slow.

It is already useless.

Propagate the Remaining Budget

A better model starts with one deadline:

TEXT
Client deadline = 800 ms

Every hop asks:

TEXT
How much time remains?

For example:

TEXT
T+0 ms
Client → API
Remaining: 800 ms

T+80 ms
API → Order Service
Remaining: 720 ms

T+210 ms
Order → Inventory
Remaining: 590 ms

T+350 ms
Inventory → Database
Remaining: 450 ms
Diagram
DatabaseInventoryOrder ServiceAPIClientDatabaseInventoryOrder ServiceAPIClient80ms consumed130ms consumed140ms consumeddeadline in 800ms~720ms remaining~590ms remaining~450ms remaining

Now every service sees the same temporal reality.

The request is not:

TEXT
five unrelated RPCs

It is:

TEXT
one user operation
with one shrinking budget

gRPC makes this model explicit

gRPC defines a deadline as the point beyond which the client is no longer willing to wait for an RPC.

A server can determine that the caller's deadline has expired and stop doing work associated with that request. More importantly, gRPC supports propagating the remaining deadline to downstream RPCs.

Conceptually:

TEXT
User Service receives:
1.5 seconds remaining

User Service spends:
300 ms

User Service calls Billing:
~1.2 seconds remaining

gRPC's Java and Go implementations support deadline propagation directly. Its propagation behavior is also careful about clock skew: rather than blindly forwarding an absolute wall-clock timestamp between machines, the system can propagate the remaining duration after accounting for time already consumed.

That is the semantic we want regardless of transport.

The work should carry:

TEXT
remaining budget

not:

TEXT
a fresh timeout chosen by every hop

The Operation Owns the Deadline

So who owns the deadline?

Not inherently:

TEXT
the database

or:

TEXT
the payment service

or even:

TEXT
the API gateway

The deadline belongs to:

the operation whose result someone is waiting for.

The outermost meaningful caller usually knows when that result stops being valuable.

A browser might say:

TEXT
I need this response within 2 seconds.

A batch process might say:

TEXT
I can wait 30 seconds.

A reconciliation job might say:

TEXT
I can wait several minutes.

The same downstream service can therefore receive very different legitimate deadlines depending on the operation that invoked it.

That is one reason a hardcoded timeout cannot fully represent request intent.

The same dependency call may be acceptable with:

TEXT
900 ms remaining

and pointless with:

TEXT
12 ms remaining

even though the dependency itself is perfectly healthy in both cases.

Local Safety Caps Still Matter

Deadline propagation does not mean a downstream dependency should automatically consume every millisecond still available.

Suppose:

TEXT
request budget remaining = 4 seconds

but the inventory service has decided that one database lookup should never consume more than:

TEXT
500 ms

Then the effective timeout should be:

TEXT
effective DB timeout
=
min(
    remaining request budget,
    local DB safety cap
)

For example:

TS
const dbBudgetMs = Math.min(deadline.remainingMs(), 500);

The inherited deadline answers:

TEXT
When does this result become useless
to my caller?

The local cap answers:

TEXT
How long am I willing to spend
on this dependency?

Both are necessary.

A service should respect the caller's shrinking budget without surrendering its own protection against a pathological dependency.

This gives each layer freedom to be stricter.

It should not give each layer permission to silently extend the operation beyond the caller's deadline.

Queue Time Consumes the Same Budget

One of the easiest mistakes is counting only active processing time.

Suppose a request arrives with:

TEXT
500 ms

remaining.

It spends:

TEXT
450 ms

waiting for a worker thread.

When execution finally begins, only:

TEXT
50 ms

remain.

Starting a:

TEXT
300 ms

database operation at that point makes very little sense.

The service should be able to conclude:

TEXT
There is not enough time left
to begin this work.

Google's cascading-failure guidance calls out this exact kind of queueing problem: requests can spend so long waiting for resources that their callers have already abandoned them before actual processing begins.

Queueing latency is therefore not separate from request latency.

It spends the same budget.

Rejecting doomed work can increase successful throughput

Suppose a service has:

TEXT
100 worker threads

and 50 of them are executing requests whose callers have already timed out.

Operationally:

TEXT
100 threads busy

but from the user's point of view:

TEXT
50 threads producing no useful result

Stopping those requests gives the service capacity for requests whose results can still be delivered.

This is one of the most useful ways to think about deadlines:

A deadline is not only a latency control. It is also a capacity control.

Failing doomed work early can increase the number of requests that actually succeed.

Cancellation Completes the Deadline Model

A deadline expiring at the caller is only half the solution.

The cancellation needs to reach the code doing the expensive work.

Imagine:

TEXT
Client
↓
API
↓
Recommendations
↓
Database

The user navigates away.

If only the browser knows:

TEXT
request cancelled

then the API, recommendation service, and database may all continue working.

A better path looks like:

Diagram
DatabaseRecommendationsAPIClientDatabaseRecommendationsAPIClientAbandoned work stopsRequestSubrequestQueryCancelCancelCancel if supported

gRPC's cancellation guidance makes this explicit: when the caller no longer wants the result, ongoing computation should stop where practical, and cancellation should propagate through downstream work triggered by the original RPC.

That is work conservation.

Framework cancellation is not application cancellation

A framework can tell your handler:

TEXT
request cancelled

but it cannot necessarily stop arbitrary work your application already launched.

Consider:

JAVA
executor.submit(() -> generateLargeReport());

The request is cancelled.

If generateLargeReport() never checks cancellation state, it may continue running for another 30 seconds.

The transport did its part.

The application did not.

Long-running work needs a cooperative way to learn:

TEXT
this operation no longer has a useful result

and stop at safe points.

Side Effects Change Cancellation Semantics

Not every operation should be abandoned the moment the caller disappears.

Consider:

TEXT
calculate recommendations

If the caller no longer cares, stopping is generally fine.

Now consider:

TEXT
charge credit card

The request has already reached the payment provider.

The client's deadline expires.

Can we assume nothing happened?

No.

The payment may already have succeeded.

Stopping our wait for the result does not undo the external side effect.

This creates the familiar ambiguous outcome:

TEXT
client observed: timeout
server observed: success

gRPC explicitly notes this possibility: the server may complete successfully while the response reaches the caller only after its deadline, so the caller observes DEADLINE_EXCEEDED.

That gives us a critical rule:

Deadline exceeded does not necessarily mean the operation did not happen.

A useful distinction is:

WorkDeadline expiresTypical behavior
Pure computationStopResult is no longer useful
Read-only queryCancel when practicalPreserve resources
External or irreversible side effectOutcome may be ambiguousReconcile or retry idempotently

A timeout around:

TEXT
GET recommendations

and one around:

TEXT
POST charge-card

cannot automatically share the same recovery policy.

Cancellation semantics depend on what the work can change.

Retries Must Fit Inside the Original Budget

Suppose the caller gives the operation:

TEXT
1 second

The first downstream attempt consumes:

TEXT
800 ms

and times out.

Should the application launch another request with a fresh:

TEXT
800 ms

timeout?

No.

Only:

TEXT
200 ms

remain in the original operation.

If the dependency cannot realistically succeed inside those 200 milliseconds, retrying is merely additional load.

Conceptually:

TS
async function callWithRetry(deadline: Deadline) {
  for (let attempt = 1; attempt <= 3; attempt++) {
    const remaining = deadline.remainingMs();

    if (remaining < MIN_USEFUL_ATTEMPT_MS) {
      throw new DeadlineExceededError();
    }

    const timeout = Math.min(remaining, PER_ATTEMPT_CAP_MS);

    try {
      return await callDependency({
        timeout,
      });
    } catch (error) {
      if (!isRetryable(error)) {
        throw error;
      }
    }
  }
}

The retry policy operates inside the deadline.

It does not create a new one.

Retries spend both time and capacity

Suppose:

TEXT
original request budget = 900 ms

Attempt 1 uses:

TEXT
300 ms

Backoff uses:

TEXT
100 ms

Attempt 2 uses:

TEXT
300 ms

Now only:

TEXT
200 ms

remain.

The retry-count policy might say:

TEXT
one attempt left

while the time budget says:

TEXT
do not bother

Both signals matter.

AWS reliability guidance warns about this general tension: overly long timeouts retain resources during failures, while overly aggressive timeouts can trigger additional retries and make an overloaded backend even busier.

Retries therefore need at least two budgets:

TEXT
attempt budget
+
time budget

Backoff is part of user latency too

A policy such as:

TEXT
attempt 1
↓
wait 100 ms
↓
attempt 2
↓
wait 200 ms
↓
attempt 3

does not get free time during the waits.

Backoff consumes the same deadline.

Before scheduling the next retry, the system should ask whether enough time remains for:

TEXT
backoff
+
network
+
actual processing

Exponential backoff and jitter are valuable because they reduce pressure and prevent synchronized retry spikes.

But the best retry schedule in the world is useless if the successful result would arrive after the caller's deadline.

Choose Timeouts From Real Latency

Where should a value such as:

TEXT
500 ms

come from?

Ideally from some combination of:

TEXT
observed latency distribution
business latency objective
acceptable false-timeout rate
network conditions
dependency behavior

Suppose a dependency currently has:

TEXT
p50    = 40 ms
p95    = 90 ms
p99    = 180 ms
p99.9  = 420 ms

A timeout of:

TEXT
50 ms

will fail many requests that would otherwise have succeeded normally.

A timeout of:

TEXT
30 seconds

may retain resources far too long during a real dependency failure.

The value should reflect the actual latency envelope and the amount of failure the caller is willing to tolerate.

gRPC similarly recommends selecting deadlines from real application characteristics and validating them through load testing rather than relying on arbitrary constants.

Network setup is part of the time too

Remote work can include:

TEXT
DNS
TCP connection
TLS handshake
connection-pool wait
proxy/load balancer
network transit
server queue
application processing
response transit

Different libraries include different parts of that path in different timeout settings.

That is why connection timeouts and request timeouts need to be understood explicitly.

Consider:

TEXT
warm connection:
request = 40 ms

versus:

TEXT
cold connection:
DNS + TCP + TLS + request
= 130 ms

A timeout tuned only to the warm case can suddenly create false failures after a deployment, connection churn, certificate change, or network event.

Timeout design has to include the path the request actually takes.

Fan-Out Turns Deadlines Into Priorities

Suppose an API calls four services in parallel:

TEXT
Profile
Inventory
Recommendations
Pricing
Diagram

API

Profile

Inventory

Recommendations

Pricing

Each dependency might individually support:

TEXT
500 ms

requests.

But if the entire API operation has only:

TEXT
500 ms

remaining, those services do not each own a fresh 500-millisecond interval.

They all share the same deadline.

This becomes especially useful when not every dependency is equally important.

For example:

TEXT
Inventory:
required

Pricing:
required

Recommendations:
optional

With very little time left, the application may choose:

TEXT
return response without recommendations

instead of:

TEXT
miss the entire deadline

Partial results can beat complete but late results

Imagine a dashboard assembled from five services.

Four finish within:

TEXT
200 ms

One is struggling.

The total deadline is:

TEXT
700 ms

At 650 milliseconds, the API can either wait and risk failing the complete response or return:

TEXT
4 / 5 widgets

if product semantics allow graceful degradation.

That is not always acceptable.

Where it is acceptable, deadline awareness lets the system express a priority that a simple timeout configuration cannot:

TEXT
this optional component is no longer worth
risking the entire user response for

Deadlines force the architecture to answer which pieces actually matter.

Queues Need Semantic Deadlines

Synchronous RPCs make deadline propagation relatively intuitive.

Queues complicate the model.

Suppose:

TEXT
HTTP request
→ enqueue job
→ worker processes job

If the HTTP caller waits synchronously for the result, the original request deadline may still apply.

But suppose the service instead returns:

TEXT
202 Accepted

and promises that the job will complete later.

Now the HTTP request and the durable job are different operations.

The business job might have a new contract:

TEXT
Complete within 10 minutes.

Do not blindly copy a short HTTP deadline into durable asynchronous work whose purpose is specifically to outlive the original request.

A deadline belongs to a semantic operation.

When the operation changes, its deadline can change too.

Do not turn expired queued work into fresh work

For synchronous queue-backed execution, the message can carry the original deadline:

JSON
{
  "requestId": "req_92",
  "deadline": "2026-10-08T22:00:00.800Z",
  "payload": {}
}

The worker can check it before starting expensive work:

TS
if (Date.now() >= message.deadline) {
  metrics.expiredBeforeProcessing.inc();

  return;
}

That prevents this waste:

TEXT
wait 2 seconds in queue
+
perform 1 second of work
+
discover caller stopped waiting
2.5 seconds ago

A queue should not make expired synchronous work fresh again.

Deadline Propagation Is Also Backpressure

Consider an overloaded service.

As load increases:

TEXT
queue time ↑

which means:

TEXT
remaining budget at execution ↓

Eventually, some requests reach the worker with too little time remaining to justify beginning expensive processing.

Those requests can be rejected earlier.

TEXT
overload
→ queue delay
→ remaining budget shrinks
→ more doomed work rejected
→ wasted execution falls

That acts as a form of backpressure.

It does not replace explicit load shedding, rate limiting, or concurrency controls.

It complements them by preventing requests that are already temporally impossible from consuming their full execution cost.

This is why deadline propagation belongs in the same reliability conversation as queueing and overload.

Time remaining is itself a capacity signal.

A Deadline Is Not an SLO

Suppose the service-level objective is:

TEXT
99% of checkout requests
complete within 800 ms

That does not necessarily mean:

TEXT
every checkout request
must have a hard 800 ms deadline

An SLO describes:

TEXT
desired behavior across
a population of requests

A deadline describes:

TEXT
how long this specific operation
remains useful

The two should influence each other.

They are not identical.

A system may intentionally allow the slowest one percent of requests to exceed the SLO target while still having a somewhat larger hard deadline.

Likewise, a user or internal caller may have a much shorter patience window than the global service SLO.

Keeping those concepts separate prevents operational objectives from being mistaken for per-request cancellation policy.

Build the Budget From the Outside In

Suppose the product requirement is:

TEXT
Checkout should normally finish
within 1 second.

A rough budget might be:

TEXT
End-to-end budget                1000 ms
Gateway/network                    80 ms
Application overhead              120 ms
Safety margin                     100 ms
----------------------------------------
Dependency work budget            700 ms

Maybe the dependency path approximately needs:

TEXT
Inventory   ~150 ms
Payment     ~350 ms
Database    ~100 ms

with some operations executing in parallel.

These values will not be perfect.

That is not the point.

The useful property is that they were derived from one end-to-end requirement.

The weaker approach is letting every team independently choose:

TEXT
Inventory timeout = 2 s
Payment timeout   = 5 s
Database timeout  = 10 s

and then expecting the composition to meet a one-second checkout objective.

It cannot.

A distributed latency budget has to compose from the outside in.

Make Remaining Time Explicit in Code

Even without gRPC, application code can carry deadline context explicitly.

For example:

TS
class Deadline {
  constructor(private readonly expiresAt: number) {}

  remainingMs(): number {
    return Math.max(0, this.expiresAt - Date.now());
  }

  expired(): boolean {
    return this.remainingMs() === 0;
  }

  timeoutFor(localCapMs: number): number {
    return Math.min(this.remainingMs(), localCapMs);
  }
}

At the edge:

TS
const deadline = new Deadline(Date.now() + 800);

Then downstream code can derive its local timeout from the remaining budget:

TS
const timeout = deadline.timeoutFor(300);

if (timeout < 50) {
  throw new DeadlineExceededError();
}

await inventoryClient.getItem(productId, { timeout });

The transport-specific implementation can change.

The invariant remains:

TEXT
remaining budget
travels with the work

Java and gRPC can carry it through RPC context

In gRPC Java, a client can apply a deadline:

JAVA
InventoryGrpc.InventoryBlockingStub stub =
    inventoryStub.withDeadlineAfter(
        remainingMillis,
        TimeUnit.MILLISECONDS
    );

When a server receives a propagated deadline, downstream calls can inherit the shrinking budget instead of inventing a new one.

For Spring applications using ordinary HTTP rather than gRPC, the same semantics can be implemented through request context or propagated metadata.

The transport is secondary.

The important part is preserving one operation clock.

Do Not Trust External Deadlines Blindly

Suppose an external caller asks for:

TEXT
deadline = 6 hours

Should an API allow that request to reserve resources for six hours?

Probably not.

The service boundary still needs its own policy.

A reasonable model is:

TEXT
effective deadline
=
min(
    caller deadline,
    endpoint maximum
)

If:

TEXT
client asks for 30 seconds
endpoint maximum = 2 seconds

then:

TEXT
effective budget = 2 seconds

The caller communicates when the result becomes useless from its perspective.

The service still protects itself.

The inverse case matters too. An absurdly tiny external deadline may be below the minimum useful execution time for an endpoint.

Deadline propagation is not the same thing as allowing arbitrary callers to dictate internal resource policy.

Different endpoints need different contracts

Compare:

TEXT
GET /users/123

with:

TEXT
POST /reports/yearly-export

A single global:

TEXT
timeout = 5 seconds

may be too generous for the first and fundamentally wrong for the second.

The profile lookup might reasonably need:

TEXT
300 ms

The yearly report may need to become asynchronous:

TEXT
submit job
→ return job ID
→ poll or notify on completion

instead of holding one HTTP request open for minutes.

Deadline design often reveals API-design problems.

An enormous timeout is not automatically resilience.

Sometimes it is evidence that a synchronous interaction has outgrown the request/response model.

Observe Where the Budget Went

Traditional latency telemetry tells us:

TEXT
request took 780 ms

Deadline-aware observability can tell us how the request arrived at that point.

A trace might record:

TEXT
API ingress:
remaining = 790 ms

Order service:
remaining = 630 ms

Payment:
remaining = 410 ms

Payment retry:
remaining = 130 ms

Now it becomes obvious why another large retry would be pointless.

Useful metrics include:

TEXT
deadline_exceeded_total

cancelled_total

expired_before_processing_total

remaining_budget_at_dependency_call

work_completed_after_caller_cancelled

retry_skipped_due_to_deadline

That last category is particularly interesting:

TEXT
work completed after caller cancelled

directly measures wasted capacity.

Trace the time budget itself

A distributed trace could show:

TEXT
Checkout [800ms deadline]
│
├── Auth          45ms
│
├── Inventory    120ms
│
└── Payment
    ├── Queue     180ms
    ├── Attempt 1 260ms
    ├── Backoff   100ms
    └── Attempt 2 started
        with 70ms remaining

The problem is no longer vaguely:

TEXT
payment was slow

The trace exposes the real design problem:

TEXT
the second attempt began
without enough budget to be useful

That is much more actionable.

Observability should show where the time budget was spent in the same way a financial ledger shows where money went.

Deadline errors need context

Suppose:

TEXT
deadline errors ↑

That can mean:

TEXT
backend got slower

caller shortened its deadline

network latency increased

queueing increased

retry policy consumed more budget

A deadline failure is not automatically evidence that the backend itself is broken.

The contract may have become impossible.

If a caller supplies:

TEXT
deadline = 50 ms

while the service normally requires:

TEXT
100 ms

every request can fail even though the implementation has not degraded at all.

Temporal context matters on both sides of the API.

Timeouts Create Ambiguous Side Effects

Suppose:

TEXT
client deadline = 500 ms

At:

TEXT
490 ms

the payment provider completes the charge.

The response would reach the caller at:

TEXT
520 ms

but the caller stopped waiting at:

TEXT
500 ms

Observed result:

TEXT
client: timeout
server: success

The client may now retry.

Without idempotency:

TEXT
charge 1
charge 2

The deadline did not create the underlying distributed ambiguity.

It exposed it.

AWS's guidance on idempotent APIs emphasizes exactly this problem: retries are much safer when multiple attempts can be recognized as the same logical operation instead of generating repeated side effects.

For a payment call, that might mean a stable operation identifier:

TEXT
payment_request_42

survives:

TEXT
original attempt
timeout
retry
status reconciliation

Deadlines and idempotency therefore belong in the same design discussion.

One limits how long the caller waits.

The other makes uncertainty after that wait safe to recover from.

The Request Can Be Doomed While Every Service Is Healthy

This is one of the stranger properties of deadline-aware systems.

Suppose:

TEXT
API          healthy
Order        healthy
Inventory    healthy
Payment      healthy
Database     healthy

The request has:

TEXT
12 ms

remaining.

Payment normally takes:

TEXT
100 ms

Every component is healthy.

The operation is already doomed.

No resource graph needs to be red.

No service needs to be unavailable.

The problem is that the request's remaining lifetime no longer matches the work required to complete it.

This is why health alone is not enough.

Distributed systems also need temporal context.

Shared deadlines prevent locally rational, globally absurd decisions

Imagine each service team independently decides:

TEXT
Inventory timeout = 2 seconds

Payment timeout = 4 seconds

Database timeout = 10 seconds

Every choice may make perfect sense in isolation.

But distributed systems compose local policies.

If the user-facing operation is useful for only:

TEXT
800 ms

the combined behavior is nonsensical.

Deadline propagation replaces:

TEXT
independent local clocks

with:

TEXT
one shared operation clock

Each service can still impose stricter local limits.

None silently grants the request more life than the operation actually has.

The Invariants I Want

For a synchronous microservice request path, I care about guarantees such as:

TEXT
Every externally initiated request has a finite
end-to-end deadline.

Downstream calls never receive more time than the
original operation has remaining.

Local dependency timeouts may be stricter than the
remaining deadline, but may not silently extend it.

Queue time consumes the same budget as execution time.

Work that cannot realistically finish inside the
remaining budget should usually be rejected before
consuming expensive resources.

Caller cancellation propagates to cancellable
downstream work where practical.

Application code that starts long-running work must
cooperate with cancellation rather than assuming
the framework can stop it automatically.

Cancellation of irreversible side effects must not
be mistaken for rollback.

Retries and backoff must fit inside the original
deadline rather than creating a fresh time budget.

Timeout values should be grounded in observed latency
distributions and the actual network path.

Asynchronous jobs should receive deadlines based on
their own semantic lifetime rather than blindly
inheriting an HTTP request deadline.

Ambiguous timeout outcomes for side-effecting work
must be handled with idempotency or reconciliation.

Observability should show where the time budget was
spent and how much work continued after the caller
stopped waiting.

Those guarantees are much stronger than:

TEXT
every HTTP client has timeout=5000

because a timeout configuration describes one call.

The invariants describe the lifetime of the operation.

Final Thought

Microservices make time deceptively local.

Each service sees something like:

TEXT
one incoming request
one outgoing RPC
one database query

so it is natural for every team to assign its own timeout.

The caller does not experience those calls independently.

The caller experiences:

TEXT
one operation

with one point beyond which the result stops being useful.

That budget should shrink as the operation moves through the system.

Queueing spends it.

Network setup spends it.

RPCs spend it.

Database work spends it.

Retries spend it.

Backoff spends it.

And once the budget reaches zero, continuing to work is often not resilience.

It is waste.

That waste is especially dangerous under overload because it steals capacity from requests that could still succeed.

So the most useful question is not:

“What timeout should this service use?”

It is:

“How much of the caller's original time budget is left when this service begins work?”

Then comes the second question:

“If the caller no longer cares about the result, why are we still computing it?”

Those two questions connect latency design to capacity design.

They also connect deadlines to several other reliability concerns:

TEXT
timeouts
→ retries
→ idempotency
→ queueing
→ overload
→ backpressure
→ cancellation
→ observability

A deadline looks like a clock.

In a distributed system, it is more useful to think of it as a contract describing how long the system is allowed to keep spending resources on one operation's intent.

Every service in the chain should know when that contract expires.

References

  • Google SRE — Addressing Cascading Failures. Explains how requests that continue executing after their callers have exceeded their deadlines waste backend resources and can contribute to overload, and discusses propagating time budgets through RPC chains.

  • gRPC — Deadlines. Defines deadline semantics, recommends explicit client deadlines, explains expiration behavior, and documents deadline propagation using the remaining timeout rather than simply forwarding an absolute wall-clock value.

  • gRPC — Cancellation. Describes cancellation propagation through ongoing computation and downstream RPCs while emphasizing that application code remains responsible for stopping work it started.

  • gRPC — gRPC and Deadlines. Discusses deadline selection, server-side cancellation handling, and the important ambiguity that a server may complete successfully even when the caller observes DEADLINE_EXCEEDED.

  • AWS Well-Architected — Set client timeouts. Recommends explicit connection and request timeouts, warns about both excessively long and overly aggressive values, and cautions against relying blindly on client-library defaults.

  • AWS — Exponential Backoff and Jitter. Describes exponential backoff and jitter as mechanisms for reducing retry pressure and preventing synchronized retry bursts against recovering dependencies.

  • Amazon Builders' Library — Making retries safe with idempotent APIs. Explains why distributed operations that may be retried need stable request identity when duplicate side effects would be unsafe.