Why Can a “Successful” Kubernetes Rolling Deployment Still Drop Requests?

The deployment finishes successfully:

deployment "checkout-api" successfully rolled out

Every new Pod reports:

READY 1/1
STATUS Running

The old ReplicaSet scales to zero, Kubernetes reports exactly the state we wanted, and yet production traffic during the rollout contains:

502
502
connection reset
503

At first, that feels contradictory. If the rollout succeeded, why did requests fail?

Because Kubernetes answering:

“Did I successfully replace the old Pods with healthy new Pods?”

is not the same as the application answering:

“Did every request continue flowing safely while that replacement happened?”

A rolling deployment coordinates several moving parts:

Deployment controller
ReplicaSets
Pods
EndpointSlices
kubelet
Service routing
external load balancer
application process
active TCP connections

Those pieces do not all change state atomically.

When an old Pod begins terminating, Kubernetes starts updating its endpoint state and initiates graceful shutdown. EndpointSlices deliberately retain terminating endpoints for a period of time, and an external load balancer may have its own propagation and connection-draining behavior. Existing TCP connections can survive yet another amount of time.

That creates a dangerous interval:

Kubernetes has decided:
"this Pod is leaving"

while

some network component or existing connection
still has work associated with that Pod

If the process disappears before that work drains, the request disappears with it.

So the interesting question is not whether Kubernetes supports rolling deployments. Of course it does.

The more useful question is:

What has to remain true between Pod deletion and the final request leaving that Pod?

RollingUpdate Protects Pod Capacity, Not Request Completion

A Kubernetes Deployment normally uses the RollingUpdate strategy. Two important controls are maxUnavailable and maxSurge:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 25%
    maxSurge: 25%

maxUnavailable limits how many desired replicas may be unavailable during the rollout, while maxSurge limits how many additional Pods may temporarily exist above the desired replica count.

With four replicas, a rollout might evolve roughly like this:

Old: 4
New: 0

then:

Old: 3
New: 2

then:

Old: 2
New: 3

and eventually:

Old: 0
New: 4

Conceptually:

That gives the Deployment controller a capacity envelope for replacing Pods.

What it does not provide is a request-completion guarantee.

Even:

maxUnavailable: 0

does not mean:

zero failed requests

It means Kubernetes is controlling Pod availability according to the rollout strategy.

Whether a request that already reached an old Pod survives termination is a different problem.

Follow One Request Through Pod Termination

Consider three replicas behind a load balancer:

Now Kubernetes decides to terminate Pod A.

At a high level, graceful termination involves several steps:

1. The Pod receives a deletion timestamp.
2. Its endpoint becomes terminating.
3. Normal readiness for the terminating endpoint becomes false.
4. The kubelet begins the shutdown process.
5. Lifecycle hooks and termination signals are processed.
6. The Pod is given its configured grace period.
7. Remaining processes can eventually be terminated forcibly if they do not exit.

The important detail is that these are not one atomic instruction broadcast to every component involved in routing traffic.

There is propagation.

That creates the central race in this article:

The application can terminate faster than traffic can disappear.

Kubernetes may already understand that the endpoint is leaving while another component still has a stale backend view, an established connection, or work already executing on the Pod.

A safe rollout therefore requires more than replacing Pods in the right order. It requires preserving the old instance long enough for the traffic path to converge.

Endpoint Removal and Load-Balancer Draining Are Different Timelines

Modern Kubernetes EndpointSlices expose useful endpoint conditions including:

ready
serving
terminating

A terminating endpoint can temporarily resemble:

{
  "ready": false,
  "serving": true,
  "terminating": true
}

That distinction is intentional.

It communicates something closer to:

Do not select me for ordinary new traffic.

But I am still alive and may be serving
work that has not finished yet.

That is very different from:

endpoint removed
→ process dies immediately

The separation gives implementations room to support connection draining.

An external load balancer is another state machine

Inside Kubernetes, EndpointSlice state can change relatively quickly.

A cloud load balancer is another system with another representation of backend health.

A path can look like:

Several components in that diagram may independently maintain some version of:

Is this endpoint still available?

Those views can converge quickly without converging instantaneously.

Google documents a concrete version of this problem for GKE Network Endpoint Groups. During workload scale-down, prematurely closing existing connections can result in 502 responses, while traffic continuing toward endpoints that have already disappeared can result in 503 responses.

That is exactly the kind of issue that makes a rollout look healthy from the Deployment controller while user requests still fail.

Existing connections complicate removal

HTTP keep-alive makes the problem more subtle.

Even if a load balancer stops establishing new connections to a Pod, it may still have an existing connection associated with that backend.

No request may be active at the exact moment termination begins, yet another request can arrive over that persistent connection shortly afterward.

Graceful draining may therefore require the application or proxy to:

stop accepting new connections
stop accepting new requests on retiring connections
allow active requests to finish
close idle connections deliberately

Stopping a listener and finishing requests are related operations.

They are not exactly the same thing.

Graceful Shutdown Is Application Behavior

Receiving SIGTERM is not an instruction to disappear as quickly as possible.

Suppose an application effectively does this:

Runtime.getRuntime()
    .addShutdownHook(
        new Thread(() -> {
            System.exit(0);
        })
    );

Or perhaps the framework receives the termination signal and immediately closes everything.

From the process's point of view, that can look perfectly orderly:

received SIGTERM
→ exited

Operationally, it may be disastrous.

Imagine:

T+0 ms      Pod termination begins
T+20 ms     application exits
T+300 ms    routing state still propagating
T+500 ms    old connection sends next request

The process is gone.

The connection is not.

A better lifecycle looks like:

receive termination
↓
stop accepting new work
↓
keep process alive
↓
finish active requests
↓
close connections
↓
exit

The key word is drain.

preStop can create a propagation window

Kubernetes supports preStop lifecycle hooks.

For example:

spec:
  terminationGracePeriodSeconds: 60

  containers:
    - name: checkout-api
      image: example/checkout:42

      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - sleep 15

A blunt:

sleep 15

is not sophisticated, but it can serve an important purpose:

Pod deletion begins
↓
routing systems get time to stop choosing endpoint
↓
process remains alive
↓
SIGTERM / application shutdown proceeds

Cloud load-balancer environments sometimes recommend this kind of delay because endpoint deprogramming takes time. GKE's guidance, for example, discusses coordinating connection draining, preStop, and the overall Pod termination grace period.

The number itself is workload-specific.

Do not copy:

sleep 120

because one documentation example or blog post happened to use it.

Measure the behavior of your own traffic path.

Blind sleep is useful, but blind

Suppose endpoint removal normally takes:

2 seconds

while the hook always sleeps:

20 seconds

You spend roughly 18 seconds waiting for no useful reason during every termination.

Now suppose a regional or load-balancer event occasionally takes:

35 seconds

The same 20-second delay is still unsafe.

Application-aware draining is more precise when practical. A simple delay remains popular because it is:

simple
predictable
easy to test

Whether that simplicity is good enough depends on the failure model.

The Termination Grace Period Is a Budget

A Pod does not have unlimited time to shut down.

Kubernetes uses a default termination grace period of 30 seconds unless you configure another value. When the grace period is exhausted, remaining container processes can eventually be terminated forcibly.

So:

terminationGracePeriodSeconds: 60

should be treated as a shutdown budget.

Conceptually:

T_grace
>=
T_preStop
+
T_application_shutdown
+
safety margin

If an external load balancer has explicit connection-drain or propagation requirements, those timings need to fit into the lifecycle as well.

Consider:

preStop = 20 sec
longest request = 45 sec
termination grace = 30 sec

That configuration contains a mathematical contradiction.

Some valid in-flight request cannot possibly finish before the process loses its grace window.

Long requests force an architectural decision

Imagine:

POST /reports/export

can take:

45 seconds

A request begins, then Kubernetes starts terminating the Pod two seconds later.

With a 30-second termination budget, something must change.

Possible choices include:

increase shutdown time

or:

redesign long work as asynchronous

or:

support cancellation and resumption

or:

make the request safely retryable

A long-running synchronous operation and a short-lived deployment lifecycle are architectural assumptions.

They need to agree.

Health Probes Have Different Jobs

Kubernetes exposes several health concepts that are easy to blur together.

Treating them as interchangeable can turn a normal degradation into an outage.

Readiness asks whether the Pod should receive traffic

A readiness probe answers something like:

Should this instance receive normal production traffic right now?

For example:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080

  periodSeconds: 5
  failureThreshold: 2

A container can be:

alive
but
not ready

and that can be completely valid.

Perhaps it is temporarily unable to accept new work but does not need restarting.

Liveness asks whether restarting would help

A liveness probe asks a much more invasive question:

Is this process unhealthy in a way that makes restart the correct recovery action?

For example:

livenessProbe:
  httpGet:
    path: /live
    port: 8080

  periodSeconds: 10
  failureThreshold: 3

Repeated liveness failure can cause the container to restart.

Kubernetes explicitly warns that badly designed liveness checks can produce cascading failures. An overloaded container fails its probe, gets restarted, remaining Pods inherit more traffic, those Pods become overloaded too, and capacity keeps shrinking.

A mechanism intended to improve resilience has become an outage amplifier.

Do not make liveness depend on every downstream dependency

Suppose /live fails whenever PostgreSQL is temporarily unavailable:

PostgreSQL outage
↓
all API Pods fail liveness
↓
Kubernetes restarts entire API fleet

Did restarting the API repair PostgreSQL?

No.

It may actually make recovery harder by producing:

connection storms
cache warmup
JVM warmup
new DNS/TLS connections

Now the incident is:

database problem
+
application restart problem

Liveness should generally represent whether restarting this container is likely to help.

Readiness can be more sensitive to temporary inability to serve traffic.

Startup probes protect slow initialization

Suppose a Spring service needs:

60 seconds

to initialize, warm caches, load configuration, and establish connections.

If liveness begins too early:

starting
→ probe fails
→ restart
→ starting
→ probe fails

the application may never finish startup.

A startup probe gives initialization its own window:

startupProbe:
  httpGet:
    path: /startup
    port: 8080

  failureThreshold: 30
  periodSeconds: 5

That configuration allows up to:

30 × 5 = 150 seconds

for startup before Kubernetes treats it as failed.

Each probe has a different semantic job.

Combining them all into one /health endpoint with identical behavior throws those semantics away.

Readiness Must Mean Ready for Production

A new Pod can be technically reachable long before it is safe to send real traffic to it.

Imagine /ready returns:

200

the moment the HTTP server binds to port 8080.

The process is alive, but perhaps it has not yet:

warmed local caches
initialized expensive clients
loaded reference data
established connection pools
completed JIT warmup

The rollout can now replace warm instances with cold-but-ready instances:

The Deployment succeeds.

Latency spikes.

Requests may time out.

Liveness may even begin failing if the cold instance becomes overloaded immediately.

Readiness should therefore mean:

This instance can actually handle production traffic at an acceptable level.

not merely:

The process opened a socket.

Readiness also applies during runtime

Kubernetes continues running readiness probes after startup, so an instance can temporarily leave normal traffic rotation if it becomes unable to serve safely.

That is powerful.

It also requires careful capacity reasoning.

Suppose every Pod marks itself unready when its connection pool becomes heavily utilized:

traffic spike
↓
Pod A not ready
↓
Pod B receives more traffic
↓
Pod B not ready
↓
Pod C receives all remaining traffic
↓
Pod C not ready

A protective mechanism can remove the whole fleet from service.

Readiness needs semantics tied to the behavior of the system, not merely a threshold inside one process.

Application-aware drain state

An application can make its own lifecycle explicit:

RUNNING
↓
DRAINING
↓
TERMINATED

When draining begins:

/ready → 503

/live → 200

existing requests → continue

new expensive work → rejected

A simplified Spring component might track that state:

@Component
public class DrainState {

    private final AtomicBoolean draining =
        new AtomicBoolean(false);

    public void startDraining() {
        draining.set(true);
    }

    public boolean isDraining() {
        return draining.get();
    }
}

Readiness can reflect it:

@GetMapping("/ready")
public ResponseEntity<Void> ready() {

    if (drainState.isDraining()) {
        return ResponseEntity
            .status(503)
            .build();
    }

    return ResponseEntity.ok().build();
}

Shutdown can then follow:

mark draining
→ wait for active requests
→ close dependencies
→ exit

Modern Kubernetes already marks terminating endpoints as not ready for normal Service traffic, so application-aware draining is not simply about manually telling Kubernetes that deletion has started.

Its value is broader:

control new application work
expose internal drain state
finish active operations
coordinate external systems

Kubernetes manages endpoint lifecycle.

The application still owns application semantics.

Rollout Capacity Still Needs Headroom

A rollout can be perfectly compliant with its Deployment strategy and still overload the application.

Suppose the service normally runs:

4 replicas

and each replica can handle:

250 req/s

Total nominal capacity:

1,000 req/s

Production traffic:

850 req/s

Now imagine the rolling configuration permits one unavailable Pod.

Temporary serving capacity becomes:

3 × 250 = 750 req/s

while traffic remains:

850 req/s

The rollout is behaving exactly as configured.

The service is overloaded.

maxUnavailable describes allowable Pod unavailability.

It does not know how much spare traffic capacity the workload has.

maxSurge can preserve serving capacity

For a four-replica latency-sensitive service, a strategy such as:

strategy:
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

allows Kubernetes to start another Pod before retiring an old one:

4 old
↓
4 old + 1 new
↓
3 old + 2 new
↓
...
↓
4 new

That can preserve nominal HTTP capacity.

It also creates temporary demand elsewhere:

extra CPU
extra memory
extra database connections
extra Kafka connections
extra cache connections

Suppose every Pod opens:

30 DB connections

Surging several new Pods can put pressure on PostgreSQL even while the HTTP tier looks safer.

Every availability control consumes capacity somewhere.

New Pods can create dependency storms

Imagine ten Pods start together.

Each establishes:

50 DB connections

Startup suddenly requests:

500 connections

The rollout protected application capacity while hammering the database.

The same pattern can occur with:

Redis
Kafka
DNS
TLS handshakes
configuration services

Adding application compute does not automatically add capacity to every shared dependency.

Deployment headroom needs to be evaluated end to end.

Long-Lived and Background Work Need Their Own Drain Semantics

Normal HTTP request/response traffic is only one workload.

A Pod may also own connections or work that lasts minutes or hours.

Long-lived connections

Consider:

WebSocket
Server-Sent Events
gRPC streaming

Waiting for every connection to end naturally may make a rollout practically impossible.

A drain policy might instead:

stop accepting new streams
send reconnect instruction
allow a short grace period
close remaining streams
let clients reconnect elsewhere

That requires client behavior too.

For some protocols, graceful rollout is not purely a server operation. It is a contract between server and client.

Background workers

Suppose a Pod is consuming Kafka records and currently processing:

event 4812

A bad termination path is:

SIGTERM
→ exit immediately
→ event partially processed

A safer lifecycle is:

receive termination
↓
stop polling for new records
↓
finish current batch
↓
commit appropriate offsets
↓
close consumer
↓
exit

That work still needs to fit inside:

terminationGracePeriodSeconds

or the kubelet can eventually terminate the process anyway.

Graceful shutdown is workload-specific.

Sidecars add another shutdown order

A Pod may contain:

application
service-mesh proxy
logging agent
security sidecar

Now process ordering matters.

If the network proxy exits before the application finishes its outgoing requests, the application may remain alive but lose the ability to complete its work.

Modern Kubernetes has improved sidecar lifecycle behavior, but the architectural lesson remains:

A Pod can contain several processes whose shutdown order affects correctness.

In environments where injected sidecars participate in request handling, lifecycle behavior needs to account for those containers too.

PodDisruptionBudgets Solve a Different Problem

A PodDisruptionBudget can limit how many Pods are made unavailable by supported voluntary disruption operations.

For example:

apiVersion: policy/v1
kind: PodDisruptionBudget

metadata:
  name: checkout-api

spec:
  minAvailable: 3

  selector:
    matchLabels:
      app: checkout-api

For a four-replica service, this says that supported voluntary eviction should preserve at least three available replicas.

That is useful.

It is not:

a guarantee that Pods cannot disappear

Hardware failure does not consult the PDB first.

Neither do:

kernel crash
network loss
OOM
power loss

The difference between voluntary and involuntary failure matters.

PDB is not RollingUpdate configuration

These settings can look similar:

Deployment maxUnavailable

and:

PodDisruptionBudget

but they apply to different control paths.

A Deployment's rolling-update settings govern how the Deployment controller replaces Pods for the application's own rollout.

A PDB constrains supported voluntary eviction workflows such as node drains.

A production service may need both.

They are not interchangeable.

Graceful Shutdown Cannot Carry Correctness

Everything discussed so far assumes Kubernetes gets to perform an orderly lifecycle.

Real failures also include:

node disappears
VM power failure
kernel panic
network partition
hardware failure
process crash

There may be no useful preStop.

No generous drain window.

No friendly SIGTERM.

Graceful shutdown improves planned lifecycle events.

It does not replace correctness mechanisms for abrupt instance loss.

Requests or jobs may still need:

idempotency
retries
transactional boundaries
reconciliation

because some processes will eventually disappear without announcing themselves.

Do not put essential state only in shutdown hooks

Suppose critical correctness depends on:

SIGTERM
→ flush essential state
→ exit

What happens when the machine loses power?

The hook never runs.

Shutdown hooks are appropriate for:

draining
cleanup
closing resources
best-effort finalization

They should not be the only place the system establishes irreversible correctness.

Critical state should already be durable before termination begins.

Test Rollouts With Real In-Flight Traffic

This test:

kubectl rollout restart deployment/checkout-api
kubectl rollout status deployment/checkout-api

tests whether Kubernetes can complete the rollout.

It does not fully test the application's request lifecycle.

A stronger experiment generates traffic continuously while rolling the service repeatedly.

For example:

100 req/s normal requests
+
several 10-second requests
+
persistent keep-alive connections

during:

rollout restart

Then measure:

HTTP errors
connection resets
p95
p99
in-flight request completion
active connection count
Pod termination timing
EndpointSlice state

Now the test is evaluating the real promise:

deployment should not break user requests

instead of merely:

deployment eventually becomes Ready

That distinction is the heart of the article.

Watch EndpointSlices during termination

The lifecycle is directly observable:

kubectl get endpointslices \
  -l kubernetes.io/service-name=checkout-api \
  -o yaml

A terminating endpoint may show:

conditions:
  ready: false
  serving: true
  terminating: true

while a replacement endpoint reports:

conditions:
  ready: true
  serving: true
  terminating: false

Kubernetes' graceful-termination documentation demonstrates this transition explicitly.

Seeing it happen in a real rollout makes the traffic handoff much easier to understand than treating Pod lifecycle as an abstract diagram.

Instrument Shutdown as a Production Operation

If graceful shutdown matters to availability, shutdown itself deserves observability.

I would emit lifecycle events such as:

{
  "event": "shutdown_started",
  "pod": "checkout-7c84d",
  "activeRequests": 17
}

then:

{
  "event": "readiness_disabled",
  "pod": "checkout-7c84d"
}

then:

{
  "event": "shutdown_drain_progress",
  "activeRequests": 3
}

and finally:

{
  "event": "shutdown_complete",
  "activeRequests": 0,
  "durationMs": 4832
}

Useful metrics include:

shutdown_duration
requests_active_at_shutdown
requests_completed_during_drain
requests_aborted_during_shutdown
forced_termination_count

Those signals let operators answer questions that a normal Deployment dashboard cannot:

How many requests were active when termination began?

How long does drain usually take?

Are Pods reaching zero active requests?

How often are we force-killed?

Do rollout errors correlate with one particular shutdown stage?

If graceful rollout is important enough to engineer, it is important enough to measure.

A More Intentional Deployment Configuration

For a latency-sensitive HTTP service, a reasonable starting structure might resemble:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: checkout-api

spec:
  replicas: 4

  strategy:
    type: RollingUpdate

    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

  template:
    metadata:
      labels:
        app: checkout-api

    spec:
      terminationGracePeriodSeconds: 60

      containers:
        - name: app
          image: checkout-api:42

          ports:
            - containerPort: 8080

          startupProbe:
            httpGet:
              path: /startup
              port: 8080

            periodSeconds: 5
            failureThreshold: 24

          readinessProbe:
            httpGet:
              path: /ready
              port: 8080

            periodSeconds: 5
            failureThreshold: 2

          livenessProbe:
            httpGet:
              path: /live
              port: 8080

            periodSeconds: 10
            failureThreshold: 3

          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - sleep 10

This is not a universal production template.

The useful part is being able to explain why each value exists:

maxUnavailable:
How much real serving capacity can disappear?

maxSurge:
How much temporary resource overhead is safe?

startupProbe:
How long does real startup require?

readiness:
When can this instance safely take production traffic?

liveness:
When would restart genuinely improve the situation?

preStop:
How much propagation or drain time is required?

terminationGracePeriod:
How long can the complete shutdown lifecycle take?

Configuration should encode measured lifecycle behavior.

It should not be a collection of values copied from another application whose traffic path, request duration, load balancer, and dependencies may be completely different.

Failure Patterns I Would Avoid

Several patterns make me especially nervous because they collapse different lifecycle concerns into one crude signal.

Liveness and readiness mean the same thing

livenessProbe:
  httpGet:
    path: /health

and:

readinessProbe:
  httpGet:
    path: /health

with identical semantics.

Temporary inability to accept traffic does not necessarily mean restarting the process will help.

Readiness becomes true when the port opens

port opened
=
production ready

even though caches, pools, reference data, or expensive clients are still initializing.

Immediate exit on SIGTERM

SIGTERM
→ exit now

with no drain window for active requests or persistent connections.

Grace period shorter than legitimate work

terminationGracePeriodSeconds: 5

for requests that can take 30 seconds.

The configuration has guaranteed that some valid work cannot complete gracefully.

Huge preStop sleep with no measurement

sleep 120 because a blog said so

rather than measuring actual endpoint-removal and drain behavior.

Assuming a PDB prevents downtime

A PDB constrains supported voluntary disruptions.

It does not make nodes or processes immortal.

Treating rollout completion as an application SLO

kubectl rollout status = success

does not prove:

zero customer-visible failures

The first is Kubernetes object state.

The second is a user-experience property.

The Failure Timeline I Want

A healthy termination path looks something like this:

The unsafe version changes only one ordering relationship:

That ordering difference explains how a Deployment can be technically successful while requests still fail.

Kubernetes replaced the Pod correctly.

The traffic handoff was not complete when the process disappeared.

The Invariants I Want

For a Kubernetes HTTP service, I care about guarantees such as:

A terminating Pod must stop receiving ordinary new
traffic before its process disappears.

In-flight requests must have enough time to finish
or have explicit retry-safe behavior.

The termination grace period must exceed the real
drain and application-shutdown requirements.

New Pods must not become ready until they can
handle production traffic safely.

Temporary overload must not cause liveness checks
to restart otherwise recoverable Pods.

A rolling update must preserve enough real serving
capacity, not merely enough Ready objects.

External load-balancer draining must be coordinated
with Pod termination.

Planned graceful shutdown must not be the only
mechanism protecting correctness from abrupt failure.

Long-lived connections need an explicit reconnect
or drain policy.

Background workers must stop acquiring new work
before shutdown and finish or safely release the
work they already own.

Sidecars involved in request delivery must remain
available long enough for application drain.

Deployment success must be measured with real user
traffic, not only Kubernetes resource state.

Shutdown must be observable enough to tell whether
requests were drained or merely interrupted.

Those guarantees are much stronger than:

strategy: RollingUpdate

because the strategy tells Kubernetes how to replace objects.

The invariants describe what users should experience while that replacement happens.

Final Thought

Kubernetes is very good at replacing Pods.

But a Pod is not a request.

A Deployment controls ReplicaSets. A ReplicaSet controls Pods. EndpointSlices communicate endpoint state. The kubelet manages container termination. A cloud load balancer maintains another view of the available backends. Your application owns active requests and background work. Clients and proxies may retain persistent connections of their own.

A zero-downtime rollout requires those systems to agree on one basic ordering:

stop sending new work
↓
finish existing work
↓
terminate process

If the ordering becomes:

terminate process
↓
eventually stop sending work

requests get dropped.

The Deployment can still report complete success.

That is why I no longer think of graceful Kubernetes deployment merely as:

“Kubernetes keeps one Pod running while another starts.”

That describes the capacity side of the problem.

The fuller model is:

“The old instance must remain useful until every system that can still route or retain work for it has safely moved on.”

That includes readiness, EndpointSlice state, connection draining, SIGTERM, preStop, termination budgets, external load-balancer propagation, in-flight request tracking, long-lived connections, background workers, sidecars, and enough spare capacity for the rollout itself.

When those pieces are aligned, rolling deployment becomes what people often assume it already guarantees:

a way to change production software without users noticing.

References

• Kubernetes — Pod Lifecycle. Documents graceful Pod termination, termination grace periods, EndpointSlice behavior during termination, and eventual forced shutdown of containers that remain alive beyond the grace period.

• Kubernetes — Explore Termination Behavior for Pods and Their Endpoints. Demonstrates endpoint transitions during Pod deletion, including ready: false, serving: true, and terminating: true, and explains how those conditions support connection draining.

• Kubernetes — EndpointSlices. Defines the ready, serving, and terminating endpoint conditions used by routing and draining implementations.

• Kubernetes — Deployments. Documents RollingUpdate, maxUnavailable, and maxSurge, including their role in controlling replica replacement.

• Kubernetes — Liveness, Readiness, and Startup Probes. Defines the separate semantics of readiness, liveness, and startup checks and warns that badly designed liveness checks can create cascading failures under load.

• Google Kubernetes Engine — Ingress Configuration and Graceful Shutdown. Documents real 502 and 503 failure modes during Pod scale-down and explains how connection draining, preStop, endpoint removal, and termination grace periods need to be coordinated.

• Kubernetes — PodDisruptionBudget. Documents how PDBs constrain voluntary Pod disruption and why they address a different problem from Deployment rolling-update configuration.