Where Does Kafka’s “Exactly Once” Guarantee Actually End?

“Exactly once” sounds wonderfully simple: an event arrives, you process it, and the result happens once—never zero times and never twice.

Distributed systems rarely give us a guarantee that broad.

Kafka does provide exactly-once processing semantics in important scenarios. Its transaction model can atomically connect consumed offsets with records produced back into Kafka, while idempotent producers prevent broker-level retries from creating duplicate records. Apache Kafka's own design documentation describes exactly-once processing for read-process-write workflows where both the input and output remain inside Kafka. Kafka Design and Message Delivery Semantics

The confusion begins when that guarantee gets mentally expanded into:

“If my application uses Kafka exactly-once semantics, everything caused by the event happens exactly once.”

That is not what Kafka guarantees.

Consider a consumer that performs this sequence:

Kafka
  ↓
consume payment event
  ↓
update PostgreSQL
  ↓
send email
  ↓
call payment API
  ↓
commit Kafka offset

Only some of those operations belong to Kafka's transaction model. The database, payment provider, mail system, filesystem, or any other external system does not automatically become part of the Kafka transaction simply because the event originated in Kafka.

That gives us a much more useful engineering question:

Where does Kafka's exactly-once guarantee end, and what correctness mechanism takes over after that boundary?

That question tells us far more about a production architecture than simply asking whether exactly_once_v2 is enabled.

“Exactly Once” Describes More Than One Guarantee

The phrase exactly once is overloaded.

Someone saying:

we have exactly once

might mean any of these:

Producer retries do not create duplicate Kafka records.

A consume → process → produce workflow commits atomically.

Application state is updated once.

A database row changes once.

A customer is charged once.

An email is sent once.

An event can never be observed again.

Those are not equivalent guarantees.

The first two can be properties of Kafka itself under the right configuration and topology. The others involve application semantics or external systems that Kafka does not control.

That distinction is the foundation of everything that follows. Before designing for "exactly once," we have to define exactly once where.

Producer Idempotence Solves One Specific Duplicate Problem

Start with a producer sending:

order.created

Kafka successfully appends the record, but the acknowledgment back to the producer disappears.

From the producer's perspective:

timeout

does not tell us whether the broker stored the message.

Without protection, the application may resend and create two records:

order.created
order.created

Kafka's idempotent producer is designed for precisely this broker-level ambiguity. The broker tracks producer identity and sequence information so retrying the same send does not append another copy.

That is valuable, but notice what it solves:

transport retry
→ duplicate Kafka append

It does not solve:

application creates the same
business event twice

For example:

producer.send(
    new ProducerRecord<>(
        "payments",
        paymentId,
        paymentEvent
    )
);

// application loses its own state

producer.send(
    new ProducerRecord<>(
        "payments",
        paymentId,
        paymentEvent
    )
);

Those are two application-level send operations. Kafka cannot infer that they represent one business intent.

That requires a stable business identity.

Producer idempotence is therefore an important guarantee, but it is not a universal deduplication system.

Kafka Transactions Join Output With Consumer Progress

Producer idempotence becomes even more powerful when combined with Kafka transactions.

Suppose a service consumes from:

orders

and produces to:

validated-orders

The workflow is:

read order
→ validate
→ produce validated order
→ commit consumed offset

Without atomicity, this sequence can fail in an uncomfortable place:

read A
→ produce B
→ crash
→ offset not committed
→ read A again
→ produce B again

Kafka transactions solve this read-process-write case by allowing the produced records and consumed offsets to participate in the same Kafka transaction.

Conceptually:

producer.beginTransaction();

for (
    ConsumerRecord<String, Order> record
    : records
) {
    ValidatedOrder result =
        validate(record.value());

    producer.send(
        new ProducerRecord<>(
            "validated-orders",
            record.key(),
            result
        )
    );
}

producer.sendOffsetsToTransaction(
    offsets,
    consumer.groupMetadata()
);

producer.commitTransaction();

Kafka can now treat:

produce output
+
advance consumer position

as one atomic operation.

Either both commit:

output visible
offset advanced

or neither does:

output hidden from transactional readers
offset not advanced

This is the heart of Kafka's exactly-once processing model. The consumed offsets themselves live in Kafka, which is what allows the transaction coordinator to atomically manage both progress and output.

The Guarantee Depends on Transaction-Aware Consumers

A Kafka transaction can contain records that are later aborted.

Those records can still physically exist in the log.

A consumer configured with:

read_uncommitted

may observe them.

A consumer using:

read_committed

will expose committed transactional records while withholding records belonging to aborted or still-open transactions.

So the architecture is not merely:

transactional producer

It also depends on downstream readers honoring the transactional isolation model.

Infrastructure guarantees often require cooperation from both ends of the boundary.

Kafka Also Has to Fence Stale Producers

Exactly-once processing has another distributed-systems problem to solve: ownership after failure.

Suppose processor A temporarily loses connectivity. Another instance, processor B, is started and takes over the work.

Processor A was not actually dead.

It eventually reconnects.

Without protection, both instances could believe they are allowed to continue producing transactional output for the same logical processor.

Kafka uses transactional producer identity and producer epochs to fence stale producers. Once a newer producer instance takes ownership of a transactional identity, the older producer can no longer continue committing as though it still owns that work.

This is another useful reminder that exactly-once semantics involve more than message deduplication.

They also require controlled ownership across failures.

The Boundary Ends When Work Leaves Kafka

Up to this point, Kafka controls the important pieces:

input offsets
transaction state
output records
producer identity

That is why it can coordinate them.

Now add PostgreSQL.

Suppose a consumer looks like:

for (
    ConsumerRecord<String, OrderEvent> event
    : records
) {
    orderRepository.updateStatus(
        event.key(),
        "PAID"
    );

    consumer.commitSync();
}

Consider the failure point:

consume event
→ update database
→ database commits
→ process crashes
→ Kafka offset never commits

After restart, Kafka sends the event again.

This is ordinary at-least-once processing.

We could reverse the order:

consume event
→ commit Kafka offset
→ update database

but now a crash between the offset commit and database update creates the opposite failure:

Kafka thinks event completed
database never changed

That leaves us with the classic tradeoff:

process then commit
→ at least once
→ duplicates possible

versus:

commit then process
→ at most once
→ loss possible

The hard problem is making:

database commit
+
Kafka offset commit

atomic.

They belong to different transactional systems.

Wrapping the Kafka portion in:

producer.beginTransaction();

does not enlist PostgreSQL.

And wrapping the database work in:

BEGIN;
UPDATE orders ...;
COMMIT;

does not enlist Kafka's transaction coordinator.

A better mental model is:

Kafka transaction
├── output Kafka records       covered
├── consumed Kafka offsets     covered
├── Kafka state/changelog      covered
├── PostgreSQL update          outside boundary
├── payment API call           outside boundary
├── email send                 outside boundary
└── file write                 outside boundary

Kafka's exactly-once guarantee is real.

It has a boundary.

Database Side Effects Need Their Own Idempotency

One of the most common strategies is to accept at-least-once delivery while making the database operation safe to repeat.

Suppose each event carries:

event_id = evt_84219

The database can track processed events:

CREATE TABLE processed_events (
    event_id UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ
        NOT NULL DEFAULT now()
);

Then process the dedupe marker and business effect inside the same database transaction:

BEGIN;

INSERT INTO processed_events(event_id)
VALUES ('84219000-...')
ON CONFLICT DO NOTHING;

If that operation indicates the event already existed:

skip business mutation

Otherwise:

UPDATE orders
SET status = 'PAID'
WHERE id = 'order_42';

COMMIT;

The key property is:

dedupe marker
+
business mutation

commit atomically.

A Spring-style version might look like:

@Transactional
public void handle(PaymentCompleted event) {

    boolean firstTime =
        processedEventRepository
            .insertIfAbsent(event.eventId());

    if (!firstTime) {
        return;
    }

    orderRepository.markPaid(
        event.orderId()
    );
}

Kafka can now redeliver:

evt_84219
evt_84219
evt_84219

without applying the business effect more than once.

This is often described as effectively-once behavior:

transport may deliver more than once
business effect becomes idempotent

For many applications, that is the guarantee that actually matters.

Sometimes the operation is already idempotent

Not every event requires an explicit dedupe table.

This operation:

UPDATE orders
SET status = 'PAID'
WHERE id = 42;

has a useful property: executing it twice produces the same final state.

Compare it with:

UPDATE accounts
SET balance = balance - 100
WHERE id = 42;

Executing that twice absolutely does not produce the same result.

The business operation determines how much deduplication machinery is required.

Useful patterns include:

SET state = X

UPSERT using a stable business key

INSERT protected by a unique constraint

compare-and-set transition

dedupe event ID

version check

This is one reason "exactly once" cannot be treated purely as Kafka configuration.

The semantics of the business operation matter.

Business Identity Matters More Than Message Identity

Suppose Kafka contains:

event ID = A
payment ID = payment_123

and later:

event ID = B
payment ID = payment_123

The Kafka events are different.

The business operation may be the same.

If we deduplicate only on:

event ID

then:

A != B

and both operations execute.

Sometimes the correct idempotency identity is therefore:

payment_123

rather than:

Kafka message A

Other examples might be:

refund_839
invoice_2026_1041
reservation_611

The right identity corresponds to the business intent that should happen once, not necessarily the transport record carrying that intent.

This distinction appears in retry-safe HTTP APIs, webhook consumers, and event-driven systems for the same reason.

Infrastructure identity and business identity are not always the same thing.

External APIs Need Their Own Correctness Protocol

Now consider a consumer handling:

payment.requested

and calling a payment provider:

async function handle(event: PaymentRequested) {
  await paymentProvider.charge({
    customerId: event.customerId,
    amount: event.amount,
  });

  await commitKafkaOffset();
}

The failure sequence is straightforward:

payment provider charges customer
→ response succeeds
→ process crashes
→ offset remains uncommitted
→ Kafka redelivers
→ charge attempted again

Kafka transactions cannot roll back the first charge.

The payment provider sits outside Kafka's atomicity boundary.

If the provider supports idempotency keys, use a stable business operation identity:

await paymentProvider.charge(
  {
    customerId: event.customerId,
    amount: event.amount,
  },
  {
    idempotencyKey: `payment:${event.paymentId}`,
  },
);

Now:

Kafka delivery 1
Kafka delivery 2
Kafka delivery 3

all refer to one payment operation.

This is not Kafka exactly-once semantics.

It is a separate correctness mechanism composed with Kafka delivery semantics.

That is the larger lesson:

End-to-end correctness is often built from several guarantees, each owned by a different system.

Some Effects Cannot Be Made Exactly Once

Email makes the limitation especially clear.

Imagine:

Kafka event
→ SMTP server
→ recipient

The mail server accepts the message.

Your consumer crashes before persisting completion.

After restart, you cannot necessarily answer with certainty:

Did the recipient receive the email?

Retrying may send two messages.

Not retrying may result in zero if the previous attempt did not actually complete.

Unless the external system participates in a deduplication or idempotency protocol, that ambiguity cannot be eliminated by a Kafka configuration.

The acceptable behavior depends on the effect.

For a marketing notification:

occasional duplicate

may be tolerable.

For:

charge customer $500

it usually is not.

The architecture has to follow the business consequence of duplication or loss.

There is no universal transport setting that turns every irreversible real-world side effect into an atomic operation.

Keep the Transaction Inside Kafka When You Can

One useful architecture is to keep the first processing boundary entirely inside Kafka.

Instead of:

Kafka input
→ database directly

use:

The first processor can use Kafka's transactional read-process-write model.

The database writer can then separately implement:

idempotent upsert
+
deduplication

This does not create one magical global transaction.

It creates two correctness boundaries that can each be reasoned about clearly.

That is often a much better architecture than trying to pretend one system owns atomicity everywhere.

Kafka Streams makes the Kafka side of this pattern particularly convenient:

Properties props = new Properties();

props.put(
    StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
    "exactly_once_v2"
);

Operations involving:

Kafka inputs
Kafka-backed state stores
Kafka outputs
consumer progress

can participate in the processing transaction.

But code like:

stream.peek((key, value) -> {
    stripe.charge(value);
});

does not make the Stripe call transactional.

The Kafka topology may be exactly-once from Kafka's perspective while the external effect still requires its own idempotency strategy.

That distinction should be explicit in production architecture documentation.

Replay Changes What “Once” Means

Kafka deliberately allows consumers to rewind and replay records.

That is one of its strengths.

Suppose an operator resets a consumer group to yesterday and reprocesses:

5 million events

Should the system:

send 5 million emails again?

Probably not.

Should it:

rebuild an analytics projection?

Very possibly.

That tells us something important:

Exactly-once processing cannot mean “this event can never execute again for all time.”

Replay is an intentional capability.

The business layer needs to distinguish:

recompute derived state

from:

repeat an irreversible effect

That may mean replay consumers suppress email, payment, or notification side effects. It may mean historical reprocessing writes into a new projection. It may mean idempotency records survive longer than Kafka retention.

The right behavior depends on what "replay" is supposed to accomplish.

Exactly Once Still Requires Deterministic Processing

There is another subtle boundary.

Suppose a processor receives:

price.updated

and calls an external exchange-rate API.

On the first attempt:

USD/EUR = 0.90

The process crashes.

One minute later, replay produces:

USD/EUR = 0.91

Even if duplicate Kafka output is correctly suppressed, the computation itself may differ because the external state changed between attempts.

If deterministic replay matters, the relevant external input may need to become part of the event:

{
  "orderId": "42",
  "amountUsd": 100,
  "exchangeRate": 0.9,
  "exchangeRateTimestamp": "..."
}

Now the processor can replay the same logical input rather than consulting mutable outside state every time.

Exactly-once processing and deterministic processing are related.

They are not the same guarantee.

Real Systems Compose Multiple Guarantees

A complete event-driven system might look like:

Different boundaries may intentionally use different semantics:

API → Kafka
idempotent request + producer safety

Kafka → Kafka
transactional exactly-once processing

Kafka → PostgreSQL
at-least-once delivery + idempotent DB transaction

Kafka → payment provider
at-least-once delivery + provider idempotency key

Kafka → email
at-least-once, duplicates possible

historical replay
explicitly supported while one-time side effects are suppressed

That can still be an extremely reliable architecture.

The goal is not to force one semantic onto every component.

The goal is to know precisely what each boundary guarantees.

Uber's ad pipeline is a good example

Uber's advertising pipeline is useful because duplicate processing can have financial consequences.

Its architecture combines Kafka and Flink transactional processing with stable record identifiers, downstream deduplication, and idempotent or upsert-style writes.

The important pattern is not:

Kafka exactly once
→ everything solved

It is:

transactional stream processing
+
stable identifiers
+
deduplication
+
idempotent downstream writes

Different layers contribute different guarantees.

Uber's Chaperone auditing system demonstrates the same idea from another direction. The system persisted auditing messages and offsets in a write-ahead log, retried after failure, and used UUID-based deduplication at the receiving side.

Again, the result came from combining mechanisms rather than enabling one global "exactly once" switch.

A Guarantee Matrix Is Better Than Saying “Exactly Once”

If I were reviewing an event-driven architecture, I would much rather see:

than:

Kafka = exactly once

The matrix tells us what happens during failure.

The label does not.

It also makes design disagreements much easier to resolve. Instead of arguing over whether the whole architecture is "exactly once," engineers can ask:

Which boundary is too weak?

Which failure are we protecting against?

What duplicate is unacceptable?

What loss is unacceptable?

Which system owns the idempotency mechanism?

Those are questions architecture can actually answer.

Test the Crash Points

Exactly-once claims should be tested by injecting failure at the uncomfortable moments.

For a Kafka-to-database consumer:

crash after poll

crash during processing

crash after database commit

crash before Kafka offset commit

crash after offset commit

rebalance during processing

database timeout with ambiguous outcome

For Kafka transactional processing:

crash after producing output

crash before transaction commit

abort transaction

restart producer with same transactional identity

read output using read_committed

For an external API:

request reaches provider

provider completes successfully

response disappears

consumer crashes

Kafka redelivers event

That last scenario is especially useful because it quickly exposes the difference between:

Kafka correctness

and:

business correctness

A happy-path integration test can tell you that records flow.

A crash-point test tells you what the system actually guarantees.

Observe Logical Operations, Not Just Kafka Records

Suppose Kafka reports:

1,000,000 consumed records

while the payment provider reports:

1,003,412 calls

That difference matters.

Useful metrics include:

events_consumed_total

events_reprocessed_total

duplicate_events_detected_total

business_operations_applied_total

idempotency_hits_total

transaction_aborts_total

reconciliation_mismatch_total

If the dashboard only shows:

consumer throughput

then a repeated external business effect can hide behind perfectly healthy Kafka metrics.

The observability model should reflect the correctness model.

If the architecture claims one logical payment despite multiple possible deliveries, monitor:

logical payments

separately from:

Kafka attempts

The same applies to database updates, webhooks, emails, or any external operation where redelivery matters.

Stronger Semantics Have a Cost

Kafka transactions introduce coordination overhead.

Transactional processing requires additional protocol work, and a consumer using read_committed cannot expose records belonging to an open transaction until that transaction reaches a committed state.

That means the architecture should not begin with:

Can we enable exactly once?

A better question is:

What failure are we trying to prevent,
and what guarantee does the business require?

For some workloads:

at least once
+
idempotent upsert

is simpler and completely correct.

For others:

Kafka transactional processing

may remove significant application complexity.

For financially sensitive effects, the stronger guarantee may justify considerably more engineering.

For an analytics counter where a very small amount of duplicate tolerance is acceptable, full end-to-end deduplication may not be worth its operational cost.

Correctness should be intentional.

It does not need to be maximal everywhere.

The Invariants I Want

For a Kafka-based system, I care about guarantees like:

Producer retries must not create accidental
duplicate Kafka records.

A Kafka read-process-write transaction must atomically
commit output records and consumed offsets.

Consumers relying on transactional output must not
process aborted transactional records.

A replayed Kafka event must not duplicate irreversible
business effects.

Every external side effect needs an explicit retry
and idempotency strategy.

Database deduplication must commit atomically with
the database mutation it protects.

Business-operation identity must remain stable
across redelivery.

Historical replay must not accidentally repeat
one-time real-world actions.

External nondeterministic state must not silently
change the meaning of replay when deterministic
processing is required.

The architecture must document exactly where
Kafka's transactional guarantees stop.

Those are much stronger than:

processing.guarantee = exactly_once_v2

because configuration describes one component.

The invariants describe the system.

Final Thought

Kafka exactly-once semantics are real.

They are also very easy to overinterpret.

Kafka can solve an exceptionally difficult problem:

consume input
→ compute
→ produce output
→ advance offset

without exposing partially committed Kafka results after failures.

Producer idempotence protects broker writes from retry duplication. Transactions make output records and consumed offsets atomic. read_committed lets downstream consumers honor that transaction boundary. Producer fencing prevents stale instances from continuing to write after ownership changes.

Those are strong guarantees.

Then your program executes:

database.update(...);

or:

stripe.charge(...);

or:

email.send(...);

and you have crossed the boundary.

Kafka cannot roll those operations back. It cannot automatically determine whether they already happened, and it cannot give an arbitrary external service transactional semantics that service does not participate in.

From that point forward, correctness comes from another mechanism:

idempotency
deduplication
unique constraints
upserts
outboxes
inboxes
reconciliation
business state machines

That is not a weakness in Kafka.

It is the nature of distributed systems.

Every guarantee exists inside some boundary.

Experienced system design is largely about knowing exactly where that boundary is.

So when someone says:

“Kafka gives us exactly once.”

the most useful follow-up question is not:

“Which configuration did you enable?”

It is:

“Exactly once… until where?”

That question usually reveals the real architecture.

References

• Apache Kafka — Design and Message Delivery Semantics. Kafka's official documentation describes idempotent production, consumer offsets, transactions, read_committed, and exactly-once semantics for read-process-write workflows that remain inside Kafka.

• Apache Kafka — KafkaProducer API and Producer Configuration. Covers producer idempotence, transactional IDs, transactional recovery, producer fencing, and the producer configuration behind transactional delivery.

• Confluent — Transactions in Apache Kafka. Explains Kafka's atomic read-process-write model, transaction coordination, zombie fencing, transactional overhead, and commit-latency tradeoffs.

• Confluent — Building Systems Using Transactions in Apache Kafka. Describes the transaction boundary directly: Kafka transactions coordinate Kafka operations but do not extend to arbitrary external databases, HTTP calls, or side effects.

• Uber Engineering — Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot. Shows how Kafka and Flink transactional processing can be combined with stable UUIDs, deduplication, and downstream idempotent or upsert behavior.

• Uber Engineering — Introducing Chaperone. Describes an auditing architecture combining a write-ahead log, retries, UUIDs, and downstream deduplication to construct exactly-once auditing behavior on top of at-least-once delivery.