← Blog

Distributed Systems · Kafka · Transactional Outbox

How Do You Commit a Database Change and Publish an Event Without Lying to Either System?

Updating a database and publishing an event looks like one operation in application code, but PostgreSQL and Kafka do not share a transaction. This article explains the dual-write problem, transactional outbox and inbox patterns, idempotent consumers, ordering, CDC with Debezium, reconciliation, sagas, event versioning, and how to make cross-system failure recoverable instead of ambiguous.

Aashaya Aryal

May 6, 2026 · 22 min read

Reflective segmented head sculpture standing in front of a large city building.

An order service processes a successful payment and needs to make two facts visible:

TEXT
1. The order is PAID in PostgreSQL.
2. An OrderPaid event exists in Kafka.

In application code, those operations can sit beside each other and look almost atomic:

JAVA
orderRepository.markPaid(orderId);

kafkaTemplate.send(
    "orders",
    orderId,
    new OrderPaid(orderId)
);

The code is adjacent.

The commits are not.

PostgreSQL can successfully commit the order update and then the process can die before Kafka receives anything. Reverse the order and Kafka can receive OrderPaid before the database transaction ever commits. In either case, both systems can be behaving correctly according to their own rules while the distributed system tells two different stories about the same payment.

That is the dual-write problem.

It is tempting to treat this as an ordering question:

Should I publish before or after the database commit?

Neither ordering solves it. The real problem is that two independent systems are being asked to participate in one logical operation without sharing one atomic transaction.

A better question is:

How can the database transaction durably record that an event must eventually be published?

That is where the transactional outbox pattern begins.

Two Correct Systems Can Still Disagree

Consider an order service with PostgreSQL and Kafka:

Diagram

Order API

PostgreSQL

Kafka

Fulfillment

Email Service

PostgreSQL has transactions.

Kafka has its own durability and transaction mechanisms.

Each system may be perfectly healthy, but this logical operation:

TEXT
PostgreSQL commit
+
Kafka publish

is not automatically one transaction.

That leaves four possible states:

DatabaseKafkaMeaning
Not updatedNo eventNothing happened
UpdatedEvent publishedDesired state
UpdatedNo eventDownstream misses a real state change
Not updatedEvent publishedDownstream believes a state change that never committed

The last two states are the dangerous ones.

Commit first, publish second

Suppose the database commits first:

TEXT
UPDATE order = PAID
        ↓
COMMIT
        ↓
process dies
        ↓
Kafka publish never happens

PostgreSQL says:

TEXT
order = PAID

while downstream systems say:

TEXT
we never heard about it

That can mean fulfillment never starts, analytics never sees the payment, inventory-related work never happens, or the customer never receives the notification expected from the event.

The local database is correct.

The distributed workflow is incomplete.

Publish first, commit second

Reverse the order:

TEXT
Kafka publish succeeds
        ↓
process dies
        ↓
database update never commits

Downstream services receive:

TEXT
OrderPaid

while the source of truth still says:

TEXT
PENDING

We did not eliminate the inconsistency.

We merely chose its opposite direction.

AWS describes the same failure boundary in its transactional outbox guidance: independently updating a database and notifying another system creates a window where one operation can succeed while the other does not.

The architecture therefore needs either to make the inconsistent states impossible or to make them durably detectable and repairable.

Durable Intent Is the Missing Piece

A natural response is to retry the Kafka publish:

JAVA
try {
    kafkaTemplate.send(event);
} catch (Exception e) {
    retry();
}

Retries are useful when Kafka is temporarily unavailable or the network fails.

They do not solve the crash gap.

Consider:

TEXT
database commit succeeds
        ↓
process dies before retry state is persisted

After the service restarts, what durable state says:

TEXT
Order 42 still needs an event.

Nothing does.

The retry intention existed only in application memory.

That is the key insight behind the outbox:

The fact that an event still needs to be published must itself be durable.

Instead of trying to make PostgreSQL and Kafka commit together, record the future publication inside the transaction PostgreSQL already knows how to make atomic.

Put the Event Inside the Database Transaction

Instead of:

TEXT
transaction:
    update order

after transaction:
    publish Kafka event

use:

TEXT
transaction:
    update order
    insert outbox event

Both writes now belong to the same database transaction.

Diagram

One Database Transaction

Application

orders

outbox_events

A simplified outbox table might be:

SQL
CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(150) NOT NULL,
    payload JSONB NOT NULL,
    aggregate_version BIGINT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);

Then the service can perform the business mutation and create the outgoing event together:

JAVA
@Transactional
public void markOrderPaid(
    UUID orderId,
    Payment payment
) {
    Order order =
        orderRepository.findById(orderId)
            .orElseThrow();

    order.markPaid(payment.id());

    orderRepository.save(order);

    OutboxEvent event =
        OutboxEvent.create(
            UUID.randomUUID(),
            "order",
            orderId.toString(),
            "OrderPaid",
            order.version(),
            serialize(
                new OrderPaid(
                    orderId,
                    payment.id(),
                    order.version()
                )
            )
        );

    outboxRepository.save(event);
}

The database can now guarantee:

TEXT
order update commits
AND
outbox event commits

or:

TEXT
neither commits

AWS's transactional outbox guidance describes exactly this structure: the business state change and event record participate in the same local database transaction.

The important improvement is not that Kafka suddenly became transactional with PostgreSQL.

It did not.

The improvement is that a successful database commit can no longer leave the system with no durable memory that publication still needs to happen.

Publication Becomes Recoverable Work

After the database transaction commits, the state might be:

TEXT
orders:
    order_42 = PAID

outbox_events:
    evt_892 = OrderPaid(order_42)

Kafka may still know nothing about the event.

That is okay.

The guarantee we wanted is:

If the database says the business change happened, durable evidence also exists saying the corresponding event must be published.

A separate process can perform that delivery later.

Diagram

one transaction

same transaction

Order Service

Business Tables

Outbox

Outbox Relay

Kafka

Consumers

Publishing is no longer a fragile continuation of the original request.

It is durable work.

A polling relay is the simplest version

A relay can periodically read unpublished rows:

SQL
SELECT *
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;

Then publish them:

TS
for (const event of events) {
  await kafka.send({
    topic: topicFor(event.eventType),
    messages: [
      {
        key: event.aggregateId,
        value: event.payload,
      },
    ],
  });

  await markPublished(event.id);
}

The lifecycle is easy to reason about:

TEXT
created
→ pending
→ published
→ cleaned up later

If Kafka is unavailable, the rows remain in the database.

When Kafka recovers, the relay continues.

The outbox has turned the database into a durable handoff point between the synchronous business transaction and asynchronous messaging.

Parallel relays need claiming

One relay may eventually become a throughput bottleneck.

If several workers all execute:

SQL
SELECT *
FROM outbox_events
WHERE published_at IS NULL
LIMIT 100;

they can select the same rows.

A PostgreSQL pattern such as:

SQL
SELECT *
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

lets workers divide available rows while skipping rows another transaction has already claimed.

Diagram

Outbox

Relay 1

Relay 2

Relay 3

Kafka

This scales the relay.

It still does not make publication exactly once.

There is another crash window.

The Outbox Prevents Loss, Not Duplication

Consider the relay sequence:

TEXT
1. Read evt_892.
2. Publish evt_892 to Kafka.
3. Kafka acknowledges.
4. Relay crashes.
5. published_at was never updated.

After restart:

TEXT
evt_892 still looks unpublished

so the relay publishes it again.

Diagram
KafkaRelayOutboxKafkaRelayOutboxCrash before DB updateevt_892Publish evt_892ACKevt_892 againPublish evt_892 again

The outbox solved:

TEXT
event permanently lost

It did not automatically solve:

TEXT
event published more than once

AWS explicitly warns about duplicate publication in outbox implementations and recommends idempotent consumers.

The practical delivery contract is therefore usually:

TEXT
at least once

rather than:

TEXT
exactly once everywhere

That is not a failure of the pattern.

It is the intended trade: prefer a duplicate that can be detected over a state change whose event was silently lost forever.

Implementation note

An outbox moves the architecture from an ambiguous dual-write failure to durable retry. Downstream idempotency is what makes that retry safe.

Idempotency Has Two Identities

Every outbox event should have a stable event identifier:

TEXT
evt_892

That identifier must survive every publication retry.

Do not generate a new ID each time the relay attempts delivery.

A message might look like:

JSON
{
  "eventId": "evt_892",
  "eventType": "OrderPaid",
  "aggregateId": "order_42",
  "aggregateVersion": 7,
  "payload": {
    "orderId": "order_42",
    "paymentId": "pay_81"
  }
}

A consumer can now recognize:

TEXT
evt_892 already processed

instead of treating each broker delivery as a distinct business event.

The inbox pattern complements the outbox

A consumer can persist processed event IDs:

SQL
CREATE TABLE processed_events (
    consumer_name VARCHAR(100) NOT NULL,
    event_id UUID NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),

    PRIMARY KEY (
        consumer_name,
        event_id
    )
);

Then combine the dedupe marker with the consumer's business mutation in one local transaction:

JAVA
@Transactional
public void handle(OrderPaid event) {

    boolean inserted =
        inboxRepository.insertIfAbsent(
            "fulfillment",
            event.eventId()
        );

    if (!inserted) {
        return;
    }

    fulfillmentRepository.createFor(
        event.orderId()
    );
}

The important part is not the table name.

It is that:

TEXT
processed-event marker
+
business mutation

commit atomically.

Now:

TEXT
delivery 1
delivery 2
delivery 3

can all converge to:

TEXT
one fulfillment record

This gives us a useful producer-consumer pairing:

TEXT
producer:
transactional outbox

consumer:
transactional inbox / idempotent effect

Event identity is not always business identity

Suppose the consumer receives:

TEXT
evt_A → RefundRequested(refund_42)
evt_B → RefundRequested(refund_42)

Those are different events.

They may still describe the same business operation.

If the consumer deduplicates only on:

TEXT
event_id

both requests can execute.

For irreversible or financial effects, the system may also need a stable business identifier such as:

TEXT
refund_42
payment_82
invoice_2026_1993

A database can enforce that identity directly:

SQL
CREATE UNIQUE INDEX uniq_refund_operation
ON refunds(external_refund_id);

These mechanisms answer two different questions:

TEXT
Event deduplication:
Have I processed this transport event?

Business idempotency:
Have I already performed this logical action?

Strong systems often need both.

Ordering Needs an Explicit Contract

Reliable delivery does not automatically imply useful ordering.

Suppose one order generates:

TEXT
v5 OrderPaid
v6 OrderCancelled

but downstream receives:

TEXT
OrderCancelled
→ OrderPaid

A naïve consumer can finish in the wrong state.

AWS's outbox guidance calls out ordering as a separate concern for exactly this reason.

A useful event envelope can contain:

TEXT
aggregate_id
aggregate_version

For example:

JSON
{
  "aggregateId": "order_42",
  "aggregateVersion": 7
}

Using:

TEXT
Kafka key = aggregateId

allows events for the same aggregate to map to the same Kafka partition.

Diagram

order_42 v5

order_42 v6

order_77 v3

Kafka Partition 1

Kafka Partition 2

Kafka can then preserve ordering within the partition.

That does not provide:

TEXT
one global order for every event

It provides something usually more useful:

TEXT
ordered changes for order_42

Consumers can defend against stale versions

Suppose a consumer already stores:

TEXT
order_42 version = 7

and then receives:

TEXT
version = 6

It can ignore the stale transition:

TS
if (event.aggregateVersion <= current.version) {
  return;
}

A version gap can also be meaningful:

TEXT
current = 5
incoming = 7

That suggests version 6 may be missing or delayed.

Depending on the architecture, the consumer can wait, re-fetch canonical state, or reconcile.

The important point is that ordering correctness should not depend entirely on hoping every delivery path behaves perfectly forever.

Make the contract observable in the event itself.

Polling and CDC Solve the Same Handoff Differently

Polling is straightforward, but it repeatedly asks:

TEXT
Anything new?
Anything new?
Anything new?

Another option is change data capture.

A CDC system reads committed database changes from the database log and turns outbox inserts into downstream messages.

Diagram

business + outbox transaction

Application

PostgreSQL

WAL / Change Log

Debezium

Kafka

Debezium provides an outbox event router for exactly this style of architecture. The application still performs:

TEXT
business mutation
+
outbox insert

inside one database transaction.

Debezium handles the relay from committed outbox changes into Kafka.

CDC moves complexity rather than deleting it

Application-level polling introduces:

TEXT
poll scheduler
claiming
relay workers
publisher checkpoints

CDC introduces different infrastructure:

TEXT
database WAL / transaction log
Debezium connector
connector offsets
Kafka Connect
connector configuration

That may be a very good trade, particularly when the organization already operates CDC infrastructure.

It is still a trade.

The right question is not:

Which approach is more modern?

It is:

Which delivery mechanism can we operate, observe, recover, and replay confidently?

CDC does not require publishing raw database mutations

CDC creates a temptation to expose every row update directly:

JSON
{
  "before": {
    "status": "PENDING"
  },
  "after": {
    "status": "PAID"
  }
}

Sometimes that is exactly the desired contract.

Often it leaks the physical database model into every downstream consumer.

A semantic outbox event says:

TEXT
business fact:
OrderPaid

instead of:

TEXT
database fact:
orders.status changed from PENDING to PAID

That difference matters.

The first contract describes the domain.

The second exposes the producer's current storage representation.

Debezium's outbox router supports the semantic approach: the application writes an explicit event envelope, and CDC transports that event rather than asking consumers to infer business meaning from arbitrary row mutations.

The Outbox Becomes Production Infrastructure

Once important transactions create outbox rows, the table stops being a temporary implementation detail.

It participates in:

TEXT
write throughput
indexes
storage growth
replication
vacuum or compaction
publisher throughput
retention
cleanup
recovery

Suppose the system creates:

TEXT
10,000 events/sec

If cleanup stops, that becomes:

TEXT
864 million rows/day

The outbox cannot be treated as:

TEXT
a little temporary table

just because rows are eventually eligible for removal.

Monitor age, not only row count

Suppose:

TEXT
100,000 unpublished events

exist.

Is that bad?

Not necessarily.

If the relay normally publishes:

TEXT
200,000 events/sec

the backlog could disappear almost immediately.

If the oldest of those rows has been waiting:

TEXT
45 minutes

the situation is very different.

Useful telemetry includes:

TEXT
outbox_pending_count
outbox_oldest_pending_age
outbox_publish_rate
outbox_creation_rate
outbox_publish_failures
outbox_duplicate_publications

I would also watch the relationship between:

TEXT
creation rate
and
publish rate

because backlog is fundamentally a rate imbalance.

A queue-like system with a large count but positive drain capacity can be healthy.

A smaller queue whose oldest item keeps aging may be stuck.

The relay is allowed to fall behind

One of the benefits of the pattern is deliberate decoupling.

If Kafka is unavailable for 30 seconds, the business transaction can still commit:

TEXT
database transaction succeeds
→ event waits durably
→ Kafka recovers
→ relay catches up

That can be much better than turning every temporary broker outage into an order-creation outage.

But durable buffering is not infinite capacity.

The database needs enough space and write headroom to absorb realistic interruptions, and operators need to know how long the relay can remain unavailable before backlog becomes dangerous.

Cleanup is part of the design

After publication, rows might be:

TEXT
deleted immediately
retained temporarily
archived
partitioned by time
moved into audit history

Deleting aggressively keeps the active table smaller.

Retaining rows can make debugging and reconciliation much easier.

The right policy depends on:

TEXT
event volume
audit requirements
database capacity
replay strategy
recovery needs

At high volume, time-based partitioning can make cleanup safer than continuously issuing very large delete workloads.

The outbox has a lifecycle.

Design it deliberately.

Kafka Transactions Do Not Remove the Database Boundary

It is natural to ask why Kafka transactions do not solve this directly.

Kafka transactions can atomically coordinate operations such as:

TEXT
consume Kafka record
+
produce Kafka records
+
commit Kafka consumer offsets

when the relevant work remains inside Kafka's transactional boundary.

They do not automatically make:

TEXT
PostgreSQL COMMIT

part of the same transaction.

So:

TEXT
Kafka transaction
+
PostgreSQL transaction

still gives you two independently committing systems.

The dual-write boundary remains.

Confluent's transaction documentation is explicit about this distinction: Kafka transactional semantics do not automatically extend to arbitrary external databases or HTTP services.

What about two-phase commit?

Traditional distributed transaction protocols such as two-phase commit can coordinate participating resources:

TEXT
coordinator:
    prepare database
    prepare second resource

    if both prepared:
        commit both

That provides stronger atomic coordination when every participant supports the same protocol.

Many modern brokers, cloud services, NoSQL systems, and third-party APIs do not participate in one shared XA-style transaction, and AWS specifically presents transactional outbox as an alternative when distributed two-phase commit is unavailable or undesirable.

Even where distributed transactions are available, they introduce tighter coordination among participants and more complicated recovery behavior.

For many event-driven systems, the preferred model becomes:

TEXT
local ACID transaction
+
durable event intent
+
asynchronous publication
+
idempotent consumption

It deliberately gives up instantaneous cross-system atomicity in exchange for recoverability and looser coupling.

Outbox and Saga Solve Different Problems

Now extend the workflow:

TEXT
Order Service
→ Payment Service
→ Inventory Service
→ Shipping Service

Each service owns its own state.

An outbox can ensure that each service reliably publishes the state transitions that commit inside its own database.

It does not make the entire workflow one ACID transaction.

Diagram

Order Created

Payment Authorized

Inventory Reserved

Shipment Created

Suppose payment authorization succeeds and inventory reservation later fails.

Rolling back the order service's PostgreSQL transaction cannot undo:

TEXT
external payment authorization

The workflow needs a compensating business action:

TEXT
release authorization

That is saga territory.

AWS's guidance makes the same distinction: outbox solves reliable communication of a local committed state change, while saga-style coordination is relevant when one business transaction spans independently owned services.

Outbox solves local publication consistency. Saga solves multi-service business consistency over time.

They are complementary patterns.

Compensation is not rollback

Suppose:

TEXT
charge customer
→ inventory reservation fails
→ refund customer

The final balance may look similar to a world in which no charge occurred.

The history is not the same.

These are two facts:

TEXT
payment charged
payment refunded

That difference matters for:

TEXT
audit
notifications
financial reporting
support
compliance

Distributed workflows often move forward through compensation rather than pretending the past never happened.

Reconciliation Is the Safety Net

Even with:

TEXT
outbox
idempotency
CDC
Kafka
sagas

I still want reconciliation.

Why?

Because sufficiently complex systems eventually fail in ways the original design did not enumerate.

Suppose:

TEXT
orders marked PAID = 10,000
OrderPaid events observed = 9,998

The architecture should be able to answer:

TEXT
Which two?

and then repair them.

Stripe's Ledger work is a strong example of treating correctness as more than preventing bad writes. Stripe describes immutable records of money movement along with completeness checks that compare expected producer state with the events Ledger actually received.

That suggests a broader reliability principle:

Correctness means not only preventing divergence, but also detecting, explaining, and repairing divergence when prevention fails.

Reconciliation can be simple

Suppose every paid order should have one corresponding outbox event.

A periodic query can check:

SQL
SELECT o.id
FROM orders o
LEFT JOIN outbox_events e
    ON e.aggregate_id = o.id::text
   AND e.event_type = 'OrderPaid'
WHERE o.status = 'PAID'
  AND e.id IS NULL;

The expected result is:

TEXT
0 rows

Anything else can trigger:

TEXT
alert
→ investigate
→ repair

The same principle can continue downstream:

TEXT
paid orders
vs.
fulfillment records

or:

TEXT
refund records
vs.
payment-provider refund IDs

Now the architecture is not merely reliable in theory.

It is auditable.

Repair must be safe

If reconciliation discovers:

TEXT
order_42 = PAID
but no OrderPaid was observed

the system needs a repair path.

That repair may involve re-emitting an event.

If replaying that event can accidentally charge a customer twice or create duplicate fulfillment, the system is not really repairable.

This is why stable event IDs and stable business operation IDs matter operationally, not just academically.

A mature system should be able to say:

TEXT
we can safely replay this

without fear that recovery will create a second incident.

Events Are Durable Contracts

Outbox rows may survive deployments.

That means producers and consumers from different application versions can coexist with the same backlog.

Suppose version 1 writes:

JSON
{
  "eventType": "OrderPaid",
  "orderId": "42"
}

while version 2 expects:

JSON
{
  "eventType": "OrderPaid",
  "orderId": "42",
  "paymentId": "pay_11",
  "currency": "USD"
}

If a publication backlog survives the deployment, the system may contain:

TEXT
old event envelopes
+
new consumer code

at the same time.

The event schema therefore needs the same compatibility discipline as a database schema or public API.

A useful envelope might contain:

JSON
{
  "eventId": "evt_892",
  "eventType": "OrderPaid",
  "schemaVersion": 2,
  "aggregateId": "order_42",
  "aggregateVersion": 7,
  "occurredAt": "2026-10-02T23:43:00Z",
  "payload": {}
}

Events are durable contracts.

Treat them that way.

Event time and publish time are different

Suppose an event occurs at:

TEXT
10:00:00

but Kafka is unavailable and the event is finally published at:

TEXT
10:02:30

Those timestamps mean different things:

TEXT
occurred_at
=
when the business fact happened

published_at
=
when transport completed

Collapsing them into one value can make downstream analytics interpret delivery delay as business timing.

The outbox model naturally gives the system a place to preserve both.

A good outbox event describes a business fact

I prefer:

TEXT
OrderPaid
InvoiceIssued
ShipmentCreated
SubscriptionCancelled

over:

TEXT
orders_row_updated

because the first set tells consumers what happened without exposing how the producer currently stores it.

That separation preserves service autonomy.

The database schema can evolve without forcing every downstream consumer to understand every physical column.

Commands and events are different

A command such as:

TEXT
PayOrder

means:

Please attempt this action.

An event such as:

TEXT
OrderPaid

means:

This action already happened.

That distinction matters enormously.

An event describing a committed fact should not escape before the corresponding source-of-truth transition has actually committed.

The outbox aligns naturally with that rule:

TEXT
commit business fact
+
record event describing that fact

inside the same transaction.

The Local Atomicity Boundary

The most useful mental model for the pattern is this:

Diagram

Local ACID Transaction

Business State

Outbox Event

Relay / CDC

Kafka

Consumer

Consumer Database

Inside the transaction boundary:

TEXT
atomic

Outside it:

TEXT
asynchronous
retryable
potentially duplicated
eventually consistent

Once that boundary is explicit, the architecture becomes much easier to reason about.

The outbox does not automatically guarantee:

TEXT
exactly-once delivery
exactly-once business effects
global ordering
instant publication
downstream availability
correct event schemas
infinite backlog capacity
cross-service ACID rollback

It gives us something narrower and extremely useful:

If the business transaction commits, durable evidence of the event exists in that same transaction.

Everything after that still needs engineering.

Failure matrix

I like evaluating the design by crash point:

FailureIntended result
Crash before DB transactionNothing committed
Business write failsOutbox insert rolls back
Outbox insert failsBusiness mutation rolls back
Crash immediately after commitOutbox row remains for later publication
Kafka unavailableOutbox backlog accumulates
Relay publishes then crashesDuplicate publication may occur
Consumer crashes after processingEvent may be redelivered
Duplicate arrivesConsumer handles it idempotently
Events arrive lateVersions and timestamps preserve meaning
Relay stops for hoursBacklog remains observable and drainable

If every row has an intentional answer, the system is becoming much more trustworthy.

A Practical Architecture

For a Spring, PostgreSQL, and Kafka service, a practical design could look like:

Diagram

HTTP Request

Spring Service

PostgreSQL

orders

outbox_events

Debezium / Outbox Relay

Kafka

Consumer Service

processed_events

Consumer State

The core guarantees would be:

TEXT
Business mutation and outbox insert commit together.

A committed business mutation always leaves
durable event intent.

Publishing may happen more than once.

Every event has a stable event ID.

Every irreversible business operation has
a stable business identity.

Consumers tolerate redelivery.

Ordering-sensitive aggregates carry explicit versions.

Outbox backlog and oldest pending age are monitored.

Reconciliation detects missing downstream state.

Repair and replay are safe.

That is substantially stronger than:

JAVA
save();
publish();

even though the latter looks much simpler.

The extra machinery is not ceremony.

It exists because the simple version depends on there never being a failure between two independent commits.

Distributed systems do not offer that promise.

When an Outbox Is Not Worth It

The outbox is a useful pattern.

It is not a mandatory accessory for every service.

Do not add:

TEXT
outbox
Kafka
CDC
saga
reconciliation

to a trivial workflow merely because the architecture uses microservices.

If a synchronous call satisfies the requirement and asynchronous decoupling adds no meaningful value, the simpler system may be better.

If the requirement is:

TEXT
write business data
+
emit best-effort analytics

and losing an occasional analytics event is explicitly acceptable, a transactional outbox may cost more than the guarantee is worth.

Likewise, if the source platform already exposes a trustworthy change-stream mechanism that provides the semantics the system actually needs, a custom relay may be unnecessary.

Reliability architecture can become complexity architecture when patterns are added without a corresponding failure requirement.

Use the outbox when the consistency invariant justifies it:

TEXT
If this business state commits,
the corresponding event must not be forgotten.

That is a strong and specific reason.

The Invariants I Want

For database-to-event consistency, I care about guarantees like:

TEXT
A committed business state transition must always
leave durable evidence that its event needs publishing.

A rolled-back transaction must not emit an event
describing a state that never existed.

Relay failure must not permanently lose publication intent.

Duplicate publication must not create duplicate
business effects.

Event identity must remain stable across retries.

Business-operation identity must survive redelivery
when the real-world action must happen only once.

Ordering-sensitive aggregates must carry explicit
ordering or version information.

Outbox backlog must remain bounded, observable,
and recoverable.

Event schemas must remain compatible across
deployments and publication backlogs.

Cross-service workflows must use explicit
compensation rather than pretending a local rollback
can undo remote actions.

The system must be able to reconcile source-of-truth
state against emitted and consumed events.

Repair and replay must be safe enough to use
during an incident.

Those are the guarantees that matter.

Not:

TEXT
Kafka send returned success.

Final Thought

The dual-write problem exists because application code makes two independent systems look closer together than they really are:

TEXT
database.save()
kafka.send()

The lines are adjacent.

The transactional boundaries are not.

Between them can live:

TEXT
process crash
network failure
broker outage
timeout
deployment
machine restart

Any one of those failures can split reality.

Publishing first does not solve it.

Publishing second does not solve it.

Retrying from memory does not solve it.

Kafka transactions do not automatically make an unrelated PostgreSQL commit part of Kafka's transaction.

The key move is different:

Do not require the database and broker to commit together. Require the database transaction to durably record what must eventually be published.

That transforms:

TEXT
two independent atomic writes
to two different systems

into:

TEXT
one atomic local transaction
+
one recoverable asynchronous delivery

The price is eventual consistency.

The benefit is that failure becomes replayable rather than ambiguous.

Once publication becomes asynchronous, duplicates become part of the contract, so consumers need idempotency. Ordering needs identity and versions. Backlogs need capacity planning and monitoring. Multi-service workflows need compensation. Critical systems still need reconciliation.

That may sound like more machinery than:

JAVA
save();
publish();

It is.

The machinery exists because the simple version was relying on something the distributed system could never guarantee: that nothing would fail between two independent commits.

The transactional outbox does not eliminate failure.

It does something more useful.

It makes the failure recoverable without lying about which state actually committed.

References

  • AWS Prescriptive Guidance — Transactional outbox pattern. Defines the dual-write inconsistency problem and describes storing business changes and outgoing events in one database transaction. It also discusses duplicate publication, ordering, idempotent consumers, and saga coordination.

  • AWS Compute Blog — Implementing the transactional outbox pattern. Walks through how independent database and broker writes can diverge and uses an outbox as the durable handoff between business state and asynchronous messaging.

  • Debezium — Outbox Event Router. Documents a CDC-based implementation in which Debezium captures committed outbox-table changes and converts them into messages for downstream consumers.

  • Confluent — Building Systems Using Transactions in Apache Kafka. Explains the scope of Kafka transactions: Kafka inputs, outputs, and offsets can participate in the transaction, but arbitrary external databases and HTTP services do not automatically join it.

  • Stripe Engineering — Ledger: Stripe's system for tracking and validating money movement. Demonstrates the complementary role of reconciliation through immutable records, completeness checks, and tooling that makes discrepancies detectable and explainable.