Why Can a Cassandra Data Model Work for Months and Then Collapse at the Tail?

A Cassandra table can look excellent when it launches. Writes are fast, median reads are fast, nodes appear balanced, CPU is moderate, and disk usage looks perfectly reasonable.

Then months pass.

A few partitions become enormous. One customer becomes dramatically more active than everyone else. Deletes and TTL expirations accumulate. Compaction begins falling behind. One replica develops much worse latency than its peers. Average request latency still looks acceptable, but the tail starts moving:

p99

40 ms
→ 150 ms
→ 800 ms
→ several seconds

Adding nodes may barely help. The cluster can still have plenty of aggregate CPU, memory, and disk while a small number of partitions produce most of the pain.

The problem is not necessarily that Cassandra suddenly became slower. The workload may simply no longer resemble the distribution the original schema assumed.

That is one of the most important things I have learned to keep in mind with Cassandra:

A data model can be correct for today's access pattern and structurally wrong for tomorrow's data distribution.

Cassandra does not hide modeling mistakes behind a powerful general-purpose query planner. The partition key is an architectural decision. Clustering layout is an architectural decision. TTL and deletion behavior are architectural decisions. Over time, those choices determine where data accumulates, which replicas receive traffic, how many SSTables a read may touch, how many tombstones must be scanned, how expensive compaction becomes, and which requests dominate the tail.

I have worked with Cassandra as part of distributed-data systems, but the large-scale failure modes discussed here are grounded in published engineering work from teams including Discord, Uber, Netflix, and the Apache Cassandra project itself.

The interesting question is therefore not merely:

“How do I model a Cassandra table?”

It is:

“What happens to this model after years of skew, retention, deletion, compaction, repair, and growth?”

Cassandra Makes Data Shape an Architectural Decision

Cassandra's write path is one of its strengths. A write is generally appended to the commit log and applied to an in-memory memtable. Later, the memtable is flushed to an immutable SSTable on disk.

Conceptually:

The system does not need to rewrite existing disk pages in place for every update.

Reads have a different cost profile.

The newest logical value may need to be reconstructed from information spread across:

memtable
+
SSTable A
+
SSTable B
+
SSTable C

Cassandra may need to inspect and reconcile several versions before it can return the current result. Apache Cassandra's documentation explains that one purpose of compaction is reducing this read amplification by merging SSTables and discarding obsolete versions when it is safe to do so.

That creates an important long-term property:

The cost of a Cassandra read depends partly on the history of writes, updates, deletions, expiration, and compaction that produced the current storage state.

A query can therefore be cheap today and expensive later even when the CQL itself has not changed.

That is why Cassandra modeling is not just about satisfying a query syntactically. It is about controlling the physical shape that query will encounter after the data has aged.

Token Balance Is Not Workload Balance

Suppose messages are stored like this:

CREATE TABLE messages_by_channel (
    channel_id UUID,
    bucket_date DATE,
    message_id TIMEUUID,
    author_id UUID,
    content TEXT,

    PRIMARY KEY (
        (channel_id, bucket_date),
        message_id
    )
);

The partition key is:

(channel_id, bucket_date)

That looks reasonable. Messages for one channel and one time bucket remain together, and a query such as:

SELECT *
FROM messages_by_channel
WHERE channel_id = ?
  AND bucket_date = ?
ORDER BY message_id DESC
LIMIT 50;

fits the model directly.

Now compare two channels.

The partitioning function can distribute their partition keys evenly across the token ring.

It cannot make the workloads represented by those keys equally large.

That distinction matters enough to state plainly:

Token balance is not workload balance.

Consistent hashing helps distribute many logical partitions across a cluster. It does not split one unusually hot logical partition into many independent workloads simply because that partition receives far more reads or writes than everyone expected.

A cluster can therefore look beautifully balanced in aggregate while one replica set is doing dramatically more useful work than another.

Hot Partitions Turn Logical Skew Into Noisy Neighbors

Imagine four Cassandra nodes:

Storage might be reasonably balanced while request volume looks more like:

Node 1:  10k reads/s
Node 2: 190k reads/s
Node 3:  12k reads/s
Node 4:  11k reads/s

The cluster has spare aggregate capacity.

Node 2 does not.

Discord encountered this kind of problem in its Cassandra-backed message storage. Messages were partitioned using a channel identifier plus a time bucket, but popular channels generated dramatically more activity than ordinary ones. Discord described these as hot partitions: intense access to one channel and bucket could increase latency on the replicas responsible for that partition and also affect unrelated queries sharing those machines.

That is why:

add more nodes

does not automatically solve:

one logical partition receives
too much traffic

The cluster may gain aggregate capacity while the hot partition still maps to one replica set.

Quorum can widen the latency surface

Now suppose:

RF = 3

and reads use:

QUORUM

Conceptually:

The exact latency behavior depends on the read path, consistency level, speculation, and which replicas satisfy the request. The architectural point is that replicated reads create dependencies across several machines.

Discord noted that its quorum usage made replica health relevant to user-visible latency: hot-partition pressure on the nodes serving that data could affect the request surface more broadly.

Consistency level is therefore not only about correctness semantics.

It also influences the shape of latency.

A hot partition can slow cold partitions

The noisy-neighbor effect is easy to underestimate.

You might reasonably think:

One celebrity channel is hot, so that channel will be slow.

But a node serves many partitions:

If one hot partition consumes disproportionate:

CPU
disk I/O
request concurrency
heap
I/O queue capacity

then unrelated partitions on the same node compete with it.

Logical skew has become physical noisy-neighbor behavior.

Wide Partitions Age Differently Than Small Ones

Traffic skew is one failure mode.

Partition growth is another.

A partition that begins at:

10 MB

may eventually reach:

500 MB

or several gigabytes depending on the schema and workload.

Large partitions can complicate:

read amplification
compaction
repair
streaming
backup and migration tooling
memory pressure

Apache Cassandra's leveled-compaction documentation highlights how fundamental the partition boundary is: a sufficiently large partition can force an SSTable beyond the nominal target size because Cassandra does not split that partition across separate SSTables merely to satisfy an SSTable sizing target.

Once the partition key is chosen, many later storage operations inherit that decision.

This matters even when normal online reads still look acceptable. A partition size that barely affects one paginated application query can become painful during repair, migration, streaming, backup processing, or compaction.

Netflix's recent Cassandra data-movement work is a good example. The workload was not the normal application serving path, yet skewed and wide partitions still caused operational problems, including out-of-memory failures in an older processing model for some large datasets. The replacement architecture moved partition-sensitive work into a more scalable execution model so processing could better follow actual partition shape.

The partition is therefore not just a serving concern.

It propagates into almost every operational system that has to touch the data later.

Bucketing Is a Prediction About Future Distribution

A common solution to wide partitions is time bucketing.

Instead of:

partition = device_id

use:

device_id + day

or:

device_id + hour

That bounds how much time-series data accumulates in one partition.

It also quietly makes a prediction:

One day—or one hour—is an appropriate bucket size for every device.

Suppose:

Device A
20 measurements/day

while:

Device B
50,000,000 measurements/day

The same time bucket produces radically different physical partitions.

A larger bucket risks wide partitions.

A smaller bucket reduces partition size but increases partition count and forces longer-range queries to fan out across more keys.

Cassandra modeling is therefore not merely:

partition or not?

It is often:

Where is the useful tradeoff
between partitions that are too wide
and queries that are too fragmented?

Static bucketing works when the workload distribution stays reasonably close to the assumptions used to choose the bucket.

Growth can invalidate those assumptions.

Time bucketing trades size for fan-out

Suppose:

partition = customer_id + day

A one-day query fits one partition.

A 30-day query may need:

30 partitions

A one-year query may need:

365 partitions

The partition-size problem has been bounded by moving complexity into the read path.

Hash bucketing trades hot writes for fan-out

Another option is:

partition =
customer_id + hash(event_id) % 16

Writes can now spread across sixteen partitions rather than concentrating on one key.

Good.

But reading all events for that customer may now require:

16 partition reads

followed by merging or ordering results.

We solved:

single hot write partition

by introducing:

read fan-out

That might be an excellent trade.

It is still a trade.

The familiar Cassandra rule:

model around your queries

is useful, but incomplete.

A stronger version is:

Model around your queries and the future statistical distribution of the data those queries will touch.

That second half is where long-lived problems often begin.

Deletes Become Tombstones Before They Become Free Space

Now add deletion.

A developer might expect:

DELETE FROM messages
WHERE ...;

to mean:

the data disappears

A replicated distributed database cannot safely make deletion that simple.

Cassandra records a tombstone: timestamped metadata representing the fact that a value was deleted.

Conceptually:

Why keep metadata for something that no longer exists?

Because one replica may have been unavailable during the deletion.

If the healthy replicas simply forgot the value immediately while an unavailable replica still retained the old version, that stale copy could later reappear.

The tombstone is therefore not merely garbage.

It is correctness metadata.

Tombstones make deletion part of the read path

Suppose a partition once contained:

1,000,000 rows

and most of them later expire through TTL or are deleted.

A query asks for the newest 50 live rows.

Logically, the system needs to return:

50 rows

but the storage engine may need to inspect far more dead state while determining what remains visible.

The table may contain relatively little useful live data while still having an expensive read path because the history of deletion remains encoded in its SSTables.

Apache Cassandra documents that tombstones persist until they can be safely purged through compaction. TTL expiration produces tombstones too.

That means:

TTL is not free cleanup.

A TTL policy creates future deletion and compaction work.

Discord found this at the end of a migration

One of the more memorable details from Discord's Cassandra migration appeared when the migration had reached roughly:

99.9999%

and then stalled.

The remaining token ranges contained extremely tombstone-heavy partitions that Cassandra had not yet compacted away. Discord had to compact those ranges before the migration could finish.

That is a useful illustration of how long-term storage history can remain hidden until a different operational workload touches the entire dataset.

The data model had worked for years.

The tombstones did not necessarily make every normal application request fail.

But historical storage debt still existed, waiting for an operation that could no longer avoid it.

Compaction Pays the Deferred Storage Bill

Because SSTables are immutable, updates and deletions create newer information instead of rewriting all older files in place.

Over time:

SSTable 1
SSTable 2
SSTable 3
SSTable 4
SSTable 5
...

may all contain pieces relevant to a logical read.

Compaction merges them:

Apache Cassandra describes two major benefits:

reduce read amplification
+
reclaim obsolete versions and tombstones

Fewer SSTables need to be consulted, and obsolete storage can eventually be reclaimed when correctness rules allow it.

The catch is that compaction itself consumes:

CPU
disk I/O
temporary disk space

Cassandra therefore performs continuous background work simply to keep foreground reads healthy.

When compaction falls behind

Suppose write throughput rises and SSTables arrive faster than compaction can process them.

The system begins moving in this direction:

pending compactions ↑

SSTables per read ↑

read amplification ↑

disk I/O ↑

foreground latency ↑

At the same time, compaction is trying to catch up:

background I/O ↑

The node is now performing more maintenance precisely while foreground work is becoming more expensive.

Discord described this dynamic directly. When compaction fell behind, reads became more expensive, while aggressive catch-up compaction could itself contribute to worsening latency. At times the team removed a node from normal traffic so it could compact, then let it catch up through hinted handoff—a procedure they nicknamed the gossip dance.

That is a good example of:

Maintenance debt becoming request latency.

Compaction strategy determines which amplification you accept

There is no universally best compaction strategy.

Different approaches trade among:

read amplification
write amplification
space amplification

in different ways.

Apache Cassandra's Unified Compaction Strategy documentation discusses this explicitly. More leveled behavior can improve read amplification and latency while increasing write costs, while more tiered behavior may favor writes at the cost of more expensive reads.

That means:

read-heavy serving

and:

TTL-heavy time-series storage

do not automatically deserve identical compaction behavior.

Compaction strategy is not housekeeping that can be chosen independently of the data model.

It is part of the model.

Repair Makes Retention a Correctness Decision

Replication gives Cassandra fault tolerance.

It also means replicas can diverge.

Nodes can be unavailable during writes. Network partitions happen. Disks fail. Machines are replaced. Anti-entropy repair exists to reconcile replica state over time.

Repair also consumes real resources:

network
disk
CPU
I/O
operator attention

Uber has written about the operational difficulty of keeping repair healthy at large scale, including scenarios where slow anti-entropy repair contributed to replica inconsistency and data resurrection.

That phrase is worth understanding.

How deleted data can return

Imagine:

RF = 3

with:

Replica A: value X
Replica B: value X
Replica C: value X

Replica C goes offline.

Now X is deleted:

Replica A: tombstone X
Replica B: tombstone X
Replica C: old value X

Eventually deletion metadata can become eligible for purging after the configured grace period and other safety conditions.

If the stale replica has not been reconciled appropriately before the deletion information disappears, the old copy can later become dangerous because the system has lost the metadata proving that it should remain deleted.

Apache Cassandra's documentation explains that tombstone grace exists partly to give unavailable replicas time to recover and learn about deletions. Cassandra also supports mechanisms that tie tombstone purging more closely to repaired SSTables.

That makes a parameter associated with cleanup part of the correctness model.

Reducing tombstone retention because:

tombstones are hurting reads

without understanding repair guarantees can trade:

performance problem

for:

data correctness problem

That is rarely a good trade.

Hinted handoff and repair solve different problems

Cassandra can retain hints for temporarily unavailable replicas and deliver those mutations when the replica returns.

That is useful for short-lived outages.

It is not a complete replacement for anti-entropy repair.

A useful mental model is:

hinted handoff
=
short-term missed-write delivery assistance

while:

repair
=
long-term replica reconciliation

The two mechanisms are related.

They are not interchangeable.

Tail Latency Is Where Background Costs Become Visible

Suppose:

99% of partitions
→ 15 ms

while:

1% of partitions
→ 800 ms

The average is only about:

23 ms

That dashboard might look healthy.

A user whose data consistently maps to the slow partition experiences an entirely different database.

This is why I care about:

p50
p95
p99
p99.9

but also about the dimensions behind them:

latency by partition
latency by coordinator
latency by replica
largest partitions
hottest partitions

A global percentile tells us a tail exists.

It does not necessarily tell us what physical data shape is producing it.

JVM pauses appear in the tail too

Classic Cassandra deployments run on the JVM, so heap behavior and garbage collection also enter the latency model.

Discord reported spending substantial operational effort on GC and heap tuning. Pauses could produce meaningful latency spikes, including extreme cases that required manual intervention.

Again:

average CPU

can look normal.

A runtime pause on one important replica at the wrong moment can appear primarily in:

p99
p99.9

Tail latency is often where workload skew, compaction, repair, GC, and unusual partition shapes finally become visible together.

That is why I increasingly think of Cassandra p99 as partly a data-model metric.

A stable median with a worsening tail may indicate:

hot partitions
wide partitions
tombstone-heavy reads
slow replicas
compaction interference
GC pauses
repair activity
partition skew

The pathological data lives in the tail long before the median admits there is a problem.

Changing the Traffic Can Be Better Than Changing the Database

Discord eventually moved its message store from Cassandra to ScyllaDB.

The more interesting lesson is that the team explicitly recognized that changing the database alone would not remove the hot-partition problem.

A hot logical key remains hot.

Discord therefore added a data-service layer in front of the storage system.

One important feature was request coalescing.

Before:

1000 callers
→ 1000 database queries

After:

1000 callers
→ 1 database query
→ result shared with callers

They also used consistent-hash routing so requests for the same channel tended to reach the same data-service instance, increasing the chance that duplicate concurrent reads could be coalesced.

That leads to a broader point:

Not every database bottleneck has to be solved inside the database.

Sometimes the better answer is to reshape the traffic before it reaches storage.

A hot read can sometimes be mitigated through:

request coalescing
cache
upstream batching
rate limiting
routing

rather than asking the database to execute the same expensive operation thousands of times.

Storage architecture and request architecture are connected.

Every Partitioning Strategy Moves Complexity Somewhere

Suppose the original model is:

PRIMARY KEY (
    (customer_id),
    timestamp
)

It works beautifully.

Then one enterprise customer becomes 1,000 times larger than the median customer.

The architecture now has several possible directions.

Static time bucketing

Use:

customer_id + month

or:

customer_id + day

This bounds individual partition growth but causes wider time-range queries to fan out.

Hash bucketing

Use something like:

customer_id + shard_number

Writes distribute across several logical partitions.

Reads now have to query and merge them.

Adaptive partitioning

Treat unusually hot identities differently from ordinary ones.

That can better match real skew, but it introduces routing and metadata complexity.

Request coalescing

Leave the physical model unchanged while reducing duplicate concurrent reads.

Caching

Shift repeated hot reads away from Cassandra.

Migrate the representation

Sometimes the original partitioning model has simply reached the end of its useful life.

None of these choices is free.

They move complexity among:

write distribution
read fan-out
routing
metadata
cache consistency
migration cost

The right design depends on which costs the real workload can tolerate.

Model the Worst Partition, Not the Median One

Suppose monitoring says:

median partition size = 5 MB

That sounds excellent.

Now add:

p99 partition size = 800 MB

maximum partition size = 19 GB

You are looking at a very different system.

For Cassandra, I care about distributions such as:

partition bytes
rows per partition
reads per partition
writes per partition
tombstones scanned
SSTables touched

not merely cluster-wide averages.

The dangerous partition is usually not the median partition.

Review the schema with future questions

Before approving a new Cassandra model, I would want answers to questions like these:

Those questions are much more valuable than:

Does this CQL query work?

The query working is the beginning.

Deletes belong in schema review too

If a table retains 30 days of data using TTL, schema review should not stop at:

How many writes per second?

It should also ask:

How many expirations per second?

How many tombstones will accumulate?

Which compaction strategy fits
the retention lifecycle?

How often do queries scan
expired ranges?

How frequently can repair complete?

When is deletion metadata safe
to remove?

TTL changes the storage workload.

Retention design is data-model design.

Test Distribution, Not Just Volume

A misleading performance test might generate:

1,000,000 evenly distributed requests

while production traffic looks more like:

50% of requests
→ 0.1% of keys

Those are not equivalent workloads.

A realistic Cassandra load test should vary both volume and shape:

uniform key distribution
Zipfian / skewed distribution
hot-key bursts
large partitions
TTL-heavy data
delete-heavy data
compaction under foreground load
repair under foreground load

and observe:

p50
p95
p99
p99.9

SSTables per read
tombstones scanned
pending compactions
disk utilization
GC pauses
replica latency

A system can survive the average production request rate and still fail the real production distribution.

That distinction is especially important for Cassandra because many of its pathological cases are localized rather than cluster-wide.

A perfectly even benchmark is almost designed to hide them.

Diagnose the Shape Before Tuning the Knobs

When Cassandra latency increases, the number of available tuning knobs can be dangerous.

It is easy to change:

heap
compaction throughput
concurrent reads
cache settings
consistency level
gc_grace_seconds

and temporarily move the symptom.

Before changing configuration, I would rather establish:

Which queries are slow?

Which partitions are involved?

How large are those partitions?

How many SSTables does the read touch?

How many tombstones are scanned?

Which replicas are slow?

Is compaction behind?

Is repair running?

Does the problem correlate with
specific logical keys?

Then tune or remodel.

Otherwise, configuration turns into archaeology: change enough unrelated settings, wait for behavior to move, and try to infer what the original problem might have been.

The data shape should be part of the diagnosis from the beginning.

Adding nodes is not a universal fix

Growing from:

20 nodes

to:

40 nodes

can help when the bottleneck is:

aggregate storage

aggregate throughput

well-distributed traffic

It may do much less for:

one enormous partition

one hot logical key

one pathological query

one tombstone-heavy read

one overloaded replica set

Nodes add aggregate capacity.

They do not automatically reshape the logical partitioning scheme.

That is why horizontal scalability and good partition design are complementary rather than interchangeable.

Sometimes the Correct Fix Is a Migration

This is the uncomfortable answer.

If the logical partition key no longer matches the workload, runtime tuning may not be able to fix the underlying problem.

Changing:

PRIMARY KEY

usually means creating a different table.

A migration may require:

new schema
dual writes
historical backfill
validation
read cutover
old-table retirement

In other words, the same staged migration discipline used for other live schema changes.

Discord eventually migrated trillions of messages from Cassandra to ScyllaDB while also introducing changes in the surrounding data-service architecture. During the migration, it dual-wrote data, built custom historical migration tooling, compared reads across the old and new stores, and only later shifted production traffic.

The database changed.

The more important point is that the architecture around the database changed too.

The workload had evolved enough that the original physical and operational assumptions no longer matched reality.

Sometimes the most responsible performance optimization is not another configuration flag.

It is accepting that the data model needs a new generation.

Cassandra Did Not Fail

It is tempting to read stories like Discord's and conclude:

Cassandra is bad

That is not the interesting lesson.

Discord chose Cassandra because it needed scalability and fault tolerance, and the system grew from billions to trillions of messages. The difficult behavior emerged from the interaction of:

massive growth
workload skew
hot partitions
wide partitions
compaction
JVM behavior
repair
operational maintenance

rather than one simplistic database flaw.

Uber, meanwhile, continues to operate Cassandra for mission-critical workloads at enormous scale and has invested substantially in repair, node replacement, consistency behavior, and operational reliability.

The more useful takeaway is:

Distributed databases do not remove data-model constraints. They make different constraints important.

Cassandra's architecture gives you powerful control over how data is distributed and retrieved.

That control is also responsibility.

The schema encodes assumptions about future workload shape.

Those assumptions need to remain true—or evolve when production proves otherwise.

The Invariants I Want

For a Cassandra-backed system, I care about guarantees such as:

No logical key should create
unbounded partition growth.

One hot key should not be able
to saturate an entire replica set.

TTL and deletion behavior should not create
unbounded tombstone scanning.

Compaction should be able to keep up
with steady-state writes.

Repair should complete frequently enough
to preserve replica and deletion correctness.

Tail latency should be observable by meaningful
partition or workload dimensions.

Partition-size distributions should be monitored,
not only total cluster storage.

Adding nodes should not substitute for fixing
pathological partition design.

Migration, backup, streaming, and repair tooling
must handle the largest partitions in the system,
not only typical ones.

Load tests should reproduce realistic workload skew,
not merely realistic total request volume.

Those are architectural guarantees.

They are not Cassandra configuration settings.

The configuration exists to support the data model.

It cannot make a structurally pathological workload disappear.

Final Thought

Cassandra is often introduced with a compelling scaling story:

distribute data
replicate it
add nodes
scale horizontally

That story is real.

The difficult part begins one level above the token ring.

Your workload is not evenly distributed merely because hashes are.

Partitions do not remain small merely because they started small.

Deleted data does not immediately disappear.

Compaction is not optional housekeeping.

Repair is not only an operational chore.

TTL is not merely a retention setting.

And average latency is not enough to tell you whether a handful of pathological partitions are quietly destroying the tail.

A Cassandra model can therefore work beautifully for months.

It can work beautifully for years.

Then growth changes the distribution. One customer becomes enormous. One channel becomes viral. Historical deletes accumulate. Compaction starts falling behind. Repair takes longer. One partition becomes several orders of magnitude larger than the median.

The assumptions encoded in the original partition key finally become visible in production latency.

That is why the Cassandra modeling question I care about most is not:

“Will this query be fast?”

It is:

“What will the worst version of this partition look like after the system has grown for several years?”

If the architecture still looks healthy under that scenario, you probably have something durable.

If it does not, the problem may already exist.

You simply have not accumulated enough data to see it in the tail yet.

References

• Discord Engineering — How Discord Stores Trillions of Messages. Describes Discord's Cassandra message-storage growth, hot partitions, quorum-sensitive latency, compaction backlog, JVM garbage-collection problems, tombstone-heavy partitions, migration tooling, request coalescing, and consistent-hash routing.

• Apache Cassandra Documentation — Compaction overview. Explains immutable SSTables, tombstones, TTL expiration, compaction, gc_grace_seconds, and why deletion metadata must remain long enough to preserve distributed correctness.

• Apache Cassandra Documentation — Leveled Compaction Strategy. Documents the interaction between partition size and SSTable targets, including the behavior of exceptionally large partitions.

• Apache Cassandra Documentation — Unified Compaction Strategy. Describes tradeoffs among read amplification, write costs, and leveled versus tiered compaction behavior.

• Uber Engineering — How Uber Optimized Cassandra Operations At Scale. Covers operating Cassandra for mission-critical workloads at large scale, including repair, node replacement, replica inconsistency, and data-resurrection problems associated with slow anti-entropy repair.

• Netflix Technology Blog — The Evolution of Cassandra Data Movement at Netflix. Describes large-scale data-movement failures involving skewed and wide Cassandra partitions and a redesign intended to process those partition shapes more safely.