Why Doesn’t More Concurrency Mean More Throughput in a Java Service?

A Java service starts getting slower under load, so the first instinct is often to add concurrency. The thread pool has 50 workers; increase it to 100 and throughput improves. Increase it to 200 and throughput improves a little more. Then increase it to 500 and something changes: latency explodes, the database connection pool fills, requests begin waiting, memory pressure rises, garbage collection becomes more noticeable, and some callers time out even though the JVM is still perfectly capable of creating more threads.

That progression exposes one of the easiest performance traps in server-side Java:

Concurrency is not the same thing as capacity.

Threads determine how much work can be in progress. They do not determine how much useful work the complete system can finish. A service may be capable of scheduling hundreds of platform threads—or hundreds of thousands of virtual threads—while remaining constrained by eight CPU cores, 30 database connections, one contended lock, a rate-limited API, or a storage system with a finite operations-per-second ceiling.

This is why the most useful concurrency question is usually not:

“How many threads should this service have?”

It is:

“Which resource actually limits throughput, and what happens when concurrency exceeds that resource’s useful capacity?”

Once that question becomes the starting point, Java concurrency tuning becomes much less mysterious.

Throughput, Latency, and Concurrency Are Connected

Suppose a service completes an average request in:

50 ms

and sustains:

200 requests / second

Approximately:

10 requests

must be in progress at any given time.

That relationship comes from Little’s Law:

concurrency ≈ throughput × latency

or more conventionally:

L = λW

where:

L = average number of concurrent requests
λ = throughput
W = average request duration

OpenJDK uses this relationship directly when explaining virtual threads. A service processing 200 requests per second at 50 ms average latency needs roughly 10 concurrent requests. If the same service wants to reach 2,000 requests per second without changing latency, it needs roughly 100 requests in flight.

So increasing concurrency absolutely can increase throughput. The important qualification is that it works only while additional concurrency allows useful work that was previously waiting unnecessarily.

If the bottleneck is idle CPU while requests block on I/O, more concurrency can help considerably. If the bottleneck is already a saturated database, adding more callers mostly gives the database a longer line.

Concurrency Helps When Work Can Overlap

Consider a request that spends:

5 ms computing
45 ms waiting for a database

A synchronous worker spends most of its lifetime waiting:

With only one worker, the CPU is idle for much of the request. Adding concurrent requests allows other work to use the CPU while one request waits on the database or network:

This is why a thread-per-request service may benefit from substantially more concurrent requests than physical CPU cores. The workload spends significant time blocked, so concurrency overlaps the waiting with useful execution elsewhere.

Now change the workload. Suppose each request instead performs:

50 ms of CPU-heavy computation

on a machine with eight cores.

Eight compute-heavy tasks can keep eight cores busy. Eight hundred compute-heavy tasks do not manufacture 792 additional CPUs:

Past useful CPU parallelism, the operating system has more runnable work than processors. Scheduling and context switching increase, but the amount of actual compute capacity does not.

OpenJDK makes this distinction explicitly in its virtual-thread design: virtual threads improve the scalability of workloads with many concurrent tasks that spend substantial time waiting, but they do not make CPU-bound computation execute faster.

Concurrency helps when work can overlap. It does not create more capacity inside the bottleneck itself.

The Thread Pool May Not Be the Pool That Matters

Consider a typical Spring service:

Suppose Tomcat allows:

200 request threads

while HikariCP provides:

20 database connections

and almost every request requires database access.

The effective architecture eventually becomes:

200 request threads
        ↓
20 database connections

A large number of request threads may be runnable from the HTTP server's perspective while unable to perform the next useful operation because every database connection is occupied.

A simplified configuration might look like:

server:
  tomcat:
    threads:
      max: 200

spring:
  datasource:
    hikari:
      maximum-pool-size: 20

Changing Tomcat to:

max: 500

does not give the database more capacity.

It gives another 300 callers permission to compete for the same 20 connections.

This is why tuning only the most visible executor can be misleading. The thread pool is only one concurrency boundary among several.

More Database Connections Can Make Things Worse

The obvious response is to enlarge the database pool too.

Sometimes that is exactly the right change. But a connection pool is another concurrency control, not a throughput generator.

HikariCP's pool-sizing guidance deliberately pushes against the assumption that more connections are always better. A relatively small saturated connection pool can outperform a huge pool because the database itself has finite CPU, storage, cache, locking, and I/O resources.

Imagine a database with eight CPU cores.

At moderate concurrency:

Query 1 → CPU
Query 2 → CPU
Query 3 → disk wait
Query 4 → CPU
...

a relatively small set of active connections can keep the hardware busy.

At extreme concurrency:

500 active queries

those queries compete for the same:

CPU
buffer cache
locks
disk
memory
I/O bandwidth

Eventually useful throughput stops increasing while latency continues climbing.

The critical point is that the database pool should not be sized according to how many application threads can exist. It should be sized according to how much concurrent database work the datastore can execute efficiently for the workload in question.

The same principle applies to almost every shared dependency.

Find the Knee of the Throughput Curve

Imagine a load test produces this result:

From 10 to 100 concurrent requests, additional concurrency produces substantial useful throughput.

Between 100 and 200, throughput barely moves:

850 req/s → 880 req/s

while p95 latency almost triples:

240 ms → 700 ms

At 400 concurrent requests, throughput actually falls while latency reaches seconds.

The system has crossed the knee of the curve.

Beyond that point, extra concurrency creates more of this:

queueing
contention
memory pressure
context switching
timeouts

and less additional useful throughput.

That point is far more useful operationally than the maximum number of threads the JVM can technically create.

Capacity planning is about finding the region where concurrency still produces useful work with acceptable latency and enough headroom—not discovering how much simultaneous work can exist before the process collapses.

Queueing Is Where Latency Hides

A conventional executor has two important controls:

workers
queue

For example:

ExecutorService executor =
    new ThreadPoolExecutor(
        32,
        32,
        0L,
        TimeUnit.MILLISECONDS,
        new ArrayBlockingQueue<>(1000)
    );

At low traffic, the path is simple:

request arrives
→ worker available
→ execute immediately

Under overload:

workers busy
→ enter queue
→ wait
→ execute later

The service may still complete requests successfully, but their latency now includes time spent waiting before useful execution even begins.

Suppose the service has:

32 workers
1,000 queued requests
100 ms service time

A request near the back of the queue may wait for seconds before a worker ever sees it.

If the client's deadline is:

1 second

the server may eventually spend CPU, database capacity, and downstream calls completing a request whose caller has already abandoned it.

That is wasted capacity.

Google SRE guidance describes this broader overload pattern: when steady-state capacity is insufficient, queues grow, latency increases, and queued work consumes additional resources. Smaller queues can sometimes produce a healthier service because excess work is rejected before it spends most of its deadline waiting.

A queue is therefore not free capacity.

It is stored waiting.

Unbounded Queues Turn Overload Into Memory Pressure

Java makes it easy to create executors whose queue can grow extremely large:

ExecutorService executor =
    Executors.newFixedThreadPool(32);

Conceptually, this gives the service:

32 workers
+
a queue that can continue growing

If incoming work consistently exceeds what those workers can complete, the queue becomes the pressure-relief mechanism.

The progression is often:

latency rises
    ↓
queued objects accumulate
    ↓
memory rises
    ↓
GC pressure increases
    ↓
OutOfMemoryError becomes possible

A bounded executor makes the saturation point explicit:

ThreadPoolExecutor executor =
    new ThreadPoolExecutor(
        32,
        32,
        0L,
        TimeUnit.MILLISECONDS,
        new ArrayBlockingQueue<>(128),
        new ThreadPoolExecutor.AbortPolicy()
    );

Once all workers are occupied and 128 additional tasks are waiting, the service refuses more work.

That can feel harsh because a request is deliberately rejected even though the JVM has memory remaining and could technically store it in another object.

But under sustained overload:

fast rejection

can be considerably healthier than:

wait 15 seconds
then time out anyway

The bounded queue turns overload into a visible admission-control decision rather than allowing it to hide inside latency and heap growth.

Locks Can Make a Highly Concurrent Service Effectively Serial

Now imagine CPU and database capacity are both healthy, but part of the application looks like this:

public synchronized void process(Order order) {
    validate(order);
    calculate(order);
    persist(order);
}

Every request must enter the same monitor.

One hundred threads may exist.

One thread makes progress through the critical section.

Ninety-nine wait.

Adding another hundred threads simply creates another hundred contenders.

This is another reason:

thread count

is not equivalent to:

parallelism

A service can be extremely concurrent from the scheduler's perspective and effectively serial at the resource that matters.

Ask What the Threads Are Waiting On

Contention is not always as obvious as one giant synchronized method.

Threads may wait on:

connection pools
shared maps
database row locks
transaction locks
rate limiters
filesystem operations
single-threaded SDK internals
logging appenders
hot cache keys
shared queues

When increasing concurrency stops helping, one of the best diagnostic questions is:

What are the threads actually waiting on?

A thread dump can tell a much richer story than aggregate CPU usage:

"http-nio-8080-exec-52"
WAITING
at com.zaxxer.hikari.pool.HikariPool.getConnection(...)

"http-nio-8080-exec-53"
BLOCKED
at com.example.InventoryService.reserve(...)

"http-nio-8080-exec-54"
RUNNABLE
at com.example.PricingEngine.calculate(...)

Those are three different capacity problems.

The first points toward database or connection-pool saturation.

The second points toward lock contention.

The third may genuinely be CPU-bound.

Increasing the global thread count treats all three as though they were the same problem.

They are not.

This is why thread tuning should usually follow bottleneck identification rather than precede it.

Concurrency Changes JVM Memory Pressure Too

More concurrent work also means more simultaneously live application state:

request objects
JSON trees
buffers
temporary collections
database results
logging data

Even if each request is individually modest, increasing the number of in-flight requests changes allocation rate, heap occupancy, object lifetime, and garbage-collection behavior.

At scale, GC becomes part of end-to-end latency.

Uber documented this class of problem in large Java and Scala workloads. In one Presto coordinator scenario, long garbage-collection pauses stopped request processing, increased latency, and contributed to user-visible failures. Their JVM tuning reduced pause behavior and improved latency and reliability.

The lesson is not:

“If Java is slow, tune the garbage collector.”

The more useful lesson is:

Concurrency changes how much live work the JVM must manage, so memory and GC belong in the same performance model as CPU, threads, queues, and downstream dependencies.

A service can move through bottlenecks like this:

Fixing one capacity constraint often does not remove the limit.

It reveals the next one.

Virtual Threads Change the Cost of Waiting

Java 21 finalized virtual threads through JEP 444.

Traditional platform threads are relatively expensive because they map closely to operating-system threads. Virtual threads are much lighter, and the JVM can suspend them during supported blocking operations so the carrier thread is available for other work.

That allows straightforward blocking-style code while supporting very high I/O concurrency:

try (var executor =
         Executors.newVirtualThreadPerTaskExecutor()) {

    Future<Customer> customer =
        executor.submit(() ->
            customerClient.fetch(customerId)
        );

    Future<List<Order>> orders =
        executor.submit(() ->
            orderClient.fetch(customerId)
        );

    return new Dashboard(
        customer.get(),
        orders.get()
    );
}

For I/O-heavy services, this is a major improvement. The application no longer needs a large platform-thread pool merely to represent large numbers of tasks that spend most of their time blocked.

But it is important to separate two ideas:

Virtual threads make waiting cheaper.

and:

Virtual threads increase downstream capacity.

Only the first is true.

Virtual threads make threads cheap. They do not make scarce dependencies unlimited.

Cheap Threads Do Not Make Dependencies Cheap

Suppose:

100,000 virtual threads

eventually execute:

jdbcTemplate.query(...);

while HikariCP has:

maximumPoolSize = 20

The effective database concurrency remains approximately:

20

The other tasks wait for a database connection.

OpenJDK explicitly warns against recreating the old thread-pool model by pooling virtual threads merely to control concurrency. Virtual threads should generally be created per task; scarce resources should be protected with resource-specific mechanisms such as semaphores.

For example:

private final Semaphore downstreamSlots =
    new Semaphore(20);

public Result callDependency() throws Exception {
    downstreamSlots.acquire();

    try {
        return dependency.call();
    } finally {
        downstreamSlots.release();
    }
}

Now these can both be true:

application task concurrency = very high

and:

downstream concurrency = deliberately bounded

That separation is powerful because one arbitrary thread-pool size no longer has to stand in for several unrelated resource limits.

Virtual threads may actually make poor capacity planning easier to expose. With platform threads, a service might accidentally have a natural limit of:

200 threads

Only 200 operations can block simultaneously.

Move to virtual threads and suddenly:

10,000 tasks

can wait cheaply.

That is excellent if platform threads were the bottleneck.

It is dangerous if all 10,000 tasks can now place demand on the same finite dependency.

Without the explicit boundary:

cheap concurrency
→ more simultaneous downstream demand
→ dependency saturation

Virtual threads therefore make capacity reasoning more important, not less. They remove one scarcity—the operating-system thread—and make the remaining scarcities easier to reach.

Tail Latency Usually Warns You Before the Average Does

Average latency can remain deceptively healthy during the early stages of saturation.

Suppose:

p50 = 90 ms
p95 = 240 ms
p99 = 4.2 seconds

Half of requests are still fast.

One percent of callers are experiencing a completely different service.

This often happens when queueing, connection waits, lock contention, or occasional GC pauses affect requests unlucky enough to arrive when a constrained resource is saturated.

For performance-sensitive services, I care about:

p50
p95
p99
p99.9

together rather than relying primarily on:

average latency

because saturation often appears first in the tail.

That tail is also where downstream deadlines and retries begin to matter. A small fraction of very slow requests can create timeouts and additional attempts even while average latency still makes the service look healthy.

Throughput Is Limited by the Narrowest Resource

Imagine a request path with:

HTTP worker capacity:       500
CPU useful parallelism:       8
DB pool:                     30
Downstream API concurrency:  50

Which value should the application's thread pool use?

That question assumes one concurrency number can represent several different resources.

The architecture is actually closer to:

Different phases of a request compete for different resources.

For CPU-heavy work:

8 cores

may be the meaningful parallelism limit.

For database-heavy work:

30 connections

may dominate.

For remote I/O:

50 allowed downstream calls

may be the boundary that needs protection.

That is why resource-specific concurrency limits are often more useful than one global thread count. The application can represent a large number of tasks while independently controlling how much pressure each scarce dependency receives.

Bulkheads Protect Independent Failure Domains

Suppose the same service calls:

Payment API
Recommendation API
Email API

If every operation shares one executor:

ExecutorService executor =
    Executors.newFixedThreadPool(300);

a slow recommendation service might eventually occupy most or all workers with blocked recommendation requests.

Payment requests then lose execution capacity even though the payment provider itself is healthy.

A bulkhead separates those capacities:

With platform threads, those boundaries might be represented by separate executors where appropriate.

With virtual threads, resource-specific semaphores are often a cleaner representation:

Semaphore paymentLimit =
    new Semaphore(40);

Semaphore recommendationLimit =
    new Semaphore(100);

Semaphore emailLimit =
    new Semaphore(20);

Now one unhealthy dependency cannot consume every unit of concurrency available to unrelated work.

The interesting architectural idea is not the semaphore.

It is the isolation of failure domains.

Load Shedding Can Preserve Useful Throughput

Suppose the service can sustainably complete:

1,000 requests / second

but receives:

1,500 requests / second

If all 1,500 are admitted into a large queue:

queue grows
→ latency grows
→ deadlines are missed
→ clients retry
→ additional work arrives

The system may eventually complete fewer than 1,000 useful requests per second because overload consumes resources that could have been used for work capable of finishing.

Early admission control can preserve the healthy portion:

Google SRE describes this broader cascading-failure pattern: excessive queues, thread starvation, missed deadlines, and retries can reinforce one another during overload.

A bounded service may therefore be more reliable than one that accepts unlimited concurrency and hopes a queue will absorb the difference.

The counterintuitive result is important:

Rejecting some work can allow the system to complete more useful work.

A Better Java Concurrency Design

Consider a simple request flow:

public OrderView loadOrder(String orderId) {
    Order order = orderRepository.find(orderId);
    Pricing pricing = pricingClient.get(orderId);
    Inventory inventory = inventoryClient.get(orderId);

    return new OrderView(
        order,
        pricing,
        inventory
    );
}

The pricing and inventory calls are independent, so we can overlap them:

try (var executor =
         Executors.newVirtualThreadPerTaskExecutor()) {

    Future<Pricing> pricing =
        executor.submit(() ->
            pricingClient.get(orderId)
        );

    Future<Inventory> inventory =
        executor.submit(() ->
            inventoryClient.get(orderId)
        );

    Order order =
        orderRepository.find(orderId);

    return new OrderView(
        order,
        pricing.get(),
        inventory.get()
    );
}

This can improve request latency because independent I/O happens concurrently.

Now suppose the inventory provider supports only:

50 safe concurrent calls

That capacity limit belongs at the dependency boundary:

private final Semaphore inventoryLimit =
    new Semaphore(50);

private Inventory getInventory(String orderId)
        throws Exception {

    if (!inventoryLimit.tryAcquire(
            100,
            TimeUnit.MILLISECONDS)) {

        throw new ServiceBusyException(
            "Inventory capacity exhausted"
        );
    }

    try {
        return inventoryClient.get(orderId);
    } finally {
        inventoryLimit.release();
    }
}

The architecture now expresses two separate ideas:

Create concurrency freely where it is cheap.

Bound concurrency where resources are scarce.

That is considerably more meaningful than:

thread pool size = 200

because the limit now corresponds to an actual capacity constraint.

How I Would Diagnose a Slow Java Service

If increasing threads stops increasing throughput, I would not immediately tune the executor again.

I would first ask where requests are spending time.

A useful investigation spans several layers:

Then I would load test across increasing concurrency instead of performing one giant stress run.

Not:

Can it survive 1,000 requests?

but:

What changes at
10 → 25 → 50 → 100 → 200 → 500
concurrent requests?

For each level, I would correlate:

throughput
p50 / p95 / p99 latency
CPU
runnable threads
queue depth
active DB connections
DB connection wait time
GC pause time
downstream latency
rejection rate

The source article's diagnostic framing is exactly right here: the useful question is where time is going, and the bottleneck often becomes obvious only when throughput, latency, CPU, queues, database waits, GC, and downstream timing are viewed together.

A thread count without those measurements tells you how much concurrency the system permits.

It does not tell you whether that concurrency is useful.

The Goal Is Sustainable Throughput, Not Maximum Concurrency

Suppose two configurations produce:

100 concurrent requests:
throughput = 950 req/s
p99 = 300 ms

and:

500 concurrent requests:
throughput = 975 req/s
p99 = 4 seconds

Technically, the second system delivers 25 additional requests per second.

Operationally, I would usually prefer the first.

The 2.6% throughput improvement came with more than a 13× increase in p99 latency, greater queueing, less headroom, more live memory, and probably a much sharper failure response if a dependency slows even slightly.

Production systems need slack.

Running directly on the saturation point means ordinary variation—a slower query, cache miss, deployment, GC cycle, or traffic burst—can push the service beyond the knee.

The ideal concurrency setting is therefore rarely:

the largest number that did not crash

It is closer to:

the concurrency level that produces
strong sustainable throughput,
acceptable tail latency,
and enough headroom for variation

That is capacity engineering rather than thread tuning.

The Invariants I Want

When designing a high-concurrency Java service, I care about properties such as:

CPU-bound work should not run with unbounded parallelism.

Database concurrency should remain inside
the datastore's sustainable capacity.

Slow dependencies should not consume all capacity
needed by unrelated operations.

Queues should remain bounded.

Requests that cannot finish inside their deadline
should not sit indefinitely waiting for execution.

Increasing concurrency should be validated against
throughput and tail latency rather than assumed
to improve them.

Virtual threads should not turn cheap waiting
into unlimited downstream pressure.

Saturation should produce controlled rejection
or backpressure rather than process collapse.

Those are system-level guarantees.

A thread count is only one implementation detail used to achieve some of them.

This distinction becomes even more important with modern Java. Virtual threads allow the application to stop treating operating-system threads as the scarce resource for I/O-heavy tasks. That is a major improvement because the application can now model concurrency more directly.

But it also removes an accidental capacity limit that used to protect downstream systems.

The architecture therefore needs to make the real limits explicit.

Final Thought

Concurrency is seductive because the first few increases often work beautifully.

One request waits on I/O, so run another.

Ten requests wait, so run more.

The server completes more work with the same hardware, and the performance improvement feels almost linear.

Then the relationship changes.

The CPU fills. The database connection pool saturates. Requests wait behind one lock. A downstream API begins throttling. Queueing dominates response time. More in-flight work increases heap occupancy enough for garbage collection to affect the tail. At that point, additional concurrency does not create additional capacity; it creates more competition for capacity that already exists.

Modern Java makes this distinction especially important.

Virtual threads let applications represent enormous numbers of concurrent I/O-bound tasks without paying the old cost of dedicating an operating-system thread to every blocked operation. That removes a major scalability constraint and makes straightforward synchronous code viable at much higher concurrency.

But a million virtual threads still execute on finite CPU cores.

They still query finite databases.

They still compete for finite connection pools.

They still call APIs with finite quotas.

They still allocate memory in a finite heap.

So when a Java service stops scaling, the useful question is not:

“How can I add more threads?”

It is:

“What resource did adding threads expose as the real bottleneck?”

Once that answer is measurable, concurrency tuning stops being guesswork.

It becomes capacity engineering.

References

• OpenJDK — JEP 444: Virtual Threads. The Java platform's design document for virtual threads, including Little's Law, the distinction between concurrency and CPU parallelism, guidance against pooling virtual threads, and the recommendation to use explicit constructs such as semaphores when access to scarce resources must be bounded.

• Google SRE — Addressing Cascading Failures. Explains how queue growth, thread starvation, missed deadlines, overload, and retries interact during saturation, and why accepting unlimited queued work can make server behavior progressively worse.

• Uber Engineering — Tricks of the Trade: Tuning JVM Memory for Large-scale Services. Describes production cases where JVM garbage-collection behavior materially affected throughput, latency, and reliability across large Java and Scala infrastructure.

• HikariCP — About Pool Sizing. Explains why increasing database connection counts does not necessarily improve throughput and can substantially worsen performance once the database's useful parallel capacity is exceeded.