Thread Pools & Executors — Reusing Threads Instead of Creating Them

A ride-matching service gets a request for every rider. The obvious design starts a new thread for each request. That works in a demo but fails under load. Every thread costs memory for its stack and time to create, and ten thousand requests mean ten thousand threads competing for a handful of cores.

A thread pool keeps a fixed set of threads and feeds them tasks from a queue. Java's Executor framework packages that idea, and Python's concurrent.futures mirrors it. This lesson shows how to submit work to a pool, get results and failures back, shut the pool down, and set its size and limits so it stays up under load.

This builds on Multithreading & Concurrency Basics, especially Callable, Future and daemon threads. Every output below was produced by running the code on Java 21 and Python 3.11. Where timings appear, they are rounded or labeled illustrative.

You'll be able to: explain what a pool saves over a thread per task, and show tasks reusing a few threads; predict where a task's exception goes with execute() and with submit(); shut a pool down so that queued work either finishes or is reported; spot the unbounded queue in newFixedThreadPool and the unbounded threads in newCachedThreadPool, and replace them with a bounded ThreadPoolExecutor; size a pool for CPU-bound and I/O-bound work, and recognise a pool starvation deadlock; choose between fixed-rate and fixed-delay scheduling.

Table of contents

1. Why not a thread per task?
2. The Executor framework
3. Where task exceptions go: execute vs submit
4. Shutting a pool down
5. What the Executors factories hide
6. Bounded queues and rejection policies
7. Sizing a pool, and pool starvation
8. Scheduled pools: fixed rate vs fixed delay
9. Virtual threads
10. Mental-model summary
11. Gotcha checklist
12. Check yourself
13. Sources

1. Why not a thread per task?

Starting a thread for every request has four costs that grow with load:

• Memory. Each platform thread reserves a stack, typically around 1 MB on 64-bit Linux. Ten thousand threads reserve gigabytes.
• Creation time. Creating and destroying an OS thread for every request is slow compared with handing the task to a thread that already exists.
• Context switching. With far more runnable threads than cores, the CPU spends its time switching between them instead of working (Basics, section 2).
• No limit. During a traffic spike, the service keeps creating threads until the process runs out of memory. Nothing tells callers to slow down.

A pool fixes all four. A fixed number of threads is created once, and those threads take tasks from a queue. A restaurant works the same way: it employs a fixed kitchen staff and queues the orders, rather than hiring a new chef for each order. Here are nine ride requests handled by a pool of three threads:

Output:

9 ride requests ran on 3 threads:
  pool-1-thread-1
  pool-1-thread-2
  pool-1-thread-3

Analysis. Nine tasks ran, but only three threads ever existed. Each thread took a task from the queue, ran it, and came back for the next. Python's ThreadPoolExecutor behaved the same way; it names its threads from the thread_name_prefix argument (pool_0, pool_1, pool_2).

2. The Executor framework

The framework in java.util.concurrent is a few types:

Two ways to hand over work:

• execute(Runnable) returns nothing. Use it when nobody needs the outcome.
• submit(Runnable | Callable) returns a Future. Its get() waits until the task ends, then returns the result or throws the task's exception (Basics, section 6).

Python's ThreadPoolExecutor has only submit(), which always returns a Future. Calling submit() and ignoring the Future is the Python equivalent of execute().

3. Where task exceptions go: execute vs submit

The same failing task, handed over both ways:

Output (Java):

-- execute()
Exception in thread "pool-1-thread-1" java.lang.IllegalStateException: mail server down
	at Main.lambda$main$0(Main.java:7)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
-- submit()
submit() printed nothing; isDone = true
get() reveals it: java.lang.IllegalStateException: mail server down

Output (Python):

submit() printed nothing; done = True
result() reveals it: RuntimeError('mail server down')

Analysis. With execute(), the exception escaped the task and killed the worker thread. The default handler printed its stack trace, and the pool replaced the thread. With submit(), nothing printed at all. The Future caught the exception and kept it, and it only appeared when get() was called. Python's submit() behaves the same way: the failure stays silent until someone calls result().

Intuition. Mechanism. submit() wraps your task in a FutureTask, whose run() catches any exception and stores it as the Future's outcome [1]. The worker thread sees a task that returned normally. execute() runs your Runnable directly, so an exception propagates to the thread's uncaught exception handler.

Concrete bite. A common bug is calling pool.submit(this::sendInvoice) and throwing the Future away. If sendInvoice throws, there is no log line and no stack trace. The code looks just like the execute() version, but it fails silently.

4. Shutting a pool down

Pool threads are not daemon threads, so a pool that is never shut down keeps the JVM running after main ends. There are three ways to stop a pool, and they differ in what happens to work still in the queue:

Output (Java):

shutdown() returned at once
report-3 rejected after shutdown
report-1 finished
report-2 finished
awaitTermination: all done = true
export interrupted
shutdownNow: 2 queued tasks never started

Output (Python):

shutdown(wait=False) returned at once
report-3 rejected after shutdown
report-1 finished
report-2 finished
export finished
cancel_futures: 2 queued tasks never started

Analysis.

• shutdown() returned at once without waiting for anything. It only stopped new submissions, which is why report-3 was rejected. The two reports already queued still ran.
• awaitTermination() is the call that waited. It returned true because everything finished within the 5-second limit.
• shutdownNow() interrupted the running export and returned the two tasks that had never started, so the caller can log or retry them.
• Python's shutdown(cancel_futures=True) also drops queued tasks, but it cannot interrupt a task that is already running, so export finished [8].

Intuition. Mechanism. shutdown() starts an orderly shutdown in which "previously submitted tasks are executed, but no new tasks will be accepted"; it "does not wait" [2]. shutdownNow() interrupts the running tasks. That only stops a task that responds to interruption; sleep does, which is why export stopped. Since Java 19, ExecutorService is AutoCloseable: close() shuts down and waits, so a try-with-resources block does both [2]. Python's with ThreadPoolExecutor(...) block does the same.

Concrete bite. A common mistake is to treat shutdown() as "wait for everything to finish", and then read results that are not ready yet. The calls that wait are awaitTermination() and close().

5. What the Executors factories hide

These factories are convenient because they choose defaults for you, and the method names don't mention them [3]. Here is what 1,000 slow tasks do to the first two:

Output:

fixed(2):  threads = 2, waiting in queue = 998
cached:    threads = 1000, waiting in queue = 0

Analysis. The fixed pool kept its 2 threads and queued the other 998 tasks. That queue has no limit, so under sustained overload it grows until the heap runs out. The cached pool did the opposite: it created a thread for every task, 1,000 of them, and queued nothing. During a traffic spike, that is the thread-per-task design from section 1 all over again. Neither pool ever refused a task.

6. Bounded queues and rejection policies

ThreadPoolExecutor's constructor makes you choose every setting: the core number of threads, the maximum number, how long extra threads may sit idle, the queue, and what to do when everything is full [4]. With two threads and a queue of two, the pool can hold at most four tasks:

Output (Java):

task 1 accepted
task 2 accepted
task 3 accepted
task 4 accepted
task 5 rejected: pool and queue are full
task 6 rejected: pool and queue are full
task 5 ran on main: the caller slowed down

Output (Python):

task 1 accepted
task 2 accepted
task 3 accepted
task 4 accepted
task 5 rejected: pool and queue are full
task 6 rejected: pool and queue are full

Analysis. Tasks 1 and 2 went to the two threads, tasks 3 and 4 filled the queue, and tasks 5 and 6 were rejected with RejectedExecutionException: the default AbortPolicy. The second pool used CallerRunsPolicy. When that pool was full, task 5 ran on main, the thread that submitted it. While main was busy running the task, it could not submit any more, so the submitting code slowed down to the pool's pace. Python's executor has no bounded queue, so the Python version limits the number of tasks in progress with a BoundedSemaphore, which has the same effect.

Intuition. Mechanism. For each new task, the pool tries four things in order: run it on a free core thread; otherwise put it in the queue; otherwise start an extra thread, up to the maximum; otherwise hand it to the rejection policy [4]. The built-in policies:

Concrete bite. With an unbounded queue, the third step ("start an extra thread") never happens, because the queue is never full. A ThreadPoolExecutor(2, 10, …, new LinkedBlockingQueue<>()) never grows past 2 threads. The maximum only matters with a bounded queue.

7. Sizing a pool, and pool starvation

How many threads should a pool have? It depends on what the tasks do while they run. Brian Goetz gives this rule of thumb [5]:

threads ≈ cores × target CPU utilisation × (1 + wait time ÷ compute time)

• CPU-bound tasks (parsing, image resizing, number crunching) hardly wait, so the wait-to-compute ratio is close to 0. That gives about one thread per core; Goetz suggests the number of cores plus one. More threads than that only add switching.
• I/O-bound tasks (calling an API, querying a database) mostly wait. A task that waits 90 ms for every 10 ms of computing has a ratio of 9, so on 4 cores about 40 threads are needed to keep the CPUs busy.

The number must also fit whatever the tasks wait for. Forty threads sharing a database connection pool of 10 connections will spend most of their time waiting for a connection. Measure, then adjust.

A pool also creates a hazard that separate threads don't have. If tasks wait for other tasks in the same pool, they can occupy every thread while the tasks they are waiting for sit in the queue:

Output:

timed out: the inner task is queued behind the task waiting for it

Analysis. The outer task took the pool's only thread, submitted the inner task, and waited for it. The inner task could only run on that same thread, which was busy waiting. Without the 1-second timeout, both tasks would wait forever. With a pool of n threads, n such outer tasks running at the same time cause the same hang.

Intuition. Mechanism. This is a thread starvation deadlock: a task waits on work that needs a pool thread, and every pool thread is busy waiting [5]. No lock is involved, so lock-based deadlock detection does not see it.

Concrete bite. Tests with a large pool and light load rarely trigger it. It appears in production, when enough outer tasks arrive at the same time.

8. Scheduled pools: fixed rate vs fixed delay

A ScheduledExecutorService runs a task after a delay, or repeatedly. "Run it every 500 ms" can mean two different things, and the difference shows when the task itself takes time. Here the task takes 300 ms:

Output:

fixed rate  started at ~0 ms
fixed rate  started at ~500 ms
fixed rate  started at ~1000 ms
fixed delay started at ~0 ms
fixed delay started at ~800 ms
fixed delay started at ~1600 ms

Analysis. Fixed rate started runs at 0, 500 and 1,000 ms: the period is measured from the start of one run to the start of the next, so the runs stay in step with the clock. Fixed delay started them at 0, 800 and 1,600 ms: the 500 ms delay is counted from the end of one run (at 300 ms) to the start of the next. Python's standard library has no periodic scheduler, so the Python version writes out both loops by hand.

Intuition. Mechanism. With scheduleAtFixedRate, if a run takes longer than the period, later runs "may start late, but will not concurrently execute" [6]. With scheduleWithFixedDelay, the gap after each run is always the full delay. If a run throws an exception, both kinds stop silently: all later runs are cancelled.

Concrete bite. A session cleaner on a fixed rate that sometimes runs longer than its period will start its next run immediately, with no pause in between. On a fixed delay, it always gets the full pause.

9. Virtual threads

Java 21 adds virtual threads: threads managed by the JVM instead of the OS [7]. When a virtual thread waits on I/O or sleep, the JVM sets it aside and frees the OS thread underneath for other work. That makes them cheap enough to create one per task, with Executors.newVirtualThreadPerTaskExecutor(). The Java guide's Concurrency: High-Level & Virtual Threads runs 10,000 of them and shows when a synchronized block pins one to its OS thread.

For pool design, virtual threads change one thing. For work that mostly waits, you no longer need to size a pool to save threads; you create one virtual thread per task. Two things stay the same. CPU-bound work still needs cores, so it still belongs on a pool of about one platform thread per core. And the resources that tasks wait for still need limits. A Semaphore (Locks & Semaphores) can cap the number of database connections in use, however many threads there are. Python's closest equivalent is asyncio, which is a different programming model.

10. Mental-model summary

11. Gotcha checklist

✅ Check yourself

One check per objective. Answer before you open anything.

📚 Sources

1. java.util.concurrent.FutureTask, Java SE 21 API — https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/FutureTask.html
2. java.util.concurrent.ExecutorService, Java SE 21 API (shutdown(), shutdownNow(), awaitTermination(), close()) — https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/ExecutorService.html
3. java.util.concurrent.Executors, Java SE 21 API — https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/Executors.html
4. java.util.concurrent.ThreadPoolExecutor, Java SE 21 API (queuing, rejected tasks, the four policies) — https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html
5. Brian Goetz et al., Java Concurrency in Practice (Addison-Wesley, 2006), ch. 8, "Applying Thread Pools": §8.1.1 "Thread starvation deadlock" and §8.2 "Sizing thread pools".
6. java.util.concurrent.ScheduledExecutorService, Java SE 21 API — https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/ScheduledExecutorService.html
7. JEP 444: Virtual Threads — https://openjdk.org/jeps/444
8. Python 3 documentation, concurrent.futures — ThreadPoolExecutor and Executor.shutdown(cancel_futures=…) — https://docs.python.org/3/library/concurrent.futures.html

Your Turn

Before you move on, check your understanding with the coach — explain the idea, apply it, weigh the trade-offs, then defend your reasoning.