Runtime concurrency: threads, processes, async and GIL
Why you care
The Python concurrency question comes up in almost every backend interview at a product company, and getting it wrong costs the offer: not because the interviewer expects you to have memorised the CPython source, but because the answer predicts whether you will reach for async on a CPU-bound job and wonder why it got slower. In a TCS ETL role you probably never had to choose; the script ran sequentially and that was fine. Product companies run 50k concurrent requests and need you to know why threads do not double CPU throughput, what the GIL prevents, and when async is the only sane choice.
Explain like I'm 5
Imagine a commercial kitchen with one stove (the GIL). Three chefs (threads) share the kitchen: only one can cook at a time. If a chef is waiting for the oven timer to ring — a network call, a disk read — they step away from the stove and another chef can use it. That is why threads help for I/O: the stove is free while you wait. CPU work is different: the chef using the stove just stands there, and the other chefs queue behind, so two chefs doing pure CPU work take longer than one — the queue adds overhead with no gain. Processes are separate restaurants, each with their own stove: truly parallel. Async is one chef so organised that she never stands idle: she starts a sauce, hands the pan off with "await", chops vegetables while waiting, and returns the moment the oven signals done.
Core concept
Three primitives, three problems:
- Threads (
threading,concurrent.futures.ThreadPoolExecutor) share heap memory and overlap I/O-bound work because the GIL is released during OS syscalls. They do not parallelise pure Python CPU work: only one thread holds the GIL and executes bytecode at a time. - Processes (
multiprocessing,concurrent.futures.ProcessPoolExecutor) each have their own interpreter and their own GIL: true CPU parallelism. The cost is startup time (~50–100 ms per process, illustrative) and pickling every argument and result. - Async (
asyncio) keeps everything on one thread. A coroutine yields at everyawait; the loop picks the next ready task. It is the most memory-efficient model for large numbers of concurrent I/O operations because there is no OS thread per connection. It gives you nothing for CPU-bound work.
Decision rule. Use threads when you have a few blocking calls and want overlap without library rewrites. Use processes when the work is CPU-bound pure Python. Use async when you have many concurrent I/O operations you can control end-to-end, and every dependency in the call chain has an async-compatible client. If you are unsure and the job is small, stay synchronous: a plain for loop is easier to debug than a stalled event loop.
Under the hood
The GIL is a mutex in CPython's interpreter that protects reference-count operations. CPython tracks how many names point to each object with a plain integer; two threads decrementing that counter simultaneously without a lock would free an object while another thread was using it. The GIL is held for the lifetime of a bytecode instruction and released before every OS syscall — file read, socket write, time.sleep — and reacquired on return. Every 5 ms (sys.getswitchinterval()) CPython signals other threads to compete; the running thread finishes its current bytecode and may yield. This signal-and-yield is what causes CPU-bound threads to be slower than serial code: extra context switches with no parallelism.
Python 3.13 added an experimental free-threaded build (a separate python3.13t binary; PYTHON_GIL=0 keeps the GIL off in it). Python 3.14 makes it officially supported via PEP 779, but it remains a separate build — the standard GIL interpreter is still the default and the one shipped by most platforms. Extension modules that access Python objects without their own internal locking automatically re-enable the GIL with a warning. Production use depends on your full dependency tree being updated.
Minimal example A coroutine object is not a running task
import asyncio, warnings
async def upload(key):
return f'uploaded {key}'
# Calling an async function returns a coroutine object; it does not run
coro = upload('orders.csv')
print('type:', type(coro).__name__)
# Forgetting await: Python warns at garbage-collection time
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter('always')
dead = upload('settlements.csv')
del dead
import gc; gc.collect()
print('warning:', w[0].category.__name__ if w else 'none')
# Run it properly
result = asyncio.run(upload('audit.csv'))
print('result:', result)
coro.close()type: coroutine
warning: RuntimeWarning
result: uploaded audit.csvIn production Parallel S3 downloads with a thread pool
A Jenkins ETL job downloads five daily settlement files from S3 and inserts them into Postgres. Replacing the sequential loop with ThreadPoolExecutor halves wall-clock time because the GIL is released during network I/O. sqlite3 stands in for Postgres; file reads simulate the S3 download.
import sqlite3, csv, concurrent.futures
from pathlib import Path
for i in range(1, 6):
Path(f'settlement_{i}.csv').write_text(
'id,amount\n' + '\n'.join(f'{i*100+j},{j*10}' for j in range(1, 4))
)
db = sqlite3.connect(':memory:')
db.execute('CREATE TABLE settlements (file TEXT, id INT, amount INT)')
def load_file(filename):
with open(filename, newline='') as f:
return [(filename, int(r['id']), int(r['amount'])) for r in csv.DictReader(f)]
files = [f'settlement_{i}.csv' for i in range(1, 6)]
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool:
all_batches = list(pool.map(load_file, files))
for batch in all_batches:
db.executemany('INSERT INTO settlements VALUES (?,?,?)', batch)
db.commit()
count = db.execute('SELECT COUNT(*) FROM settlements').fetchone()[0]
print('rows loaded:', count)
print('files processed:', len(all_batches))rows loaded: 15
files processed: 5See it happen Three concurrency models on one diagram
- Two threads share one GIL: only one executes Python bytecode at a time.
- Thread 1 hits a network call, releases the GIL; thread 2 runs Python while thread 1 waits for I/O.
- Processes each have their own GIL and truly run in parallel on separate CPU cores.
- Async: one event loop thread. Coro A runs until it awaits, then hands control back to the loop.
- The loop immediately picks up coro B. Both coroutines make progress with zero OS threads beyond one.
In your world
The situation
Your Jenkins job downloads five daily pricing CSV files from S3 sequentially, parses each, and inserts into a Postgres staging table. Total runtime is 8 minutes, mostly network-wait during S3 reads.
What actually breaks
Finance waits 8 minutes every morning for the staging table. You try two parallel Jenkins jobs but hit a file-naming collision on the shared download path: zero rows load, build failure.
What you'd do on Monday
Replace the sequential loop with ThreadPoolExecutor. Each thread downloads and parses one file; the GIL is released during S3 reads so all five overlap, cutting runtime from 8 minutes to roughly 2 (illustrative). Stream directly into memory to avoid collision:
import concurrent.futures, boto3, csv, io, psycopg
def fetch_and_parse(key):
body = boto3.client('s3').get_object(Bucket='pricing', Key=key)['Body'].read()
return [row for row in csv.DictReader(io.StringIO(body.decode()))]
keys = ['cat_a.csv', 'cat_b.csv', 'cat_c.csv', 'cat_d.csv', 'cat_e.csv']
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool:
batches = list(pool.map(fetch_and_parse, keys))
with psycopg.connect(DSN) as conn:
for rows in batches:
conn.executemany(INSERT_SQL, rows)Imagine you work at Zomato
The numbers
50,000 concurrent HTTP requests arrive at the 1–2 PM lunch window. Each request enters a Python handler that makes three downstream I/O calls (menu service, inventory, personalisation) averaging 100 ms each. Sequential per request: 3 × 100 ms = 300 ms. One OS thread per request: 50,000 threads × 64 KB minimum stack = 3.2 GB stack allocation. The OS scheduler managing 50,000 runnable threads adds 20–30% CPU overhead from context switches alone (illustrative), even when all threads are idle waiting for the same three downstream services.
What the Scenario-A answer does here
The Scenario A fix — switch from sequential to ThreadPoolExecutor — works well up to a few hundred concurrent requests. At 50k the thread count approaches the OS limit (typically 32k–64k on Linux without ulimit tuning), and the memory pressure from thread stacks turns a latency problem into an OOM crash.
What flips at this scale
At this scale, async is the design. One event loop handles all 50,000 handlers concurrently; each await yields control while the handler waits. Thread overhead: one OS thread. Memory per waiting handler: one task object (~few KB) instead of a 64 KB stack.
The trade-off you'd defend
I would defend async over threads for a high-concurrency request handler, trading the higher refactor cost and the need for async-compatible clients (httpx, asyncpg) for an order-of-magnitude reduction in memory footprint. For an internal tool handling ten concurrent requests, I would keep threads: simpler to test and no library rewrites.
Illustrative; 50k concurrent sessions and 64 KB stack sizes are representative of production-scale web services, not Zomato-published figures.
Threads vs Processes vs Async
| Axis | Threads | Processes | Async |
|---|---|---|---|
| GIL | shared | each has own | single thread, no contention |
| CPU-bound work | no speedup (GIL) | true parallelism | no speedup |
| I/O-bound work | overlaps (GIL released) | overlaps | optimal (many tasks, one thread) |
| Startup cost | low (microseconds) | high (~50–100 ms, illustrative) | low (coroutine object) |
| Memory per unit | ~64 KB stack (OS) | separate interpreter | ~few KB per task |
| Shared heap | yes | no (pickle boundary) | yes |
Gotchas
A blocking call inside an async function stalls the entire event loop
An async handler takes 2 seconds per request; asyncio.get_event_loop().is_running() is True but no other coroutine makes progress during those 2 seconds.
Calling time.sleep(), a synchronous requests.get(), or any blocking OS call directly inside async def does not yield to the event loop. The loop is stuck until the call returns.
Replace with the async equivalent (asyncio.sleep, httpx.AsyncClient) or offload with await asyncio.to_thread(blocking_fn, arg) (Isolate blocking work with executors).
Threads do not speed up pure Python CPU work
Adding a second thread to a CPU-bound loop makes total runtime longer, not shorter.
The GIL serialises bytecode execution; the second thread adds context-switch overhead with no parallel progress.
Use ProcessPoolExecutor for CPU-bound work, or switch to a C-extension that releases the GIL internally (NumPy operations, hashlib, zlib).
asyncio.get_event_loop() in a new thread raises RuntimeError
Code that works in the main thread raises RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-0_0'.
Python 3.10+ no longer creates an event loop on demand in threads that did not start with asyncio.run.
Call asyncio.run(coro()) to create and run a new loop in that thread, or use asyncio.to_thread() from within an already-running loop (Isolate blocking work with executors).
Tricky questions
Yes — that is what threading is for.
For pure Python code, no. The GIL means one thread runs bytecodes at a time. Threads do overlap for I/O waits and for work in C extensions that release the GIL. Whether you get parallelism depends on what the thread does, not just that it exists.
Yes — async is the modern choice, so it must be better.
For a small number of concurrent operations the difference is negligible. Async wins at scale — thousands of connections — because it avoids OS thread-scheduling overhead and per-thread stack memory. For ten requests, a thread pool is as fast and simpler.
Scenario question
You're handed a Python microservice that makes three external API calls per incoming request using the synchronous requests library. It handles 200 requests per second and is hitting its CPU ceiling. Where is the bottleneck and what do you change?
Model answer
I'd profile first to confirm where the CPU goes. If the three API calls dominate latency, the service is I/O-bound, not CPU-bound — the "CPU ceiling" is thread context-switch overhead from each request holding a thread across three serial waits. I'd switch to httpx.AsyncClient with asyncio.gather to fan out the three calls in parallel per request (Choose gather or a task group), cutting latency from three serial times to one. If a call can't be made async, I'd isolate it with asyncio.to_thread (Isolate blocking work with executors), not rewrite everything.
Interview corner
CommonWhat is the GIL and why does it matter for concurrency?
TrickWhy is calling an async function without await a silent bug?
CommonWhen would you choose processes over threads?
DesignHow would you design a service handling 10k concurrent WebSocket connections?
asyncio.to_thread (Isolate blocking work with executors).”Memory hook
The kitchen image holds everything: one stove (GIL) means two chef-threads take turns, not doubling kitchen output. A second restaurant (process) is the only way to truly cook in parallel. Async is one very organised chef who never stands idle at the oven.
Threads overlap I/O; processes parallelise CPU; async handles many concurrent I/O tasks on a single thread — never confuse which tool is which.
Self-check
Can two Python threads execute Python bytecode at the same time?
No. The GIL allows only one thread to execute bytecodes at a time. They can overlap when one is blocked on I/O.
What does calling an async function return?
A coroutine object. The function body does not run until the coroutine is awaited inside a running event loop.
What does
asyncio.run(coro())return?The coroutine's own return value; it creates a new event loop, runs it to completion, and closes it.
If you call
time.sleep(5)inside anasync def, what happens to the event loop?The loop blocks for 5 seconds. All other tasks also stop progressing. Use
await asyncio.sleep(5)instead.When does
ProcessPoolExecutoroutperformThreadPoolExecutor?When the work is CPU-bound pure Python, where the GIL would prevent thread parallelism.
Write a bench.py file with:
import concurrent.futures, time
def cpu_sum(n):
return sum(i * i for i in range(n))
N, TASKS = 500_000, 4
t0 = time.perf_counter()
for _ in range(TASKS):
cpu_sum(N)
serial_s = time.perf_counter() - t0
t0 = time.perf_counter()
with concurrent.futures.ThreadPoolExecutor(max_workers=TASKS) as pool:
list(pool.map(cpu_sum, [N] * TASKS))
thread_s = time.perf_counter() - t0
print('threads slower than serial:', thread_s > serial_s)threads slower than serial: TrueRecord the ratio thread_s / serial_s. Then add a ProcessPoolExecutor version inside a if __name__ == '__main__': block in bench.py and run python bench.py. Record the process-pool ratio. Explain in one sentence why the process result differs.