Book 01 · Programming FundamentalsPage 42 of 56

Parallelism

In Turkish: Paralellik
Pronunciation: PAIR-uh-lel-iz-um

Updated October 3, 20262 min read

In short

Parallelism is running several computations at literally the same time, on multiple CPU cores, GPUs or machines, so a large job finishes faster.

What is parallelism in programming?

A modern computer has many cores, and a GPU has thousands of small ones. Parallel code splits work so these run simultaneously: resizing a thousand images on eight cores, or multiplying the large matrices inside a neural network on a GPU. Data parallelism applies the same operation to different pieces of data; task parallelism runs different tasks at once.

Most languages offer tools for it. Threads can run in parallel in Java, C#, Go, Rust and C++. In Python, the standard CPython interpreter's global interpreter lock (GIL) has long let only one thread run Python code at a time, so CPU-heavy work uses separate processes through multiprocessing or concurrent.futures, although newer versions offer an experimental build without the GIL.

Speed-ups have limits. Amdahl's law, from 1967, points out that the part of a program that must run in sequence caps the total gain: if a tenth of the work can't be parallelized, no number of cores makes it more than ten times faster. Splitting work, moving data and combining results also cost time, so small jobs can get slower when parallelized.

A common misconception is that parallelism and concurrency are the same. Concurrency is structuring a program to deal with many things at once, which works even on one core by switching between tasks. Parallelism is actually doing several things at the same instant, which needs multiple processors.

Key takeaways

Parallelism runs computations at the same time on multiple processors.
Data parallelism splits the data; task parallelism runs different tasks.
CPU-bound Python work usually uses processes because of the GIL.
Amdahl's law: the sequential part limits the maximum speed-up.
Concurrency is about structure; parallelism is about simultaneous execution.

Example

Using every CPU core for a CPU-heavy job (Python)python

from concurrent.futures import ProcessPoolExecutor
import math

def count_primes(limit):
    return sum(1 for n in range(2, limit) if all(n % d for d in range(2, math.isqrt(n) + 1)))

chunks = [200_000] * 8

if __name__ == "__main__":
    # Each chunk runs in its own process, so all cores work at the same time
    with ProcessPoolExecutor() as pool:
        results = list(pool.map(count_primes, chunks))
    print(sum(results))

Readers ask

What is the difference between concurrency and parallelism?

Concurrency means handling multiple tasks in overlapping time, possibly by switching between them on one core. Parallelism means executing multiple tasks at the same instant on multiple cores. Code can be concurrent without being parallel.

Does parallel code always run faster?

No. Splitting the work and combining the results costs time, the sequential part can't be sped up, and shared data needs coordination. For small or I/O-bound tasks, parallelism may bring little or nothing.

Why are GPUs good at parallelism?

A GPU has thousands of simple cores designed to run the same operation on many pieces of data at once. That fits graphics, matrix math and neural network training very well.

Often compared

Concurrency vs ParallelismConcurrency is handling many tasks in overlapping time, even on one core, while parallelism is running several tasks at the same instant on multiple cores.