Parallelization in Python

Python is often praised for its simplicity and readability, but when it comes to optimizing parallel code execution, it raises complex questions. This article explores the fundamental differences between multithreading, asynchronous programming, and multiprocessing in Python. It also highlights the limitations imposed by the Global Interpreter Lock (GIL) and the challenges associated with multiprocessing.

Multithreading vs Asynchronous Programming vs Multiprocessing

What is Multithreading?

Multithreading involves running multiple threads within the same process. These threads can execute on different physical or logical CPU cores while sharing the same memory space, which facilitates communication and reduces memory usage. In theory, it is possible to allocate a dedicated thread to each CPU core, allowing multiple tasks to run in parallel and resulting in significant performance gains.

However, in Python, multithreading is heavily limited by the GIL (Global Interpreter Lock). This lock prevents multiple Python threads from executing bytecode simultaneously, even on a multi-core processor. For I/O-bound tasks (such as network requests or I/O operations), multithreading remains effective because the GIL is released during these calls. Beyond these scenarios, only Python extensions written in C that explicitly release this lock can fully utilize multiple cores. But then, can it truly be considered Python parallelization if achieving it requires external extensions that are not written in Python?

Code example:

Python

import threading

def worker():
    print("Thread en cours d'exécution")

if __name__ == "__main__":
		threads = []
		for i in range(5):
		    t = threading.Thread(target=worker)
		    threads.append(t)
		    t.start()
		
		timeout = 10
		still_alive = False
		for t in threads:
		    t.join(timeout)
		    if t.is_alive():
				    still_alive = True
		
		if still_alive:
			raise TimeoutError

What is Asynchronous Programming?

Asynchronous programming, often referred to as “concurrent programming,” allows tasks to be executed in an interleaved manner by relying on rapid context switches. This creates the illusion of tasks being executed in parallel. It achieves this by running I/O-bound operations non-blockingly, using an event-driven model. Unlike multithreading, which relies on multiple concurrent threads, asynchronous programming uses a single thread and coroutines to manage multiple tasks concurrently. This approach is more efficient than multithreading in the context of I/O-bound operations, though it often requires a higher level of development expertise.

In Python, asynchronous programming is supported by libraries such as asyncio, anyio, or trio. Due to the GIL, multithreading offers no real advantage over asynchronous programming for I/O-bound tasks. In such cases, asynchronous programming is generally the better choice, as it ensures greater compatibility with future language developments, especially if the GIL is eventually removed in upcoming versions.

Code example:

Python

import anyio

async def worker():
		await anyio.sleep(1)
		print("Tâche asynchrone terminée")

async def main():
    with anyio.fail_after(10) as timeout:
		    async with anyio.create_task_group() as tg:
		        tg.start_soon(worker)
		        tg.start_soon(worker)
		        tg.start_soon(worker)

if __name__ == "__main__":
		anyio.run(main)

What is Multiprocessing?

The only way to parallelize CPU-bound tasks in Python for a long time has been to use multiprocessing, which involves creating multiple child processes, each with its own memory space and, more importantly, its own interpreter and GIL. Each process operates with its own threads, effectively reduced to a single thread in Python, but they run in parallel, allowing for significant performance gains in this case.

Since this approach is necessary in Python more than in other languages, the multiprocessing module aims to provide an API very similar to that of the threading module. This design attempts to hide many complexities specific to multiprocessing, particularly regarding communication between child processes and resource management. Despite this effort, we believe that trying to transparently equate these two concepts is sometimes tedious or even risky, as they are fundamentally different.

We will also explore other techniques available in Python to work around the GIL. Unfortunately, these techniques remain experimental or not widely adopted at this time.

Code example :

Python

from multiprocessing import Process

def worker():
    print("Processus en cours d'exécution")

if __name__ == "__main__":
		processes = []
		for i in range(5):
		    p = Process(target=worker)
		    processes.append(p)
		    p.start()
		
		timeout = 10
		still_alive = False
		for p in processes:
		    p.join(timeout)
		    if p.is_alive():
				    still_alive = True
				    p.terminate()
		
		if still_alive:
				raise TimeoutError

The Global Interpreter Lock: A Barrier to Parallelization

The Global Interpreter Lock (GIL), as its name suggests, is an internal lock specific to the CPython interpreter that restricts parallel execution of Python threads. Its primary role is to ensure that only one thread can execute Python bytecode at a time, even on multi-core systems.

Other languages also implement similar mechanisms, such as:

Ruby: The Global VM Lock (GVL) is conceptually equivalent to the GIL.
R: Its threading model is also highly limited. Parallel execution is only achievable through native code execution or explicitly adapted packages.
PHP: The Zend Engine implementation, used as an Apache module, employs a similar but slightly more permissive concept.
Lua: Intrinsically single-threaded, Lua doesn’t have an exact equivalent to the GIL, but running multiple Lua scripts in parallel within the same interpreter is impossible.
JavaScript: Like Lua, its execution model is single-threaded, relying on an event loop, making it effectively asynchronous programming.

Why Does the GIL Still Exist?

The GIL significantly simplifies the implementation of the CPython interpreter and Python extensions written in C, which are often not “thread-safe.” By maintaining a global lock, it avoids the need for complex synchronization mechanisms around shared memory management.

Removing the GIL would require profound changes, not only to CPython itself but also to the majority of third-party libraries, whether or not they rely on Python extensions written in other languages. Many of these libraries depend on assumptions tied to the GIL’s presence. These changes would include overhauling memory management models and introducing synchronization mechanisms within CPython. More importantly, it would necessitate revisiting all third-party libraries, some of which may rely on improper multithreading synchronization methods that, until now, have been harmless thanks to the GIL.

What’s Next for the GIL?

In recent years, many efforts have been made to reduce or completely eliminate the limitations imposed by the GIL:

IronPython / Jython

Several alternative Python interpreters have been developed to address some of the limitations of CPython:

IronPython: Designed to run on the .NET Framework, this interpreter does not use a global lock, enabling true multithreaded execution for CPU-bound tasks.
Jython: An implementation of Python on the JVM (Java Virtual Machine), it leverages the JVM’s thread management mechanisms and, therefore, does not rely on a global lock.

These interpreters provide interesting alternatives to CPython, particularly in scenarios where the absence of a global lock is critical for performance. However, many Python extensions written in C are not (or are only partially) supported by these interpreters. This includes a significant number of libraries that contribute to Python’s popularity today, such as pandas or numpy, often making the transition to these interpreters impractical.

Software Transactional Memory (STM) with PyPy

Software Transactional Memory (STM) is an approach that manages synchronization and concurrency without relying on explicit locking mechanisms like the GIL. STM is inspired by transactions in databases: concurrent operations are executed within isolated “transactions,” and at the end of each transaction, a validation mechanism ensures that the changes made did not result in conflicts. If conflicts are detected, the transactions are automatically replayed. This approach enables better utilization of multi-core processors for CPU-bound Python applications.

While CPython does not natively support STM, projects like PyPy have explored this approach. For example, PyPy STM manages concurrency by avoiding the limitations of the GIL while maintaining compatibility with most standard Python programs.

Python

# Les threads ou processus exécutant cette transaction 
# peuvent fonctionner de manière concurrente sans verrou explicite. 

def transaction():
    with stm.atomic():
        x.value += 1
        y.value -= 1

The Major Drawback of STM

The primary disadvantage of STM lies in its complexity and its performance cost when conflicts are frequent or the number of simultaneous transactions is high. Currently, support for STM in the Python ecosystem remains limited and exploratory.

Subinterpreters (Python 3.9)

Since CPython 3.9, via the C API for Python extensions, it has become possible to create subinterpreters that no longer depend on the parent interpreter’s GIL, with each having its own global lock. This feature represents a significant advancement. However, its practical use remains limited: it is primarily designed for Python extensions in C, and its integration requires explicit management of communication and synchronization between subinterpreters. Similar to the multiprocessing approach, shared data must be exchanged through mechanisms like queues, UNIX sockets, or shared memory segments, which increases implementation complexity. As a result, despite its potential, this approach has not yet gained significant adoption in the standard Python ecosystem.

NoGIL (Python 3.13)

PEP 703, titled “Making the Global Interpreter Lock Optional in CPython”, proposes to:

Remove the GIL optionally.
Provide backward compatibility.
Minimize the impact on single-threaded performance.

Without the GIL, there would no longer be a need for multiprocessing or multi-interpreter approaches and their associated complexities. However, as mentioned earlier, removing the GIL requires a complete review of all existing libraries, including both pure Python libraries and Python extensions, which will now need to be thread-safe. While auditing CPython’s standard library alone is a significant challenge, Python’s success has led to hundreds of thousands of libraries available on PyPI. Many of these will likely never be audited or updated.

This change could also affect single-threaded applications, as existing thread safety would need to be reinforced, even though PEP 703 includes optimizations for this scenario. In other words, while this change is a major step forward, we are still far from being able to fully benefit from it.

Challenges of Multiprocessing

Although multiprocessing circumvents the GIL and enables effective parallelization of CPU-bound tasks, it introduces several challenges for which the multiprocessing module provides some tools:

Communication Between Processes and Data Management

Unlike a single-process approach, data must be explicitly transferred between child processes and the parent process, as they do not share the same memory space.

To achieve this, one or more of the following techniques can be used:

Shared Memory

The multiprocessing module provides tools such as Value and Array for sharing data between processes, which helps minimize data copying and improve performance.

Python

import time

from multiprocessing import Process, Value


def writer(shared_value):
    """Processus qui écrit dans la valeur partagée."""
    for i in range(5):
        shared_value.value = i * 3  # Modifie la valeur partagée
        print(f"[Writer] Écrit : {shared_value.value}")
        time.sleep(0.5)  # Simule un délai entre les écritures

def reader(shared_value):
    """Processus qui lit la valeur partagée."""
    for i in range(5):
        print(f"[Reader] Lu : {shared_value.value}")
        time.sleep(0.7)  # Simule un délai entre les lectures

if __name__ == "__main__":
    shared_value = Value('d', 0.0)  # 'd' signifie double (c type: double)

    writer_process = Process(target=writer, args=(shared_value,))
    reader_process = Process(target=reader, args=(shared_value,))

    writer_process.start()
    reader_process.start()

    writer_process.join()
    reader_process.join()

    print("Communication via Value terminée.")

Sockets
Sockets enable data exchange using network primitives, either between remote machines or on the same machine. In the latter case, it is preferable to use UNIX sockets, which provide better performance by bypassing the IP layer.

Python

import os
import socket
import multiprocessing


def child_sender(socket_path):
    """Processus qui envoie des messages."""
    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_sock:
        client_sock.connect(socket_path)
        for i in range(5):
            message = f"Message {i} depuis le processus sender"
            print(f"[Sender] Envoi: {message}")
            client_sock.sendall(message.encode('utf-8'))
        client_sock.sendall(b"END")  # Signal de fin
        print("[Sender] Tous les messages ont été envoyés.")

def child_receiver(socket_path):
    """Processus qui reçoit des messages."""
    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as server_sock:
        server_sock.bind(socket_path)
        server_sock.listen(1)
        print("[Receiver] En attente de connexion…")
        conn, _ = server_sock.accept()
        with conn:
            print("[Receiver] Connexion acceptée.")
            while True:
                data = conn.recv(1024)
                if not data or data == b"END":
                    break
                print(f"[Receiver] Reçu: {data.decode('utf-8')}")
        print("[Receiver] Fin de la réception.")

if __name__ == "__main__":
    socket_path = "/tmp/unix_socket_example"

    if os.path.exists(socket_path):
        os.remove(socket_path)

    receiver_process = multiprocessing.Process(target=child_receiver, args=(socket_path,))
    sender_process = multiprocessing.Process(target=child_sender, args=(socket_path,))

    receiver_process.start()
    sender_process.start()

    sender_process.join()
    receiver_process.join()

    if os.path.exists(socket_path):
        os.remove(socket_path)

    print("Communication terminée.")

Queues and Pipes
For simpler communication, the multiprocessing module provides queues (multiprocessing.Queue), which offer a straightforward producer/consumer-style interface, and pipes (multiprocessing.Pipe), which provide a higher-level interface similar to sockets.

Python

import multiprocessing as mp
import time


def producer(queue: mp.Queue):
    """Produit des données et les place dans la queue."""
    for i in range(5):
        data = f"Message {i}"
        print(f"Producer: envoi {data}")
        queue.put(data)  # Envoie les données dans la queue
        time.sleep(1)  # Simule un délai pour produire les données
    queue.put(None)  # Indique la fin de la production

def consumer(queue: mp.Queue):
    """Consomme les données de la queue."""
    while True:
        data = queue.get()  # Récupère un élément depuis la queue
        if data is None:  # Si le message est None, on arrête
            print("Consumer: fin des données")
            break
        print(f"Consumer: reçu {data}")
        time.sleep(2)  # Simule un traitement des données

if __name__ == "__main__":
    queue = mp.Queue()

    producer_process = mp.Process(target=producer, args=(queue,))
    consumer_process = mp.Process(target=consumer, args=(queue,))

    producer_process.start()
    consumer_process.start()

    producer_process.join()
    consumer_process.join()

    print("Tous les processus sont terminés.")

Synchronization

Just like in multithreading approaches, primitives such as locks, semaphores, and events are necessary to coordinate processes. However, it is important to keep in mind that these primitives, while easy to use, also rely on the inter-process communication techniques mentioned earlier. Needless to say, these primitives have a much higher cost than their counterparts that share the same memory space.

And What About Memory?

Unlike threads, child processes do not share their memory space, which inevitably leads to increased memory usage

A Concrete Example: Logging

One common pitfall when starting with the multiprocessing approach is using the logging module. Unfortunately, this is one of those subtle issues that might not come to mind initially when optimizing performance. And yet, it often becomes essential when building a serious application.

To convert a single-threaded Python application with a logger into one using the multiprocessing approach, it is necessary to configure the logger in a specific way to avoid concurrent access issues. This ensures consistent and complete logs, particularly when they are being written to disk. Below is an example of an additional configuration to include in your application:

Python

import contextlib
import logging
import multiprocessing as mp

class _LogQueueHandler(logging.Handler):
    def emit(self: Self, record: LogRecord) -> None:
        pass

    def handle(self: Self, record: LogRecord) -> bool:
        logger = getLogger() if record.name == "root" else getLogger(record.name)
        if logger.isEnabledFor(record.levelno):
            logger.handle(record)
        return True

# Chaque processus fils doit initialiser son propre logger de la manière suivante
def _init_logger_child(q_logging: mp.Queue, root_log_lvl: int, loggers_config: dict):
		log_config = {
        "disable_existing_loggers": False,
        "handlers": {
		        "queue_hdl": {
		            "class": "logging.handlers.QueueHandler",
                "queue": q_logging
            }
        },
        "root": {
             "handlers": ["queue_hdl"],
             "log_level": root_log_lvl
        },
        "loggers": loggers_config
     }
     logging.config.dictConfig(log_config)

# Le parent peut, de son côté, utiliser ce contextmanager pour transférer les
# messages de log issus de ces processus fils
@contextlib.contextmanager
def _init_logger_parent_listener(q_logging: mp.Queue):
		listener = logging.handlers.QueueListener(q_logging, _LogQueueHandler())
		try:
				listener.start()
				yield
		finally:
				listener.stop()

if __name__ == '__main__':
		# Initialiser le logger principal
		logging.basicConfig()
		
		q_logging = mp.Queue()
		
		def child_process(q_logging: mp.Queue, root_log_lvl: int, loggers_config: dict):
				_init_logger_child(q_logging, root_log_lvl, loggers_config)
				# for job in todo_jobs:
				#     job()
		
		timeout = 10
		with _init_logger_parent_listener(q_logging):
				children = []
				for i in range(10):
						child = mp.Process(target=child_process, args=(q_logging, getLogger().getEffectiveLevel(), {})
						child.start()
						children.append(child)
				
				for child in children:
						child.join(timeout)
						if child.is_alive():
								child.terminate()

With this configuration, each log message generated by the child processes is sent through a queue to the parent process, which is the only one that actually logs messages in the application.

Conclusion

The choice between multithreading, asynchronous programming, and multiprocessing depends on the specific needs of your application.

In Python, the limitations of the GIL and the complexities of multiprocessing show that parallelization is not always straightforward. It’s easy to assume that using a multithreading approach will automatically improve performance, but in reality, it often offers no advantage, or even less, compared to an asynchronous approach. However, future developments, especially the removal of the GIL, will undoubtedly transform how parallelized applications are developed in Python.

For now, a deep understanding of existing approaches remains essential to optimizing your Python applications.

My Recommendations

For long-running calculations or persistent applications:

I/O-bound tasks: Favor asynchronous programming.
Specifically, I recommend using the anyio library, which allows your projects to remain compatible with both asyncio and trio while maintaining their robustness. This approach also ensures that your code will stay robust regardless of the Python implementation you use, with or without the GIL.
CPU-bound tasks: Use multiprocessing to work around the GIL.
Consider using a more advanced third-party module like joblib, which provides a high-level interface for job management with features like caching. Don’t hesitate to rely on a higher-level library than the default multiprocessing module to avoid many issues inherent to this approach. Some libraries, for instance, handle problems like the one encountered with the logging module mentioned earlier, either directly or indirectly.

For short-running calculations or ephemeral applications:

If you’re looking for a quick way to develop and test different approaches, you might consider using the concurrent.futures module. It provides a relatively simple abstraction for parallelization and allows you to quickly choose between multithreading and multiprocessing approaches.

Python

import time

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed

def worker(n):
    """Simule une tâche prenant du temps."""
    print(f"Thread {n} : démarrage")
    time.sleep(2)  # Simule un travail
    print(f"Thread {n} : terminé")
    return f"Résultat du thread {n}"

if __name__ == "__main__":
    # Utiliser:
		#  – ThreadPoolExecutor pour l'approche multithreading
		#  – ProcessPoolExecutor pour l'approche multiprocessing

    # Crée un pool de threads avec 3 threads maximum		
    with ThreadPoolExecutor(max_workers=3) as executor:
        # Soumet les tâches au pool d'exécution
        futures = [executor.submit(worker, i) for i in range(5)]

        # Récupère les résultats des futures au fur et à mesure qu'elles se terminent
        for future in as_completed(futures):
            print(future.result())

    print("Toutes les tâches sont terminées.")

Parallelization in Python

Multithreading vs Asynchronous Programming vs Multiprocessing

What is Multithreading?

Code example:

What is Asynchronous Programming?

Code example:

What is Multiprocessing?

Code example :

The Global Interpreter Lock: A Barrier to Parallelization

Why Does the GIL Still Exist?

What’s Next for the GIL?

IronPython / Jython

Software Transactional Memory (STM) with PyPy

Communication Between Processes and Data Management

Synchronization

Conclusion

My Recommendations

For long-running calculations or persistent applications:

For short-running calculations or ephemeral applications:

Additional Links

You may like to read:

Help & Info