Threads, Locks, Thread Safety & Synchronization

Python’s threading module spawns native OS Kernel Threads. Because all threads inside a single Python process share the same heap memory space, concurrently modifying shared data structures without synchronization primitives causes severe Race Conditions and memory corruption.

This chapter details native thread execution, race conditions, synchronization primitives (Lock, RLock, Semaphore, Event, Condition), and thread-safe data structures (queue.Queue).

1. Race Conditions on Shared Memory

When multiple threads read and write to a shared variable concurrently, the order of execution becomes non-deterministic:

Race Condition Timeline:

Thread A                                           Thread B
-----------------------------------------------------------------------------------
1. Read balance = 100
                                                   2. Read balance = 100
3. Calculate: 100 + 50 = 150
                                                   4. Calculate: 100 - 30 = 70
5. Write balance = 150
                                                   6. Write balance = 70  <-- OVERWRITES THREAD A WRITE!
(Final Balance: 70! Expected Balance: 120! DATA CORRUPTION HAS OCCURRED!)

Even simple Python operations like counter += 1 are not thread-safe because they compile into multiple bytecode opcodes (LOAD_FAST, BINARY_OP, STORE_FAST), allowing thread context switches to occur mid-operation!

2. Synchronization Primitives

CPython provides OS-level synchronization primitives via the threading module:

import threading

# 1. Mutex Lock (threading.Lock)
lock = threading.Lock()

def safe_increment():
    with lock:  # Acquires lock; guarantees mutually exclusive execution!
        global counter
        counter += 1

Primitive Overview:

• Lock (Mutex): Standard mutual exclusion lock. Can only be acquired by one thread at a time.
• RLock (Re-entrant Lock): Can be acquired multiple times by the SAME thread without deadlocking. Essential for recursive functions or nested method calls.
• Semaphore(value): Maintains a counter allowing up to value threads to enter a critical section simultaneously (useful for rate-limiting pool connections).
• Event: Thread signaling mechanism (event.wait(), event.set()).
• Condition: Complex synchronization allowing threads to wait until a specific boolean state condition is signaled (cond.wait(), cond.notify_all()).

3. Thread-Safe Queues (queue.Queue)

Never build custom list queues with manual locks for producer-consumer workflows. Use queue.Queue:

• Thread-Safe: Uses internal Lock and Condition variables to synchronize put() and get() calls safely across threads.
• Backpressure Support: queue.Queue(maxsize=100) blocks producer threads when the queue is full, preventing unbounded memory growth.

import queue
import threading

work_queue = queue.Queue(maxsize=50)

def consumer():
    while True:
        item = work_queue.get()
        if item is None:  # Sentinel exit signal
            break
        process(item)
        work_queue.task_done()

4. Production Deadlock Prevention

A Deadlock occurs when Thread A holds Lock 1 and waits for Lock 2, while Thread B holds Lock 2 and waits for Lock 1.

Deadlock Prevention Invariant:

Always acquire multiple locks in a strict, consistent global order across all threads (e.g. always acquire lock_a before lock_b).