Property-Based Testing & Hypothesis Fuzzing

Example-based unit tests (assert add(2, 3) == 5) only test inputs that developers explicitly remember. Property-Based Testing (PBT) inverts this paradigm by testing invariant mathematical properties against hundreds of automatically generated randomized inputs using hypothesis. When a failure occurs, hypothesis executes a Shrinking Engine to find the minimal reproducing test case.

This chapter details Property-Based Testing vs Example-Based Testing, hypothesis data strategies, shrinking engine mechanics, and stateful testing.

1. Example-Based Testing vs. Property-Based Testing

Testing Paradigm Comparison:

1. EXAMPLE-BASED TESTING (Standard Unit Tests):
   Developer picks 3 specific inputs: add(0, 0), add(2, 3), add(-1, 1).
   - Limitation: Misses edge cases (Unicode surrogates, NaN, Integer overflows, Empty byte strings).

2. PROPERTY-BASED TESTING (Hypothesis Fuzzing):
   Developer asserts general invariants:
    - Invariant 1: encode(decode(x)) == x (Round-trip)
    - Invariant 2: len(sort(x)) == len(x) (Invariance)
   Hypothesis generates 100+ random inputs testing boundary limits!

2. Defining Properties with hypothesis Strategies

Use @given and strategies from hypothesis.strategies to generate test data:

from hypothesis import given, strategies as st

# Test Invariant: Sorting a list preserves length and maintains sorted order
@given(st.lists(st.integers()))
def test_sort_properties(xs: list[int]):
    sorted_xs = sorted(xs)

    # Invariant 1: Length is preserved
    assert len(sorted_xs) == len(xs)

    # Invariant 2: Result is ordered
    assert all(sorted_xs[i] <= sorted_xs[i + 1] for i in range(len(sorted_xs) - 1))

Core Strategies (st):

• st.integers(), st.floats(allow_nan=False), st.text(), st.binary()
• st.lists(st.text(), min_size=1)
• st.builds(UserModel, name=st.text(), age=st.integers(18, 100))

3. The Hypothesis Shrinking Engine

When hypothesis discovers a failing input (e.g., a 500-character complex string containing special characters that crashes your parser), it doesn’t just print the 500-character blob.

It executes Automated Shrinking:

Hypothesis Shrinking Loop:

[ Failing Test Input Found! (e.g. List of 250 complex integers) ]
                             |
                             v (Shrinking Engine iteratively simplifies input)
  - Step 1: Reduce list length (250 -> 100 -> 10 -> 2 -> 1)
  - Step 2: Simplify integer values (984214 -> 100 -> 1 -> 0)
  - Step 3: Verify minimal failing input!
                             v
[ Output Minimal Reproducing Example: test_func([-1]) ]

Shrinking simplifies complex failing inputs down to the absolute smallest minimal reproducing test case (e.g. [0] or ""), making debugging trivial!

4. Production Invariants for Property-Based Testing

When writing property-based tests, look for 4 common invariant patterns:

1. Round-Trip / Metamorphic: decode(encode(data)) == data (Serializers, Encoders, Compression).
2. Idempotency: f(f(x)) == f(x) (Formatters, Cleaning scripts, Normalizers).
3. Test Oracle: custom_fast_sort(x) == python_stdlib_sort(x) (Comparing optimized algorithms against trusted reference implementations).
4. Invariance: count(x) == count(process(x)) (Operations that must preserve size or count).