Property-based testing: How to find bugs your unit tests miss

Property-based testing is a software testing technique that checks whether a program satisfies general rules, called properties, across many automatically generated inputs. Instead of writing tests for individual examples, developers define how their code should behave and let a testing framework search for inputs that break those rules.

As AI coding agents take on more development work, the ability to validate code against clearly defined requirements is becoming increasingly valuable. Agents can generate and modify code quickly, but getting those changes validated and into production remains a challenge. Property-based testing provides a way to check implementations against broader behavioral requirements, uncovering edge cases that neither a developer nor an agent anticipated. Researchers are already using coding agents to generate property-based tests and find bugs in widely used software libraries.

With property-based testing, you can spend less time guessing which inputs might cause problems and more time defining what your software needs to guarantee. In this guide, you’ll learn how to write those guarantees as executable tests, use automatically generated inputs to uncover hidden bugs, and build continuous validation into your CI/CD pipeline.

How does property-based testing work?

Conventional unit tests check whether specific inputs produce expected outputs. Property-based tests check broader rules that should hold across many different inputs, even when the exact output changes. For example, instead of asserting that sorting [3, 1, 2] produces [1, 2, 3], a property-based test might verify that any sorted list is in ascending order and contains exactly the same elements as the original, including duplicates.

A property-based testing framework such as Hypothesis can generate hundreds of different lists and check whether each one satisfies both rules. It might try empty lists, duplicates, negative numbers, or much larger collections than a developer would ordinarily include in a unit test. The goal is to find inputs that violate the property, exposing bugs that hand-picked examples might miss.

Example-based unit tests	Property-based tests
Test specific, hand-selected inputs	Generate inputs from a defined range
Check expected outputs for each example	Check rules that should hold across different inputs
Verify known behaviors and edge cases	Explore cases the developer may not have anticipated
Report the input that failed	Can simplify failing inputs to aid debugging

Property-based testing doesn’t replace unit testing. Both approaches can test individual functions, and they’re often used together. The main difference is how developers specify expected behavior and how the tests explore possible inputs.

Property-based testing frameworks automate three important parts of the testing process:

1. Generate inputs. You define the kinds of values your code should handle, such as integers, strings, lists, or more complex data structures. The framework produces different inputs within those constraints.
2. Check a property. The framework runs your code with each generated input and checks whether an assertion holds.
3. Shrink failures. If a generated input causes a failure, the framework tries progressively simpler inputs that reproduce the same problem.

Shrinking makes failures easier to investigate by reducing complicated inputs to smaller examples that still reproduce the bug. The diagram below shows how generation, checking, and shrinking work together.

The important output of this process is a counterexample: a specific input that breaks a rule the code is supposed to follow. By reducing that input to its simplest form, the framework gives developers a reproducible failure they can investigate and fix.

Property-based testing example: Finding a bug with Hypothesis

Suppose you’re building a function that normalizes search queries before processing them. It removes leading and trailing spaces and replaces repeated spaces between words with a single space, so variations in formatting produce consistent results.

A simple Python implementation might look like this:

# normalizer.py

def normalize_query(query):
    return query.strip().replace("  ", " ")

The strip() method removes whitespace from the beginning and end of the string, while replace() converts pairs of spaces into single spaces.

For ordinary inputs, the function appears to work correctly.

normalize_query("hello  world")
# "hello world"

normalize_query("  hello world  ")
# "hello world"

Both inputs return the expected result. For the cases we’ve checked so far, the implementation appears to work.

Start with conventional unit tests

We can write a few tests to check the expected behavior.

from normalizer import normalize_query

def test_normalize_query():
    assert normalize_query("hello  world") == "hello world"
    assert normalize_query("  hello world  ") == "hello world"
    assert normalize_query("hello world") == "hello world"

All three assertions pass, but they cover only the spacing patterns we anticipated. Other combinations could behave differently, and choosing more examples still means guessing which inputs to test.

Define a property and generate inputs

Rather than trying to anticipate every spacing pattern, we can test a rule that should hold for any valid input: normalizing a string twice should produce the same result as normalizing it once.

This property is called idempotence. A normalization function should finish its work on the first pass, so applying it again shouldn’t change the output.

With Hypothesis, we can test this property against automatically generated strings.

First, install the dependencies:

pip install pytest hypothesis

Then create a test file:

# tests/test_normalizer.py

from hypothesis import given, strategies as st
from normalizer import normalize_query

@given(st.text(alphabet="abc ", max_size=40))
def test_normalization_is_idempotent(query):
    normalized = normalize_query(query)
    assert normalize_query(normalized) == normalized

The @given decorator tells Hypothesis to generate inputs for the test.

The st.text() strategy generates strings from the specified alphabet. Here, we’ve limited the characters to a, b, c, and a space, with a maximum length of 40 characters. Hypothesis can produce empty strings, individual words, and strings containing different combinations of letters and spaces.

Run the test with:

python -m pytest -v tests/test_normalizer.py

Hypothesis generates strings, runs the normalization function, and checks whether normalizing the output again changes it.

It doesn’t need to know what the normalized string should look like for every possible input. The assertion describes a general rule that the function must satisfy.

Understand the failure

The test exposes the problem with consecutive spaces. A small counterexample that demonstrates the failure is:

query = "a   a"

normalize_query(query)
# "a  a"

normalize_query(normalize_query(query))
# "a a"

The first pass reduces three consecutive spaces to two. The second pass reduces the remaining pair to one.

The function violates the idempotence property because it hasn’t fully normalized the string after the first pass.

Hypothesis attempts to shrink failing inputs, removing unnecessary characters and simplifying the input while preserving the failure. For example, a complicated failure could be reduced to a much smaller counterexample:

Original failing input: "find     old   builds"
Shrunk counterexample: "a   a"

Both inputs violate the idempotence property, but the smaller example makes the underlying problem easier to identify.

By reducing the input to just a few characters, Hypothesis makes the problem much easier to spot. We can see exactly why the first pass leaves an extra space and why the second pass changes the output.

Minimized counterexamples are particularly useful when coding agents are debugging their own implementations. Instead of passing a large, complicated failure back to an LLM, developers can provide a concise example that isolates the problem. The agent has less irrelevant information to process and a clearer target for its next attempt.

Fix the implementation and rerun the test

The problem comes from how str.replace() handles repeated spaces. It replaces each occurrence of two consecutive spaces, but a single pass doesn’t necessarily collapse a longer sequence into one space.

We can fix the implementation with a regular expression that matches one or more consecutive spaces.

# normalizer.py

import re

def normalize_query(query):
    return re.sub(r" +", " ", query.strip())

The expression r" +" matches any sequence of one or more ordinary spaces. The replacement converts the entire sequence into a single space, regardless of its original length.

Now the previously failing input produces the expected output:

normalize_query("a   a")
# "a a"

normalize_query(normalize_query("a   a"))
# "a a"

The function satisfies our idempotence requirement for this case, and we can rerun the property-based test to explore other generated inputs.

We can also expand the test to generate arbitrary strings rather than restricting it to a small alphabet.

@given(st.text())
def test_normalization_is_idempotent(query):
    normalized = normalize_query(query)
    assert normalize_query(normalized) == normalized

Hypothesis can now try a much wider variety of strings, including characters and spacing patterns that weren’t covered by the original tests.

The bug also provides a useful case to preserve as a conventional unit test, preventing the same problem from returning:

def test_repeated_spaces_regression():
    assert normalize_query("a   a") == "a a"

The unit test checks the specific bug that was fixed, while the property-based test keeps searching for other inputs that change when normalized a second time. Both are useful because idempotence alone doesn’t guarantee correct behavior. A function that returns every string unchanged would also pass the property-based test, even though it doesn’t normalize anything.

What kinds of properties can you test?

Idempotence is one useful property, but there are several others you can look for when testing your code.

Property	What it checks	Example
Idempotence	Repeating an operation doesn’t change its result	normalize(normalize(x)) == normalize(x)
Round trip	An operation and its inverse preserve information	decode(encode(x)) == x
Invariant	A condition remains true after an operation	Sorting preserves the number and frequency of list elements
Relationship	Different operations produce equivalent outcomes	Reversing a list twice returns the original list

Properties are especially helpful when calculating the exact expected result would be difficult, but certain relationships are easy to specify.

For example, you may not know the expected output of a complicated data-transformation function for every possible input. You might still be able to verify that it preserves the number of records, maintains valid relationships, or produces the same result when an operation is repeated.

Choosing the right property requires care. A weak property might pass even when the implementation is incorrect. A sorting function that deletes half the input could satisfy an assertion that its output is ordered. Checking that the output also preserves the original elements would catch the problem.

Property-based testing vs. fuzz testing and mutation testing

Property-based testing overlaps with several other testing techniques, but each offers a different way to find defects or evaluate test quality.

Technique	How it works	Primary purpose
Example-based testing	Runs selected inputs and checks expected outputs	Verify known behaviors and edge cases
Property-based testing	Generates inputs and checks general behavioral rules	Find inputs that violate specified properties
Fuzz testing	Feeds varied, often unexpected inputs into software	Find crashes, invalid behavior, and robustness or security issues
Mutation testing	Deliberately changes source code and runs existing tests	Find weaknesses in the test suite

Property-based testing and fuzz testing aren’t mutually exclusive. Both explore input spaces automatically, and some fuzzing tools can check application-specific properties rather than looking only for crashes or exceptions.

The distinction is often in how developers approach the test. Property-based testing typically begins with an explicit behavioral rule, while many fuzzing workflows emphasize generating inputs that exercise unusual execution paths and reveal robustness problems.

Mutation testing approaches the problem from another direction. Instead of changing the inputs, it deliberately modifies the program to check whether existing tests detect the change.

These techniques can complement one another. A property-based test might discover a previously unknown edge case, while mutation testing can help determine whether the suite would detect particular defects in the implementation.

When should you use property-based testing?

Property-based testing is most valuable when a function accepts many possible inputs but has clearly defined behavioral requirements.

Good candidates include:

• Parsers and serializers: Check that valid inputs survive conversions without losing or changing information.
• Validation functions: Check that accepted and rejected values follow the specified rules, including boundary conditions.
• Data transformations: Verify that normalization, filtering, sorting, or aggregation preserves required properties.
• Numerical and algorithmic code: Test mathematical relationships, bounds, and invariants across generated inputs.
• Stateful systems: Generate sequences of operations and verify that valid state transitions preserve important constraints.

In each case, you can describe how the code should behave without specifying the expected output for every possible input.

Property-based testing is less straightforward when correct behavior depends on complex external systems, subjective requirements, or expensive side effects. Generating hundreds of test inputs for a function that sends emails or makes live payment requests would require substantial isolation and test setup.

The technique also depends on developers specifying the right properties. A test can pass consistently while checking an incomplete or incorrect requirement. Property-based testing expands the range of inputs explored, but it doesn’t eliminate the need to understand what the software is supposed to do.

Property-based testing for agent-generated code

Property-based testing is especially useful in agent-driven development, where coding agents can produce implementations faster than developers can anticipate every possible failure. Rather than relying solely on the examples an agent includes in its own tests, developers can define properties that express the intended behavior and use generated inputs to check whether the implementation satisfies them. Property-based testing is a natural complement to intent-driven development, providing a way to verify behavioral requirements even when agents determine how to implement them.

Coding agents can also help identify properties and write the tests themselves. Developers still need to verify that those properties reflect the actual requirements. An agent that misunderstands a requirement could produce code and tests that agree with each other while both are wrong. Reviewing the properties and running them in CI provides an additional layer of validation as agents produce and modify code.

Popular property-based testing tools

You can add property-based testing to an existing test suite using frameworks such as:

• Hypothesis (Python) integrates with pytest and provides strategies for generating primitive values, collections, structured data, and more complex inputs. It also supports shrinking, stateful testing, and reusable test settings.
• fast-check (JavaScript and TypeScript) provides input generators, property assertions, and shrinking, with integrations for common JavaScript test runners.
• jqwik (Java) integrates with the JUnit ecosystem and supports property definitions, generated inputs, shrinking, and stateful testing.

These frameworks work with existing test runners, so property-based tests can live alongside conventional tests rather than requiring a separate testing process.

Running property-based tests in CI/CD

Once a property-based test catches a bug locally, you want it to keep checking that behavior as the code changes. Running the same tests in CI helps prevent regressions and gives the framework more opportunities to discover unexpected failures.

Property-based tests can run alongside conventional unit tests in the same CI job. For our Python example, we can configure CircleCI to install pytest and Hypothesis, execute the tests, and collect the results.

Create a requirements.txt file containing:

pytest
hypothesis

Then add the following configuration to .circleci/config.yml:

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/python:3.12
    steps:
      - checkout

      - run:
          name: Install dependencies
          command: pip install -r requirements.txt

      - run:
          name: Run tests
          command: |
            mkdir -p test-results
            python -m pytest -q --junitxml=test-results/junit.xml

      - store_test_results:
          path: test-results

workflows:
  test:
    jobs:
      - test

With the project connected to CircleCI, the pipeline runs the tests when code changes are pushed. Each execution allows Hypothesis to generate inputs and check the properties against the current implementation.

If a property fails, pytest marks the test as failed. The CircleCI job fails as well, and the job output includes the counterexample reported by Hypothesis.

When the property fails in CI, the job output shows the counterexample Hypothesis found.

The --junitxml option produces a JUnit XML report, which CircleCI collects through the store_test_results step. Developers can inspect passing and failing tests in the CircleCI Tests tab and use the job output to investigate the failure.

The example-based test passes on the buggy normalizer, while the property-based test finds the repeated-space bug.

For more on configuring reports, see the CircleCI test results documentation.

Balance test coverage and execution time

By default, Hypothesis targets 100 generated examples per property-based test, although failures, shrinking, and other settings can affect how many times a test executes.

For small, isolated functions, these tests can run quickly enough to include in every pull-request pipeline. More expensive properties may need tighter limits so testing doesn’t delay feedback.

You can control the number of generated examples using Hypothesis settings:

from hypothesis import given, settings, strategies as st

@settings(max_examples=1000)
@given(st.text())
def test_normalization_is_idempotent(query):
    normalized = normalize_query(query)
    assert normalize_query(normalized) == normalized

Increasing the number of examples gives Hypothesis more opportunities to explore the input space, but also increases test execution costs.

For larger projects, a practical strategy is to run a bounded set of property-based tests on every pull request and schedule more extensive runs against the main branch. Hypothesis settings profiles can help configure different execution budgets for each environment.

Because CI environments may not preserve Hypothesis’s local database of previous failures between runs, important counterexamples should also be captured in reproducible tests.

Property-based testing best practices

Writing effective property-based tests requires more than generating large quantities of data. The value comes from matching the generated inputs and assertions to meaningful software requirements.

A few practices make these tests more useful and maintainable.

1. Start with simple, high-value properties. Round trips, idempotence, and invariants are good starting points because developers can usually state the expected behavior clearly.

2. Generate valid and relevant inputs. Use strategies to represent the values your code is designed to handle. Include invalid inputs when testing rejection or error handling, but don’t mistake unsupported inputs for implementation bugs.

3. Test meaningful behavior. A property should catch defects that matter. Avoid assertions that simply restate implementation details or check only that a function returns a value.

4. Preserve important counterexamples. When a generated input reveals a real bug, retain a targeted regression test so the same behavior remains covered.

5. Keep tests reproducible. Prefer isolated functions and controlled dependencies. Use the framework’s reproduction and example-replay features to investigate failures rather than ignoring intermittent results.

6. Manage execution costs. Start with fast-running properties, set appropriate example limits, and reserve expensive tests for longer-running workflows when necessary.

It’s also important to keep conventional tests. An idempotence property can confirm that normalizing a string twice produces the same output, but it cannot by itself establish that the output is correct. Specific examples remain valuable for verifying external requirements and independently known expected outputs.

Put property-based testing to work across the development loop

Property-based testing gives you a way to turn behavioral requirements into automated checks that keep looking for unexpected failures as your code evolves. You don’t need to overhaul your test suite to benefit. Start with one function, define a property it must satisfy, and use generated inputs to find cases your existing tests might miss.

Property-based tests are especially valuable while coding agents are actively developing and revising code. Chunk sidecars provide isolated environments where agents can test their working changes, using counterexamples to identify and fix problems before code is pushed. Once changes reach CI, CircleCI pipelines can run the same properties alongside broader test suites, providing independent validation before changes are merged.

By checking the same behavioral requirements throughout development, you give agents the feedback they need to improve their implementations and developers greater confidence in the code they deliver.

Get started with CircleCI to make property-based testing part of your continuous validation workflow.