---
url: https://thekevinscott.github.io/testing-conventions/explanation/mutation.md
description: >-
  Why mutation testing is the verification rung coverage can't reach, why the
  gate is binary, not a score, and the engines that run it.
---

# Mutation

Coverage tells you which lines your tests *ran*. **Mutation testing** tells you whether your tests
would *notice if those lines were wrong* — the question coverage can't answer. It's the
verification rung above the [coverage floor](./coverage): the top of the
[unit ladder](./#the-unit-ladder-exist-→-run-→-verify). The [workflow](../reference/workflow) runs it
on every pull request, diff-scoped to the changed lines.

## Coverage runs your code; mutation checks your tests

A coverage report says a line executed during a test. It says nothing about whether any test
*asserted* on the result. A test that calls your function and checks nothing still covers every line
it touches — 100% coverage, zero verification. Mutation testing closes that gap by deliberately
breaking the code and watching what your tests do.

A **mutant** is your code with one small, deliberate fault introduced — `+` becomes `-`, `>=` becomes
`>`, `return x` becomes `return null`. The tool makes the mutant, runs your unit suite, and records
the outcome:

* Tests **fail** → the mutant is **killed**. Good: a test caught the bug.
* Tests **pass** → the mutant **survived**. Bad: a real, bug-shaped change slipped past every test.

Unlike coverage, the result can't be satisfied by executing code — only by asserting on it.

## The example coverage can't catch

```ts
// adult.ts
export function isAdult(age: number): boolean {
  return age >= 18;
}
```

```ts
// adult.test.ts
import { expect, test } from 'vitest';
import { isAdult } from './adult';

test('isAdult', () => {
  expect(isAdult(20)).toBe(true);
  expect(isAdult(10)).toBe(false);
});
```

That test has **100% line and branch coverage** — there isn't even a branch to miss. Now mutate `>=`
into `>`:

```ts
return age > 18;
```

`20` and `10` still return the same answers, so every assertion still passes — the mutant
**survives**. Coverage said "perfect"; mutation testing found a real off-by-one at the boundary. The
only thing that kills it is the assertion you didn't write:

```ts
expect(isAdult(18)).toBe(true); // now `>` and `>=` diverge
```

## Why a number won't do: equivalent mutants

You might expect a mutation-score floor like "≥ 95%". Mutation score — the fraction of mutants a
suite kills — is a poor *target*, for one dominant reason: **equivalent mutants**. Some mutations
produce code that is *semantically identical* to the original (`x = x + 0`, a boundary that can't be
reached). No test can ever kill them — and detecting which survivors are equivalent is formally
undecidable. So for any given file you don't actually know what the maximum achievable score is; a
fixed floor can be unreachable through no fault of your tests. It's the "last 20% is defensive
trivia" problem from coverage, but worse — you often can't tell a real gap from an equivalent mutant
without reading it. A score also isn't comparable across the per-language engines.

So the gate isn't a percentage. It's **binary and diff-scoped**:

> **No unexplained surviving mutant on the lines a change touched.**

A survivor you've confirmed is equivalent or deliberately defensive gets a reasoned, line-scoped
[exemption](../guide/configure#exempt-specific-lines-coverage-mutation) — the same mechanism every
other check uses; every other survivor must be killed. It's the mutation analog of the coverage
philosophy — *"zero survivors except what you exempted with a reason"*, not *"hit a number you
can't reach"* — and it ports cleanly across languages, where a score does not. Diff-scoping is
what makes a binary gate tractable: whole-tree mutation is too slow to run, so the job runs on
pull requests only, over the `<base>...HEAD` changed lines.

## The last mile of a subprocess

One survivor class is not a gap and not an equivalent mutant: the function that actually spawns.
cargo-mutants replaces a whole function body with a plausible return value, so a wrapper around
`cargo llvm-cov` yields `Ok(String::new())` — and no unit test can tell the difference, because
calling the wrapper at all runs the subprocess. The tiers that would notice are `tests/` and the
e2e suite, and the unit tier deliberately does not run them.

The answer is not an exemption per wrapper. An exemption records a judgement, and a hundred copies
of "this is a subprocess seam" records a missing rule. Nor is it "inject the `Command`" — that is
the right move one level up, and it is what makes the argv assembly assertable, but injection
bottoms out: something has to hold the `.output()` call. So the rule names that bottom. A
**branch-free** function that runs a `Command`, directly or by delegating to one in the same file,
has no unit-tier contract for its own body, and cargo-mutants' `FnValue` mutant on it is dropped.

Branch-free is what keeps this from swallowing real gaps. A body that decides *and* spawns keeps
every mutant, so the pressure to split it stays: move the decision into a function that returns the
argv, assert that, and what is left delegates and qualifies. The gate rewards the split rather than
waiving the function.

This is Rust-only, and deliberately. It keys on a cargo-mutants genre; Stryker and cosmic-ray
produce their own mutators over these wrappers, and whether the same argument holds for each is a
question about that engine's operator set, not about the language. A TypeScript or Python wrapper
that hits the equivalent still takes a line-scoped exemption.

## A Python annotation is metadata, not behavior

The second survivor class the tool recognizes is an annotation. `def f(x: dict | None) -> int |
None` carries two `|` operators, and cosmic-ray rewrites each of them the way it rewrites any
other: `dict + None`, `dict & None`, eleven variants per site. None of them runs. An annotation is
type metadata, and the interpreter is free never to evaluate it — `from __future__ import
annotations` (PEP 563) keeps the whole annotation a string, and from Python 3.14 on laziness is
the default (PEP 649), where an annotation is evaluated only if something asks for it. A mutation
of code that never executes cannot fail a test, so every one of those mutants survives a suite
that pins the function's behavior completely.

That is the textbook equivalent mutant, and the conclusion the gate draws from it is the whole
reason the gate is binary: a survivor no test can kill is not a measurement. So the tool does not
hand this one to the consumer as an exemption to write. An exemption records a judgment the author
made, and a hundred copies of "this is an annotation" records a missing rule — the same argument
the subprocess seam settles above. The adapter skips the mutants inside each annotation's span
before the suite runs, so they are never judged and never cost a suite run.

Scoping to the span rather than the line is what keeps the rule from swallowing real gaps. A
default value shares the `def` line with its annotation and *is* evaluated when the `def` runs, so
`n: int = 10 - 4` keeps every mutant it has; so does a `cast(int | None, x)` whose arguments the
interpreter really does evaluate. Python is also the only arm that needs this: Stryker and
cargo-mutants read a type as a type and never offer a mutant inside one, while cosmic-ray matches
operators over the token tree, where a type and an expression look alike. The drop brings the
Python arm to the behavior the other two engines already have.

## A pass names its evidence

The check has three green outcomes, and they are different facts — so the run reports which one
it earned. A run that tested mutants states the count:

> `unit mutation: no surviving mutants — every mutation was caught (6 mutant(s) tested)`

so the pass carries its own evidence, and the count is always at least one. A diff-scoped run
whose changed source lines hold no mutant site — a signature move, a `const` value, a tests-only
edit, a Python annotation-only edit, a declaration-only module's mutants dropped before judging —
runs the engine, finds nothing to judge, and says so:

> `unit mutation: the engine found no mutants to test`

And a run whose changed lines hold no source files at all — a docs-only or workflow-only pull
request — skips the engine. TypeScript skips it the same way when the diff touches only
declaration-only modules, filtered out before Stryker ever runs:

> `unit mutation: no mutatable changed lines — engine not run`

All three pass (an empty diff owes no mutation run; this is reporting, not gating), but the log
tells a validated pass from a vacuous one: a check that has only ever printed the last two lines
has never killed a mutant, and the first behavior-touching pull request is where an environment
problem would surface. In Rust the zero-mutant line is additionally held to proof: cargo-mutants
applies the changed-line filter itself, so the tool cross-checks a zero-mutant report against
the crate's own mutant list, and a filter that dropped real mutants on changed lines is a hard
error, not a pass.

## The engines

Each language wraps its standard engine; the tool drives the engine, and you provide the test
runner that runs your own suite:

| Language | Engine | You provide |
| --- | --- | --- |
| TypeScript | [Stryker](https://stryker-mutator.io/), driven via its Node API by an adapter bundled in the npm package | `vitest` |
| Python | [cosmic-ray](https://github.com/sixty-north/cosmic-ray), driven via its library API by an adapter bundled in the wheel, with a baseline run that requires the clean suite to pass; each mutant's run ends at its first failing test | `pytest` |
| Rust | [cargo-mutants](https://github.com/sourcefrog/cargo-mutants), provisioned on first use (a pinned `cargo install` into the tool's own cache) and run from there; concurrent invocations share one provisioning install rather than each racing to install their own | the cargo toolchain that builds and tests your crate |

A run lists each survivor with its file, line, and mutation — the engine's mutator and the source
that mutator produced — and fails on any un-exempted one. Survivor paths are reported relative to
the scanned path, so exemptions address the same paths every other check uses.

Each engine runs where its ecosystem expects: cargo-mutants at the crate root, cosmic-ray at the
scanned path, and Stryker at the **package root** — the nearest directory at or above the scanned
path holding a `package.json`. cargo-mutants addresses files relative to the crate's **cargo
workspace root**, so the tool rebases the diff it feeds the engine onto that root and rebases the
engine's reported paths back — a crate that is a member of a larger workspace is diff-scoped,
gated, and reported exactly as a standalone crate is. A [`[rust] features`](/reference/config#rust-features)
list rides on every cargo invocation the run makes, so the crate's test targets are **built** with
the features too: an integration test that names a feature-gated item compiles, and the unmutated
baseline the whole run rests on builds. All three mutate the tree **in place**: Stryker applies each mutant
to the package's real files (a backup lives under `.stryker-tmp` for the duration of the run, and
the run restores every file when it ends), so everything a run touches resolves exactly as it does
in your editor — a source that reaches above the scanned path (`import pkg from '../package.json'`,
a shared `../tsconfig`) resolves through the real tree, the package's `tsconfig.json` is read where
it lies, and tooling resolves from the package's own `node_modules`. Mutation stays scoped to the
scanned path, and the scanned path's colocated unit suite is what judges each mutant; the package's
other suite tiers (`tests/`) stay out of the run. Rust reaches that scope with
`--cargo-test-arg --lib`: binary unit tests and the `tests/` tree do not judge library mutants.
A survivor under `--lib` means the library unit tier does not assert the mutated line. An
integration or e2e test may legitimately assert it; a narrow line-scoped exemption can record
that test and reason. The scan path narrows **which** of your test
files judge the mutants, and your runner keeps deciding what counts as a test and where it is
rooted — so a `vitest.config.ts` at the package root whose `include` names `src/**/*.test.ts`
resolves that pattern against the package root, exactly as it does when you run vitest yourself.
The run mutates production source alone and leaves your test files intact: a `*_test.py`,
`test_*.py`, or `conftest.py`; a `*.test.*` or `*.spec.*`, and anything under `__tests__/`; a
Rust `#[cfg(test)]` module — at any depth beneath the scanned path, not just its top level.
In-place mutation is built for the ephemeral CI
checkout the workflow runs in; run it locally on a clean working tree, since the sources hold
mutants while the run is live.

Each engine's own configuration is the tool's, and the test runner's configuration is yours.
cosmic-ray's whole config (module paths, excludes, timeout, test command) is rendered in-process
by the bundled adapter, so a cosmic-ray config file in the tree plays no part in the run; Stryker
is driven through its Node API with the adapter's options — the runner, in-place execution, the
mutate scope — taking precedence over a `stryker.conf.json`, which still supplies anything the
adapter leaves unset. The runner judging each mutant resolves its configuration exactly as in
your own runs: pytest finds its rootdir, config, and `conftest.py` files with its own upward
search from the scanned path, and vitest resolves the package's `vitest.config.*` with its own
upward search from Stryker's package-root, in-place run.

## Timeouts: a mutant timeout is inconclusive, a baseline timeout is fatal

A mutant whose run outlasts its budget **times out**. A timeout is *inconclusive* — the tool
counts it as neither a caught mutant nor a survivor, so a diff-scoped run where every mutant on
the touched lines times out passes with zero findings. This holds across engines: cargo-mutants
signals it with its own timeout exit status, and cosmic-ray reports the mutant with no usable
outcome; both drop out of the survivor set.

A **baseline** timeout is the opposite — a loud error. The baseline is the clean, unmutated suite;
a suite that can't finish in its budget is untrustworthy as a judge of any mutant, exactly as a
baseline that *fails* is. So an unmutated suite that times out or ends abnormally stops the run
with an error. A silently timed-out baseline would instead let every mutant time out and drop,
and slip an empty survivor set through as a false pass.

Each engine scopes the per-mutant timeout to the clean suite's own measured runtime: a slow suite
earns a proportionally larger budget, and only a mutant that runs *far* longer than the clean
suite is judged hung — so a legitimately slow suite keeps its budget instead of hitting a fixed
ceiling.

## Why it matters for agents

An LLM can reach 100% coverage with assertion-light tests; published benchmarks put the mutation
score of LLM-generated tests around 40% despite high coverage. Mutation score is the signal an agent
**can't** satisfy by executing code — only by writing assertions that pin behavior. That makes it far
more resistant to gaming than a coverage number, which is the most exploitable target you can hand an
optimizer.

## Where it fits

Mutation measures the same colocated unit suite the other checks already do. Because it re-runs
that suite many times, it rewards exactly what [isolation](./isolation) and the coverage floor
already enforce: fast, deterministic, isolated unit tests. A suite that passes those is already
mutation-ready.