Mutation
Coverage tells you which lines your tests ran. Mutation testing tells you whether your tests would notice if those lines were wrong — the question coverage can't answer. It's the verification rung above the coverage floor: the top of the unit ladder. The workflow runs it on every pull request, diff-scoped to the changed lines.
Coverage runs your code; mutation checks your tests
A coverage report says a line executed during a test. It says nothing about whether any test asserted on the result. A test that calls your function and checks nothing still covers every line it touches — 100% coverage, zero verification. Mutation testing closes that gap by deliberately breaking the code and watching what your tests do.
A mutant is your code with one small, deliberate fault introduced — + becomes -, >= becomes >, return x becomes return null. The tool makes the mutant, runs your unit suite, and records the outcome:
- Tests fail → the mutant is killed. Good: a test caught the bug.
- Tests pass → the mutant survived. Bad: a real, bug-shaped change slipped past every test.
Unlike coverage, the result can't be satisfied by executing code — only by asserting on it.
The example coverage can't catch
// adult.ts
export function isAdult(age: number): boolean {
return age >= 18;
}// adult.test.ts
import { expect, test } from 'vitest';
import { isAdult } from './adult';
test('isAdult', () => {
expect(isAdult(20)).toBe(true);
expect(isAdult(10)).toBe(false);
});That test has 100% line and branch coverage — there isn't even a branch to miss. Now mutate >= into >:
return age > 18;20 and 10 still return the same answers, so every assertion still passes — the mutant survives. Coverage said "perfect"; mutation testing found a real off-by-one at the boundary. The only thing that kills it is the assertion you didn't write:
expect(isAdult(18)).toBe(true); // now `>` and `>=` divergeWhy a number won't do: equivalent mutants
You might expect a mutation-score floor like "≥ 95%". Mutation score — the fraction of mutants a suite kills — is a poor target, for one dominant reason: equivalent mutants. Some mutations produce code that is semantically identical to the original (x = x + 0, a boundary that can't be reached). No test can ever kill them — and detecting which survivors are equivalent is formally undecidable. So for any given file you don't actually know what the maximum achievable score is; a fixed floor can be unreachable through no fault of your tests. It's the "last 20% is defensive trivia" problem from coverage, but worse — you often can't tell a real gap from an equivalent mutant without reading it. A score also isn't comparable across the per-language engines.
So the gate isn't a percentage. It's binary and diff-scoped:
No unexplained surviving mutant on the lines a change touched.
A survivor you've confirmed is equivalent or deliberately defensive gets a reasoned, line-scoped exemption — the same mechanism every other check uses; every other survivor must be killed. It's the mutation analog of the coverage philosophy — "zero survivors except what you exempted with a reason", not "hit a number you can't reach" — and it ports cleanly across languages, where a score does not. Diff-scoping is what makes a binary gate tractable: whole-tree mutation is too slow to run, so the job runs on pull requests only, over the <base>...HEAD changed lines.
The last mile of a subprocess
One survivor class is not a gap and not an equivalent mutant: the function that actually spawns. cargo-mutants replaces a whole function body with a plausible return value, so a wrapper around cargo llvm-cov yields Ok(String::new()) — and no unit test can tell the difference, because calling the wrapper at all runs the subprocess. The tiers that would notice are tests/ and the e2e suite, and the unit tier deliberately does not run them.
The answer is not an exemption per wrapper. An exemption records a judgement, and a hundred copies of "this is a subprocess seam" records a missing rule. Nor is it "inject the Command" — that is the right move one level up, and it is what makes the argv assembly assertable, but injection bottoms out: something has to hold the .output() call. So the rule names that bottom. A branch-free function that runs a Command, directly or by delegating to one in the same file, has no unit-tier contract for its own body, and cargo-mutants' FnValue mutant on it is dropped.
Branch-free is what keeps this from swallowing real gaps. A body that decides and spawns keeps every mutant, so the pressure to split it stays: move the decision into a function that returns the argv, assert that, and what is left delegates and qualifies. The gate rewards the split rather than waiving the function.
This is Rust-only, and deliberately. It keys on a cargo-mutants genre; Stryker and cosmic-ray produce their own mutators over these wrappers, and whether the same argument holds for each is a question about that engine's operator set, not about the language. A TypeScript or Python wrapper that hits the equivalent still takes a line-scoped exemption.
A Python annotation is metadata, not behavior
The second survivor class the tool recognizes is an annotation. def f(x: dict | None) -> int | None carries two | operators, and cosmic-ray rewrites each of them the way it rewrites any other: dict + None, dict & None, eleven variants per site. None of them runs. An annotation is type metadata, and the interpreter is free never to evaluate it — from __future__ import annotations (PEP 563) keeps the whole annotation a string, and from Python 3.14 on laziness is the default (PEP 649), where an annotation is evaluated only if something asks for it. A mutation of code that never executes cannot fail a test, so every one of those mutants survives a suite that pins the function's behavior completely.
That is the textbook equivalent mutant, and the conclusion the gate draws from it is the whole reason the gate is binary: a survivor no test can kill is not a measurement. So the tool does not hand this one to the consumer as an exemption to write. An exemption records a judgment the author made, and a hundred copies of "this is an annotation" records a missing rule — the same argument the subprocess seam settles above. The adapter skips the mutants inside each annotation's span before the suite runs, so they are never judged and never cost a suite run.
Scoping to the span rather than the line is what keeps the rule from swallowing real gaps. A default value shares the def line with its annotation and is evaluated when the def runs, so n: int = 10 - 4 keeps every mutant it has; so does a cast(int | None, x) whose arguments the interpreter really does evaluate. Python is also the only arm that needs this: Stryker and cargo-mutants read a type as a type and never offer a mutant inside one, while cosmic-ray matches operators over the token tree, where a type and an expression look alike. The drop brings the Python arm to the behavior the other two engines already have.
A pass names its evidence
The check has three green outcomes, and they are different facts — so the run reports which one it earned. A run that tested mutants states the count:
unit mutation: no surviving mutants — every mutation was caught (6 mutant(s) tested)
so the pass carries its own evidence, and the count is always at least one. A diff-scoped run whose changed source lines hold no mutant site — a signature move, a const value, a tests-only edit, a Python annotation-only edit, a declaration-only module's mutants dropped before judging — runs the engine, finds nothing to judge, and says so:
unit mutation: the engine found no mutants to test
And a run whose changed lines hold no source files at all — a docs-only or workflow-only pull request — skips the engine. TypeScript skips it the same way when the diff touches only declaration-only modules, filtered out before Stryker ever runs:
unit mutation: no mutatable changed lines — engine not run
All three pass (an empty diff owes no mutation run; this is reporting, not gating), but the log tells a validated pass from a vacuous one: a check that has only ever printed the last two lines has never killed a mutant, and the first behavior-touching pull request is where an environment problem would surface. In Rust the zero-mutant line is additionally held to proof: cargo-mutants applies the changed-line filter itself, so the tool cross-checks a zero-mutant report against the crate's own mutant list, and a filter that dropped real mutants on changed lines is a hard error, not a pass.
The engines
Each language wraps its standard engine; the tool drives the engine, and you provide the test runner that runs your own suite:
| Language | Engine | You provide |
|---|---|---|
| TypeScript | Stryker, driven via its Node API by an adapter bundled in the npm package | vitest |
| Python | cosmic-ray, driven via its library API by an adapter bundled in the wheel, with a baseline run that requires the clean suite to pass; each mutant's run ends at its first failing test | pytest |
| Rust | cargo-mutants, provisioned on first use (a pinned cargo install into the tool's own cache) and run from there; concurrent invocations share one provisioning install rather than each racing to install their own | the cargo toolchain that builds and tests your crate |
A run lists each survivor with its file, line, and mutation — the engine's mutator and the source that mutator produced — and fails on any un-exempted one. Survivor paths are reported relative to the scanned path, so exemptions address the same paths every other check uses.
Each engine runs where its ecosystem expects: cargo-mutants at the crate root, cosmic-ray at the scanned path, and Stryker at the package root — the nearest directory at or above the scanned path holding a package.json. cargo-mutants addresses files relative to the crate's cargo workspace root, so the tool rebases the diff it feeds the engine onto that root and rebases the engine's reported paths back — a crate that is a member of a larger workspace is diff-scoped, gated, and reported exactly as a standalone crate is. A [rust] features list rides on every cargo invocation the run makes, so the crate's test targets are built with the features too: an integration test that names a feature-gated item compiles, and the unmutated baseline the whole run rests on builds. All three mutate the tree in place: Stryker applies each mutant to the package's real files (a backup lives under .stryker-tmp for the duration of the run, and the run restores every file when it ends), so everything a run touches resolves exactly as it does in your editor — a source that reaches above the scanned path (import pkg from '../package.json', a shared ../tsconfig) resolves through the real tree, the package's tsconfig.json is read where it lies, and tooling resolves from the package's own node_modules. Mutation stays scoped to the scanned path, and the scanned path's colocated unit suite is what judges each mutant; the package's other suite tiers (tests/) stay out of the run. Rust reaches that scope with --cargo-test-arg --lib: binary unit tests and the tests/ tree do not judge library mutants. A survivor under --lib means the library unit tier does not assert the mutated line. An integration or e2e test may legitimately assert it; a narrow line-scoped exemption can record that test and reason. The scan path narrows which of your test files judge the mutants, and your runner keeps deciding what counts as a test and where it is rooted — so a vitest.config.ts at the package root whose include names src/**/*.test.ts resolves that pattern against the package root, exactly as it does when you run vitest yourself. The run mutates production source alone and leaves your test files intact: a *_test.py, test_*.py, or conftest.py; a *.test.* or *.spec.*, and anything under __tests__/; a Rust #[cfg(test)] module — at any depth beneath the scanned path, not just its top level. In-place mutation is built for the ephemeral CI checkout the workflow runs in; run it locally on a clean working tree, since the sources hold mutants while the run is live.
Each engine's own configuration is the tool's, and the test runner's configuration is yours. cosmic-ray's whole config (module paths, excludes, timeout, test command) is rendered in-process by the bundled adapter, so a cosmic-ray config file in the tree plays no part in the run; Stryker is driven through its Node API with the adapter's options — the runner, in-place execution, the mutate scope — taking precedence over a stryker.conf.json, which still supplies anything the adapter leaves unset. The runner judging each mutant resolves its configuration exactly as in your own runs: pytest finds its rootdir, config, and conftest.py files with its own upward search from the scanned path, and vitest resolves the package's vitest.config.* with its own upward search from Stryker's package-root, in-place run.
Timeouts: a mutant timeout is inconclusive, a baseline timeout is fatal
A mutant whose run outlasts its budget times out. A timeout is inconclusive — the tool counts it as neither a caught mutant nor a survivor, so a diff-scoped run where every mutant on the touched lines times out passes with zero findings. This holds across engines: cargo-mutants signals it with its own timeout exit status, and cosmic-ray reports the mutant with no usable outcome; both drop out of the survivor set.
A baseline timeout is the opposite — a loud error. The baseline is the clean, unmutated suite; a suite that can't finish in its budget is untrustworthy as a judge of any mutant, exactly as a baseline that fails is. So an unmutated suite that times out or ends abnormally stops the run with an error. A silently timed-out baseline would instead let every mutant time out and drop, and slip an empty survivor set through as a false pass.
Each engine scopes the per-mutant timeout to the clean suite's own measured runtime: a slow suite earns a proportionally larger budget, and only a mutant that runs far longer than the clean suite is judged hung — so a legitimately slow suite keeps its budget instead of hitting a fixed ceiling.
Why it matters for agents
An LLM can reach 100% coverage with assertion-light tests; published benchmarks put the mutation score of LLM-generated tests around 40% despite high coverage. Mutation score is the signal an agent can't satisfy by executing code — only by writing assertions that pin behavior. That makes it far more resistant to gaming than a coverage number, which is the most exploitable target you can hand an optimizer.
Where it fits
Mutation measures the same colocated unit suite the other checks already do. Because it re-runs that suite many times, it rewards exactly what isolation and the coverage floor already enforce: fast, deterministic, isolated unit tests. A suite that passes those is already mutation-ready.