Mutation Testing

Automatically mutate the production code and check whether the test suite catches each mutation.

Why

100% line coverage can be achieved with tests that execute every line but assert nothing meaningful — coverage measures which lines ran, not whether the test would notice if the line were wrong. Mutation testing fixes this by changing the code (a + to a -, true to false, removing a condition) and re-running the tests. If the tests still pass, the mutation "survived," which means the test suite is blind to that change — the assertion is missing or too weak. It is the most rigorous way to measure test quality, and it exposes the false confidence of high coverage on weakly-asserted code.

When

Run mutation testing periodically (weekly or per release) on the most critical modules — not on every save, it is too slow. Use it to audit a suite that has 90%+ coverage but still has regressions, to find where the assertions are hollow.

How

1. Choose a mutation tool for your stack
   - JS/TS: Stryker JS
   - Java: PIT (Pitest)
   - Python: mutmut or cosmic-ray
   - Go: go-mutesting

2. Configure the mutators and the scope
   - Enable arithmetic (+ → -), boolean (true → false), conditional, and boundary mutators
   - Scope to the business-logic directory, not config or entry points (too slow otherwise)
   - Set a timeout per mutant (e.g. 5s) so a hanging mutant does not stall the run

3. Run the mutation analysis
   - The tool compiles each mutant, runs the test suite, and records killed/survived
   - This is N × test-suite-time, so start with a small module and a fast suite

4. Read the mutation score and the surviving mutants
   - Mutation score = killed / total; aim for > 80% on critical modules
   - For each surviving mutant, read the diff and the test that should have caught it
   - Most survivors are missing assertions, off-by-one checks, or boundary conditions

5. Fix the survivors by strengthening tests, not by weakening mutators
   - Add the missing assertion that would kill the mutant
   - Re-run to confirm the mutant is now killed
   - Do not disable a mutator just because it produces survivors — that hides the weakness

Example

// src/lib/pricing.ts
export function finalPrice(base: number, member: boolean): number {
  if (member) return base * 0.9;
  return base;
}

// src/lib/__tests__/pricing.test.ts  (weak — 80% coverage, 0% mutation)
test("member gets a discount", () => {
  finalPrice(100, true);  // no assertion! line runs, coverage ticks up
});

// Run: npx stryker run
// Stryker mutates "base * 0.9" → "base * 0.8", "if (member)" → "if (false)"
// The weak test calls finalPrice(100, true) but never checks the return value,
// so EVERY mutant survives. Mutation score: 0%.

// Fix: add the assertion
test("member gets 10% off", () => {
  expect(finalPrice(100, true)).toBe(90);  // now kills the * 0.8 mutant
});
test("non-member pays full price", () => {
  expect(finalPrice(100, false)).toBe(100); // kills the if(false) mutant
});
// Re-run Stryker → mutation score: 100%. The suite now actually verifies behavior.

Common Problems

⚠Running mutation testing on the entire codebase at once takes hours and produces thousands of survivors, overwhelming the team — start with one critical module and fix it before expanding scope.
⚠Disabling mutators that produce survivors ("we don't test boundary conditions anyway") inflates the score without improving the suite — the weakness is still there, just unmeasured.
⚠Treating the mutation score as a gate that must hit 100% before merge is counterproductive — some mutants are equivalent (the mutation produces identical behavior) and cannot be killed, capping the realistic score around 80-90%.
⚠Running mutation tests on every PR slows CI to a crawl and frustrates developers — mutation testing is an audit tool, run periodically, not a per-commit gate.

Best Practices

✓Scope mutation testing to the modules with the highest defect density or the most critical business logic — a 90% mutation score on the payment module is worth more than 60% across the whole repo.
✓Use Stryker's incremental mode (or PIT's history) so only mutants in changed code are re-analyzed, cutting run time from hours to minutes on incremental changes.
✓For each surviving mutant, decide consciously: strengthen the test (preferred), mark as equivalent (if truly so), or accept as low-risk — never silently ignore survivors.
✓Run mutation testing on a weekly or per-release cadence and track the score trend — a falling score means new code is being added with weaker assertions than the baseline.