Mutation Testing

What it does

This skill enables AI agents to perform mutation testing, a technique for evaluating test suite quality. It works by introducing small changes ("mutations") into source code and verifying that existing tests fail when these mutations are present. If a mutation survives (tests don't catch it), this indicates gaps in test coverage or weaknesses in the tests themselves, helping to improve software reliability. The skill supports implementations for JavaScript/TypeScript, Java, and Python.

When to use it

• Evaluating the effectiveness of your existing test suite.
• Identifying code paths that are not adequately tested.
• Improving test quality metrics and ensuring tests actually verify behavior.
• Validating critical business logic is well-tested.
• Measuring real test coverage beyond simple line coverage.

Key capabilities

• Generates mutations (small changes) in source code.
• Verifies that existing tests fail when mutations are introduced.
• Identifies "survived mutants" indicating gaps in test coverage or ineffective tests.
• Supports Stryker for JavaScript/TypeScript, PITest for Java, and mutmut for Python implementations.
• Allows marking equivalent mutants to exclude them from analysis.

Example prompts

• "Run mutation testing on the payment_processing module."
• "Analyze test coverage using mutation testing and report any survived mutants."
• "Evaluate the quality of our unit tests with a mutation testing run, focusing on critical business logic."

Tips & gotchas

• Mutation testing can be slow, so focus on critical modules rather than running it against all code.
• Aim for an 80%+ mutation score on important code but don't expect or strive for 100% everywhere.
• Review and address any survived mutants to improve test quality.