Mutation Testing Rust in Earnest

14 February 2018

It’s been a while since I last suggested Mutation Testing in Rust, almost two years ago. Since then I got sidetracked a lot, and later lost interest. Just one more cool project I couldn’t afford to take on. But as things go, my interest in mutation testing was rekindled, and I decided to give it a shot and do more than blogging about it.

So without further ado, here’s the code. As predicted in 2016, it consists of a helper library and a procedural macro to insert mutations into code. There are still parts missing (for example the code that runs the tests), but at least we have a way to insert mutations into Rust code. This in itself is quite cool.

As I wrote last time, the ways to mutate Rust code without needing full type inference nor running afoul of the various checkers are few. I even made some errors when formulating stuff, because for example, we can not switch && and || for code may rely on the fact that the right-hand side is only evaluated when the left-hand side evaluates to true, and we should honor the original invariants as much as possible. However that still leaves us with a few tricks we can pull.

First, if we can find out if the return type implements Default (even on a best-effort basis with false negatives), we can insert an early return Default::default(). We can also check the inputs if they match the output type and return them instead. Lastly, we may still switch argument order, e.g.

fn foo(x: usize, y: usize) -> bool {
    let (x, y) = if MU::now(42) { (y, x) ] else { (x, y) }
    ..
}

Note that we have to take care to correctly reproduce mutability.

We can also change if and while conditions and change integer literals (e.g. to 0 or 1), or change comparisons to our own mutated versions. The mutagen plugin implements a few of those changes and will gradually implement most of them. While I focused on the plugin first, the other components are at least as important as introducing the mutations.

Running the tests for now is done by hand, ideally we want to have a test coordinator that runs and records the original tests’ runtime so we can run the tests again on mutated code, killing the process if a test takes more than e.g. 3× the original runtime. While an initial implementation may just repeatedly fork the test executable, we may want to speed things up by replacing the test harness with something we can control and run many mutations within the same process until we hit a timeout, removing the cost of a fork for each new mutation. But even something that runs all mutations without further modification would be a great first step.

Then we’ll need a reporting component that will show what mutations survived, or even show what tests killed what mutations, perhaps even what tests were covered completely by another test, so the covered test can be removed without ill effect.

TL;DR: Mutation testing is definitely coming to Rust! And you can be a part of it – I will be glad to mentor anyone wanting to join the project. Look at the issue list and see if you find something you’d like to work on! Thanks!