In general, it is not algorithmically possible to always prove a program’s correctness or incorrectness. So, in practice, a program’s correctness is usually judged by testing the program on test cases from some test suite. How can we gauge the effectiveness of a test suite? Practitioners use metrics like code coverage, that is, the proportion of the code that is executed in at least some of the test cases. However, this metric is not well correlated with test suite effectiveness: some test suites have high code coverage, but fail to detect faults. A more effective technique is mutation testing, where we add faults into different parts of the code and see if the tests detect these faults. However, this technique is very time-consuming and thus rarely used in practice. It is therefore desirable to come up with a measure of test suite effectiveness that is more accurate than code coverage and more practical than mutation testing.

The authors take into account that, for each test, some methods are invoked directly while other methods are invoked indirectly via a chain of method calls. They show, crudely speaking, that the length of this chain--which they call “stack distance”--is highly correlated with test suite effectiveness in detecting the method’s faults. To be more precise: for this correlation to appear, we need to slightly modify the definition of the length, for example, we should not count calls to external libraries (since these libraries are supposed to be already tested), we should not count calls to constructor methods (such methods rarely contain faults), and so on.

An even better prediction of possibly faulty methods comes if we also take into account different versions of code coverage and other easy-to-compute characteristics, and train a neural network to predict the method’s faultiness based on all of this information. This leads to a practically useful alternative to mutation testing.

The paper is intended for specialists and practitioners in software engineering who are familiar with the basics of statistics. I recommend it to both theoreticians and practitioners.