Testing software remains a complex business, so finding different means for doing so is important. Metamorphic testing belongs in one’s toolkit, along with unit tests and property-based testing. Thus, overviews are particularly welcome.

Here, many of the most prolific authors on the subject have gotten together to review the topic, its challenges, and the opportunities still open to be tackled. Unfortunately, the resulting paper is deeply flawed.

For example, a proper definition of metamorphic testing (MT) doesn’t occur until page 4, but an understanding of MT is used throughout pages 2 and 3. Some of the obvious context behind MT, such as there being no specification for a program available, are not stated explicitly. Concretely, the oracle problem would be quite different if a specification were available. The example after Definition 3 (p. 5) is confusing because the notation uses comma-separated tuples for two different things; it is not incorrect, just a poor choice of notation. Definition 4 is vague about the roles of f and P. In other words, the authors assume too much implicit knowledge on the part of the reader.

Surprising for a paper in ACM Computing Surveys, the English is quite rough. And the tone is odd. For example, Section 3, “Frequently Misunderstood Concepts in MT,” comes off as being very defensive, while some of the challenges in Section 5 are so vague as to be quite hard to understand. There is also a fair amount of repetition (for example, the Siemens programs are mentioned four separate times). Worse still: Example 3 (p. 10) represents bad numerical analysis that is taught as incorrect to undergraduate students.

Someone interested in a survey of metamorphic testing, including some of the more speculative material covered here, would be better advised to read [1].