Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
Search
DeFlaker: automatically detecting flaky tests
Bell J., Legunsen O., Hilton M., Eloussi L., Yung T., Marinov D.  ICSE 2018 (Proceedings of the 40th International Conference on Software Engineering, Gothenburg, Sweden, May 27-Jun 3, 2018)433-444.2018.Type:Proceedings
Date Reviewed: May 20 2020

One of the final steps in producing a new version of a software tool is to make sure the old functionality was not lost when the new functionality was added. A common approach to this problem is regression testing, that is, running the test cases used for testing the old functionality on the new version. If a regression test fails, further expensive and time-consuming investigation and/or additional testing may be needed to make sure that the old functionality is still viable. However, having one of the regression test cases fail may not actually mean that old functionality was lost. The test may be a flaky test, that is, a test case that “can non-deterministically pass or fail when run on the same version of the code.”

The DeFlaker tool is an interesting approach to the early identification of many flaky tests. The tool uses differential coverage analysis of Java code to make sure that the failing test actually executes some of the code that implements the new functionality. This is done by checking statement-level coverage and/or class-level coverage.

Extensive experiments demonstrate that this approach does in fact identify which tests are flaky. The authors compare DeFlaker with other approaches, including JaCoCo, Cobertura, and Clover, and it performed well.

This is an interesting approach. DeFlaker is available under the MIT License. This paper is a good introduction to regression testing, differential coverage analysis, and tool evaluation issues. The reference list is excellent.

Reviewer:  David A. Gustafson Review #: CR146974 (2009-0224)
Bookmark and Share
  Featured Reviewer  
 
Testing And Debugging (D.2.5 )
 
 
Software Development (K.6.3 ... )
 
 
Verification (B.1.4 ... )
 
Would you recommend this review?
yes
no
Other reviews under "Testing And Debugging": Date
Software defect removal
Dunn R., McGraw-Hill, Inc., New York, NY, 1984. Type: Book (9789780070183131)
Mar 1 1985
On the optimum checkpoint selection problem
Toueg S., Babaoglu O. SIAM Journal on Computing 13(3): 630-649, 1984. Type: Article
Mar 1 1985
Software testing management
Royer T., Prentice-Hall, Inc., Upper Saddle River, NJ, 1993. Type: Book (9780135329870)
Mar 1 1994
more...

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright 1999-2024 ThinkLoud®
Terms of Use
| Privacy Policy