Software is notoriously difficult to get right even for experienced, professional software developers using modern software engineering practices. If these experts can’t produce correct programs, what chance do mere scientists have of writing scientific programs that will produce accurate, correct results, especially if modern development practices are not followed? Errors in scientific software have led to controversies over results and even the retraction of results in some cases.
The chasm in the title refers to the gap between current software engineering practices and the development methods used in large-scale scientific programming. The survey reviews the work and literature that addresses this gap, summarizing problems encountered when software is used in scientific work. It reviews current state-of-the-art methods in scientific programming reported in case studies, and describes ongoing work to understand and support the requirements of scientific programming projects. Finally, it explores how both software engineering and scientific research practice may require changes to make software more effective for scientific programming.
The survey has seven sections. The first is an introduction.
Section 2 summarizes problems encountered when software is used in scientific research. These go beyond mere bugs in the programs, including things such as the peer-review process discouraging maintenance of software, which might discover defects that could require retraction of published results. Other problems include the unavailability of software used in experiments hindering repeatability of results, and updates to the software changing results.
Several case studies showing current software development processes in scientific programming are discussed in Section 3.
Interestingly, agile methods have been tried, and while some agile methods seem to fit well with scientific programming, others do not. For example, an agile approach to requirements specification and planning works well to elicit and prioritize requirements, while test first development seems to be less effective.
Section 4 discusses quality assurance practices. Since the goal of a scientist is to produce science not software, scientific software is only a means to this end. Standard software engineering practices do not always fit well within the constraints of the scientific domain. Quality is also impacted because of poor dissemination of software quality practices among developers of scientific software and because scientists overestimate their ability to produce quality software. This is compounded by the difficulty of telling when a test result is wrong due to an incorrect theory, a design flaw in the model, or just a defect in the program.
Component architectures, design patterns, and re-factoring and re-engineering techniques as currently used in scientific programming are discussed in Section 5. Data quality issues are briefly covered in Section 6.
The concluding Section 7 notes that the problems aren’t really new and aren’t going away anytime soon, as there are not any easy straightforward solutions to them.
The themes identified in this survey suggest that both software engineering practices and scientific research need to be adapted to address the challenges of repeating, reproducing, and verifying software-based science.
The survey is well written and thoughtful. A comprehensive bibliography of 194 references is included. Scientists developing or using software in their work should read this survey. It is also recommended to software engineers working with scientists, as well as researchers looking for ways to improve the development of scientific software.