In this survey paper, using over 150 selected articles between 2012 and 2017, the authors develop a framework to classify cloud testing research and to identify research trends and gaps in this area. The authors define six primary motivating categories for these works: test perspective, test design, test execution, test objective, test evaluation, and test domain.

Through this work, the authors have uncovered that the most actively investigated area is test execution, as the cloud offers the distinct “possibility to develop and maintain costly test infrastructures and to leverage on-demand scalable resources for configuration (by using cloud virtualization) and [scale (performance)].” Test objective for “performance, functional, security, reliability, and elasticity” is the next most investigated area with respect to cloud testing research, as it allows for expansive combinatorial testing and highly scaled parallel executions.

Not surprisingly, the most often tested scenarios include mobile and web applications. Gaps in this testing focus include the Internet of Things (IoT) and networking. Furthermore, the study uncovers that i) there is very limited research on understanding and organizing for testing perspectives, thereby indicating that there is still a lack of proper conceptualization; ii) test evaluation activities such as test monitoring, coverage, and related analytical techniques haven’t received significant attention; and iii) innovations in testing infrastructures to “support the assessment of cloud testing outcomes, possibly along different validation metrics,” are also inadequately addressed.

Cloud testing research should “be more strictly intertwined with the progress of research and developments in cloud computing: the [research provides] approaches and methodologies for developing, validating, measuring, and certifying applications, frameworks, tools, and infrastructures, [and the developments provide] the resources and facilities to assess, simulate, or emulate real-world scenarios.”

In summary, the authors present a good but tightly sandboxed survey vis-à-vis this nascent and rapidly evolving field. Virtual and remote testing are rapidly gaining attention within large organizations to enable separations along design, manufacturing, testing, shipping, and commissioning/deployment activities; each of these activities could possibly be geographically distributed or geo-separated with respect to capabilities and cost optimizations, and the cloud is a huge part of this expansion. Such topics are not too easy to contextualize (with data) within a theoretical study, and hence are more suited to experimental research within these large organizations themselves. Nevertheless, this is a good baseline for researchers in the area.