All Collections
Test Design Principles and Helpful Tips
Prioritization: how should we get the biggest "bang for the buck" in a limited number of software tests?
Prioritization: how should we get the biggest "bang for the buck" in a limited number of software tests?

Why you should consider investing in intelligent test design to maximize the overall testing ROI.

Conor Wolford avatar
Written by Conor Wolford
Updated over a week ago

The above chart is based on averages presented in the following four studies, each of which measured how many test inputs needed to be included together in a single test case in order to trigger defects in production in mature, well-tested applications.

Works Cited:

Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.

Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA, Goddard SFC 4-6 December, 2002. 

NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.

Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006. 

Summary of Pairwise Testing and the Theory Behind Why It Works So Well

This fun presentation explains what pairwise testing (and closely-related test design approaches) are all about and why they work.  The presentation also explains some of the more-thorough combinatorial testing approaches (e.g, 3-way, 4-way, 5-way, and 6-way testing).  If you're looking for a photo-rich introduction to these concepts, this is it!

This article, written by three PhDs and the CEO of Hexawise, is a concise introduction to efficient and effective test design strategies.

"Combinatorial Software Testing" IEEE, August 2009.

Did this answer your question?