There are various measures that testers use to determine the completeness of their testing. The most common is “When all of the test cases that we have designed and built have run successfully and there are no outstanding severity one or severity defects, we are done testing”. The second most common is “When we run out of time of time and resources, that’s when we are done testing”. While the second is more honest, the first leads to a fundamental question. That is, quantitatively, how did you measure your coverage?
Most testers respond that they try to have a test for each line item in the requirements. However, that is highly dependent on the experience of the tester. Two or more testers will come up vastly different sets of tests based on their experience in testing, in the application, in the technology the application runs on, and even how they were feeling the day they designed the tests. It is also highly dependent on the level of detail and clarity of the requirements – a notoriously ambiguous and incomplete source.
The measurement of coverage needs to quantitative. It needs to have the same meaning across all classes of applications and all classes of technologies. There are two basic perspectives on test coverage: Black box (i.e. requirements based) and White box (i.e. code based). Within each of these there are different levels of coverage.
Figure 1 below shows some sample levels of black box and white box coverage. For more information please see the “How do you know when you are done testing?” paper on our web site at www.BenderRBT.com.
Keeping it simple, let’s us a basic white box coverage measure of statements and branches. Our experience shows that production data, taken as is, generally gives very low coverage. In one case we took six million transactions from a production stream and ran them against the application with a code coverage monitor on. The coverage was less than 30%. It would not be unusual to find that the coverage from production data was closer to 20%. Production data tends to be very weak in exception testing and timing dependent tests. It also tends to be highly redundant, testing the same paths again and again.
By using even basic approaches to test design, such as pair wise testing supported by full constraint specification, you can raise the coverage percentage into the 60’s. It would do this with far fewer tests. Going to fully sensitized variations via Cause-Effect Graphing raises the function coverage to 100% and the code coverage to about 90%. It actually does this with the fewest number of tests compared to any of the other test design approaches since the test libraries are highly optimized thanks to algorithms adapted from hardware logic testing. In the above case we were able to fully test the application with about six thousand tests, including all exception paths.
It is interesting to note that most applications go into production with less than 50% code coverage at the end of all of their testing. Therefore, using reasonable approaches to synthetic test data design leads to more effective testing, due to increased coverage, and more efficient testing, due to far fewer tests.
