1. It is a measure of how program code is covered for a given program execution.

2. Coverage is typically measured at the level of textual lines of code.

3. When a program is run to completion, the coverage measure states the percentage of program lines that are covered, i.e., executed, during the run.

4. If all lines of code are executed, coverage is 100%; if only half the lines are executed, coverage is 50%.

1. Lines of code can go unexecuted for a number of reasons, including the following.

   1. Uninvoked functions -- code in a function body is not covered if the function is never called during a program run

   2. Untaken conditional branches -- depending on the values assigned to program variables, not all alternative branches of conditional statements may be covered in a particular program run.

   3. Unexecuted loop bodies -- if a loop test never evaluates to true the loop body will not be executed.

2. In a testing context, uncovered code means there are insufficient test cases to fully exercise the code being tested.

1. See the 309/doc/ page.

2. And note that code coverage is NOT required for Milestone 7, though it will be for a later milestone.

1. Code coverage is used to ensure that black box tests are adequately cover code.

2. There are many different coverage measures.

3. The bottom line is to ensure some measure of coverage.

4. During and after a test execution run, the coverage measures are applied to determine how much of the tested code is covered.

5. What follows is a discussion of the different coverage measures, from weakest to strongest.

1. Function (method) coverage.

2. Statement coverage

3. Branch coverage

4. Decision coverage

5. Loop coverage

6. Define-use (d-u) coverage

7. All path coverage

8. Exhaustive coverage

    public static int f(int i, int j) {
        int k;
        if (i > j) {
            i++;
            j++;
        }
        k = g(i,j);
        if ((k > 0) && (i < 100)) {
            i++;
            j++;
        }
        else {
            i++;
        }
        return i+j+k;
    }

    static int g(int i, int j) {
        return i-j+1;
    }

1. Each function is called at least once.

2. Very large-grain measure.

3. Not adequate for final tests.

4. Can be done with one test case for function f.

1. Every statement is executed at least once.

2. Can be done with two test cases for f.

1. The true/false direction of each branch is taken at least once, including branches that have no code in them, as in an else-less if statement.

2. This requires four test cases for f.

1. Each distinct control path is traversed.

2. Requires four cases for f.

1. The boolean logic of each condition is fully exercised.

2. Requires at least four cases in f.

1. Cover every path for every variable between a definition of that variable (i.e., assignment) and a use of that variable, without an intervening definition.

2. D-u for i requires three paths in f.

3. D-u for j requires two paths in f.

1. There are a number of code coverage tools available for Java, and other languages, links to which are in the 309 doc directory.

2. One of the best of the tools is Cobertura.

3. There is an example of Cobertura in 309/examples/cobertura

   1. The example code is that shown above as the common example for the different coverage measures.

   2. It can be run in conjunction with JUnit tests, providing an HTML report of the code covered when the tests are executed.

   3. There is an ant script to build and run the example, which is invoked simply by typing "ant" on the command line.

   4. The examples files are the following:
      • CoverageExample.java -- the code to be tested and measured for coverage.
      • CoverageExampleTest.java -- the JUnit tester for the code.
      • build.xml -- the ant build script
      • build.properties -- the properties file for the build, which defines file paths for the build.

   5. The results of running the example are in these subdirectories:
      • reports/cobertura-html -- the coverage report
      • reports/junit-html -- a junit testing report

   6. As these reports suggest, Coberatura is well integrated with JUnit, and produces result reports for both test coverage and the JUnit test execution.

4. You can modify the ant scripts for use in other contexts by changing the fileset list in build.xml, and the directory paths in build.properties.

5. Alternatively, you can use Coberatura in an IDE, per the IDE's conventions.

1. When a testing tool such as Cobertura reports that branch coverage is not attained, the code can be white-box analyzed to determine test cases to add to meet the branch coverage requirement.

2. Specifically, test values must be added to exercise fully the boolean logic of all in conditional statements.

3. The use of a truth table can help in this task.

4. As an example, here is the truth table for the four alternatives in the boolean logic of the conditional statement "if ((k > 0) && (i < 100)) ...":

   k > 0	i < 100	(k > 0) && (i < 100)	i	j	Remarks
   0	0	0	1	2	i < j means k <= 0
   0	1	0	100	101	i < j means k <= 0
   1	0	0	100	100	i >= j mean k > 0
   1	1	1	2	1	i >= j mean k > 0

1. Change line 22 of CoverageExample.java from

   return i+j+k;
   

   to

   return i+j-k;
   

   1. This is an example of a bug in the code not properly implementing what the code is supposed to do.

   2. The result is the coverage tests all still succeed, but three out of the four test cases in CoverageExampleTest.java fail

2. Change line 55 of CodeCoverageExampleTest.java from

   assertEquals(-2, ce.f(-2,-1));
   

   to

   assertEquals(-2, ce.f(2,-1));
   

   1. This is an example of a bug in the testing code not properly testing code that is correct.

   2. In particular, the expected results are not correct with respect to the test plan.

   3. The unit tests fail but the coverage tests still succeed

3. Further change line 55 of CodeCoverageExampleTest.java from

   assertEquals(-2, ce.f(2,-1));
   

   to

   assertEquals(-2, ce.f(-2,1));
   

   1. This is another example of a bug in the testing code not properly testing code that is correct.

   2. Here the inputs are not correct with respect to the test plan.

   3. The unit tests succeed but the coverage tests still succeed

1. Some very interesting research results on test coverage are presented in the paper

   "Test coverage and post-verification defects: A multiple case study"
   by Audris Mockus, Nachiappan Nagappan, Trung T. Dinh-Trong,
   Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement,
   October 2009

2. It is available at the ACM digital library at
   
   http://portal.acm.org/citation.cfm?id=1671248.1671276&coll=ACM&dl=ACM&CFID=80887391&CFTOKEN=40233171
   
   access to which is free from campus computers (or anywhere to ACM digital library subscribers).

3. The paper authors are from Microsoft research and Avaya, two very large companies.

4. Key observations and conclusions from the paper are the following (emphasis mine):

   1. "Despite dramatic differences between the two industrial projects under study we found that code coverage was associated with fewer field failures ... . This strongly suggests that code coverage is a sensible and practical measure of test effectiveness."

   2. "[They found] an increase in coverage leads to a proportional decrease in fault potential."

   3. "Disappointingly, there is no indication of diminishing returns (when an additional increase in coverage brings smaller decrease in fault potential)."

   4. "What appears to be even more disappointing, is the finding that additional increases in coverage come with exponentially increasing effort. Therefore, for many projects it may be impractical to achieve complete coverage."

5. Bottom line -- more coverage means fewer bugs, but it costs to get there.

1. Acceptance testing is Performed by end user.

2. Acceptance tests are based on the HCI of the software, as opposed to the code- level API of the software.

3. Despite the HCI versus API distinction, the basic structure of a test case is the same for acceptance and functional unit tests.

4. Namely, an acceptance test case is defined as

   1. a set of inputs for a user-level operation

   2. the expected outputs when the operation is performed with the given inputs

   3. a remark about what the test case is intended to do

5. There is a handout on acceptance testing that provides further details on the specific structure of the acceptance test plans we'll use in 309.