CSC 307 Lecture Notes Week 10

CSC 307 Lecture Notes Week 10
Introduction to Code Coverage

Milestone 10 Summary
1. Due 11:59PM Thurs 11 June
2. Deliverables:
  1. finished implementation of requirements subset
  2. JML and unit tests for 8 to 12 methods
  3. 100% code coverage for tested methods
Final exam.
1. Cummulative
2. Both paper and computer-based
3. See the final exam overview handout for details.
What is code coverage?
1. It is a measure of how program code is covered for a given program execution.
2. Coverage is typically measured at the level of textual lines of code.
3. When a program is run to completion, the coverage measure states the percentage of program lines that are covered, i.e., executed, during the run.
4. If all lines of code are executed, coverage is 100%; if only half the lines are executed, coverage is 50%.
How code goes "uncovered".
1. Lines of code can go unexecuted for a number of reasons, including the following.
  1. Uninvoked functions -- code in a function body is not covered if the function is never called during a program run
  2. Untaken conditional branches -- depending on the values assigned to program variables, not all alternative branches of conditional statements may be covered in a particular program run.
  3. Unexecuted loop bodies -- if a loop test never evaluates to true the loop body will not be executed.
2. In a testing context, uncovered code means there are insufficient test cases to fully exercise the code being tested.
Coverage Tool Resources
1. See the 307/doc/ page.
2. And note that code coverage is NOT required for Milestone 7, though it will be for a later milestone.
Where code coverage fits into testing.
1. Code coverage is used to ensure that black box tests are adequately cover code.
2. There are many different coverage measures.
3. The bottom line is to ensure some measure of coverage.
4. During and after a test execution run, the coverage measures are applied to determine how much of the tested code is covered.
5. What follows is a discussion of the different coverage measures, from weakest to strongest.
Code coverage measures.
1. Function (method) coverage.
2. Statement coverage
3. Branch coverage
4. Decision coverage
5. Loop coverage
6. Define-use (d-u) coverage
7. All path coverage
8. Exhaustive coverage

Here is a common example that will be used to illustrate the different types of coverage.

    public static int f(int i, int j) {
        int k;
        if (i > j) {
            i++;
            j++;
        }
        k = g(i,j);
        if ((k > 0) && (i < 100)) {
            i++;
            j++;
        }
        else {
            i++;
        }
        return i+j+k;
    }

    static int g(int i, int j) {
        return i-j+1;
    }

Function coverage.
1. Each function is called at least once.
2. Very large-grain measure.
3. Not adequate for final tests.
4. Can be done with one test case for function f.
Statement coverage.
1. Every statement is executed at least once.
2. Can be done with two test cases for f.
Branch coverage.
1. The true/false direction of each branch is taken at least once, including branches that have no code in them, as in an else-less if statement.
2. This requires four test cases for f.
All path coverage
1. Each distinct control path is traversed.
2. Requires four cases for f.
Decision coverage
1. The boolean logic of each condition is fully exercised.
2. Requires at least four cases in f.
D-u coverage
1. Cover every path for every variable between a definition of that variable (i.e., assignment) and a use of that variable, without an intervening definition.
2. D-u for i requires three paths in f.
3. D-u for j requires two paths in f.
Coverage tools.
1. There are a number of code coverage tools available for Java, and other languages, links to which are in the 307 doc directory.
2. One of the best of the tools is Cobertura.
3. There is an example of Cobertura in 307/examples/cobertura
  1. The example code is that shown above as the common example for the different coverage measures.
  2. It can be run in conjunction with JUnit tests, providing an HTML report of the code covered when the tests are executed.
  3. There is an ant script to build and run the example, which is invoked simply by typing "ant" on the command line.
  4. The examples files are the following:
    - CoverageExample.java -- the code to be tested and measured for coverage.
    - CoverageExampleTest.java -- the JUnit tester for the code.
    - build.xml -- the ant build script
    - build.properties -- the properties file for the build, which defines file paths for the build.
  5. The results of running the example are in these subdirectories:
    - reports/cobertura-html -- the coverage report
    - reports/junit-html -- a junit testing report
  6. As these reports suggest, Coberatura is well integrated with JUnit, and produces result reports for both test coverage and the JUnit test execution.
4. You can modify the ant scripts for use in other contexts by changing the fileset list in build.xml, and the directory paths in build.properties.
5. Alternatively, you can use Coberatura in an IDE, per the IDE's conventions.

Details of branch (aka, decision) coverage in Cobertura and other coverage tools

When a testing tool such as Cobertura reports that branch coverage is not attained, the code can be white-box analyzed to determine test cases to add to meet the branch coverage requirement.
Specifically, test values must be added to exercise fully the boolean logic of all in conditional statements.
The use of a truth table can help in this task.

As an example, here is the truth table for the four alternatives in the boolean logic of the conditional statement "if ((k > 0) && (i < 100)) ...":

k > 0	i < 100	(k > 0) && (i < 100)	i	j	Remarks
0	0	0	1	2	i < j means k <= 0
0	1	0	100	101	i < j means k <= 0
1	0	0	100	100	i >= j mean k > 0
1	1	1	2	1	i >= j mean k > 0

Some mutations to illustrate different forms of test failure.
1. Change line 22 of CoverageExample.java from
```
return i+j+k;
```
  to
```
return i+j-k;
```
  1. This is an example of a bug in the code not properly implementing what the code is supposed to do.
  2. The result is the coverage tests all still succeed, but three out of the four test cases in CoverageExampleTest.java fail
2. Change line 55 of CodeCoverageExampleTest.java from
```
assertEquals(-2, ce.f(-2,-1));
```
  to
```
assertEquals(-2, ce.f(2,-1));
```
  1. This is an example of a bug in the testing code not properly testing code that is correct.
  2. In particular, the expected results are not correct with respect to the test plan.
  3. The unit tests fail but the coverage tests still succeed
3. Further change line 55 of CodeCoverageExampleTest.java from
```
assertEquals(-2, ce.f(2,-1));
```
  to
```
assertEquals(-2, ce.f(-2,1));
```
  1. This is another example of a bug in the testing code not properly testing code that is correct.
  2. Here the inputs are not correct with respect to the test plan.
  3. The unit tests succeed but the coverage tests still succeed
Recent research on code coverage.
1. Some very interesting research results on test coverage are presented in the paper
  "Test coverage and post-verification defects: A multiple case study"
  by Audris Mockus, Nachiappan Nagappan, Trung T. Dinh-Trong,
  Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement,
  October 2009
2. It is available at the ACM digital library at
  
  http://portal.acm.org/citation.cfm?id=1671248.1671276&coll=ACM&dl=ACM&CFID=80887391&CFTOKEN=40233171
  
  access to which is free from campus computers (or anywhere to ACM digital library subscribers).
3. The paper authors are from Microsoft research and Avaya, two very large companies.
4. Key observations and conclusions from the paper are the following (emphasis mine):
  1. "Despite dramatic differences between the two industrial projects under study we found that code coverage was associated with fewer field failures ... . This strongly suggests that code coverage is a sensible and practical measure of test effectiveness."
  2. "[They found] an increase in coverage leads to a proportional decrease in fault potential."
  3. "Disappointingly, there is no indication of diminishing returns (when an additional increase in coverage brings smaller decrease in fault potential)."
  4. "What appears to be even more disappointing, is the finding that additional increases in coverage come with exponentially increasing effort. Therefore, for many projects it may be impractical to achieve complete coverage."
5. Bottom line -- more coverage means fewer bugs, but it costs to get there.