Demo of EMMA coverage tool

EMMA is a tool for measuring coverage of Java software. Such a tool is essential for detecting dead code and verifying which parts of your application are actually exercised by your test suite and interactive use.


Sample source code

This demo uses a sample application called "CircleConverter". The source code including the emma tool is available as a zip file:    CoverageDemo.zip [498K]

Download the zip file and unzip it into a folder of your choice.  Open a command window in that folder then proceed with the steps below.

This demo is for Windows platforms, but can easily be modified to run under Unix.



First be sure to delete any coverage data file from previous runs:

>del coverage.em coverage.ec coverage.txt

Start by compiling the example source code:

>del *.class
>javac CircleConverter.java

To get the basic idea of how the Emma works, you can run your app using "on the fly" coverage and perform manual testing:

>java -cp TestTools\emma.jar;. emmarun -cp . CircleConverter

This will cause the application GUI to appear.  Enter a 3 into the radius text field and press Enter.  The value for area will appear in the area text field.

App screenshot

Press Alt-X to exit the app.  Emma will print a message telling you where the coverage report was created.

EMMA: writing [txt] report to [...coverage.txt] ...


By default the text coverage report is generated in the current directory:

>type coverage.txt
[EMMA v2.0.4217 report, generated Fri Mar 04 12:26:30 PST 2005]
-------------------------------------------------------------------------------
OVERALL COVERAGE SUMMARY:

[class, %]      [method, %]     [block, %]      [line, %]       [name]
100% (4/4)      77%  (10/13)    74%  (261/352)! 78%  (60/77)!   all classes

OVERALL STATS SUMMARY:

total packages: 1
total classes:  4
total methods:  13
total executable files: 2
total executable lines: 77

COVERAGE BREAKDOWN BY PACKAGE:

[class, %]      [method, %]     [block, %]      [line, %]       [name]
100% (4/4)      77%  (10/13)    74%  (261/352)! 78%  (60/77)!   default package
-------------------------------------------------------------------------------

So by entering a single test data value, you achieved 74% block coverage.


That gives you the idea of how it works.  But we don't want to manually run our tests every time.  Ideally we would like the coverage test to be performed every time we run our automated unit tests.

First compile your unit tests:

>javac -classpath TestTools\junit.jar;. TestCircle.java

Then enter this monstrous command (normally you will include this in your test script):

>java -cp TestTools\emma.jar;TestTools\junit.jar;. emmarun -cp TestTools\junit.jar;. -report html -properties emmaconfig.txt -filter -junit* -filter -Test* junit.textui.TestRunner TestCircle
..
Time: 0.172

OK (2 tests)

EMMA: writing [html] report to [F:\CoverageDemo\coverage\index.html] ...


There are several command line parameters in the above example that customize the results the way we desire.

    -filter -junit*     tells Emma to ignore coverage of JUnit framework classes.
    -filter -Test*      tells Emma to ignore coverage of our unit test classes.
    -report html          causes an html report to be generated instead of the default text output.

    -properties=emmaconfig.txt  Tells Emma to use a configuration file we provide.

The file "emmaconfig.txt" contains a single line to specify that we want depth of coverage reported to the method level:
    report.html.depth=method


Look in the directory "coverage" and open the "index.html" file in a browser.  You'll see the overall summary, and near the bottom of the page the breakdown by package.  The CircleConverter app has only one package, the "default package."  Click on the "default package" hyperlink and another page appears with the details for the Circle.java file:

COVERAGE BREAKDOWN BY SOURCE FILE

name class, % method, % block, % line, %
Circle.java 100% (1/1) 71%  (5/7) 66%  (73/110) 68%  (17/25)




The report shows that our unit test accomplished only 66% block coverage.  To determine why the coverage is so low, you can examine the TestCircle.java unit test.


The next step is to include the GUI part of the application.  You are probably testing your GUI with an automated GUI robot.  "On-the-fly" coverage testing won't work for this situation because it will report on the robot, not on the GUI.  Instead, we will run a separate "instrumentation" step to prepare the target classes for testing.

>java -cp TestTools\emma.jar;. emma instr -ip . -m overwrite -filter -Test*
EMMA: processing instrumentation path ...
EMMA: instrumentation path processed in 1031 ms
EMMA: [5 class(es) instrumented, 0 resource(s) copied]
EMMA: metadata merged into [F:\CoverageDemo\coverage.em] {in 422 ms}

This step has added the Emma instrumentation code to our class files. 

Now we can run our automated GUI tests as we normally do.
>java  -cp .;TestTools\emma.jar TestTools.SwingRobot CircleConverter CircleTest.rsf
EMMA: collecting runtime coverage data ...
EMMA: runtime coverage data merged into [F:\CoverageDemo\coverage.ec] {in 141 ms}

You can watch the automated robot script exercise the GUI.


We now have to issue a separate command to generate the report:

>java -cp TestTools\emma.jar emma report -r html -in coverage.em -in coverage.ec -sp .
EMMA: processing input files ...
EMMA: 2 file(s) read and merged in 31 ms
EMMA: writing [html] report to [F:\CoverageDemo\coverage\index.html] ...

If you examine the html report closely, you will see it contains only coverage from the GUI test run, not the unit tests. If you want to combine both of them, you need to run the unit tests again (after they have been instrumented).  Then run the report again.

>java -cp TestTools\junit.jar;TestTools\emma.jar;. junit.textui.TestRunner TestCircle
.EMMA: collecting runtime coverage data ...
.
Time: 0.219

OK (2 tests)

EMMA: runtime coverage data merged into [F:\CoverageDemo\coverage.ec] {in 219 ms}

>java -cp TestTools\emma.jar emma report -r html -in coverage.em -in coverage.ec -sp .
EMMA: processing input files ...
EMMA: 2 file(s) read and merged in 47 ms
EMMA: writing [html] report to [F:\CoverageDemo\coverage\index.html] ...

Now the html report will show the total of all previous testing runs, both GUI and junit.


COVERAGE BREAKDOWN BY SOURCE FILE

name class, % method, % block, % line, %
Circle.java 100% (1/1) 86%  (6/7) 75%  (82/110) 80%  (20/25)
CircleConverter.java 75%  (3/4) 83%  (5/6) 88%  (213/242) 88%  (46/52)






You can also produce a more detailed report from "on-the-fly" manual tests. (The following example shows commands for Unix.)
$ del coverage.em coverage.ec coverage.txt
$ javac CircleConverter.java
$ java -cp TestTools/emma.jar emma instr -ip . -m overwrite
EMMA: processing instrumentation path ...
EMMA: instrumentation path processed in 74 ms
EMMA: [5 class(es) instrumented, 0 resource(s) copied]
EMMA: metadata merged into [/media/JDALBEYCSC/CoverageDemo/coverage.em] {in 325 ms}

$ java -cp TestTools/emma.jar:. CircleConverter
EMMA: collecting runtime coverage data ...
(Manual tests are run here)
EMMA: runtime coverage data merged into [/media/JDALBEYCSC/CoverageDemo/coverage.ec] {in 110 ms}

$ java -cp TestTools/emma.jar emma report -r html -in coverage.em -in coverage.ec -sp .
EMMA: processing input files ...
EMMA: 2 file(s) read and merged in 7 ms
EMMA: writing [html] report to [/media/JDALBEYCSC/CoverageDemo/coverage/index.html] ...


Here's an example for a NetBeans project.