Writing System Test Cases

System tests are the third phase in the Testing Lifecycle.  System Tests are the test of the end-user functionality. System Tests verify the correct functioning of all the required features as given in the specification document. 

Since prior testing phases have tested the internal logic of the application, System Tests should not repeat detailed, exhaustive testing.  Instead System Tests verify that all subsystems are cooperating successfully to yield the final desired features.

System tests are usually "black box" tests since we are testing the application without seeing the source code.  Create the test cases following the guidelines in the textbook, the black box techniques studied in class, as well as your own experience or intuition about verifying program correctness.  Number each test case and write it in HTML (or Wiki) format.

Prepare the test cases according to these directions:  System Test Case Format

The QA manager is responsible for creating the Test Matrix, which is a grid with Requirement Numbers on one axis and Test Case Numbers on the other.  It shows which test cases cover which requirements.  Use this  Test Matrix Template which shows which test cases were written by which team member and has a link to the test cases.   (Alternate HTML Test Matrix Template)

Quality Assurance Criteria



Deliverable: System Test Scripts

For this deliverable your team is to write test scripts for each of the test cases you created above.  The goal is to create a completely automated System Test.

The easiest way to automate your tests is to write an Ant script that runs the test AND compares the results, perhaps by using Unix "diff."  This approach assumes you have a console driver for you app to which you can provide test input, and capture the text output.  There are examples of this approach in the CrazyEights and Hangman case studies. Tip: If there are dates, times, or version numbers in your output causing 'diff' to fail, you can remove them with Ant's replaceregexp task.

For GUI applications, you can use the SwingRobot test tool provided by Dr. Dalbey. You must create specialized test scripts which simulate keyboard and mouse operations.  For Swing-based apps, you can use the Abbot/Costello framework, which is more powerful but a little finicky.  The CrazyEights case study has a good example of Costello in action.

Both of the previous approaches can be integrated into your NetBeans-based project.
There are video demos of both techniques:
  SystestScripting.ogv  (52MB!!!)
  SystestGUIscripting.ogv (20MB)

Quality Assurance Criteria

The following approach is now deprecated for this class:

A more primitive approach, which Ant has made obsolete, is to automate your text-based console application by redirecting an ASCII text file with input data and capturing the output,
for example: java MyApp < TestDataFile1.dat > TestOutput1.txt 
To verify the results the test the output is compared against the expected results, called an "oracle," using a file comparison utility such as Unix 'diff' or DOS 'fc'.  For example,
FC TestOutput1.txt TestOracle1.txt
The test manager then has to create Unix shells scripts or DOS batch files that invoke the test tools and run the test scripts.  Organize the test scripts, the "setup" files (if needed), and the "oracle" files into a suitable directory hierarchy.  Create a ZIP file containing the entire test directory structure (without any application files), and place the zip file under a new link on your team home page called System Test Scripts.

Here are some examples:

runtest17.zip      a sample test script, oracle, and batch file
LOC.zip [228K]   This example shows how to organize the folders for the application source code (in packages), system tests, and unit tests.
Revelation9 This web site includes example System Test Cases and System Test Scripts for a GUI app.

It should be possible to run all the tests by invoking a single command. You may be asked to demonstrate your test scripts to the class during lab.

Document History
5/8/2011  Added new Ant-based techniques.
4/11/04 JD Changed title, clarified wording in first paragraph, changed name of hyperlinks.

CPE 206 Home