System tests are the third phase in the Testing Lifecycle.  System Tests are the test of the end-user functionality. System Tests verify the correct functioning of all the required features as given in the specification document. 

Since prior testing phases have tested the internal logic of the application, System Tests should not repeat detailed, exhaustive testing.  Instead System Tests verify that all subsystems are cooperating successfully to yield the final desired features.

The QA manager is responsible for creating the Test Matrix, which is a grid with Requirement Numbers on one axis and Test Case Numbers on the other.  It shows which test cases cover which requirements.  Use this  Test Matrix Template which shows which test cases were written by which team member and has a link to the test cases.   (Alternate HTML Test Matrix Template)

• Each test case is in HTML (or Wiki) format, posted on team web page as described above.
• Each test case has all four required sections.

Content  

• Every functional requirement in the specification has at least one test case.
• Purpose provides specific detail of the behavior or function being tested.
• Actual input data and expected output data are provided.  The expected output is precise and detailed enough to be implemented as an "oracle" for automated testing.  The data needs to run on the actual software being released.
  
• User Action description is clear and specific enough for independent testers to follow the directions.
• The test case will correctly verify the requirement.
• Cross-reference is correct.

The easiest way to automate your tests is to write an Ant script that runs the test AND compares the results, perhaps by using Unix "diff."  This approach assumes you have a console driver for you app to which you can provide test input, and capture the text output.  There are examples of this approach in the CrazyEights and Hangman case studies. Tip: If there are dates, times, or version numbers in your output causing 'diff' to fail, you can remove them with Ant's replaceregexp task.

For GUI applications, you can use the SwingRobot test tool provided by Dr. Dalbey. You must create specialized test scripts which simulate keyboard and mouse operations.  For Swing-based apps, you can use the Abbot/Costello framework, which is more powerful but a little finicky.  The CrazyEights case study has a good example of Costello in action.

Both of the previous approaches can be integrated into your NetBeans-based project.
There are video demos of both techniques:
  SystestScripting.ogv  (52MB!!!)
  SystestGUIscripting.ogv (20MB)

• There is a test script for every system test case.
• The test script implements the complete test case.
  
• All the test scripts are organized with the NetBeans project hierarchy.
  
• The complete automated System Tests can be run using the NetBeans build.xml Ant script.
• An automated "diff" step shows whether the test pass or fail.
  

The following approach is now deprecated for this class: A more primitive approach, which Ant has made obsolete, is to automate your text-based console application by redirecting an ASCII text file with input data and capturing the output, for example: java MyApp < TestDataFile1.dat > TestOutput1.txt  To verify the results the test the output is compared against the expected results, called an "oracle," using a file comparison utility such as Unix 'diff' or DOS 'fc'.  For example, FC TestOutput1.txt TestOracle1.txt The test manager then has to create Unix shells scripts or DOS batch files that invoke the test tools and run the test scripts.  Organize the test scripts, the "setup" files (if needed), and the "oracle" files into a suitable directory hierarchy.  Create a ZIP file containing the entire test directory structure (without any application files), and place the zip file under a new link on your team home page called System Test Scripts. Here are some examples: runtest17.zip      a sample test script, oracle, and batch file LOC.zip [228K]   This example shows how to organize the folders for the application source code (in packages), system tests, and unit tests. Revelation9 This web site includes example System Test Cases and System Test Scripts for a GUI app. It should be possible to run all the tests by invoking a single command. You may be asked to demonstrate your test scripts to the class during lab.