How to Test Before

Submitting to Bender

Overview

Successfully passing tests run by a robot like Bender requires a systematic and thorough approach to testing. This page describes such an approach. It assumes you're testing on Unix, which you must do prior to Bender submission to be sure everything is correct.

Step 0 – What not to do

First step is what not to do. Do not..

A. Run tests just on a local computer, and hope that they'll run on Unix also. Bugs may be hidden on your local machine, and show up on Unix, especially if you're using C.

B. Check for accuracy by "eyeballing" output from your program to see if it looks right. If you aren't running "diff" (see below) you will miss errors.

C. Rough out a test or two and call it good enough. Testing is a thoroughgoing discpline. See here for more information.

Step 1 – Build your test cases

Write a series of test files, using the approach and philosophy described here. Name them in orderly fashion, e.g. test1.in, test2.in, etc. You may also use any test cases I supply, but note that for many assignments (I'll tell you which) Bender uses additional tests that are not publicized. You need to write more tests of your own.

Make your test cases thorough. If you come to me asking why Bender is bouncing you when you think your program is correct, the first question I will ask is "Please show me your test cases and your script for running them." (See below.) The following are unacceptable answers to this request:

"I don't have actual test files but I checked it really carefully."

"I looked over the outputs and it's right."

"I ran them on my Windows machine so it should be fine."

"I just used the test cases you gave." (Unless the assignment permit this)

and the ever-popular

"Test cases???"

Step 2 – Verify your test cases

A. Run my sample executable on your test files. If the executable accepts input from the keyboard, redirect from the file, and save the ouput (which would otherwise appear on screen) thus:

[my executable] < test1.in > clint1.out

B. Run your executable to create your version of the output:

[your executable] < test1.in > my1.out

C. Compare the two using "diff" with the "-bw" option which makes diff a little more tolerant about blank lines and spaces. (Bender runs it this way.)

diff -bw clint1.out my1.out

Any output at all from diff is a bad thing. No output means you have a perfect match. Diff shows lines that differ between your file and mine. Lines in the first file (mine in the example above) are marked with a preceding "<" and those in the second file are marked with a preceding ">".

Step 3 – Make the tests easy to run

You end up running a lot of commands to do testing, even with the prewritten input files like test1.in. You have to run your program, run mine, save outputs, do a diff, and repeat for as many test cases as you have. Any time you find yourself repeating the same Unix commands over and over, it's time for a shell script. A shell script is a text file that contains a program. You write it using vi, just like your C or Java program. But, the commands in the shell script aren't C or Java; they're Unix commands. You type into the shell script file, using vi, exactly the set of Unix commands you want to run, and save it. For instance, just as an example, you might fill a shell script file called "runtests" with these 4 commands.

echo Doing test1

myexec < test1.in > my1.out

clintsExec < test1.in > clints1.out

diff -bw my1.out clints1.out

Once you've saved "runtests" as a file containing the 4 Unix commands, you need one more step:

chmod 700 runtests

This makes the runtests text file runnable like a program such as a.out. You have now created a shell script. You may run the shell script as a command in its own right, just as though it were an executable:

runtests

All the Unix commands in the shell script will be performed in sequence. It's like a little Unix "player piano". You just run the script file with that one command and all those commands you put into the runtests file get run in order (including the "echo", which we'll talk about in a moment). You can watch all the commands go by on the screen, with your hands off the keyboard. Any output that results from the commands (especially diff) will show on the screen, for you to diagnose. In effect, you now have your own small Bender, though Bender himself is actually a pumped-up 241-line shell script.

You'll probably want to do several different input files in your script, with diffs on each output. In that case, it's easy to get lost in the output, not knowing which diff command in the script produced which output line. That's where the "echo" comes in. *All* this command does is print back whatever you gave for arguments, mindlessly. So you can write:

echo Mary had a little lamb

(Try it!) And the command will print "Mary had a little lamb". You use this like a printf in scripts, to tell what part of the script is running. When the script reaches the echo command, it just prints on the screen whatever the echo says to print, and you can tell where in the script you are by seeing where the echo prints occur.