2.2.5 Test Generation Details

2.2.5.1 Test Generation Process

RAINBOW uses the following procedure when generating a test. The filled out Basic Test Creation form seen in figures 6 & 8 from Section 2.2.2 and Section 2.2.3 will be used as an example. They has also been duplicated below for convenience.

Figure 6: Filled in Basic Test Creation

Figure 8: Question Type Breakdown

The test is generated using the following procedure.

1. The given question bank is filtered by the given keywords (if any). In this case, the pool of questions would now contain only those with the Mutability, Constructors, and Interfaces keywords.
2. All questions not in the given difficulty range are then removed. Now, only questions with a difficulty of 2, 3, or 4 and match the aforementioned keywords remain.
3. From the remaining questions, every possible combination of questions that meets the time constraint will then be generated. In this example, the tests the remain would all be between 100 and 120 minutes.
4. These tests are then shuffled and each combination is given a score based on how closely it follows the breakdowns the user specified. This measure is the sum of the differences between the given combination's actual breakdown and the one specified by the user. For example, if a test was found that was 10% true-false questions, 60% multiple choice, and 30% code questions, its difference score would be 20. This is because the user specified that the test should consist of 70% multiple choice and only 20% code questions.
5. Either the first test with a difference score of zero is selected, or the test with the lowest difference score is selected. In the case of the latter, the user will be notified that their constraints could not be met and where the given test deviates from their secification.

2.2.5.2 Test Generation Errors

As mentioned above, if RAINBOW cannot generate a test that completely meets the user's specifications, they will be prompted with the dialog seen in figure 17 when the generate button is pressed.

Figure 17: Test Generation Error

The message displayed is dependent upon the manner in which RAINBOW failed to meet user specifications. Below is the set of error strings that can be displayed, where X represents the value RAINBOW used and Y is the value the user specified.

• There were not enough questions in the question bank to generate the test
• The test is X minutes too long
• The test is X minutes too short
• X% of the test is instead of Y%
• X% of the test is of difficulty instead of Y%
• X% of the test is of keyword instead of Y%

It is important to note that RAINBOW will never user a keyword, question type, or difficulty that it was alloted no room for. For example, RAINBOW will never generate a test with code questions unless the user told it to do so. In case of an error, only percentages will vary, not types.