2.2.1 Automatic Test Generation

Test generation begins in the Test Generation Wizard. The Wizard allows one of two different test generation methods: automatic test generation, and manual test generation. Manual test generation allows the instructor to begin creating a test from scratch; the test initially has no questions. In order to complete a manually-generated test, the instructor must search for and select all questions to put in the test.

Automatic test generation allows the instructor to specify what type of test he requires through various criteria. The software then creates a test automatically given the criteria.

2.2.1.1 Basic Selection Criteria

Step 1 - Enter Test Name

This step, illustrated in Figure 2.2.1.1.i1, asks the instructor for four pieces of information:

• name of the test
• class to which the test will be given
• test generation mode
• total point value

Figure 2.2.1.1.i1: Enter test name and general information

When the Test Generation Wizard first appears, the instructor enters the name of the new test in the new test name text box. He then enters the class to which this test will be given in the combo box. He can either select a known class from the list, or type in any class. A class will be in the list if:

• it has at least one question pertaining to it in the local question bank, or
• if another test has been created for it and that previously-created test is in the local question bank.

The class entered in the combo box does not restrict the instructor from giving the test to other classes; it is simply a property of the tests that is supplied for informational purposes, like the test name; it class has no effect on test generation.

The instructor selects the Automatic test generation radio button.

Next, the instructor fills in the Total point value text box. The Total point value text box allows the instructor to specify the amount of points the test should be worth. This is not a generation criteria, because any test can be given any point value without changing what questions are included on the test.

The instructor proceeds to step two by clicking Next. The instructor may not proceed to step two until all three text boxes have a value and are not blank. Because the test name is not the name of the file which the test is stored in, the test name can contain any character, but cannot be blank. The total point value must be a positive integer. The class of the test can also contain any character, but cannot be blank. The instructor must specify a test generation mode before proceeding as well.

Step 2 - Select Classes to Include in Test Question Pool

The next step involves selecting which classes to include in the test. If a class is included in the test, then questions pertaining to it are considered for inclusion in the test. It is illustrated in Figure 2.2.1.1.i2. The classes that are excluded from the test are listed on the left in the Excluded list. The classes that are already considered for use in the test are listed on the right in the Included list. At the beginning of the step, the Included list is completely full, and the Excluded list is completely empty. "Completely full" refers to the state of containing every class in the local question bank. A class is "in" the local question bank if it has at least one question pertaining to it in the local question bank.

Figure 2.2.1.1.i2: Select which classes to include

To exclude a class from the test, the instructor selects in the Included list the class he wants to exclude; this causes the class to be highlighted. To move the item into the Excluded list, the instructor then clicks on the button with one left-pointing arrow. Clicking on the button with the two left-pointing arrows will cause all classes to be excluded.

Reversing the above process requires the instructor to click on the question in the Excluded list. Then the instructor clicks on the button with one right-pointing arrow. The class is now included in the test. Clicking on the button with two right-pointing arrows will cause all classes to be included. Clicking on the button with two left-pointing arrows will cause all classes to be excluded.

When the instructor has finished selecting which classes to include and which to exclude, he clicks Next to proceed to the next step.

Every class has one or more topics associated with it. For example, a class about basic arithmetic may include the topics multiplication, addition, subtraction, and addition. For each class that was included in step two, the associated topics can be included or excluded with the next step.

Step 3 - Select Question Topics to Include in Test Question Pool

This step allows the instructor to select which topics to include in the test. It is illustrated in Figure 2.2.1.1.i3. By default, each topic appears in the Included list. The instructor selects which topics to exclude from the test by moving the relevant topics into the Excluded list. The process to select relevant topics is identical to the process used previously to select relevant classes.

Figure 2.2.1.1.i3: Select which topics to include

The test generator will not make any attempt to cause topics to appear equally in the test. For example, if the topics included on the test are Addition and Subtraction, then 75% of the questions may be related to Addition, and the other 25% would be related to Subtraction. It is also possible that none of the questions would be related to Addition, and all of the questions would be related to Subtraction. This behavior may be changed by using the Advanced Weight Specification Dialog, which can be accessed by clicking the Advanced... button. This feature is discussed in 2.2.1.2 Advanced Selection Criteria.

Once the instructor has finished selecting the appropriate topics, he clicks Next for step 4 of the Test Generation Wizard.

Step 4 - Specify Question Types, Test Length, and Difficulty

This step asks the instructor for three pieces of information:

• which types of questions to include
• the length of the test in minutes
• the difficulty level of the test questions
  

This step is illustrated in Figure 2.2.1.1.i4.

Figure 2.2.1.1.i4: Enter test length and question types

For the ten text fields in this step, the instructor may enter a range of acceptable values, instead of exact values, to make the generation more flexible. If an exact value is entered, then this value is taken to mean the range (x-2) to (x+2), where x is the exact value entered.

The Point Value of Each Section button allows the instructor to specify the amount of points that each section of the test is worth, where each section is comprised of all the questions of a particular type (the True/false section and Matching section, for example). If the point values specified for each section do not add up to the total point value specified in step one, then the point value entered in step one is ignored. The process of specifying the point values of each section is similar to the process of specifying the weights for question topics. This process (the Point Value Specification) is covered in 2.2.1.2 Advanced Selection Criteria.

One text box in this step is labeled Length of test in minutes. The the expected time taken for each question in the test will add up to a sum in the given range. For example, if the range is 5-10, then the possible tests generated could include:

• 5 one-minute questions (5 minutes total)
• 2 five-minute questions (10 minutes total)
• 3 three-minute questions (9 minutes total)
• 1 seven-minute question and 1 one-minute question (8 minutes total)

In each case, the total expected time is within the range of 5-10 minutes. This text box may not be left blank.

The Length of test in minutes feature operates on the estimated time taken attribute of questions in the question bank. The Length of test in minutes feature does not limit the student to the amount of time given; the value or range is only used as a criteria for finding questions. The estimated time taken attribute is a value supplied by the instructor upon the creation of the question.

The Allowed difficulties text box specifies the acceptable range of difficulty levels of each question on the test. If the range is 4-9, then no question on the test will be level 3 or easier, and no question will be as hard as level 10.

The ten text boxes in this step must contain integers, or they must contain ranges defined in terms of integers. The instructor may not use real numbers. They may not be blank. No integer in the Average difficult or Allowed difficulties fields may be less than 1 or greater than 10. Hence, the following values in either of the two difficulty fields are considered invalid:

• 0-5 (the lower limit is too small)
  
• -1-9 (the lower limit is too small, negative in this case)
  
• 1-11 (the upper limit is too large)
• 0-12 (the lower limit is too small and the upper limit is too large)
• 0 (the exact value is too small)
• 11 (the exact value is too large)
  

The integer(s) in the Length of test in minutes field must be positive. The following values in this field are considered invalid:

• 0-100 (the lower limit is not positive)
  
• 0-9999999 (the lower limit is not positive)
  
• -20-20 (the lower limit is not positive)
  
• -25--20 (both the lower limit and the upper limit are not positive)

Default Criteria Values

The table below indicates the default values for every field or criteria that can be entered in the Test Generation Wizard.

Wizard Flexibility

The instructor does not have to follow the four steps of the Test Generation Wizard in the order discussed in this document. Clicking on a step name on the left side of the Wizard window is equivalent to clicking on Next or Previous repeatedly to arrive at the step from the current step. For example, clicking on 1. Enter test name (step one) while in the 3. Select topics step (step three) is equivalent to clicking on Previous twice.

Unless some required field is empty, the instructor may click Finish at any point in time, using either default values for all of the unvisited steps or instructor-given values for steps that have already been visited. No text fields may be left blank before Finish is clicked or the step is completed. This should not create much frustration for the instructor because every field also has a default value.

In most cases, returning to any step that was previously visited will cause the values entered during the first visit to be retained, unless

1. the value entered had been invalid (including the condition of being blank), or
2. some values entered in a different step invalidate the values in the returned-to step.

The first case is only possible when the instructor enters an invalid value in step four, clicks Previous, and then returns to the step later. The second case only occurs when the instructor selects question topics to include in step three, then the instructor returns to step two and excludes any classes whose topics were included during step three.

When values cannot be retained, these solutions are used. In the first case, the offending fields are reset to their default values. In the second case, invalidated question topics are removed from the Included list.

2.2.1.2 Advanced Selection Criteria

The dialog used to specify topic weights is illustrated in Figure 2.2.1.2.i1. This is the Advanced Weight Specification dialog. This dialog is only for specifying topic weights in step three of the Test Generation Wizard.

Figure 2.2.1.2.i1: Advanced Weight Specification

It is possible that the weights of the question topics are not important, and the instructor does not want to restrict the criteria with weights. If so, the instructor clicks the bottom option titled Use "don't care" weights.

All weights will have a specific default value, explained in the Default Weight Values subsection. All weights must be non-negative and can be either an integer or a real number.

To set a topic's weight, the instructor makes sure Specify relative weights is selected. The instructor then clicks the topic whose weight he wants to change. The current weight of that topic will appear in the text box near the bottom of the dialog. The instructor enters the new weight in the text box, and repeats the process until all weights have been set to their desired values. When finished, the instructor clicks OK.

The dialog used to specify the point value of each section is illustrated in Figure 2.2.1.2.i2. This is the Point Specification dialog.

Figure 2.2.1.2.i2: Point Value Specification

It is possible that the point value of each test section is not important. If so, the instructor clicks the Don't care radio button near the bottom of the window and clicks OK. All point values must be positive integers. To set the point value of a section, the instructor makes sure Specify point value of each subsection is selected. The instructor then clicks the section name whose point value he wants to change. The current point value of that topic will appear in the text box near the bottom of the dialog. The instructor enters the new point value in the text box, and repeats the process until all point values have been set to their desired values. When finished, the instructors clicks OK.

Weight Values Flexibility

It is not realistic to expect the topic weight values to be followed exactly in the generated test; for this reason, a margin of error is allowed. This margin of error is a number chosen by the test generation system and is as small as possible to allow the system to generate a test. The margin of error cannot be specified by the instructor, and is not told to the instructor.

If the only way to generate a test with the given criteria is to use a very large margin of error, then the Length of the test in minutes is decreased slightly, as well as the quantity of each question type, and the test generator attempts to generate another test with the new criteria. The margin of error is considered "very large" when it renders meaningless the actual weights given by the instructor.

If the point value of a section is not divisable by the number of questions that appear that section after test generation, then the point value of that section is increased to the next divisable number, and the point value of the section is divided between all the questions in that section equally. For example, if there are 11 True/false questions, and the instructor specified a point value of 15 for that section, then the point value is automatically increased to 22 and each True/false question is worth two points. The instructor can change the point value of any question after the test has been generated.

Default Weight Values

The default weight of a topic is 10. The default point value of the sections are:

If a question type is not checked in step four of the dialog, then it will not be shown in the Point Value Specification dialog, and the default point value of its section will be irrelevant. Here are some more Default Criteria Values that pertain to Weight and Point Specification.