CPE/CSC 484 W07 Assignment 4: Usability Evaluation

Assignment

This assignment consists of two parts.

Part 1: Development of the Evaluation Materials

This task is to be performed in a team of about 3-5 people, preferably the same as previous assignments.

Part 2: Individual Evaluations

Individually, you will use the developed materials and perform an evaluation of a system.

Either part may be related to your final project, so it is advisable to use the same core team for this assignment and the final project.

Goals and Objectives

• The goal of this assignment is to allow students to get practical experience with the design, planning, conduct, and aftermath of a usability evaluation for an existing system.

Description

Your task in this assignment is to perform a usability evaluation of an existing system. This system ideally should be related to the topics of the other assignments and the final team project. Due to time and resource constraints, your evaluation will most likely be of the "quick and dirty" type, trying to identify the most problematic issues with low overhead.

Your team may also do a usability evaluation for their final project. However, there should not be any significant overlap between this assignment and the project. In particular, you should probably not select the same system for A4 and the project, and you should use different evaluation approaches. For example, you can do a "quick and dirty" (heuristic) evaluation for this assignment, and a more elaborate one for the final project. If you believe that it is appropriate to select the same system for both this assignment and the project, you need to explain and justify this, and work out a comprehensive strategy for both evaluations.

Team Tasks

In Part 1, your team needs to decide upon a system, identify the main tasks of interest, and then develop the material to be used in the actual usability evaluation. Use the following points as guidelines when writing the deliverable for Part 1:

• System Identification: What system are you evaluating?
• Tasks and Activities: What do you want the participants to do with the system? Be sure to identify at least three to five primary ones, and as many additional ones as you feel are appropriate.
• Metrics: Try to develop measurable criteria. If you feel that subjective impressions are critical, provide guidance to the user on how to express their impressions (e.g. on a scale from 0-5, or by providing a set of terms to choose from).
• Evaluation form: Create an evaluation form from your metrics that the participants will use during the actual heuristic evaluation of the system. Even though your own team members may be the participants, try to formulate the questions and instructions in such a way that the intended group(s) will be able to understand them. Consider the information you would like to get from the participants, but avoid biased ("leading") questions or statements. If you would like to use the Websurveyor online questionnaire system, please let me know.
• Assessment: Based on the individual suggestions and recommendations formulated in Part 2 of this assignment, put together a common set of recommendations supported by the whole team. (Note: This should not simply be a "cut and paste" from the individual recommendations. You also need to consider potential conflicts in recommendations, and determine how to deal with such conflicts.

Individual Tasks

In Part 2, each member of the team performs a separate usability evaluation by performing the tasks identified according to the material developed in Part 1. Each individual must also formulate a set of recommendations aimed at improving the user interaction and interface of the evaluated system.

Background Material

There is an abundance of material available on this topic; a list of references is given below. You can also refer to the material used by others, such as Hoang Bao's tablet PC questionnaire, the Internet2 high-bandwidth file transfer done by Gigi Choy and Rachelle Hom, or Melissa Toy's usability evaluation for the Amgen Web site from Winter 2005. The material can be found on the 484 Blackboard discussion forums under the W06 and W05 quarters.

• "Interaction Design" Textbook Chapters 10-14
• "Resonant Interface" preprint Chapter 8
• Chapter 8 of User-Centered Website Development by McCracken & Wolfe
• Chapters 24, 25 and 33 of Human-Computer Interaction by Preece et al.
• Readings on Guidelines and Principles:
  • Ben Shneiderman's Designing the User Interface
    • Chapter 1 - Guidelines documents bibliography
    • Chapter 2 - Principles, Golden Rules and Guidelines
• http://www.usability.gov/guidelines/
• Website Usability Evaluation:
  • http://webtango.ischool.washington.edu/
  • http://zing.ncsl.nist.gov/WebTools/tech.html
  • http://zing.ncsl.nist.gov/WebTools/WebSAT/websat_rules.html
  • http://www.waller.co.uk/eval.htm
• Heuristic Evaluation:
  • http://www.useit.com/papers/heuristic/
• examples of good/bad interfaces:
  • http://www.baddesigns.com/index.shtml
  • http://pixelcentric.net/x-shame/
  • http://digilander.libero.it/chiediloapippo/Engineering/iarchitect/shame.htm
  • http://digilander.libero.it/chiediloapippo/Engineering/iarchitect/mfame.htm
  • http://www.commarts.com/interactive/index.html
  • http://useit.com
  • http://askTog.com

Submission and Deadline

For Part 1, submit electronic versions of the documents you produced to the Blackboard discussion forum for Assignment 4. Create a thread for your team with a meaningful subject, a brief description of the topic and your approach, and a list of team members. You can post the materials as one large document or as several separate documents; use the guidelines above for the overall structure. If the structure for your materials is different, it might be useful to explain the structure you're using.

Submit the individual evaluations as replies to the previously established team thread. This way, the individual materials are grouped together with the respective team materials. You can submit supporting documents such as questionnaires as hardcopies, but your evaluation document should be submitted on Blackboard.

The deadline is Thursday, March 8, end of lab period.

Grading Criteria

• Usability method selected (difficulty, appropriateness)
• Supporting material (questionnaire, consent form)
• Data used for the evaluation
• Evaluation of data
• Assessment
• Presentation of the results
• Recommendations