Assignment 4: User Data Collection


Team Assignment


This assignment is to be performed in a team of about 3-5 people, preferably the same team as the project.


Goals and Objectives

  • Get practical experience with the design, planning, and conduct of data collection activities.

  • Perform an evaluation of the collected data with particular emphasis on usability.

  • Utilize the collected data to improve the design of a system or product.

  • Collect information for adjustments to the design and development process used.

Description


In this assignment you will select a topic or task you need user input for, choose a data collection method appropriate for this purpose, set up a data collection station, and conduct a sample session with outside participants, which should be as close to the intended users as possible. If possible, the topic should be related to your team project; if your team feels that it is not suitable, you can select a different topic for the user data collection task, but please talk to me first.Look at this activity from the perspective of team members who need to justify to their boss the overhead and expenses involved in this data collection activity. If you add up the actual expenses, the time spent by the developers, the time spent by the participants, and possibly the costs affiliated with the use of external participants, you will quickly see that such activities can become quite expensive. So it is very important to determine what you want to achieve, to carefully plan the setup and procedures, and to perform a thorough evaluation of the collected data, and to complete this assignment professionally.


Data Collection Station


Throughout the quarter, teams will set up their data collection activities in the HCI lab. In order to gather experience with the HCI lab facilities, all teams are expected to do their data collection in the lab. One of the teams is working on usage scenarios for the Morae usability suite, and it will be available to other teams. If your data collection activity requires both parts of the lab or is sensitive to noise and distractions, you can schedule the lab for your exclusive use during times when it is not occupied. Contact me for a reservation. While some team members guide the data collection activities at their station, the other team members will act as participants at the station of another team. You can use your storyboards from the previous assignments to explain the product and task to the participants. One goal of this trial run is to give everybody in class an impression of different data collection techniques. You need to do the actual data collection wit participants outside of this class, with participants that should be as close as possible to the intended users of your system (or the one that you’re evaluating). You should collect data from at least 20 participants - the more the better. If this is impractical due to the complexity of the task or the data collection method you are using, talk to me beforehand. We will discuss some practical aspects of this event in class. Video, Audio, and Computer Activity Recording

The most frequently used methods are video recording (either with a video camera or a Webcam), audio recording (with a tape recorder, dictation devices, or electronic voice recording), and the logging of computer activities (key strokes, mouse movements). Morae and similar usability evaluation suites provide these capabilities in an integrated manner. During the S12 quarter, I expect all teams to use Morae for their data collection activities in order to collect experience with it. We can also get additional equipment from Cal Poly's Media Services department; please let me know as soon as possible if your team requires specialized equipment.


Participant Consent


Experiments involving humans (and possibly also those involving animals) are subject to some laws and guidelines. Cal Poly has a Human Subjects Committee, which maintains a Web page on this topic at http://rgp.calpoly.edu/indexHS.html. While classroom activities like ours are exempt from the more stringent requirements (such as approval of the experiments), you need to be aware of these policies, and some of the laws that they are based on. In general, you should tell participants in advance what you expect them to do, what the potential difficulties and risks are, and what you are planning to do with the data collected. This is of course most essential if you are collecting personal data. I strongly recommend refraining from collecting potentially sensitive data like home addresses, home phone numbers, racial or ethnical background, religion or faith, sexual orientation, etc. Additionally, I cannot possibly imagine a scenario for this exercise where it would be legitimate for you to collect data like Social Security, credit card, or bank information.For the more common activities like questionnaires, surveys, or focus groups, it is common practice to use "informed consent" forms that participants sign at the beginning. You can find examples on the Human Subjects Committee Web page (see above, or http://rgp.calpoly.edu/formsHS.html) for a template to be used on anonymous questionnaires).


Submission and Deadline


You can view the deliverable for this activity as a report to your boss, or to venture capitalists who might be interested in providing funding for your project. Of course the main emphasis is to convince them that your overall project is great, but in this particular part, you can actually collect data that show them what others think of it.At the end of the next lab period after the data collection station event, post the materials you used on the Trac Wiki or an alternative repository if confidentiality is required. This should include:


  • A brief description of the topic, and the reasons you selected it.
  • A description of the data collection technique you chose, and why.
  • Materials used during the data collection itself, such as consent forms, explanations of the procedure, questionnaires, feedback forms, etc.
  • Excerpts from protocols of data collection activities. If possible, include some typical events, and also some unexpected ones.
  • A summary of the overall outcome, including common themes among the activities and reactions of the users, observations that confirmed your expectations, and surprises that you didn't expect. It is also possible that you may have conflicting evidence, which could indicate diversities in the participants, or deficiencies in your data collection technique; you should address these.
  • An evaluation of the overall data collection activity, including suggestions for improvement, and an assessment of the suitability of the data collection technique you chose.

As with the previous exercises, I will collect feedback from the teams. Please return the feedback sheets as soon as possible after the event. We will make the feedback available to the respective teams. This information is also very useful for accreditation purposes, so please return the completed forms after you've had a chance to look at them.Grading Criteria

Criteria Collection Station


  • Difficulty of chosen collection method
  • Consent form (specific to the method)
  • Script, schedule, questionnaire etc available
  • Setup of the station (equipment, facilities)
  • Interaction with the participants
  • Cleanup after the event

Criteria Documents


  • Quality of the documents
  • Capturing of the data (e.g. audio/video recording, notes, computer)
  • Organization of data (spreadsheed, data base)
  • Evaluation of data
  • Discussion of the results
  • Presentation of the results
  • Evaluation of the overall experiment

Some materials from last year's projects and assignments are available on the respective team Wikis; see http://users.csc.calpoly.edu/~fkurfess/Courses/484/S12/Project/Project/Teams.html for the ones from last year, and http://users.csc.calpoly.edu/~fkurfess/Courses/Recent-Courses.shtml for a list of courses going back to 2000 (not all projects are available anymore, and some are on confidential repositories).