CSC 484 Lecture Notes Week 7, Part 1

CSC 484 Lecture Notes Week 7, Part 1
Data Gathering (Part 1)
Data Analysis (Part 2)

Relevant reading.
1. Textbook Chapters 7 and 8
2. Selected portions of Chapters 13 and 14.
3. Weeks 7 and 8 research reading (one paper for two weeks, given the extra chapter reading):
  
  "Integrating statistics and visualization: Case studies of gaining clarity during exploratory data analysis", by Adam Perer and Ben Shneiderman, University of Maryland; Proceedings of the SIGCHI conference on Human factors in computing systems, 2008, ACM.
4. Certain teams should read ahead, based on the type and scale of data gathering you're doing; in particular,
  1. The 2d3d team should read Chapter 13 on the DECIDE evaluation framework.
  2. The swat team should do an early read of Section 14.3 on field studies.
Introduction to Chapter 7 (Section 7.1).
1. The chapter discusses planning and conducting data gathering activities.
2. The book considers data gathering for two steps of the ID process: requirements and usability evaluation.
3. Our focus at this point of 484 is on evaluation.
4. Chapter 7 covers three specific data gathering techniques:
  1. in-person interviews
  2. questionnaires
  3. (non-intrusive) observation
5. Additional data gathering techniques are presented in Chapters 12 (which we've already covered), 13, and 14.
Four key data gathering issues (Section 7.2).
1. Setting goals (Section 7.2.1).
  1. This is very important to do this at the outset.
  2. When a team is fully immersed in a project, it can be easy to forget to state the high-level goals clearly for the purposes of data gathering.
    1. Be completely clear on tasks users will be asked to do, and how you will measure their performance.
    2. Be clear on the most important things you need to know from the participating subjects.
  3. In 484, project goals are defined in two specific places:
    1. You define overall project goals in the Milestone 2 deliverable.
    2. You define usability study goals in the Milestone 3 deliverable.
2. The relationship with participants (Section 7.2.2).
  1. Establish and maintain a professional relationship with the study participants.
    1. In 484, you will have study subjects sign an informed consent form, including the 484 student who act as subjects.
    2. See www.calpoly.edu/~sdavis/human2.htm for a template.
    3. Note that subject anonymity is most likely not necessary for the 484 studies.
      1. If you take photos or video of usability study sessions, anonymity is difficult to maintain.
      2. If you obtain qualitative results from user questionnaires, you may want to clarify their responses, as long as such clarification does not adversely influence the data analysis.
3. Triangulation (Section 7.2.3).
  1. This just means using more than one ¹ data gathering technique.
  2. Doing so provides more useful and believable results.
  3. For the 484 studies, most teams will use
    1. questionnaires,
    2. subject performance data,
    3. other forms of observation,
    4. possibly in-person interviews with study subjects.
4. Pilot studies (Section 7.2.4).
  1. This is a small separate study, run at the beginning of a larger data collection activity.
  2. The pilot study is used to "debug" data gathering techniques, before proceeding with the full study.
  3. For example, a pilot questionnaire can be sent to a small group of pilot participants, to determine if it's understandable and asks the right questions.
  4. I personally think that pilot studies are an indispensable tool when a team is embarking on the study of an area that is new to the team members performing the study.
  5. In 484, we will not have time to do any pilot studies.
Data recording (Section 7.3).
1. The forms are well known, i.e.,
  1. Hand-written, PDA, or laptop-recorded notes, taken by evaluation team members.
  2. Questionnaires, and other forms of user-recorded data.
  3. Still photographs.
  4. Audio recording.
  5. Video recording.
2. Noteworthy considerations:
  1. Always ask permission of interviewees.
  2. Avoid adding bias by asking leading questions, giving suggestive gestures or body language.
  3. Any of the above forms of explicit data recording may distract the flow of user study, particularly video.
  4. Having one team member ask questions and another record data can be useful.
  5. Transcribing recorded data can be time consuming -- have at least some form of an agenda.
3. Table 7.1 (book page 297) has a comparison of the pros and cons of the different forms of data recording.
4. In your 484 usability studies, think over the pros and cons, and use the forms of data gathering that are appropriate to your circumstances.

Interviews (Section 7.4).

The book defines an interview as a "conversation with a purpose".
It notes four general types of interview (Sections 7.4.1 - 7.4.4).
1. Unstructured -- open-ended discussion with interviewees
2. Structured -- a predetermined set of questions
3. Semi-Structured -- a combination of structured and unstructured
4. Group -- conducted with multiple interviewees, e.g., focus groups
Planning and conducting an interview (Section 7.4.5).
1. Even unstructured interviews should have a plan.
2. Use open-ended questions in unstructured interviews, when you don't know in advance all of the answers to expect.
3. Use closed questions in a structured interview, where the subjects selected from a fixed set of answers.
4. A "closed" question is one for which you have a fixed set of answers.
5. The book has some additional guidelines on pages 304 - 307.
Other forms of interview (Section 7.4.6).
1. The book mentions phone and online interviews as possibly useful.
2. In my experience, these are no substitute for face-to-face meeting.
The book also mentions the reasonably obvious point that interviews can be "enriched" with prototypes and other product-appropriate materials (Section 7.4.7).

Table 1 (in these notes) summarizes important properties of the different types of interview.

	Unstructured	Structured	Semi-Structured
Replicatable	Not easily	Yes	Somewhat
Amenable to Statistical Analysis	No	Yes	Somewhat
Easily Transcribable	No	Reasonably	Somewhat
Type of Planning	General Agenda	Rigid Agenda	Rigid then General
Type of Questions	Open-ended	Fixed Answer Set	Combination

Table 1: Properties of different interview types.

At this point, it looks like most 484 usability studies will make limited use of in-person interviews.
1. The swat team will in fact conduct interviews with out-of-class subjects; they can use questionnaires with the 484 students who participate in the study during week 10.
2. Other teams can employ in-person interviews as you see fit, but you will use in-person questionnaires.

Questionnaires (Section 7.5).
1. Much the same form of questions as in a structured interview.
2. The questions asked must be very clear and unambiguous, given that an interviewer is not present to clarify.
3. Motivation is also an issue with questionnaires versus in-person interviews.
  1. I.e., it can be easier to encourage subject responses in person.
  2. This can be mitigated by having questionnaire respondents fill them out in person, as will be the case with the monitored 484 usability studies.
Questionnaire design (Section 7.5.1).
1. Ask for demographic data if appropriate, though it is probably not relevant in 484.
2. Points to consider:
  1. Clear instructions -- provide them up front, including any necessary definitions.
  2. Question ordering -- ask most important questions first.
  3. Different versions of the questionnaire -- consider if you need them.
  4. Keep it short and sweet -- even in monitored studies, users quickly grow weary of long questionnaires.
3. Regarding question ordering, you can have bifurcation points as appropriate.
  1. E.g., "If X is true answer the following questions, otherwise proceed to question N."
  2. This is less likely to be useful in the 484 use of questionnaires during the usability studies.
4. See book pages 313 - 314 for a general example.
Question response formats (Sections 7.5.2).
1. Check boxes and ranges
  1. Select appropriately, based on your knowledge of subjects.
  2. Be careful to avoid overlaps.
  3. Avoid annoyingly long selection lists (use the dreaded 7+/2 rule).
2. Rating scales
  1. Two common ones are Likert and semantic differential.
  2. The book goes over details on pages 313 - 317.
Administering questionnaires (Section 7.5.3).
1. Return rates vary widely.
2. 484 is a somewhat specialized case in that most subjects will be asked to complete questionnaires in person, before departing the study venue.
Online questionnaires (Section 7.5.4).
1. There are a number of tools and templates available.
2. The book has details on pages 317 - 321.
3. Each team can consider if an online questionnaire format is appropriate.
Questionnaire use in 484.
1. As noted in the Milestone 3 writeup, all 484 teams must gather data with one or more questionnaires.
2. You can use questionnaires in two modes:
  1. as an integral part of the prototype-based ² usability study, to gather data about aspects of prototype use;
  2. as an adjunct to the prototype-based study, to gather additional data, most likely of a qualitative nature.
3. You will use multiple questionnaires when you have different user groups from whom to gather data, e.g.,
  1. The gatekeeper team will gather data from room-entry users (484 students) and administrative users (Byron, Greg).
  2. The 2d3d team may have a different form of questionnaire for outside- class subjects versus 484 student subjects.
  3. The menupad team may have a separate restaurant-owner questionnaire.
Observation (Section 7.6).
1. For all 484 teams, qualitative observation is a secondary form of data gathering, if used at all.
2. Quantitative observation via interaction logs may be useful for some teams, e.g., 2d3d and mobility.
3. Each team should consider what form of recorded observations it needs to do.
4. The most important consideration is to be as unobtrusive as possible.
5. The book has details on pages 321 - 342, some of which may be appropriate to your team's work:
  1. Field observation (Section 7.6.1).
  2. Observation in controlled environment (Section 7.6.2).
  3. Indirect observation via tracking user activities (Section 7.6.3).
Choosing and combining techniques (Section 7.7).
1. As noted in the preceding discussion of these notes, the questionnaire is the only required technique for 484.
2. Each team should carefully and thoughtfully consider what other data gathering techniques may be useful for your usability study.
3. Pages 342 - 346 of the book have some summarizing discussion.

Footnotes:

¹ Precisely defined, "triangulation" means the use of exactly three points to determine a geometric location; the malapropistic use here means simply "more than one".

² For 2d3d team, substitute the term "game-based" for "prototype-based".