Usability Testing Pitfalls
One of the applications I've been involved with developing at work is a reporting app for marketing. The app basically allows marketing users to keep track of lead vendor data, such as cost per lead, conversion rates, monthly budgets, etc.
When the marketing manager first approached us about the project, the requirements were a development team's dream. Very straightforward. The manager gave us the reports as an Excel file, and asked us to turn them into a web app. All the calculations were already built into the spreadsheets. Moreover, this particular manager happens to be an Excel power user with a strong visual design sense, so he had already helped us out by making the reports look good. We figured we would pretty much copy the spreadsheets.
We implemented the application along the lines suggested by the spreadsheets. We didn't do any kind of up-front discovery, or spend any time finding out exactly how the marketing users actually use the various reports. We just looked at the calculations the Excel reports were doing and emulated them. When we had questions, we just called the manager and asked. Everything seemed to be pretty good.
What we thought the tasks were
We wanted to make sure everything was in fact OK before releasing the reporting app, so we conducted some usability tests. As you may know, to conduct usability tests, you identify a representative set of realistic user tasks, and then observe the users trying to use your application to complete the tasks. So we came up with various tasks that we thought were probably representative, given the app's feature set, such as:
- Add a new contact to a given vendor
- Enter the vendor's self-reported lead counts for such-and-such week
- Bring up the Lead Cost to Date report for March 2005
We conducted the usability tests, and for the most part the users were in fact able to complete the tasks. There were some exceptions, but they were nothing major, and the solutions were more or less obvious once we observed the problems. The users indicated that they really liked the app, and that they thought it would help them out. So we made the fixes and then went into QA.
It was about this time that we wanted to look into expanding the scope of the app, and so we decided that that was big enough to warrant doing some discovery. We decided to use contextual inquiry as a way to understand the users' work.
Here's where it gets interesting. :-)
