Wow, that was a surprising contextual inquiry
My contextual inquiry was with the marketing manager. It was a real eye-opener. I realized that my team (including moi) did not understand what the users were doing with the reports, and that lack of understanding was reflected in the developer-centric way we designed the app. (By "developer-centric", I mean that the design reflected the development team's ideas on what would be sensible, and that those ideas weren't strongly informed by an understanding of the user's actual work practice.) I'll explain what I mean.
When I arrived at the interview, the marketing manager said, "Oh, you're in luck. Today we're going to balance the monthly budget." I was pretty excited about that, because I didn't know that this was a use case for our report. So the manager pulled up his Excel budget spreadsheet, which contains a list of lead vendors, along with various stats including the monthly budget.
Then I watched what was involved in actually balancing the budget. The manager would pull budget from one vendor so that he could give it to another. He did this based on vendor performance. If a given vendor has a history of overpromising on the number of leads it will deliver, then the manager has no choice but to decrease the lead cap and reallocate the budget to a vendor who is being more successful at reaching its cap. It's very important to the manager that the vendors generally reach their caps. If they don't, then we won't have enough leads for the month. If the shortfall is sufficiently great, the situation can be highly visible: "highly" meaning visible enough to impact the price of the stock. (That's pretty visible...) So what the manager would do is look at his list of vendors, reduce the lead cap for one vendor by some amount, and increase the lead cap for another vendor by some other amount that would preserve the overall monthly budget. (There's more to it than that, but that's the basic idea.)
After a few minutes of watching him balance the budget, it occurred to me that our current reporting app wouldn't correctly support his way of doing the work:
- In our app, the budget report is read-only. Lead caps are set on individual vendor pages. That means that in our app, the manager would have to keep bouncing back and forth between the report and the individual vendor pages.
- We built our app based on a snapshot of the vendor data. When I watched the manager work on the budget, I saw that he relied heavily on the ability to highlight certain cells in the spreadsheet as tentative. He would make them yellow so he would know to go back and call the vendor or otherwise confirm the data, and he did this enough times to suggest that the app should support this sort of activity. But the snapshot upon which we built the app had no such highlighting, because the manager gave us a "completed" report.
Despite the fact that the usability tests seemed to go well, and despite the fact that the users liked the app, it was obvious that what we were about to release would fall short, at least with respect to supporting budget balancing. (And I suspected we probably missed other things as well.) I mentioned my concerns to the marketing manager. While he agreed that current version had some issues, he thought that we should go ahead and release it anyway as the app would still be useful. So fortunately, we were close enough. Whew.
