Comments on: Measuring Gov 2.0 (Via Web 1.0): Forrester

By: Gwynne

Gwynne — Sun, 12 Apr 2009 00:11:00 +0000

Jed, Since I got you on the ropes, I am gonna finish you off with my right cross.

I don't think that user testing has to be project based. If we only talk to and interact with our users at the beginning of a project, we miss improvement opportunities. The question is how to illicit the nuggets? In the past, I have tested very narrow--but critical--questions in the hallway during a conference of my target group armed with a clipboard and some full color screen shots.

These conversations taught me--and my fellow testers--boatloads about people's expectations and the way they worked. A little, cheap interaction, especially if frequent, while being anecdotal and hard to quantify, can go a long way. Frankly, I wish we did this all the time!

See also Gerry McGovern; for example, Too Close to Your Website, Too Far from Your Customers.

Boom! (okay, don't think I delivered the knockout, since we are really in agreement, but it's a fun analogy)

By: Jed

Jed — Sat, 11 Apr 2009 23:48:00 +0000

I’m against the ropes!

I’m apparently using a very narrow definition of user testing that excludes a/b and multivariate testing, which I tend to look at as extensions of analytics. I’ve probably got my definition wrong.

While I think gathering users into a room to do card sorting exercises and examine paper prototypes can be extremely valuable at the outset of a large project, I think analytics (complete with a/b testing etc) is ultimately a better resource for ongoing in

Like Gwynne, I’m crazy about multivariate testing, and I’m thrilled to know that tools like Web Site Optimizer are on govt web managers’ radars. If we can allow persistent cookies, we’ll gain access to a very low cost way to user test via analytics.

And Andy, there’s some best practices gold at webcontent.gov. With regards to specifically to web 2.0, I’m hoping that the Social Media Subcouncil Wiki will become a great resource as well.

By: Andrew Krzmarzick

Andrew Krzmarzick — Sat, 11 Apr 2009 21:45:00 +0000

Gwynne,
Thanks for your thoughts. I agree with your summary – all of these evaluative tools/methods are good in combination. The “who” (champion for project, contributors to dynamic content, and the constituents/customers/citizens that are the ultimate end users) that comes right after the “why” (mission, goals, objectives) in implementing social media is critical to ensuring that the final “product” is appropriate and effective. Another question for both of you: What are less costly ways of accomplishing user testing?

By: Gwynne

Gwynne — Sat, 11 Apr 2009 20:40:00 +0000

@Jed, I respectfully disagree with the relative value of user testing. While it can be costly, I have NEVER EVER sat through a user testing session without being floored or surprised by something. Something that turned my thinking on it’s face.

User testing can be done simply using paper prototypes or by doing live A/B or multivariate testing. I am dying to try the google tools [see http://www.google.com/websiteoptimizer once the TOS issues are resolved.

Heuristics are good, analytics are good, user testing is good.

By: Andrew Krzmarzick

Andrew Krzmarzick — Fri, 10 Apr 2009 12:30:00 +0000

Jed – Thanks for your feedback. Where can one find those best practices that you cite? Is there a place where agencies can review and replicate the work of their compatriots? If so, I’d love to see it and learn more.

By: Jed Sundwall

Jed Sundwall — Thu, 09 Apr 2009 22:50:00 +0000

Thanks for posting this, Andrew. I agree that there is a need for multiple levels/flavors of analysis, but I’m a big fan of heuristic analyses and detailed anayltics. I’d prefer a combination of the two over user testing in most instances.

User testing is extremely costly to execute. It’s hard to fathom its cost across all government sites. Instead, it’d be great if we could provide incentives for agencies to adopt best practices identified by site analytics and heuristic analysis.