Indolering.com » summer09

Marketing UI Research Paper: Some Love From ToCHI

Zachary Lym — Wed, 19 May 2010 21:30:41 +0000

One vote for, one vote in the middle, and one vote against publication. Essentially, they wanted more formal psych study about video and learning which requires a different kind of statistical validity than what usability provides. My error was writing this like a psych study (which I have written before) and not like a usability study. My one solid “yes” vote, the editorial encouragement, and my usability mentors all tell me the paper is worthy of publication -just in a more usability oriented journal.

It’s pretty clear that author’s based their judgment on usability testing standards and their external validity,

“It does not help to compensate for a sloppy experimental design and by calling it a “qualitative study.”

Ouch, that’s a zinger! But I can go one further: the test didn’t have an experimental design- at least by the standards I was taught in my psych research class. No, I did an industry standard iterative usability study which does, ahem, provide statistically valid results (I had nearly 15 test subjects in the study, btw).

We get to see what is going on inside their heads, not just A/B comparisons like psych studies rely on. If I wanted to do a psych study I would have just tested people with using Apple’s iPhone commercial-Apple probably did similar iterative studies like the one I did. Indeed, if I they had published any of their data I would have considered an A/B study aimed to get statistical significance with controls and the like.

My one vote for acceptance talked only about the subject and wanted more treatment of side issues and other usability oriented discussions, accepting the iterative test design for what it is. S/he also wrote “It is also well written and easy to read” giving me a 4/5 marks on the readability, the in between vote gave me a 3/5, and the hard no gave me a 2/5…

I do admit, now that I am a year or so further along in my cognitive psych studies my preliminary thinking on human perception and learning was fundamentally broken. It followed a simplified cognitive psych model that usability practitioners use. I often cringe during usability discussions and the (mis)interpretation of the users upper level behavior (thinking, decision-making, understanding) and how it’s linked to lower level processes (perception, attention, and recognition). It’s also what bothers me about Malcom Gladwell’s work, which is a pretty unpopular position : /

Part of it is bloat- even my recent Review of Cognitive Psychology class didn’t discuss some established science I have been reading about on the side because it took 10 years for that research to become established. My professor knew about the research, but the book and class structure hadn’t been updated.

However, I think the fundamental issue is that usability is an engineering discipline and we abstract the lower level processes away because we don’t need that extra information to know that a user doesn’t get what s/he is being asked to do. For example, most UI modeling assumes we can’t think about two things at once. But we do. It also assumes we can’t take in more than stream of information at a time. But we do, at some level we “resolve” “all” information otherwise the “cocktail effect” (when you pick out your name at a chatty party) wouldn’t exist. But even that metaphor isn’t an exact fit to the actual physical processes that are going on.

As they pointed out, “The study is organized and presented more as a pilot study, which is a perfectly legitimate step to take in a research program, but it is just a beginning. My recommendation is in agreement with the reviewers, that you should consider continuing the work but … as a more full-fledged research project.” Now that Ubiquity is hibernating and I don’t care about marketing research there isn’t much reason to do a larger study.

In the end, I just want the paper in some research database so people know what to expect and how to do it. I plan to do some lite editing (as this post probably needs) and I will resubmit to a journal more on the usability side of life.

Got my paper back from the editor

Zachary Lym — Sun, 22 Nov 2009 20:49:11 +0000

Every page looks like this. That’s what paying for editorial help gets you: real advice.

Sessions 6, 7, 8, & 9

Zachary Lym — Sun, 11 Oct 2009 22:26:51 +0000

Despite my best efforts on the documentation front, users still failed to grasp the simplistic syntax scheme of Ubiquity. I suspected this was due to poor contextual “documentation.” So I designed an experiment testing out a simpler skin.

Below is an expert user, a former networking technician for Microsoft and an admin for their XBox Live division whom is familiar with command lines. He totally missed the command premise, he mistook Ubiquity for the search box, also note how the user doesn’t read the suggestion text.

While trying to customize a skin I accidentally got Ubiquity without any styling… and found it very refreshing.

I also used Enso on my media server and took note of the lite nature of text decorations, syntax highlighting in the text entry field, as well as the auto-completion.

: From the Humanized website

For test 7 I switched the skin to a modified Simpliquity skin with much larger font and a white background. I also attempted to hack syntax highlighting. However, I was unable to get delimiter highlighting working in the suggested arguments area in time for the tests. Never-the-less, it seems to have helped.

The user picked up the command structure immediately. The user still thinks the optional augments are not optional -a mistake several past participants made as well. The most minimalist way to solve this is by having a vanilla command and command with optional arguments in the suggestion list, with a fuller explanation in the help text.

email this page

email message to name or email address

User #8 threw a wrench into my theory that syntax highlighting would convey the command and argument structure. Although he did functionally score better than user 6 it was clear that I prompted him to search for weather in NYC. Although, if you watch closely, he first types “ne” instead of nyc, mimicking the autosuggets. Additionally, he also nails the modifier in “email to” after watching the video a second time through.

The confusion from #8 was enough that I tested another person. Here is a Linux admin, getting it right out of the gate. Big thanks to satyr for getting delimiter highlighting working! (It’s in the nightlies, or as an xpi here.)

Users 7 & 9 are Linux users, while 6 & 8 are Windows admins. Despite all being familiar with command lines, obviously the Linux admins will have a better pre-existing mental model. Will see if future tests show continue to show an improvement.

Give feedback people! Even mean feedback! Especially mean feedback!

Update, Oct 24th

Just noticed that the compressor didn’t provide sound, grrrrr! Will investigate…

Sessions 4 & 5

Zachary Lym — Sat, 05 Sep 2009 20:08:01 +0000

Couple more sessions using the new translate commercial.

Users are unable to tell the difference between the two video clips, so slight differences are a no no. Repetition may be key as users just don’t have time to digest what happened the first time around and are unable to spot differences between the two clips.
Users still see Ubiquity as command specific as opposed to a generalized command framework. This is despite the generalized example command.

Will try two separate, unrelated commands next time. The lack of users ability to generalize from the suggestions (despite attempts to do so) is also evidence for code highlighting. Videos after the jump.

*Update* Here is the latest incarnation, now with real FF Meta!

You need a password for these, email me.

Learning Ubiq P1: Minimalist Documentation

Zachary Lym — Mon, 13 Jul 2009 21:22:39 +0000

It has been over a month since my last post -testing has taken a backseat to reading research in social learning theory and technical communications. But now I have some more solid ideas I want to share in this 3-part series.

Today I am going to talk about a typical users, how they learn, and the principals of minimalist documentation. If you want to skip the fluff, head to the minimalist style guidelines for Ubiquity’s documentation.

Reasoning

(skip this if your not a psych geek ; )

Albert Bandura defined three cognitive prerequisites to learning:

Motivation
Attention
Retention

Tutorials and other traditional documentation which strive to give conceptual overviews are the best way to learn. Volumes of psychological research prove that the thorough processing prompted by a conceptual overview improves retention and performance.

However, the typical setting for traditional psychological tests of learning guarantee both motivation and attention -such as work training. Web browsers are a glass pane- their purpose is to deliver content, their utility invisible, their new features just smudges on the glass, obscuring the view.

Watching professors at school is a lesson on how people interact with FireFox and obstructions to their goals. If FireFox doesn’t start immediately they become flustered and switch to IE. If FireFox asks to apply an update s/he quickly clicking “cancel” to whatever pops up. They are trying to teach a class, not learn new skills or perform maintenance!

Even in my usability studies, when participants actually have the goal of learning the interface, they are seemingly incapable of reading the documentation. Only one user, who was very insecure about his technological prowess, methodically read the tutorial. After that, his performance was great. However, that means that the tutorial only works for 10% of a dedicated user-base, while 90% are left in the dark.

The test was a bit skewed; users were artificially motivated, but they also had the discoverability problem which Taskfox will fix, but many still had either poor performance or didn’t value the functionality that Ubiquity added.

So what do we do about this? The most important area is the auto-suggest, it must be magical. I have some ideas on how to add to that, but first some lower-level stuff must be fixed. The most boring, but essential topic, is how we configure our documentation.

About Minimalist Documentation

The minimalist documentation style, as outlined by Carroll in The Minimal Manual, shows massive improvements in real-world tests where motivation is not guaranteed. It has subsequently become the dominant paradigm in the technical communication field. The overall guiding principals of this movement are:

Procedural Instruction
Minimal Wording
Error Recovery
Guided Discovery

Procedural Instruction

By focusing on procedural, task based instruction, minimalist documentation caters to how users naturally browse documentation and think about their problems.

Users interaction with documentation occurs when they are inhibited in performing a task. When users to kill the paper clip, they do not take a tutorial on how to use word or read from the beginning of the manual- they skip to the “paper clip” chapter hoping to find something about turning it off.

Thus minimalist documentation is focused on examples of tasks, there is no attempt to teach the underlying models of operation. Real users are trying to do something, not understand the grand scheme of things.

Secondly, minimalist documentation is as non-sequential as possible. Each example is independent of the previous example, requiring no knowledge that the user likely skipping anyway.

Minimal Wording

Minimalist documentation attempts to cut down the thicket of prose obstructing the users view of a potential solution. Users are skimming, facilitate that by leaving only the most essential wording, writing inverted pyramid style, breaking rules of grammar if it increases clarity, and bolding key verbs is used to help users pick out important information.

Error Recovery

Attempt to provide clear checkpoints and examples so a user can make sure that they have carried out the example correctly. Screenshots should crop non-relevant information (unless it visually orients the user) and emphasize important areas.

If something is particularly tricky or can cause harm, warn users and give options of undoing the damage or reseting the environment.

Guided Discovery

Ubiquity guides users to the functions that they want, and keeps the unnecessary ones hidden from view via it’s auto-suggest. I will offer some suggestions on how to improve auto-suggest in a more automated way in Part 3.

Read (the much better, imho) Part 2

Go to the Ubiquity Minimalist Documentation guidelines