It’s pretty clear that author’s based their judgment on usability testing standards and their external validity,

“It does not help to compensate for a sloppy experimental design and by calling it a “qualitative study.”

Ouch, that’s a zinger!  But I can go one further: the test didn’t have an experimental design- at least by the standards I was taught in my psych research class.  No, I did an industry standard iterative usability study which does, ahem, provide statistically valid results (I had nearly 15 test subjects in the study, btw).

We get to see what is going on inside their heads, not just A/B comparisons like psych studies rely on.  If I wanted to do a psych study I would have just tested people with using Apple’s iPhone commercial-Apple probably did similar iterative studies like the one I did.  Indeed, if I they had published any of their data I would have considered an A/B study aimed to get statistical significance with controls and the like.

My one vote for acceptance talked only about the subject and wanted more treatment of side issues and other usability oriented discussions, accepting the iterative test design for what it is.  S/he also wrote “It is also well written and easy to read” giving me a 4/5 marks on the readability, the in between vote gave me a 3/5, and the hard no gave me a 2/5…

I do admit, now that I am a year or so further along in my cognitive psych studies my preliminary thinking on human perception and learning was fundamentally broken.  It followed a simplified cognitive psych model that usability practitioners use.  I often cringe during usability discussions and the (mis)interpretation of the users upper level behavior (thinking, decision-making, understanding) and how it’s linked to lower level processes (perception, attention, and recognition).  It’s also what bothers me about Malcom Gladwell’s work, which is a pretty unpopular position : /

Part of it is bloat- even my recent Review of Cognitive Psychology class didn’t discuss some established science I have been reading about on the side because it took 10 years for that research to become established.  My professor knew about the research, but the book and class structure hadn’t been updated.

However, I think the fundamental issue is  that usability is an engineering discipline and we abstract the lower level processes away because we don’t need that extra information to know that a user doesn’t get what s/he is being asked to do.  For example, most UI modeling assumes we can’t think about two things at once.  But we do.  It also assumes we can’t take in more than stream of information at a time.  But we do, at some level we “resolve” “all” information otherwise the “cocktail effect” (when you pick out your name at a chatty party) wouldn’t exist.  But even that metaphor isn’t an exact fit to the actual physical processes that are going on.

As they pointed out, “The study is organized and presented more as a pilot study, which is a perfectly legitimate step to take in a research program, but it is just a beginning. My recommendation is in agreement with the reviewers, that you should consider continuing the work but … as a more full-fledged research project.”  Now that Ubiquity is hibernating and I don’t care about marketing research there isn’t much reason to do a larger study.

In the end, I just want the paper in some research database so people know what to expect and how to do it.  I plan to do some lite editing (as this post probably needs) and I will resubmit to a journal more on the usability side of life.

This post is password protected. To view it please enter your password below:

Password:

Every page looks like this.  That’s what paying for editorial help gets you: real advice.

Below is an expert user, a former networking technician for Microsoft and an admin for their XBox Live division whom is familiar with command lines.  He totally missed the command premise, he mistook Ubiquity for the search box, also note how the user doesn’t read the suggestion text.

While trying to customize a skin I accidentally got Ubiquity without any styling… and found it very refreshing.

I also used Enso on my media server and took note of the lite nature of text decorations, syntax highlighting in the text entry field, as well as the auto-completion.

For test 7 I switched the skin to a modified Simpliquity skin with much larger font and a white background.  I also attempted to hack syntax highlighting.  However, I was unable to get delimiter highlighting working in the suggested arguments area in time for the tests.  Never-the-less, it seems to have helped.

The user picked up the command structure immediately.  The user still thinks the optional augments are not optional -a mistake several past participants made as well.  The most minimalist way to solve this is by having a vanilla command and command with optional arguments in the suggestion list, with a fuller explanation in the help text.

email this page

email message to name or email address

User #8 threw a wrench into my theory that syntax highlighting would convey the command and argument structure.  Although he did functionally score better than user 6 it was clear that I prompted him to search for weather in NYC.  Although, if you watch closely, he first types “ne” instead of nyc, mimicking the autosuggets.  Additionally, he also nails the modifier in “email to” after watching the video a second time through.

The confusion from #8 was enough that I tested another person.  Here is a Linux admin, getting it right out of the gate.  Big thanks to satyr for getting delimiter highlighting working!  (It’s in the nightlies, or as an xpi here.)

Users 7 & 9 are Linux users, while 6 & 8 are Windows admins.  Despite all being familiar with command lines, obviously the Linux admins will have a better pre-existing mental model.  Will see if future tests show continue to show an improvement.

Give feedback people!  Even mean feedback!  Especially mean feedback!

Update, Oct 24th

Just noticed that the compressor didn’t provide sound, grrrrr!  Will investigate…

]]> http://indolering.com/ubiquity/sessions6789/feed/ 0 http://indolering.com/ubiquity/summer09-4-5/ http://indolering.com/ubiquity/summer09-4-5/#comments Sat, 05 Sep 2009 20:08:01 +0000 Zachary Lym http://www.indolering.com/?p=202 Couple more sessions using the new translate commercial.

1. Users are unable to tell the difference between the two video clips, so slight differences are a no no.  Repetition may be key as users just don’t have time to digest what happened the first time around and are unable to spot differences between the two clips.
2. Users still see Ubiquity as command specific as opposed to a generalized command framework.  This is despite the generalized example command.

Will try two separate, unrelated commands next time.  The lack of users ability to generalize from the suggestions (despite attempts to do so) is also evidence for code highlighting.  Videos after the jump.

*Update* Here is the latest incarnation, now with real FF Meta!

You need a password for these, email me.

After I updated Ubiq I started getting these reminders notifications:

This is a common UI design pattern, one that is well intentioned but awkward.  About as well intentioned and awkward as the first time I asked a girl out- via “secret admirer” in fifth grade.  Just asking someone on a date is the quickest way to a fake phone number. The girl only knows your interested, not if your worth it.  Hell, she may even know your worth it, but is to distracted with other things.

Just throw it all out there, show the user what is valuable about the command and they will follow along:

Search faster: google search keywords

Secondly stick to seducible moments.  Try a reminder in to Google search-bar auto-suggest, or some extra highlighting of unused commands within ubiquity when opened on related sites.  This is something I touched on earlier in error pages.

At best, a manual can be oriented to predefined tasks and known problems, at worst it is a dense technical read with a broad overview intertwined with nitty-gritty details.  Neither approach considers how people actually work with documentation.  Firstly users never consult a manual unless they encounter a problem they cannot solve on their own.  And even if they do consult some form of help reference,

“[Users] encounter a usability problem on average about once every 75 minutes and typically spend about a minute looking for a solution” -When and How People Seek Help with Computer Applications

I call this the 1st rule  of triage support, a problem or question must be resolved within 60 seconds or it won’t be resolved unless the problem is really bad.  This law explains why Twitter and Get Satisfaction are so successful- users find the most up-to-date answer upfront instead of  guessing which chapter is most likely to contain the desired solution or digging through a forum posts.

Documentation and Help resources need to be thought of in holistic terms, as a part of the user experience spectrum instead of a disparate collection of tools.  To push us closer to that less abstracted understanding we can change the definition to user support. Abandoning the terminology of help and documentation and thinking in terms of user experience and support includes more than just manuals and forums.  This changes the strategic importance of different items, magnifying some things and reducing others.

Don’t think of Documentation

As best psychologists can tell, humans recognize and categorize things in two ways, as described by instance theory and abstraction theory.  Instance theory is that we store a central prototype for a category, with all sub-items being variations upon that prototype- an instance of it. ((Anderson, John. Cognitive Psychology and its Implications. New York: Worth Publishers, 2004.))  I believe we generally think of documentation via the instance method method because documentation also has a literal meaning of recording details and specifications, as in “The case was very well documented” and due to way developers implement documentation resources.

Documentation’s central prototype is a manual, of which there are many variations; wikis, FAQ, knowledge bases, etc.  Thinking in this manner will bring up related help prototypes (through spreading activation) such as a mailing list and it’s variations, such as a forum.  Such thinking prevents us from improving upon the satus-quo, it starts our thought process at a solution and walks backwards into the problem.

Abstraction theory holds that we string together different categories via shared characteristics. What do manuals, wiki’s, FAQ, knowledge bases, mailing lists, forums, GetSatisfaction, IRC, and web-chat all have in common?  What is “documentation” in the context of the user experience?

All of these things are just forms of user support.  I want to talk and think about user support as a collection of actions, products, and features compensating for when the user interface falls short of perfection.  This starts thinking at the problem and designs solutions for it.  Solutions which I call support structures.

I think that this seemingly small tweak in definition leads to big changes in our reasoning- and the effectiveness of support structures saving users from the alpha nature of Ubiquity.

Support Structures

The knowledge base/wiki

If a user fails to find relevant results on their first search, 50% will not search again ((

“47% of the users who failed only tried the search engine a single time. Another 30% tried twice. Less than 25% tried more than twice to get the search engine to produce a successful result…

Now, the designers of many of the sites we tested went to great lengths to get users to continue searching. They put in encouraging search tips that said “Try a new search using different terms.”

However, we did not see any evidence that these tips encouraged any user to search again. They pretty much assumed that the first (maybe second) try was the best they were going to get…

These results indicate that designers get one, possibly two chances to help users find their content with Search. If most of the users don’t find what they want in the first try, it doesn’t seem likely they will ever find it.” -

-People Search Once, Maybe Twice))

Yet most online documentation only offers links or a search box at the end of the page.  Sadly, the user is more likely to work around the problem or drop use of a feature -like Taskfox,

For these work-around episodes, the mean task time was 20 minutes and the median task time was 10 minutes. We estimate that use of available functions, had the participants known of them, would have reduced task times by 75 percent. -When and How People Seek Help with Computer Applications

So why not try to engage the user by offering a GSFN submission or chat support at the bottom of every page?

Click to enlarge

The Herd

The herd delivers information about a command, will show up high in google searches for the command, and will get a lot more traffic than any knowledge base we erect.  So why not make examples editable and sync them with the knowledge base?

Click to enlarge

This is extremely evolutionary, re-purposing material in pre-existing support structures.  It’s not a technological leap but a cognitive one.  Just as Ubiquity is only a cognitive leap- using a very old interface in places where it’s best suited for a task.

Error Pages

A seducible moment is when a user’s goal changes from performing their primary task to fixing the whatever is preventing them from accomplishing their primary task.

Google found that by placing the spell correction feature not only at the top, but also at the bottom of a search they were able to double it’s usage.  Users skimmed over what could have helped them originally because it was at the top.

Error pages could exploit this by offering a “best effort” landing page along with re-purposed knowledge-base materials.

Guided Exploration

Only a fraction of users have add-ons  due to the high barrier of actually adding them.  If we don’t just automatically add top-tier 3rd party commands we could at least offer them up on error pages.

However, we are limiting ourselves.  Google has replaced bookmarks, Wolfram is attempting to solve problems, and Bing wants to answer questions.  Isn’t Ubiquity not only a shortcut for these functions but also a replacement?  Why not embed suggestions where it is relevant, be it landing pages, a custom search page, or within the website itself?  Normal Google searches through Ubiquity could be redirected to a customized Google Custom Search page re-skinned with some JS magic:

Click to enlarge

This might seem hokey, and it’s effectiveness would be very limited in getting new commands onto Ubiquity.  But think of how the click-through rates and keywords from this “feature” can be used to improve the parsers logic.  User support isn’t just about teaching users, it’s a part of the user experience, it’s part of the testing cycle which improves Ubiquity.

Developer Support Structures

A more complex interface (such as MediaWiki, Trac, and Merc) filters out users who probably shouldn’t be patching code, logging bugs, or writing developer documentation.

However, the core developers are overworked trying to get the parser working,  internationalize it, integrate Taskfox, architect Jet Pack, design the UI, and help out on other Mozilla projects.  As a result, the core commands (among other things) have gone stale.  Commands are simple enough that even transient volunteers could improve them immensely.

I think a modified rule for these smaller items could be “Contributing shouldn’t take more than 60 seconds.”  Quality code takes longer than a minute- but hasn’t everyone found themselves spending hours on Wikipedia when they meant to just fix a minor thing?  Even hosting the commands on as individual GitHub projects would go long way towards increasing their exposure.

~

Please, leave negative feedback as well, challenge whatever I have to say.  I don’t get much feedback other than general “good job” stuff.  Not being a programmer, in mountain view, or being as visible as I could be really limits my understanding of the broader picture.  I often wonder if what I am doing is what what is most beneficial, or if I should change tactics or focus.  Take a moment to help me correct that : )

Example of the help page.

Unfortunately, the short-form videos embedded into the help pages were not working.  It would have been interesting to see if the these would have solved his problem.
He then navigated to the Google command help page where he took a cursory glance of the whole page and even moused over Ubibot’s speech bubble. Either he didn’t read the example or didn’t register the differences in the example and his mistakes because he again type in just google and waited.
This is one of the few cases where it would have been nice to have eye tracker data.  We would know if the user had read the suggestions at all but they were ineffectual or had skipped over them entirely, etc.
After an awkward wait, the user finally tried putting in a space followed by a search term. He then proceeded to gripe about how it wasn’t any better than the search bar.  The map function was interesting, but not that much of an improvement.  This strongly suggests that any video marketing clips needs to use page keywords and browser cookies in selecting which video to show.
The user seems to have either not read or not understood the commercials or the help information.  So, how do I make that easier?

• The intro commercial should be larger.
• Despite what my designer tendencies, I need to rip out the intro bubble and push the core help information to the top.
• I think that the in-line suggestions should be highlighted, somehow.  Perhaps in yellow instead of the muted gray.  Yellow has special properties that, like grey, let people skim over it but, unlike gray, still bring attention to itself.

If I get another user who has similar problems with the text, but is helped by the short help videos, maybe animating the example pictures will grab the users attention- something that the written instructions and static images fail to do.
Finally, I will try to hack together some kind of eye tracking solution. There are a few open source projects that do eye tracking, sadly they only work on Windows (and not on VMWare)- so I need to repartition my computer.
I think that my built-in iSight camera won’t be of high enough resolution to capture the corneal activity. So maybe I need to apply to the the Mozilla things and get an infrared camera, or even rent some higher=end eye tracking solutions.
If anyone knows some HCI professionals in the Seattle area who could loan me some eye tracking equipment for the weekend- let me know.