Transferable Research Skills (Part A)

MGUIDE is my most up to date research work, and I am very proud of what I have accomplished. However, I’ve become eager to outgrow the domain and transfer my research skills to the digital media world. I am interested in any form of digital interactive applications (websites, social networks, interactive-tv, search engines, games, etc). I am highly experienced in using the following techniques for user research: 

A: User Research

A1.0 Quantitative Research

Surveys/Questionnaires (Online and Offline):

Post-test and pre-test questionnaires provide real insights into user needs, wants and thoughts. I use powerful statistics (e.g., Cronbach's Alpha) to ensure that the questionnaires I create, are both reliable and valid. I can apply these skills into any domain with minimum adaptation time.

Performance Measures:

Performance measures, like for example, time to complete a task, the number of errors conducted, scores in retention tests, etc,  provide strong indication of how easily people can achieve tasks with a system. If this data are correlated with other objective or subjective measures they can provide deeper user insights than surveys/questionnaires alone.

Log File Analysis:

A log is a file that lists actions that have occurred. Both quantitative and qualitative data can be stored in a log file for later analysis. I use device/system logs to automatically collect data such as time to complete a task, items selected on the interface, keys pressed etc.

---

A2.0 Qualitative Research:

Focus Groups:

I mainly use focus groups for requirements gathering, either through the introduction of new ideas and discussion and/or the evaluation of low-fidelity prototypes.

Direct Observation:

One of the most common techniques for collecting data in an ethnographic study is direct, first-hand observation of participants. I am experienced in using direct observation for note taking in both indoor and outdoor environments.I find gaining an understanding of users through first-hand observation of their behaviour while they use a digital system, genuinely exciting. During my work in MGUIDE direct observation was used to uncovered a number of interesting user-insights that were then correlated with user views collected from the questionnaires.

User Interviews & Contextual inquiry:

Other common ethnographic techniques are user interviews and contextual Inquiry. I use extensively open-ended interviews  i.e., interviews where the interviewees are all asked the same-open ended questions, in both field and lab conditions. I like the particular style as it is faster and can be more easily analysed and correlated with other data.

Think Aloud Protocol:

Think-aloud is a technique for gathering data during a usability testing session. It involves participants thinking aloud as they are performing a set of specified tasks. I have used think-aloud very successfully in navigation tasks, where participants had to verbalise their answers to navigation problems as those presented by two interactive systems.

---

A3.0 Quantitative & Qualitative Research

Usability and Accessibility testing:

Lab-based and Field-based testing are the most effective ways of revealing usability and accessibility issues. I am experienced in conducting and managing lab and field testing. I use scenario-based quantitative and qualitative methods for my research.

Continues in the next post

Universality of Research Methods & Techniques

I thought that the universality of methods for research was a fundamental fact of modern science. Isn’t it obvious that having successfully applied quantitative/qualitative research in one domain means that your skills can be applied to any other domain with minimal adaptation time? Is there a real difference between applying qualitative research in a complex avatar-system like MGUIDE and an e-commerce web site? For example, If you apply techniques like unstructured interviews wouldn’t you follow the same principles to design the interviews in both domains?

Or even using more complex techniques like eye tracking  and emotion recognition, aren’t these domain-independent? Consider for instance, my combined emotion recognition + face detection technique for accessibility research, described in the previous post. The technique was developed with avatar-based interfaces/presentation systems in mind. Adapting the technique to different domains is a matter of defining the aspects of the interface you wish to research. The quantitative data that you will collect are the same (emotion intensities, etc), the qualitative of course will differ because the interfaces are different. In general once you establish the objectives/goals of the research, deciding which  techniques you will use (and modifying them if necessary  to suit your needs) is easy and the process is domain-independent.

 

Eye tracking used in completely different contexts: a) a 3D avatar-based world and b) a web page.

I am not sure why some people insist otherwise and focus so much on the subject matter. I have to agree that having expertise in a certain area, means that you can produce results fairly quickly. However this is a process easily learnt. Is domain expertise the most important quality a user researcher should have? Or should he perhaps have a solid research skills-set to start from and the willingness to learn more about established techniques and explore new ones?

Video Games & Online Games

This post in an attempt to disambiguate the domain of virtual humans. Most people have never heard the term “Virtual Human” before, but they all play games (online or offline) and they all have interacted with some limited form of a VH on the web.

Computer games (online and offline) are the closest thing to the domain of Virtual humans.

Online games (e-gaming)

You can argue that online games are much simpler than video games, but they are progressively getting much more complicated. As in video games, fully-fleshed avatars are widely used to immerse the player into the scenario. Below is an example of a poker game I found from a company called PKR. Notice the use of body language, face expressions, etc to create a fully realistic poker simulation.  

Video Games:

Below is a screenshot from my favourite game Mass Effect:

Source: http://www.jpbrown.co.uk/reviews.html 

Notice the use of dialogue wheels to simulate dialogues between the avatars. There is an excellent analysis of the particular style of conversation here: 

However, in contrast to most current video games Virtual humans engage players in actual dialogue, using speech recognition, dialogue system technology, and emotional modelling to deepen the experience and make it more entertaining. Such technologies have started only recently to find their way into video games. Tom Clansy’s End War game is using speech recognition to allow users to give commands to their armies.

Source: http://www.the-chiz.com/

Some games go as far as using full natural language processing:

Virtual Humans on the Web:

There are a lot of very superficial virtual humans on the web. This is perhaps one of the main reasons that they have failed so far to become a mainstream interface. What virtual humans should be, is about the whole thing: emotion modelling, cognition speech, dialogue, domain strategies and knowledge, gestures etc. Avatars like Anna of IKEA are mere drawings with very limited dialogue abilities, and are simply there to create a more interesting FAQ (Frequently Asked Question) Section. There is still someway to go before we will see full-scaled avatars on the web, but we will get there.

 

  Source: http://www.ikeafans.com/forums/swaps-exchanges/1178-malm-bed-help.html

 

E-Learning Prototype

Below is the prototype of a e-learning system that I was asked to do by a company. As I can not draw, I decided to use Microsoft Word to communicate my ideas. There should be a good storyboarding tool out there that could help me to streamline the process.

The design below is based on existing and proven technologies that can be easily integrated into existing e-learning platforms. Codebaby, a company in Canada is already using avatars (such as those shown in my design) [1] in e-learning very successfully for several years. The picture in the last screen of the design is a virtual classroom [2] created in the popular Second Life platform.

 

Compare my solution with a “conventional” e-learning platform shown below. Although I do include several GUI (Graphical User Interface) elements in my work, it is obvious that : a) my interface is minimalistic with fewer elements on the screen. b) accessibility is greater, as instead of clicking on multiple links in order to accomplish tasks, you can simply “ask” the system using the most natural method you know - “natural language”. The benefits of avatar-assisted e-learning will become evident when the web progresses from its current form to Web 2.0 and ultimately to Web 3.0. For now, such solutions should at least be offered as an augmentation to “conventional” GUI-based interfaces. All companies want something more, for example something that would add easier access to module contents and  the “WOW” factor to their products. They just don’t know what it is until you show it to them.

 

Source: http://www.accessplanit.com/accessplan_lms_screen_shots

 

Although the proposed design is based on mature and well-tested technologies, I can understand if someone wants the purely GUI solutions. In fact, I would be more than happy to assist them. I have been working with GUI interfaces for several years, long before I developed an interest for avatar technologies. I developed my first e-learning tool back in 1998 (12 years ago). It was an educational CD-ROM about the robotic telescope platform of Bradford University.

 

[1] http://www.codebaby.com/showcase/

[2] http://horizonproject.wikispaces.com/Virtual+Worlds

Heuristics vs. User Research

People keep asking me about the W3C accessibility guidelines – a set of heuristics that should aid designers towards more accessible web sites. Of course these are not the only guidelines out there, BBC has it own accessibility guidelines and  there are several for web usability as well. Although I am familiar with the W3C guidelines, I didn’t use them in my MGUIDE work because I didn’t find them relevant. The reason is that the W3C guidelines are written specifically for web content and not for multimodal content. The research is the area of virtual humans provide more relevant heuristics, but there is still room for massive additions and improvements. Instead of heuristic evaluation, I decided to built my own theoretical framework to guide my research efforts. The framework is based on the relevant literature in the area and on well documented theories of human cognition. It provides all the necessary tools for iterative user testing and design refinement. 

There is no doubt that relying on user testing is costly and lengthy. This becomes even more difficult, when you have to deal with large groups of people as I did in MGUIDE. However the cost and time can be minimised with the use of proper tools. For example, the global lab project has created a virtual lab (on the popular Second Life platform) in which projects are accessible to anybody, anytime, and from anywhere. New research methods like eye tracking and emotion recognition, can reveal user insights with a relatively small group of people and with minimal effort. Soon enough perhaps, tools will include routines that calculate deep statistics with minimal intervention. User testing has definitely some way to go before it becomes mainstream, but I am sure we will get there.

Until then, Inspection methods (e.g., cognitive walkthroughs, expert surveys of the design, heuristic evaluations etc) are used to replace user testing. In such a process, some high level requirements are usually prototyped, and then judged by the expert against some established guidelines. A major problem with this approach though, is that there are over 1,000 documented design guidelines [1]. How do you choose which is one is proper given the specific context? It is my understanding that each institute/professional uses a set of best-practice guidelines, adapted from the relevant literature and from years of experience. However, even if these guidelines have worked in the past it doesn’t mean they will work again. Technology is progressing extremely fast, and people become more knowledgeable, and more accustomed to technology every single second. Therefore, even when inspection methods are used some form of user testing is necessary. A focus group for example, with a couple of users can provide enough user-insights to amend a design as necessary.

[1]http://www.nngroup.com/events/tutorials/usability.html 

 

 

Emotion Recognition for Accessibility Research

There are a number of quantitative techniques that can be used in the user research of avatar-based interfaces. Apart from the “usual” techniques for gathering subjective impressions (through questionnaires, tests, etc) and performance data, I also considered a more objective technique based on emotion recognition. In particular, I thought of evaluating the accessibility of the content presented by my systems through the use of emotion expression recognition. The main hypothesis is that the perceived accessibility of the systems' content is evident in the user's emotional expressions. 

If you think about it for a while, the human face is the strongest indicator of our cognitive state and hence, how we perceive a stimuli (information content, image, etc). Emotion measures (both quantitative and qualitative) can provide data that can augment any traditional technique for accessibility evaluation (e.g., questionnaires, retention tests, etc). For example, with careful logging you can see which part of your content is more confusing, which part requires the users to think more intensively, etc. In addition to the qualitative data, the numeric intensities can be used for some very interesting statistical comparisons.  Manual coding of the video streams is no longer necessary, as there are a number of tools that allow automating analysis of face expressions. To my knowledge the following tools are currently fully functional:

1) Visual Recognition

2)   SHORE

The idea is fully developed, and I am planning the release of the paper very soon. Finally, If we combine this technique with eye-tracking we can reveal even more user-insights about avatar-based interfaces. We could try for instance to identify, what aspect of the interface make the user to have the particular face expression (positive or negative). For example, one of the participants in my experiments mentioned that she couldn’t pay attention to the information provided by the system, because she was looking at the guide’s hair waving. To such a stimuli humans usually have a calm expression. This comment is just an indication of the user-insights that can be revealed, if these techniques are successfully combined.

Accessibility/Universal Access

I recently found a good resource [1] on accessibility from a company called Cimex that says what most designers and UX specialists fail to see – when you design for accessibility you do not cater only for less able users. You are making sure that your content is open and accessible to a variety of people and machines using whatever browser or method they choose.

Now, caring for a variety of people of different physical, cognitive emotional and language background and the methods they choose to use you end up with Universal Access.   

Using traditional interfaces is difficult to achieve the goals of Universal Access. Virtual humans as interfaces hold a high potential of achieving the goals of UA as the modalities (e.g., natural language processing, gestures, facial expressions and others) used in such interfaces are the ones our brains have been trained to understand over thousand of years. Virtual humans can speak several languages with a minimal effort (see the Charamel showroom). Their body and face language can be adjusted easily to highlight the importance of a message. Sign-language can be used to communicate with less-able users (no other interface can currently accomplish that). Accurate simulation of interpersonal scenarios (e.g., a sales scenario) can guarantee that your message gets across as effectively as it would if a real person would speak it.

In my work I did go as far as Universal accessibility by comparing the effects of virtual human interfaces on the cognitive accessibility of information under simulated mobile conditions, using groups of users with different cultural and language background. In order to make the information easier to access, I used a variety of methods found in the VH interfaces (e.g., natural language processing, gestures, interpersonal scenario simulation and others). By making the main functions of your system easier to access you ultimately make the interface easier to use and hence, it was natural to investigate some usability aspects as well (e.g., ease of use, effectiveness, efficiency, user satisfaction, etc). These are all aspects of the user experience (UX), i.e., the quality of experience a person has when interacting with a product. I can not release any more information at this stage, as the necessary publications have not yet been made.

In the future I believe that the existing technologies will merge into two mainstream platforms: a) Robotic assistants from the one and b) software assistants/virtual human interfaces from the other. Accessing the services these systems will offer will be as easy as our daily interactions with other people. The barriers that exist today (cognitive, physical, etc) will become a “thing” of the past.