Giannis Doumanis

My Complete Portfolio

2011-01-15T16:59:00.001-08:00

My complete portfolio can be found here. I recently realised that my blog although impressive, it has a lot of text. Hence, I decided to create a professional portfolio (with a nice and clean design) to demonstrate my skills and abilities more effectively. Please feel free to download it.

iPAD – Multitouch Web (Part B)

2010-08-30T12:44:00.001-07:00

Source: Some blog

Research

There is already a usability study of the IPad. from the Nielsen Norman group here. A summary of the study can be found here. As Neilsen admits the study is preliminary, but the resulting usability insights serve as a good foundation for the design of the myriads of applications that will follow the release of the device on the global market.

The study is very generic – it tested several applications and web sites running on the iPad device. As every digital project requires a unique testing context that takes into account its unique range of parameters, more focused studies are necessary. Again existing research methods and techniques must be tailored accordingly to take into consideration the multi-touch style of interaction.

Take as an example, m-commerce, which according to many is the next “big” thing in the mobile world. In my opinion, multi-touch web pages if done correctly, hold a high potential to make our transactions easier than even our desktop computers. How would you design an m-commerce web site in order to achieve such a goal? The guidelines discussed in Part A of this post, are a good place to start in order to construct some initial prototypes. However, as these guidelines are far from best-practises, an iterative cycle of participatory design workshops for a few days is necessary in order to agree on a final prototype.

Gathering user insights after the web site is released, currently seems challenging. The usability studies I’ve read so far, use lab-based testing with one-to-one sessions. However testing with real-users in lab-conditions is always expensive. Cheaper techniques, like remote usability testing currently seem very hard to implement. Then, do existing tools for split or multivariate testing (e.g., Google Web site Optimizer) work on a multi-touch interface? These tools are optimized for a mouse-based environment, and I am not convinced that they can be used effectively on multi-touch. For instance, does Google Web Site optimizer registers multi-touch gestures, as well as clicks when it comes to measuring the success of a web site? Nevertheless, it will be very interesting to see how these techniques for research will be adapted, to serve the new environment in the years to come.

iPAD – Multitouch Web (Part A)

2010-08-29T09:40:00.001-07:00

Image via Wikipedia

I finally had the chance the test the new IPad device. I spent some time trying to figure out if it really worth spending 400 pounds on this device. Here are my findings:

1) The device is excellent for gaming. It is perhaps one of the best gaming devices I’ve ever used. The integrated gyroscope means, that are no annoying arrow keys to use while playing games (all you have to do is to turn the device)

2) For i-reading, although the screen is very clear the absence of integrated back stand (like Samsung’s UMPC devices) make it very hard to hold it for a very long time.

3) From all the applications I tested I only found one of particular interest. It is an application that shows you the star constellations based on your geographical position.

4) The device has to flash support, which means that the majority of the WWW content is out of reach. Advocates of the device say that as the Web is progressively moving to the HTML 5.0 standard, that will soon stop be an issue. Advocates of Flash say that Flash can not die, as it is an integral part of the web. Who is right, and who is wrong only time will tell. For now all I can say is that if I buy the IPad, my favourite episodes of “Eureka” on Hulu are out of reach for good.

5) The device is multi-touch, which means that it is very hard to operate on mouse-based web pages. As the majority of mobile platforms are now moving towards multi-touch, what does this mean for designers, IA, researchers and other stakeholders? Below I attempt to present some of the possible implications for designers, IA and researchers.

Design

Size of Finger

Web pages on touch-sensitive devices are not navigated using a mouse. They are controlled with human fingers, many of which are much fatter than a typical mouse pointer. No matter what apple says about an “ultimate browsing experience” on IPad, clicking on small text links with your finger is painful, and sometimes practically impossible. As touch-sensitive devices become more popular, this could mean the end of traditional text-links and their replacement by big touchable buttons.

Secondary Functions

The “fat finger” problem discussed above, and the limited screen estate, also mean that we can not cram thousand of features (or ads) in a tight frame as we would in a desktop web page. The design of web pages should be focus on the essential elements, and it should avoid wasting user attention on processing secondary functions.

Hover effects and Right Clicks

Without a mouse-based interface, you can’t use any mouse over effects. Elements that we are so used to interact with on mouse-driven interfaces, like menus that pop-up when you hover you mouse over a link, or right clicks, do not exist. Apple has a number of replacements in place, like holding your finger down on a link to get a pop-up menu, but they only make clicking itself more complex.

Horizontal Vs. Vertical Styling

Due to the ability to easily switch between vertical and horizontal orientations, web sites will have to automatically adapt their styling to look accordingly in both orientations. Seamless presentation in both landscape and portrait mode is one of the most fundamental guidelines when it come to designing for the IPad/IPhone devices.

3D Objects

Apple is trying to push designs that immediate tangible things – real world interfaces that are easy to understand and familiar in their use. If you create a magazine application make it look like a real magazine, or if you make a word processor make it look like a type writer. Could iPad be a significant milestone towards a more three-dimensional WWW? Some web 3D applications like Second Life, could certainly benefit from the mouse-less interface, as touching and tilting make it much easier to interact with 3D worlds than mousing and keyboarding. In mainstream websites, 3D elements (e.g., material surfaces and SFX) will probably be used widely as an “invitation to touch”, but never as a basic metaphor.

Information Architecture

Multi-touch presents a unique set of challenges for information architects. The limited screen size and the size of the human finger tip, means that limited number of actions needed to complete one task (its tiresome to swipe and touch too often) pushes the IA to create a dead simple architecture with minimal number of actions. Information aggregation, will play a very important role into creating architectures that minimize input and maximize output.

Under the above rule, several more “human-like” modalities of communication, such as speech recognition, text to speech processing, emotion recognition, natural language processing etc, are likely to find their place into the multi-touch web. Multi-touch seems to me the perfect vehicle towards the right direction, away from the dominance of GUI interfaces and towards a more natural way of interacting with computers.

Continues in the next post

Cognitive Walkthrough - ICT Virtual Human Toolkit

2010-08-17T18:04:00.001-07:00

As part of the MGUIDE project, I had to complete the cognitive walkthrough of the ICT Virtual Human Toolkit. This toolkit is a collection of the state-of-the-art-technologies including: speech recognition, automatic gesture generation, text to speech synthesis, 3D interfaces, dialogue model creation to name but a few. Current users of the toolkit include, CSI/UCB Vision Group at UC Berkeley, Component Analysis Lab at Carnegie Mellon University, Affective Computing Research group at MIT Media Lab and Microsoft Research.

An assemble of some of the characters created by the toolkit.

Source: University of Southern California Institute for Creative Technologies

The main idea behind the evaluation, was to provide usability insights on what is perhaps the most advanced platform for multimodal creation on the planet today. The process was completed successfully with 2 experts, and revealed a number of insights that were documented carefully. These insights will be fed to the design of the Talos Toolkit – among the MGUIDE deliverables was an authoring toolkit to aid the rapid prototyping of multimodal applications with virtual humans. Talos is currently just an architecture (see here), but the walkthrough of the ICT toolkit provided some valuable insights that should guide its actual design. However the MGUIDE project was completed, with the development of Talos set for the future goals of the project.

I applied the cognitive walkthrough, exactly as I would applied it in any other project. I performed a task analysis first (i.e., i established the tasks I wanted to perform with the toolkit I broke them into actions) and then, I asked the following questions at each step:

1) Will the customer realistically be trying to do this action?

2) Is the control for the action visible?

3) Is there a strong link between the control and the action?

4) Is feedback appropriate?

Ultimate – A prototype search engine

2010-08-09T14:38:00.001-07:00

Lately, I have been experimenting again with Axure on a prototype search engine called Ultimate. The engine is based on actual user requirements collected through a focus group study. I decided to prototype Ultimate, in order to perfect my skills in Axure. The tool enabled me to construct a high-fidelity and fully functional prototype within a few hours. Some of the features of the engine are:

It relies on full natural language processing to understand the user’s input.
The search algorithm is based on a complex network of software agents – automated software robots programmed to complete tasks (e.g., monitor the prices of 200 airlines, get ratings from tripadvisor.co.uk, etc) - to deliver accurate and user-tailored results.
Some of the engine functionalities are discussed in the user journey shown below. The full functionalities are well documented, but for obvious reasons I can not discuss them in this post.

Screenshots:

User Journey:

Research:

Run a usability testing of the above design against a more “conventional” search engine (e.g., Skyscanner ), and I am certain that the results will show the clear superiority of Ultimate. Of course, there is the need for a careful usability study in order to compare the designs but I am certain that Ultimate is superior in all usability metrics.

Bespoke Research Solutions

2010-08-06T08:54:00.001-07:00

The research methodologies discussed in previous posts provide an excellent basis to start from, but the web is changing rapidly. Rich Internet Applications (Flash, Silverlight, Ajax, etc), new interaction methods (e.g., multi-touch, gesture recognition, etc) will flood the web are already here. Can we apply what we know in terms of user- research in these new environments? Consider as an example a corporate web site, a mobile artificial intelligent assistant, and an RIA application. Conducting a usability study in the first scenario, is perhaps straightforward. However what techniques and measures are the most relevant to the second and third scenarios? What can we apply in order to ensure that we can indeed gather rich user insights? What are the most relevant tools to deploy? - Siri is a mobile application, while the rest are desktop applications. Unfortunately I don’t have the answers, as I have never attempted a similar study before. I can only imagine that existing techniques would have to be tailored to the unique and complex range of variables. Therefore, research adaptation is far more important that the domain itself.

Present:


Source: HalCrow	Source: Some Blog	Source: Microsoft

Future (Aurora) ??:

I like thinking about the future, a lot! Aurora from Mozila labs, is a project that aims to redesign the way we browse the web. Currently, It is merely a concept, but it gives a pretty good idea of how the future will be like. There is an excellent critique of the Aurora project here. If the future will be similar to Aurora (i.e., reinventing the user interface), what does this mean for user-research? We need to fundamentally re-think the way that we conduct user research. New techniques will have to be invented and existing ones revisited. Innovation and creativity, will distinguish the companies that survive from the ones that will go out of business.

Aurora (Part 1) from Adaptive Path on Vimeo.

User Research Deliverables

2010-08-05T09:45:00.001-07:00

Different institutes require user research deliverables to be formatted differently. In MGUIDE I had to provide both within the allocated budget and timeframes:

Industry: Usability reports with actionable recommendations were delivered to each of the companies involved in the MGUIDE project. Each company wanted user-insights on the particular piece of technology that contributed to the project. For example, the following recommendation was of particular interest to the text-to-speech company:

Actionable Recommendations:

Provide a visible and easy to use method for users to decrease/increase the rate of the Text-to-Speech output while the application speaks.

-Users will likely find the output more natural and easy to understand

Note: I can not release any detailed research findings as they are the property of the institutes that supported the project.

Personas: Personas are a technique used to summarize user-research findings. In my understanding, personas are made-up people used to represent major segments of a product’s target audience. There is an excellent explanation of personas here. Personas are easy to construct, and a great way to distil research findings into a simple and accessible form.

Source: WebCredible

Academia: In academia statistical significance is of major importance. I presented deliverables in similar format to those above, but accompanied by statistics in the proper format (e.g., F (1, 14) = 7.956; p < 0.05). Statistics appear to be of little interest to the industry though.

Transferable Research Skills (Part B)

2010-08-05T06:48:00.001-07:00

Remote Usability Testing:

Remote testing, is about conducting usability testing without having participants come into the lab. Although there are several tools and web services on the market, I prefer to work with userfeel because of the low cost, and their massive network of testers from all over the globe.

A/B and Multivariate Testing:

A/B and multivariate testing, is about testing different versions of the same design, in order to see which performs the best. I use this technique in all of my usability tests, either by differentiating my designs across one variable (i.e., A/B testing) or more (i.e., multivariate testing).

Co-Discovering Learning:

My approach to co-discovering learning, is as follows: I usually ask two or more users to perform a task together, while I observe them. I encourage them to converse and interact with each other to create a “team spirit”. In some cases, I also allow note taking (e.g., when the content is technical/complex). The technique can yield some really powerful results, as it is more natural for users to verbalise their thought during the test.

Participatory Design:

Participatory design, is about involving users into the design and decision-making process into an iterative cycle of designing and evaluation. I usually conduct a short participatory design session, prior all of my usability evaluations. In these sessions, the usability issue of a prototype system are determined, and the changes to accommodate for these issues are made. The refined system is then used in the actual usability evaluation.

A4.0 Inspection Methods

Cognitive Walkthrough:

The cognitive walkthrough is a method of “quick and dirty” usability testing requiring a number of expert evaluators. A list of tasks and the actions to complete them is created. The evaluators step through each task, action by action, noting down problems and difficulties as they go. I can use cognitive walkthroughs on a number of digital interfaces, ranging from web sites to complex authoring toolkits.

Heuristic Evaluation:

Heuristic evaluation is about judging the compliance of an interface against a number of recognized usability principles (i.e., the Heuristics). I used this method extensively in the evaluation of e-learning prototypes during my teaching at Middlesex University.

A5.0 Advanced Usability Techniques (in training)

Eye Tracking:

Eye tracking is a technique that pinpoints where the users look on a system and for how long. I am currently talking with Middlesex University in order to get training on using eye-tracking as a usability testing technique. We plan to conduct a series of eye-tracking session in Middlesex state-of-the-art usability labs, using the MGUIDE prototypes.

Emotion Recognition & Eye Tracking

This is a technique I developed during the MGUIDE project. I discuss it in detail here. It was developed with avatar-based interfaces/presentation systems in mind, but it is universal in nature. It is based on the hypothesis that the perceived accessibility of a system’s content is evident in the user's emotional expressions. The combined “Emotion Recognition and Eye-tracking” technique will be validated in a lab-based study that will be performed at Middlesex University.

A5.0 Audits

Accessibility Audit:

In accessibility audit, an expert checks the compliance of a web site with established guidelines and metrics. The W3C WAI are the most widely used guidelines in accessibility audits. My approach for accessibility evaluation is framework-based (see here), but a) I haven’t applied my framework with disabled users and b) the W3C WAI heuristics are very well established. Although I have a good knowledge of the W3C WAI heuristics, I have never performed an accessibility audit before.

Transferable Research Skills (Part A)

2010-08-04T17:36:00.001-07:00

MGUIDE is my most up to date research work, and I am very proud of what I have accomplished. However, I’ve become eager to outgrow the domain and transfer my research skills to the digital media world. I am interested in any form of digital interactive applications (websites, social networks, interactive-tv, search engines, games, etc). I am highly experienced in using the following techniques for user research:

A: User Research

A1.0 Quantitative Research

Surveys/Questionnaires (Online and Offline):

Post-test and pre-test questionnaires provide real insights into user needs, wants and thoughts. I use powerful statistics (e.g., Cronbach's Alpha) to ensure that the questionnaires I create, are both reliable and valid. I can apply these skills into any domain with minimum adaptation time.

Performance Measures:

Performance measures, like for example, time to complete a task, the number of errors conducted, scores in retention tests, etc, provide strong indication of how easily people can achieve tasks with a system. If this data are correlated with other objective or subjective measures they can provide deeper user insights than surveys/questionnaires alone.

Log File Analysis:

A log is a file that lists actions that have occurred. Both quantitative and qualitative data can be stored in a log file for later analysis. I use device/system logs to automatically collect data such as time to complete a task, items selected on the interface, keys pressed etc.

A2.0 Qualitative Research:

Focus Groups:

I mainly use focus groups for requirements gathering, either through the introduction of new ideas and discussion and/or the evaluation of low-fidelity prototypes.

Direct Observation:

One of the most common techniques for collecting data in an ethnographic study is direct, first-hand observation of participants. I am experienced in using direct observation for note taking in both indoor and outdoor environments.I find gaining an understanding of users through first-hand observation of their behaviour while they use a digital system, genuinely exciting. During my work in MGUIDE direct observation was used to uncovered a number of interesting user-insights that were then correlated with user views collected from the questionnaires.

User Interviews & Contextual inquiry:

Other common ethnographic techniques are user interviews and contextual Inquiry. I use extensively open-ended interviews i.e., interviews where the interviewees are all asked the same-open ended questions, in both field and lab conditions. I like the particular style as it is faster and can be more easily analysed and correlated with other data.

Think Aloud Protocol:

Think-aloud is a technique for gathering data during a usability testing session. It involves participants thinking aloud as they are performing a set of specified tasks. I have used think-aloud very successfully in navigation tasks, where participants had to verbalise their answers to navigation problems as those presented by two interactive systems.

A3.0 Quantitative & Qualitative Research

Usability and Accessibility testing:

Lab-based and Field-based testing are the most effective ways of revealing usability and accessibility issues. I am experienced in conducting and managing lab and field testing. I use scenario-based quantitative and qualitative methods for my research.

Continues in the next post

Universality of Research Methods & Techniques

2010-08-02T17:48:00.001-07:00

I thought that the universality of methods for research was a fundamental fact of modern science. Isn’t it obvious that having successfully applied quantitative/qualitative research in one domain means that your skills can be applied to any other domain with minimal adaptation time? Is there a real difference between applying qualitative research in a complex avatar-system like MGUIDE and an e-commerce web site? For example, If you apply techniques like unstructured interviews wouldn’t you follow the same principles to design the interviews in both domains?

Or even using more complex techniques like eye tracking and emotion recognition, aren’t these domain-independent? Consider for instance, my combined emotion recognition + face detection technique for accessibility research, described in the previous post. The technique was developed with avatar-based interfaces/presentation systems in mind. Adapting the technique to different domains is a matter of defining the aspects of the interface you wish to research. The quantitative data that you will collect are the same (emotion intensities, etc), the qualitative of course will differ because the interfaces are different. In general once you establish the objectives/goals of the research, deciding which techniques you will use (and modifying them if necessary to suit your needs) is easy and the process is domain-independent.

Eye tracking used in completely different contexts: a) a 3D avatar-based world and b) a web page.

I am not sure why some people insist otherwise and focus so much on the subject matter. I have to agree that having expertise in a certain area, means that you can produce results fairly quickly. However this is a process easily learnt. Is domain expertise the most important quality a user researcher should have? Or should he perhaps have a solid research skills-set to start from and the willingness to learn more about established techniques and explore new ones?

Video Games & Online Games

2010-07-31T18:26:00.000-07:00

This post in an attempt to disambiguate the domain of virtual humans. Most people have never heard the term “Virtual Human” before, but they all play games (online or offline) and they all have interacted with some limited form of a VH on the web.

Computer games (online and offline) are the closest thing to the domain of Virtual humans.

Online games (e-gaming)

You can argue that online games are much simpler than video games, but they are progressively getting much more complicated. As in video games, fully-fleshed avatars are widely used to immerse the player into the scenario. Below is an example of a poker game I found from a company called PKR. Notice the use of body language, face expressions, etc to create a fully realistic poker simulation.

Video Games:

Below is a screenshot from my favourite game Mass Effect:

Source: http://www.jpbrown.co.uk/reviews.html

Notice the use of dialogue wheels to simulate dialogues between the avatars. There is an excellent analysis of the particular style of conversation here:

However, in contrast to most current video games Virtual humans engage players in actual dialogue, using speech recognition, dialogue system technology, and emotional modelling to deepen the experience and make it more entertaining. Such technologies have started only recently to find their way into video games. Tom Clansy’s End War game is using speech recognition to allow users to give commands to their armies.

Source: http://www.the-chiz.com/

Some games go as far as using full natural language processing:

Virtual Humans on the Web:

There are a lot of very superficial virtual humans on the web. This is perhaps one of the main reasons that they have failed so far to become a mainstream interface. What virtual humans should be, is about the whole thing: emotion modelling, cognition speech, dialogue, domain strategies and knowledge, gestures etc. Avatars like Anna of IKEA are mere drawings with very limited dialogue abilities, and are simply there to create a more interesting FAQ (Frequently Asked Question) Section. There is still someway to go before we will see full-scaled avatars on the web, but we will get there.

Source: http://www.ikeafans.com/forums/swaps-exchanges/1178-malm-bed-help.html

E-Learning Prototype

2010-07-30T18:28:00.000-07:00

Below is the prototype of a e-learning system that I was asked to do by a company. As I can not draw, I decided to use Microsoft Word to communicate my ideas. There should be a good storyboarding tool out there that could help me to streamline the process.

The design below is based on existing and proven technologies that can be easily integrated into existing e-learning platforms. Codebaby, a company in Canada is already using avatars (such as those shown in my design) [1] in e-learning very successfully for several years. The picture in the last screen of the design is a virtual classroom [2] created in the popular Second Life platform.

Compare my solution with a “conventional” e-learning platform shown below. Although I do include several GUI (Graphical User Interface) elements in my work, it is obvious that : a) my interface is minimalistic with fewer elements on the screen. b) accessibility is greater, as instead of clicking on multiple links in order to accomplish tasks, you can simply “ask” the system using the most natural method you know - “natural language”. The benefits of avatar-assisted e-learning will become evident when the web progresses from its current form to Web 2.0 and ultimately to Web 3.0. For now, such solutions should at least be offered as an augmentation to “conventional” GUI-based interfaces. All companies want something more, for example something that would add easier access to module contents and the “WOW” factor to their products. They just don’t know what it is until you show it to them.

Source: http://www.accessplanit.com/accessplan_lms_screen_shots

Although the proposed design is based on mature and well-tested technologies, I can understand if someone wants the purely GUI solutions. In fact, I would be more than happy to assist them. I have been working with GUI interfaces for several years, long before I developed an interest for avatar technologies. I developed my first e-learning tool back in 1998 (12 years ago). It was an educational CD-ROM about the robotic telescope platform of Bradford University.

[1] http://www.codebaby.com/showcase/

[2] http://horizonproject.wikispaces.com/Virtual+Worlds

Heuristics vs. User Research

2010-07-30T10:13:00.001-07:00

People keep asking me about the W3C accessibility guidelines – a set of heuristics that should aid designers towards more accessible web sites. Of course these are not the only guidelines out there, BBC has it own accessibility guidelines and there are several for web usability as well. Although I am familiar with the W3C guidelines, I didn’t use them in my MGUIDE work because I didn’t find them relevant. The reason is that the W3C guidelines are written specifically for web content and not for multimodal content. The research is the area of virtual humans provide more relevant heuristics, but there is still room for massive additions and improvements. Instead of heuristic evaluation, I decided to built my own theoretical framework to guide my research efforts. The framework is based on the relevant literature in the area and on well documented theories of human cognition. It provides all the necessary tools for iterative user testing and design refinement.

There is no doubt that relying on user testing is costly and lengthy. This becomes even more difficult, when you have to deal with large groups of people as I did in MGUIDE. However the cost and time can be minimised with the use of proper tools. For example, the global lab project has created a virtual lab (on the popular Second Life platform) in which projects are accessible to anybody, anytime, and from anywhere. New research methods like eye tracking and emotion recognition, can reveal user insights with a relatively small group of people and with minimal effort. Soon enough perhaps, tools will include routines that calculate deep statistics with minimal intervention. User testing has definitely some way to go before it becomes mainstream, but I am sure we will get there.

Until then, Inspection methods (e.g., cognitive walkthroughs, expert surveys of the design, heuristic evaluations etc) are used to replace user testing. In such a process, some high level requirements are usually prototyped, and then judged by the expert against some established guidelines. A major problem with this approach though, is that there are over 1,000 documented design guidelines [1]. How do you choose which is one is proper given the specific context? It is my understanding that each institute/professional uses a set of best-practice guidelines, adapted from the relevant literature and from years of experience. However, even if these guidelines have worked in the past it doesn’t mean they will work again. Technology is progressing extremely fast, and people become more knowledgeable, and more accustomed to technology every single second. Therefore, even when inspection methods are used some form of user testing is necessary. A focus group for example, with a couple of users can provide enough user-insights to amend a design as necessary.

[1]http://www.nngroup.com/events/tutorials/usability.html

Emotion Recognition for Accessibility Research

2010-07-28T17:25:00.001-07:00

There are a number of quantitative techniques that can be used in the user research of avatar-based interfaces. Apart from the “usual” techniques for gathering subjective impressions (through questionnaires, tests, etc) and performance data, I also considered a more objective technique based on emotion recognition. In particular, I thought of evaluating the accessibility of the content presented by my systems through the use of emotion expression recognition. The main hypothesis is that the perceived accessibility of the systems' content is evident in the user's emotional expressions.

If you think about it for a while, the human face is the strongest indicator of our cognitive state and hence, how we perceive a stimuli (information content, image, etc). Emotion measures (both quantitative and qualitative) can provide data that can augment any traditional technique for accessibility evaluation (e.g., questionnaires, retention tests, etc). For example, with careful logging you can see which part of your content is more confusing, which part requires the users to think more intensively, etc. In addition to the qualitative data, the numeric intensities can be used for some very interesting statistical comparisons. Manual coding of the video streams is no longer necessary, as there are a number of tools that allow automating analysis of face expressions. To my knowledge the following tools are currently fully functional:

1) Visual Recognition

2) SHORE

The idea is fully developed, and I am planning the release of the paper very soon. Finally, If we combine this technique with eye-tracking we can reveal even more user-insights about avatar-based interfaces. We could try for instance to identify, what aspect of the interface make the user to have the particular face expression (positive or negative). For example, one of the participants in my experiments mentioned that she couldn’t pay attention to the information provided by the system, because she was looking at the guide’s hair waving. To such a stimuli humans usually have a calm expression. This comment is just an indication of the user-insights that can be revealed, if these techniques are successfully combined.

Accessibility/Universal Access

2010-07-27T15:31:00.001-07:00

I recently found a good resource [1] on accessibility from a company called Cimex that says what most designers and UX specialists fail to see – when you design for accessibility you do not cater only for less able users. You are making sure that your content is open and accessible to a variety of people and machines using whatever browser or method they choose.

Now, caring for a variety of people of different physical, cognitive emotional and language background and the methods they choose to use you end up with Universal Access.

Using traditional interfaces is difficult to achieve the goals of Universal Access. Virtual humans as interfaces hold a high potential of achieving the goals of UA as the modalities (e.g., natural language processing, gestures, facial expressions and others) used in such interfaces are the ones our brains have been trained to understand over thousand of years. Virtual humans can speak several languages with a minimal effort (see the Charamel showroom). Their body and face language can be adjusted easily to highlight the importance of a message. Sign-language can be used to communicate with less-able users (no other interface can currently accomplish that). Accurate simulation of interpersonal scenarios (e.g., a sales scenario) can guarantee that your message gets across as effectively as it would if a real person would speak it.

In my work I did go as far as Universal accessibility by comparing the effects of virtual human interfaces on the cognitive accessibility of information under simulated mobile conditions, using groups of users with different cultural and language background. In order to make the information easier to access, I used a variety of methods found in the VH interfaces (e.g., natural language processing, gestures, interpersonal scenario simulation and others). By making the main functions of your system easier to access you ultimately make the interface easier to use and hence, it was natural to investigate some usability aspects as well (e.g., ease of use, effectiveness, efficiency, user satisfaction, etc). These are all aspects of the user experience (UX), i.e., the quality of experience a person has when interacting with a product. I can not release any more information at this stage, as the necessary publications have not yet been made.

In the future I believe that the existing technologies will merge into two mainstream platforms: a) Robotic assistants from the one and b) software assistants/virtual human interfaces from the other. Accessing the services these systems will offer will be as easy as our daily interactions with other people. The barriers that exist today (cognitive, physical, etc) will become a “thing” of the past.

MGUIDE Development Process

2010-07-26T15:16:00.001-07:00

I thought it would be a good idea to try to explain the methodologies followed in the development of the MGUIDE prototypes. Having a focus mainly on the research outcomes, the development methodology followed was of little concern to the involved stakeholders. Trying to create interpersonal simulations like the ones found in real-life is a process mostly compatible with the a Scrum development methodology (shown below). I am planning to create a paper on the topic, and hence I will not say much in this post.

Source: Wikipedia

Gathering the requirements of the users can be done using a variety of ways. I followed a combine literature-user evaluation approach. One of my earliest prototypes was developed using guidelines found in the literature. The prototype was then evaluated with actual users and a set of new requirements was developed. These requirements are what the SCRUM refers to as the “product backlog”. Each spring (usually in my case 1-3 months) a set of the requirements were developed and tested, and then were replaced by a new set of requirements. Doing simulations of interpersonal scenarios gives you the freedom to augment the product backlog with new requirements quite easily. Using methods of research like direct observation and note taking, you can take notes on the interactions found in the scenarios that you want to simulate. My scenario was a guide agent and hence, I went to a number of tours where I made a number of interesting observations. Most of my findings were actually developed in the MGUIDE prototypes, but there are others that still remain in the “product backlog”. Of course these requirements and the work that was done in the MGUIDE is enough to inform Artificial intelligence models of behaviour in order to create completely automated systems.

This iterative process was then repeated prior the actual user research stage, where the full set-up of the MGUIDE evaluation stage was tested. I used a small group of people that tried to find bugs in the software, problems with the data gathering tools and others. The problems were normally corrected on site and the process was repeated again. Once I ensured that all my instruments were free of problems, the official evaluation stage of the prototypes started.

Closing this post, I must highlight the need for future research in gathering data about different situations where interpersonal scenarios occur. In reality different situations produce different reactions in people and this should be researched further. Only through detailed empirical experimentation we can ensure that future avatar-based systems will guarantee superior user experiences.

Speech Recognition

2010-07-23T08:40:00.001-07:00

In order to successfully simulate an interpersonal scenario with a virtual human, you need speech recognition (in real-life we speak to each other and not click on buttons or use text). For this reason, I have been following closely the evaluation of the speech recognition industry for some time now.

During the MGUIDE project I successfully integrated speech recognition into one of my prototypes. I used Microsoft Speech Recognition Engine 6.1 (SAPI 5.1) with dictation grammars which I developed using the Chant GrammarKit, in pure XML. The grammars look like this:

<RULE name="Q1" TOPLEVEL="ACTIVE">
<l>
<RULEREF NAME="want_phrases"/>to begin
<RULEREF NAME="want_phrases"/>to start
<RULEREF NAME="want_phrases"/>to start immediately
</l>
<opt>the tour </opt>
<opt>the tour ?then</opt>
</RULE>

I also voice-enabled the control of the interface of my system, so if you would say “Pause” the virtual guide would pause its presentation. I briefly tested both modes with one participant in the lab. In the dictation mode, with just a couple of minutes of training Microsoft’s engine performed with 100% accuracy within the constrains of the grammar. For completely unknown input, the engine performed with less than 40% accuracy. In CnC mode, the engine worked with 100% accuracy without any training. Of course, SAPI 5.4 in Windows 7 offer much better recognition rates in both dictation and CnC modes. I haven’t tried SAPI 5.4 but is within my plans for the future.I think that true speaker-independent (i.e., without training) recognition in indoor environments, is only 5 years away, at least for the English language.

In mobile environments, Siri appears to be the only solution out there that realises the idea of a virtual assistant on the go using speech recognition. Siri works uses dynamic grammar recognition, similar to my approach. If you say something within the constrains of the grammar the accuracy of recognition reaches 100%. However, as in the case of my prototype, if you say something outside the grammar files the recognition results can be really funny.

Statement to Siri: Tell my husband I’ll be late

Reply: Tell my Husband Ovulate (he he he)

Source: http://siri.com/v2/assets/Web3.0Jan2010.pdf

Terminology:

Dictation Speech Recognition: Refers to the type of speech recognition where the computer tries to translate what you say into text

Command and Control mode (CnC): This type of speech recognition is used to control applications

MGUIDE Project Funding

2010-07-23T03:22:00.001-07:00

As people keep asking me about the funding of the MGUIDE project, I thought to post this in an attempt to clarify the situation further. The MGUIDE was a large and very sophisticated project and money come from a variety of resources. The project started in 2007 and until 2008, Middlesex University was the main funding body and my last official employment institute. The package from Middlesex University covered my project expenses for that year and required me to perform maximum 15 hours of teaching/week. Two other Universities and six companies also provided support in the form of know-how, and funding for tools and hardware. From 2008 until June 2010, I was able to secure funding from an angel investor and thankfully the continued support of the companies and universities. The idea with the MGUIDE was and still remains to develop a commercial product out of it. However, because of the bad economic climate, my investor decided not to proceed any further. I still hope that this work will appeal to a company and I will be able to see MGUIDE as an application/system for Ipad or any other tablet-based computer system.

Project Management - E-Learning Projects

2010-07-21T16:29:00.001-07:00

This post is not related to the MGUIDE project, but to the work I did at Middlesex University. Most of the modules I taught there were project based. Usually I had to guide several groups of students (20-30 students) into the design and development of projects. A particular project had to do with the design and implementation of e-learning games for autistic children. Each of my students was given a case study describing the requirements of the particular autistic children (as these were captured from the teachers of the children, like for instance, that the children needs help in understanding emotion expressions), and had to produce a game under my guidance. The game was hen sent to the relevant school for full-scale evaluation. Each semester I usually ended up marking 100-200 games with at least 80% of them being top class. Below is an example of the projects produced under my guidance. All material is copyrighted by Middlesex University so please ask before you copy anything:

All multimedia elements (including designs) have been produced by my students using Adobe Photoshop. The tools needed for the game development along with best practice techniques, were discussed in detail in the class by myself.

View Full Album

Each game was evaluated in the class (by myself and the students). The games were then sent to the schools for formal evaluation by the children and their teachers. Below is a sample of the heuristic evaluation that was performed in the class:

Evaluation comments

Negative Comments

Positive Comments

Background
First scene using a suitable background and clearly stating your own title for the topic and clear instructions

Text
A variety and clear use of Text (Spelling and Grammar?) Too small – Too large – not clear- inappropriate words used

Art Assets

2010-07-19T16:50:00.001-07:00

Below are samples from the art work I completed in the MGUIDE project. Although I have several years of experience in designing using Adobe Photoshop, I do not consider myself a designer. Design is interesting, but I prefer programming and user-based research. However, if a project requires me to produce art assets I am perfectly capable of accomplishing that as well.

View Full Album

Experiment 4 setup

2010-07-02T09:45:00.000-07:00

Due to demand, I decided to provide some information on the set-up of my experimental work during the evaluation stage of MGUIDE. The information below is the briefing participants had to read for experiment 4. Please note, that the main technique for data collection in experiment 4, is think aloud protocol. I conducted the testing with two user groups of 6 participants.

The purpose of this experiment is to investigate the possible effects of two mobile systems for path finding of variable complexity, on your ability to find your way in the castle. You will have to use the systems to navigate along two different routes visiting a number of landmarks in turn (10 to maximum 18), using the system A on one, and the system B on the other. The total duration of the experiment doesn’t exceed 20 minutes per route. For the purpose of the experiment I have created two video applications representing in detail each route of the castle. At each video-clip you will hear the question “What would you do at the particular point, if you were in the castle?” You must answer the question based on the visual (i.e., gestures and landmarks) and/or audio instructions delivered by the system.

For example:

Given this instruction:From where you are, if you look on your right, you will see two chimneys. Opposite the chimneys there is a path that leads to another square. Please follow this path until you will come across a house with a black front-door!

And this clip:

You will have to answer: “I will follow the path on the right of the tree until I see the house with the black front-door”. The next video will show the result of this action (i.e., that you have moved towards the house with the black door), and will pose you a new navigational challenge.

User Research – General Setup

2010-06-30T18:08:00.000-07:00

User research in the MGUIDE project can be divided into the following stages:

Requirements Gathering and specification: A prototype system was constructed based on recommendations from the literature on animated agents. The prototype was evaluated in the actual castle of Monemvasia with a number of participants. You can find more details here. The lessons learned were presented during the M.Phil presentation and influenced the design of the final five (5) prototypes.

Design and Prototypes: Based on requirements gathered from the pilot in the castle of Monemvasia, five(5) prototypes were developed. A number of novelties were achieved during the developed of the prototypes (e.g., an algorithm for natural language understanding, the design of Talos – a toolkit for system prototyping and research, and others). Although the initial idea was to continue the evaluation in the castle, due to lack of resources and time it was decided to simulate the conditions in the lab. There is an on-going debate on whether mobile applications should be tested in labs or fields. For instance, in 2009 70% of the developed systems were evaluated under lab conditions using a variety of techniques.

Evaluation: The evaluation carried out in Greece and the UK, followed the same setup. I used detailed panoramic photography and high definition video-clips to represent in high detailed all locations and attractions of the castle. The lab was a simple room in which each user participated individually. In general the approach was successful, as participants could follow the same routes and watch the same presentations about the locations of the castle, from the comfort of their chair.

Examples of the panoramas used in the evaluations is shown below:

View Full Album

User Research Stage

2010-06-30T16:46:00.000-07:00

I recently completed the evaluation stage of all five (5) prototypes of the MGUIDE project. The purpose of the studies was to provide accessibility and usability insights into the design process of animated agents for mobile applications, as well as, to support some understanding of the psychology of the users of such systems. One hundred (100) real-world users participated. My role included the following:

Design and apply various user -centred design methods (UCD) for research including: structured questionnaires, retention tests, interviews, think aloud protocols, observations, performance measures, scenario-based usability testing, to name but a few.

Note: For a complete list of the research techniques I excel at, see here (part A) and here(part B)

Design, organize and manage five user research studies (with 100 real – world users) across two countries (Greece and the UK)

Continuously refine of the prototypes based on feedback collected by the users

Regularly update all project sponsors and stakeholders of the progress made

Statistically analyze the quantitative data using various statistical tests

Rapidly analyze the qualitative data

Develop recommendation/requirements to impact the design of animated agents for mobile applications

Produce documentation and presentation of findings

Write papers for publication in relevant conference proceedings and academic journals.

As no publications have been made yet, I can not provide any further details on my experimental work. Once the publication process is completed I will post further details on this very important stage of my work.

I want to thank all the students/family/friends/other people that honoured my work with their presence. Their contribution is valuable.

Complex System Architecture 2

2010-06-09T12:45:00.000-07:00

Below is the final architecture of the Talos toolkit - my authoring toolkit for rapid prototyping of Virtual guide systems and research. The design is complete with a number of modules that need detail explanation. Some of these ideas were implemented in MGUIDE, but implementing the full toolkit is a task best suited for a team.

The only other toolkit in existence (free of charge for research) is the ICT Virtual Human Toolkit. I have performed a cognitive walkthrough of the ICT toolkit, and fed the results into the design of Talos. I discuss my findings on a paper that will be published soon.

Please note: For obvious reasons I can not provide any documentation on the workflow of the Talos toolkit. The purpose of the diagram is ONLY to illustrate the complexity of my work.

Complex System Architecture

2010-06-06T18:02:00.000-07:00

Below, you will find some sample from the complex system and subsystem architecture I did for the MGUIDE project. In particular, a) the general system overview of all prototypes b) is the workflow of the MGUIDE’s Natural Language Understanding (NLU) module. For obvious reasons I can not provide any documentation on the workflow of the two architectures below. The purpose of the diagrams are ONLY to illustrating the complexity of my work.

Apart from the semantic analyser (shown in the first diagram as experimental) everything else was fully prototyped and evaluated in the user-research stage of the project.

Rapid Prototyping, Which tool?

2010-06-04T18:04:00.000-07:00

Below is an experiment to simulate one of the MGUIDE prototypes using Axure RP Pro. I wanted to see if Axure can be used in such applications, and how fast I could implement a working prototype.The dialogue window is clickable, as well as the buttons. All standard GUI (Graphical User Interface) elements of the MGUIDE interface can be quickly and easily implemented using this tool. The environment is drag drop, supporting adding interactions using visual elements (e.g., If then else statements with a few clicks). However, the absence of an internal scripting language means that I am limited by the number of interactional elements the program has already build in. I can’t build my own interactions, like for example the dialogue branching I easily implemented using VB.NET. Neither can I import my 3D avatar, for lets say a formal evaluation. In conclusion, Axure appears to be a tool to be used very early in the product cycle and mostly for low-fidelity prototypes. Later on, you you will need to build something complex in order to communicate your ideas more effectively. This is where I come in! I can built highly complex prototypes, in the same amount of time using VB.NET/Adobe Director with just a few lines of code. However, if a a stakeholder is happy with this tool, I can be happy as well. The tool can be used virtually by anybody, let alone myself.

System Preferences Wireframe

Main Screen Wireframe

Sample Portfolio 2

2010-05-31T18:02:00.000-07:00

Several more prototypes can be constructed and evaluated around the virtual guide agent with minimal effort. However, I think its time to present some architectural designs on which the system was based upon:

Architecture-related software: The architectures below are made mainly using MindJet Mindmanager.

Hub-and-Spokes like Dialogues:

Prototype 1 features a Finite State Machine (FSM) dialogue manager capable of dynamically displaying questions based on the user’s selection and the current context. The questions cover a very broad range of the possible questions/clarifications that a user can ask after a presentation for a location. This FSM was used in most systems except prototype 4 where the user could ask questions using natural dialogue. The flow diagram below gives an idea of how dialogue is brunched within the system.

Please note that only the branching for questions 1 and 4 is shown in the diagram below. In addition, although not shown the final hubs all terminate in the “Dialogue Terminate” state.

Short explanation:

The user listens to an introductory script from the guide agent and then, they can either select to listen to the presentation about a location or have a look around. Once the presentation is complete, a hub with questions is loaded from where the user can select what to ask . After hearing the guide’s answer, the user can either return to the main hub (i.e., by selecting “I want to ask something else”), from which they can ask another question, or enters a deeper hub with more options to choose from. The user can either exhaust all questions available to them and proceed to the next location, in which case the dialogue enters the “Dialogue Terminate State” or simply tap the button next on the interface of the system to proceed.

System Architecture:

This is one of my first attempts to design Talos – an authoring toolkit for Virtual guide development and research. The architecture is far from complete, but it serves as an excellent basis for designing the final architecture of the toolkit. Unfortunately, I can not provide any explanation on the toolkit or its modules, as I have to publish a paper on it. However, the main idea of Talos is to provide a cheap solution for researchers wishing to create virtual guide prototypes, as well as, for content writers (e.g., guide book writers) wishing their content to be presented through a truly multi-modal medium.

View Full Album

Software Structure:

1) This is my first attempt to implement a simple UI to fully/partially automate the development of Virtual humans. This prototype has many interesting features, that implemented during my M.Phil years (2003-2006), like for instance a Cyc query builder aimed to automate the transformation of questions into a format suitable to be inserted into CyN AIML scripts.

2) A system structure of a system I built during the M.Phil years (2003-2006)

View Full Album

Sample Portfolio

2010-05-29T18:01:00.000-07:00

Information architecture is the “art and science of organizing and labelling websites, intranets, online communities and software to support usability.”[1]. MGUIDE takes information architecture to the next level by suggesting human modalities (gestures, facial expressions, emotion recognition, real-time user information adaptation, etc) as an alternative method for supporting this goal. I have done extensive experimental work on trying different human modalities and methods for ensuring effective information architecture. Technology nowadays in the avatar domain is so advanced, that we have the ability to simulate various human-to-human communication scenarios, in order to ensure that our message will get across effectively. Such a scenario is for example, the use of humour/serious messages to attract attention when someone is now paying attention to what we are saying. The list of these scenarios is endless and so is the experiments that can be performed to gather some evidence on their application in the virtual world. I truly believe that the future of Web is 3D and hence, avatar-based applications will be the next “big thing”. No matter what we do, no matter how well we will design a web site it can not be compared with a humanised interface for information retrieval and processing. Human gesturing, natural language processing, emotion recognition, and other human modalities if all combined together correctly, they can create the most powerful paradigm for our proliferated information age.

Hence, the MGUIDE is not related to the “traditional” web page model. It goes towards the Web 2.0 applications, i.e., applications with high interactivity. Any architectural designs that you will find related to MGUIDE, are complex software blueprinting, dialogue hubs and branching and others. However, this doesn’t mean that I can’t do information architecture for web sites/internet applications or that I haven’t done it in the past. Actually I find it relatively easy, compared to the complexity of the work I did in MGUIDE. Back in the old days – I am working in the new media/education industry for almost 10 years - we used to do information architecture on the fly. I have done it for large companies in Greece and international, on both multimedia productions and web sites. However, back then I was responsible for a web site from its “birth” until its full deployment, which means that I have learn about information architecture, the old school way – through designing, developing and deploying large multimedia projects and web sites. Below you will find a sample of this work.

Web Sites: Companies include:

Elnet Site, Metrolife – Emporiki, Marketing Lead and others

View Full Album

CD/DVD ROM

Companies include: Centric multimedia, Grecotel, Metrolife – Emporiki and others

View Full Album

References

[1] http://iainstitute.org/documents/learn/

Prototype 5

2010-05-29T06:29:00.000-07:00

This prototype is similar to the others, but it:

1) Features only brief descriptions about four locations of the castle of Monemvasia. This content has been carefully crafted to ensure that it is short (but not to short) in terms of length, analytical, and most importantly simple to understand.

2) The Loquendo Kate text-to-speech voice speaks too fast even in low speed mode. To fix this, I have introduced a 1 sec delay between each sentence of the description. That makes the presentation to flow more naturally and the text easier to understand.

3) Features two types of virtual guides:

a) One virtual guide, that uses full gestures and happy facial expressions. This guide always looks at the user.

b) Another virtual guide that uses no gestures and her face appears always serious. This guide looks away from the user most of the times.

The screenshots below demonstrate both guides:

Prototype 2

2010-05-28T08:56:00.000-07:00

Prototype 2 (and prototype 1) were evaluated in Greece. The idea here was to build a system that offers information of variable complexity about the castle and most importantly allows the visitor to navigate freely in the castle.

A number of solutions were considered in order to allow free navigation:

a) Real – time landmark recognition. This feature is shown below (move to slider 10 1.58). The idea here, is to train the system to recognize a particular location (simply by taking a photo) and allow the user to initiate a presentation about the location by pointing the camera of the device to it. That would be extremely neat to have in the current prototype 2 implementation but the technology is probably still very expensive to purchase.

b) QR-CODE technology. This feature is shown below (Greek language only). The idea here, is to tag each location on the castle with a QR-CODE (similar to the product bar code but it can hold textual as well as numerical information) and allow the user to initiate a presentation about a location simply by photographing the QR-CODE of the location. A more advanced version of the technology, allows a similar functionality to the real-time landmark recognition (i.e., real-time video QR-CODE recognition). It would be interesting to come up with a study to compare the performance and usability of both technologies. At the moment apart from cost (QR-CODE is far cheaper) I see no core differences between the two technologies.

Prototype 4 (Technical updates)

2010-05-26T05:15:00.000-07:00

After a series of tests with the Antelope tagger, I found that it still has a lot of bugs. Hence, I decided to use the Stanford parser as a tagger in order to perform shallow parsing to the returned VPF trigger. It seems to be working OK, though its much slower than the Antelope tagger.

I also changed the flat textual database with the an XML one that provides a more structured way to access its contents. For example,

<sentances id="2">
<text>Can we begin the tour please</text>
<predicates2>begin</predicates2>
<predicates2>please</predicates2>
<Deep_Syntax name="Subject">We</Deep_Syntax>
<Deep_Syntax name="Subject">Tour</Deep_Syntax>
</sentances>

A pilot with three participants showed that the search algorithm works effectively. In addition, the following design alternations were suggested:

1) Break down the presentations each user has to listen (and ask questions) into parts. That way question asking can be focused on certain parts and not on the whole of the presentation

2) Create a castle window that will help users visualize the presented information. I used detailed 3D panoramic pictures from each location in the castle, to help them achieve this goal (see screenshot)

3) Include a stop and pause button on the interface. Participants can pause the text to speech synthesis or simply stop it if they find i annoying.

4) Search google and wikipedia using natural language for words and terms that are unknown to the user. This feature was simulated in the current system.

This system will be evaluated with 15 participants at Middlesex University.

Future Work:

1) Extend the current algorithm to include semantics. Such system will match the user’s input with existing triggers in the DB at the semantic level.

2) Replace the XML database with an SQL one. Although the XML DB works fine, it has to load all phrases and parses into RAM. With a static MySQL database I can avoid that.

3) In the current implementation, if the system fails to match an input with a trigger in the DB, it will return “I am not sure if I understand your question please re phrase”. Parsing of the input though, is usually successful. The idea here, is to allow the system to actually learn the questions the previous users asked. For example, if the first user asks “What is a pergola?” and the system doesn’t have an answer, it should be able to dynamically update its DB’s and return an answer by searching the WWW. That way, the more questions users ask the more “intelligent” the system it becomes.

Videos – Prototype 4 (2nd Demo)

2010-05-24T17:26:00.001-07:00

The video below demonstrates the two processing layers I constructed upon the VPF for better language understanding. An early development of the algorithm can be found here. Please note that the few minutes of delay at the beginning of the video are because of the initialization of the tagger.

This system uses shallow parsing and deep syntactic processing to match the user’s input with the database. In particular, the following steps are taken to find the database phrase closest to the input:

Stage 1: Shallow Parsing

Replace contractions

[didn't, 'll, 're, lets, let's, 've, 'm, won't, 'd, 's, n't] with [did not, will, are, let us, let us, have, am, will not, would, is, not]

Remove unnecessary words and POS

[ok, yes, no, hmm, yeah, uh, huh, to, Um, Oh, Alas, Oh, Eh, er, uh, uh huh, um, well]

[Article, Preposition, Conjunction, Determiner, Modal, Interjection, Numeral, Punctuation]

Tag the user’s input with its Part of Speech (POS).
Tag the VPF match of the user’s input with its Part of Speech (POS)
Filter both input and VPF match, based on a list of the global keywords returned by the VPF Web Service.
Compare what is left for POS and values. For example, for my question “Does the castle has any other gates?” only the (gates castle) keywords are returned.
If the comparison is successful allow the output (i.e., a script containing all the synchronized animations, speech, etc) of the VPF service to be executed by the system
If the comparison fails, pass the input to Stage 2 for deep syntactic processing.
Stage 2: Deep Syntactic Processing
Fully Parse the user’s input and extract its predicates and Deep Dependencies (e.g., Subject, DirectObject, etc). For example, my phrase “I would like a brief description about all walls!” fails in the first stage of processing and parses like:

like(Subject: I, DirectObject: description, SpaceComplement: about walls)

If it is a single predicate sentence, conduct 10 similarity tests, to check for similarities between the parsed input and the pre-parsed sentences in the database.
If a match is found, query the VPF with the match.
Return the output (i.e., a script containing all the synchronized animations, speech, etc) and execute it in the system.
If it is a double predicate sentence, conduct 9 similarity tests, to check for similarities between the parsed input and the pre-parsed sentences in the database
If a match is found query the VPF with the match.
Return the output (i.e., a script containing all the synchronized animations, speech, etc) and execute it in the system.
If this stage fails, ask the user to rephrase or to move on to another question.
I also experimented with the semantic parsing, but the Antelope parser is currently experimental. I am planning to add a third stage for full semantic processing in the current algorithm once Antelope’s parser is fully matured.

Videos – Prototype 4 (Original Version)

2010-05-11T14:51:00.001-07:00

This is a demo of the original version of prototype 4. The system was initially constructed with the aim of comparing a natural method of communication with the guide vs. a menu of predefined phrases. This system also uses speech recognition (not shown in this demo) with dynamic grammars. However, because of the time needed to evaluate the system per participant (more than two hours) we decided to drop it and replace it with a simpler system (shown in screenshot 1).

This demo doesn’t use AIML but rather access the VPF service (http://vpf.cise.ufl.edu/VirtualPeopleFactory/) through an API provided by its creator Brent Rossen. VPF uses a similar approach to AIML, but the patterns don’t have to be said verbatim to match. This means that if the system has a trigger (i.e., a question) that is even remotely similar to what you said, it will match it and return it associated speech (i.e., answer). This is one step further to language understanding without any linguistic processing, but of course it has its limitations. I will discuss these limitations in another post.

Screenshot 1: The simpler system that replaced Prototype 4.

Videos – Prototype 1 (Part 2 Greek Only)

2010-05-07T10:16:00.001-07:00

This is the second part of the vide for prototype 1. The system has just completed the presentation for location 1, and asks the user to move on to location 2. Notice how the agent reacts at the beginning of the presentation (she can not see the user through the camera). The reaction is similar to the one you would expect from a real guide, if she saw that someone from the group is not listening to what she is saying.

Videos – Prototype 1 (Part 1 Greek Only)

2010-05-07T09:19:00.001-07:00

This is the first part of the two videos for Prototype 1. The guide’s task is to provide navigation instructions on predetermined routes, as well as personalised information for specific locations in the castle of Monemvasia.

When the system loads the user is given a choice between three information scenarios: Architecture, History and Biographical. The user can also customize the appearance of the agent and other system settings, but this doesn’t show on the video. During a presentation the agent can utilize information from the Face-Detection module, and react if for example, the user is standing too far away from the camera. Finally, notice the use of FSM (Finite State Machine) in the construction of the dialogues. With the proper authoring tool such dialogues are extremely easy to make and can cover a whole range of dialogue phenomena.

Another idea I experimented for a while was the use of emotional responses as a method to guide how a presentation evolves about a location. For example if the user is too bored of the provided information, the agent can try to either provide alternative information or speed up the pace. However, its impossible to implement such approach in the existing Script-based systems. Possibly a KB is needed to dynamically create the contents of each presentation, but how the agent augments the story with non-verbal behaviour is an open question.

Videos – Prototype 3 (Clothes)

2010-05-06T23:51:00.001-07:00

This is the same prototype system (Prototype 3) but with a clothed avatar. The clothing is completely dynamic and very demanding in hardware resources. It won't even open on the UMPC. The run the clothe animations smoothly (along with the avatar ones) you need a quad-core system. In my opinion the avatar looks more realistic with the clothes on than without. The clothes are available free-of-charge at http://virtual-guide-systems.blogspot.com/2009/09/current-developments-haptek-clothing.html

Videos – Prototype 3

2010-05-06T05:52:00.001-07:00

Below is a link to a video from the third prototype of the MGUIDE project. The guide’s task is to provide navigation instructions based on photographs of landmarks shown in her background. The only draw back of the video is the lost of Simon's lip-syncing and smooth animations due to the limitations of the screen capture software, that had trouble to handle everything at the same time. I will use a video camcorder to show a user in action once I will begin the evaluation of the system.

On the UMPC device I get 35 FPS, which is impressive if you think about the hardware limitations. The only way to get the FPS that high, is to have the face-detection module of the system on at all times. Although this should have had the opposite effect (i.e., to decrease the fps) for a very strange reason it boosts the engine’s FPS to maximum. Without the face-detection I am getting 4-10 FPS. I am not sure why it happens but I am glad it does.

MGUIDE Components

2010-05-03T22:08:00.001-07:00

Below are a number of components I used in the development phase of MGUIDE. I am giving them away for free, all you have to do is to email me at virtual.guide.systems at googlemail.com.

1) AIML Control (.dll): This control can be added into any authoring environment (e.g., Flash, Director, etc) as long as you have the latest .NET framework installed. The control can handle Unicode characters and has full Javascript support.

2) WebCam Capture (.dll). A simple control that allows to add web cam support into any project.

3) Face Detection (.dll). A control that allows you to add face detection into any project. It uses the Intel’s OpenCV library for face detection and a .NET wrapper from Mr.Oshikiri. The control can detect right, left, far_away, normal and close. It currently uses OpenCV 1.0, and if you want to upgrade it to the latest OpenCV 2.0 you will need the latest .NET wrapper from here

4) QRCode (.dll). The control allows you to add QR-CODE recognition into any project. It uses two commercial components from a company called PartiTek.

a) PtImageRW.dll

b) PtQRDecode.dll

You will need to purchase these components if you want to make the control to work.

5) Subtitles (.dll). A control that allows you to add speech subtitles to any project. The control uses a number of commercial components from Chant.

a) Chant.Shared.dll

b) Chant.SpeechKit.dll

c) DNSpeechKit.dll

d) NSpeechKitLib.dll

and an XML file as a subtitles feed. You will need to purchase Chant SpeechKit if you want to make this component to work.

Finally don't forget my free offering on Haptek Clothing at: http://virtual-guide-systems.blogspot.com/2009/09/current-developments-haptek-clothing.html

MGUIDE Project Investors and Supporters

2010-05-03T00:45:00.001-07:00

I wish to thank the following for supporting my work:

Individuals:

1) Ms Maria Tsouros - Graphic Design

2) Dr Katerina Theodoridou

3) Ms Fotini Singel, Proof-Reading of Greek texts.

4) Ms Arxontoula Tsakou – Recording of Video Clips from the Castle of Monemvasia

Universities:

1) Virtual Experiences Group, University of Florida (Mr Brent Rossen)

2) Mr Mark Chavez, Nanyang Technological University

3) Middlesex University, School of Engineering and Information Sciences.

Companies:

1) Haptek (3D Avatar Engine)

2) Chant ( Interfaces for Speech recognition and Text to Speech generation)

3) Loquendo (Text to Speech Engine)

4) Proxeme (Natural Language Processing Platform)

5) Panoramic Applications

6) PartiTek (QR-Code Recognition and generation)

Final System Configuration

2010-01-01T03:40:00.000-08:00

Developing and running virtual human applications on mobile devices is a challenging task. Mobile phones because of the extremely limited hardware resources are out of the question (unless of course you like these bad-looking flash-based avatars). UMPC and notebooks are the best choice for such systems. After reviewing several models I ended up with the Samsung Q1 Ultra Premium (1.33 Intel Core Solo Processor):

As the device is not available in Europe, I had to buy it from Ebay and import it in the UK (It cost me close to 900 USD). I also thought to upgrade the device to 2GB Ram but after watching this video I realised that I would better ask an engineer to do it. I only added a 360 degrees web cam (for face detection and QR-CODE reading) as the integrated ones are very low resolution. I installed the systems successfully and I am ready to move on to the final stage of my work – The evaluation Stage of the prototypes. The first two prototypes (Prototype 1 and Prototype 2) will be evaluated in my home country Greece.

Immortal Avatars - A very unique use of avatar technology

2009-12-14T03:38:00.000-08:00

Sometimes I wonder if there will be any limits in the human imagination. Engineers at the University of Illinoy Chicago, are trying to immortalize human beings through the use of avatar technology. The realization of such attempt is the Alex the avatar(show above). Alex is the digital double of Alex Pothen the former director of the National Science foundation. All I can say is WOW!!! Just imagine in 10-20 years from now to be able to transfer yourself into an AI brain and to live forever as a digital entity. Would you do it if you had the opportunity? Only the idea creeps me out but future generations may consider it an acceptable practice. The whole concept reminds me of the movie the Lawnmower man where the main character becomes so intelligent through the use of Virtual Reality technology that manage to transfer himself into the WWW.

PopSImmci's Future Of: Imm

http://science.discovery.com/videos/popscis-future-of-immortal-avatars.html

Avatar Movie

2009-12-14T03:32:00.000-08:00

I am truly shocked and amazed. Avatar is the BEST movie I have ever seen. The Na'vi characters are simply breath-taking and way out of the uncanny valley(see below for a definition). That's an important aspect of every system that integrates Virtual human technology. The more life-like a character is, the more believable it becomes and hence, it is easier for peopel to accept it in its assigned role (guide, actor, etc). Avatar is the most tangible evidence for that. So what can we (the poor Virtual human developers) do? How can we develop systems with Virtual Humans that people will accept? If you have access a computer with 80000 processors (that's what James Cameron used for the creation of avatar), you are in HEAVEN. If not, then what? No matter how beautiful and life-like your creation is, it will only go as far as the real-time engine can take it. I have reviewed several real-time engines in the past-years and nothing comes close to Avatar. Second Life has become popular despite its crapy graphics. But has anybody ever used Second Life for a serious application? There have been a number of research attempts for training people, but there is nothing serious for the common user (other than silly things to "kill" his/er time).

Def: The uncanny valley refers to the negative effect that is created when something approaches human (in appearance) but isn't quite there; it creates this creepiness. You can find an explanation of the term from James Cameron himself here: http://news.discovery.com/videos/tech-avatar-motion-capture-mirrors-emotions.html

Industry Survey

2009-12-01T03:25:00.000-08:00

Below is a list of Character Engines I compiled a long time ago:

Open-source

1) The EMBR project (http://embots.dfki.de/EMBR/)

2) Ogre3D- www.ogre3d.com. An open Source real-time 3D engine but without any kind of web player for content delivery.

Real-Time engines

1) Haptek - www.haptek.com The full suite of tools is the range of 7,0000-10,000 USD

2) Charamel - http://www.charamel.de/. A German avatar company, with very realistic characters but also very expensive tools. A very interesting product of this company, is the Charavirld that allows actual actors to controls Virtual characters in real-time. They ask 10,000 Euros for their main development platform along with the player. However, for research purposes they offer a 6 month demo license for only 300 Euros

3) QEDSOFT - http://www.qedsoft.com. A french company with a very interesting real-time 3D engine. Although I am not sure, their tools must be very expensive.

Real-time but without a real-time web player

1) DA Group - http://www.digital-animations.com/. The creators of Ananova. The company offers a range of real-time content creation tools but there is no player for real-time delivery.

2) Codebaby - http://www.codebaby.com/. A canadian company offering 3D characters through flash.

3) Cantoche - http://www.cantoche.com/. A french company with cartoon-like avatars again relying onto flash for content delivery.

4) VCom - http://www.vcom3d.com. An amazing technology for 3D character creations again using flash for content delivery

5) IMS Interactive - http://www.ims3d.com/ The company uses shockwave 3D for content delivery

6) Visage - http://www.visagetechnologies.com. A Swidish company with an interesting technology but without a web player for content delivery

7) Virtuoz- http://www.virtuoz.com. Another French company

8)SimGraphics - http://www.simg.com. A California company using a Wizard-Of-Oz aporoach for brining 3D characters to life.

Non real-time engines (Psedo 3D)

1) Guile 3D - www.guile3d.com. Recently the company created Denish a 3D photorealistic character with amazing visuals. There is no information for any kind of Web player for content delivery

2) Media Semantics - http://www.mediasemantics.com/. A very interesting and cheap technology using XML for character control and flash for content delivery

3) Gizmoz - http://www.gizmoz.com/. A photorealistic technology using flash for content delivery

4) http://www.karigirl.com. A virtual girlfriend

Face animation only

1) Crazytalk - http://www.reallusion.com/crazytalk/. A face-only creating tool using mainly flash for content delivery

2) http://www.lifemi.com/. Another face-only company using flash for web-content delivery

3) FaceFX-http://www.oc3ent.com/. A face-only tool mainly used for game development

Avatar communities

1) SecondLife http://secondlife.com/ Everyone knows secondlife

2) The Bluemars project (http://www.bluemarsonline.com/). In terms of graphics blue mars is far far better than SL. Download the installer (be carefull is massive 1,01 GB) and try it. The 3D worlds are simply AMAZING.... (that's me avatar in BM by the way)

3) http://www.imvu.com/. Very similar to Second Life with far more amazing graphics

4)Google Lively - http://www.lively.com. A rather unsuccesfull attempt of google to mimic secondlife

5) Soon Startrek online - http://www.startrekonline.com/

6) Entropia Universe. Mainly for Sci-Fi funs http://www.entropiauniverse.com/index.var

7) ActiveWorlds - http://www.activeworlds.com/

8) Kaneva - https://www.kaneva.com

9) The new Amsterdam - https://www.kaneva.com

Character Languages

Several attempts have been made to standarize avatar creation. Currently there are two major trends, one in Japan and the other in the Western World

1) Behaviour Markup language - http://wiki.mindmakers.org/projects:bml:main

2) Multimodal Presentation Markup Language 3D (MPML3D) -

http://research.nii.ac.jp/~prendinger/MPML3D/MPML3D.html

Haptek Characters

2009-09-30T22:56:00.000-07:00

Haptek is the best example of how overpriced products can fail your business. Their real-time 3D engine, although originally designed for Windows 95 (can you believe that? 14 years ago) and its still the best in the market. I heavily modified their fullBod character for the needs of my project, and now comes with:

1) 2000 animations that covers almost every project needs.

2) Dynamic clothes pants, top and shoes. Character clothes are separate models animated to flow naturally with every character movements (even walking). Clothes have more than 2000 animations to achieve that. Animations and clothe took more than a year to complete.

3) Several textures for both top and pants.

4) A script parser that allows automatic creation and cleaning of Haptek scripts.

The Talos script parser as I call it, features the following:

a) Automatic creation of script ready to be executed by the Haptek engine. For example, Talos will convert the following text into a script that accurately animates the character, top, pants and shoes in sync with the text:

(open_tag) BOOKMARK mark=‘anim,suggest’ (close_tag) Can I suggest you to have a look on wall (open_tag)BOOKMARK mark=‘anim,show_sea’(close_tag) that goes down the bold cliff and towards the sea?

Note: Open_tag and close_tag stand for the standard opening and closing tags. For some reason I can not display them here, correctly.b) Accurate extraction of tags from already tagged texts. These tags (shown above in red) can easily be used in Talos to tag texts in any language.

c) Single/batch conversion of any text to .ogg format (the sound format used by the Haptek engine).

d) Ability to use SAPI 5.0 compatible tags in your texts. Usage of Loquendo compatible tags is also possible. These tags add realistic effects (e.g., sneezing, laughter) to Loquendo TTS voices (e.g., (http://tts.loquendo.com/ttsdemo/default.asp?page=id&language=en) but work only with the Loquendo voices.

e) Limited automatic tagging of texts. Talos script parser allow you to automatically and accurately tag a given text. Currently works only with a limited set of texts, but I am working hard to extend it.

If you want to use the character in your projects please email me at: virtual.guide.systems at googlemail.com. An example of a script created by the script parser:

#Haptek Version= 2.00 Name=LocC_HB.hap HapType= script FileType= text

## world It

##prereq= none

\clock [t= 0.0] \load [file= [sounds\LocC_HB.ogg]]

\clock [t= 0] \SetSwitch [figure= fullBod switch= during state= start]

\clock [t= 3] \SetSwitch [figure= fullBod switch= propably state= start]

\clock [t= 5.1] \SetSwitch [figure= fullBod switch= used state= start]

\clock [t= 9] \SetSwitch [figure= fullBod switch= when state= start]

\clock [t= 11] \SetSwitch [figure= fullBod switch= retake state= start]

\clock [t= 13.1] \SetSwitch [figure= fullBod switch= converted state= start]

\clock [t= 18] \SetSwitch [figure= fullBod switch= retake state= start]

\clock [t= 21] \SetSwitch [figure= fullBod switch= and_the state= start]

\clock [t= 23] \SetSwitch [figure= fullBod switch= become_again state= start]

\clock [t= 26.4] \SetSwitch [figure= fullBod switch= after state= start]

\clock [t= 31] \SetSwitch [figure= fullBod switch= while state= start]

\clock [t= 35] \SetSwitch [figure= fullBod switch= today state= start]

\clock [t= 38.75] \SetSwitch [figure= fullBod switch= from_the state= start]

\clock [t= 40.85] \SetSwitch [figure= fullBod switch= name state= start]

\clock [t= 42.7] \SetSwitch [figure= fullBod switch= contradictory2 state= start]

\clock [t= 44.6] \SetSwitch [figure= fullBod switch= exist_near state= start]

\clock [t= 48] \SetSwitch [figure= fullBod switch= large2 state= start]

\clock [t= 51] \SetSwitch [figure= fullBod switch= find_series state= start]

\clock [t= 52.8] \SetSwitch [figure= fullBod switch= important state= start]

\clock [t= 57] \SetSwitch [figure= fullBod switch= whole state= start]

\clock [t= 62] \SetSwitch [figure= blackboard switch= enter state= on]

\clock [t= 62] \settexture [figure= blackboard tex= museum1.jpg]

\clock [t= 63.9] \SetSwitch [figure= fullBod switch= museum1 state= start]

\clock [t= 66.1] \settexture [ figure= blackboard tex= museum2.jpg]

\clock [t= 66.1] \SetSwitch [figure= fullBod switch= museum2 state= start]

\clock [t= 68.3] \SetSwitch [figure= fullBod switch= as_well state= start]

\clock [t= 78] \SetSwitch [figure= blackboard switch= enter state= close]

\clock [t= 80] \SetSwitch [figure= fullBod switch= plateia_9 state= start]

\clock [t= 87] \SetSwitch [figure= fullBod switch= contradictory2 state= start]

\clock [t= 91] \SetSwitch [figure= fullBod switch= propably state= start]

\clock [t= 98] \SetSwitch [figure= fullBod switch= named state= start]

\clock [t= 101] \SetSwitch [figure= fullBod switch= no_help state= start]

\clock [t= 105] \SetSwitch [figure= fullBod switch= plateia_10 state= start]

\clock [t= 109] \SetSwitch [figure= fullBod switch= which state= start]

\clock [t= 114] \SetSwitch [figure= fullBod switch= which state= start]

\clock [t= 118] \SetSwitch [figure= fullBod switch= point state= start]

\clock [t= 124] \SetSwitch [figure= fullBod switch= of_course_left state= start]

\clock [t= 132] \SetSwitch [figure= fullBod switch= if_else state= start]

\counter [name= [LocC_HBcounter] i0=0 f0= 1]

Current Developments - Computer Vision

2009-09-30T03:24:00.000-07:00

OpenCV is an open source library for robotic vision from Intel. Using its head-tracking functionality, I was able to construct a simple algorithm that detects, the location of the user's head in the 3D environment (right,left,far-away, close, etc). By using this information, you can then implement some really fantastic scenarios in any programming environment. For example in my prototypes I have the following scenarios:

Face recognition is also possible with OpenCV but it doesn't offer much more than face detection. A good idea would be to integrate both into a single component. Ideally such a component would lock onto a face, and provide its coordinates to the application. Finally somehow such a component should also integrate emotion recognition abilities, an important attribute for Virtual Humans.

A VH looking at the direction of my face:

Natural Language Processing

2009-09-30T03:22:00.000-07:00

VPF is a web service where "you can create virtual people for a variety of uses. Currently the most common use of the Virtual People Factory is to create Virtual Patients for Medical and Pharmacy education.".

Other free VH hosting services are the following:

a) Personality Forge Engine

b) PandoraBots

Although I am sure you can cite more advanced system that these, compared to just mere claims, these three systems are: a) publicly available, b) fully-functional c) free-to-use. Although all three systems, do not actually process language, VPF is by far the most effective of all. As VPF people were kind enough to provide me with their script-matching algorithm and a fully functional API I was able to address this limitation.

VPF currently relies on a score-matching algorithm that determines the likelihood a user's input to match with a trigger (i.e., a phrase input in the system by the content developer). In my approach simple keyword matching is the last stage of processing, if all other stages fail. In particular the algorithm in:

Stage 1.

Compares the user's input and the trigger returned by the VPF for common tokens and POS. If comparison is successful (all tokens match), returns the answer associated with the VPF trigger. If comparison fails (keywords are not equal) it moves on to the next stage.

Stage2.

Conducts a series of predicate tests on the input against a DB of predefined phrases. If comparison is successful it passes the phrases to VPF for 100% matching. If comparison fails, it moves on to the final stage where the system reverts back to the simple VPF keyword matching

Stage 3.

VPF keyword Matching

Semantic processing and comparison was also implemented as an additional stage in the algorithm but as it has some problems that need to be addressed first, it was decided not be used in the final prototype. The plan is to integrate the VPF script matching algorithm to the existing one , and create a four-stage approach (with Semantic Processing included) that will enable the NLU component of Talos authoring tool to full process the user's input before matching it with a trigger.

Another fully developed idea for Talos is the creation of a dialogue manager based on HTN (Hierarchical Task Networks) but as it is currently only on paper, I would prefer not to discuss it any further.

Code for Stage 1

Sub Syntactic_Keyword_Processing(ByVal userinput As String)
 'we need to load the tagger first
 Try
 tagger_counter += 1
 If tagger_counter = 1 Then
 load_tagger()
 Else
 'tagger is loaded don't loaded again
 End If
 'remove panctuation and contractions first
 Dim contractions As List(Of String) = New List(Of String)(New String() _
 {"didn't", "'ll", "'re", "lets", "let's", "'ve", "'m", "won't", "'d", "'s", "n't"})
 Dim word_contractions As List(Of String) = New List(Of String)(New String() _
 {"did not", "will", "are", "let us", "let us", "have", "am", "will not", "would", "is", "not"})
 Dim end_line As String = "[\,\?\!\.]|\s+$"
 Dim start_line As String = "[\,\?\!]|^\s+"
 Dim userinput2 As String = Regex.Replace(userinput, end_line, "")
 Dim userinput3 As String = Regex.Replace(userinput2, start_line, "")
 'remove contractions
 For Each item As String In contractions
 If userinput.Contains(item) Then
 Dim cont_position As Integer = contractions.IndexOf(item)
 Dim what_word As String = word_contractions.Item(cont_position)
 userinput3 = Regex.Replace(userinput3, item, Space(1) & what_word)
 Exit For
 End If
 Next
 Dim ask_step As String = "step_1"
 Select Case ask_step
 Case "step_1"
 hr.addUserInput(userinput)
 Dim response As String = hr.findResponses(Current_Script)
 Dim index As Integer = hr.findMostRelevantResponse()
 VPF_trigger = hr.getResponseMatchedSentence(index)
 If VPF_trigger <> "" Then
 'Perform syntactic comparison between the input and the trigger
 syntactic_keyword_comparison(userinput3)
 If comparison = "Sucessful" Then
 _answer = hr.getResponseMatchedSpeech(index)
 input_list.Clear()
 trigger_list.Clear()
 Else
 ask_step = "step_2"
 End If
 End If
 End Select
 Select Case ask_step
 Case "step_2"
 If new_question <> "" Then
 hr.addUserInput(new_question)
 Dim response As String = hr.findResponses(Current_Script)
 Dim index As Integer = hr.findMostRelevantResponse()
 _answer = hr.getResponseMatchedSpeech(index)
 input_list.Clear()
 trigger_list.Clear()
 End If
 End Select
 Catch ex As Exception
 output.Clear()
 output.Text += ex.Message + Environment.NewLine
 End Try
 End Sub

Current Developments - Talos Authoring tool

2009-09-29T03:19:00.000-07:00

Talos: A Virtual Human Authoring tool

Talos is an authoring environment designed to enable content developers to create Virtual Human systems for the domain of mobile guidance. Although the system has been designed with the paper guide book creators in mind, it can also be used in other application domains as well.The final design of Talos is very complex and includes several different modules. I think that it will take a programming team of 5 people 1 year of intense work to complete.

Talos come as an idea when I realised that there are virtually no tools in the market that would enable me to rapidly create the prototypes I needed for my research work. From the final design of Talos, I was only able to implement some of its ideas, those that needed to implement my final prototype systems.

1 ) A script parser to automatically create Haptek scripts. A screen-shot of the parser is show below:

The tool is described in more detail in the topic "Current Developments - Haptek characters". The script parser of the full Talos environment follows a similar approach, but in a fully automated and real-time fashion. Talos will accept pure text as input, and generate full character performances as an output. Of course the problem is this approach, is how to make the character pointing correctly to objects in its background. This requires the character to have knowledge of its environment. My design in this area is incomplete. A possible solution, would be to divide the background of the character into segments and associate each segment with a keyword. I need to do more research on this area.
2) A simple UI (User Interface tool) that enabled me to create without too much effort, the AIML KBs needed for my final prototype systems. In more detail the UI includes:
a) An AIML\XML KB creator.
b) An AIML creator for existing question-sets (e.g., questions that you may already have in XML format)
c) A translator for both AIML/XML KBs
d) A scene character modifier where you can modify various scene and character settings (e.g., probs, backgrounds, etc)
e) A GPS integration module, where you can assign Long and Lat coordinates to various scripts and tests them in real-time
f) A speech recognition module where you can load a grammar and test it
g) A Cyc-creator module. This module was supposed to automate the creation of CYC queries suitable for insertion into AIML scripts. However, the API of ResearchCyc is extremely tough to crack (perhaps because it is pure Java) and I ended up with just a simulation.
From the above I only used "a","b",and "c" in the creation of my final prototypes. The rest of the features were not of any value to the actual development, but they lead to several improvement in the final Talos design. The UI tool is available for free. If you want to experiment with Virtual Humans, AIML and XML KB's, making location-sensitive scripts for your characters that's the best way to start.

A screenshot of the UI tool

Current Developments - Haptek Clothing

2009-09-29T03:17:00.000-07:00

After spending several months building these clothes I realised that they are too complex to be handled by the limited hardware of the UMPC device. Hence, I decided to give them for free. The clothes come with several textures, animations, morphs, etc. The clothes are designed for the character(called the guide) I use in my systems. I will possibly release the character once I am done with the evaluation stage of my project.

You can find them here:

http://uploading.com/files/45e3e6fm/clothes.zip/

Current Developments - MGUIDE Prototypes

2009-09-28T03:04:00.000-07:00

Prototype 1:

1. A 3D Agent with more than 2000 gestures and several face expressions. The agent uses this body and facial language to augment the location presentation and navigation instructions provided.

2. A 3D agent that is aware of its environment and can dynamically evoke the attention of the user during a presentation for a location. For example if the user is poking around for a certain period of time, it can request the attention of the user to the presentation.

3. A 3D agent with fully dynamic 3D clothing with changeable textures during system configuration from the user.

4. A 3D agent capable of using additional multimedia information (on a 3D board) to enhance further the transmitted information (mainly in the presentation mode).

5. A Finite State Machine (FSM) dialogue manager capable of dynamically displaying questions based on the user’s selection and the current context. The questions cover a very broad range of the possible questions/clarifications that a user can ask after a presentation for a location.

6. 12 information scenarios based on what the castle has to offer to the potential visitor both culturally and historically. The total content (presentations and questions) is more than 10 hours long.

7. Designed but not implemented, customization of the agent voice.

Screenshot 1: The animated agent points to an image on the 3D board

Screenshot 2: The animated agent gestures as she speaks

Prototype 2:

Similar to the first system but with one additional feature - QR Code based navigation. A QR-Code is a bar-code capable of storing up to 4,296 characters in a simple geometrical shape. The system uses a QR-Code recognition algorithm to recognize the locations that the user is currently in (the user must photograph the QR-Code in order for the system to process it).

Prototype 3:

Similar to the other systems but it focuses only on the provision of navigation instructions. At the moment the system uses only photographs of landmarks but other more automated methods (e.g., GPS positing) have also been considered.

Prototype 4:

It features one information scenario only, along with the characteristics of the first prototype and:

2) Dynamic changing voice recognition grammars that allows the user to interact with the system using only h/er voice.

3) Natural language processing abilities using the Stanford/Link Parser. The system utilizes a novel algorithm that allows it to conduct predicate analysis and score keyword matching (if the first stage fails).The second stage of analysis is conducted by a secondary web system (Virtual People Factory).

4) A highly experimental search/comparison algorithm utilizing Semantic interpretation of the user’s input.

Screenshot 3: The system preferences of prototype 3 (Natural Language Processing version)

Development of Virtual Humans - Early Years

2009-09-27T20:54:00.000-07:00

Domain: Mobile Guides

After the first 1 and 1/2 years of my M.Phil work, I finally found heaven. The area of mobile guides is a relatively simple domain but with much to offer. Tourism in several countries (e.g., Greece) is considered a vital part of the economy. In 2005 I begun working on a system that a) provides navigation instructions in a specific area (i.e., a medieval castle in Greece) b) provides information on selected locations of the castle (e.g., historical information). The system was authored in Macromedia Director MX 2004 multimedia environment and used:

a) The AIMLPad ActiveX control

b) The Maddy character from DAGroupPlc.

c) A location-sensing algorithm that triggered specific scripts at specific coordinates in the castle.

d) A speaker independent ASR engine provided by Babel technologies ().

The system accepted input through a menu of static English phrases and buttons. It generated no output apart from the minor movement of the character's body on the screen. After field testing the system, I realised that a) the GPS component because the rocky environment of the castle was not working. b) The ASR engine could not distinguish the user's voice from the noise in the environment. These two components were removed and a formal evaluation took place in November 2005. The goal of the evaluation was to investigate how such an agent would affect the accessibility and usability of a mobile guide system.

In parallel with the developments of the prototype I also had the opportunity to investigate Research-Cyc the most advanced Knowledge-Base on the planet. In parallel with the developments of the prototype I also had the opportunity to mess around with Research-Cyc, the most advanced Knowledge-Base on the planet. The design of the system is shown below.

The idea was to built a KB that would contain all the castle knowledge and integrated with the prototype system.Back then, this was an ideal solution for me, as with a simple fact in KB (Yannis Ritsos is a poet) the system could answer questions like:

“Who is Yannis Ritsos”

“What is the profession of Yannis Ritsos” (poet is a profession)

“Name a poet of Greece” (Greece is a nation and Greeks its inhabitants)

I used the CynD AIML Interpreter. The system uses specially modified AIML to access the KB NL facilites, query the KB and then return the results. For example:

Given the below AIML code and the sentence:

WHAT COUNTRIES BORDER *

(#$isa #$Country )

Consulting cyc returns: What countries border ?

The system calls the ResearchCyc parser (through the custom cycquestion tag), translate the input into its CycL (the language in which the KB is written) representations:

((#$and

(#$isa ?WHAT-ONE-1 #$Country)

(#$politiesBorderEachOther #$Greece ?WHAT-ONE-1))

(#$and

(#$bordersOn #$Greece ?WHAT-ONE-1)

(#$isa ?WHAT-ONE-1 #$Country))

(#$and

(#$isa ?COUNTRY #$IndependentCountry)

(#$politiesBorderEachOther #$Greece ?COUNTRY))

(#$and

(#$bordersOn #$Greece ?COUNTRY)

(#$isa ?COUNTRY #$IndependentCountry)))

And then iterate through the list of parses, and try to query the KB to get a possible answer. In the particular example the system returns:

Consulting Cyc returns:

0 (((?COUNTRY . Bulgaria))((?COUNTRY . Macedonia))((?COUNTRY . Albania))((?COUNTRY . Turkey))).

1. (((?COUNTRY . Bulgaria))((?COUNTRY . Macedonia))((?COUNTRY . Albania))((?COUNTRY . Turkey)))

2. (((?WHAT-ONE-1 . Bulgaria))((?WHAT-ONE-1 . Macedonia)) ((?WHAT-ONE-1 . Albania))((?WHAT-ONE-1 . Turkey)))

3.(((?WHAT-ONE-1 . Bulgaria))((?WHAT-ONE-1 . Macedonia))((?WHAT-ONE-1 . Albania))((?WHAT-ONE-1. Turkey)))

Finally it calls the NL generator of the KB and translate the results back to English:

Consulting Cyc returns: 0 Bulgaria Macedonia Albania Turkey. 1 Bulgaria Macedonia Albania Turkey. 2 . Bulgaria Macedonia Albania Turkey 3. Bulgaria Macedonia Albania Turkey

Problems:

1) CynD will try to query the KB for every single interpretation of the user's question. In the above example, the KB returned the same answer for all parser interpretations. Some kind of tool was needed that would automatically generate AIML categories incorporating all possible parser interpretations, allow the system's author to select the one that produced the answer he expected and make the system's answer more natural. There was no time for that.

2) In an open Q-A dialogue with a system in my domain (mobile guides), the user could ask questions about random objects that he may encounter in his path-something like "What is that?" for a church or a particular building. The disambiguation of "that" to the actual object name and location in the physical environment is an extremely difficult problem -- one of the many that NLP has to face in the future. At the moment, there is no location technology that can allow a computer to distinguish between two objects located at a close distance in a psychical environment.

3) My work focuses on animated agents. If the dialogue with the system is unknown... how could I possibly generate the proper animations for each output of the CyND engine?

5) Hardware resources. Such a system demands massive hardware resources... resources in a M.Phil project are always scarce.

Lessons learned:

1) The pilot evaluation suggested a) several improvements in the experimental design b) several design improvements of the prototype.

2) The need to produce more user-insights. This could only be accomplished, by developing more prototypes varying different attributes of virtual humans (e.g., competence, modality of communication, etc.).

3) The need for technological differentiation. Its impossible to continue the simplistic approach of this prototype, as the research will lead me nowhere. The domain of VH is extremely comparative area with groups of researchers that produce massive contributions to both technology and knowledge every single month.

4) My rather bumpy experience in developing the prototype also suggested the need for an authoring tool that would enable non-programmers to rapidly develop complex Virtual Human systems.

Development of Virtual Humans - Early Years

2009-09-26T19:38:00.000-07:00

The idea, was to develop and evaluate a conversational sales assistant that would assist users in all stages of the CBB (Consumer Buying Behaviour Model) - a model used to describe the entire range of consumer needs in on-line commerce environments. The algorithm alone, would have been enough for a M.Phil contribution. Hence, with great enthusiasm, I started my work at Middlesex. The first task in my research list, was the development of a fully-working prototype. I designed a rather complex system (shown below along with an explanation) and begun looking for proper authoring tools.

A discourse manager module can not be evaluated if it is not part of a general dialogue system . The organization of system needed to test this module is shown below: Starting at the top of the figure, the user communicates with the system textually through a standard desktop PC. The input is parsed to a sequence of speech acts based on the syntactic and semantic form of the utterance and sent to the discourse manager. The discourse manager sets up its initial conversation state and passes the sequence to the context for identification of any lexical information (e.g. names, features etc), and then hands the acts to the reference resolution. The component has two duties. First, it assigns the correct referent to anaphoric referring expressions (e.g. the pronoun her in the sentence Anne asked Edward to pass her the salt refers to Anne). Then, if it is necessary it does illocutionary remapping of the speech acts assigned by the parser as needed to fit discourse and reference cues. For instance, an utterance that consist of a REJECT “no” followed by a REQUEST “go via bank” will have the letter REQUEST remapped into the REJECT; it is essentially the content of the REJECT, not a separate REQUEST. After this processing, reference returns the speech act(s), now in an internal format, to the discourse manager for further disposition. These speech acts however, are only the surface acts (i.e., literal acts) of the utterance. Oftentimes, a surface speech act has a deeper intended or indirect meaning (i.e. an illocutionary act). Take, for example, the utterance, “Can you pass the salt?” which on the surface looks like a QUERY speech act (i.e., asking about your ability to pass the salt). However, the indirect speech act is most likely a REQUEST (i.e. a request to pass me the salt). The discourse manager detects and interprets indirect speech acts through an intention recognition process done in conjunction with the Task Manager. More specifically, it sends each of the postulated acts to the Task Manager and asks whether this interpretation would ‘make sense’ given the current domain and planning context. Each of these postulates are then scored by the Task Manager which allows the Discourse Manager (DM) to find the correct interpretation of the surface speech act (i.e., the intended problem solving act). In addition, with its indirect speech acts duties, the discourse manager must convert the speech act semantics from a linguistic knowledge representation (produced by the parser) to a domain specific, problem-solving knowledge representation (used by the Task Manager for intention recognition). For example, the linguistic semantics of the utterance “Buy the HP Palm from PC-World” might be:

(buy1
:agent SYSTEM
: theme HPPalm
: goal PC-World
The domain specific, problem-solving knowledge representation would correspond to some action in the domain (purchase) with some corresponding parameters. For example:
(purchase
:product HPPalm
:from-merchant PC-World
:price ?p)

The ?p is a variable, since the price was not explicitly mentioned in the utterance.

The output of the task manager and the interface agent is a set of surface speech acts that the system wants to communicate to the user. The generation module achieves this communication using the graphical user display. This module also uses prioritized rules to match requested speech acts to appropriate means of expressing them. For example a request to the generator to inform user about a specific product will result in telling the display to show the image of the product, and simultaneously provide a textual description. The above architecture is by no means complete. We suspect that several modifications to the existing components and addition of new ones can be made during the course of development. However, it serves as a good example of the great complexity of the project.

I spent several months search the WWW. I have a massive URL library from companies that probably no longer exist. An example is: http://www.agentscape.de/, that asked me 15,000 USD for their authoring tools. To that, add, a) a University refused to spent any more money on me (it was already paying me a scholarship) b) a supervisor that kept insisting to follow that path, and you end-up with the perfect recipe for career destruction. Under the massive pressure, instead of quitting, I decided to move to a less complex domain and area of research. It was clear, that this project was better suited for a team of researchers with a perhaps unlimited budget. When I think about it today, it is still unknown, why my supervisor failed to see something so obvious!!!!

Development of Virtual Humans - Early Years

2009-09-25T18:19:00.000-07:00

I think, its a good idea to start this post, with a simple, non-technical definition for a virtual human. A virtual human is an intelligent system capable of providing and accepting information through a full-range of human modalities (e.g., speech,gesture, face expressions, etc).

Constructing an intelligent VH with a general intelligence is a scary idea that should never be pursued. Instead I am more fun of the the robot-slaves idea, i.e., virtual humans (in physical or holographic form) that ONLY look intelligent in a specific domain (e.g., tour-guides, sales, etc)

I initially started working with VH back in 2002, during my Master's degree. The demo I constructed, the e-briefing room is shown below, along with a brief description of its functionality:

The “e-Briefing Room” service provides a tool for effectively educate customers with complex and high-end products on the Web. This vehicle allow customers to fully interact with three-dimensional (3D) models of products online, and also access personalized services on demand, which will be provided by a three-dimensional (3D) talking virtual sales-assistant.Through this interactive technology, the e-briefing room makes it easy for customers to access information on demand. Start your experience by directly selecting a product category of your choice, or activate Derek to listen a detail presentation about the functionalities of the service.

The demo was extremely simple (but... I did it only for my Master's degree). Its only difference from a static web-page, was the 3D head (from an Australian company called famous3D). Back in those days I wasn't even aware of the existence of ALICE But once you got the bug of VH technology you can not easily stop. The simple "e-briefing room" was enough to start me wondering what could be next in the exciting world of Virtual Humans.

In 2003 I was given the opportunity to study for an M.Phil degree at Middlesex University in London. My decision was to continue working on Virtual Humans for electronic commerce.