Development of Virtual Humans - Early Years

Domain: Mobile Guides

After the first 1 and 1/2 years of my M.Phil work, I finally found heaven. The area of mobile guides is a relatively simple domain but with much to offer. Tourism in several countries (e.g., Greece) is considered a vital part of the economy. In 2005 I begun working on a system that a) provides navigation instructions in a specific area (i.e., a medieval castle in Greece) b) provides information on selected locations of the castle (e.g., historical information). The system was authored in Macromedia Director MX 2004 multimedia environment and used:

a) The AIMLPad ActiveX control

b) The Maddy character from DAGroupPlc.

c) A location-sensing algorithm that triggered specific scripts at specific coordinates in the castle.

d) A speaker independent ASR engine provided by Babel technologies ().

The system accepted input through a menu of static English phrases and buttons. It generated no output apart from the minor movement of the character's body on the screen. After field testing the system, I realised that a) the GPS component because the rocky environment of the castle was not working. b) The ASR engine could not distinguish the user's voice from the noise in the environment. These two components were removed and a formal evaluation took place in November 2005. The goal of the evaluation was to investigate how such an agent would affect the accessibility and usability of a mobile guide system.

 

In parallel with the developments of the prototype I also had the opportunity to investigate Research-Cyc the most advanced Knowledge-Base on the planet. In parallel with the developments of the prototype I also had the opportunity to mess around with Research-Cyc, the most advanced Knowledge-Base on the planet. The design of the system is shown below.

 

 

The idea was to built a KB that would contain all the castle knowledge and integrated with the prototype system.Back then, this was an ideal solution for me, as with a simple fact in KB (Yannis Ritsos is a poet) the system could answer questions like:

 

“Who is Yannis Ritsos”

“What is the profession of Yannis Ritsos” (poet is a profession)

“Name a poet of Greece” (Greece is a nation and Greeks its inhabitants)

I used the CynD AIML Interpreter. The system uses specially modified AIML to access the KB NL facilites, query the KB and then return the results. For example:

Given the below AIML code and the sentence:

WHAT COUNTRIES BORDER *

(#$isa #$Country )

Consulting cyc returns: What countries border ?

 

The system calls the ResearchCyc parser (through the custom cycquestion tag), translate the input into its CycL (the language in which the KB is written) representations:

((#$and

(#$isa ?WHAT-ONE-1 #$Country)

(#$politiesBorderEachOther #$Greece ?WHAT-ONE-1))

(#$and

(#$bordersOn #$Greece ?WHAT-ONE-1)

(#$isa ?WHAT-ONE-1 #$Country))

(#$and

(#$isa ?COUNTRY #$IndependentCountry)

(#$politiesBorderEachOther #$Greece ?COUNTRY))

(#$and

(#$bordersOn #$Greece ?COUNTRY)

(#$isa ?COUNTRY #$IndependentCountry)))

And then iterate through the list of parses, and try to query the KB to get a possible answer. In the particular example the system returns:

 

Consulting Cyc returns:

0 (((?COUNTRY . Bulgaria))((?COUNTRY . Macedonia))((?COUNTRY . Albania))((?COUNTRY . Turkey))).

1. (((?COUNTRY . Bulgaria))((?COUNTRY . Macedonia))((?COUNTRY . Albania))((?COUNTRY . Turkey)))

2. (((?WHAT-ONE-1 . Bulgaria))((?WHAT-ONE-1 . Macedonia)) ((?WHAT-ONE-1 . Albania))((?WHAT-ONE-1 . Turkey)))

3.(((?WHAT-ONE-1 . Bulgaria))((?WHAT-ONE-1 . Macedonia))((?WHAT-ONE-1 . Albania))((?WHAT-ONE-1. Turkey)))

Finally it calls the NL generator of the KB and translate the results back to English:

 

Consulting Cyc returns: 0 Bulgaria Macedonia Albania Turkey. 1 Bulgaria Macedonia Albania Turkey. 2 . Bulgaria Macedonia Albania Turkey 3. Bulgaria Macedonia Albania Turkey

Problems:

1) CynD will try to query the KB for every single interpretation of the user's question. In the above example, the KB returned the same answer for all parser interpretations. Some kind of tool was needed that would automatically generate AIML categories incorporating all possible parser interpretations, allow the system's author to select the one that produced the answer he expected and make the system's answer more natural. There was no time for that.

 

2) In an open Q-A dialogue with a system in my domain (mobile guides), the user could ask questions about random objects that he may encounter in his path-something like "What is that?" for a church or a particular building. The disambiguation of "that" to the actual object name and location in the physical environment is an extremely difficult problem -- one of the many that NLP has to face in the future. At the moment, there is no location technology that can allow a computer to distinguish between two objects located at a close distance in a psychical environment.

 

3) My work focuses on animated agents. If the dialogue with the system is unknown... how could I possibly generate the proper animations for each output of the CyND engine?

 

5) Hardware resources. Such a system demands massive hardware resources... resources in a M.Phil project are always scarce.

Lessons learned:

1) The pilot evaluation suggested a) several improvements in the experimental design b) several design improvements of the prototype.

2) The need to produce more user-insights. This could only be accomplished, by developing more prototypes varying different attributes of virtual humans (e.g., competence, modality of communication, etc.).

3) The need for technological differentiation. Its impossible to continue the simplistic approach of this prototype, as the research will lead me nowhere. The domain of VH is extremely comparative area with groups of researchers that produce massive contributions to both technology and knowledge every single month.

4) My rather bumpy experience in developing the prototype also suggested the need for an authoring tool that would enable non-programmers to rapidly develop complex Virtual Human systems.

Development of Virtual Humans - Early Years

The idea, was to develop and evaluate a conversational sales assistant that would assist users in all stages of the CBB (Consumer Buying Behaviour Model) - a model used to describe the entire range of consumer needs in on-line commerce environments. The algorithm alone, would have been enough for a M.Phil contribution. Hence, with great enthusiasm, I started my work at Middlesex. The first task in my research list, was the development of a fully-working prototype. I designed a rather complex system (shown below along with an explanation) and begun looking for proper authoring tools.

 

A discourse manager module can not be evaluated if it is not part of a general dialogue system . The organization of system needed to test this module is shown below: Starting at the top of the figure, the user communicates with the system textually through a standard desktop PC. The input is parsed to a sequence of speech acts based on the syntactic and semantic form of the utterance and sent to the discourse manager. The discourse manager sets up its initial conversation state and passes the sequence to the context for identification of any lexical information (e.g. names, features etc), and then hands the acts to the reference resolution. The component has two duties. First, it assigns the correct referent to anaphoric referring expressions (e.g. the pronoun her in the sentence Anne asked Edward to pass her the salt refers to Anne). Then, if it is necessary it does illocutionary remapping of the speech acts assigned by the parser as needed to fit discourse and reference cues. For instance, an utterance that consist of a REJECT “no” followed by a REQUEST “go via bank” will have the letter REQUEST remapped into the REJECT; it is essentially the content of the REJECT, not a separate REQUEST. After this processing, reference returns the speech act(s), now in an internal format, to the discourse manager for further disposition. These speech acts however, are only the surface acts (i.e., literal acts) of the utterance. Oftentimes, a surface speech act has a deeper intended or indirect meaning (i.e. an illocutionary act). Take, for example, the utterance, “Can you pass the salt?” which on the surface looks like a QUERY speech act (i.e., asking about your ability to pass the salt). However, the indirect speech act is most likely a REQUEST (i.e. a request to pass me the salt). The discourse manager detects and interprets indirect speech acts through an intention recognition process done in conjunction with the Task Manager. More specifically, it sends each of the postulated acts to the Task Manager and asks whether this interpretation would ‘make sense’ given the current domain and planning context. Each of these postulates are then scored by the Task Manager which allows the Discourse Manager (DM) to find the correct interpretation of the surface speech act (i.e., the intended problem solving act). In addition, with its indirect speech acts duties, the discourse manager must convert the speech act semantics from a linguistic knowledge representation (produced by the parser) to a domain specific, problem-solving knowledge representation (used by the Task Manager for intention recognition). For example, the linguistic semantics of the utterance “Buy the HP Palm from PC-World” might be:

(buy1

:agent SYSTEM

: theme HPPalm

: goal PC-World

The domain specific, problem-solving knowledge representation would correspond to some action in the domain (purchase) with some corresponding parameters. For example:

(purchase

:product HPPalm

:from-merchant PC-World

:price ?p)

The ?p is a variable, since the price was not explicitly mentioned in the utterance.

The output of the task manager and the interface agent is a set of surface speech acts that the system wants to communicate to the user. The generation module achieves this communication using the graphical user display. This module also uses prioritized rules to match requested speech acts to appropriate means of expressing them. For example a request to the generator to inform user about a specific product will result in telling the display to show the image of the product, and simultaneously provide a textual description. The above architecture is by no means complete. We suspect that several modifications to the existing components and addition of new ones can be made during the course of development. However, it serves as a good example of the great complexity of the project.

I spent several months search the WWW. I have a massive URL library from companies that probably no longer exist. An example is: http://www.agentscape.de/, that asked me 15,000 USD for their authoring tools. To that, add, a) a University refused to spent any more money on me (it was already paying me a scholarship) b) a supervisor that kept insisting to follow that path, and you end-up with the perfect recipe for career destruction. Under the massive pressure, instead of quitting, I decided to move to a less complex domain and area of research. It was clear, that this project was better suited for a team of researchers with a perhaps unlimited budget. When I think about it today, it is still unknown, why my supervisor failed to see something so obvious!!!!

Development of Virtual Humans - Early Years

I think, its a good idea to start this post, with a simple, non-technical definition for a virtual human. A virtual human is an intelligent system capable of providing and accepting information through a full-range of human modalities (e.g., speech,gesture, face expressions, etc).

Constructing an intelligent VH with a general intelligence is a scary idea that should never be pursued. Instead I am more fun of the the robot-slaves idea, i.e., virtual humans (in physical or holographic form) that ONLY look intelligent in a specific domain (e.g., tour-guides, sales, etc)

I initially started working with VH back in 2002, during my Master's degree. The demo I constructed, the e-briefing room is shown below, along with a brief description of its functionality:

The “e-Briefing Room” service provides a tool for effectively educate customers with complex and high-end products on the Web. This vehicle allow customers to fully interact with three-dimensional (3D) models of products online, and also access personalized services on demand, which will be provided by a three-dimensional (3D) talking virtual sales-assistant.Through this interactive technology, the e-briefing room makes it easy for customers to access information on demand. Start your experience by directly selecting a product category of your choice, or activate Derek to listen a detail presentation about the functionalities of the service.

The demo was extremely simple (but... I did it only for my Master's degree). Its only difference from a static web-page, was the 3D head (from an Australian company called famous3D). Back in those days I wasn't even aware of the existence of ALICE But once you got the bug of VH technology you can not easily stop. The simple "e-briefing room" was enough to start me wondering what could be next in the exciting world of Virtual Humans.

In 2003 I was given the opportunity to study for an M.Phil degree at Middlesex University in London. My decision was to continue working on Virtual Humans for electronic commerce.