A Platform for Conversational Experiences

A multi-purpose platform for conversational AI agents, which provides the functionality necessary to simulate natural conversation in diverse situations. Our platform employs tools such as intention recognition, interaction control, motion generation, conversational proficiency assessment, Human-in-the-Loop machine learning and dialogue control software, which choreographs all platform functions.

• Human intention recognition：Recognizes user intention by extracting features from speech, language, facial images, gaze information, etc. as input.
• Interaction control：This tool provides a set of functions related to timing control, for instance determining whether a user intends to continue or end an utterance, or resolving the problem of speech overlap whenever it occurs.
• Motion generation service： Generates various gesture trajectories based on discourse and syntactic information, as well as the rapidly-developing flow of conversation. Produces nonverbal motions such as synchronous facial expressions and mimics the body language of an active listener, i.e. through head-nodding motions activated during user speech.
• Conversational skill assessment service: This service assesses the conversation ability of both the user and the agent. This service is expected to be used not only for English conversation learning support applications, but also for a wide range of communication ability assessment.
• Human-in-the-Loop Machine Learning：Crowdsourcing enables the collection of large amounts of data concerning pattern recognitions in areas such as human intention recognition and conversational language proficiency assessment. Cloud workers perform data-labeling tasks and produce high-quality data that is then used in the process of creating dialogue scenarios for our conversational AI agent platform.
• Dialogue control： A dialogue control system manages all of the tools described above. Users can write dialogue scenarios according to their respective business logic, and realize conversational interaction through video conferencing and XR devices.

Conversation patterns currently supported by our platform include:

• Interview-style questioning, which elicits and extracts appropriate information from the user 
• Information expounding, which actively caters to and addresses user interests – e.g. a museum guide. 

In the near future, we plan to support a variety of conversation patterns that are adaptable for use in various industries. Our platform will enable the efficient and cost-effective joint decision-making between humans and AI agents at point of sale and retail sites.

A Conversation-Oriented Communication Framework

For the use of our dialogue system, we have designed and developed the Duplex Communication and Transactional Streaming Framework (DUCTS – joint development with Intelligent Framework Research Institute), an open source communication framework that allows mutual modules to be processed robustly and without delay, even when accessed by multiple users simultaneously.

A voice search engine (Pull-based information transmission), such as a smart speaker, can be built on a conventional web paradigm, since it operates by 1) receiving a user’s voice query and 2) consequently responding with the appropriate search result. In a more “conversational” setting however, it is not obvious who – the system or the user – should take their turn to speak, causing difficulties in handling regular, linear server-client communications. 

DUCTS natively supports conversational interaction and advanced asynchronous communication. It consists of a server and client (currently compatible with mobile devices) Web, and VR/AR) that enable communication based on fine data units such as those delimited by utterance clauses. DUCTS supports text, voice and multimodal data. The cloud system also scales according to the number of concurrent user connections.