We announce the release of InproTKs, which is a set of extension modules for InproTK that allow for easier integration of multimodal sensors for situated (hence the s) dialogue. Also included in this release is a module that can directly use the Google Web Speech interface, allowing a dialogue system to have access to a large-domain speech recognition engine.
You can read about it in our 2014 SigDial paper. An explanation is given below. Download instructions can be found at the end of this page.
InproTK is an implementation of the incremental unit processing architecture described in Schlangen & Skantze (2011). It includes modules for several components, including an incremental version of Sphinx ASR, incremental speech synthesis using Mary TTS, as well as some example modules for getting you up and running with your own incremental dialogue system.
The extensions released with InproTKs make it possible to get information into InproTK from outside sources (even from network devices on various platforms), as well as get information from InproTK out to other modules.
Below is an example. A human user can interact with the system using speech and gestures. InproTK provides the ASR (speech recognition) as a native module. Using the extensions in InproTKs, information from a motion sensor (such as a Microsoft Kinect) can be fed into an external processing module which, for example, can detect certain gesture types, like when the human is pointing to an object. That information can then be sent to InproTKs using the extension modules (denoted in the figure as a Listener). That information can then be used to help the dialogue system make decisions. Information can also be outputted to external modules (such as the logger in the figure).
There are three methods which one can use to get information into and out of InproTK: XmlRpc, the Robotics Service Bus, and InstantReality. Modules in InproTKs exist for each type, a module for getting data into InproTKs known as Listeners and modules for getting modules out of InproTKs, known as Informers. Each method now be briefly explained.
XmlRpc is a remote procedural call protocol that can be used to send information across processes (potentially running on different machines). Libraries exist in most programming languages.
Robotics Service Bus (from their website) “The Robotics Service Bus (RSB) is a message-oriented, event-driven middleware aiming at scalable integration of robotics systems in diverse environments. Being fundamentally a bus architecture, RSB structures heterogeneous systems of service providers and consumers using broadcast communication over a hierarchy of logically unified channels instead of a large number of point-to-point connections. Nevertheless RSB comes with a collection of communication patterns and other tools for structuring communication, but does not require a particular functional or decomposition style.”
RSB has bindings for Java, Python, and C/C++ languages.
InstantReality (from their website) “The instantreality-framework is a high-performance Mixed-Reality (MR) system, which combines various components to provide a single and consistent interface for AR/VR developers.”
InstantReality only handles Java and C++, but can be used directly with the InstantReality player where one can create a virtual reality scene.
Download Instructions
The InproTKs is found in the InproTK git reposity under the “develop” branch.
Open a terminal and run the following two commands:
>git clone https://[your-user-name]@bitbucket.org/inpro/inprotk.git
>git fetch && git checkout develop
This will reveal the inpro.io package in the src folder. The package contains a README file that explains examples. You will need the instantreality.jar from the InstantReality website (simple download and install the software and you will find the jar included) as well as the protobuf jar from the Google protobuf website. (Eclipse users can import the InproTK as a project; note that the inpro.io.instantreality package might be excluded from the build because it relies on a jar that we cannot distribute; please add the two above jars to the classpath and include the inpro.io.instantreality package).