Multimodal Interaction Research Group
Our MMIG group looks at all aspects of multimodal HCI (Human-Computer Interaction). With multimodal interaction, we mean humans using machines via several senses. For instance, people can touch, feel, smell and hear computer outputs, and computers can decode how people speak, wave hands, or look at things.
Our group approaches research through analytic and constructive means. We build prototypes, study behavior of people, invent new technologies, carry out laboratory tests, and do generally crazy stuff.
Some of our research topics are listed below, together with a few key research questions.
Selected papers on these topics are found in the Publications section, and specific topics in the Projects section.
Automotive interaction, new applications of modalities for communication between cars, their users, information and communication systems and smart traffic systems. Interaction needs and technologies in autonomous driving. Multimodal systems for control and maintenance of industrial automation, production and production plant development, technology use as extensions of human sensing and cognition.
Mobile and wearable device interaction. Developing multimodal kinaesthetic interfaces by combining eye gaze and hand motion as the kinaesthetic input. Multimodal VR, HMDs, fogscreens, interaction with novel 3D UIs. Contactless mid-air tactile feedback, super-wide FOV optics for HMDs, direct retinal feedback for HMDs. Design and research of haptic feedback for non-contact interaction.
How to improve virtual reality locomotion techniques while minimizing simulator sickness?
Visualization and imaging based on intelligent tactile surfaces. Imagery-based mental visualization and interaction, brain-computing and mental interfaces. Information transfer and integration, perceptual modifiers (materials, methods, and techniques). Learning how to integrate the virtual information with reality without overwhelming users.
How to utilize all human senses while providing complex information? What techniques for human-device interaction are suitable for AI systems in commercial and personal space? Do users of virtual reality remember visual information better if it is combined with a matching scent?
Research on multimodal XR interaction technologies for medicine and therapy. Research on new technologies to improve human awareness of environmental factors and health hazards. Utilizing modern user interaction technologies to enable users to have more efficient and natural interaction. Research on multimodal (visual, auditory, haptic, scent) technologies for creating eating experiences that promote health and wellbeing.
Can restorative virtual environments replicating real natural environments such as forests bring therapeutic benefits?
Studying user behavior in basic multimodal interaction task, such as improving speed and accuracy of pointing with good quality feedback, using haptics and other technologies. Studying timing requirements. Experimenting with new uses of existing techniques for user interaction, such as analyzing gaze data to disambiguate intent, to improve accuracy in pointing. Categorizing the user context to streamline the interaction.
How much do we know about what the user is trying to achieve?
Developing new technique to monitor and regulate how actuation signals are mediated from the source (actuation mechanism) to the point of (skin) contact. Identifying and enhancing actuation signals within an interaction system to actively remove environmental noise and other inefficiencies found in conventional actuation devices, ensuring the intended signal reaches the point of contact with minimum attenuation or interference.
Haptic Mediation: “A process of effectively relaying the actuation signal from the source (actuator) to the necessary point of contact (respective area of the skin), mitigating environmental noise and other internal and external inefficiencies within the system.” From: Farooq, 2017
Designing and fabricating novel actuation components to provide more efficient multimodal systems. Fundamental research focused on creating a new generation of efficient actuation components that are targeted towards greater energy transfer as compared to replicating existing physical parameters (i.e. acceleration, displacement, frequency), which are outside human perceptual threshold.