Aaron Quigley, Professor CSE, UNSW

Machine Learning methods in Intelligent Interfaces: Challenges and Opportunities

Abstract

Opisthenar (UIST 2019) introduced a vision-based technique to recognize static hand poses and dynamic finger tapping gestures without instrumenting the fingers. This relies on a camera on the wrist, with a view of the back of the hand. The oblique angle and placement of the camera make typical vision-based techniques difficult to adopt. Our alternative approach observes small movements and changes in the shape, tendons, skin and bones on the area. Our trained networks recognize both hand poses and dynamic finger tapping gestures. Follow up work in (UIST 2020) advances this with a richer 3D hand pose estimation framework called DorsalNet. This is a two-stream convolutional neural network to regress finger joint angles from spatio-temporal features of the dorsal hand region (the movement of bones, muscle, and tendons).

RadarCat (UIST 2016, Interactions 2018, IMWUT 2018 UbiComp 2019) is a small, versatile system for material and object classification which enables new forms of everyday proximate interaction with digital devices. RadarCat exploits the raw radar signals that are unique when different material and objects are placed on the sensor. By using machine learning techniques, these objects can be accurately recognized. An object’s thickness, state (filled or empty mug) and different body parts can also be recognized. This gives rise to research and applications in context-aware computing, tangible interaction (with tokens and objects), and in industrial automation (e.g., recycling), or laboratory process control (e.g., traceability). While AquaCat (MobileHCI 2017 workshop) is a low-cost radar-based system capable of discriminating between a range of liquids and powders. Further in Solinteraction we explore two research questions with radar as a platform for sensing tangible interaction with the counting, ordering, identification of objects and tracking the orientation, movement and distance of these objects. We detail the design space and practical use-cases for such interaction which allows us to identify a series of design patterns, beyond static interaction, which are continuous and dynamic with Radar.

Bio

Aaron’s research interests include discreet computing, global HCI, pervasive and ubiquitous computing and information visualisation. He is an ACM Distinguished Speaker and the general co-chair for the ACM CHI Conference on Human Factors in Computing Systems in Yokohama Japan in 2021. Until June this year he was Chair of Human Computer Interaction in the School of Computer Science at the University of St Andrews, director of the Scottish Informatics and Computer Science Alliance (SICSA), board member for ScotlandIS and the DataLab.