Grand Challenge — Foundations of Multimodal Interactions

Returning for a new season, our first event featured Paul Liang (MIT Media Lab), discussing multimodal and multisensory AI systems.

Abstract

Building multisensory AI systems that learn from multiple sensory inputs — text, speech, video, real-world sensors, wearable devices, and medical data — holds great promise across many scientific areas, from supporting human health and well-being to enabling multimedia content processing and enhancing real-world autonomous agents. In this talk I discuss my research on the machine-learning principles of multisensory intelligence. Specifically, I present a theoretical framework formalising how modalities interact to give rise to new information for a task. These interactions are the basic building blocks in all multimodal problems, and quantifying them lets users understand their multimodal datasets and design principled approaches to learn these interactions. Finally, I discuss collaborative efforts to scale AI to many modalities and tasks for real-world impact on affective computing, mental health, and cancer prognosis.

Speaker

Paul Liang is an Assistant Professor at MIT Media Lab and EECS. His research advances the foundations of multisensory artificial intelligence to enhance the human experience. He is a recipient of the Siebel Scholars Award, Waibel Presidential Fellowship, Facebook PhD Fellowship, Center for ML and Health Fellowship, Rising Stars in Data Science, and three best-paper awards. He also received the Alan J. Perlis Graduate Student Teaching Award for teaching courses on multimodal machine learning.