Grand Challenge — Multimodality as a Design Feature of Human Language

Our invited guest was Prof. Aslı Özyürek, Director of the recently founded Multimodal Language Department at the Max Planck Institute for Psycholinguistics in Nijmegen. Her research uses cross-linguistic, developmental, and experimental approaches to study speech, gesture, and sign, and challenges the traditional view of language as a unimodal communication system.

Abstract

Most theories about the nature of human language — how it is processed, acquired, and evolved — have assumed that language can be characterised by what is expressible in speech or text: a single channel, arbitrary form–meaning mappings, and sequential organisation. However, all the world’s languages embody visible components: spoken languages are accompanied by manual and facial gestures, and language can be expressed by signed as well as spoken languages.

In this talk I explain how taking into account these universally shared visible components, and their modality-specific characteristics such as iconicity, visible indexicality, and simultaneity (e.g. saying “play” in speech while performing a piano-playing gesture at the same time), changes the way we view and revisit our assumptions about language structure, use in interaction, (neural) processing, acquisition, and evolution. Presenting evidence from research in different spoken and signed languages, I argue that a multimodal view of language can explain its adaptive structure, resilience, and learnability across contexts, cultures, and individual variation.