Grand Challenge — Typology of the Visual Language(s) of Comics as a Window into a Multimodal Language Faculty

The Visual Language Lab (Tilburg University) discussed the multimodal nature of language by drawing insights from the visual languages used in comics and their relationships to the structures of spoken languages.

Abstract

Multimodality necessitates both the incorporation of all natural modalities into a common architecture and a rethinking of modalities’ fundamental relations to each other. Here we explore typological relations between the vocal and graphic modalities using the TINTIN Corpus, an annotated corpus of 1,030 comics from 144 countries. First, Neil Cohn outlines a multimodal cognitive architecture and its reconceptualization of diversity, universals, and relativity, and demonstrates the tensions between these issues in the framing structure of 76,000 panels from the corpus. Next, in “Whoosh! Visual depictions of motion events”, Irmak Hacımusaoğlu explores the lexicalized patterns of depicted motion events in 315 comics, and how basic linguistic constructs of motion events are typologically represented in graphic depictions. Finally, in “Iconicity and frequency of lines depicting motion in comics”, Ana Krajinović examines motion-event cues by comparing four types of symbolic motion lines by average size and frequency, and their diagrammatic and residual imagic iconicity. Altogether, we highlight how similar typological issues persist in the “visual languages” of comics and those of other linguistic systems.