Many of the texts that are translated today are multimodal in one way or another: they consist of various interrelated modes, for example, image and word or speech and body language. In multimodal texts, such as films or illustrated texts, verbal language is accompanied by other kinds of visual and/or auditive information that interact with words and influence the eventual interpretation of the multimodal text. Human social interaction and therefore also interpreting constitute of multimodal communication between the co-participants. The translation and the translational enquiry of texts and interaction – be it by an individual translator or by a team – needs to include a careful consideration of all of the modes involved and the ways in which the modes combine to make meaning. In addition to the multimodal forms of interlingual translation, the study of multimodal meaning construction is notably relevant in the field of accessibility: audio description, for instance, entails converting visually represented information into a verbal form, and requires an in-depth understanding of the multimodal entity.
The MULTI (Multimodality in Translation and Interpreting) research group investigates the implications of multimodal meaning construction in translation and interpretation. The group examines how different aspects of multimodality affect the work of translators and interpreters on the one hand, and the users of these products on the other. The research interests of the group cover a variety of languages (Finnish, English, Russian, German, Hungarian, Italian, and Spanish), and areas of multimodal intercultural communication, such as interpretating, audiovisual translation, collaborative translation process, audio description, and the translation of illustrated texts and comics.
MULTI’s research focuses on
- developing novel research approaches that can address the diversity of multimodal data
- reviewing the current state of research on multimodal translation within translation studies and promoting awareness of the importance of such research to the discipline and the disciplines close to it
- envisioning and encouraging new ways of thinking about and conceptualizing translation from the viewpoints of multimodality, materiality and intermodal and intersemiotic translation
- developing interdisciplinary research approaches that allow for the emergence of new and broadened perspectives into multimodal translation.
MULTI studies the communicative functions of the different modes in relation to each other and the socio-cognitive basis of translatorial action, in particular how
- a translation interacts with other modes as a part of a multimodal orchestration
- meanings transfer from one mode to another
- material media shapes the contents of the multimodal orchestration
- meanings are constituted and negotiated multimodally in interaction between translating or interpreting co-participants.
Directors
Maija Hirvonen (professor)
Eliisa Pitkäsalo (university lecturer, docent, Sarjis)
Members
Anne Ketola (university researcher, PhD, Sarjis)
Riitta Oittinen (emerita university lecturer, docent)
Laura Hekanaho (post-doctoral researcher)
Chiara Galletti (doctoral researcher)
Riku Haapaniemi (doctoral researcher)
Tero Ikävalko (doctoral researcher)
Mira Kainulainen (doctoral researcher, Sarjis)
Betta Saari (doctoral researcher)
Frederike Schierl (doctoral researcher)
Affiliated groups
Eliisa Pitkäsalo, Laura Kalliomaa-Puha, Anne Ketola, Vaula Haavisto, Kirsi Günther, Mira Kainulainen, Jan Pitkäsalo
Sarjis is searching for ways to make legal documents more accessible and find out whether the visualisation of documents can improve the accessibility of information. Making diffcult content (e.g. concepts) easier to grasp makes the text easier to understand and, therefore, supports the client’s autonomy and participation in society. In its current projects, Sarjis is translating various kinds of documents into comics.
Tampere Accessibility Unit (TACCU)
Tampere Accessibility Unit is multidisciplinary unit conducting accessibility-related research and education at the Tampere Universities. The work is characterized by cutting-edge research, novel education contents as well as active collaboration with external stakeholders, such as associations and companies that work with accessibility.
Funded projects
From the margins to the masses: Standard practices and innovative uses of technology in augmenting different abilities of people in the world of work (NewWorkTech, 2024-2027, Horizon Europe research & innovation action).
Maija Hirvonen (Coordinator, PI) & Plural research center, Roope Raisamo & TAUCHI research center (Tampere University, FI), University of Oulu (FI), University of Copenhagen (DK), University of Warwick (UK), Consiglio Nazionale Delle Ricerche (IT), European Platform for Rehabilitation (BE), Associazione Italiana Persone Down (IT), Instituttet for Blinde og Svagsynede (DK), Autism Foundation Finland (FI), Be My Eyes Aps (DK).
NewWorkTech engages in research-based enhancement of the work-related capacities of a) people with disabilities and b) the general workforce, with particular focus on technologically mediated tasks and interaction. The project covers the entire arc of development from empirical research into how people with disabilities – as forerunners of technology use – perform tasks and interact at work, to theoretical innovation regarding the nature of socio-material assemblages as well as what constitutes technology, to policy recommendations, and to the development of new technological solutions, including AI-based technologies. We consider people with disabilities to be experts in using technical systems to improve their work performance, and through an analysis of a variety of use cases, will identify elements of successful use as well as factors that cause problems for interaction or the performance of tasks. Empirical results will also be generated by surveys and co-creation activities. People with disabilities will be engaged in NewWorkTech in line with the principle “nothing about us without us”. NGO partners will be actively involved as facilitators in the empirical research and as drivers of application and dissemination. Through our own technological development effort, attention to the ethical use of technologies, and the dissemination of the knowledge we generate, the results will feed into improved usage methods and functionality of work-related technologies, as well as knowledge about the required skills. The smart use of technology thus achieved will help create employment opportunities for people with disabilities, improving their quality of life and reducing inequalities while also helping alleviate the current shortage of labour in Europe, and open possibilities for more efficient and creative use of technology by everyone, whether they have physical or cognitive limitations or not.
Guided Audio Captioning for Complex Acoustic Environments
Annamaria Mesaros (PI), Irene Martin Morato, Manu Harju (Audio Research Group), Maija Hirvonen, Riku Haapaniemi (MULTI). Jane and Aatos Erkko Foundation (2022-2023)
The project develops a novel approach for machine learning to automatically produce linguistic descriptions of everyday acoustic environments. The general aim is to increase the understanding of how people (sighted and blind) interpret the world through sound and to teach intelligent machines to simulate it.
Using Language to Interpret Unstructured Data
Tuomas Virtanen (PI, Audio Research Group), Parthasaarathy Ariyakulam Sudarsanam, Huang Xie (Audio Research Group), Maija Hirvonen, Laura Hekanaho, Frank Emmert-Streib (Predictive Medicine and Data Analytics Lab), Nadeesha Perera, Pavel Davidson (Data Science). Emil Aaltonen Foundation (2021-2023)
The interdisciplinary project developed deep learning methods to compute unstructured data, such as images and sounds, into linguistic representations by learning from human multimodal communication and intermodal translation. This multidisciplinary research project combined signal processing, machine learning, health technology, translation studies and linguistics. The project was carried out at Tampere University, the Faculty of Information Technology and Communication Sciences (ITC).
Graphic Justice
Eliisa Pitkäsalo (PI) and the Sarjis research group. Academy of Finland (2020–2024)
Our multidisciplinary research group examines how the comprehensibility of different types of legal documents – such as contracts or court rulings – could be improved by using comic-style communication. Official documents can often be difficult for clients to understand, due overly complicated language, unfamiliar terms, etc. Misunderstandings can impede people from accessing their social rights. The Graphic Justice research project aims to tackle this problem by examing whether a comic-style document can be legally binding and whether visual presentation of information can make the document easier to understand and hence improve the client’s access to justice. The project is carried out in collaboration between researchers from the Faculty of Social Sciences (SOC) and the Faculty of Information Technology and Communication Sciences (ITC). The Principal Investigator of the project is Eliisa Pitkäsalo (ITC).
Word to Image
Laura Kalliomaa-Puha (PI), Eliisa Pitkäsalo, Anne Ketola, Vaula Haavisto, Kirsi Günther, Mira Kainulainen, Jan Pitkäsalo. Kone Foundation (2020–2023)
The Word to Image research project examines the visualization of social welfare documents. The language of institutional texts can often be difficult to comprehend, especially for clients with limited language or reading skills or different types of cognitive challenges that affect reading. The research project examines if the comprehensibility – and, hence, accessibility – of social welfare documents could be improved by converting the documents into comics. Further, the project sheds light on what the creators of comic-style documents need to take into account in order to produce documents that are legally binding also in a visual format. One example of a comic-style document the project is working on is an agreement for a supervised exchange, created in collaboration with the Finnish Federation of Mother and Child Homes and Shelters. The project is carried out in collaboration between researchers from ITC and SOC. The Principal Investigator of the project is Laura Kalliomaa-Puha.
MUTABLE: Multimodal Translation with the Blind
Maija Hirvonen (PI). Academy of Finland (2017–2020)
MUTABLE studies the collaboration and interaction in audio description teams that consist of blind and sighted members. The project sheds new light on collaborative translation processes, shared cognition, and multimodal interaction.
Ongoing dissertations and other research projects
The One Who Speaks Last Speaks for All. Mediating Mechanisms at Play in the Collaborative Indirect Translation of Multimodal Children’s Storybooks (doctoral dissertation)
Chiara Galletti
The dissertation project deals with a relatively common practice in children’s literature: the indirect translation of multimodal storybooks. This practice involves the collaboration between a professional in charge of producing a preliminary word-for-word translation crib and a storyteller in charge of shaping the voice of the story in view of the read-aloud situation. Applying Andrew Chesterman’s (2019) notion of consilience, the theoretic framework includes the studies on indirect translation carried out by the IndirecTRans international research network (Assis Rosa, Pięta and Bueno Maia 2019) and Marco Dallari’s narrativity studies in the field of multimodal children’s literature (2012). The empirical investigation concentrates on a small-scale corpus of multimodal storybooks translated indirectly by the Italian children’s author Bianca Pitzorno. The methodology involves a combination of contextual and textual descriptive analysis.
Materiality of texts in translation studies (doctoral dissertation)
Riku Haapaniemi, Finnish Cultural Foundation grant
This dissertation project discusses the concept of materiality in the study of translation, utilising perspectives from semiotics and textual theory. Materiality is identified as a common concern across all kinds of textual communication: modes of expression must always have material forms that can be identified by recipients, and the materiality of these forms affects the meaning constructed from the text. Translation Studies has a unique perspective on material processes of textual communication, and a material approach is useful in studying a variety of different translational phenomena.
Comic documents as an instrument of improving accessibility and social inclusion (doctoral dissertation)
Mira Kainulainen, Sarjis research group
The topic of the doctoral thesis is comic documents as an instrument of accessibility and social participation. Comic documents are comic versions of text documents. The goal of the research is to further develop functional translation theory for intersemiotic translation and accumulate information on the characteristics and functionality of comic documents from the viewpoint of their users. Accessible communication improves users’ social participation because it helps them to understand communication about and for them.
The role of interaction and language use in the construction of audio described visual art museum experiences of visually impaired people (doctoral dissertation)
Betta Saari, Emil Aaltonen Foundation grant
The dissertation project studies how social interaction and language use construct visual art museum experiences of visually impaired people on audio described and guided art museum tours. The multidisciplinary project is part of fields of linguistics, accessibility research and experience research. From the viewpoints of social interaction and language use, I examine how experiences of visual art are formed as cooperation on guided, interactive and audio described art museum tours. The project increases knowledge about the collaborative construction of a visual art experience of visually impaired people through social interaction.
The role of the audio describer’s voice in an art experience mediated by audio description
Tero Ikävalko
In this project, statistical analysis of voice features is used to analyse how the characteristics of an audio describer’s voice affect the art experience of audio description users.
Machine Translation Potential (un)limited? – Investigating the quality of machine-translated vs. human-translated subtitles
Frederike Schierl
The doctoral dissertation project looks into the reception of machine-translated and human-translated subtitles. Educational videos have become an important tool for learning over the last decade, fostered even more since the last pandemic. Still, contents might not always be available in the user’s language, resulting that audiovisual translation, mostly in the form of subtitling, is needed. Nowadays, the content to be translated exceeds human translators’ capacities, therefore machine-translated subtitling can provide a low-threshold support and access to content which would otherwise not be accessible. The dissertation project is built as a reception study based on the 3R concept (response, reaction, repercussion, Gambier 2009), and it investigates visual attention allocation, cognitive effort in processing, and attitudes and expectations towards subtitling in general and machine-translated subtitles in particular. The aim of the research is to gain a more thorough, i.e. better, insight into the usability, viability and potential of machine-translated subtitles in the context of accessibility.
Key Publications
You can find the up-to-date research listings on each researcher’s Tunicris profile. You can access Tunicris by clicking on the researcher’s name at the section titled Research community.
2023. Haapaniemi, Riku. Translation as meaning-construction under co-textual and contextual constraints: A model for a material approach to translation. Translation Studies. DOI: 10.1080/14781700.2022.2147988
2023 (forthcoming). Anne Ketola, Eliisa Pitkäsalo & Robert de Rooy. Contracts via comics: Migrant workers and Thai fishing vessel employment contract. Teoksessa Comics and Migration: Practices and Representation, toim. Ralf Kauranen, Olli Löytty, Aura Nikkilä & Anna Vuorinne. London: Taylor & Francis.
2023 (forthcoming). Pitkäsalo, Eliisa & Riitta Oittinen. On Translation and Censorship in Children’s Literature. Teoksessa Routledge Handbook of Translation and Censorship, toim. Denise Merkle & Brian Baer. London: Taylor & Francis.
2022. Hirvonen, Maija & Betta Saari. Scripted or spontaneous? Two approaches to audio describing visual art in museums. Perspectives: Studies in Translatology, (E-pub ahead of print) https://doi.org/10.1080/0907676X.2022.2046816
2022. Pitkäsalo, Eliisa, Anne Ketola, Vaula Haavisto & Laura Kalliomaa-Puha. Image analysis as a visualization tool – Translating contracts into comics. Teoksessa Research Handbook on Contract Design, toim. Marcelo Corrales, Helena Haapio & Mark Fenwick. 347–364. Cheltenham: Edward Elgar Publishing.
2021. Ketola, Anne. Visual explicitation in intersemiotic translation. STRIDON: Studies in Translation and Interpreting 1 (1): 103–122.
2020. Hirvonen, Maija & Tuija Kinnunen (toim.). Saavutettava viestintä: Yhteiskunnallista yhdenvertaisuutta edistämässä. Helsinki: Gaudeamus. Sisältää artikkelit: Hirvonen, Kinnunen & Tiittula; Pitkäsalo & Kalliomaa-Puha; Hirvonen & Tiittula
2020. Hirvonen, Maija & Mari Wiklund. From image to text to speech – The effects of speech prosody on information sequencing in audio description. Text & Talk 41, 3, p. 309-334.
2020. Pitkäsalo, Eliisa (2020). Traduction intersémiotique et contexte : des contrats en bande dessinée en tant que documents juridiques accessibles. Meta: Translators’ Journal 65:1
2020. Pitkäsalo, Eliisa & Laura Kalliomaa-Puha (2020). Sarjakuvasopimukset – esimerkki oikeudellisen tekstin visualisoinnista. In Maija Hirvonen and Tuija Kinnunen (eds.): Saavutettava viestintä: Yhteiskunnallista yhdenvertaisuutta edistämässä. Gaudeamus.
2019. Pitkäsalo, Eliisa & Laura Kalliomaa-Puha. Democratizing access to Justice: the comic contract as intersemiotic translation. Translation Matters 2 (2019).
2018. Hirvonen, Maija. Audiodeskription und Sichtdolmetschen: Translation über Sinnesgrenzen. In Kvam, Sigmund, Parianou, Anastasia, Schopp, Jürgen & Solfjeld, Kåre (eds.): Spielräume der Translation: Sprach- und translationswissenschaftliche Zugänge. Münster: Waxmann, 100–124.
2018. Hirvonen, Maija & Tiittula, Liisa. How are translations created? Analysis of multimodal interaction as a methodology to study a team translation process. Linguistica Antverpiensia, Special issue “Methods for the Study of Multimodality in Translation” 17, 157–173.
2018. Hirvonen, Maija & Schmitt, Reinhold. Blindheit als Ressource: Zur professionellen Kompetenz eines blinden Teammitglieds bei der gemeinsamen Anfertigung einer Audiodeskription. Gesprächsforschung 19, 449-477.
2018. Jimenez Hurtado, Catalina, Anne Ketola &Tiina Tuominen (toim). Methods for the Study of Multimodality in Translation. Linguistica Antverpiensia – Themes in Translation Studies (17/2018).
2018. Ketola, Anne: Word–Image Interaction in Technical Translation. Students Translating an Illustrated Text (väitöskirja)
2018. Oittinen, Riitta & Eliisa Pitkäsalo. Creating Characters in Visual Narration: Comics and Picturebooks in the Hands of the Translator. In Search of Meaning and Identity Through Language; Literary, Linguistic, and Translational perspectives. Toim. Hanne Juntunen, Kirsi Sandberg & Kübra Kocabaş. Tampere Studies in Language, Translation and Literature, Series A5.
2018. Pitkäsalo, Eliisa. Images without words. Focus group as a method of experiencing comics. Methods for the Study of Multimodality in Translation. Toim. Catalina Jimenez Hurtado, Anne Ketola and Tiina Tuominen. Linguistica Antverpiensia -Themes in Translation Studies (17/2018).
2017. Hirvonen, Maija & Igareda, Paula. Imaginary places – Verbalization of setting in the English, German and Spanish audio descriptions of Slumdog Millionaire. mtm – A translation journal 9, 341–364.
2017. Ketola Anne. Using Translation Research to Model Word Image Interaction. New Voices in Translation Studies 17, 82–104.
2017. Oittinen, Riitta, Anne Ketola & Melissa Garavini. Translating Picturebooks. Revoicing the Verbal, the Visual and the Aural for a Child Audience. New York: Routledge.
2017. Pitkäsalo, Eliisa. Conveying Nonverbal Messages in Translated Comics. In: Sándor Maticsák & László Keresztes (eds.) Folia Uralica Debreceniensia 23. Debrecen: A Debrecen Egyetemi Kiadó. 233–243.
2016. Ketola, Anne. Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9 (1), 67–81.
2016. Pitkäsalo, Eliisa & Nina Isolahti (toim.) Kääntäminen, tulkkaus ja multimodaalisuus. Menetelmiä monimuotoisten viestien tutkimiseen. Tampere Studies in Language, Translation and Literature.
Other activities
MULTI participates in organising a digital study module called “Accessibility in a Digital Society” that has been available since 2021 under the life-long learning program of Tampere University. More information at the TACCU website (Tampere Accessibility Unit).
Contact
Maija Hirvonen, professor
maija.hirvonen@tuni.fi
+358 50 318 25 03
Eliisa Pitkäsalo, university lecturer, docent
eliisa.pitkasalo@tuni.fi
+358 50 318 1238
Anne Ketola, university researcher, PhD
anne.ketola@tuni.fi