Perspectivas en el análisis de la multimodalidad
desde los inicios al estado del arte
Keywords:
multimodality. semiotics. typography. speech synthesis and automation. medicine. hypertext. learning. translation. corpus analysis.Abstract
This article gives a broad historical overview from the early beginnings of multimodal analysis to currently burning issues and future perspectives in the study of multimodality. In contrast to other publications, the historical beginning of multimodality research is seen with Lessing’s Treatise on Laocoon (1766), so what we see today is the revival of a debate that was partly already discussed among the German classicists. A fervently discussed topic today is how modes relate as sub-modes to other modes and within different media. The meaning potential of a mode changes according to the salience of the mode within a semiotic construct as well as through the loss or gain of meta-functional meaning in the technological development of the mass-media. This is exemplified for typography. New topics and trends in current multimodality research are portrayed, and recent findings in a range of studies in applied research fields, such as translation in hospitals, hypertext design, museum design, film sub-titling and comics translation are presented and reviewed. Finally, future perspectives on how to put multimodality research on a more empirical basis, e.g. through multimodal corpus analysis, are outlined.
Downloads
References
ALIAS, S. (2004). ‘‘A semiotic study of Singapore’s Orchard and Marriott Hotel’’, en K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives, pp. 55-79. Londres: Continuum.
BALDRY, A. (Ed.) (2000a). Multimodality and multimediality in the distant learning age. Compopasso: Palladino Editore.
BALDRY, A. (2000b). ‘‘English in a visual society: Comparative mediality’’. en A. Baldry (Ed.), Multimodality and multimediality in the distant learning age, pp. 41-89. Campopasso: Palladino Editore.
BALDRY, A. (2004). ‘Phase and transition, type and instance: Patterns in media texts as seen through a multimodal concordancer’, en K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives, pp. 83-108. Londres: Continuum.
BALDRY, A. & Thibault, P. (2004). Multimodal transcription and text analysis. Londres: Equinox Publishing.
BATEMAN, J., DELIN, J. & HENSCHEL, R. (en prensa). ‘‘Multimodality and empiricism: Preparing for a corpus-based approach to the study of multimodal meaning-making’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 65-88. Amsterdam & Philadelphia: John Benjamins.
BESKOW, J. (2003). Talking heads - Models and applications for multimodal speech systhesis. Tesis doctoral no publicada. Department of Speech, Music and Hearing, KTH, Stockholm. http://www.speech.kth.se/~berkow/thesis
BÜHRIG, K. (en prensa). ‘On the «multimodality» of interpreting in medical briefings for informed consent: using diagrams to impart knowledge’ en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 227-214. Amesterdam & Philadelphia: John Benjamins.
CLARK, R.C. & MAYER, R. (2003). E-learning and the science of instruction. Proven guidelines for consumers and designers of multimedia learning. San Francisco, CA: Jossey-Bass/Pfeiffer.
COLBY, J. (2002). ‘‘Multimodality: The new wave of mobile interaction’’, en Speech Technology Magazine, November/December 2002. http://www. speechtechmag.com
DAVIS, G. & MASON, P. (2004). ‘‘Trying on voices: Using questions to establish authority, identity, and recipient design in electronic discourse’’, en P. Levine & R. Scollon (Eds.), Discourse & technology. Multimodal discourse analysis, pp. 47-58. Georgetown, Washington, D.C.: Georgetown University Press.
DE BEAUGRANDE, R.A. & DRESLER W.U. (1981). Einführung in die Textlinguistik. Tübingen: Niemeyer.
DE JONG, M. & VAN DER GEEST, T. (2000). ‘‘Characterizing web heuristics’’, en Technical Communications 3, 311-324.
DE SAINT-GEORGES, I. (2004). ‘‘Materiality in discourse: The influence of space and layout in meaning making’’, en P. Levine & R. Scollon (Eds.), Discourse & technology. Multimodal discourse analysis, pp. 71-87. Georgetown, Washington, D.C.: Georgetown University Press.
ECO,U. (1980). Il nome della rosa. Milano: Bompiani.
ECO, U. (1983). The name of the rose. New York: Harcourt. Echkrammer, E.M. (en prensa). ‘‘Drawing on theories of inter-semiotic layering to analyse multimodality in medical self-counselling texts and hypertexts’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on Multimodality, pp. 211-226. Amsterdam & Philadelphia: John Benjamins.
EVERTS, E. (2004). ‘Modalities of turn-taking in blind/sighted interaction: better to be seen and not heard?’, en P. Levine & R. Scollon (Eds.), Discourse & technology. Multimodal discourse analysis, pp.128-145. Georgetown, Washington, D.C.: Georgetown University Press.
FARKAS, D. & J. FARKAS (2000). ‘‘Guidelines for designing web navigation’’, en Technical Communication, 3, 341-358.
FILLIETTAZ, K. (2004). ‘‘The multimodal negotiation of service encounters’’, en P. Levine & R. Scollon (Eds.), Discourse & technology. Multimodal discourse analysis, pp. 88-100. Georgetown, Washington, D.C.: Goergetown University Press.
GOETHE, J. W. (1798/1998). ‘‘Über Laokoon’’, en F. Apel (Ed.), Goethe - Ästhetische Schriften 1771 - 1805, pp. 489-500. Frankfurt a. Main: Deutscher Klassiker Verlag.
HALLIDAY, M.A.K. (1961). ‘‘Categories of the theory of grammar’’, en Word, 17, 3, 242-292.
HALLIDAY, M.A.K. (1975). Language as social semiotic: towards a general sociolinguistic theory. Columbia: Hornbeam Press.
HALLIDAY, M.A.K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Arnold.
HALLIDAY, M.A.K. & HASAN, R. (1976). Cohesion in English. London: Longman.
HODGE, R. & KRESS, G. (1988). Social semiotics. Cambridge: Polity Press.
HOFINGER, A., & VENTOLA, E. (en prensa). ‘‘Multimodality in operation: Language and Picture in a Museum’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 193-2210. Amsterdam & Philadelphia: John Benjamins.
IEDEMA, R. (2003). ‘‘Multimodality, resemiotization: Extending the analysis of discourse as multi-semiotic practice’’, en Visual Communication, 2, (1): 29-57.
JAKOVI-MIRWALD, C. (1997). Buchmalerei; ihre Terminologie in der Kunstgeschichte. Berlin: Reimer.
JEWITT, C. (2004). ‘‘Multimodality and new communication technologies’’, en P. Levine & R. Scollon (Eds.), Discrouse & technology. Multimodal discourse analysis, pp. 184-195. Georgetown, Washington, D.C.: Georgetown University Press.
JOHNSTON, A. (2004). ‘‘Files, forms, and fonts: Mediational means and identity negotiation in immigration interviews’’, en P. Levine & R. Scollon (Eds.), Discourse & technology. Multimodal discourse nalysis, pp. 116-127. Georgetown, Washington, D.C.: Georgetown University Press.
JONES, R. (2004). ‘‘The problem of context in computer-mediated communication’’, en P. Levine & R. Scollon (Eds.), Discourse & technology. Multimodal discourse analysis, pp.20-23. Georgetown, Washington, D.C.: Georgetown University Press.
KAINDL, K. (en prensa). ‘‘Multimodality in the translation of humour in comics’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 173-192. Amsterdam & Philadelphia: John Benjamins.
KALTENBACHER, M. 2003. ‘‘Language learning via CD-Rom - Old wine in new bottles’’ en D. Newby (Ed.), Mediating between theory and practice in the context of different learning cultures and languages, pp. 171-175. Strasbourg: Council of Europe Publishing.
KALTENBACHER, M. (en prensa). ‘‘Multimodality in language teaching CD-ROMs’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 119-136. Amsterdam & Philadelphia: John Benjamins.
KENDON, A. (1990). Conducting interaction: Patterns of behavior in focussed encounters. Cambridge: Cambridge University Press.
KOK, A. (2004). ‘‘Multisemiotic mediation in hypertext’’, en K. O’Halloran (Ed), Multimodal discourse analysis: Systemic functional perspectives, pp. 131-159. London: Continuum.
KRESS, G., JEWITT, C., OGBORN, J. & TSATSARELIS, C. (2001). Multimodal teaching and learning. The rhetorics of the science classroom. London and New York: Continuum.
KRESS, G. & VAN LEEUWEN, T. (1990). Reading images. Geelong: Deaking University Press.
KRESS, G. & VAN LEEUWEN, T. (1996). Reading images: The grammar of visual design. London: Routledge.
KRESS, G. & VAN LEEUWEN, T. (2001). Multimodal discourse. The modes and media of contemporary communication. London: Arnold.
KRESS, G. & VAN LEEUWEN, T. (2002). ‘‘Colour as a semiotic mode: Notes for a grammar of colour’’, en Visual Communication, 1, 3, pp. 343-368.
LEE, J. (ED.) (1997). Intelligence and multimodality in multimedia interfaces: Research and applications. Menlo Park, CA: AAAI Press.
LEMKE, J. (2002). ‘‘Travels in hypermodality’’, en Visual Communication, 1 (3), pp. 299-325.
LESSING. G.E. (1766/1990). ‘‘Laokoon: Oder über die Grenzan der Melerei und Poesie’’, en W. Barner (Ed,.), Gotthold Ephraim Lessing - Weke und Briefe, 5/2, pp. 11-321. Frankfurt a. Main: Deutscher Klassiker Verlag.
LEVINE, P. & SCOLLON, R. (Eds.) (2004). Discourse & technology. Multimodal discourse analysis. Georgetown, Washington, D.C.: Georgetown University Press.
LIM, V.F. (en prensa). ‘‘Problematising ‘semiotic resource’ ’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 51- 63. Amsterdam & Philadelphia: John Benjamins.
LUTZ, B. (1998). ‘‘Hypertext: das Medium und der Autor. Ein Erfhrungsbericht’’, en B. Ketteman, M. Stegu, & H. Stöckl (Eds.), Mediendiskurse. VerbalWorkshop Graz 1996, pp. 151-162. Frankfurt am Main: Peter Lang Verlag.
MAZAL, O. (1985). Das BUch der Initialen. Wein: Edition Tusch.
MCNEILL, D. (1992). Hand and mind. What gestures reveal about thought. Chicago: University of Chicago Press.
MCNEILL, D. (Ed.) (2001). Language and gesture. Cambridge: Cambridge University Press.
MUNTIGL, P. (en prensa). ‘‘Modelling multiple semiotic systems’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 31-59. Amsterdam & Philadelphia: John Benjamins.
MURRAY, G. & COSTANZO, T. (1999) ‘‘Usability and the web: An overview’’, en Network Notes 61. Information Technology Services. National Library of Canada. http://www.collectionscanada.ca/9/1/pl-260-e.html
NIJHOLT, A. (2003). ‘‘Multimodality and ambient intelligence’’, en W.F.J. Verhaegh, E.H.L. Aarts & J. Korst (Eds.), Algorithms and ambient
intelligence. Dordrecht et al.: Kluwer Academic Publishers. (Disponible en: http//doc.utwente.nl/fid/1144).
O’HALLORAN, K. (2004a). ‘‘Visual semiosis in film’’, en K. O’Halloran (Eds.), Multimodal discourse analysis: Systemic functional perspectives, pp. 109-130. London: Continuum.
O’HALLORAN, K. (Ed.) (2004b). Multimodal discourse analysis: Systemic functional perspectives. London: Coninuum.
O’TOOLE, M. (1994). ‘‘Opera Ludentes. The Sydney Opera House at work and play’’, en K. O’Halloran (Ed.l), Multimodal discourse analysis: Systemic functional perspectives, pp. 11-27. London: Continuum.
PANG, A. (2004). ‘‘Making history in ‘From Colony to Nation’’: A multimodal analysis of a museum exhibition in Singapore’, en K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives, pp. 28-54. London: Continuum.
RAISAMO, R. (1999). Multimodal human-computer interaction: a constructive and empirical study. Tesis doctoral no publicada. Department of Computer Science, University of Tampere. http://acta.uta.fi/pdf/951-44-4701-6.pdf
RHEINDORF, M. (en prensa). ‘‘The multimple modes of dirty dancing: a cultural studies approach to multimodal discourse analysis’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.). Amsterdam & Philadelphia: John Benjamins.
SAJOR, M. (2002). Multimodality: The coming revolution. Wireless Future Magazine, September/Octber 2002. http://www.wirelessfuturemagazine.com
SCHILLER, F. (1793-1962). ‘‘Über das Pathetische’’, en B. von Wiese (Ed.), Schillers Werke, 20 - Philosophische Schriften, 1. Teil, pp. 196-221. Weimar: Böhlau.
SCOLLON, R. & LEVINE, P. (2004). ‘‘Multimodal discourse analysis as the confluence of discourse and technology’’, en P. Levine & R. Scollon (Eds.), Discourse & technology. multimodal discourse analysis, pp. 1-6. Georgetown, Washington, D.C.: Georgetown University Press.
STÖCKL, H. (1998). ‘‘Bild-Text-Verknüpfung’’, en B. Kettemann, M. Stegu & H. Stöckl (Eds.), Mediendiskurse. Vergal-Workshop Graz 1996, pp. 73-92. Frankfurt am Main; Peter Lang Verlag STÖCKL, H. (en prensa). ‘‘In between modes: Language and image in printed media’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 9-30. Amsterdam & Philadelphia: John Benjamins.
TAYLOR, C. (2000). ‘‘Text analysis and translation: An ineractive, self-access computer application incorporating a functional approach’’, en A. Baldry
(Ed.), Multimodality and multimediality in the distant learning age, pp. 295- 309. Compopasso: Palladino Editore.
TAYLOR, C. (en prensa). ‘‘Multimodal text analysis and subtitling’’, en E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality, pp. 153-172. Amsterdam & Philadelphia: John Benjamins.
THIBAULT, P. (2000). ‘‘The multimodal transcription of a television advertisement: Theory and practice’’, en A. Baldry (Ed.), Multimodality and multimediality in the distant learning age, pp. 311-385. Campopasso: Palladino Editore.
USABILITY.GOV (2004). Methods for designing usable web-sites. Collecting, writing and revising content. US Department of Health and Human Services (Ed.). http://www.usability.gov/methods
VAN DER GEEST, T. (2001). Web site design in communication design. Amsterdam: Benjamins.
VAN DER GEEST, T. & SPYRIDAKIS, J. (2000). ‘‘Developing heuristics for web communication’’, en Technical Communication 3, pp. 301-310.
VAN LEEUWEN, T. (1999). Speech, music, sound. London: Macmillan.
VAN LEEUWEN, T. (en prensa). Introducing social semiotics. An introductory textbook. London: Routledge.
VENTOLA, E. & KALTENBACHER, M. (2003). ‘‘Lexicogrammar and language teaching materials - A social semiotic and discourse perspective’’, en J.E. Joyce (Ed.), Grammar in the language classroom, pp. 158-201. Singapore: SEAMEO Regional Language Centre.
VENTOLA, E., CHARLES, C. & KALTENBACHER, M. (Eds.). (en prensa). Perspectives on multimodality. Amsterdam: Benjamins.
VERGILIUS, P. M. (1988). ‘‘Aenis’’, en J. Götte (Ed.), München & Zürich: Artemis Verlag.
WILLIAMS, T. (2000). ‘‘Guidelines for designing and evaluation the display of information on the web’’ en Technical Communication, 3, pp. 383-396.
WINCKELMANN, J.J. (1755-1969). Gedanken über die Nachahmung der griechischen Werke in der Malerei und Bildhauerkunst. Stuttgart: Reclam.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The authors retain the copyright and guarantee RALED the right to be the first publication of the work as well as a Creative Commons Attribution License that allows others to share the work with recognition of authorship and the initial publication in this journal.
