This is an essay I wrote for my MA TESOL module on Issues in EAP. Unfortunately, it did not score well. It did not meet the requirement of the University (Nottingham) of links the theory to practice. I wonder why I was not aware of this – I think during my previous MA the criteria were different and I had not adapted. Also, I admit getting carried away with the theory – I enjoyed reading about the development of discourse (Bazerman) and exploring Bakhtin and looking at Saussere. I also tried non-standard style with the opening of quotes that (I felt) encaptured the perspective on the topic which were then elucidated throughout the essay.
But hey, the essay is about the risks of deviating from standard discourse – in some way, scoring only 53 epitomised the risk being elaborated upon – expressing meaning in a different way may make it harder to be understood.
Anyway, by putting it here, perhaps it will be useful to someone if they stumble upon it and read it.
Academic Discourse Deviation: risks, opportunities and consequences of multimodality.
“Multimodal, dialogic texts contest the primacy of the essayist, monologic approach to writing … in HE”
“Digital academic texts have the potential to disrupt our ways of making arguments and describing ideas.”
Lillis (2003) suggests multimodal and dialogic forms of meaning making offer an alternative to the standard monologic, text-based academic discourse. McKenna (2015) suggests that differences may relate to ‘ways of making arguments and describing ideas’. ‘Academics are increasingly turning to digital spaces’ (McKenna 2015) and the affordances of technology have reduced the primacy of text-based knowledge representation through language; other forms, especially image, are becoming increasingly common and are found in combination with other forms. The consequences of changes in meaning representation from, for example, text to image or image with text to moving image (video) with text are important as:
“A semiotic is hard pressed to provide an unproblematic transparent and ‘direct’ translation for meaning made in another semiotic.”
Latour, 1992 (in Idema 2003)
New ways of making meaning and representation are available through multimodality but knowledge created using a different semiotics would inevitably have a different meaning.
Structure of essay
This essay seeks to identify what the consequences of adapting multimodality within academic discourse are and to elucidate what the key considerations for choosing to utilize multimodal opportunities are from the perspective of the scientist, researcher or student. The argumentation is as follows: first the setting of the analysis in an academic literacies perspective is introduced. This essentially sees writing as a social activity. Next, ‘multimodality’ and its roots in Hallydian social semiotics and systemic functional grammar are covered briefly and linked to more recent concepts such as ‘multimodal discourse analysis’, intersemiotic complementarity (Royce 2007) and resemiotization (Idema 2003).
Academic discourse is described in terms of the historical development of the genre with particular reference to Bazerman’s analysis of the ‘scientific report’ published in 1988. The academic research paper is considered as an ‘utterance’ in the Bakhtinian sense and acknowledges a ‘dialogic’ form of meaning making.
These aspects are related to the reader’s schema of the genre. Discourse deviation (Cook 1994) is understood as challenging the schemata of the reader.
The possible consequences of changing the mode from still image to moving image, to represent Event Related Potentials, are analysed as an example. Some generalizations of the risks and opportunities are made and the main considerations for a student, researcher or scientist are discussed.
- Academic Literacies
Prior to Academic Literacies (AL) the framework for assessing and developing student academic writing was deemed insufficient. The prevailing approach reflected a ‘normative attitude that discourses (of …) are not open to negotiation or criticism’ (Canagarajah 2002). AL rejects the notion that suppression of alternative discourses is necessary and beneficial to the academic community. It recognizes that writing is a social process: ‘based in social practice, historical experience and interests’. AL argues ‘vernacular of L1 may enhance expression’ (Canagarajah 2002). Furthermore, it is an approach to writing that allows for the representation of individual identity representation and with it, more flexible discourse is allowable.
1.1 Examples of progressive multimodal academic discourse
Kairos (kairos.technorhetoric.net) is an online journal that publishes peer-reviewed articles. Other forms of meaning making being experimented include the ‘intertext’ by Colleen McKenna (2015). This prezi (more commonly used for presentations as an alternative to using PowerPoint) covers the same topic area as the text-based version published in Lillis et al. ‘Working with Academic Literacies: Case Studies Towards Transformative Practice’, 2015. Another notable publication was the PhD thesis produced entirely in comic book format by Nick Sousanis, defended in 2014 (Sousanis 2015).
- Multimodality – opportunities
The term ‘communicative competence’ was introduced in 1972 by Hymes (in Royce 2007). He stated that people need to understand how the community of speakers uses the language to be able to communicate effectively.
Savignon (1972) (in Royce 2007) articulated the idea of communicative competence as a target for second language learners:
“Ability to function in a truly communicative setting – that is, in a dynamic exchange in which linguistic competence must adapt itself to the total informational input, both linguistic and paralinguistic”.
‘Total informational input’ introduces the notion of dealing with different forms of integrated meaning receptively and being able to combine various modes, linguistic and other nonverbal forms, to express meaning. This reverberates strongly with the notion of multimodal communicative competence.
However, the starting point to describe the concept of multimodality often refers back to Hallyidian social semiotics (Halliday 1978) (for example as introduced in Idema (2003), Camciottoli & Fortanet-Gomez (2015). He stated that language is functional, semantic (can make meaning), semiotic (making meaning meanings from the total possible options, and meanings ‘generated and exchanged are motivated by their social and cultural contexts.
Royce summarises (2007):
“This social semiotic view of communication implies that whether a text contains only verbal, or both verbal and visual modes, it can and should be viewed as embodying the pattern of purposeful choices made by it’s constructors in order to make meanings for others it receive and respond to in some way.”
2.1 Inter-semiotic complementarity and resemiotization
To express meaning with multiple modes, an author needs to consider how different modes create meaning together and how meaning changes when one mode is used instead of an alternative. For example, words and a visual image can produce meaning together as ‘one domain is mapped onto the other’. This follows ‘conceptual metaphor theory’ (Lakoff and Johnson 1999). This form of analysis considers that the one form (visual) could be more dominant than the other (verbal) or vice-versa, or the meaning could be complimentary.
Choices should take into consideration the gains and losses of using a particular mode: ‘potential of different semiotics’, what they ‘enable and/or constrain’ (Kress & van Leeuwen 1996). Regarding using language:
“Natural language contains no continuum of degree.”
“The natural world is about kinds and degrees.”
“Representation of the world requires a combination (of modes).” Lemke, 2004
Lemke illustrates the very necessity of multimodal expression to represent the world in a meaningful way. Language categorizes objects and things. Modifiers can be used to give more precise meaning but there cannot be a continuum of degree. Language forces categorization upon descriptions of reality that are then understood, shared and become accepted as accurate representations of reality.
In some cases, an alternative mode would offer an alternative representation of reality and facilitate the production of research in that area using a similar mode of representation to share research findings. Yu & Owyong (2011) investigated the use of symbols (for example ‘CO2’) in chemistry. It is argued that once fixed, this symbolic representation helped chemistry develop as a paradigm (Yu & Owyong 2011).
- Discourse deviation – risks
Multimodality is about representing knowledge in different ways that can facilitate, inhibit and direct future knowledge creation.
Bazerman (1988) explored how the nature of the experimental research paper from its beginnings in 1665 to the 1980s. The development of the research is explained in terms of the changing expectations of the scientific community. The findings within the reports are ‘socially legitimized’ in ‘socially orgainsed forms’ and may or may not be ‘accepted into the codified literature’ (Bazerman 1988).
Bakhtin in ‘The problem of Speech Genres’ notes that a genre is a relatively stable type of ‘utterance’ linked to a particular sphere of communication. The ideas expressed within an utterance are a chain of expression full of dialogic meaning making. All utterances, even when adhering to a genre contain elements of individual style (Bakhtin 1986). This lends support to using alternative modes of representation and thus evolving a discourse. But as noted: ‘not all genres are conducive to individual style.’
Kress (2003) highlights ‘the broad move from the now centuries-long dominance of text to the medium of the image’. Kress notes that people from different cultures may interpret the images differently. This can be viewed through the application of different schema that is used to project an expected structure onto a social event (Whitney 2001). These schemata also carry shared assumptions and understandings (content schema) as well as rhetorical schema (Gurkan 2012). ‘Readers interpret the meaning of a text through (own) shared values, customs and assumptions (Gurkan 2012). Multimodality offers ways to express meaning that may be unfamiliar; the audience schemata may not be activated. This poses a risk for the writer who wishes to place the new idea within a set of shared knowledge. The greater the discourse deviation (Cook 1994) the greater the risk that the idea will not be understood. If the unfamiliar can be understood then meaning is transferred, if not, then ‘confusion’ is the likely result and possibly rejection of the validity of the idea (Stockwell 2006).
- Using moving image
When there is a need to express meaning with a ‘continuum of degree’ the use of image should be welcomed as the categorisations (and therefore approximations) that take place when language is used, distort reality when represented and reality is distorted again in more unpredictable ways – by the reader.
Moving image offers a way to present changes through time on a continuum of degree. Moving image could not only be a video or model of reality changing in real-time or slowed-down / speeded up depending on the time scale of the observation but also of the correlations between variables. A third variable, often ‘time’ can be represented by movement. This resemiotization changes the meaning making process from one of looking at multiple graphs representing different points in time to simply observing one. This could have significant consequences beyond simply concerned with using image. Firstly, multiple variables can be presented and relationships between them shared easily. The continuum of degree is significant when the additional variable is time. Looking at multiple graphs required the reader to remember the previous graph and relate the next graph to it. The discrete points on time measure moments in time rather then giving the complete view of what occurred.
Therefore, utilizing multimodality to represent multiple modes simultaneously without the need to use language to describe the interaction between the modes could better represent an accurate version of events to another interested scientist. The process of knowledge sharing could be enhanced; ideas can be validated or invalidated through replication more easily. The area of research investigated here points to the end of the rigid discourse of the behaviourist psychologist’s report, as suggested by Bazerman (1988). Investigating brain activity opened up a new way to understand psychology that has led to a different form of empirical experiment.
4.1 Event Related Potentials (ERPs)
Brain scanning technology measuring Event Related Potential (ERPs) has become widespread. These are represented in through a variety of graphics in research papers. The spatio-temporal nature of the information presents a challenge. The ERP readings occur at different points on a map of the brain and also at different points in time.
The graphic Fig 1 below shows the level of ERP at 3 points in time. The colours are used to represent the levels of ERP taken from scalp surface measurements. However, this is represents a continuous change over time. The interpreter of the information would need to conceptualize the continuous change by inferring the changes from one time frame to the next. Also, the time frames are already covering increasingly longer periods of time: the image on the left 100ms, then 200ms and then 300ms. The conceptualizing of the actual event becomes increasingly difficult.
Fig 1. From Cohn et al. (2012), “Voltage maps illustrate the differences across the scalp surface of ERPs evoked by panels in Structural Only minus Semantic Only sequences (top row), and Semantic Only minus Structural Only sequences (bottom row) at the 300–400 ms (N300), 400–600 ms (N400) and 600–900 (late negativity) time windows.”
A moving image of the two cases could assist the interpreter to some extent. The meaning represented would have greater ‘semiotic closeness’. Creating a moving image that allows the reader to control the timescale could be created and embedded into the research report [for example, in Miller 2016, published in Kairos].
Inferencing from schema is not eradicated; it is transferred from the readers, applying individual and varied schemata to make inferences, to the researcher (the writer). The original recordings are from points in time and are not continuous. A moving image created from the measurements taken would require inference of missing data in order to create a moving image. There is a risk of presenting an event that is an inaccurate representation of the data recorded.
There are already concerns over the validity of ERP related studies:
“Neuroimaging is definitely the hottest topic in cognitive science. If you produce a study that is accompanied by at least one high-resolution scan of some aspect of the brain, preferably also with some parts of it lit up (and all of it, of course, in vivid colours!), people will know that this is serious matter.”
ERP research suffers from two serious issues: no one is sure exactly what causes the ERP measurements, ‘what N400 actually indexes’ (Kutas, VanPattern and Kluender 2006) and that identical patterns of ERP activity does not necessarily mean the same neural processing took place.
He mentions the use of ‘vivid colours’ as a way to increase the validity of the findings. Here, he is not only referring to how technological advances enable areas of research to become possible, he is noting the effects of mode of representation on perceived validity of research. The use of moving image could further attract more research into an area that measures and represents something (ERP) that is not even fully understood.
Language is now losing its place of prominence as a way of communicating meaning. Technology now allows the creator of the text to utilize the various modes available. Perhaps, with the advent of multimodality, language should and other forms of meaning should be studied under the umbrella of ‘Semiotics’ originally put forth over 100 years ago by Saussure (lectures noting ‘semiotics’ were presented to students from 1906-1911). However, given the primacy of text (language) over the intervening 100 years the study of language has been most fully developed. Multimodality research has started to address the in-balance and study different semiotic systems and importantly the interaction effects of using different modes together.
The effects of these changes can be seen in terms of gains and losses, opportunities and risks. Knowledge creation and sharing is affected by changing modes and historical developments of discourse indicate that these changes have consequences on areas of research pursued.
The more immediate consequences on the individual scientist, researcher or student relate to making appropriate choices. It is important to understand how the discourse operates to facilitate the communication of ideas. A discourse is never fixed and any deviation from the standard discourse is possible but any deviation is a risk. The research findings may not be understood in a consistent way – any change in mode, changes the meaning understood. Meanings become fixed and accepted overtime. The meanings understood and shared by a community are settled upon through a dialogic negotiation (Bakhtin 1986). Assuming multimodal offerings (including moving image and sound) become the norm after sometime, the complimentary meaning created by using the modes in conjunction with each other cannot fully be predicted. The meaning created through ‘juxtaposition’ for example can only be stated with certainty after the negotiation has taken place. If the value of this element causes unpredictable meaning outcomes, it may become the norm to ignore or avoid using such an element all together. In this sense, declaring the meaning created by intersemiosis is premature. The scientist sharing his research should not only pay attention to the gains and losses of each mode but also be aware that multimodal adoption of the kind published on the online journal Kairos, is still in the early days of the process. Almost all academic papers submitted for publication are still produced on a page, are written within the margins of a page despite the near absence of need to actually print it onto paper.
Furthermore, the community of scientists is now international, with researchers coming from varied backgrounds. The community may ultimately settle on a meaning that is prevalent in one culture over another. For example, Kress (2003) speaks of a culture that is familiar with a writing system from left to right to interpret the juxtaposition of image to text in a different way to an alternative writing system. An image on the right of the text may be considered as being more prominent and important if the reader’s native writing system is written from right to left as opposed to left to right. The eventual meaning settled upon (if any) would depend on the community itself. Meanings of elements are not directly imposed upon a community of users (Saussure).
Finally, the writer would need to consider their own identity within the community of scientists and academics that is their audience. Using moving image in a published research paper that is not the accepted form of meaning representation could for example, be not only more difficult to understand, but risk the researcher being seen as an outsider. Being the pioneer within the field may assist credibility in the field or it may cause the research to be respected by some disciplines that have long since agreed on a fixed discourse. It may be worth noting that those in the community that a novice researcher wishes to impress may have more conservative ideas regarding the forms of acceptable meaning presentation for example, multiple annotated stills to present gesture versus embedding the actual three-second video of a gesture taking place.
5.1 Other constraints on multimodality
There are other constraints that have not been considered fully in this paper. Two constraints worth a brief mention are: the value institutions place on paper-based records and access.
A new university library completed recently in 2009 for Georgia Southern University boosted enough storage space for 70 years collection growth capacity (Lilli 2013). It was expected that the publication would be in duplicate form – one paper-based version as well as an online (‘e-paper’) version (Lili 2013). This predicts that in 2079, academic papers will still be restricted to modes within the margins of the page. I personally feel that this university library will never use up all the available space.
Universities all around the world may have Internet access although in some places the connectivity speed may be slow and intermittent. This may mean that printable versions are still important for readers in these regions for sometime to come. However, in terms of the university in the developing world with tight budgets, when making a purchasing decision regarding buying an online version or paper-based versions or both, it is likely that buying an online version would be cheaper, even if the reduced cost is solely related to reduced storage costs and transportation costs. Increased accessibility due to cost concerns may in fact encourage the proliferation of publication solely for the online space if problems related to Internet connectivity are overcome.
This essay has attempted to identify the consequences of adapting multimodality within academic discourse and to elucidate key considerations for choosing mode with respect to the meaning created and represented, from the perspective of the scientist, researcher or student.
Bakhtin M., (1986), Speech Genres and Other Late Essays, translated by McGhee V., University of Texas Press.
Bazerman C., (2000), Shaping Written Knowledge: The Genre and Activity of the Experimental Article in Science, WAC Clearing House, Landmark Publications in Writing Studies: http://wac.colostate.edu/books/bazerman_shaping/ Originally published in Print 1988, by University of Wisconsin Press, Madison, Wisconsin.
Camiciottoli C., Fortanet-Gomez I., (Eds.) (2015), Multimodal Analysis in Academic Settings: From Research to Teaching, Routledge.
Canagarajah S., (2002), Multilingual Writers and the Academic Community: towards a critical relationship, Journal for Academic Purposes, 1, p29-44.
Cohn N. et al., (2012), (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension, Cognitive Psychology, 65, 1-38.
Cook G., (1994), Discourse and Literature, Oxford, Oxford University Press.
Dörnyei, Z. (2009), The Psychology of Second Language Acquisition, Oxford, Oxford University Press.
Gurkan S., (2012), The Effects of Cultural Familiarity and Reading Activities on L2 Reading Comprehension, 3rd Int. Conference on New Horizons in Education INTE, 55, p1196-1206.
Halliday, M.A.K., (1978), Language as Social Semiotic: The Social Interpretation of Language and Meaning, Edward Arnold, London.
Idema R., (2003) Multimodality, resemiotization, extending the analysis of discourse as multi-semiotic practice, Visual Communication, 2(29).
Kress, G., van Leeuwen, T., (1996), Reading Images: The Grammar of Visual Design. Routledge.
Kress, G., (2003), Literacy in the New Media Age, London: Routledge.
Kutas, M., C. K. VanPatten, and R. Kluender (2006), ‘Psycholinguistics electrified II (1994–2005)’ in M. J. Traxler and M. A. Gernsbacher (eds.): Handbook of Psycholinguistics, 2nd edn. London: Academic Press, 659–724.
Lemke J., The Literacies of Science. In Saul E., (Ed), (2004), Crossing Borders in Literacy and Science Instruction: Perspectives on Theory and Practice, International Reading Association.
Lili L., (2013) The Future of Academic Libraries in the Digital Age. In Trends, Discovery and People in the Digital Age, p253-268.
Lillis, T. (2003). Student writing as “academic literacies”: Drawing on Bakhtin to move from critique to design. Language and Education, 17(3), 192-207.
Liu Y., Owyong Y., (2011), Metaphor, multiplicative meaning and the semiotic construction of scientific knowledge, Language Sciences, 33(5), p822-834.
McKenna C., (2015), Digital Writing as Transformative: instantiating academic literacies in theory and practice. In Lillis et al., Working with Academic Literacies: Case Studies Towards Transformative Practice. Perspectives on Writing. Fort Collins, Colorado: The WAC Clearinghouse and Parlor Press. Available at http://wac.colostate.edu/books/lillis/
And accompanying ‘intertext’ prezi located here: https://prezi.com/ux2fxamh1uno/digital-writing-as-transformative/
Miller B., et al. (2016), The Roots of an Academic Genealogy: Composing the Writing Studies Tree, Kairos: A Journal of Rhetoric, Technology and Pedagogy, 20(2). Available at: http://kairos.technorhetoric.net/20.2/topoi/mill-et-al/future-directions.html
Royce T., Bowcher W. (Eds.), (2007), New Directions in the Analysis of Multimodal Discourse, Lawrence Erbaum Associates, Inc. Chp. 12 Royce T., Multimodal Communicative Competence in Second Language Contexts
Saussere, F. de. (1983), Course in General Linguistics. Tr. Roy Harris, London: Duckworth.
Sousannis N., (2015), Unflattening, Cambridge, MA, Harvard University Press.
Stockwell P., (2006), Schema Theory: Stylistic Applications, Encyclopedia of Language and Linguistics (Second Edition), p8-13.
Whitney P., (2001), Schemas, Frames and Scripts in Cognitive Psychology, International Encyclopedia of the Social and Behavioural Sciences, p13522-13526.