The complexity of the distinctive discourse of science has long been recognized as a challenge to learning for many students and particularly those from low socio-economic backgrounds and/or those whose first language is not the official language of instruction (Nunes et al., 2017; Teese, 2013). There has been a prodigious amount of research into the complexity of scientific discourse and its accessibility to students. Arguably the most educationally influential research has emerged from systemic functional linguistics (SFL) through work on language led by Halliday (Halliday & Martin, 1993), Martin (Martin, 2017; Martin & Veel, 1998) and Lemke (1990, 2004). Significant in this work from the perspective of student negotiation of scientific discourse has been the description of the linguistic construction of technicality and the textual resources for the aggregation of meaning indicating how this distinguishes scientific from everyday discourse about phenomena, thus identifying the linguistic nature of the complexity confronting students (Martin, 2020).
The complexity of scientific images and of the composite image-language ensembles that confront science students has not yet been similarly specifically delineated. Seminal studies of the meaning-making resources of images deriving from or related to the systemic functional linguistic tradition that have included images from science have given little attention to the verbiage that frequently occurs within such images. Science images in these studies have not been discussed in the context of the discourse of science and the relative ideational complexity of such images has not been addressed (Bateman, 2008; Höllerer et al., 2019; Kress & van Leeuwen, 2021). In their seminal work on the grammar of visual design Kress and van Leeuwen (2006) emphasized single structure images (Doran, 2019). Kress and van Leeuwen (2006) described the depiction of ideational meanings in terms of ‘narrative structures’ (which construe activity), classificational structures (which construe class-subclass relations), analytical structures (which principally construe compositional relations) and symbolic structures. It is unclear in the Kress and van Leeuwen approach whether images can contain, and accord the same status to, more than one of these structures (Doran, 2019). Kress and van Leeuwen (2006) do indicate that timelines seem to occupy an intermediate position between the narrative and the analytical and that genealogies and evolutionary trees and graphs may blur the boundaries between the static and the dynamic (Kress & van Leeuwen, 2006, pp.101-103). They also suggest that ‘embedding’ can occur in diagrams and provide one example where a narrative structure is embedded within an analytical structure (p. 51). But certainly, the overwhelming emphasis in their work is on single structure images. This seems to have influenced many subsequent pedagogically oriented publications using their approach which also limit their discussion to single structure images (Callow, 2013; Callow, 1999; de Silva Joyce & Gaudin, 2007; Knain, 2015). However, deploying multiple structures in a single image is a frequent and significant resource in science research and education, one that contributes substantially to the complexity of multimodal scientific discourse, and one that students increasingly need to negotiate as they progress through high school. The nature of the complexity of visualizations and the verbiage incorporated within them has been broached in very few of the substantial number of systemic functional semiotic studies of school science discourse (Doran, 2017, 2019; Martin, 2020; Unsworth, 2020; Unsworth & Cleirigh, 2009a, 2009b). Doran and Martin described the representation of phenomena in these multimodal assemblages in science texts from the perspective of field, which is a resource for construing phenomena as activities alongside the taxonomies of items involved in these sequences (organised by both classification and composition), along with associated properties (Doran & Martin, 2021). They pointed out that these multimodal assemblages frequently simultaneously represent two or more of these aspects of field and that the images combined with annotations and interpolated text blocks aggregate meaning in highly condensed synoptic portrayals, all of which contributes to the nature of the complexity students need to negotiate (Doran, 2019; Martin, 2020).
This paper outlines ongoing research building on these studies to develop a framework describing options for the co-articulation of image and language and how this aggregates the ideational meanings construing field in the infographics that high school science students need to interpret and to create (Martin & Unsworth, forthcoming; Martin, Unsworth & Rose, in press). The emerging framework is used firstly to investigate variation among textbook infographics in their deployment of image-language co-articulation in the multimodal representation of complex scientific phenomena such as mitosis in cell DNA replication. Secondly, infographics constructed by senior high school students in response to examination questions are examined to investigate the relationship between their co-articulation of image-language resources and the quality of their responses.
Theoretical Framework
Infographics in School Science
Over nearly two decades now research has documented the dominance of the image as the rhetorical locus of ‘trade’ and textbooks in science (Bateman, 2008; Bezemer & Kress, 2010; Danielsson & Selander, 2016; Kress, 2005; Peterson, 2016) to the extent that in some cases traditional ‘running text’ has been elided in favour of image-based portrayals that may include annotations, interpolated text blocks and a caption (Danielsson & Selander, 2016; Martin & Rose, 2012). Such multimodal ensembles are also increasingly characterizing the communication of science on websites and in the popular press as well as in brochures and other publications by government and semi-government authorities (Polman & Gebre, 2015), which has inspired science educators to adopt the creation of infographics, as a learning experience for students (Gebre & Polman, 2016; Ozdamli & Ozdal, 2018; Walsh & McGowan, 2017). However, the use of multimodal student created representations linking image and language has been increasingly advocated in research over many years as a core aspect of student enquiry to enhance science learning, albeit that this has occurred predominantly in the primary (elementary) and junior secondary school years (McDermott & Hand, 2013; Tytler, Prain, & Hubber, 2018). Nevertheless, this kind of image-prominent combination of various forms of visualizations and verbiage also typifies the expected form of student constructed responses to many of the short answer questions in senior high school final year science examinations in countries like the United Kingdom, Singapore, Australia and New Zealand. Notwithstanding the increasing prominence of infographic portrayals in textbooks and other learning materials, and the reliance of the multimodal representations that students are encouraged to construct on the effective co-articulation of image and language, little research attention has been given to how these image and language components of infographics combine in the construction of meaning to inform the development of students’ critical learning to read from such representations and the development of their effective creation of science infographics (McDermott & Hand, 2013, 2016; Polman & Gebre, 2015).
Deployment of Image and Language in Infographic Design
Recent and ongoing semiotic research analysing infographics in high school science textbooks and related websites has started to document the repertoire of options for ways in which images and language are deployed in these multimodal representations (Martin & Unsworth, forthcoming; Martin et al., in press; Unsworth, 2020). The image component of infographics can consist of one or more of the several types of images such as photographs, drawings, diagrams, graphs etc., or a combination of the different image types. Figure 1 shows examples of high school science textbook infographics depicting the components and functioning of the ear (Bennett, 2017, p. 108) and the phases of mitosis in the replication of cell DNA (Chidrawi et al., 2013, p.19).
|
|
Figure 1. Science Textbook InfographicsSource: Rickard, G. (2017) & Chidrawi et al. (2013). |
The verbiage components can include annotations and/or co-text, which can consist of a caption and/or one or more interpolated text blocks. Examples of interpolated text blocks can be seen in the infographic on the greenhouse effect in Figure 3. The co-text as caption or interpolated text block relates to the image as a whole, whereas annotations relate to specific parts of the image. In Figure 1, the annotations refer to particular parts of the image as specified by the connecting lines. The captions (located below the image of the ear and to the top and right of the diagram of mitosis) refer to the image as a whole.
A simplified representation of options for the deployment of images and verbiage in infographics and for the aggregation of ideational meaning construing field is indicated in Figure 2.
|
|
Figure 2. Infographics and AggregationSource: Prepared by the author. |
The left facing brace from ‘infographic’ in Figure 2 indicates that infographics consist of at least one image as well as verbiage. The left facing brace from ‘image’ along with the square brackets indicates that the image component may consist of only one of the image types or any combination of any number of the image types. The left facing brace from ‘verbiage’ along with the square bracket indicates that the verbiage component may consist of co-text or annotation or both, and similarly the co-text may consist of a caption or interpolated text or both. Here we will interpret infographics from the perspective of their representation of meanings construing field, following Doran and Martin (2021). As we have noted, infographics may construe one or more of the dimensions of field including activity, classification and composition of entities and properties of activity or entities, and these may be represented in images only, verbiage only, or in combinations of image and verbiage. Part of the complexity of some infographics is their representation of two or more of these field dimensions simultaneously and with the same status. This aspect of complexity we refer to as aggregation – the extent to which activity, classification, composition and properties are combined in an infographic.
Aggregation through Accumulation and Integration
The system for aggregation in Figure 2 interfaces the options for inclusion of the technicality construing field (composition, classification, activity and property) with two options for combining those meanings in infographics: (1) accumulation; (2) integration.
Accumulation refers to aggregation of different types of meaning (composition, classification, activity and property) from two or more macro-groups, each consisting of an image +/- annotations and +/- interpolated text. Integration refers to aggregation within a macro-group through (a) incorporation of two or more of the dimensions of activity, classification, composition and property within one image; (b) representation of different dimensions of meaning construing field in the image and in the annotations; (c) depiction of activity as a verbal representation of entity and a visual representation of an action process. Aggregation in infographics may involve either or both of the options of accumulation and integration to a greater or lesser extent, hence in Figure 2 these options are represented as a cline.
Aggregation through accumulation is exemplified in Figure 3 in infographics portraying the greenhouse effect (Lofts & Evergreen, 2015, p. 228) and the conduction of nerve cell impulses in the axon (Huxley & Walter, 2019, p. 260).
In the greenhouse effect portrayal in Figure 3 both image macro-groups represent the activity of the energy radiated from the sun to the earth and the reflection of some radiation from the earth’s surface only in the images. The image on the right-hand side also includes some composition of the continent showing entities of a house, factory, tower block, car and cattle with some activity in the form of emissions from the first four entities. But it is the combination of the two images that adds the comparative classification of the environment before and after the introduction of agrarian and industrial development. The aggregation of meaning across two macro-groups can also be seen in the infographic about the conduction of nerve impulses in the axon in Figure 3. The picture macro-group of the axon at the top depicts the essential compositional meanings. The soma is an expanded portion of cytoplasm that contains the nucleus (the nucleus is shown as a darker circle, but not labelled). The elongated section extending to the right is the axon, which is the main pathway for the conduction of nerve impulses to the axon terminal. Other components of the neuron are depicted but not named. The second macro-group consisting of a thin black action line and below that three parallel horizontal bands each with an aligned annotation conveys the activity. The rows, read from top to bottom, indicate the progressive depolarizing the repolarizing of successive sections of the axon as the means for conducting the nerve impulse.
High school science textbooks are replete with infographics aggregating meaning by depicting two or more of the field dimensions of activity, classification composition and property within the same macro-group. This can occur within one image depiction. An area where this kind of aggregation of meaning is essential to the representational construal of the phenomena is that of biological processes such as mitosis, which is depicted in Figure 1. This infographic includes one diagram macro-group consisting of four rounded imagic micro-groups along with a number of verbal micro-groups as annotations (Martin & Unsworth, forthcoming; Martin et al., in press). Each of the micro-groups depicts compositional entities within the cell at the different phases of mitosis and the micro-groups are linked by arrows depicting progression from one phase to the next. The depiction of the changing compositional relations in the cell is interconnected with the depiction of the activity of cell transformation – through the Prophase, Metaphase, Anaphase and Telophase phases. This progressive activity is depicted partly by the arrows between the successive images of the cell. But each cell image also depicts the distinctive components of the cell at that phase. For example, from the Prophase to the Metaphase, the nuclear membrane and defined nucleus in the Prophase disappears; the chromosomes are re-configured to line up around the equator of the cell; the centrioles change position to the poles of the cell; and the spindle fibres change from clustering around the nuclear membrane to radiating from centrioles at both poles of the cell.
|
|
Figure 3 Aggregation through AccumulationSource: Lofts & Evergreen (2015) & Huxley & Walter (2019). |
This shows that while activity can be represented imagically by vectors (Kress & van Leeuwen, 2006), it can also be inferred from successive similar images when items either appear or disappear, or reappear in a different form (Painter, Martin, & Unsworth, 2013).
A second way in which aggregation can occur through integration is when one or more aspects of the field construal are conveyed by the image and one or more different aspects of the field construal are conveyed by the verbiage. In the infographic about the ear in Figure 1, the image depicts the composition of the ear only. There are no vectors (action lines) or other visual depiction of activity representing the functioning of the ear. The bolded headings of the annotations name the components of the ear. The main part of the annotation may extend the composition taxonomy by naming sub-components and/or indicating the function of the component. For example, the ossicles are composed of the hammer, anvil and stirrup, and the semi-circular canals ‘give us our sense of balance’. But the annotations can also realize activity as in ‘It (the outer ear) funnels sound into the ear canal’ and ‘Sound reaching the eardrum makes it vibrate’. The realizations of activity in the annotations can be as figures (clauses) such as ‘It (the outer ear) funnels sound into the ear canal’ or ‘Sound reaching the eardrum makes it vibrate’. But activity can also be realized by an activity entity (in the form of a noun group) such as ‘Vibrations of the eardrum’ in the annotation of the ossicles, which is a condensation of the construction of this activity as a figure or clause in the eardrum annotation (‘Sound reaching the eardrum makes it vibrate’). The annotations also construct properties such as the tightness of the skin stretched over the eardrum or the oval window as a ‘thin’ layer of tissue. The infographic about the ear does not include classification. Options for verbal designation of classification in annotations include specifying the superordinate, naming the sub-types and the criteria for categorization. Additional to the construal of these aspects of field, the verbiage can also realize causal relations. In Figure 1, for example, in the annotation on the cochlea, ‘Vibrations cause this fluid to move…’.
The third way in which aggregation occurs through integration is through meaning-making at the intersection of image and language. The journal limitation on the number of Figures does not permit illustration of this, however a clear example is provided in an infographic photosynthesis (Huxley & Walter, 2019, p.113). This representation of photosynthesis shows the intermodal realization of activity – with events of carbon dioxide absorption and oxygen discharge realized imagically by brown and blue arrows respectively and one of the participating entities (the leaf) also realized imagically, while the other participating entities (carbon dioxide and oxygen) are realized through verbiage.
Method
This qualitative study adopted a multimodal discourse analysis approach to investigate firstly, the relative extent to which the technicality of mitosis was represented in junior high school textbook infographics and the nature of the co-articulation of image and language in this representation, and secondly, how this image-language coarticulation was related to the quality student representations of the technicality of ionic bonding in infographic responses to an examination question. The following research questions are addressed:
- How can a framework for the co-articulation of image and language in aggregating ideational meaning be used to evaluate the relative complexity and accessibility of school science infographics?
- How is the quality of student constructed science infographics related to their co-articulation of image and language resources?
Corpus
The textbook corpus consists of eight high school textbooks which are commonly used in Australian biology classrooms in Year 10 and in Years 11 and 12 (Armstrong et al., 2019; Borger, 2015; Chidrawi, Davis, Farr, Lampman, Matchett, & Young, 2013; Huxley & Walter, 2019; Kinnear, 2016; Linstead, Clarke, Devline, Madden, Rickard, & Spenceley, 2012; Lofts & Evergreen, 2015; Silvester, 2016). The infographics dealing with mitosis were selected from each text. The four Year 10 texts were used for detailed comparative analysis.
The student constructed infographics were selected from a corpus of twenty, year 11 chemistry students’ responses to the following examination question dealing with ionic bonding:
When an experimenter adds 50g of solid sodium nitrate (NaNO3, an ionic salt) to 100ml room temperature water it will all readily dissolve. Explain how this occurs with reference to the bonding between particles. (use a diagram to aid your explanation.)
The student responses were rated by the teacher on a scale of one to five. One five-rated response and one two-rated response were selected for comparison.
Data Analysis
The infographics on mitosis from the eight textbooks in the corpus were analysed to determine the components of the cell and each moment (separate event) of the activity of mitosis represented in each infographic in the images and/or the language (Unsworth, 2020). Then all of the ideational meanings realized by the images and verbiage in the eight infographics construing cell components and the activity of mitosis were compiled to form a comprehensive composite mapping of the compositional relations and of the activity construed in all eight infographics taken together. The composite map of all cell components represented across the corpus texts comprised: Cell (cell walls, cytoplasm, plasma membrane); Nucleus (nucleolus, nuclear membrane, chromosomes (chromatids, centromere), DNA (chromatin), mitotic spindle (centrioles, microtubules/fibres)). The composite map of all activity in mitosis across the corpus texts comprised: Interphase (chromosomes replicate); Prophase (chromosomes condense/become visible, nucleolus disappears, nuclear membrane disappears, pairs of centrioles separate, microtubules/fibres extend from centrioles, spindle fibres attach to centromere, network of spindle fibres forms); Metaphase (chromosomes line up along the equator of the cell); Anaphase (spindle fibres/microtubules contract, each centromere divides, chromatids separate, chromatids move to opposite ends of the cell); Telophase (chromosomes uncoil and become less visible, nuclear membrane re-forms, spindle fibres disappear); Cytokinesis (cytoplasm divides, new cell membrane forms, two daughter cells form).
In the next step, for each of the four year ten texts, the extent of correspondence of the representation of cell components and the extent of correspondence of the momented activity (separate events) represented in the image(s) were mapped against the composites for composition and activity. A similar procedure was used to map the representation of components and activity in the verbiage in each year ten text (conflating the results for the verbiage as interpolated text, annotation and caption).
The student constructed infographic responses to the examination question on ionic bonding were compared in terms of the extent and effectiveness the students’ deployment of options for the use of image and language in representing and aggregating ideational meaning construing field in infographics, as summarized in Figure 2. The comparative analysis considered the relative use of accumulation and integration in the aggregation of meaning including the related selection of type and function of image(s) and the nature and extent of the use of annotation and text blocks as well as the co-articulation of image and language in combining and condensing meaning.
Results
Complexity and Accessibility of Year Ten Textbook Infographics on Mitosis
Three of the four year ten textbooks rely entirely on the annotated images and captions of their infographic portrayals to explain mitosis (Chidrawi et al., 2013; Lofts, 2015; Silvester, 2016). Each includes very minimal traditional running text, which does not explain the process, so only the infographics were analysed. The fourth year 10 textbook (P) is the only one in which the mitosis infographic is accompanied by running text that is an explanation of the process of mitosis (Linstead et al., 2012). This analysis indicated for each year ten text the extent to which composition and activity from the composite accounts was represented and how it was distributed across the modes of image and language. The results for the representation of cell components and the activity of mitosis are shown in Table 1.
Table 1
|
||||
|
Textbook Infographic |
O |
N |
J |
P |
|
Representation of Cell Composition |
||||
|
Image or Verbiage |
10 |
12 |
11 |
11 |
|
Image + Verbiage |
7 |
11 |
5 |
10 |
|
Verbiage only |
1 |
1 |
0 |
0 |
|
Image only |
2 |
0 |
6 |
1 |
|
Representation of Activity in Mitosis |
||||
|
Image or Verbiage |
11 |
16 |
11 |
15 |
|
Image + Verbiage |
8 |
11 |
3 |
9 |
|
Verbiage only |
1 |
4 |
2 |
5 |
|
Image only |
2 |
1 |
6 |
1 |
|
Source: Prepared by the author |
||||
Table 1 shows in the first row of the representation of cell composition, the number of compositional relations construed by either image or verbiage; the second row shows the number construed by both image and verbiage, followed by those construed by verbiage only and then by image only. While all texts include nearly the same number of cell components, clearly, the N and P texts both depict and name nearly all such components, as indicated in the second row. The intermodal construal of composition confirms deeper taxonomic relations in the N and P texts compared with O and J. The components in the former texts are also verbally identified, whereas, in text J in particular at least half of the components are only depicted in the image. As there is no accompanying running text explanation of mitosis, this means that student interpretation of and learning from text J relies heavily on augmentation of information from the teacher and/or alternative texts.
The most comprehensive and explicit construal of activity occurs in the N and P infographics, as shown in the first and second rows of Table 1 in the section on the representation of activity in mitosis. In these texts the only activity that is construed by image only is ‘spindle fibres disappear’. There is a good deal of concurrence in the construal of activity in image and verbiage, as shown in the second row, and the remainder are primarily construed explicitly in language. While the O infographic has less commitment to activity in either image or verbiage than the N and P texts as indicated in the first row, there is a good deal of concurrence in what is committed in O as shown in row two. The J infographic has the same overall level of commitment to activity as O, but little concurrence, with a significant proportion of the activity committed in images only. The pattern of representation of composition and activity is very similar in the J text.
In summary, all four texts represent similar depths of compositional taxonomic relations, but the N and P texts also name nearly all of the represented components. The N and P texts also provide the most extensive depiction of activity and with a good deal of concurrence between the image depiction and the verbal realization of the activity. The J text relies more on image depiction only for compositional relations and activity and hence its use is heavily dependent on augmentation through teacher explication and some form of additional notation and/or alternative learning materials.
Deployment of Infographic Resources in High and Low Scoring Student Responses
The response from a higher achieving student, Amri is compared with that of a lower achieving student, Sim, to show how Amri’s response more effectively deploys a greater range of the options for aggregating meaning in infographics. Noting the conceptual errors in Sim’s response, the attention here will focus on the more apposite choice of diagrams by Amri, as well as the more extensive and effective use of annotation for multiple purposes and the cohesion between the interpolated text blocks and the two diagram macro-groups. The infographic responses provided by Amri and Sim are shown in Figure 4.
The mention by both students of ion-dipole bonding suggests their understanding that when an ionic compound such as sodium chloride or sodium nitrate is added to water, the positive ends of the water molecules are attracted to the negatively charged chloride or nitrate ions and the negative ends of the water molecules are attracted to the positively charged sodium ions.
Amri’s diagrams albeit non-canonical, are two quite separate, highly abstract and decontextualized conceptual microscopic depictions of the processes of dissociation. The separation of the depictions of what happens with the cation and the anion in two discrete drawings emphasizes the focus on communicating a conceptual explanation. Both of these drawings include the representation of an ionic lattice for sodium nitrate, albeit in a minimalist highly schematic form with no detail, which may reflect the unfamiliarity of students with the complex nature of the sodium nitrate lattice. Each of these two diagram macro-groups represent both composition and activity. Aggregation through integration occurs in the image depiction of composition and activity. Composition includes the lattice and the depiction of the anion and cation as well as the accurate representation of the water molecules, with colour used to further distinguish the hydrogen and oxygen. In representing activity, the drawings emphasize the cation and anion being removed from the edge of the lattice and detailed depiction indicates the activity through the wavy lines implying movement between the cation and the oxygen end of the water molecule, in the left-hand diagram. In Amri’s infographic aggregation also occurs through accumulation as both diagram macro-groups contribute different aspects of composition and activity in the phenomenon of ionic bonding.
Sim’s diagram is one macro-group consisting of four drawings connected by arrows. Although also representing a microscopic view, the representation of the containers provides a more macroscopic contextualization. The drawings do include composition and an attempt to represent activity. A key limitation of Sim’s explanation is the omission of the ionic lattice in either the language or the image. The first of Sim’s drawings includes a fair representation of the water molecules. In the second drawing Sim attempts to represent the sodium nitrate in a similar fashion to the water molecule rather than as a lattice. In relation to activity, the lines above drawing two appear to represent sodium nitrate being put into the water – again a macroscopic rather than microscopic activity. Beyond that, the intention to represent activity is indicated by the arrows representing sequence between each drawing and there is clearly a change to what represents the sodium nitrate and water molecules from drawings two through four. Activity that might be inferred from drawing two to drawing three is that the previously represented water molecules appear to no longer be intact in drawing three and there appear to be some detached nitrate ions also in drawing three. However, the nature of the activity represented is unclear due to inaccurate compositional meaning depicted. Drawing three, for example, seems to have several depictions of what appears to be one oxygen atom connected to one hydrogen atom. It is not clear what is being represented by the blue and green circular depictions in the fourth diagram. The representation of activity shows the sequence of the experiment rather than focussing on the activity involved in dissociation.
|
|
Figure 4. High and Low Scoring Explanations of water as a solvent for sodium nitrateSource: Pseudonyms are used in the main text. The source cannot be cited in accordance with research ethics agreement. |
The extensive and varied types of annotation in Amri’s diagrams deal with composition as well as activity. Aggregation therefore occurs through integration by representing these different meanings that construe technicality in either the verbiage or the images or both. Key items represented are named or indicated by symbols such as H or O. In addition, activity is either named as an activity entity (‘attraction’) or the action is described as a figure (clause) as in ‘the cation is removed from the lattice’. The latter is positioned as an un-numbered caption adjacent to the left-hand image. A similar un-numbered caption, ‘the anion is removed from the lattice’ is located below and to the left of the right-hand image. The other annotations are connected to the relevant image elements by arrows. By contrast Sim uses almost no annotation at all except for a one sentence sub-posed caption-like annotation ‘bonds between H2O and NaNo3 break making bonds with each other’ (the inaccuracy of this annotation betrays Sim’s lack of understanding of dissociation). Amri includes some information in the diagram which is not mentioned in the main text such as the naming of the cation and anion. No additional information is provided in Sim’s diagram. The positioning of the diagrams in Amri’s response in relation to the text blocks is also highly cohesive. The final sentence of the top text block indicating the activity of the water molecule dipoles pulling the ions from the lattice is then elaborated visually and verbally in the diagrams below it by ‘momenting’ - showing diagrammatically the constituting sub-activity. Cohesion is also achieved in the bottom text block as the beginning of the first sentence ‘condenses’ the prior multimodal explanation succinctly as ‘The above process’.
In seeking to provide an explanation of the phenomenon described in the assessment task, Amri was able to select quite extensively and deploy effectively multimodal options for representation that typify infographics. Amri made more extensive use of a variety of types of annotation to construct both composition and activity relations, whereas Sim made very minimal use of annotation. The commonality and complementarity of meaning representation across image and language was effective in Amri’s deployment of these modes whereas Sim’s diagram showed only the structure of the water molecule and the sequence of the experiment in addition to what was in the text. Amri’s coordination of annotated diagrams and interpolated text blocks was also highly cohesive. Amri was more effectively able to deploy the resources of language and of image to construct a clear and accurate representation, albeit avoiding the challenge of including a detailed representation of the ionic lattice for sodium nitrate. While Sim’s apparent misconceptions about the nature of dissociation and lack of knowledge about the sodium nitrate lattice are central to the inadequacy of the response, it is also clear that this student’s repertoire of language resources and resources for constructing infographic responses to assessment items are much more limited than those of Amri.
Discussion
Implications for Transdisciplinary Semiotic and Science Education Research
It is only recently that attention has been drawn to the significance of multi-structure images in science (Doran, 2019; Martin, 2020), whereby two or more of the ideational meanings construing phenomena (composition, classification, activity, and property) are represented with equal status in the one visualization. The research reported in this paper extends this recognition to propose a framework that describes options for the co-articulation of image and language to construct such combinations of different kinds of meanings in condensed synoptic image-language ensembles through strategies of accumulation and integration. While these visualizations incorporating language within image-focused portrayals increasingly characterize science textbooks (Bezemer & Kress, 2010; Bateman, 2008), previous multimodal analyses of such science texts have not addressed how these portrayals represent the meaning complexes involving simultaneous representation of two or more of the different types of meaning construing field (Danielsson & Selander, 2016; Knain, 2015; Peterson, 2016; Polias, 2015).
Few studies of student created multimodal representations have addressed the nature and extent of the co-articulation of image and language, and where this has been addressed the semiotic basis for its examination has been severely limited. For example, Gebre and Polman (2016) investigated the extent to which high school student created infographics included meanings represented by ‘non-text representations’ that were not already communicated by other such representations or by language, and McDermott and Hand (2013, 2016) designed an intervention to build student awareness of ways to improve the integration of non-text modes in their multimodal representations, however in their scale for assessing ‘non-text’ integration, each use of a non-text mode was awarded a point for any of the following characteristics: next to the text, referred to in the text, including a caption, scientific accuracy, completeness (amount of detail), and originality (created by the student and not adopted from another source). The first three criteria only are inter-modal linking devices, but it is not clear whether the text refers to main text accompanying the multimodal representation, interpolated text blocks as part of the multimodal ensemble or an annotation. There is no consideration of the ways in which the combination of image and language construct the ideational meanings relevant to the topic, much less the aggregation of ideational meanings building complex knowledge of field. The use of the framework for image-language co-articulation in aggregating meaning in infographic responses outlined in this paper suggests the potential for extending transdisciplinary studies in educational semiotics and science education. Such transdisciplinary approaches to investigations into students’ creation of multimodal representations in science learning could provide greater specification of students’ use of the resources of language and image and their co-articulation in communicating the students’ understanding of complex phenomena. Hence enhancing the capacity of the research findings to inform pedagogy that would improve students’ interpretation and creation of infographic representations.
Pedagogic Implications
The critical interpretation and the creation of complex infographics that represent two or more of the dimensions of ideational meaning construing field (activity, composition, classification, and property) through the combined resources of image and language, are crucial aspects of the multimodal disciplinary literacy of science to which students need to be effectively apprenticed. These highly synoptic displays of knowledge where well supported by teacher scaffolded reading, explicit teaching about how the resources of image and language combine to construct meaning, and critical classroom discussion, can work effectively for students as summative aggregations of accumulated knowledge; where supportive reading and classroom interaction critically addressing the multimodal ‘constructedness’ of infographics does not occur, the highly condensed and often implicit representation of meaning in infographics may well function as impenetrable obstacles to learning, especially for students from low socio-economic backgrounds and/or those whose first language is not the language of instruction. Similarly, explicit pedagogic support for students learning to create infographics through teacher deconstruction, explanation and critical comparison of multiple infographics on the same topic, teacher demonstration of infographic construction, explication of the co-deployment of image and language, joint teacher-student construction of infographics, peer co-construction of infographics and opportunities for student independent construction with detailed individual feedback, can enable students to participate effectively in this disciplinary representational practice and may well enhance their achievement in external examinations requiring these kinds of student constructed responses to short answer questions.
Explicit attention to developing students’ capacity to interpret and create infographic representations of phenomena is more effective when treated as a fundamental part of science pedagogy – rather than being seen as learning science literacy in isolation from the teaching and learning experiences of the science classroom. Such an 'infused' approach to disciplinary literacy development (Martin et al., in press; Unsworth et al., forthcoming) proposes a pedagogic orientation combining the fostering of student enquiry with the teacher’s role of providing guidance through interaction in the context of shared experience (Martin & Rose, 2005). In our further research, we discuss the facilitative role of an accessible and shared metalanguage (including metalanguage derived from SFL) that describes the meaning-making affordances of different semiotic modes as teachers’ support students to interpret and create different levels of representation involving multiple modes in order to show casual relations among changing structures in complex activities (Unsworth et al., forthcoming). An effective pedagogic model for this involves teaching/learning cycles (Martin et al., in press; Rose & Martin, 2012). These cycles unfold in a series of steps – building field knowledge, teacher scaffolded reading, teacher modelled deconstruction of sample texts and demonstration of text creation, joint teacher-student text construction, and ultimately independent student construction. Various re-conceptualizations of such a cycle have emphasized that it can be entered at different phases, with differing emphases on interpretation and production, and with phases being recycled according to the needs of the student group (Rose & Martin, 2012; Unsworth, 2001). Crucially these infused multimodal disciplinary literacy approaches are informed by explicit linguistic and visual semiotic descriptions of the meaning making resources of language and image. Such descriptions are in turn informed by the analytic framework outlined in this paper as a support for pedagogic development rather than a direct component of pedagogy. It is hoped that emerging semiotic studies of school science infographics and their relationship to science learning and teaching as described herein will foster greater research and pedagogic attention to the significant role of infographics in high school science teaching and curriculum.
Funding: This research was supported by an Australian Research Council Linkage Projects grant (LP160100263) funded by the Australian Government. The views expressed herein are those of the author and are not necessarily those of the Australian Research Council.
The original paper was received on January 7th, 2021
The reviewed paper was received on April 14th, 2021
The paper was accepted on June 8th, 2021
References
Armstrong, Z., Deeker, W., Madden, A., McMahon, K., Naughton, K., Siwinski, S., & Wheeler, A. S. (2019). Pearson Biology 12: New South Wales student book: Melbourne, Victoria: Pearson Australia.
Bateman, J. (2008). Multimodality and Genre: A Foundation for Analysis. London: Palgrave Macmillan.
Bennett, A. (2017). Pearson science. 9, T.C (2nd edition.. ed.): Melbourne, Victoria: Pearson Australia.
Bezemer, J., & Kress, G. (2010). Changing Text: A Social Semiotic Analysis of Textbooks. Designs for Learning, 3, 10-29.
Borger, P. (2015). Nelson biology VCE units 1 & 2 (3rd ed.): South Melbourne, Victoria: Nelson Cengage Learning.
Callow, J. (Ed.). (1999). Image matters: Visual texts in the classroom. Sydney: Primary English Teaching Association.
Callow, J. (2013). The shape of text to come: How image and text work: Primary English Teaching Association of Australia.
Chidrawi, G., Davis, A., Farr, R., Lampman, K., Matchett, B., & Young, P. (2013). Nelson iscience 10. Melbourne: Nelson Cengage
Martin, J. R., & Unsworth, L. (forthcoming). Reading Images for Knowledge Building: Analyzing Infographics in School Science London: Routledge.
Martin, J. R., Unsworth, L., & Rose, D. (in press). Condensing meaning: Imagic aggregations in secondary school science. In G. Parodi (Ed.), Multimodality: From corpus to cognition. London: Bloomsbury.
Painter, C., Martin, J. R., & Unsworth, L. (2013). Reading Visual Narratives: Image Analysis of Children’s Picture Books. London: Equinox.
Danielsson, K., & Selander, S. (2016). Reading Multimodal Texts for Learning–a model for cultivating multimodal literacy. Designs for Learning, 8(1), 25-36.
de Silva Joyce, H., & Gaudin, J. (2007). Interpreting the Visual: A Resource Book for Teachers. Sydney: Phoenix.
Doran, Y. J. (2017). The discourse of physics: Building knowledge through language, mathematics and image. London: Routledge.
Doran, Y. J. (2019). Building knowledge through images in physics. Visual Communication, 18(2), 251-277.
Doran, Y. J. & J. R. Martin (2021). Field relations: understanding scientific explanations. Teaching Science: Knowledge, Language, Pedagogy. K. Maton, J. R. Martin and Y. J. Doran. London, Routledge: 105 - 133.
Gebre, E. H., & Polman, J. L. (2016). Developing young adults’ representational competence through infographic-based science news reporting. International Journal of Science Education, 38(18), 2667-2687.
Halliday, M. A. K., & Martin, J. R., (Eds.). (1993). Writing science: Literacy and discursive power. London: Falmer Press.
Höllerer, M. A., van Leeuwen, T., Jancsary, D., Meyer, R. E., Andersen, T. H., & Vaara, E. (2019). Visual and multimodal research in organization and management studies: Routledge.
Huxley, L., & Walter, M. (2019). Oxford Biology for Queensland Units 1 and 2. Melbourne: Oxford University Press.
Kinnear, J. (2016). Nature of biology. 1 : VCE units 1 and 2 (Fifth edition.. ed.): Milton, Qld : John Wiley & Sons Australia Ltd.
Knain, E. (2015). Scientific literacy for participation: A systemic functional approach to analysis of school science discourses: Springer.
Kress, G. (2005). Gains and losses: New forms of texts, knowledge, and learning. Computers and composition, 22(1), 5-22.
Kress, G., & van Leeuwen, T. (2006). Reading Images: The grammar of visual design (2 ed.). London: Routledge.
Kress, G., & van Leeuwen, T. (2021). Reading Images: The grammar of visual design (3 ed.). London: Routledge.
Lemke, J. (1990). Talking science: Language, learning and values. Norwood, New Jersey: Ablex.
Lemke, J. (2004). The literacies of science. In E. W. Saul (Ed.), Crossing borders in literacy and science instruction: Perspectives on theory and practice (pp. 33-47). Newark DE: International Reading Association.
Linstead, G., Clarke, W., Devline, J., Madden, D., Rickard, H. C., & Spenceley, M. (2012). Pearson science. 10 : S.B. Port Melbourne, Vic: Pearson Australia.
Lofts, G., & Evergreen, M. J. (2015). Science quest 10 : Australian curriculum (2nd ed.): Milton, Qld.: Jacaranda.
Martin, J. R. (2017). Revisiting field: Specialized knowledge in secondary school science and humanities discourse. Onomazein, 111-148.
Martin, J. R. (2020). Revisiting field: Specialized knowledge in secondary school science and humanities discourse. In – Martin, J. R., K. Maton & Y. Doran (Eds.), Accessing Academic Discourse: Systemic Functional Linguistics and Legitimation Code Theory (pp. 114-148). London: Routledge.
Martin, J. R., & Rose, D. (2005). Designing Literacy Pedagogy: scaffolding asymmetries. In J. Webster, C. Matthiessen & R. Hasan (Eds.), Continuing Discourse on Language (pp. 251-280). London: Continuum.
Martin, J. R., & Rose, D. (2012). Genre and texts: living in the real world. Indonesian Journal of Systemic Functional Linguistics, 1(1), 1-21.
Martin, J. R., & Veel, R. (Eds.). (1998). Reading Science: Critical and functional perspectives on discourses of science. London: Routledge.
McDermott, M. A., & Hand, B. (2013). The impact of embedding multiple modes of representation within writing tasks on high school students’ chemistry understanding. Instructional Science, 41(1), 217-246.
McDermott, M. A., & Hand, B. (2016). Modeling scientific communication with multimodal writing tasks: Impact on students at different grade levels. In B. Hand, M. McDermott & V. Prain (Eds.), Using multimodal representations to support learning in the science classroom (pp. 183-211). Cham, Switzerland: Springer.
Nunes, T., Bryant, P., Strand, S., Hillier, J., Barros, R., & Miller-Friedmann, J. (2017). Review of SES and science learning in formal educational settings. London: University of Oxford/Education Endowment Foundation.
Ozdamli, F., & Ozdal, H. (2018). Developing an instructional design for the design of infographics and the evaluation of infographic usage in teaching based on teacher and student opinions. EURASIA Journal of Mathematics, Science and Technology Education, 14(4), 1197-1219.
Peterson, M. O. (2016). Schemes for Integrating Text and Image in the Science Textbook: Effects on Comprehension and Situational Interest. International Journal of Environmental and Science Education, 11(6), 1365-1385.
Polias, J. (2015). Apprenticing students into science: Doing, talking, writing and drawing scientifically. Stockholm: Hallgren and Fallgren.
Polman, J. L., & Gebre, E. H. (2015). Towards critical appraisal of infographics as scientific inscriptions. Journal of Research in Science Teaching, 52(6), 868-893.
Rickard, G. (2017). Pearson Science 9. Melbourne, Pearson.(p.108)
Rose, D., & Martin, J. R., (2012). Learning to write, reading to learn: Genre, knowledge and pedagogy across the curriculum. London: Equinox.
Silvester, H. (2016). Oxford science 10 : Victorian curriculum. Melbourne: Melbourne Oxford University Press Australia.
Teese, R. (2013). Academic Success and Social Power: Examinations and Inequality. North Melbourne: Australian Scholarly Publishing.
Tytler, R., Prain, V., & Hubber, P. (2018). Representation Construction as a Core Science Disciplinary Literacy. In K.-S. Tang & K. Danielsson (Eds.), Global Developments in Literacy Research for Science Education (pp. 301-318). Cham, Switzerland: Springer.
Unsworth, L. (2001). Teaching Multiliteracies Across the Curriculum: Changing contexts of text and image in classroom practice. Buckingham, United Kingdom: Open University Press.
Unsworth, L. (2020). Intermodal relations, mass and presence in school science explanation genres. In Michele Zappavigna & S. Dreyfus (Eds.), Discourses of hope and reconciliation: J. R. Martin’s contributions to Systemic Functional Linguistics (pp. 131-152). London: Bloomsbury Academic.
Unsworth, L., & Cleirigh, C. (2009a). Multimodality and reading: The Construction of meaning through image-text interaction. In C. Jewitt (Ed.), Handbook of Multimodal Analysis (pp. 151-164). London: Routledge.
Unsworth, L., & Cleirigh, C. (2009b). Towards a relational grammar of image-verbiage synergy: Intermodal representations. In S. Dreyfus, S. Hood & M. Stenglin (Eds.), Semiotic Margins. University of Sydney: Australian Systemic Functional Linguistics Association (www.asfla.org.au).
Unsworth, L., Tytler, R., Fenwick, L., Humphrey, S., Chandler, P., Herrington, M., & Pham, L. (forthcoming). Multimodal Literacy in School Science: Transdisciplinary Perspectives on Theory, Research and Pedagogy: Routledge.
Walsh, E. M., & McGowan, V. C. (2017). ‘Let your data tell a story:’climate change experts and students navigating disciplinary argumentation in the classroom. International Journal of Science Education, 39(1), 20-43.