The impact of subtitles and transcripts in disciplinary videos for EAL learners

Existing research on instructional videos primarily explores subtitles and transcripts in vocabulary and listening comprehension for English as an Additional Language (EAL) learners, with limited focus on content and language integrated learning (CLIL). This pilot case study investigates how these tools mediate postgraduate EAL learners’ mathematics learning, spotlighting their video consumption approaches and thinking processes. Findings from video-stimulated recall (VSR) and semi-structured interviews highlight the distinct advantages of both transcripts and subtitles in facilitating CLIL. While transcripts provide a robust textual reference, enabling learners to navigate through complex mathematical concepts efficiently, bilingual subtitles foster translanguaging and crosslinguistic strategies, empowering learners to leverage their full linguistic and cognitive repertoire for understanding mathematical concepts. Our findings enhance and refine existing knowledge of video designs, highlighting the crucial, complementary roles of subtitles and transcripts for EAL learners, with implications for developing and implementing instructional videos, endorsing strategic use of both tools to accommodate varied learning needs.


Introduction
The use of videos in education has become increasingly popular, particularly with the rise in digitally enabled instructional approaches such as flipped learning and hybrid learning.Among the various types of instructional videos employed in higher education, lecture recordings have been generally recognized as effective learning tools.Nonetheless, a noticeable disparity persists in educational experiences between international and domestic students in Australia, particularly within the tertiary sector, concerning the use of learning resources.Such a divergence in satisfaction rating has been widening since 2019, with international students rating (74%) lower than domestic students (81%) (Department of Education, 2022).This disparity is most pronounced within the science and mathematics domains (Department of Education, 2022).
Given the significant number of international students who speak English as an additional language (EAL) in Australian higher education, researchers are motivated to investigate the effectiveness of different instructional designs used in lecture recordings for EAL learners (Mayer et al., 2014).While much previous research has adopted either media-or instruction-centered approaches, there is a need for more learner-centered investigations which capture the influences of the linguistic and cultural backgrounds of non-English speaking students.
This study examines the lived experiences of EAL learners during their mathematics learning with lecture recordings.Through observing their video-watching behaviors, it explores their engagement with subtitles and transcripts, and the cognitive processes associated with these practices.Using video-stimulated recall (VSR) and semi-structured interviews within a case study design, the study delves into the complexities of why and how these assistive language tools influence their learning experiences and understanding of the subject-specific content in mathematics.

Background Learning mathematics with instructional videos for EAL learners
Guided by the cognitive theory of multimedia learning (CTML) model (Mayer, 2014), previous design-based research has invested a significant amount of effort in examining the effectiveness of language supporting tools in instructional videos, such as subtitles, captions, narrations, and transcripts to support EAL learners (Bensalem, 2017;Liao et al., 2020;Teng, 2020).However, due to the existence of various boundary conditions found in general research on CTML principles (Fyfield et al., 2022), and the lack of learner-centered studies around non-English speaking students, there remains a compelling need for further research to capture the nuanced interactions between the language factors, cultural backgrounds, and instructional designs of videos.Moreover, while many studies related to the use of subtitles and transcripts have focused on their values in supporting language learning and improving language comprehension (Liao et al., 2020;Metruk, 2018;Teng, 2020), only a limited number of studies have explored their potential values in supporting the learning of subject content and subject-specific language.Thus, there exists an imperative to expand our understanding of how these language aids may facilitate the mastery of subject content through the mediation of language.
Contrary to the commonly held belief that "Mathematics is a universal language," a large volume of research has shown that learning mathematics in a non-native language can be challenging and hence affect learners' performance and engagement (Moschkovich, 2015).Although the challenges associated with acquiring content knowledge through an additional language have been well-documented in the research field of content and language integrated learning (CLIL), this concept has rarely been applied to scrutinize the current video designs guided by the CTML principles.Therefore, exploring the efficacy of some of these video designs from the perspective of content and language integrated learning may result in valuable and useful insights.

Conceptual Framework
With a long tradition of examining multimedia learning from a cognitivist lens, Mayer's (2014) Cognitive Theory of Multimedia Learning (CTML) model affords a comprehensive overview of how visual and auditory information are processed by the human mind and hence explains the functionality of subtitles and transcripts during activity based on their impacts on learners' cognitive loads.Although the information processing of multimodal inputs from lecture videos can be explained by the CTML framework, how this process is influenced by other contextual factors such as one's language and cultural background has not yet been fully explored within this model.Thus, an additional theoretical lens is required to capture the contextualized nature of mathematics learning.
In the context of content and language integrated learning, content learning is scaffolded through the use of language (Jäppinen, 2005) and other tools with an aim to facilitate learning from lower-order thinking skills (e.g., remembering, comprehending) to higher-order thinking skills (e.g., analyzing, evaluating, and creating) (Coyle, 2007).Thus, language assistive tools adopted in the video designs should also facilitate learners in mathematical thinking beyond the comprehension level.In this sense, Vygotsky's (1978) sociocultural theory (SCT), which is commonly regarded as a fundamental principle underpinning CLIL, offers a potential lens in conceptualizing language(s) as a key mediational tool for higher-order thinking with the assistance of subtitles and transcripts in mediating this process.
More importantly, the notion of genetic method in Vygotsky's (1978) sociocultural theory allows the investigation of a wide range of personal and contextual factors such as EAL learners' language background, prior knowledge, experience, and culture that influence their approach to video learning in mathematics and their perceived effectiveness of language supporting designs (Authors, 2023).Therefore, it provides a theoretical foundation to explore the dynamic interplay between linguistic, cognitive, and sociocultural elements in shaping this learning experience.

Qualitative data collection
To understand video learning as a real-life experience and the context pertinent to it, a case study design that captures the answers to our "how" and "why" questions (Yin, 2018) was considered to be appropriate.Two participants, Siyi and Yihe (pseudonyms), who both spoke Chinese as their first language, were recruited from two postgraduate mathematics units at an Australian university.
To gather information about how the use of subtitles and transcripts supports our EAL learners' learning of mathematics, participants were invited to perform video-stimulated recalls (VSR) after a 30-minute video watching session.Each of them engaged with a selected lecture recording from their current mathematics units via the Echo360 platform, wherein transcripts were concurrently displayed adjacent to the recording.The platform afforded learners the autonomy to enable subtitles and manipulate their aesthetic properties, such as size, background transparency, color, and position.Stimulated recall is an introspective method which prompts individuals to reflect on their thought processes and strategies when engaged in an activity (Henderson & Tallman, 2006).Immediately after the video watching session, VSR and semi-structured interviews were conducted in participants' native language (Chinese) to provide them with the opportunities to delve into a more detailed discussion about their experience and how the assistive tools facilitated the meaning-making process of both subject content and language.
Ethics approval was provided by the institutional human research ethics committee before data collection.

Reflexive thematic analysis
To provide a multifaceted and nuanced lens for interpreting qualitative data, reflexive thematic analysis (TA) was used to leverage the first author's personal experience as an EAL learner and her professional knowledge as an educator.Reflexive TA views meaning and knowledge as situated and contextual, and hence conceptualizes researcher subjectivity as "a source of knowledge" (Braun & Clarke, 2021, p. 334).The first author's teaching and learning experience not only enhanced her ability to discern subtleties in learners' understanding and behaviors, but also provided her contextual understandings to inductively identify the social, cultural and cognitive factors which were relevant and meaningful within the context being studied (Braun & Clarke, 2019, 2021).Therefore, reflexive TA unfolded the complexity of this research topic by the power of the author's sensitivity and knowledge toward the possible issues and themes emerging from the data.(Braun & Clarke, 2021).
The data analysis was conducted with multiple stages.It was initiated with a thorough process of familiarization and reflexive thematic development.The first author began by carefully immersing herself in the data, reading through the video-stimulated recall (VSR) recordings and the interview responses provided by the participants.In the next stage, interview transcripts were coded to identify key concepts emerged from their responses and constructed into themes.Through iterative development, the author refined and clarified the key themes and their contextualized meaning acknowledging Braun and Clarke's (2019) call for reflexive engagement with the data and their contextualized interpretation.

Findings and Discussion
As a prevalent learning strategy employed by EAL learners, translanguaging is an approach that has been observed to enhance their understanding by leveraging language resources from both their native language (L1) and the additional/target language (L2) to aid comprehension and learning in CLIL (Nikula & Moore, 2019).Throughout the study, participants consistently demonstrated a higher reliance on language assistive tools (e.g., subtitles and transcripts) compared to other video design features (e.g., graphics and visual flow) emphasizing the effectiveness of these tools in supporting their mathematics learning process.It was confirmed in the indepth interviews that a substantial portion of their learning time was dedicated to "comprehending subjectspecific content and language" while engaging with the lecture recordings.Therefore, the focus of the following analysis is on how these two EAL learners used language supporting tools to facilitate the sense-making process of both the subject content and the subject-specific language.

Transcript vs subtitle
Subtitles and transcripts are essential language supporting tools in multimedia learning, especially for EAL learners (Grgurović & Hegelheimer, 2007).While subtitles provide real-time textual representations of dialogues in videos, transcripts offer a detailed written account of all spoken content, enabling viewers to methodically review and revisit material at their own pace (Grgurović & Hegelheimer, 2007).Both tools aim to assist viewers' language comprehension and hence their understanding towards the video content.
During the video-watching sessions, it was observed that the participants frequently paused the lecture recordings to search for subject-specific terms in the transcript displayed on the side panel (adjacent to the video) and then translated them into Chinese.In the VSR and subsequent interview sessions, the participants mentioned that although they could make educated guesses about the meanings of these words, confirming their exact meanings in mathematics required contextual information from the preceding dialogue to ensure precise understanding within the subject or topic.Thus, the participants found transcripts to be more useful than subtitles, attributing it to transcripts' capacity to display an extensive portion of the dialogue at once, thereby providing a broader textual context that facilitates a deeper and more contextually informed understanding of the mathematical terms or concepts being taught.On the other hand, the transient and rapid display nature of subtitles, which are typically presented in a swift and continually changing manner, rendered reading more challenging and hence resulted in superficial processing of information (Liao & Kruger, 2023;Liao et al., 2021) and difficulties in grasping a concept.
The two participants in this study highlighted the advantage of transcripts over subtitles as it provided more textual information of the surrounding text.However, the participants also pointed out some drawbacks on learning with transcripts.One disadvantage was the time required to locate specific words or phrases by scrolling through the transcript compared to searching in synchronous subtitles.Subtitles, with their succinct and immediate presentation of textual content synchronized with spoken words, provided a more streamlined approach to vocabulary learning by offering real-time textual counterparts (Grgurović & Hegelheimer, 2007), thereby facilitating a more immediate translation process.
Our study showed that the two participants watched only 10% to 20% of the lecture recordings, on average, within the 30-minute activity session.The participants identified language comprehension as a major obstacle impeding their learning.In seeking their input on potential improvements to the video designs, the participants emphasized the distinct advantages of both transcripts and subtitles in facilitating mathematics learning and suggested the potential benefit of enabling the automatic-scrolling function of transcripts.

Monolingual vs bilingual subtitle
Another intriguing observation made during the study pertained to the participants' utilization of an automated browser plug-in that enabled synchronous Chinese subtitles.When questioned about the rationale behind their use of bilingual subtitles in both languages, participants emphasized the significance of employing translingual approaches in the realm of mathematics learning.One participant elucidated, "In some cases, a mathematical concept may possess different names in English and Chinese.I often compare both names to determine which name aligns more closely with its intended meaning."Subsequent interviews provided further insight into the participants' perspectives on the value of having comparative texts in both languages.
My mother tongue is Chinese, and my English is not good, so when I think deeply, I can only think in Chinese, and I can't think in English at all…Language is part of a culture...I can't leave the Chinese culture or my language (Chinese) behind.It's hard for me to think outside these boxes.
Vygotsky posited a fundamental interconnection between language and cognitive development, asserting that language serves not merely as a communication tool but also as a critical mediator in thinking processes, influencing and shaping our cognitive activities (Vygotsky, 1978).Despite the body of work in CTML that presents mixed evidence on the use of bilingual subtitles in terms of cognitive loads (Liao et al., 2020;Wang & Pellicer-Sánchez, 2022), our study shows that the use of bilingual subtitles holds significant value when assisting EAL learners in the content and language integrated learning settings.It fosters a deeper understanding of the subject matter by affording cognitive flexibility and enabling EAL learners to access a wealth of information beyond the confines of a single language.

Conclusion
This pilot study confirms that bilingual subtitles and transcripts can support EAL learners' meaning-making process during video learning.While this finding is not surprising, it serves as a reminder to educators and educational designers that these functions, along with the availability of language customization options and other more advanced functions (e.g., automatic-scrolling transcript), are important considerations in the design of instructional videos.Moreover, given the specific focus of this study on mathematics education, it is noteworthy how the two participants strategically used these assistive language tools to support not only their language comprehension but also their in-depth understanding of the subject-specific content through translingual processing, which we argue could further support critical thinking and reasoning.This warrants particular attention, as non-language educators might otherwise assume that assistive language tools are merely important for language comprehension, without recognizing the significant roles they may have in fostering CLIL.Also, our study has demonstrated the value of examining the effectiveness of video designs from a sociocultural perspective, perceiving them as mediational tools that are influenced by a wide range of noncognitive factors in the video learning process.