Examining students’ behavioural engagement in lecture videos with and without embedded quizzes in an online course.

This study reports part of a larger study that explores three technologies – Padlet, Panopto videos and Google Docs – and their effectiveness in enhancing university students’ engagement in online learning. The current paper explores the role of interactive quizzes embedded in recorded lectures in enhancing student engagement in higher education focused on behaviour engagement. Panopto video analytics and a survey were used as a source of data. Descriptive statistics were used to analyse the data, including percentage and frequency. The analyses focused on 59 university students’ behavioural engagement in the recorded videos with and without embedded quizzes in relation to their video dropout, average viewing time, completion rate and frequency of viewing. The paper highlights the benefits of incorporating interactive quizzes within lecture videos, including increased motivation, engagement, and academic performance. However, this study also shows some interesting results on students’ behavioural engagement when it comes to learning online. Overall, findings suggest that the incorporation of quizzes in lecture videos can be an effective tool for increasing student engagement and improving learning outcomes in online courses. Implications for online course design and future research are discussed.


Introduction
Student engagement is crucial in achieving better learning outcomes in higher education.In recent years, educational technologies such as video platforms and online quizzes have become popular tools for increasing student engagement.One such platform is Panopto, which provides a range of features to create and manage video content, including interactive quizzes.Panopto quizzes are designed to be integrated into video content to engage students while watching and to assess their learning outcomes.This study aims to explore the influence of Panopto quizzes on students' behavioural engagement in higher education.
According to previous studies, interactive technologies such as quizzes in video content have been shown to improve student engagement (Getenet et al., 2022a;Jones et al., 2022;Schindler et al., 2017).For example, Jones et al. (2022) embedded quizzes in lecture videos to test students' understanding of concepts as they were covered and their impact on their learning.However, few studies have investigated the specific influence of video embedded quizzes on students' behavioural engagement in higher education.This study seeks to fill this gap in the literature by examining how Panopto quizzes influence students' behavioural engagement, including their attendance, participation, and attention span during lectures.We focused on the information acquired from Panopto analytics and survey questions.By understanding the impact of Panopto quizzes on students' behavioural engagement, educators can make informed decisions about the use of this tool to enhance student learning outcomes.Specifically, we aim to answer the following research question: • Is there a difference in the level of student behavioural engagement in lecture videos with and without Panopto quizzes?
This study has important implications for educators to design courses seeking to optimise student engagement and learning in online course environments.By identifying the potential benefits of video-embedded quizzes, educators in higher education may be able to design more effective course materials that promote student engagement and facilitate learning.Additionally, this study may provide insight into the broader question of how to design effective online learning environments that promote student engagement and success.

Student engagement
Higher education places a significant emphasis on student engagement due to its strong connection with academic outcomes such as retention, grade point average, and graduation rates (Hu & McCormick, 2012).However, engagement has various meanings and interpretations, resulting in diverse views about its definition and the interrelatedness of various factors (Dixson, 2015;Lawson & Lawson, 2013).While some studies define engagement in relation to the extent or quality of students' commitment and active involvement in learning (e.g., Garrison, 2018;Redmond et al., 2018), others define it as students' social interaction with teachers and peers (Lawson & Lawson, 2013).
Recently, researchers have shifted their focus from examining only students' cognitive processes to exploring other aspects of engagement and how technology can facilitate students' engagement with learning (Redmond et al., 2018).Garrison's (2011) Community of Inquiry Framework for e-learning, which incorporates social, cognitive, and teaching presence, has been shown to enhance students' educational experiences when used in online course design (Garrison, 2018).However, the concept of engagement still requires further explanation, as noted in a review of multidimensionality in online learning by Fredricks et al. (2004).
To provide a more comprehensive framework, Martin and Borsup (2022) integrated the cognitive, affective, and behavioural dimensions with learners' background, learner characteristics, learning environment, and personal environment as facilitating factors for positive learning outcomes.Similarly, Redmond et al. (2018) proposed an online engagement framework for higher education with five dimensions: behavioural, emotional, cognitive, collaborative, and social engagement.Although the authors did not develop an instrument for measuring the dimensions of engagement, they provided indicators for each dimension that could be used to develop measurement methods.The authors suggested that the framework could serve as an "audit tool or point of reference" (p.196) for institutions seeking to enhance student engagement in online learning.Redmond et al. (2018) showed that social engagement involves purposeful and trusting relationships with others, cognitive engagement encompasses the active process of learning, and behavioural engagement entails demonstrating positive learning behaviours and attitudes.Collaborative engagement involves developing different relationships and networks that support learning, including collaboration with peers, educators, industry, and the educational institution, while emotional engagement relates to students' feelings or attitudes towards learning.The framework suggested by Redmond et al. (2018) is particularly relevant to this study because it provides a clear definition of each dimension of engagement, and its context is specific to higher education.In this study, students' behavioural engagement is considered for further scrutiny in relation to quiz embedded lecture videos.

Behavioural engagement
Behavioural engagement, defined as actions and behaviours taken that promote students' thinking and learning, is considered a fundamental form of engagement (Bowen, 2005;Redmond et al., 2018;).Redmond et al. (2019) emphasised the importance of behavioural engagement in enhancing students' interest in learning and academic skills.Fredricks et al. (2004) described behavioural engagement as "doing the work and following the rules" (p.65).This would include students accessing materials, attending online classes, and completing and meeting assessment due dates.Measuring student engagement has become challenging, and several frameworks have been used recently.Bote-Lorenzo and Gomez-Sanchez (2017) identified 16 indicators to measure student engagement in an online course, while Li and Tsai (2017) analysed 14 indicators related to time spent on educational resources.Singh et al. (2018) proposed a content engagement score to measure student engagement towards specific content.Engagement indicators such as the number of views or clicks, length/duration of viewing time, and completion rate and frequency of viewing can also be used to measure students' behavioural engagement (Breslow et al., 2013;Guo et al., 2014;Jordan, 2014;Kim et al., 2014;Koller et al., 2013).Table 1 further summarises these indicators with their corresponding definition, descriptions and authors.

Viewing duration
The amount of time students spend watching a video The average amount of time students spend watching a video influences their engagement, and attention tends to decline after the first 10 minutes.Krumme et al. (2012), Wu et al. (2018)

Time of viewing
The time during a program when students watch a video The U-shape nature of video consumption shows that more videos are watched at the beginning and end of a program, and clearly establishing time parameters may encourage students to watch videos completely.

Viewing dropout
The percentage of students who start watching a video but leave before playing entirely and the length of time learners spend watching lectures The percentage of students who start watching but do not finish a video and establishing time parameters may be effective in getting students to complete video lectures.

Lecture videos
The use of lecture videos has become increasingly prevalent in higher education, with many institutions incorporating them into their course materials to supplement or replace traditional in-person lectures (O'Callaghan et al., 2017).While the benefits of lecture videos are numerous, including flexibility and convenience, one challenge is ensuring that students remain engaged and focused while watching them.Research has shown that students may be prone to distraction or multitasking while watching lecture videos, which can lead to decreased retention of the material and academic outcomes (Mayer & Moreno, 2003;Szpunar et al., 2013).To address this issue, some educators have begun to incorporate interactive elements into their lecture videos, such as quizzes or interactive simulations, in order to promote students' behavioural engagement and improve learning outcomes.One such tool is Panopto quizzes.These quizzes are embedded within lecture videos and allow students to pause and answer questions related to the material covered in the video.This provides an interactive element to the video, potentially increasing student engagement and promoting retention of the material.Research has suggested that interactive elements can be effective in promoting student engagement and improving learning outcomes.For example, a study by Kizilcec et al. (2013) found that incorporating interactive elements into online lectures improved student engagement and retention of the material.

Quizzes in lecture videos
Relatively little research has been conducted on the specific effectiveness of video-embedded quizzes in promoting student engagement and learning.Video-embedded quizzes are an interactive tool used in lecture videos to help students engage with the content and assess their understanding of the material (Getenet et al., 2022b).These quizzes are typically included within the video and allow students to pause and answer questions related to the content they just learned.The purpose of these quizzes is to help students stay engaged while watching lecture videos, as well as to reinforce the concepts covered in the video (Jones et al., 2021).They also provide immediate feedback to students, allowing them to identify areas where they may need further review or clarification and to test their understanding of concepts (Jones et al., 2021).Video-embedded quizzes can take many different forms, including multiple-choice, true/false, or short-answer questions.They can also be timed or untimed, depending on the preference of the educator.Educators can choose to make these quizzes mandatory or optional, depending on their teaching goals and objectives.The present study seeks to address this gap in the literature and contribute to our understanding of the effectiveness of video-embedded quizzes as a tool for promoting student behavioural engagement and improving learning outcomes in online course environments.

Method
This study is part of a larger research project that aims to investigate the impact of three digital tools -Padlet, Panopto videos, and Google Docs -on students' engagement in online learning at university.Specifically, this current study focuses on examining students' behavioural engagement with recorded lecture videos, with and without quizzes, using Panopto analytic data.The study was conducted on 59 students who were enrolled in a mathematics course at a regional university in Australia, from February to June 2022.The study analysed data from 18 recorded lecture videos, some with embedded quizzes (9) and others without quizzes (9).All videos were uploaded and made available through the Panopto video platform, which is integrated into the Canvas Learning Management System.Panopto offers various features such as recording and uploading lectures or presentations, embedding videos with content, and providing minute-by-minute analytics for each video.This feature allows researchers to analyse students' engagement levels in real time and gain valuable insights into their learning behaviour.

Participants and data source
There was a total of 59 students enrolled in the course.Throughout the 13-week semester, nine lecture videos with and without quizzes of varying durations were uploaded at different intervals.The length of the recorded lectures ranged from 16.9 minutes to 108.9 minutes, as presented in Table 2.This data was collected by order of views and downloads.The reliability of the items for each construct was conducted (N=3).The Cronbach's Alpha is greater than 0.70 (α= 0.96), which indicates high overall internal consistency among the three items representing each engagement dimension and fulfilling the minimum required value α=.70 (Hair et al., 2017); hence, the survey items were reliable to use in the study.
Furthermore, demographic information, including gender, age, and mode of study, was collected.The survey was distributed to 59 students; however, 13 students responded to pre (9 between 16 -24 and 4 between 25-34 years of age.Only 4 students responded to the post survey, and all were in the age range of 25 to 34 years.Of the survey participants, 1 was female and 3 were males, and most of the participants were studying off-campus (N=54) and full-time (N=5).

Data Analysis
In this study, university students' lecture video-watching patterns were analysed to understand and provide a unique opportunity for students to engage with recorded lecture videos.Video analytics and reporting from Panopto and a survey were the sources of our data.The Panopto analytics provides detailed reports on audience engagement and comprehensive insights into viewing behaviour for every video on the Panopto portal.The behavioural engagement indicators described in Table 1 were used to guide our analysis of the data from Panopto analytics.In addition, descriptive statistics examined participants' agreement with the survey questions on their behavioural engagement due to their experience of watching lecture videos with embedded quizzes in the online learning context.

Results
This section reports the results of the survey and Panopto analytics.The survey was conducted at the beginning and at the end of the 13-week semester where students rated the video-embedded quizzes as a learning tool.Panopto video analytics were analysed to observe the behaviour of the students on lecture recording with and without embedded Quizzes.

Survey results
The students were asked to place themselves on a continuum (5=strongly agree to 1=strongly disagree) regarding their overall level of agreement on the influence of video-embedded quizzes on their behavioural engagement in online learning.The students generally reported near the agreement level in the three items.The students' self-rating results of the mean (3.75) and standard deviation (0.95) across the three items are shown in Table 3.

Panopto analytics results
Lecture videos from the mathematics course were categorised into with and without quizzes.The videos were stream-lined by views and downloads and the percentage completed is shown in Figure 1 below.Through these graphs, it is shown that overall, there were more views of the videos with quizzes compared to those without quizzes.It is also observed that in videos without quizzes, students either dropped out early (not reaching 50%) or finished more than 75%, whereas in videos with quizzes, the dropout was consistent.However, the dropout below 50% was the same but the students who got engaged had a higher completion rate.Table 4 shows the average calculation of 18 videos, where 9 were with Quizzes and 9 without Quizzes.Some of these videos had more than one Quiz, which were multi-choice or true and false type only.Out of the four categories students dropping out of the lecture videos was common and was not dependent on whether there was a Quiz or not.On the other hand, the mean completion between 50% to less than 95% was quite low.On the assumption that 50% is a good engagement indicator.The average minutes delivered of videos without Quizzes was 1.2 times that of those with Quizzes, which does not show a significant difference to students' behaviour.However, the mean completion of greater than 95% is more with videos with quizzes compared to those without.

Discussion
This study results suggest that incorporating video-embedded quizzes in lecture videos can enhance student engagement in online courses.Participants in the video-embedded quizzes condition reported higher levels of engagement through their completion rate, as these quizzes assessed their understanding of the course materials.
In contrast, the non-video-embedded quizzes condition showed lower student engagement with lecture videos (see Figure 2).These findings align with previous research demonstrating the benefits of embedded quizzes, which have been shown to increase engagement and improve learning outcomes (Getenet al., 2022b).Moreover, the analysis revealed a notable number of students viewing over 75% of the lecture videos when quizzes were present.The distribution of video completion was slightly more spread out with quizzes, but the mean completion rates (7 for quizzes and 6.4 without quizzes) were not significantly different, with median completion rates of 7 and 6, respectively.The findings further indicate that the use of video-embedded quizzes helped maintain student engagement because of their interactive nature, particularly when they were able to complete over 50% of the video content (see Figure 3).Further analysis revealed that students actively engaged with the embedded quizzes to facilitate their understanding of new concepts within the recorded lectures.Video-embedded quizzes represent a basic level of interactivity that can change video viewers from passive to active learners and it often increases the length of the video with additional few minutes.Jones et al. (2021) reported that it took around five minutes per video and this is similar to the authors' experiences.Panopto only offers four types of quiz questions: True/False, multiple choice, multiple select and fill in the blank.These quiz types are sufficient for short and sharp knowledge checks only.However, additional interactive quiz formats, like drag-and-drop and matching questions, could be adapted in Panopto to enhance a more interactive nature and enhance learning engagement.While Panopto lacked these features, video-embedded quizzes with futures in this study offered opportunities for students to engage with video content.

Limitations and future studies
This study has some limitations that should be acknowledged.Firstly, the reliance on analytic data and a relatively small sample size of survey responses may restrict the generalizability of the study's conclusions.To address these limitations, future research should aim to replicate these findings in various course contexts, with larger and more diverse participant samples.This would help to establish the robustness and applicability of the results across different educational settings.Furthermore, it would be valuable for future studies to investigate the long-term effects of video-embedded quizzes on learning outcomes and the retention of material.Examining how the use of quizzes impacts knowledge retention over an extended period can provide deeper insights into their effectiveness as a learning tool.Finally, it is important to consider potential confounding variables in future research.Factors such as individual differences in prior knowledge or motivation may have influenced the outcomes but were not accounted for in this study.Addressing these variables would enhance the validity and comprehensiveness of future investigations.Regarding the quiz types used in this research, it is worth noting that the study was limited to multiple-choice and true or false quizzes.Exploring different quiz formats could provide additional insights and analytics, offering a more comprehensive understanding of their effectiveness in assessing learning.

Conclusion
In this study, the incorporation of quizzes in lecture videos is found to be a valuable approach for enhancing student engagement and assessing learning.By integrating quizzes into the video content, educators can effectively promote active learning and ensure that students grasp and retain important concepts covered in the lecture.This immediate review of knowledge shows engagement within the learning process and allows students to apply their understanding to subsequent assessments.The flexibility provided by these quizzes enables students to learn and apply concepts effectively.While this study has its limitations, it serves as a steppingstone for future research to expand upon these findings.By addressing the identified limitations and exploring the suggested areas for further investigation, we can continue to advance our understanding of effective educational practices.
Overall, the results of this study demonstrate the potential benefits of incorporating quizzes into lecture videos, offering educators a valuable tool for enhancing student engagement and promoting meaningful learning experiences.

Figure 1 :
Figure 1: The percentage completion of each Lecture videos by students with and without Quizzes

Figure 3 :
Figure 3: Students completing more than 75% of the recorded videosThese findings support the work ofJones et al., (2021) andKizilcec et al. (2013) recommending the use of interactive elements in videos to keep students behaviourally engaged.As shown in the current study's survey results, students see quizzes in videos as helpful for improving their academic skills, agency and familiarity with online learning norms.As videos are a common tool for teaching online, it is important to understand how to increase their value for online students.Building upon the work ofGuo et al. (2014) which identified viewer behaviours related to videos without interactive elements, this study shows that in videos that include Quizzes, students are more likely to watch until the end of the video.The data also suggest that students will watch less than half a video whether a quiz is embedded or not.This confirms earlier work byGuo et al. (2014) and Geri et al. (2014)  which related viewer drop-out rates to lengthy videos or those that students consider to be irrelevant or misrepresented by the title.The current study suggests that shorter videos with well-placed quizzes can encourage students to watch the entirety of a video.