Diverse definitions of engagement: Personalised learning analytics to support staff and students

Although teachers design learning experiences, their pedagogical and pastoral connections to students and teaching are often unaccounted for in learning analytics approaches. What is needed for analytics to reconnect teachers and students at a unit and program level, and help unit and program coordinators support those students who need it most? We present the approaches and findings from a pilot initiative where a freely available learning analytics platform allowed unit coordinators to define their own contextually unique measures of engagement and allowed program coordinators to see across units. We discuss the forms of outreach afforded by the initiative, the support provided to coordinators, and the implications of learning analytics that are not one-size-fits-all on using data meaningfully to support human connection.


Background Learning analytics and student support
Learning analytics have been a part of the higher education landscape for over a decade. Catalysed by the increased availability of data about students and their learning behaviours, early work in the field focused on supporting students by ostensibly predicting their future performance. The field has also addressed analytics around social networks, discourse, assessment and feedback, learning design, and even student affect (Joksimović et al., 2019). Throughout this journey of learning analytics research and practice, the fundamental aim has always been to understand and enhance student learning. To do this, learning analytics relies on a complex interplay of sociotechnical elements including data, software and algorithms, teachers, university administration, policy, ethics, and, of course, students. Work around predictive modelling, for example, can involve machine learning algorithms that consider demographic and learning behaviour data from learning management systems, presenting to teachers a visual representations of predicted student academic outcomes that can then inform educational interventions (Herodotou et al., 2019). These interventions can range from automated, system-generated 'nudges', to personalised messages designed by teachers, to invoking institutional student support mechanisms (Wong and Li, 2018;Arthars et al., 2019;Lim et al., 2019).
Although learning analytics holds much promise, it has faced a number of challenges. Some notable criticisms include the lack of consideration of learning design or learning and teaching context and whether teachers and their pedagogical knowledge are involved in the use of learning analytics (Guzman-Valenzuela et al., 2021). Predictive analytics have also been challenged about their reliance on standardised variables or indicators, with an emphasis that instructional contexts are critical to consider when using learning analytics, and that generic one-size-fits-all models of student success are overly crude and ineffective (Gasevic et al., 2016). Moreover, there have been impassioned calls to rediscover the humanity in learning and teaching through data, and reconnect teachers with students at a human level through an ethic of care (Parkes et al., 2020): … interventions should start with (and be built around) human interactions. Genuine staff-student interaction is increasingly difficult to achieve in [higher education]'s massification environment where students are often reduced to mere numbers… however, [learning analytics] offers staff an opportunity to initiate contact with those specific students who may benefit from such human interaction. (p. 120).

Context and urgency of using program-level analytics for student support
Indeed, highly individualised teacher-driven approaches to supporting students are a fixture of many higher education teachers' roles. Rarely, though, are there initiatives implemented across multiple units of study or across departments or faculties. This is an area where the increased availability of data and learning analytics may have a key role to play. Additionally, recently introduced federal legislation in Australia affecting access to the Higher Education Contribution Scheme (HECS-HELP), which is offered to domestic students to reduce and offset the cost of their course fees, has produced a more urgent need to support students in a targeted way. The Job-ready graduates package (JRG), announced in 2020 by the former federal government, introduced several reforms including restricting access to HECS-HELP if students do not maintain a pass rate of 50% for their enrolled units. When these reforms were announced it was quickly identified that teachers, being the people that students have the most and closest contact with at university, were likely best placed to provide additional support to those who may be impacted by these changes. However, with the impacts only being realised at program level across multiple units of study, it was important to both address the nuances of teaching and curriculum design within individual units as well as be able to measure and support student engagement across units.

Measures of student (dis)engagement
Kahu's (2013) framework views engagement as a multilevel phenomenon of socio-cultural processes. These processes can be influenced by institutional and personal factors, and are obviously embedded within a wider social context. Things that can influence engagement include structural influences such as a student's family, support and workload, and the curriculum, assessment and policies of an institution. In addition, there are factors such as the teaching staff, their workload, students' motivation, personal identity, and feelings of selfefficacy. Finally, engagement also includes behavioural and cognitive factors such as ability to concentrate, participate and interact.
In this paper, we present the methodology and early findings for a pilot initiative to measure and support student engagement across programs. Importantly, our approach privileged the nuances of individual units of study as determined by teachers and their understanding of how the design of their curriculum related to indicators of student disengagement. We describe how various educational contexts informed the indicators and rules used to identify disengaged students at a unit level and how this was then aggregated to provide tailored support at a program level. Internally the goals of this were to address the immediate needs of supporting students before they were impacted by the new punitive legislation, and here we also demonstrate that individualised approaches to learning analytics can be meaningfully implemented at scale across multiple units.

A teacher-driven approach to measuring and supporting student engagement From unit-level to program-level learning analytics for student support
In the initial pilot in semester 2, 2021, the Bachelor of Science (BSc) and the Bachelor of Liberal Arts and Sciences (BLAS) were involved. The BSc has a pool of options rather than a set of core units and so an analysis of enrolment patterns was performed to identify the minimum number of units to ensure >95% of the total cohort was covered in at least two units of study. In the BLAS, there is a compulsory unit and so this and the most commonly co-enrolled units were selected. Ensuring that a majority of students were in at least two units was intended to provide accurate engagement information across their studies rather than just in individual units. In semester 1, 2022, the Bachelor of Engineering, Bachelor of Advanced Computing, Bachelor of Arts, and Bachelor of Economics were added following a similar model to ensure adequate coverage across the major liberal studies degrees with large domestic cohorts. Alongside the approach of picking commonly-taken units, students in these degrees often take units from other faculties and this assisted ensuring coverage whilst minimising the overall coordination effort. Each unit coordinator was supported to identify appropriate engagement indicators for their units and to provide personalised communications for students flagged by these indicators. For each of the degrees, a senior faculty academic leader (the 'program coordinator') was also provided with aggregated data across their degrees so that students appearing to be disengaged across their studies could be contacted by an experienced academic advisor.

Helping teachers define, measure, and support (dis)engagement
Our underlying assumptions were that unit coordinators knew their units and cohorts better than someone outside the unit, and that they would have a better idea of what might constitute (dis)engagement. In large universities such as ours there is a temptation to use institution-wide indicators such as LMS access or predictive analytics to identify at-risk students, but broad-brush approaches run the risk of targeting the wrong students and missing those who are really struggling. They also ignore important variations in approaches by individual teachers and unintentionally separate teachers from their responsibility in supporting students by 'outsourcing' it to central teams. As Kift (2008) powerfully stated, a successful transition is "everybody's business". Each unit coordinator in the pilot met with a member of the project team to discuss potential indicators and rules that might be used to identify disengaged students in their respective units at two points in the semester. The first point ('early') was in the lead up to census date and the second point ('late') was towards the end of semester when students would be beginning to prepare for exams or submitting final assignments. Our aim was to minimise extra workload for teachers, so we focused on data already available for each unit; according to Kahu's (2013) framework, these were primarily related to behavioural engagement and proximal consequences of engagement.
Once we had settled on combinations of indicators (e.g. quiz score or attendance count) and rules (e.g. less than 50%) to be used in each unit, the project team arranged for this data to be made available in the Student Relationship Engagement System (SRES) in an aggregated form. Typically, this form of nuanced (that is, not one-size-fits-all) learning analytics is difficult to achieve with large cohorts, across multiple units and programs. SRES was critical here because of its ability to allow unit coordinators to select, collect, and analyse data that is meaningful for them in their teaching and unit requirement contexts (Liu, D., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A, 2017). The system then allowed the project team to combine these engagement measures across units to identify students who may be disengaged across programs.
Across the 46 units in the pilot (23 each semester), there were a wide variety of different indicators and rules chosen by unit coordinators to determine student (dis)engagement (Table 1), closely linked to the learning design of each unit. Because student contact was made at two points in each semester, coordinators were also able to adjust the indicators and rules according to the most relevant data at those points in time. The indicators selected early in the semester differed substantially from those selected late in the semester ( Table 2). The wide variety of indicators selected was also apparent, as was the nuance of most indicators towards the learning design of each unit.

Supporting students, at scale
As discussed above, once the individual unit data were aggregated at both the 'early' and 'late' stages and across all units, program coordinators prepared targeted outreach through SRES. This took the form of a personalised email, complemented by a text message directing students to check their email. These messages provided information to students about the possible impact of the JRG on their study progression, student support programs, an encouragement to contact individual unit coordinators, and an offer to support the student directly. These emails were crafted to be caring and human and in the authentic 'voice' of the program coordinator. 'Early' emails included information about census date, and 'late' emails included both information about the upcoming discontinue fail date, and final exam or assessment support. Unit coordinators were also provided with information about students who showed disengagement indicators in their individual units of study. The emails sent by program coordinators resulted in an average open rate of just under 30%, and approximately 20% of the students contacted responded either with further information or a request for support. Student responses indicated that disengagement was primarily due to (a) personal or health challenges with COVID a major factor, but that (b) work commitments and (c) problems with enrolment systems also affected their engagement. Program coordinators supported these students individually, referring them to other services where relevant.

Discussion and future work
The variability of indicators chosen by unit coordinators to measure student (dis)engagement in this pilot shows the importance of teacher-driven use of learning analytics. Across the two semesters, 14 distinct engagement identifiers (Table 2) were decided upon in concert with unit coordinators across 46 units of study. This revealed that teachers have very different conceptions of what constitutes positive engagement in their contexts, linked to varying teaching approaches, unit requirements, class types, assessment tasks, etc. LMS logins or pageviews were a dominant indicator used in 'early' semester 2 2021 in units where other indicators could not be easily identified, but as noted above are broad-brush, and feedback from unit coordinators suggested that they had little predictive value. Useful analytics are the product of intentional design and many of the units in the pilot already had highly individualised but long-standing context-driven approaches to supporting student engagement using SRES. While this project illustrated the need for a good deal of support when implementing meaningful engagement support strategies, there is a clear benefit to teachers, units, and students in the reflective process required to develop these. In turn, approaches to utilising learning analytics must afford flexibility in order to be appropriately informed by teachers and their learning designs. Future work in this space should involve the evaluation of the extent indicators recognise student (dis)engagement within individual teaching contexts.
The 14 distinct engagement identifiers can be separated into two types (Table 2) and a pattern in their use was seen; behavioural identifiers were more likely to be used in the early stages of semester, and achievement identifiers in the later stages. This may largely be due to the availability of data, with behavioural data more prevalent earlier in the semester, but could also relate to a tendency to set 'easy' assessments earlier in the semester which are considered less reliable indicators of eventual success. We propose here that we collectively may be over-privileging achievement as an indicator for engagement, but also reflect on the fact that while achievement is the ultimate indicator for success in individual units of study, students' eventual success at university is a combination of academic achievement but also the sense of belonging they develop through their education journeys. This involves genuine connections and care, a process that can be supported by using data in a nuanced and flexible way.