Developing a learning analytics resource with meaningful data for first year teachers

It is known that due to the large, diverse sets of data captured in learning analytics, clear display of information is crucial to its success. Here, we describe a concise learning analytics resource, the Cohort Snapshot, that has been developed using a human centred approach, with input from experienced academics from across the institution. Meaningful data is presented as a unit level summary and includes program enrolment, student demographics, grade distributions and LMS activity. The data that was used to develop the resource was accessed via the university’s data warehouse which was synthesised using the R programming language. The Cohort Snapshot provides teaching academics, including sessional staff, access to data, to allow the adaption of teaching pedagogy to meet the needs of increasingly diverse student cohorts in a regional Australian university.


Introduction
Learning analytics are applied in higher education for a variety of purposes from providing information on student demographics to monitoring student engagement and patterns of study and success. The usage and reliance on an ever-widening choice of data is only growing (Kitto et al., 2020), however the practical outcomes can be underwhelming. Part of the problem is that technical experts in the field of data science often develop tools without adequate input from teaching academics (Kitto et al., 2020;Jivet et al., 2017). As a result, decisions around the development of tools can be driven by data-related factors rather than the practical context. Thus, there are calls to ensure that educational scenarios inform the data generation rather than technology functionality (Kitto et al., 2020;Jivet et al., 2017) by taking a human-centred approach to learning analytics (Buckingham Shum et al., 2019) and incorporating a breadth of experience in learning analytics development (Sarmiento & Wise., 2022). There is extensive data available, however, how to curate and present the data to teaching academics remains a challenge.
Charles Sturt University is a large regional university with a high proportion of students from a low socioeconomic (LSES) background and regional, rural and remote (RRR) students; two groups of students who have been shown to be less likely to complete their program compared to individuals from metropolitan areas (Department of Education, Skills & Employment [DESE], 2019). Having a high proportion of students from non-traditional backgrounds presents pedagogical challenges in teaching. In 2008, the Bradley Review recommended that participation of students from under-represented backgrounds including RRR, LSES and Australian First Nations individuals be increased in higher education to promote economic sustainability in Australia. The proportion of students from non-traditional and disadvantaged backgrounds continues to increase (National Data -NCSEHE). With widening participation, teaching practices need to be adapted to meet the needs of increasingly diverse cohorts of students to increase student retention and success of equity groups in Australian universities (Devlin et al., 2012).
It is essential that teaching academics are engaging and provide inclusive learning environments to support students as they transition to university. Universities continue to investigate ways to improve student experience and engagement, particularly in first year, to increase overall student success and progression (Kift, 2015). Providing students with a sense of connectedness and belonging impacts student engagement and student success (Kahu & Nelson, 2018) and students from non-traditional and under-represented backgrounds require additional support (Devlin et al., 2012). Student data can be used to provide teaching academics with deeper insights into the diversity of their cohorts and to create a more inclusive teaching environment (Muir et al., 2019).
Here, we model the design and distribution of a concise learning analytics resource, the Cohort Snapshot, that has been developed using a human-centred approach with input from experienced academics from across the institution. Much of the data presented in the Cohort Snapshot is available to staff across multiple platforms, however, accessing the information required a significant time investment and/or data requests. Sessional or new academics face additional challenges including learning what information is available, identifying the correct tools and obtaining authorisation for access, which may not always be granted (Baik et al., 2018). As a result, unit coordinators have varying levels of understanding of the learning needs of their cohorts. . Some academics are left in the dark, others in dappled light with only a few living in sunshine. We should all be able to enjoy the sunshine. In addition, access to this information needs to be timely. The Cohort Snapshot is composed of data analyses of regularly updated data from across multiple sources to provide an overview of a student cohort in a unit of study. Providing teaching academic staff with a simple tool for reviewing students within their cohort will enable staff to employ adaptive and inclusive teaching pedagogies as the number of students from diverse backgrounds continues to increase. Learning analytics data from a range of sources are used to synthesise a single resource that provides a meaningful unit overview; delivering unstructured data to teaching academics as actionable information

Methods
In 2021, the Cohort Snapshot was developed to share with academics who teach into approximately 100 first year units as part of a larger retention project (Linden, 2022). Development of the Cohort Snapshot was based on a theoretical framework, SHEILA (Supporting Higher Education to Integrate Learning Analytics) (Tsai et al., 2018). The key dimensions from this framework that we have implemented pertain to understanding the context (information poor sessional and new academics), effectively engaging stakeholders, establishing monitoring frameworks and understanding the capacity to effect change. Central to the design was understanding the internal capacity for change needed rapid synthesisation of appropriate data sources into a resource that was not going to overwhelm already busy academic staff. Stakeholder engagement was implemented through training workshops for sessional staff to use the Cohort Snapshot as well as information sheets, along with the use of an iterative design process implemented over three semesters where we monitored feedback from various stakeholders such as Division staff, school leadership and staff from across the three faculties and made improvements for the next session. Feedback included making headings clearer as well as some more specific session information in some sections. The data that was used to develop the resource was accessed via the university's data warehouse and included students' academic progress, unit and program enrolments, student demographics, LMS activity and grades. All data was presented for a given unit and compared against the university average. A data scientist synthesised the templates using the R programming language to write scripts that summarised and displayed the data, synthesising a clean skin html file for academic staff to view. Unit level data is displayed in html and updated regularly prior to the start of semester until census as during this time the enrolment data is constantly changing. The secure link to the html page (click here to see a mock Cohort Snapshot) was distributed via an email containing information of the interpretation and significance of the report (click here to view) and is now available to all unit coordinators in the University.

Results and Discussion
Who is enrolled?
The first section of the Cohort Snapshot lists the most common programs that the students in the unit are enrolled in and the corresponding progress rate (Figure 1). This allows academics to ensure that resources are relevant to create an inclusive environment and increasing engagement (Bakharia et al., 2016;Sanger & Gleason, 2020). Knowing who is enrolled in a unit is of particular importance for foundational units where you have large cohorts of students from many programs, with some programs having higher progress rates than others as seen in Figure 1. Academics can ensure that they have focused tutorial questions or case studies that include the various disciplines of enrolment within the unit, and adapt assessment questions and exam question options to have continuous student engagement throughout the semester.
As shown in previous studies, program enrolment knowledge will support student engagement and overall graduate learning outcomes of application of knowledge into their program (Sanger & Gleason, 2020). Feedback from academic staff suggests they were able to quickly apply small modifications that make the resources relevant for a diverse range of programs, with a focus on the programs that have a lower progress rate than others for the unit of study, "I found it helpful for preparing this weeks tute actually.

[…] I have lots of different disciplines attending my class, […] I tend to give examples from within my field or my own research but this
week I tailored it to the policing students, so I found that really helpful as did the students."

Student demographics
The cohort snapshot provides academics with a list of demographics, such as the proportion of enrolled students who are First in Family to attend university and the average age of students within the unit compared to current enrolment average at Charles Sturt ( Figure 2). This can be used by the academic at an individual level by looking at specific demographics and have a targeted approach to support services provided, for example ensuring the textbook is freely available online through library services with clear links. Academics could also ensure that if they have a high number of regional or remote students that the textbook is downloadable to avoid issues with internet during the semester of study. Demographics could also be looked at in combination with each other, for example if there is a high proportion of students that are LSES and First in Family focusing awareness on free support services, including webinars and scholarships that the university offers to support students for successful transition to university, and progression through the program. Literature shows that students without family or social history of university are unaware of the volume of assistance available and therefore creating awareness is vital for the retention of these students. The embedding of services into foundational units allows students to begin creating relationships with various services at selected timepoints in their program (Sanger & Gleason, 2020;Tualaulelei et al., 2021).

Figure 2 Student Demographics
Understanding of your program needs and support required by the various demographics can increase engagement, for example implementing links with specific services such as the Division of Library services or Academic Skills if there is a high proportion of First in Family or regional and remote students (Sanger & Gleason, 2020;Tualaulelei et al., 2021). The individual demographics within a unit allow staff to consider verification to their teaching style to help with student success (Sanger & Gleason, 2020). Feedback from staff was positive with instant actions being taken, "In terms of my cohort snapshot, I had not previously thought to specifically discuss the academic skills team to my students [...]

but now I plan to"
Charles Sturt has a high proportion of online students. As seen in figure 2, within this unit the percentage of students studying online is 98%, shown by the blue circle. Academics need to focus on this demographic with specific strategies of engagement creating opportunities for meaningful interaction and online spaces for peer to peer communicating. Academic staff have since adapted supportive learning environments through set up of Jamboards, Padlets and smaller discussion groups to create safe spaces for meaningful interactions and have reported increased levels of student engagement. Identifying and engaging with the online students improves first year experience as well as increases student success (Tualaulelei et al., 2021;Bowden, 2021;Kift, 2015).

Former Fails
Providing a list from the students who are currently enrolled who have failed the unit previously allows staff to reach out to those students. Proactive contact has been shown to increase the positive student experience in units and contribute to student success (Sanger & Gleason, 2020). By knowing students' history with the unit, academics can have a targeted approach at supporting these high-risk students from the very beginning. For example, in semester 2 2022, 46 students in one unit and 18 students in another unit were contacted by the subject tutor regarding a former fail and encouraged to access additional support services including one-on-one tutor support. Conversations with students could begin by pointing out any changes that have occurred in the unit since their last attempt. Academic feedback identifies that they are not always aware of the repeating students, "Not realising previously how many students fell into this category, the snapshot has shown me that there are quite a few in this position, and that they may require further support." It has been evidenced that starting conversations with students gives the students an opportunity to feel connected not only with the unit content by with the academic if further support is required such as conversations around available support services (Sanger & Gleason, 2020;Tualaulelei et al., 2021).

Academic History and LMS activity
The Unit Academic History section displays the unit progress rates, proportion of students who receive a zerofail grades (data not shown) and the grade distribution over the previous 3 years of the unit against the university average ( Figure 3a). Continuing academics, and more often sessional staff, do not always have easy access to the historical aspect of the unit grades or trends. Setting out clear expectations for students and creating supporting documentation and resources based on historical data is difficult if staff do not have access to it (Baik et al., 2018;Bowden, 2021;Heffernan, 2018;Muir et al., 2019). Figure 3b shows the activity of students across the first six weeks of semester and the number of interactions students have had. The LMS becomes available to students in week -2 and in that week that approximately 30% of students had accessed the LMS with 15% accessing the LMS once and 15% accessing 2-5 times. Understanding the views and contact students have with the LMS can assist teaching academics with the type and number of resources that they have available on their LMS for students at particular times during the semester. This also helps teaching academics to design activities that may help identify disengaged students that might be likely to receive a zero-fail grade (Linden et al., 2021). Importantly, high student engagement is directly linked to successful student outcomes (Foster & Siddle, 2019).

Conclusion
The Cohort Snapshot has enabled academics to make informed variations to units to ensure safe teaching spaces with meaningful interactions, targeted support and improve their teaching pedagogy to ensure student success. The Cohort Snapshot delivers data from various sources in digestible, easy to access form through clear presentation of the data to benefit teaching academics, students and the institution. The theoretical framework and the ways of informing the teaching academics of the potential application of the data is what makes this resource impactful. The resource results in a deeper level of understanding of student cohorts and can be used to create inclusive teaching environments, improve curriculum and assessment design and ultimately improve educational experiences which will result in higher completion rates and increased student satisfaction. The Cohort Snapshot is currently a live dashboard accessible to teaching academics. Future improvements will include specific examples of how the data can be implemented within units. The Cohort Snapshot ensures that key learning analytics are easily accessible to all academic staff as they can enjoy the sunshine as they work in supporting their students for successful outcomes.