Enhancing Higher Education with Learning Analytics in the digital age

Learning Analytics (LA) in higher education is emerging as a powerful tool for optimising learning and teaching outcomes at a time when technology is revolutionising the delivery of education generally. In this paper we explore LA implementations in five diverse Australian universities. Our study draws on DeLone and McLean's (2003) model of information systems success to qualitatively evaluate the various implementations. We highlight both the successes and challenges encountered by each university. In taking this approach we aim to contribute to laying a foundation for optimal use of LA systems. This research reveals a degree of pervasive uncertainty as to how Learning Analytics is defined. While substantial research exists on LA systems and their usages, there are still limited empirical studies that examine the success of LA implementations, as many Higher Education Institutions (HEIs) are still in very early stages of their LA system implementations. The outcomes of the study include recommendations for universities to consider when implementing LA systems or improving existing implementations which is an addition to both the research areas of LA research and information systems success research.


Introduction
Data analytics is a maturing discipline that has been used for many years by commercial enterprises for competitive advantage (Rubel & Jones, 2016).It is becoming a new source for transforming many human activities (Rates & Gašević, 2022).Enormous amounts of data have been collected about consumers and their behaviours and this data is analysed by tools to help refine enterprise practices (Rubel & Jones, 2016).In the last decade Higher Education Institutions (HEIs) have started using analytic tools to collect information about students and how those students navigate their way through university (Rubel & Jones, 2016).Ifenthaler & Yau, (2020) posit LA are socio-technical data-mining practices that exists within educational contexts.Through the collection and analysis of large amounts of data, educators can gain insights into innumerable elements of learning and teaching (Wong, 2017).Empirical studies have shown LA offers unique opportunities to support contemporary learning and teaching practices (Kovanovic et al., 2021;Ifenthaler & Gibson).Ifenthaler & Gibson (2020) note the field of LA research is still emerging, but it has added new frameworks, methodological approached and empirical investigations relevant to educational research.Tsai et al., (2020) confer that LA has the potential to enhance education by providing insights for learning design and pedagogy that would otherwise not be available today with the availability of data and technological innovation.LA system data also facilitates ongoing feedback between educators and students which has the potential to improve the learning experience for students (Avella et al., 2016;Wong, 2017).Over the last few years there have been many studies that examine LA tools and how they can assist with a range of activities at HEIs (Rates & Gašević, 2022).LA is becoming an emerging technology used at HEIs (Gašević et al., 2014) and the research is continuing (Clark et al., 2020).
While substantial research has been performed about LA systems and their usages, many HEIs are still in the early stages of their LA system implementations (Colvin et al., 2017;Ferguson et al., 2019).This work focusses on the success of LA implementations at HEIs but also delves into the factors that detract from system success.However, it must be noted that while educators to vast amounts of data, decisions made using LA data do not always lead to improvements (Ferguson et al., 2019).Much work in the LA literature examines the tools being used by educators rather than evaluating the use of LA within HEIs more broadly (Ferguson et al., 2019).There is strong consensus by the researchers in the field of LA, the definition of LA from the 1st conference on LAK2011 is the most fitting one.LA is "the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs" (Long & Siemens, 2011, p. 34).This study is part of a larger study which concluded that LA are essentially information systems and therefore is a system which collects, processes, evaluates, analyses and reports on data for the purpose of learning and teaching (Clark et al., 2020).This paper explores the novel application of the DeLone & McLean (2003) model of information systems success to five large, multi-campus, Australian universities.The study focusses on the LA systems in each university and how these systems can improve learning and teaching for students.The LA systems are examined in terms of participant's perceptions on whether the system is a successful implementation.The LA systems are rightly viewed as information systems.This could be useful since DeLone &McLean's ( 2003) model provided a comprehensive framework for assessing the performance of information systems in organisations.This model has been rigorously tested over many years in a range of applications, but these studies have had a quantitative focus.There are very limited studies examining LA systems success using a qualitative lens.Much of the research in LA focuses on educational examples including learning design, student engagement and student performance.Studies are lacking in areas such as looking at the perceptions of users, on whether the implementation of the LA system is effective.This study addresses that need.The potential of this field is great, though there is still a need to build the theoretical and empirical base that provides clear evaluative procedures for matching observed student interaction behaviour with course and program level learning goals and outcomes (Siemens, 2013).

Learning analytics
LA is considered a viable approach to the enhancement of learning and teaching by harnessing large amounts of data generated by learning systems (Alzahrani et al., 2021).Popular uses for LA include predicative analytics, the early identification of students at risk, improving the learning experience, monitoring student system site activity, informing pedagogy and providing feedback at scale (Tsai et al., 2021).It has been demonstrated that LA interventions assist in students learning by identifying problems early and providing the necessary support when it is needed (Wong & Li, 2020).Personalised feedback has also been suggested as a positive case for intervention using LA (Wong & Li, 2020).LA has the potential to make implicit learning design practice explicit, thus providing educators with a pedagogical context for the interpretation of LA findings which can then support intervention (Holmes, et al., 2019).Even though these studies have furthered the field of LA research, they most often examine one-off applications of LA or ad hoc systems (Nguyen et al., 2021).Educators would arguably benefit from attaining a deeper understanding of how to improve design practices to guide learners to better learning outcomes (Siemens, 2019).However, the benefits and capabilities that LA can provide are not always adaptable to all HEIs (El Alfy et al., 2019).Dawson et al., (2019) argue LA research has not risen to the challenge and delivered on its promise to improve educational outcomes.Given that Australia has a mature, sophisticated higher education sector, plus a strong commitment to the scholarship of learning and teaching.It is perhaps surprising that LA has not played a more influential role in the development of evidencebased learning and teaching strategies (Universities Australia, 2023).Australian universities have been using technology to make higher education more flexible and accessible, with the overall goal of being more productive, but they have not yet seen the full potential of LA to support decisions that are data driven.Largescale adoption of LA continues to be problematic (Dawson et al., 2019).That said, we argue LA has more value to offer HEIs than is currently afforded.We examine five detailed case studies that aim to illuminate the issues HEIs should focus on for successful implementations.

Information systems success
Researchers in the field of information systems have been evaluating information systems using models and theories for more than three decades, seeking to understand what constitutes successful information systems and information technology implementations in organisations (Gable et al., 2008;Davis, 1996;DeLone & McLean 1992).Fundamentally, an information system involves the gathering, processing, distributing, and the using of information by input, processing, and output, with a storage and feedback component (Beynon-Davies, 2013).We therefore make the case that LA can be classified as an information system therefore a Learning Analytics Information System (LAIS) by virtue of the LA systems under investigation undertaking to process, collect, evaluate, analyse, and report organisational data for the purpose of decision making (Campbell et al., 2007).Information systems implementations are not always successful (Nguyen, Nguyen & Cao, 2015).In this paper, the DeLone & McLean (2003) model is applied to five Australian implementations of LA systems utilising the body of work that exists within the information systems success literature.The authors are using the updated DeLone & McLean model (2003).The updated model is derived from the original model ( 1992) with contributions made by many information systems scholars to improve the model (DeLone & McLean, 1992).The model has been cited in thousands of papers and has been one of the most influential theories in contemporary information systems research (Nguyen, Nguyen & Cao, 2015).The model provides a solid foundation for examining the success of LA implementations, particularly in relation to the improvement of learning and teaching outcomes.DeLone & McLean (2003) use three classifications to categorise information systems success (Figure 1).The first area examines the Information Communication Technology (ICT) system or functionality of the system.The second part focuses on the usability of the system and how users interact with the system and whether the system interface is user friendly and also evaluating whether the system achieves its intended goal.The third part focusses on the overall net benefits of the system, including how the information system's overall impact is felt as both an individual and from an organisational perspective (Nguyen, Nguyen & Cao, 2015;Beynon-Davies, 2013).(2003) to an e-learning context.The authors reveal that the overall success of an e-learning application depends on achieving success at all three stages of the development of an e-learning system.That is, in system design, system delivery and the system outcome.Figure 2 shows the author's adaption of the original model as applied to e-learning systems implementation.We examine the potential of LA to deliver appropriate functionality and usability to create an information system that delivers value to students and staff to improve learning and teaching outcomes, thus helping students to engage more closely and productively with their studies.A review of other success and technology acceptance models was performed as part of a larger study which reflected on the use of (a) The Theory of Reasoned Action (TRA), (b) the Technology Acceptance Model (TAM), (c) The Unified Theory of Acceptance & Use of Technology (UTAUT), & (d) the Innovation Diffusion Theory model (IDT) (Ajzen & Fishbein, 1975;Ajzen, 1991;Rogers, 1995;Venkatesh et al, 2003) in information systems research.This study did not adapt the TRA model due to being interested in the perspectives of the use of LA by participants and not in their behavioural intentions or attitudes towards the use of that system.The TAM model has been tried and tested over the years and has only proven to successfully predict about 40% of a system's use (Venkatesh et. al, 2003).Whereas the UTAUT has been criticised for taking a narrow perspective on the use of information technology (Shachak et al, 2019).This study focuses on participant's perceptions of LA system use at different case study sites.It considers the success of the system in relation to participants' perceptions of information quality, intention to use, system quality, service quality and overall net benefits.The IDT model was not used as it was not seen as a good fit in terms of the research questions.This study focused on participant's who may or may not be individuals who are likely to accept new ideas or cope with high levels of uncertainty as for those under the IDT model (Rogers, 2003).The DeLone & McLean (2003) model was used in this study as it enables the researcher to examine more closely, the system quality, information quality, service quality, intention to use/use/user satisfaction and the net benefits of the information system.The perceptions from users using LA as in this study, can be obtained to see whether they are satisfied or dissatisfied with the system and in which way the system can be improved to achieve success.By examining the dimensions of the information systems success model an information systems implementation can be improved for all users.

Research approach
Case study research is a reliable approach used in qualitative information systems research (Myers, 1997).Case studies have been used amongst a wide range of disciplines in projects seeking to understand complex issues (Yin, 1984).Case studies are used in information systems research as they offer insights not available with other approaches (Rowley, 2002).As the generalisability of the case study approach is sometimes questioned, it is advisable to establish validity by cross-referencing multiple case studies.Multiple cases are equivalent to multiple experiments as opposed to having a single case or single experiment (Rowley, 2002).Yin (2013) believes that case studies should not be studies in isolation but rather as an interaction between the case and its context.Case studies work well with exploratory investigations where there is little or no prior knowledge of reality or of a phenomenon (Järvinen, 2001).Qualitative methods like the case study approach allows realworld events to be studied in context.This includes the cultural aspects of people, organisations, and groups (Yin, 2011).An interpretivist perspective was taken in this research which views the world as a social construction of reality that is interpreted, and experienced by people and their interactions within the wider social systems in which they exist (Cantrell, 2001).Qualitative data analysis was performed using template analysis.This method has evolved out of the wider tradition of thematic analysis (King & Brooks, 2016).Template analysis attempts to seek a balance between flexibility and structure in terms of the way it handles textual data and is one of the styles of thematic analysis (King & Brooks, 2016).The first step in template analysis involves using codes that are defined by the researcher.This involves the use of priori codes drawn from research readings of theory (Blair, 2015).The present study had priori codes at the start of the analysis which were drawn from the DeLone & McLean model (2003).These resulted from a careful study of the problem under investigation and the theoretical motives driving the study (Schwandt, 2014).Codes were then derived from the language used in the DeLone & McLean model (2003).After the codes were derived, the data was then examined and sorted into a scheme (Schwandt, 2014).The interview questions used for the DeLone & McLean (2003) model were formulated based on the work of similar qualitative studies and mapped to quantitative categories (Ojo, 2017;Hopsapple & Lee, 2006).The interview protocol included 32 questions categorised into background data, systems quality, information quality, service quality, intention to use/use, user satisfaction and net benefits (impact).In addition to the categories in the updated ( 2003) DeLone & McLean model, questions about the impact of the system on learning and teaching were also added.

University case studies
Of those approached using a random sampling method, five Australian universities agreed to participate as case studies (Seawright & Gerring, 2008).These universities were demographically diverse by chance; three of which are metropolitan and two are regional or a mix of regional and metropolitan.The demographic spread of the data sources provides an inclusive view of the LA implementations in Australian universities.University one is a regional university in an agricultural region.Given its remoteness from major population areas, it specialises in online programs and has earned a reputation for excellence.A large proportion of enrolments at this university are online students.These students come from various backgrounds, including low socio-economic segments of society, first-in-family, mature aged and professionals seeking career enhancement with further study.University two is a metropolitan public research university with a low to middle socio-economic catchment.A higher-than-average proportion of its students are first-in-family.In some cases, students have a lower entrance score than those of their counterpart universities in the same city that were not part of the study.University three is a regional public university with campuses in multiple states.Like university one, it specialises in online service delivery, for which it is a recognised leader in Australia.This university draws its students from diverse background across regional Australia, including low socio-economic status, first-infamily, mature aged and professionals.University Four is a metropolitan public research-intensive university serving a middle to high socio-economic demographic.This university is perceived locally and internationally as prestigious, attracting students with high entrance scores.University Five is also a metropolitan public research-intensive university.This university has two campuses with one being based in a central business district.

Research results
To explore the success of LA system implementations, interview participants were asked questions based on the DeLone & McLean (2003) model.The universities were chosen at random, and three to five participants were recruited from each university.The study had twenty-three participants in total.The recruitment process differed at each university.Some universities had rigorous ethics processes to follow whereas other universities had key staff who recruited the participants.Participants roles included managerial roles, teaching staff, LA support staff, user designers and data scientists.Initially respondents were asked about the types of LA systems implemented at their respective universities.

DeLone & McLean Model (2003) constructs
According to Petter et al., (2008) system quality is constituted by ease of use, system flexibility, system reliability, ease of learning, intuitiveness, sophistication, flexibility, and response time (Petter et al., 2008).Due to the interpretivist nature of this research, (allowing the participants to focus on what they considered important) not all these factors were mentioned during interviews.The DeLone & McLean (2003) model consists of six interrelated dimensions of information systems success: system quality, information quality, service quality, use, user satisfaction and net benefits.Under the constructs of information quality, system quality, and service quality, participants were concerned with ease of use, system reliability, relevance, understandability, and accuracy of system outputs.Other factors shown to be important under the constructs of use and user satisfaction included technical competence and expertise provided by system support staff and nature of use and user satisfaction.None of the participants indicated that they enjoyed using LA, which is surprising considering its purpose.Nor were they forthcoming about assessing the overall success of LA.In terms of net benefitsdecision making and net benefitslearning and teaching, participants related to the constructs of decision-making and productivity, improving learning and teaching strategies and improved learning teaching and design.Participants strongly believed that the use of LA could lead to better student outcomes for universities.

Conclusion
LA systems have the potential to improve decision-making around learning and teaching processes including improving pedagogy (Prinsloo et al., 2021).Categorising LA systems as information systems is a useful designation with researchers thus able to draw on the wide-ranging extant research on information systems success to evaluate how to best use LA systems and in their implementations.This paper presented a novel application of the DeLone & McLean (2003) model of information systems success to LA at five large, Australian universities.With said model, we evaluated how these systems could possibly improve learning and teaching for students.Some of the major issues raised by participants were explored in terms of the DeLone & McLean (2003) model constructs of system quality, information quality, service quality, intention to use/use, user satisfaction and net benefits.A major theme arising from the research is that it is not clear in the university sector (at least in the five universities studied) what is meant by the term 'Learning Analytics'.Several participants wanted clarification from the researcher at the outset as to how LA was defined.There was also a degree of uncertainty at each university as to how LA systems are properly constituted, i.e., what elements in what configuration.It is revealing that each university had a different concept of what an LA system is.Our analysis focused on the perceptions held by participants and their believes about what the LA system is.Therefore, this made it difficult to assess the success of a system, given the differing terms in which LA systems are defined.The themes emerging from system quality were the dual concerns of ease of use and system reliability.It became clear that LA systems need to be easy to use and learn as well as have reliable functionality and performance to be useful at improving learning and teaching outcomes.In the information quality dimension, understandable and accurate system outputs were expected by participants.Participants also thought that possessing data literacy skills is important to ensure the correct interpretation of results from system outputs.In the service quality dimension, training from competent support staff was viewed as necessary and participants believed that more capabilities could be offered in the LA system, but they were not sure about what new capabilities could be added.In the dimension of intention to use and actual use, the construct of intention to use was not mentioned by participants as they did not express their attitudes towards using the LA system.Some use case examples were provided including using LA data to target at-risk students.The fact of whether the use of an LA system was voluntary of mandatory did seem to affect the use of the system.Participants also expressed what they felt as limitations with the systems at their respective university.User satisfaction was assessed by participants as overall satisfaction with the system.The type of LA implementation strategy each university had was also a factor that impacted on satisfaction by participants.The net benefitsdecisionmaking dimension revealed that both improved decision-making, and informed learning design were important elements for participants.The net benefitslearning and teaching dimension had three major themes attached to it.First were that these were that LA systems need managerial support to be successful.Second, that LA can be used for activities like course re-design.Third, policy needs to be developed around data use for LA system implemented at universities.There were limitations in the qualitative use of the DeLone & McLean (2003) model and in the sample size of the study with only five universities participating.Some original constructs from the model were not useful to this study as participants did see them as important to discuss in terms of LA systems use.Also, it was difficult to determine whether the LA systems were successful due to the ambiguity surrounding the definition of LA by participants.Although, they did believe they were successful.Another finding revealed that viewing LA systems as information systems can enables them to be viewed as software and the associated practices that go with the use of the software.

Figure 1 :
Figure 1: The updated DeLone-McLean information system success model (DeLone & McLean, 2003).Most of the DeLone & McLean (2003) research to date has focused on quantitative studies, but some qualitative studies do exist.In 2008, Petter, DeLone & McLean did an extensive review of the uses of the model.Of all studies examined, only the following three qualitative studies were cited, Coombs et al., 2001, Scheepers et al., 2006; Leclercq, 2007).In another qualitative study Hosapple & Lee-Post (2006) apply the DeLone & McLean(2003)  to an e-learning context.The authors reveal that the overall success of an e-learning application depends on achieving success at all three stages of the development of an e-learning system.That is, in system design, system delivery and the system outcome.Figure2shows the author's adaption of the original model as applied to e-learning systems implementation.
Questioning also took place covering all aspects of the DeLone & McLean (2003) model including systems quality, information quality, service quality, intention to use/use, user satisfaction.Nearing the completion of the interview, participants were asked under the categorisation of net benefits to describe (a) any perceived benefits that the LA systems brought to their learning and teaching in terms of decision making and, (b) how LA benefited learning and teaching overall.Priori themes based on the DeLone & McLean (2003) model were used initially in the analysis, although two new themes emerged from the data analysis (Figure3).

Figure 3 :
Figure 3: Summary of DeLone & McLean (2003) themestemplate 2 Firstly, it was not clear to participants what was meant by the term 'Learning Analytics'.The second theme that emerged was about how there was ambiguity around what constitutes a LA system at the various case study sites.Participants referred to a variety of different systems and software which they saw as serving the function of providing LA.

Table 1 : DeLone & McLean (2003) Case Study Analysis Summary Model Construct University one University two University three University four
The analysis of the DeLone & McLean's (2003) model of IS /success is presented in Table below.