Understanding university teachers’ online pedagogies through teaching analytics: what are the possibilities?

The pedagogies used by university teachers in the online learning space influence the student learning process. Understanding teachers’ online intentions and practices is therefore a critical pursuit towards enhancing student learning experiences and outcomes. This paper explores teaching analytics and how they can help us understand online teacher decisions and practices. Using preliminary data related to online quizzes and forums, the paper illustrates the potential and challenges related to the use of teaching analytics and concludes by discussing implications and recommending further work.


Introduction
The use of learning analytics in higher education has grown rapidly over the last decade.Defined by the Society of Learning Analytics Research (SoLAR, https://www.solaresearch.org/)as, "The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs," learning analytics help educators monitor learners and the learning process to discover patterns of engagement, identify learning difficulties, predict future behaviours in order to intervene and improve teaching (Leitner, Kahili & Ebner, 2017;Dychoff et al., 2016).From a student perspective, learning analytics can provide relevant feedback on their learning progress and performance and recommend appropriate action to take.
How learners interact with an online learning environment is in large part influenced by the teachers' online pedagogy.The online pedagogy is reflected by how teachers design the online learning environment (predelivery) as well as how they facilitate the online learning process (during delivery).A critical part of understanding the teachers' online pedagogies is using teaching analytics.However, the use of teaching analytics to gain insight into teaching practices is still at its early stages.Therefore, the purpose of this paper is to further contribute to this discussion by exploring the concept of teaching analytics and how they can help us understand online teacher decisions and practices.We suggest a framework for understanding and implementing teaching analytics in a holistic and pedagogically beneficial way.Using preliminary data related to quizzes and forums, we illustrate how this might work in real life, followed by a discussion of possibilities and potential challenges.

What are teaching analytics?
Unlike leaning analytics, research on teaching analytics is still sporadic.While much of teachers' online practices can be implied from learning analytics, the information is not primarily about them but about learners.Teaching analytics differ with learning analytics in the sense that the data collected is primarily about teaching rather than learning and its purpose is to help educators analyse and improve the quality of their teaching practice, particularly in the online space (Khazairi, Sidhu & Cobb, 202;Ndukwe & Daniel, 2020).The data used focusses on both educational design and delivery.Teaching analytics bridges the gap between academic analytics, which focusses on performance of academic programs, and learning analytics which is concerned with understanding learners and learning (Ndukwe & Daniel, 2020;Bennacer, 2022).This paper is authored by academic developers and educational designers/learning experience designers, and from that specific perspective, teaching analytics helps researchers and those who support teaching to develop better strategies to support and enhance teachers' professional practice (Bennacer, 2022).The primary source of data about teacher online practices is the learning management system (LMS) as well as curriculum management systems (Stack et al., 2020).Other secondary sources are plug-in technologies and integrated software that teachers use in the course of their teaching practice; this could be portfolio tools or virtual meeting technologies such as Zoom, for example.Data from the LMS primarily includes tool usage and activity logs, and uploaded documents about the course such as course outlines.On the other hand, curriculum management systems contain information about courses and programs, providing curriculum mapping information such as learning outcomes and assessment types.From an LMS perspective, tool usage can be clustered to provide insights into pedagogical approaches.For example, courses can be categorised according to predominantly used tools: forum-based, quiz-based, wiki-based or resource based (Il-Hyun, 2016).Other course archetypes could include whether technology is used for supplementary, complementary, social, evaluative and holistic purposes (Whitner et al, 2016;see also, Park et al, 2016).Furthermore, these analytics can help us understand teaching behaviours (Bennacer, 2022) and combined with learning analytics help evaluate and understand how students interact with their learning design, with a potential for a recommendation system for teachers to improve their practice (Bennacer, 2022).

Exploring potential frameworks for meaningful application of teaching analytics
Learning analytics are most beneficial when their design and interpretation are informed by learning design (Law, Milligan, Nawaz, Corrin & Bakhataria, 2022), teaching analytics, in a similar way, should be underpinned by an understanding of what good online learning and teaching entails (Nduke & Daniels, 2020).Underpinning our use of teaching analytics with curriculum and learning design models can help us determine what data we collect and how we analyse it to identify meaningful interpretations and recommendations.To assist with this, we conceptualise the online teaching activity as having three layers: Curriculum design, learning design, and online delivery.Consequently, understanding the online teaching practice holistically entails taking these three layers of activity into consideration, although the challenge is that these activities are often done by different people especially in the online learning context.From a teaching analytics perspective we suggest this table might be helpful in conceptualising what information to collect and how to analyse it.This table unpacks what the three layers entail, what type of information can be collected, from what sources and the sort of outputs we might expect to get.These layers recognise the curriculum process in terms of the designed curriculum (curriculum design), the developed curriculum (learning design) and the delivered or implemented curriculum (delivery) with an emphasis on the linkage between curriculum, pedagogy and learning effectiveness (Boitshwarelo & Vemuri, 2017).While all these layers are important the middle layer, learning design, is pivotal, in the understanding of the teacher's decision-making process.Learning design is informed by the curriculum and in turn influences delivery or the learning process, that is, it provides a bridge between the designed or intended curriculum and the delivered curriculum (the actual learning process) (Boitshwarelo & Vemuri, 2017).The next section illustrates how initial insights about teachers' pedagogical decisions related to quizzes and forums can be gained and how further questions be asked to trigger further investigations.

Examples: Quizzes and forums
Online quizzes and discussion forums characterise many an LMS, and are perhaps the most used interactive features, where students actually do and submit something.Research attests to their widespread use (Boitshwarelo, Reedy & Billany, 2017;Stack et al, 2020;Fehrman & Watson, 2021;Wang, 2019).Used effectively, they can help to engage students meaningfully and facilitate different types of learning as per Laurillard's Conversational Framework (Laurillard, 2002;Laurillard, 2012).Selecting either online quizzes and/or online discussion as part of an online learning environment, is a design decision that reflects the intended teaching practices and determines what learners are expected to do.From this perspective, data from the LMS and analytics provides a rich source of information on practices related to the use of these two tools and associated teaching practices.
We provide an example of initial stages of an investigation into the use of quizzes and forums in terms of the design and delivery of these activities within the LMS.At a later stage of the research, we plan to focus on online learning delivery and how students engage with these tools.The expected outcome is to determine the extent and nature of these two practices and to establish any patterns of use across courses, programs or year levels providing a glimpse into pedagogical practices of staff and informing directions for further investigations.Part of the initial data is presented below.This data was retrieved and analysed through R Studio tools and is primarily from the LMS.How useful is this data in helping us understand the teacher's pedagogy?The data provides insights into institutional online teaching practices and their differences across academic units, year levels and modalities of teaching.It helps us to identify patterns of practice and/or what may seem to be anomalies, prompting further focussed investigations on aspects of the data.For example, the following are observable from this data and may prompt further questions: 1.In most academic units (ALH, CHS, CTV, EDC, and STM) there is a clear pattern of decreasing numbers of quizzes used per course as one goes up the study levels in the undergraduate space (levels one to four), in the Learn Online courses (On-campus).This is also generally the case in UniSA Online except for JUS and EDC.Just looking at this example we ask ourselves if the reduction in quizzes per course as you go up the levels could be a reflection of the change in the cognitive levels of learning outcomes, the assessment approaches and therefore the teacher's pedagogies? 2. There is generally less use of quizzes in the UniSA Online than Learn Online courses.Why might that be the case? 3. On the other hand, there is higher usage of discussion forums in UniSA Online that in Learn Online.This is probably expected as discussion forums help online students communicate, connect and collaborate with others.It might be worth investigating whether there is also a difference in the quality of discussion forums.4. Other questions may relate to whether a direct or inverse relationship exists between the use of quizzes and forums in courses.For example, in the above graphs EDC barely uses quizzes in UniSA Online but is the largest user of discussion forums.Is this indicative of the nature of their subjects and/or their pedagogy?
As evidenced by the observations and associated questions or speculations above, this initial data is useful in understanding what the extent and patterns are in the institutional use of these tools, and, more importantly, prompting us to start asking relevant why and how questions that would unveil online teaching pedagogies more deeply.At the micro level of collecting the more fine-grained why and how data, it becomes more beneficial to apply a learning design framework to help meaningfully plan, interpret and action teaching analytics.Subsequent papers will explore this avenue further using appropriate frameworks, such as Laurillard's Conversational Framework (Laurillard, 2012).
Revisiting the three-layered model presented earlier (Table 1), the data presented here with the middle layer of learning design.The why questions that need to be further explored to understand this middle layer, would mostly reside in the first layer, curriculum design.The how questions on the other hand, require both a deeper dive into the learning design as well as exploration of the delivery to understand how the design is meant to work.

Conclusions
This paper set out to make a case for a greater focus on teaching analytics to understand teaching practices.It introduced the different layers along which teaching can be understood, with a particular focus on learning design.Using preliminary data, it showed possible avenues of investigations that can lead to our understanding of teaching pedagogies, as a precursor to understanding learning.
There are challenges, however, associated with teaching analytics.Unlike learning analytics with more developed tools and metrics, teaching analytics are currently more dependent on improvising and making use of indirect data.In this initial work, we found that the data had to be wrangled and queries needed to be refined to bring it to a state where we were confident of its integrity.Additionally, because of this limited sophistication, a lot of interpretations are implied and need to be verified with more micro level data, although acquisition of this data can be work intensive as it mostly requires manual subject-by-subject extraction.
That said, we conclude by suggesting that the more data on teaching practices is sought, the greater the likelihood of the processes of acquisition, analysis, interpretation and action being refined, and perhaps even more appropriate tools developed.Bennacer (2022)'s work is an excellent example of this where he is incorporating artificial intelligence to harness the potential power of teaching analytics.In our specific case a powerful tool might include being able to easily correlate quizzes and forums with student engagement and performance, and to overlay quantitative analytics data with qualitative sentiment analysis.As it is evident, this work is only preliminary, and more empirical work is anticipated with a focus on some of the questions raised earlier, which will seek to understand the why and how of teaching practices related to quizzes and forums, potentially leading to recommendations for teaching and learning enhancement.The authors in this paper are from two universities and will be engaged in comparative studies between their institutions.Finally, we challenge other teaching and learning practitioners to intentionally embark on the exploration of teaching analytics in their contexts and share those practices broadly amongst peers.

Figure 1 :Figure 2 :
Figure 1: Quiz per Course by Level in both Blended Learning and Online Courses