Assuring and Improving Learning Outcomes

Abstract

Universities have a responsibility to measure and assure graduate learning outcomes ("Higher Education Standards Framework (Threshold Standards)," 2021; TEQSA, 2017). However, while assuring learning outcomes is a priority across the sector, many universities fail to collect meaningful data on student achievement of these outcomes (Martin & Mahat, 2017). Some institutions are investigating technologically enhanced processes to improve the efficiency and accuracy of learning-achievement data (Burrack & Musselwhite Thompson, 2021; El Marsafawy, 2022). Over the past four years, the University of Technology Sydney (UTS) Business School has made substantial enhancements to assurance of learning (AoL) practices, leveraging learning management system (LMS) functionality and data visualisation software to create a streamlined process, which better facilitates the continuous improvement of programs. This work is of particular interest to academic and professional staff involved in collecting and reporting on student achievement data, in program continuous improvement and evaluation, or in leveraging technology-enhanced learning analytics.

The UTS Business School’s AACSB accreditation necessitates the collection, tracking, and reporting of student achievement of program-level learning outcomes (AACSB, 2020). Each program identifies subjects to "develop" and "assure" learning outcomes, with student achievement measured within specific assessment tasks and collated for program-level analysis. This process can be administratively complex, and historically involved multiple manually collated spreadsheets. This represented a substantial time cost for the faculty and often resulted in unreliable data.  It was, for example, common practice to report overall subject marks rather than the specific learning outcome to determine achievement level, resulting in assessment practices that diverged from the recommended standards of constructive alignment (Biggs, 2014).

In 2021, faculty T&L and central learning technologies teams commenced a pilot using LMS tools and the interactive data visualization tool PowerBI to streamline data collection, collation, and visualisation. Canvas Outcomes set up at the faculty level allows for centralised achievement-data collection, which, when linked to student and program information in PowerBI, can be used to obtain program-level insights. Beyond administrative efficiency, the project was also designed to improve and foreground the constructive alignment (Biggs, 2014) of measured learning outcomes. The revised process requires the embedding of learning-outcome-aligned assessment criteria within rubrics, resulting in improved constructive alignment and data transparency.

The initial pilot was evaluated using Human Centred Design (IDEO.org, 2015) methods, which concluded the new system was an improvement on previous processes. Since the pilot, the faculty has completed a full rollout to over 60 subjects and 20 programs.

Once this foundational AoL process was in place, program and subject-level continuous-improvement projects could be built upon it. Previous reporting methods had limited usefulness to Program Directors and were no longer fit for purpose due to changes in AACSB requirements. Reporting processes were therefore adjusted to focus on continuous improvement of key program metrics. In the new process, Program Directors and teaching teams review program data to identify curriculum or student-experience improvements that will enhance student achievement of learning outcomes. For example, a master's program identified that a high percentage of students were falling below expectations in two outcomes. Investigating the causes identified a need for substantial changes, with new subjects taking assurance responsibilities, and revisions to the program's overall assessment strategy. Recent data shows significant improvements in the target outcomes.

This presentation will outline these updated technology-enhanced AoL processes demonstrating how the collected data can be used to improve university programs and the student learning experience.