Same graph, different data: A usability study of a student-facing dashboard based on self-regulated learning theory

Student-facing learning analytics dashboards have the potential to reconnect students with their purpose for learning, reminding them of their goals and promoting reflection about their learning journey. However, far less is known about the specifics of the relationship between different types of visualisations and data presented in dashboards and their impact on students’ motivation. In this study, we used a Human-Centred Design method across three iterations to (1) understand how students prioritise similar visualisations when presenting different data (2) examine how they interact with these, and (3) propose a dashboard design that would accommodate students’ different motivational needs. In the first iteration, 26 participants ranked their preferred visualisations using paper prototypes; in the second iteration, a digital wireframe was created based on the results from the first iteration to conduct user tests with two participants; and in the third iteration, a high-fidelity prototype was created to reflect findings from the previous iterations. Overall, findings showed that students mostly valued setting goals and monitoring their progress from a multiple goals approach, and were reluctant about comparing their performance with peers due to concerns related to promoting unproductive competition amongst peers and data privacy. Implications for educators and learning designers are discussed.


Introduction
Student-facing learning analytics dashboards have become popular in educational technology over the last decade. Dashboards have the potential to reconnect students with their purpose for learning, reminding them of their goals and promoting reflection about their learning journey. That is, assisting them to become active agents of their learning journey, or self-regulated learners. However, there are still many unknowns on the use of such dashboards to promote learning, particularly when referring to their impact on student motivation. Previous research had mixed findings; while some aspects of dashboards were found to motivate students, others were detrimental to their motivation (e.g., Corrin & de Barba, 2015). In this study, we iteratively examined students' perceptions of varied dashboard visualisations and widgets in relation to their motivation to learn, based on selfregulated learning theory.

Related work
Self-regulated learners are strategic about their approach to learning, planning, monitoring and adapting the chosen learning strategies (Zimmerman, 2008). The COPES model (Winne, 2018;Winne & Hadwin, 1998) provides a useful framework to understand how students regulate their learning. This model proposes four phases: (1) task understanding, (2) goal setting and planning actions, (3) execution of their plans to achieve their goals, and (4) adaption of their strategies when progress to achieve their goals is hindered. Students' success to regulate their learning lies on their ability to continuously control and monitor these phases during learning. Evaluations comparing their work against standards help students to check their progress towards their goals. Goals, therefore, guide students' actions and serve as a reference point for progress evaluation.
When setting goals, students set the standards they are hoping to achieve during their learning journey. These standards can be based in different reference points (Valle et al., 2003). Researchers refers to as mastery or learning goal orientation when the reference point used is a self-reference standard (i.e., increasing their own competence). For example, students aiming to learn all they can about a certain topic. On the other hand, performance goal orientation is when students' reference point for standards are other people, as they focus on demonstrating their competence to others. For example, students aiming to be the top student of their class. Goal orientation impact how students monitor and regulate their cognition, affect and behaviour during learning. Current research supports that students pursuing multiple goals are most successful (Valle et al., 2003). However, continuously controlling and monitoring their learning is an effortful enterprise. Students often lack the motivation and skills to adequately activate these regulatory skills (Zimmerman, 2008;Winne & Hadwin, 2013).
Dashboards have the potential to help students to regulate their learning, as they provide objective data related to their achievement and progress in relation to their chosen standards (Molenaar et al., 2019). Dashboards apply data science methods to process and analyse their data and present these back to them in a meaningful way (Matcha et al., 2019). Doing so, student-facing learning analytics dashboards can be considered intervention tools aiming to either instruct, support or motivate students (Pokhrel & Awasthi, 2021). Instructional interventions provide students guidance on what to do next, supportive interventions encourage them to continue their current actions, and motivational interventions have the purpose to bring their attention back to their learning experience. In this study, we aimed to examine how commonly used features in dashboards could be tailored to motivate students. This means that the focus of our dashboard was not so much to use the dashboard to guide students on what to do next, but bring their attention back to the course and provide insights into their current efforts.
Effort regulation in self-regulated learning theory is referred to students' ability to commit to achieving their goals. This could be translated to the amount of time a student dedicated to their studies, for example (de Barba et al., 2020). A useful dashboard would then present the 'right' data combined with the 'right' visualisation technique that best supports the users to evaluate their progress towards their goals. Deciding which data and visualisations to include in a dashboard is a big challenge for educators and designers. Defining the 'right' data includes dealing with data limitations related to what data is currently being recorded and can be retrieved, while defining the 'right' visualisation requires knowledge about how users will interpret the data. In the case of learning analytics dashboards, this involves knowledge about how the information being displayed will impact their learning journey. For example, previous research has found that visualisations displaying the class average as a reference point could be detrimental to students' motivation (Corrin & de Barba, 2015).
Commonly used visualisations and widgets (i.e., a simple stand-alone component) in dashboards allow students to set goals and monitor their progress, receive rewards and compare themselves to peers using leaderboards (e.g., Sahin & Ifenthaler, 2021). Goal setting and progress monitoring allow students to input their goals and monitor their progress throughout the course, either updating them themselves, or having a system updating them automatically. Virtual rewards and incentives are widely used in online learning platforms such as ClassDojo, where provides virtual points and badges to gamify the learning experience and engage online learners (Williamson, 2017). Leaderboards, on the other hand, are seen as a gamification strategy that when used in education can enhance learners' motivation through peer learning (e.g., Park & Kim, 2021). Platforms like Kahoot and Kaggle are widely adopted by online instructors, as such tools can help recreate a classroom atmosphere that allows isolated learners to visualise the participation and performance of other students to motivate their learning, improve the online study experience, and engagement (Gillett-Swan, 2017). In this paper, we investigated how students perceived these commonly used visualisations and widgets in dashboards when adapted to display data related to different goal orientations (learning and performance) and their commitment to their study so far (effort regulation).

The current study
In this study, our aim was to better understand students' perceptions and preferences of commonly used dashboard visualisations and widgets populated with different data, each representing a motivational construct. The following research questions were investigated: RQ1. How do students prioritise similar visualisations and widgets when presented with different motivational data (i.e., performance goal orientation, learning goal orientation and effort regulation)? RQ2. What are students' perceptions when interacting with similar visualisations and widgets presented with different motivational data (i.e., performance goal orientation, learning goal orientation and effort regulation)? RQ3. How can a student-facing learning analytics dashboard design accommodate students' different motivational needs?

Method
This study followed a Human-Centred Design approach, which includes the design thinking process of empathize, define, ideate, prototype, and test in iterative cycles (Giacomin, 2014). This approach allows rapid prototyping and immediate evaluation to understand users' decision-making process and expectations under different goal-oriented visualisation scenarios. As design thinking is a non-linear iterative process, we organised this study in three different iterations detailed below. All iterations were conducted online due to COVID-19 restrictions at the time of data collection.

First Iteration: Paper prototype and survey
In the first iteration, 26 participants answered a survey on Qualtrics to evaluate a series of widgets created based on paper prototypes. Participants were students from the University of Melbourne recruited via emails and announcements on the Canvas learning management system (LMS) and provided informed consent (Ethics application number 20833). The online survey, using Qualtrics, displayed and prompted participants with a combination of card sorting activities, multiple choice questions, and an open-end question, all focused on setting, visualising and managing learning goals. The card sorting activity was based on the MoSCoW method (Hudaib et al., 2018) to help us understanding widgets prioritisation. Students were prompted with different widgets for Canvas and required to categorise them as 'Must Have', 'Nice to Have', and 'No need', with multiple choice questions following up on concerns related to widgets' features. Additionally, we asked them about their willingness to share their own data with peers on the leaderboard.
Quantitative research data from the first iteration was analysed through Majority Voting Goal-Based (MVGB) technique for Requirement Prioritisation (Hudaib et al., 2018). In this analysis method, a weighted value is given to each category: 'Must have' has a weight of 3, 'Nice to have' has a weight of 1, 'No need' has a weight of -3: W= {3, 1, -3}, and the evaluation value is the partial sum of the count for each category C= {C1,C2,C3} * W= {3, 1, -3} and divided by total count. This method helped us create an ordered list based on the most important features students identified to guide their learning in the Canvas dashboards' paper prototypes.
The design of the paper prototype focused on the following widgets for students: goal setting and progress monitoring, receiving rewards on achievement milestone, and comparing themselves to peers using leaderboards. Each widget was created using data to promote a learning goal orientation, a performance goal orientation or effort regulation (for simplicity, referred to as effort goal in the visualisations), as presented in Figure 1 and described below: • Goal setting and progress monitoring (pie charts): The performance goal is set to be the result students want to attain in a subject, the learning goals are the overall learning tasks and modules to be accomplished in a subject, and the effort goal is the time they aim to spend on studying the subject every day ( Figure 1A). • Goal setting and progress monitoring (vertical bar charts): Our performance bar chart was represented in a one hundred-point system, following strategies and scales adopted by the university. The learning progress and daily effort widgets in Figure 1B show students accomplished tasks and time spent on them every day. • Goal setting and progress monitoring (calendars): Performance goals in our calendar widget ( Figure 1C) would be automatically prefilled from events added to Canvas. The calendar widget was designed to also allow students to add personal goals to it. Moreover, students would be able to set the number of hours or effort estimation they want to spend on the goals they add to the calendar. • Reward (badges): Our proposed 'performance' reward widget ( Figure 1D) appears when students' performance surpasses their goals or when they reach the top 10% of the class. The 'learning goals' reward pops up when students finish all the learning tasks in a module or achieve a certain learning milestone. The 'effort-reward' appears when students achieved the number of hours they targeted to spend on their studies. • Leaderboard (horizontal bar charts): This widget ( Figure 1E) enables students to compare their goals and learning progresses with peers. The issue of privacy was considered during this initial design, and gamification of the leader board is proposed to help solve these concerns, such as showing data anonymously or using virtual points (Park & Kim, 2021) instead of actual performance data, and showing the top performer's data only.

Second iteration: Digital wireframing and user testing
Building from analyses performed during first iteration, iteration two involved the design and creation of a wireframe based on widgets prioritisation ranking results. The created wireframe for the second round of iteration aimed at allowing students to set personalised learning goals and to track achievement progress on dashboards (i.e., user flows). Qualitative research methods such as think aloud protocol and interviews were adopted in this iteration to validate and test the wireframe prototype.
We had two participants involved in this second iteration. Participant A was doing bachelor's degree full time and could dedicate more than 30 hours per week to studying. This participant was familiar with Canvas LMS and has been consistently maintaining good grades and high Weighted Average Mark (WAM). Participant B was doing master's degree while having a full-time job. This participant was doing the degree based on personal interest in the area. Opposite to Participant A, this participant doesn't have much time to finish all the learning tasks on time due to work, but believes it's been studying enough to pass or obtain reasonably good grades in the subjects. Both participants were pursuing degrees in Information Technology-related fields; however, they are at different life stages and have different learning and performance goals.
Each participant spent over 90 minutes performing pre-established tasks using the wireframe prototype.
Participants were asked to perform three user flows (or tasks) in this iteration: (i) set a goal in the pie chart view and monitor achievement progress, (ii) set a goal in the calendar widget and monitor achievement progress and, (iii) share achievement on leader board. During the user test, participants were encouraged to articulate their thinking as they completed these tasks.

Third iteration: High-fidelity prototype
After our analyses and investigations performed in iterations one and two, we designed a high-fidelity prototype for Canvas LMS that can accommodate students with diversified backgrounds, supporting them setting and monitoring learning journeys. This iteration had no student participation, rather we present and justify the design of a dashboard built based on our findings to support and motivate students.

Students' prioritisation of visualisations and data (RQ1)
Results from the survey in the first iteration helped us identifying and ordering learning features students reported to be more motivating to them in the Canvas dashboards' prototypes (Table 1). The priority ranks for the dashboards corresponded to evaluation values given by participants (1 = most important). Overall, an average considering the priority of each type of data showed that, overall, performance goal orientation data had the highest priority (M=7.3), followed by learning goal orientation (M=8.33) and effort regulation (M=12.83) data. However, a closer inspection when considering the type of widget and visualization showed a more nuanced scenario, where participants mixed and matched the types of data: performance, learning and effort orientations. Using a calendar for setting and viewing performance and learning goals received the highest priority from students, followed by using a pie chart widget for the same purposes. Bar charts were the preferred way for students to visualise daily learning effort and achievement progress. Overall, visualisations using pie charts and calendar to present effort data were considered less important. The reward feature analysis for performance goals, as presented on Table 1, was highly ranked by students, while being rewarded by achieving learning tasks and effort goals was regarded as less important. Survey result also revealed that students were not likely to feel motivated by sharing their achievement with their peers, especially, sharing achievement about 'time spent on learning', which was regarded as 'No need' by 50% of the participants.
The leaderboard widget survey results showed that it was a debatable feature as it received negative evaluation value for both performance and effort goal (Table 1). Privacy and negative feelings related to a sense of competition were some of the main identified concerns students raised in the open-end questions when comparing performance and learning data with peers. When asked about their willingness to share their own data to populate the leaderboard, only 15.38% of all students wouldn't be willing to share any data with peers at no circumstance. Therefore, considering other responses (42.30% willing to share their data only if anonymously, 23% willing to share data if converted to virtual points) we decided to include a leaderboard identifying users using virtual points instead of their real data. Moreover, three students replied to the final open-ended question about further suggestions for the dashboard. Two of the students mentioned that peer pressure is a major concern for the goal-oriented dashboard, especially on the leaderboard feature.

Students' perceptions when interacting with the dashboard (RQ2)
In the second iteration, responding to the evaluation result, widgets with an evaluation score of > = 0 were included in the dashboard design. A calendar widget for learning and performance goals which was ranked at the top in the priority list was placed on the right side of the dashboard (Figure 2, visualization 1, 2). The pie chart widgets for learning and performance goals were considered the second priority, so were presented on the top left of the dashboard (Figure 2, visualization 3, 4). The widget to show the users' achievement was put on the top right of the dashboard (Figure 2, visualization 5), the achievement score was converted to gamified virtual points (performance score + learning progress*100) to reward students for achieving multiple goals. The bar chart widget for monitoring users' daily achievements with an average evaluation value of 1.385 was put below the pie chart with a filter dropdown to switch between the metrics ( Figure  The two participants were able to successfully complete the user tasks within the available time. The main usability issue encountered was related to the ambiguity of how the learning progress and daily learning effort were tracked in the LMS. Participant A, who has a better academic performance, was initially drawn to the leaderboard widget and mentioned: "Wow, competition?! (I) wasn't expect that" in a negative tone. On the other hand, Participant B perceived the leaderboard as a motivating feature. Regarding monitoring their performance, Participant A would prefer to use their own WAM rather than the class average showed in the dashboard, while participant B thought the average in the dashboard was a good performance measure. Both participants mentioned that although they used the LMS to access resources and submit assignments, most of their learning activities were not completed there.

Accommodating students' different motivational needs in a dashboard (RQ3)
The evaluation results from iterations one and two gave us an understanding of students' goal-setting behavior on the dashboard for the Canvas LMS, especially on how different types of goals are prioritized under different scenarios.
Across the study, students preferred data related to goal setting and progress monitoring using pie chart widgets and calendar widgets. Performance goals were more relevant when showing expected grades for the subject such as in a pie chart view. Learning goals were considered as part of a continuous process and students found more relevant to visualize that on bar charts or calendars. Setting personalised deadlines for learning tasks and assignments were also preferred by students to manage study time, instead of setting target study hours for each day. Meanwhile, achievement-oriented virtual rewards have been found to be an effective feature that can motivate students during online learning.
Based on these findings, we designed a high-fidelity prototype for Canvas LMS that can accommodate students' diversified goal orientations (Figure 3). Our new leaderboard widget was designed to use virtual points calculated from performance and learning progress data to minimize the negative influence of seeing peer's data and aimed at satisfying students' needs in wide demography. We also added a feature to let students share their achievement with customised profile images and nicknames. Students were also given the option to hide the leaderboard widget. Symbolic colour connotations were used to represent different types of goals in the final prototype, with the aim to help students classifying information on the dashboard. Learning goals were represented in purple, performance goals were represented in green and effort goals were represented in blue.

Discussion and conclusion
The investigations conducted in this study clarified some of students' preferences for data and visualisation in relation to their motivation to learn. This is particularly important when examining the use of dashboards from a self-regulated learning theory perspective, which states that students' goals guide their actions during their learning journey. Although there are limitations in the current study related to small sample size, this paper provides important contribution related to students' perceptions and preferences related to how data visualisation in learning analytics dashboards may impact their motivation and reconnect them with their purpose for learning.

Figure 3. High fidelity prototype produced in the third iteration
Overall, performance goal orientation data had the highest priority, followed by learning goal orientation and effort regulation data (RQ1). A closer inspection, however, showed that participants preferred to mix and match the type of goals across different types of visualisations. This finding is aligned with educational research in goal orientation which has found that a multiple goals approach is the most beneficial for students (Valle et al., 2003). It is interesting to note that the visualisations with highest prioritiescalendar and pie charts with performance and learning goalshelped students to identify what was coming next in their learning journey (calendar) and where they were at the moment (pie chart), in reference to both external deadlines and their own progress. Referring back to the COPES model (Winne, 2018;Winne & Hadwin, 1998), this suggests that students prioritise looking for information that will help them evaluate how they are progressing towards their goals. Only if they notice a problem in attaining their goals, data related to how they are exerting their effort becomes relevant. That is, if they are progressing well towards their goals, evaluating how they are applying learning strategies is secondary.
The cautionary tale of displaying performance data was replicated in our study when students interacted with the dashboard during user tests (RQ2). Participants reported concern in relation to the leaderboard inciting unproductive competition amongst peers. Potential negative outcomes from this could be, based on previous studies, distracting students from their own goals (Corrin & de Barba, 2015) or demotivate students if they see their performance far behind others (Sakulwichitsintu et al., 2015). This finding highlights the tension that designers face whenever displaying performance goal data comparing peers. Although these can be very useful to help students evaluate their progress, as mentioned in the previous paragraph, comparing themselves with their peers can be detrimental depending on their personal and contextual factors. Further research is necessary to investigate the specific of these personal and contextual factors so dashboards can provide personalised and adaptive data according to students' current situation.
Moreover, participants mentioned during user tests that even though they used the LMS to access resources and submit assignments, most of their learning activities were not completed there. This suggests that participants were aware of the incomplete nature of the data presented in this dashboard and took this information into consideration when interpreting the data presented. This finding complements current studies on the need to bring other data sources into the LMS (Kitto et al., 2015), or distribute student data out of the LMS (Oliveira et al., 2021), to provide more effective and comprehensive support to students.
Our suggested dashboard attempted to deal with three tensions found in the previous iterations of our study (RQ3). First, we prioritised learning and performance data related to goal setting and progress monitoring, using pie charts and calendar widgets, by placing them on a prime position in the dashboard (i.e., easy to find). Second, we decided to include the leaderboard, but with some modifications to lessen the negative impact on competition and peer pressure that this may bring to students. These included the creation of virtual points and of customised profiles to allow anonymity. Such modifications have been found to minimise the sense of inadequacy and maximise the student's experience of success and engagement (Park & Kim, 2021). Additionally, another modification to deal with the potential detrimental effect of a leaderboard, was the option for students to hide the leaderboard all together. Third, we included options for students to switch the type of data populating different graphs or widgets. The aim with this solution was to allow students to switch the data if needed. For example, after evaluating that they were failing to progress towards their goal, students could load effort data to examine their approach to learning (e.g., how frequently they were having study sessions). Ideally, an open learner model (i.e., a cumulative student model which students can access and make contributions; Bull & Kay, 2007) would inform dashboard design to promote personalisation and adaptation in real time. That is, depending on students' personal and contextual factors, certain graphs would appear, others would be hidden, and different types of data would populate them at different times.
For educators and designers involved in the development of dashboards, our findings can serve as initial guidelines to balance the type of goals represented in a dashboard. Our findings suggest that it may be useful to give emphasis to both learning and performance data to allow students to evaluate their progress towards their goal, followed by effort data, which may only be useful to them if there is a need to make adaptations to their approach to learning. In the case of presenting performance goal orientation data, additional care must be taken in regards to promoting unproductive competition amongst peers and to overstepping privacy concerns; perhaps involving students in the decisions involved in this process is a prudent first step. Future studies would benefit from applying elements of this dashboard to higher education learning settings with the aim to understand their impact on students' learning experience.