Evaluating students’ experience of simulation technology in an Australian initial teacher education program

International research illustrates the effectiveness of simulation in professional education. For pre-service teachers, such technology offers the ability to contextualise pedagogy and curriculum within the complexity of the classroom environment. Novice practitioners benefit from developing decision-making skills in a risk-free context, whilst receiving immediate feedback on their performance. The University of Newcastle has therefore embedded simSchool, a virtual classroom gaming platform) in its undergraduate teacher education programs. This paper reports pre-service teaching students’ sense of teacher efficacy before and after using simSchool, based on students’ responses to surveys before (n. 398) and after (n. 233) use of the technology. Analysis examined changes in teacher efficacy, as well as potential differences between students based on previous experience. Results will form the basis for more informed integration of simulation technology, and more targeted support for students to optimise their experience and learning.


Simulation technology in initial teacher training
Simulation technology is well established in some industries and disciplines, yet use of simulation in educational training has lagged behind somewhat, perhaps due to the complexities of creating realistic scenarios with dozens of students, to practice pedagogical strategy that is far from standardised.However, the twenty-first century has seen the development of teaching simulations that offer a virtual experience of the classroom.Simulation offers a range of advantages in initial teacher education (ITE).It allows new practitioners to try strategies without fear of adversely affecting real learners.This 'safe space' allows inexperienced preservice teachers to fail without meaningful consequences.Additionally the rich data generated during these attempts means users can benefit from fast and detailed feedback, to inform repeated attempts of the exact same scenario which of course is not possible with real students.Furthermore, this ability to duplicate scenarios means simulation technology can give large cohorts of preservice teachers the same virtual classroom experiences.Scenarios can also be customised to suit particular training needs, This enables consistent preparation of trainees and enables precise alignment with accreditation requirements.Simulation technologies have therefore become part of twenty-first century ITE.In Australia, various institutions have incorporated simulation in ITE, including the University of Wollongong (Ferry et al., 2005), Murdoch University (Whipp et al., 2019), and the University of the Sunshine Coast (Febretti et al., 2013).Publications emphasise the benefits of simulation, such as improvements in student performance (Reinke et al., 2021), increased self-efficacy (Ledger & Fischetti, 2020) and classroom management (Yilmaz & Hebebci, 2022).It is equally important to be aware of the limitations inherent in simulation technology.Most obviously, simulation is not real; thus its proper place is as a practice space prior to real classroom experience.Some features of simulation, designed to support safe practice of professional skills, are irrelevant to the real world.Preservice students are quick to point out that you cannot pause a real classroom, nor select strategies from menus.These issues highlight the importance of managing students' expectations and supporting effective engagement for students to benefit from the opportunities offered by simulated teaching experience.

The context of the current study
This paper reports on the use of one simulation technology: simSchool.This commercial product offers a gamified teaching experience in which users teach a class of avatar students whose behaviour is informed by complex AI algorithms, giving each student their own particular combination of dispositional attributes (Gibson & Halverson, 2004;Zibit & Gibson, 2005).The user delivers a predetermined lesson, attempting to maintain all students' engagement and wellbeing.This requires ongoing decision-making to evaluate and select from a wide range of interaction options.Pedagogical choices are selected from drop-down menus, giving participants experience with a range of strategies.ITE students deepen their understanding of teaching practice through repeated cycles of decision-making, experimentation, and refinement, gradually building their expertise.To succeed, students must analyse student needs, make effective instructional decisions and evaluate the impact of their actions on student learning.These skills are required by the Professional Experience component of initial teacher education in NSW (NSW Education Standards Authority [NESA], 2017), thus simulation allows preservice teachers to begin building professional experience before they enter real classrooms at the practicum stage of their training.This study reports on the integration of simSchool into the ITE program at a large regional university in Australia.The School of Education has 3,000 undergraduate and 1,000 postgraduate students.SimSchool forms part of a suite of simulation technologies, alongside SimTeach, a simulation delivered virtually by human actors, and the iCave, a 360º immersion space.In the undergraduate programs, simSchool is introduced earliest, in the first or second year.SimSchool was initially trialled in 2022, with further customisation and increased integration in 2023.In the first half of 2023, just over 400 undergraduate ITE students were enrolled in core courses that required using simSchool as part of learning and assessment.These were a secondary classroom management course, and s a K-6 Pedagogy course for early childhood and primary preservice teachers.In both courses, simSchool was integrated into tutorial activities, directed independent work, and course assessment.The secondary course used 'off the shelf' simSchool modules, of which student had to complete at least three classes and reflection questions to secure five marks towards their overall grade.The K-6 course used three modules custom designed for this course.In module one students had to think about the needs of one student, Evan, and his responses to teacher-centred instruction.Students could try different strategies, review feedback, and consider how their choices impacted Evan.In module two students taught Evan within a group, and the third module in a whole class.Students then used that feedback to redesign the lesson plan, and the simSchool team uploaded these new lesson plans for students to use in place of the previously provided materials, to teach their own lesson within the simulated environment.All students in these courses were invited to participate in the pre-and post-surveys reported in this paper.

Method
Online surveys were presented to students via Canvas, the online learning management system.Students were directed to the surveys during tutorials and given time to complete the surveys in class.This was approved by the University's ethics department (Quality Assurance Approval #306).The surveys targeted students' teacher sense of efficacy, which may increase with use of simulation technology (Theelan et al., 2019;Yilmaz & Hebebci, 2022).The pre-simSchool survey also included a scale gauging participants' epistemic beliefs about the acquisition of knowledge, which are known to relate to engagement with higher order learning (Author, date).It was anticipated that epistemic beliefs may relate to students' openness to and engagement with simulation.Following use of simSchool, students were invited to complete a second survey to evaluate their experiences.The survey materials are detailed below.

Materials
Two online surveys were created to assess students' sense of teacher efficacy before and after engaging with simSchool.The first survey also gathered basic demographics and asked participants to report their prior experience of teaching and of simulation technology, before presenting the Teachers Sense of Efficacy Scale (TSES) (Tschannen-Moran & Woolfolk-Hoy, 2001) and the Acquisition of Knowledge scale (adapted from Bendixen et al., 1998, as per Author, date).These items were rated on a six-point Likert scale, from 'strongly agree' to 'strongly disagree'.Participants also rated three standalone items about their enjoyment of new technology, motivation to learn from the experience, and belief that simulation technology can help develop skills for teaching.The second survey was delivered at the conclusion of students' use of simSchool, and repeated the TSES items alongside an adapted version of the Technology Acceptance Model (TAM) (Davis, 1989).This survey concluded with three open text items asking students what they found positive about the experience, suggestions for future use of simSchool, and whether they had any other feedback.They were also invited to recommend whether simSchool should be used more, less, or the same amount in the future.The exported survey data aligned with Canvas activity data such as pages viewed and course grade.All identifying data was then removed prior to analysis.The analysis reported here focuses predominantly on the TSES data.

Participants
The participating courses had 466 students enrolled, of whom 398 responded to the pre-simSchool survey, giving an 85% response rate.The post-simSchool survey registered 233 responses (50% response rate), including 8 duplicates (which were removed) and 6 responses with no identification, meaning they could not be matched to pre-simSchool survey responses.Of the 398 pre-simSchool survey respondents, 268 (67%) identified as female, 123 (31%) male, 5 (1%) non-binary and 2 preferred not to answer.The mean age of participants was 2.64 (s.d.4.1).The majority of respondents were enrolled in the Primary teaching degree (59%), with the remainder in the Secondary (40%) or Early Childhood and Primary (0.5%) degrees.

Analysis and results
The survey scales were checked to verify internal reliability, and showed strong reliability in most cases (see Table 1).The exception was the post-simSchool measure of TSES, which had a weak Cronbach's Alpha (.422).Removal of three items ( 5, 6 and 3), increased this to .945.Future analysis will further investigate this unusual psychometric variation.A comparable 9-item scale score was calculated for the pre-simSchool survey data, enabling direct comparison of pre-and post-scores.Pre-simSchool TSES ranged from 2.42 up to 6, with a mean of 4.54 (n.398), and the reduced 9-item measure was very similar (range 2.33 -6, mean 4.52).An independent samples T-test indicated no significant differences in pre-simSchool TSES between students in the two courses (p = .347).Post-simSchool TSES scores based on all twelve items ranged from 1-6, with a mean of 4.5; these statistics changed very little for the more cohesive 9-item scale, which ranged from 1-6 with a mean of 4.49.The TSES instrument was originally designed with three subscales, although it is not uncommon for researchers to collapse these into a single score.The subscales comprise four items each and are intended to capture more specific dimensions of teaching: student engagement, instructional strategies, and classroom management.In the pre-simSchool survey, all subscales had good internal reliability (α ≥ .799),see Table 2 below.In contrast, in the post-simSchool survey all subscales showed poor reliability (α = ≤.265).The removal of one item from each subscale greatly increased reliability.The calculation of these reduced subscales for the pre-simSchool survey data resulted in slightly lower Cronbach's alphas, yet still within the satisfactory range (α ≥ .726).These revised subscales therefore offered an opportunity to investigate a finer-grained view of participants' teacher sense of efficacy, and to compare any changes pre/post across the three subscales.While the overall descriptive data appeared to show no significant change in participants' teacher sense of efficacy, this masks variation within the sample.A two-step cluster analysis based on pre-and post-simSchool 9-item TSES scores identified a two-cluster solution of good quality.These clusters represented groups of students with quite different profiles.The larger cluster, comprising 67.3% of the sample, had a pre-simSchool TSES of 4.77, increasing slightly to a post-simSchool TSES of 4.81.In contrast, the other cluster (32.7%) had a pre-simSchool TSES of 3.98, decreasing to a post-simSchool TSES of 3.79.This may reflect two broad groups of participants, one entering with greater confidence, engaging more effectively and further building their sense of efficacy; the other entering with lower confidence, thus having poorer experiences and further diminishing their already lower sense of efficacy.Cluster membership was somewhat predicted by participants' pre-simSchool responses to three items asking about enjoyment of technology, whether simulation can help develop teaching skills, and whether they feel motivated to learn from the experience.This indicates that very simple survey measures may be effective in identifying students who are at risk of not engaging effectively with the simulation experience, and who would therefore benefit from targeted support.Cluster membership did not appear to be related to prior teaching experience, with students who reported teaching experience or not similarly likely to fall into either cluster.There was also little difference in cluster membership between students enrolled in the two participating courses.Mean age in each cluster was almost identical (21.58 and 21.62).There seemed to be a small gender effect, with female students slightly over-represented in the reduced efficacy cluster, and male students slightly more likely to be in the increased efficacy cluster.The increased efficacy cluster members reported, in the first survey, slightly more sophisticated beliefs about the acquisition of knowledge, and it is possible that this relates to their more effective engagement with the simulation experience.However, the increased efficacy cluster did not achieve higher grades than the reduced efficacy cluster; the final grade for the increase efficacy cluster was 76.5, and for the reduced efficacy cluster 74.1, although an independent samples T test showed that this difference was not significant (p = .120).

Student Feedback
Of the 238 students who completed the item asking them about future use of simulation technology in their degree, 84 recommended using it the same amount and 17 suggested using it more, meaning 42% of respondents were happy with the use of simulation in their degree.However, 55 (23%) said to use it less, and 72 (30%) said to not use it at all.When this is compared between the two clusters, students in the increased efficacy cluster were more likely to support the continuation or increase of simulation use in their degree (47%) whereas students in the decreased efficacy cluster were more likely to recommend using it less or not at all (62%).
Students were also asked to identify the positive aspects of simulation technology, and to make suggestions for future use.Their responses illustrate the diversity of their experience.While some students struggled to find any positives ('to be honest, nothing really'), the majority readily identified a range of benefits: • It was a risk free way to practise in the classroom.
• SimSchool gave me an opportunity to consider the reality of teaching.
• The simulation was quite realistic like a proper classroom setting.
• SimSchool demonstrated that every student … has different needs and learning abilities.
• [it] opened my eyes to different classroom management options I didn't know about.

Findings and discussion
The analysis reported above indicates that in some cases students' use of simSchool appears to have contributed to greater teacher sense of efficacy.This aligns with the intention of integrating simulations in ITE, and with previously published findings (Samuelsson et al., 2022).However, for a proportion of the current sample there was a drop in reported teacher sense of efficacy.At first glance this is concerning, because the use of simulation is intended to build preservice students' skills for face-to-face practicum placements.However, it is possible that this early exposure to some realities of classroom teachingserves as a 'reality check' for some preservice teachers.The results of this research illustrate the complexity of teacher sense of efficacy as a construct and highlight potentially contradictory relationships between teacher efficacy and use of simulation in preservice teacher training.This requires further examination, in order to effectively support different students to effectively build both competence and confidence prior to entering the classroom.However, it must also be acknowledged that for some students, it may be appropriate and even beneficial to experience a gentle reality check through simulated classroom teaching experiences.Over-confidence is unlikely to be helpful in developing expertise, and may set students up for disappointment when they enter a physical classroom space.Guided discussion of experiences, with tailored support as required, should enable all students to gain from the simulation experience.Some limitations should be noted.Firstly, it is not possible to know with certainty whether these changes in teacher efficacy were due to the use of simulation technology.They may reflect ordinary development of professional competence and confidence in undergraduate ITE programs.It is also unknown to what extent sense of teacher efficacy may normally vary between measurement points, and the robustness of the measure may be brought into question by the significantly weaker Cronbach's alpha of the post-simSchool TSES data.This requires further investigation.It should also be noted that the data reported here may also be influenced by self-selection of participants, because while the courses they were enrolled in were mandatory, completion of the surveys was optional.Response rates for the first survey were stronger than for the second, and so all analysis involving post-simSchool measures is limited by this factor.However, the results presented here serve as a foundation for more detailed analysis and ongoing evaluation of the integration of simulation in teacher education programs.

Future directions
The integration of simulation technologies in teacher education is ongoing in Australian universities.At this university, existing evaluation has identified opportunities to optimise students' engagement with simulation by adhering to essential principles including: early introduction, management of expectations, staff expertise, structured reflection, and integration with assessment.The next phase of delivery will focus on closer alignment with the Australian Professional Standards for Teachers (APST), which set out the required competencies of graduate teachers.Such alignment, and the evaluation of students' perceptions of the ways simulation supports their progression towards the APST, will address the contribution of simulation to pre-service teachers' professional accreditation.This forms part of a continuing program of research monitoring not only students' sense of efficacy following use of simulation but also subsequent performance in practicum placements in real schools.In future there will also be the opportunity to review longitudinal experiences of students who have engaged with various simulation technologies over a number of years.