Learning from ‘failures’ in the development of mobile and technology-enhanced learning initiatives

Mobile and digital technologies can support learning experiences outside of the traditional classroom contexts and provide opportunities for collaborative elearning initiatives. Following design-based research (DBR) principles, curriculum developers face challenges in building mobile learning opportunities that are fit for purpose, adequately supported, and embraced by instructors and students alike. However, published accounts of mobile and technology-enhanced learning initiatives are currently largely restricted to ‘successes’ and lack descriptions of the processes and challenges involved in development. Critical factors for success – derived from accounts of ‘failed’ projects – may also be useful to guide the planning of initiatives in development. I present here potential steps towards a guiding framework for mobile and technology-enhanced learning developers that incorporates both an appraisal of these critical success factors within the context of a developing initiative and the instructive value of ‘productive failure’.


The transformative potential of mobile learning
Mobile and smartphone ownership is high amongst across students at higher education institutions, with students increasingly using mobile technology to both connect with peers and access learning materials (Bernacki et al., 2020). Educational activities that incorporate mobile applications can facilitate student-directed learning and collaboration (García-Morales et al., 2021), but productively harnessing the potential for mobile learning in higher education requires an understanding of how these initiatives are integrated into the curriculum.

Contextual utility in field-based settings
Authentic, place-based experiential learning that builds understanding of ecosystem dynamics and human impacts is essential for students of ecology. Creating future environmental leaders requires fostering such understanding whilst building transferable skills in collaboration and communication. Mobile and digital technologies can enable student cohorts to share data and learning experiences and allow them to build a common understanding of global environmental challenges, thus providing tantalising opportunities for collaborative international elearning projects.
The powerful potential of mobile app technologies for field-based learning has been realised in numerous citizen science projects that seek to obtain data on species presence or habitat characteristics across broad geographic areas (e.g., iNaturalist: Unger et al., 2020). However, despite potential advantages, the use of mobile tools in field-based learning in higher education contexts appears to be limited. A systematic review in progress (Bone et al., 2022)ostensibly to examine the institutional and curriculum contexts within which successful field-based mobile learning projects are implementedfound fewer than ten studies describing field-based mobile learning projects published within the last decade. In addition, no included study reported on how institutional factors such as instructor capacity and support may have contributed to project success. Although other projects may have been trialled, implemented and/or developed, it is apparent that the publication of these ideas and unfinished projects -'failures'has been low. A 2012 review by Wu et al., found most m-learning studies focused on the development of the tool itself, rather than on student learning experiences, whilst Crompton and Burke (2018) found most (70%) studies reported positive research outcomes, whereas few (4%) reported negative outcomes. These patterns were affirmed by a rapid scan of Google Scholar results from 2018 onwards, using terms such as 'mobile learning project failure' and 'mobile learning project null results', which yielded no ASCILITE 2022 The University of Sydney e22097-2 first-page results describing failed projects.

Reporting on processes
Initiatives that seek to build technology-enhanced and mobile learning experiences in the curriculum can encounter challenges and barriers at several stages during the development process. These may include the knowledge of, and capacity for, technology-enhanced pedagogies within instructors; students' digital knowledge; the adaptability of the curriculum, and the available technological affordances and institutional support for such projects (e.g., Lašáková et al., 2017;Polly et al., 2021). These initiatives are often developed and implemented through processes that align with a design-based research (DBR) framework (described by Reeves 2006), with DBR approaches now common in educational fields (Tinoca et al., 2022). Phases in the DBR process detail iterative stages of project development that include testing and fine-tuning, the identification and resolution of issues, and reflection and enhancement ( Fig. 1).

Figure 1: Design-based research (DBR) framework. Adapted from Reeves 2006: p. 59
The DBR framework expresses the ideas that, in phases 2 and 3, solutions will be informed by existing design principles and that cycles of testing and refinement of solutions in practice will be iterative. These processes of testing and refinement will, in turn, be reflected on to enhance solutions and implementation. Thus, embedded within each step of a DBR process is a sequence of trial, error, refinement and re-trial that may not be described in print or widely disseminated; instead, publications tend to focus on the solutions implemented in practice; somewhere between P3 and P4, with a description of P1 and a description of the successful solutions developed in P2. Thus, whilst lessons are being learned at each point in this process; our next steps as a growing community are to recognise, describe and communicate these lessons to better inform practitioners.

'Productive failure' as an instructive force in technology-enhanced learning development
Within each of the descriptors of the DBR process, the idea of productive failure is implicit, without being explicitly stated. Lessons are learned at each stage of the process, but these lessons are not being disseminated through traditional publication means. 'Productive failure' as described by scholars such as Kapur (2008) is an instructive concept that incorporates learning from iterative processes of trial and error. Productive failure has a long history of learning from failure in computer science and technology development (e.g., Gregor 2006), and there is growing recognition of its utility in educational design (e.g., Henderson et al., 2022). Productive failure is inherent in open-ended problems and inquiry-based learning and is particularly relevant when incorporated within technology-enhanced educational settings that allow students greater flexibility and autonomy in their learning (Kennedy-Clark et al., 2009;Lodge et al., 2018). As a step towards identifying processes that can enable mobile learning development initiatives, it is also instructive to consider accounts of 'failed' initiatives, and backwards-map these failures to identify points at which intervention might be productive and the possibility of changing direction could be considered.
Cochrane (2012) outlines six critical factors for success in mobile learning initiatives that emphasise the need for: (1) pedagogical integration of technology; (2) lecturer modelling of pedagogical use of tools; (3) creating a supportive learning community; (4) appropriate choice of mobile devices and software; (5) technology and pedagogical support within social constructivist learning paradigms, and (6) sustained interaction and scaffolding of ontological shifts in teaching and learning. I suggest here that these critical factors may also be useful in predicting the likelihood of success for projects yet to be implemented, including those that may also exist outside existing academic development or strategic organisational initiatives.
That these challenges are not currently communicated in the literature is unsurprising; the culture of academic publication is one of positive, successful stories, and the reporting of null results discouraged (Dawson & Dawson 2018). However, the dearth of such stories to draw on means that future initiatives may be more likely to confront similar challenges and blockages and waste resources. In addition, this lack of foresight and clarity on factors contributing to mobile learning project success or 'failure' can restrict strategic implementation of mobile learning projects across curriculum contexts and institutional boundaries. I propose we reimagine how we communicate outcomes of mobile learning initiatives and present here an initial exploration of a possible framework to assess, categorise and describe the trials, tribulations and challenges of their development.

Proposed steps in mapping process
Taking the critical success factors as defined by Cochrane (2012), I first worked to define discrete subfactors that would reasonably contribute to each factor. Next, I described how these subfactors may manifest within the specific institutional context of my international mobile learning initiative in development and define what 'success' would mean. This included an appraisal of both the level of efforthuman or monetary resources, political capital or social effortrequired to reach success, as well as the consequences for the initiative of task completion or incompletion. Incorporating these measures of effort and consequence allowed these subfactors to be appraised in terms of the risk involved to the project of task incompletion. Table 1 presents a modified risk matrix based on these effort/consequence assessments, where the failure of initiative components or factors that require little effort and/or are unlikely to be critical to the overall success of the initiative would rank as presenting 'Low' risk, whereas the incompletion of tasks requiring a high degree of difficulty or effortsuch as modifying an entire degreeor that are important to the initiativefor example, securing a working technical platform -would be ranked as presenting a 'High' risk to the initiative overall. Consequences of some subfactors failing may be negligible to the overall success of the initiative, whereas others may be essential. Risk levels will also vary according to the specific context and would need to be modified by practitioners. Working to assess likelihood of specific tasks being completed may help practitioners: (a) to identify a potential path of 'least resistance' through tasksone that presents the lowest risk with the highest potential for success; (b) to identify aspects of projects that may need more attention, and (c) make decisions and set priorities for further project development or upgrades.

Describing a single critical success factor in context
Within the context of an existing project in development that seeks to develop a mobile learning system for collaborative field-based learning across international institutional contexts (Bone et al., 2020), I describe below just one of these critical success factors defining several subfactors and assessing their feasibility in context.
Factor: Pedagogical integration of technology into course and assessment 1. Subfactor: Lecturer intent to change curriculum Assessment criteria: Capacity and intent of lecturers to alter components of the curriculum, including learning activities, learning objectives, assessment tasks and weightings to suit a more social constructivist pedagogy. 2. Subfactor: Lecturer capacity for pedagogical change Assessment criteria: Prior knowledge and capacity for lecturer to be supported in making pedagogical shifts. Consider both the lecturer's space for change (within the curriculum) and their desire for change 3. Subfactor: Ease of control over curriculum redevelopment Assessment criteria: How much control does the lecturer have over curriculum changes? Are they the coordinator or part of a teaching team? How strict are departmental requirements for submission of curriculum changes? 4. Subfactor: Gap between existing and desired curriculum Assessment criteria: How much will the curriculum have to change to embed the planned technology and shift in pedagogy? Table 2 provides a brief description of the assessments at each subfactor level, and the level of risk determined.

Challenges and next steps
There remains limited discussion on the processes by which mobile and technology-enhanced learning is developed in the higher education curriculum, and limited descriptions of 'failures' in context. Disseminating lessons learned from initiatives in varying development stages, including potential barriers, challenges and strategies, can lead to more robust, sustainable and strategic implementation of mobile learning initiatives across higher education curricula. A framework incorporating lessons learned from 'failed' initiatives has the potential to guide practitioners in planning and developing new projects. Presented here is a first step towards developing such a framework, using critical success factors described in Cochrane (2012) and applying them to the context of an international collaborative m-learning project in development. The next steps will be to further develop the framework, incorporating additional information and parameters from both the exemplar project and from the broader community. I propose initial data be gleaned from the existing ASCILITE community, from both within the ML-SIG and from participants at the ASCILITE 2022 conference. Incorporating these additional data will allow the success factors and subfactors to be refined across multiple contexts, building towards testing validity and application of the framework in practice.