Teaching teachers to propose meaningful learning objectives: A MOOC case study

Learning objectives designate what students should be able to do to transform their social reality. Evidence shows tertiary education teachers usually do not know how to propose meaningful learning objectives. The aim of this study was to evaluate the effectiveness of a MOOC to teach how to formulate relevant learning objectives. This study has measured participant’s retention, performance on the topic, satisfaction and learning perception. A pilot version of this course was made available for 40 days. Out of the 176 participants enrolled, 60.8% completed the course. The average score in the initial test was 5.71, while in the final test was 8.67. 91% of the participants evaluated the course positively. Considering the high retention rate, the proficient performance at the end of the course, and the positive evaluation by participants, it was concluded that the MOOC was effective.


Meaningful learning objectives
In the 1960s and 1970s, the concept of learning objectives was widely disseminated from primary to tertiary education (Gusso, 2013). This concept was initially proposed as a description of what students should learn, defined by student's actions during classes rather than the content taught (Mager, 1962;Vargas, 1974). One of the most recognized proposals in this direction was Bloom's taxonomy (1956), which highlighted that the aim of education should not be memorization, but the development of different abilities to deal with the content (Anderson, 2005).
Since the 1960s, different terms were used indiscriminately to designate educational intention, such as: objectives, goals, intents, aims, outcomes, or tasks (Allan, 2006). And there is a relative consensus that this concept is relevant to "offer a starting point for a viable model for the design of curricula in higher education" (Allan, 2006, p. 93).
In parallel, the contributions of authors such as Paulo Freire helped to spread the idea that the purpose of Education should not be exhausted in what teachers or students do in the classroom. Educational objectives must impact the way students deal with the social needs of their community (Gusso et al., 2020). Thus, Kubo and Botomé (2001) highlighted that the proposition of meaningful learning objectives in tertiary education should consider what the learner should be able to do, after the course, to transform social reality as higher education professionals and as citizens.
Proposing meaningful learning objectives helps to increase students' perception of the relevance of courses, to establish realistic expectations regarding the subject, to define a clear criterion for the evaluation of the educational effectiveness, and to guide all the design of curricula (Cortegoso & Coser, 2011;Vargas, 2009). In addition, recent studies have demonstrated that the presentation of learning objectives has a positive effect on learning retention (Sana et al., 2020), and helps students identify what is most important in a course, helping them to better organise their own studies (Osueke, Mekonnen & Stanton, 2018).
Besides knowing what learning objectives are, it is also relevant to know what they are not. In this sense, an important contribution that helped to make more explicit the core characteristics of learning objectives was presented by Botomé in 1985, in which the author distinguishes meaningful learning objectives from false learning objectives (Cortegoso & Coser, 2011). These are described by the author as phrases that, instead of being learning objectives, only indicate teachers' intentions, activities performed in the classroom, observable actions of students in the teaching context, or even the mere description of content using verbs. Teaching this distinction to teachers contributes to qualify them to propose learning objectives more accurately (Carvalho, 2015).
The idea of learning objectives changes along the decades. Nowadays, it is not characterized as student actions in the classroom to be measured by the teacher (Kubo & Botomé, 2001). Meaningful learning objectives should therefore not be reduced to false learning objectives (Carvalho, 2015). Despite the use of different terminologies in the field of tertiary education, there is a relative consensus in the literature regarding learning objectives as the description of what learners should be able to do, after classes, in society. This conception reinforces the purpose of education as the development of abilities to transform social reality (Gusso et al., 2021). In view of these points, the importance of formulating learning objectives appropriately is evident. Yet, few teacher education programmes emphasise the formulation of meaningful learning objectives as a component of their training curricula.

Research Objective and Method
The aim of this study was to evaluate the effectiveness of a massive open online course (MOOC) for teacher training, whose general objective was to enable participants to propose meaningful learning objectives. The effectiveness was evaluated through the measure of participant's retention, performance on the topic, satisfaction and learning perception.

The MOOC 'propose meaningful learning objectives'
The main learning objective was to "propose meaningful learning objectives", and involved two specific objectives: to characterize the function of learning objectives in higher education and to characterize false learning objectives. The course was elaborated in MOOC (Massive, Open, Online Course) format and offered in a university educational platform. This platform is focused on courses for teacher training and science education and used, at the time of this study, a customized version of the learning system management (LMS) Moodle (v. 3.8). The courses designed for this platform follow principles derived from behavioural sciences to promote performance, retention, and satisfaction of participants (Gusso et al., 2021).
The course was composed of the following stages: I) completion of the participant's profile; II) initial test; III) three teaching units; IV) final test; V) course satisfaction form. Each teaching unit consisted of content in the form of text or interactive video (h5p), with integrated exercises providing immediate feedback to indicate student success or failure. In case of errors, the participant was asked to redo the activity to progress. In the teaching units, proficiency above 90% was required in the exercises of the unit for the student to progress in the course.
The initial and final tests consisted of 16 questions related to the learning objectives of the course. The initial test was inserted as a parameter of students' performance in the topics before taking the course, to allow comparison with the final performance. To be approved in the final test, the student should obtain a performance above 80%, being able to retake the test as many times as necessary until reaching the proficiency criterion. To retake the final test, the student had to wait at least 30 minutes before doing it again.
The satisfaction form contained 15 multiple-choice questions on different aspects of the course and one open question asking for participants' suggestions, criticisms, compliments, or doubts about the course.

Participants
176 participants enrolled in the course, of which, 107 concluded it with proficiency. Among the concluding participants, the average age was 27.57 years (dp = 8.39), and most (82.24%) were female. 81 (75.7%) completers were undergraduate students, 8 (7.47%) either had a PhD or were enrolled in one, as well as 8 either had a master's degree or were enrolled in one. In addition, 45(42%) people indicated that they had previous teaching or research experience. 94 (87.85%) reported having prior experience with online courses.

Procedures
The course was advertised on the website of the university and on the platform's social networks and was made available between 40 days between August and September of 2021. Data regarding the performance, retention and satisfaction of the participants were extracted from the LMS, organised and anonymised by the platform coordinator using Microsoft Excel (v. Microsoft Excel for Mac 2020). Descriptive and inferential analyses were performed using R Studio software (v.1.4).
For the comparison between the performances in the initial and final test, a one-group pretest-posttest quasi-experimental design was used, using a dependent t-test for paired samples. For effect size analysis, Hedges' g was used as a measure.

Results
The results of the course evaluation were organised into three categories: students' performance, retention and satisfaction and learning perception. Table 1 shows a comparison between the participants' scores in the pre-test (before the course), in the first and in the last attempt of the final test (in which the participant obtained a score above 80% for approval in the course). The average number of attempts to obtain a score above 80% in the final test was 2.85. It is possible to notice an increase in the average score and a decrease in the standard deviation. When comparing the scores using a dependent t-test for paired samples, significant differences were identified both between the pre-test and the first attempt of the post-test, and between the pre-test and the last attempt of the post-test. The calculation of the baseline Hedges' g with the first attempt on the final test resulted in 1.23 (large size), and with the last attempt, 2.15 (large size).

Retention
Out of the 176 enrolled in the course, 107 (60.8%) completed the final test with proficient performance. This retention rate is expressive, given that the average rate in MOOCs platforms is 6.5% (Jordan, 2014). Four possible aspects were considered as possible determinants of this result. One of the reasons that may have contributed to the high level of retention is the audience profile. 87.85% of the people indicated had already had previous experience with online courses, an aspect considered as a predictor of retention (Goopio & Cheung, 2021). Another aspect was the limited period to finish the course, which may have contributed for the participants to devote more attention to explore all the available content. A third aspect is that the data collection was conducted during a period of disruption of face-to-face teaching activities due to the COVID-19 pandemic, which may have increased student engagement in online activities. Finally, the instructional design of the course itself, which incorporates contributions from behavioural sciences to promote retention (Gusso et al., 2021). Table 2 presents the participants' perceptions regarding seven aspects of the course. In the first three rows it is possible to observe a high agreement regarding the learning perception of the three main learning objectives proposed in the course. Also noteworthy is the extremely positive evaluation of the course in relation to its teaching method (95.3% of favourable dispositions), and the high percentage of students who would recommend the course to other people (90.7%). Participants were also asked about the overall assessment of the course experience, using a five-point Likert scale (very good -poor). On this measure, 90.6% indicated favourable dispositions (57% very good and 33.6% good), 7.5% neutral dispositions, and only 1.9% unfavourable dispositions (0.9% regular and 0.9% poor). In the open question of the satisfaction form, which demanded suggestions, criticism, praise, or doubts from the participants, 92 mentions to the course, or to any of its components, were presented. Of these, 46 were compliments, 28 were suggestions, 15 criticisms and 3 doubts. Among the compliments, the most frequent topics were about the course in general (24), about the use of the h5p interactive video lesson resource (13), and about the quantity and types of exercises in the teaching units (13). The suggestions were very diverse, with the two most frequent ones being to replace the content parts in the form of text by interactive video (4), and to adapt the course to also contemplate the needs of teachers of basic education, not only of higher education (3). The main criticism was in relation to the need to retake the final test until obtaining proficiency and the absence of immediate feedbacks in the final test so that students know exactly what they are doing wrong (7).

Conclusion
Considering the high retention rate in the course (60.8%), much higher than reported in other studies with MOOCs, the students' performance in proficiency level (8.67), the very large effect size (Hedges' g=2.15), and the high indicators of satisfaction and perception of learning, it is possible to state that the online course "propose meaningful learning objectives" was effective. The results strengthen the idea that well designed online and massive courses can be a good contribution to assist university teachers training.
In addition, the number of compliments and suggestions to expand the use of the video-interactive resource is remarkable. To the participants, it seems to be more satisfying to watch video lessons in the online system than to read texts, even if they are short. It would be important to evaluate the effects of a possible replacement of texts by videos on performance, retention, and student satisfaction in future studies.