Identifying the components of foundational Artificial Intelligence (AI) literacy - Early results from a Delphi study

This article provides some initial results from the first phase of a Delphi study to identify the critical components of an Artificial Intelligence (AI) literacy curriculum. This article explores the study results that address a four-level capability model, but only the lowest level of this model. The Delphi panel comprised 17 experts in AI, and the first round of the study used a survey to gather the experts' responses in three areas that were based on previous literature - knowledge (and concepts) of AI, skills related to AI, and understanding AI in context. A structured thematic analysis revealed several themes under these categories. For foundational knowledge and concepts, it was determined that three areas were needed, namely 1) what is AI? 2) applications of AI, and 3) AI technologies. Skills were divided into cognitive and technical skills, with cognitive skills further divided into 21st-century and applied skills. Understandings comprised social issues, risks, and debates. The repeated ideas that formed these themes gave rich insights into how an AI literacy curriculum might be structured and provided a firm foundation for subsequent rounds of the study, which will involve further iteration and consolidation of these ideas.


Introduction
The concept of Artificial Intelligence (AI) literacy has become increasingly prominent in recent years.For example, The World Economic Forum has promoted the need to foster universal AI literacy.It asserts that everyone needs to become literate about AI to know when it is being used and be able to evaluate its benefits and limitations in our lives (WEF, 2022).So, while these literacies are required for everyone, fostering an understanding of AI at a young age is becoming more critical.In particular, understanding how to promote AI literacies in schools will help us respond to a future in which AI is embedded in children's everyday lives (Druga & Ko, 2021).
Recent developments in generative AI technologies have led to an increasing focus on the need for learners to become critical consumers of AI (Trust et al., 2023).Although AI literacy has been discussed for some years, the pervasive nature of contemporary AI systems means that the discussion around what AI literacy is and how it should be developed in education has become more urgent.The call is that "literacy and numeracy remain fundamental, but now we must add AI literacy" (Luckin, 2023, para. 8).The recent advances in AI have meant that AI technologies have become more accessible and in the hands of users.Users are becoming both the source and the unwitting trainers of AI technologies, and previous AI literacy frameworks need to address the fundamental shift that AI has brought where AI has moved from an abstract concept to something that we interact with daily.
To address this challenge, this article reports on the initial phase of a Delphi study intended to articulate an agenda for AI literacy that can form the basis for integrating AI into the curriculum at all levels.Given the need for AI literacy for all learners, the study takes a capability model approach, whereby the assumption is that all learners, no matter their age, should have a basic understanding of AI.The first phase of the Delphi asked experts (n=17) to provide input on a proposed AI literacy framework designed to explore what 1) knowledge and concepts, 2) skills, and 3) understandings all learners need to have to engage with AI.The focus was not on how this should be taught but instead on the different foundations needed to engage with AI, from basic to more comprehensive.This first phase of the Delphi was therefore developed broadly to start exploring AI literacy in general and to test the concepts of this capability model.Further stages of the Delphi and resulting work would then refine and expand on this model to consolidate and refine the framework.This further development will also include how this could be taught at various ages.This article focuses only on the first level of the model, which represents the foundational knowledge expected of all learners.This level represents the initial awareness for the informed use of AI.The structure of this paper first provides a brief background literature review that considers existing viewpoints on AI literacy and how it relates to the curriculum.This is followed by a rationale for undertaking this Delphi study and a statement of the research questions.The study's methodology is then explained, followed by the results we have gathered so far that address the fundamental components of a foundational AI literacy curriculum.The article concludes with some reflections on the findings of this initial round of the Delphi study and discusses how the study will progress to address its broader aims and objectives.

Approaches to AI literacy
AI literacy is considered one of the many digital literacies needed for learners to succeed in today's society (Ng et al., 2021).AI literacy, therefore, sits alongside new or 'multiple' literacies that are considered vital to construct knowledge from multiple sources and modes of representation (Seel & Winn, 2012).
AI literacy was initially conceptualised in simple terms as being able to recognise tasks that can be performed by AI and learning and investing in the human strengths that it cannot replace (Konishi, 2016).Since this concept is too broad to define a specific understanding of the components of AI literacy, many more developed definitions of AI literacy have been proposed.Long and Margerto (2020) presented five AI literacy themes expressed as a series of questions: What is AI?What can AI do?How does AI work?What should AI do?And how do we perceive AI? Subsequent suggestions have included the 'four C's' of AI literacy -concepts (how AI uses data), context (strengths and limitations of AI are context dependent), capability (how AI can be applied) and creativity (finding new ways to apply and benefit from AI technology) (Talagala, 2021), while Cui and Wheatcroft (2021) apply Haberman's categorisation of knowledge by human interests to state that AI literacy could be understood as skills and technical abilities, applications and practical considerations, and transformative thinking, experiences and considerations.Laupichler et al. (2022) have suggested understanding and critically reflecting on AI and being able to apply and interact with AI technologies both personally and professionally as critical literacies.No doubt there are many other ways of thinking about AI literacy, including identifying common aspects that apply across literacies to form 'meta literacies' that emphasise the four learning domains of cognitive, behavioural, affective, and metacognitive (Fulkerson et al., 2017).
Requirements for an AI literacy curriculum UNESCO (2022) notes that although some countries have developed elements of an AI K12 curriculum, implementation so far is limited and inconsistent, and further work must be done in this area.One widely used source for discussing an AI literacy curriculum is Long and Magerko's (2020) conceptual framework, which proposes 17 core competencies of AI literacy.However, this framework does not explore how these capabilities may intersect or what capabilities are considered core and which may be developed later.Therefore, it may serve as one starting point for considering how to approach AI literacies in education, but needs to be further expanded and developed to create an appropriate competency model and associated assessment strategies (Faruqe et al., 2022).Some AI literacies have, therefore, focused more on specific sectors.For example, Kim et al., (2021) propose an elementary school AI curriculum that builds upon the three competencies of AI Knowledge (problem-solving, reasoning, applications), AI Skill (using AI tools, coding), and AI Attitude (social impact and collaboration with AI).A pilot study with 60 students suggested that the proposed curriculum positively impacted the learners' AI literacy levels.However, while not explored, it may be considered that these competencies would be valid for a learner of any age.
An essential component of any approach to AI literacy is to take account of cultural issues.Developing AI literacy that considers cultural concepts also ensures communities in which these concepts are embedded are not negatively affected by the advance and integration of AI into society.Instead, they are empowered to inform its development and play a part in building truly responsible and representative AI.This is particularly important given we are at such a critical time in technology development (the rise of AI and its incorporation into multiple facets of society).We have the opportunity and responsibility to ensure that traditionally marginalised and underrepresented groups are given a voice and the ability to apply their values and principles to influence AI knowledge, AI literacy, and ultimately, this technology's development.
In New Zealand, a Māori perspective on the AI life cycle is based on the cultural concepts of Whakapapa (interconnection of people, planet and purpose), Whanaungatanga (creating, maintaining, and enhancing human and environmental relationships for a sustainable future) and Manaakitanga (inclusion and participation).These concepts support stakeholder identification, understanding ecosystems, and building trust and engagement in AI (World Economic Forum, 2022).To be compatible with these cultural expectations, AI should only use data if it is treated as a resource to be distributed for collective well-being (Stratton, 2022).Therefore, an essential consideration for any capability framework is to consider the cultural context it is designed for.

Approaching AI literacy from a capability model perspective
What can be drawn out from these proposals is that AI literacy encompasses multiple aspects that include understanding the role of AI in the world, knowing what it is capable of, and being able to engage with it both from a technical perspective and by applying a critical lens that addresses broader social and cultural perspectives.In this study, we have characterised these various perspectives under three broad categories: knowledge of AI, skills related to AI and an understanding of AI in its wider contexts.
One reason there are many different approaches to AI literacies is that we have to consider developmental and learning contexts when deciding what is appropriate.When developing an approach to AI literacies in education, a careful analysis of age and stage is required in developing a curriculum.For that reason, in this study, we explored AI literacy in terms of a capability model.The initial conception of this model was exploring the three categories identified above, based on different capability levels, moving from what would be expected for all learners to a deeper understanding that may be expected from learners who are specialising in AI.Therefore, in addition to the three broad categories, we address how these levels deepen as users develop their capabilities.These levels were not built around the learner's age but instead focused on how the three broad categories scaffold a deeper understanding of AI, which may move from concept to development.However, how these categories are taught would depend on the learner's age.
Framed by the existing research in AI literacy, we have considered how AI literacy can be conceptualised and what new capabilities are needed for learners to engage with the rapidly changing landscape of AI.The paper's research question is to identify the key components of an Artificial Intelligence (AI) literacy curriculum, specifically at the foundational level of a four-level capability model.The study explores what knowledge concepts, skills, and understandings all learners need to engage with AI.The following section explores the approach adopted in this study to answer this question.

Methodology
Artificial intelligence has recently become so pervasive in the education space that many voices are expressing their opinions, experiences, hopes and fears through multiple channels.With so much debate, it is hard to focus on the fundamental questions.One way of doing so is to seek experts' opinions, which is why we have chosen to undertake a Delphi Study of experts in Artificial Intelligence.Most of the experts (n=17) recruited for the study were already known to the authors as being recognised in their fields.Invitations to participate were also shared through various professional channels.Because the study focused on considering how AI literacy might be integrated into the curriculum, our experts came from education (n=11) and industry (n=5), with one respondent covering both education and industry roles.A total of 83 people opened the survey, but the majority of these did not complete it.The survey was demanding to complete and did require extensive knowledge of AI, and it appears likely that many of those who viewed the survey did not feel able to complete it effectively.Most of those who completed the survey were those we had directly invited as recognised experts in AI.Only 17 respondents fully completed the first round of the Delphi survey.Still, the detail and expertise that was provided meant that the data provided a firm foundation for proposing some initial findings to be fed back into the next round of the Delphi study.
The respondents from the education sector held roles that either taught AI, including tertiary educator (n=4, professor to lecturer) and high school or primary teacher/principal (n=3), and those that supported these teaching roles, lab technician, learning support or designer (n=4).The years of teaching were diverse, with an even split between those who have taught up to 10 years (n=8) and those who have taught 11 and more (n=7).
Those in the industry came from a diverse mix of experts that included Senior Quality Editors, Senior Principal Data Scientist, Special AI Projects Lead, and founders of businesses with AI as part of their business (n=3).Of these, one specialised in Māori cultural and AI ethics and the other privacy, cyber security and online safety.
Of the experts, most (n=12) had up to 5 years of experience in AI, while one educator had 11-20 years.Two respondents had 21-30 years of experience (one from academia and the other representing both education and industry), while one industry respondent had over 30 years of experience.The experts covered a wide range of AI expertise, including AI ethics, building AI systems and probabilistic models, and general and specialist teaching of AI concepts.
As this study has been based in New Zealand, there are a large number of experts from New Zealand (n=11) in the study, but there are also several contributors from other countries (Australia, Canada, South Africa, UK, USA).The study may, therefore, be characterised as one that deliberately focuses on New Zealand education for it to be as relevant as possible to the authors and their networks, while also considering the views of noneducators and those from other countries to ensure a broad understanding is attained.This is not the first Delphi study that has addressed AI literacy.For example, Laupichler et al. (2023) undertook a Delphi study to assess the AI literacy of non-experts.However, that study focused on a set of assessment items rather than addressing the broader concepts that comprise AI literacy and began by providing a specific model for the experts to respond to.The intention of this study explored in the article was to address a broader set of ideas about an AI literacy curriculum at multiple levels (not just non-experts).
To gather the opinions of the Delphi panel, a survey was constructed based on four capability levels.These were • Level 1: Informed -initial awareness / foundational knowledge.
• Level 3: Engaged -implementing and embedding concepts and tools.
For each capability level, we asked questions about knowledge/concepts.skills and understandings, as follows • What are the knowledge/concepts that you believe should be covered in each level of the capability model?
• What are the skills that you think should be covered in each level of the capability model?In this section, you might include a reference to any tools/activities the learner should be engaging in for each level • What are the understandings (i.e., social, cultural, privacy, security and/or ethical issues, etc.) that you believe should be covered in each level of the capability model?
The survey included further questions about the respondents' opinions about the capability model itself.
Educators were asked to provide information about how and if AI was currently being taught in their institutions, and a range of demographic data was gathered, including age, role, experience in AI, highest qualification and specialisation within the AI field.
Given length constraints, this article only discusses the responses to level 1 of the capability model.This has been chosen because level 1 constitutes the components of AI literacy that would apply to all learners, and it provides the foundation for all the higher levels of the capability model.
The data was primarily analysed using structural coding since the survey provided explicit questions that were asked, under which the results might reasonably be analysed.However, within this thematic analysis, the data revealed that, in some cases, certain themes occurred across different questions because different respondents chose to categorise them in slightly different ways.Therefore, although the high-level constructs used in the analysis were taken from the three question areas of knowledge, skills, and understandings, in some cases, the data has been categorised under a category different from the one in which it originally appeared.

Results
Table 1 shows the result of coding the survey data using thematic analysis, identifying repeated ideas by using constant comparison, clustering these into themes and then positioning these under the three constructs taken from the original survey questions.Knowledge and concepts of AI are divided into three areas of knowledgedescribing AI, applications of AI, and AI technologies.Skills are primarily divided into cognitive and technical skills, while cognitive skills are further divided into 21st-century skills and applied skills.Understandings comprise broad social issues, risks, and debates about AI.Details of how these themes have been arrived at are discussed in the next section.

Themes from the Analysis
Many of the themes identified in the data encompassed a broad range of repeated ideas that provide multiple perspectives on AI literacy.As can be seen from Table 1, the three most commonly repeated ideas were, what is AI? current applications, and ethics.Some ideas shown as repeated include some that have been clustered as 21st-century skills.Although some were only mentioned once, the common use of these, together with critical thinking in many contexts led to them being grouped together here.
This section briefly illustrates some of the themes by referring to quotes from the original data.For each source, the case number and their role are indicated.
The The other repeated ideas in this theme were key AI concepts, including: The relationship between the human brain and machine models (2 -Senior Principal Data Scientist).
The skills theme is divided into technical and cognitive skills.Within cognitive skills were a group of what are often called 21st-century skills: collaboration, communication, creativity, critical thinking, and problem-solving.Of these, problem-solving came out as the most mentioned.The other category of cognitive skills was applied skills, which included computational thinking, logical reasoning, mathematics and statistics and researching AI.Computational thinking encompasses a set of other skills that, it could be argued, overlap strongly with 21stcentury skills.
including problem-solving, algorithmic reasoning, and logical thinking.(11 -Māori cultural and AI ethics advisor) Technical skills were related to the skills required to work with coding, AI tools, and data, including being able to work effectively with prompts.Understanding data was seen as particularly important, covering how to manage and process data, how to use it to make decisions, and also be aware of broader issues such as: ownership, quality, reliability, privacy laws (2 -Senior Principal Data Scientist) In the understandings construct, the most commonly occurring theme was risk, with ethics coming to the fore, including, for example: algorithmic fairness, privacy concerns, transparency, accountability, and the potential social and economic impacts of AI deployment (4 -Associate Principal) Privacy was also highlighted separately, as was responsibility, which relates to accountability: What could go wrong?Is it the fault of AI, or the people who misuse it?(3 -Senior Technician) Bias was mentioned in five cases, including: cultural biases embedded in AI systems (4 -Associate Principal) The concept of 'error' included being aware of both the fact that AI systems could make mistakes and that they could use/generate 'fake' data.
Security also came through as a repeated idea with the risks of hacking and fraud.
Under the theme of broad social issues, the two repeated ideas centred on culture and its impact on people.This impact was seen as both positive and negative, but the emphasis was mainly on the potential negative consequences, e.g., Students should grasp the political dimensions of AI, including its impact on governance, policymaking, and power structures.This involves studying AI's influence on decision-making processes, accountability, surveillance, and the potential for bias or discrimination.(4 -Associate Principal) However, there were some more positive voices: How AI can aid and support humanity.AI works best when used alongside human knowledge (10 -Teacher) Finally, the theme addressing debates about AI covered current issues, discussion and analysis, and questions about who AI is created by, and for whom?Several areas for learners to engage in critical thinking were suggested, such as.

Critique of the model
In terms of the model itself, all but one respondent was positive about the approach of this model, including levels and three categories.One respondent did not explicitly criticise it but rather said they could not determine the underlying structure and direction intended for the development of the model' and therefore did not provide any feedback on the model itself.(7 -Senior Technician) However, there were some aspects that the respondents recommended to consider moving forward.They include that, AI is incredibly broad, and it might be useful to separate different kinds of AI in this model (9-Learning Experience Designer), and there is overlap between knowledge and understanding (15-Assistant Professor).
It was also recommended that the model must be flexible and adaptable to support the evolving nature of AI, and as a model itself (5-Technology Integrator), it would need to be expanded to provide practical details on how this would be taught or developed.In particular, five respondents (8-Business Founders, 9-Learning Experience Designers, 11-Business founders and Māori cultural and AI ethics advisors, 16 -Flexible Learning Advisors, 17-Special Projects Managers) highlighted that teaching each capability would depend on the age of the learner, for example, in particular, one respondent stated: Age maturity appropriateness is needed on some of the more complex subjects such as neural language processes, programming such as Python, concepts such as layering, or harms and ethics?(11-Business founder and Māori cultural and AI ethics advisor) Therefore, it was recommended that the later layers may be separated out based on focus and age.In addition, another respondent also highlighted that, In Aotearoa Te Ao, Māori needs to be covered at the same time to 'normalise' Māori world view and knowledge, even about the technical aspects, such as computational thinking, programming, etc….I think also there need to be statements about whose knowledge and concepts and frameworks are majoratively spoken about, to understand the issues of marginalised, incomplete, biased knowledge and data.(11-Business founder and Māori cultural and AI ethics advisor) So, while most respondents felt that the approach made sense, a few (n=3) respondents highlighted the notion of having this as a linear model; in particular, one respondent stated, The proposed … model is generally well-structured, following a clear progression from foundational knowledge to active participation.However, a few suggestions and considerations might help improve it: Overlap and Reinforcement: Reinforcing concepts at each level ensures deeper understanding and retention, especially for crucial topics like ethics and security.
Measuring Progress: Defining clear, concrete methods to assess competency at each level is crucial for tracking development and ensuring the learner is ready to progress to the next stage.
Another respondent reinforced this idea by adding, I think that people need to gain an understanding of a topic through a variety of processes such as reading/literature, reflection, and doing.Debate would also be a good activity.Ideally, each person should form some idea or conjecture that they will then test through the work they do (using a refutational model, ideally).(3-Senior Technician) Therefore, it wouldn't necessarily mean that students would progress through the different layers but instead continuously build on the layers through different approaches to cement understanding.This was echoed by another respondent who said, While the framework is useful, I think that in the case of AI literacy in particular, that learning will rely on dynamic cycles of knowledge acquisition and application, with experimentation as the cornerstone of skills development and understanding.(9-Learning Experience Designer) Two respondents (17-Special Projects Manager, 10-Primary School Teacher) found the levels and categories aligned with the refreshed New Zealand Curriculum.In particular, they found the model related well to 'The know, do, understand' framework adopted in the refresh.And that some of the later layers of the model would be covered under the specialised learning areas of the curriculum while the lower layers may sit within other learning areas.

Conclusion
Recent developments in the application of generative AI technologies have highlighted the importance of AI literacy as a core, fundamental competence.Developing an AI literacy curriculum for all levels of education is needed to prepare individuals for a future in which AI is embedded from profession to everyday life.Our next generation requires the knowledge, skills and understanding to allow them to become critical consumers of AI, and given the pervasive nature of this technology, these must begin at a young age.
This article presents several key components of an AI literacy curriculum based on the outcome of a Delphi study with insights from AI subject matter experts from both industry and education.Thematic analysis of survey responses uncovered eight themes under the three primary constructs from the survey: AI technologies, applications of AI, describing AI, cognitive skills, technical skills, broad social issues, AI debates, and risks.
Responses indicated three ideas to be the most important: knowledge of how AI is being used today (current applications of AI), understanding what AI actually is, and the ethical considerations of AI such as fairness, privacy, transparency and accountability.

As one respondent highlighted,
There are many misconceptions about AI; what it is and what it isn't.The media doesn't help much in this regard by sensationalising fairly mundane topics.The main thing that people need to understand is that the dangers of AI are not the AI per se, but the way that people can and will use it.It accelerates abilities that we as humans have had for a long time in terms of influencing behaviour.We are seeing that in the way 'the algorithm' works.So, learning about AI is as much learning about cognitive science and human behaviours as learning about technology.(3 -Senior Technician) This study has provided a framework that lays valuable groundwork for the development of an AI literacy curriculum.The capability model presented to the Delphi panel consisted of four capability levels, ranging from initial awareness and foundational knowledge to creating or applying AI (though this article only presents a discussion on responses to the first level).Areas of potential future work include refinement of the capability model across all these levels through further study iterations and development of assessment metrics to measure AI literacy proficiency (and effectiveness of the related curriculum).Another area of future work is exploring the skills educators require to efficiently and effectively convey AI literacy knowledge, skills and understanding to their students at different levels.

Table 1 : Coding of the repeated ideas, themes and constructs from the survey data using thematic analysis
Familiarity with machine learning techniques and algorithms is vital.This includes knowledge of supervised learning, unsupervised learning, reinforcement learning, and deep learning architectures like neural networks, (4-Associate Principal)The applications of AI were mostly addressed as the need to be aware of how and why AI is already being used in our daily lives.Describing AI was heavily weighted to asking the simple question 'What is AI?' reiterated in various forms by ten respondents.Understanding the history of AI was also seen as important, as well as being able to place it in relation to other technologies, for example, Defining AI and placing it in the context of broader technological evolution AI technologies theme contained many responses suggesting what types of technologies should be learned about.These were either at a high, conceptual level: Differentiating between AI, Machine Learning, and Deep Learning (12 -Senior Lecturer) Or referring to specific technologies like machine learning: