Positioning large language model artificial intelligence tools within discourse analysis: opportunities, challenges and ethical considerations

Artificial Intelligence (AI) tools based on Large Language Models (LLMs) such as ChatGPT has generated significant interested within the higher education sector. The threats, challenges and opportunities for its use in teaching and learning continue to be discussed widely, however it’s use within research and especially research involving marginalised perspectives is far less discussed. This panel will share how ChatGPT was used to add value to the conversations between researchers applying critical discourse analysis exploring Indigenous Australian perspectives with international students. As part of their study the researchers compared the efficacy of ChaptGPT and NVIVIO and the impacts on iterative discourse discussions between the researchers. While the data in this research revealed some very encouraging results, it also highlighted significant areas that need to be explored even further especially around ethical use of AI and untangling in-built biases within the tools’ algorithms.


Introduction
Tools such as NVIVO have been used successfully within various forms of qualitative studies and while the results from these tools are comprehensive, researchers often still need to engage in iterative conversations as part of the 'meaning making' process.This is often very time consuming.Various artificial intelligence tools based on large language models (LLMs) such as ChatGPT may provide an alternative and complementary approach to conduct qualitative analysis.

Existing uses of Artificial Intelligence Tools in Qualitative Studies
There is a range of discussions going on about the use of AI in teaching, learning and research.In research for example, on the one hand arguments support its use to improve efficiencies, while on the other hand, arguments sound alarms related to academic integrity and privacy.Alongside these debates exists questions around biases within various algorithms that drive these tools.To remain responsible and relevant to Indigenous perspectives, for example, we should expand the AI 'alarm bells' to include real world biases coded into AI algorithms (Raji & Buolamwini) and the creation of unprecedented manifestations of discrimination and bias (James & Whelan, 2022).Some notable examples of AI use in qualitative analysis includes, a study by Wang et al. (Wang et al., 2023) that explored the use of ChatGPT as a sentiment analyser against existing benchmark Natural Language Processing (NLP) models such as the Bidirectional Encoder Representations from Transformers (BERT) and the State of the Art (SOTA).While ChatGPT does not use the same algorithmic foundations as BERT and SOTA, this study indicate that while the use of ChatGPT was not perfect, it came very close to the sentiment analysis tools used.Similarly, computer-based NLP tools such as Topic Modelling and Word2Vec were tested by Leeson et al. (Leeson et al., 2019) to test if the results generated were conceptually similar to the open coding methods regularly used in qualitative research.The results indicate that these computer-based modelling tools provided very similar results to manual coding done by researchers.

ChatGPT use in Qualitative Data Analysis of Interview Transcripts
In a study conducted by the researchers, large, rich, descriptive textual data was collected from semi-structured interviews of international students engaged in an immersive Indigenous Orientation program funded by a research-intensive Australian university.These interviews were informed by researchers from internationalisation of the curriculum and an Indigenous scholar, all from different universities in Australia.The interviews were designed to ensure that the students were allowed to ground their experiences within their own cultural identities.About 20 students participated in pre-program and post-program interviews and generated rich narrative and reflective data.
The researchers used critical discourse analysis (CDA) (Fairclough, 2013) framework and the Theory of Communicative Action (Bonell & Melendez-Torres, 2023) to explore the data.NVIVO was used to identify common themes along the way.To identify further themes and to experiment with AI, ChatGPT was used to see if the themes matched NVIVO's and the researchers' responses.

Questions that arise from that study?
While the data analysis from the ChatGPT experiment was extremely insightful, it brought to fore some pertinent questions: 1. What types of data can we analyse through LLMs? 2. How do we verify the validity of these analyses and results? 3. Is the technology mature enough for reliability and validity of analysis?4. How do we navigate ethical considerations of LLM use for such purposes?5. Can tools such as ChatGPT be prevented from learning from our research endeavours, and is that counterproductive to help develop the tool into a mature research tool?The data that we analysed through ChatGPT required nuanced understanding about some of the concepts through intercultural and Indigenous perspectives.While the researchers themselves were able to apply existing qualitative frameworks and engage in iterative meaning making processes, the use of ChatGPT, in this particular case, had a significant impact on the way that the data was analysed and interpreted.For the researchers, it added significant value to the conversations about the data.This panel will be composed of some of the researchers in this study, as well as independent researchers who have used technology based textual analysis tools before.The panel will generate questions from the audience to participate through an interactive Q&A throughout the discussion.