Evaluating the Impact of an AI Critical Friend on Student Research Proposals, Critical AI Literacy, and Transparent Collaborative Assessment Practices

Abstract

This paper presents a work-in-progress report on an innovative approach to integrating generative AI capabilities into the assessment process, focusing on research proposal development. Specifically, the study explores the impact of incorporating an AI-powered "critical friend" that provides targeted feedback and guidance to students as they craft their research proposals. The aim is to evaluate how this AI-powered support system affects students' critical AI literacy, the quality and transparency of the collaborative assessment process, and, ultimately, the strength of the final research proposals. Through a mixed-methods approach combining surveys, focus groups, and analysis of student work and interactions with generative AI, the researchers seek to uncover key insights on the benefits, challenges, and ethical considerations of leveraging AI to enhance the research proposal development experience. The findings from this study will contribute to the growing body of literature on the role of AI in academic assessments and provide practical recommendations for educators and institutions looking to responsibly integrate these emerging technologies.