Comparative Analysis of Human Graders and AI in Assessing Secondary School EFL Journal Writing

Abstract

This study conducts a comprehensive analysis of the assessment of journal writing in English as a For-eign Language (EFL) at the secondary school level, comparing the performance of a Generative Artificial Intelligence (GenAI) platform with two human graders. Employing a convergent parallel mixed methods design, quantitative data were collected from 389 assignments of 91 students in a private school in Is-tanbul during the first semester of the 2023-2024 academic year, evaluated by both the GenAI platform and human graders. Qualitative data involved analyzing feedback from both sources. The study aimed to compare grading performance, assess the GenAI platform's consistency and effectiveness, and exam-ine feedback quality. Results indicated a high level of agreement between the GenAI platform and hu-man graders, suggesting the GenAI platform can effectively simulate an English teacher's role in an EFL context. Limitations include the restricted sample size, the study's specific context, and potential variabil-ity in evaluations. Findings highlight the potential for integrating GenAI in EFL assessment, though hu-man feedback remains crucial for personalized and emotionally supportive feedback. The conclusion emphasizes the GenAI platform's promise in enhancing feedback efficiency and comprehensiveness, while recommending future research to explore evaluation criteria, long-term impacts, and ethical con-siderations.