Beyond perception: Longitudinal insights into chatbot-assisted EFL learning

Abstract

While AI chatbots are increasingly integrated into English as a Foreign Language (EFL) instruction, many prior studies emphasize learner perceptions rather than objective linguistic development. This longitudinal study analyzed weekly chatbot interactions from first-year English majors in Japan (n = 18) over two academic terms. Using chat logs analyzed with paired t-tests, and ANOVAs, this study tracked lexical diversity, in terms of type token ratio (TTR), mean sentence length, expanded answers, self-correction, total turns, and follow-up questions. Results showed initial gains but uneven patterns across terms. In Spring, students significantly increased their total turns (t(17) = 2.42, p = .028, dz = 0.57) and demonstrated a steady rise in self-correction across sessions (F(10, 170) = 3.34, p < .001, eta^2 = .16), while sentence length and expansion trended upward without reaching significance. In Fall, lexical diversity increased significantly (t(17) = 5.51, p < .001, dz = 1.30), and follow-up questions showed a marginal increase (t(17) = 2.17, p = .055, dz = 0.51), while expanded answers declined. Overall, the data suggested recurring trends across semesters, though the specific areas of growth differed. These findings indicate that chatbots can foster early improvements in sentence elaboration, interaction, and self-monitoring, but vocabulary and expansion plateau without varied pedagogical support. Implications include diversifying chatbot tasks, setting explicit lexical goals, and integrating teacher mediation to sustain growth.