Validating automated complexity, accuracy, and fluency metrics for L2 writing analysis

Abstract

This study evaluates the construct validity of automated complexity, accuracy, and fluency (CAF) metrics for AI-assisted analysis of L2 writing. As automated feedback systems and learner analytics increasingly rely on computational indicators to evaluate learner texts at scale, it is necessary to establish whether these metrics support defensible interpretations before pedagogical or longitudinal use. Using a cross-sectional dataset of 1,000 Korean EFL learner texts and minimally corrected counterparts generated by a fine-tuned grammatical error correction model, we examined candidate CAF indicators within a structural equation modeling framework. Accuracy was operationalized using GLEU and edit/error count, fluency using raw perplexity (PPL) and delta perplexity (ΔPPL), and complexity using selected L2SCA indices. GLEU and edit/error count converged but were redundant, supporting GLEU as a single observed accuracy indicator. Raw PPL and ΔPPL also converged in isolation, but their latent fluency specification became inadmissible in the full model. For complexity, a three-indicator factor comprising MLT, MLC, and CN/C produced a stable measurement structure. In the final admissible SEM, ΔPPL showed stronger covariance with GLEU, whereas raw PPL showed stronger covariance with Complexity, indicating a trade-off between correction-normalized fluency and construct separation. The findings support a reduced framework while highlighting the need to compare PPL and ΔPPL further in longitudinal modeling.