Assessing AI-generated academic writing in Spanish: a multivariate analysis of syntactic variation across ChatGPT, Gemini, and Claude

Abstract

Research on large language models has increasingly examined their capacity to produce academically acceptable texts, yet the structural properties of AI-generated academic writing remain underexplored, particularly in Spanish. This study examines whether three proprietary large language models, ChatGPT, Gemini, and Claude, generate distinguishable syntactic configurations in academic Spanish and whether these configurations provide interpretable evidence for language assessment contexts. Using a controlled intra-task design, a corpus of 90 texts was generated under identical prompts, the same source text and task instructions as supporting material, and standardized production conditions. The design did not include a human-written baseline corpus; therefore, the study addresses inter-model syntactic variation rather than human-AI differentiation. Syntactic structure was operationalized through the Universal Dependencies framework and extracted automatically with Stanza. Normalized frequencies of coordination and subordination relations, together with mean sentence length as a structural control variable, were analyzed through Welch’s ANOVA, MANOVA, PERMANOVA, principal component analysis, linear discriminant analysis, and Mahalanobis distances with bootstrap confidence intervals. Results showed significant differences across all syntactic variables, with the strongest effects concentrated in coordination and more moderate differences in subordinate constructions. Multivariate analyses confirmed robust structural separation across models, and linear discriminant analysis achieved 91.1% classification accuracy. These findings indicate that the models generate academic Spanish through differentiated syntactic configurations. For language assessment, the results suggest that dependency-based syntactic profiling can support the interpretation of AI-assisted academic writing, inform rubric refinement for academic Spanish, and provide transparent features for automated writing evaluation and model-attribution procedures under controlled conditions.