Volume 5, Issue 1 December 2022, pp. 34–51 
Regular Articles
The processes of rating L2 speaking performance using an analytic rating scale – A qualitative exploration
Thuy Thai1, & Susan Sheehan2
1 University of Huddersfield, UK 
2 University of Huddersfield, UK
DOI: https://doi.org/10.29140/lea.v5n1.777
Abstract
In language performance tests, raters are important as their scoring decisions determine which aspects of performance the scores represent; however, raters are considered as one of the potential sources contributing to unwanted variability in scores (Davis, 2012). Although a great number of studies have been conducted to unpack how rater variability impacts on rating decisions, much still remains unclear about what raters actually do when they assess speaking performances. This paper aimed to extend our understanding of the rating process by analysing think-aloud protocols provided by 13 VSTEP speaking raters while they used an analytic rating scale to assess 15 recorded performances with varying proficiency levels. The data suggested that although all the raters seemed to experience similar stages in the rating process, differences in rating behaviour between novice raters and experienced raters exist. To finalise the scores, the raters used five different ways of deciding the scores.
Copyright
© Thuy Thai, Susan Sheehan
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Suggested citation
Thai, T., & Sheehan, S. (2022). The processes of rating L2 speaking performance using an analytic rating scale – A qualitative exploration. Language Education and Assessment, 5(1), 34–51. https://doi.org/10.29140/lea.v5n1.777
Author biodata
Thuy is a PhD holder in language testing and assessment at School of Education and Professional Development, University of Huddersfield, the UK. Her main interests are speaking assessment, rater cognition, and classroom-based assessment.
Related Articles:
Promoting acts of kindness on campus: Views of Chinese international students in the UK
Ming Cheng, Olalekan Adekola
Intercultural Communication Education Published: 30 April, 2022, Volume 5(1), 17–32. 
Tasks, self-efficacy, and L2 motivational self system in an online emergency EFL speaking class: A mixed-methods study
Nguyễn Nhật Quang, Phạm Nhật Linh, Nguyễn Thị Thu Hiền
The JALT CALL Journal Published: 2 April, 2022, Volume 18(1), 1–33.
Implementing backward design to foster intercultural communicative competence in textbook-based curricula: A proposed framework for English language practitioners
Hiba B. Ibrahim
Intercultural Communication Education Published: 30 April, 2022, Volume 5(1), 1–16.
Challenges and responses: A Complex Dynamic Systems approach to exploring language teacher agency in a blended classroom
Grace Yue Qi, Yuping Wang
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 54–82.
The effects of corpus use on learning L2 collocations
Yoshiho Satake
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 34–53.
Captions and learnability factors in learning grammar from audio-visual input
Anastasia Pattemore, Carmen Muñoz
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 83–109.
Chinese language learners evaluating machine translation accuracy
Li-Ching Chang
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 110–136.
Statistical analysis of the impact of the e-learning platform Furago on French learners’ listening skills
Albéric Derible, Éric Wiel
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 137–161.
Work integrated language learning: Boundary crossing, connectivity, and L2 affordances
Linda Febring, Alastair Henry
Migration and Language Education Published: 29 April, 2022, Volume 3(1), 1–22.
The role of literature in intercultural language education
Melina Porto, Michalinos Zembylas
Intercultural Communication Education Published: 10 December, 2022, Volume 5(3), 86–104.