Volume 5, Issue 1 December 2022, pp. 18–33 
Regular Articles
Role of expert judgement in language test validation
David Coniam 
1, Tony Lee 2, Michael Milanovic 3, Nigel Pike 4, & Wen Zhao 5
1 PeopleCert, UK 
2 LanguageCert, UK
3 LanguageCert, UK
4 LanguageCert, UK
5 School of Foreign Studies, Jinan University, CHINA
DOI: https://doi.org/10.29140/lea.v5n1.769
Abstract
The calibration of test materials generally involves the interaction between empirical anlysis and expert judgement. This paper explores the extent to which scale familiarity might affect expert judgement as a component of test validation in the calibration process. It forms part of a larger study that investigates the alignment of the LanguageCert suite of tests, Common European Framework of Reference (CEFR), the China Standards of English (CSE) and China’s College English Test (CET).
In the larger study, Year 1 students at a prestigious university in China were administered two tests – one with items based on China’s College English Test (CET), and the other a CEFR-aligned test developed by LanguageCert. Comparable sections of the CET and the LTE involved sets of discrete items targeting lexico-grammatical competence.
In order to ascertain whether expert judges were equally comfortable placing test items on either scale (CET or CEFR), a group of professors from the university in China who set the CET-based test, were asked to expert judge the CET items against the nine CSE levels with which they were very familiar. They were then asked to judge the LTE items against the six CEFR levels, with which they were less familiar. Both sets of expert ratings and the test taker responses on both tests were then calibrated within a single frame of reference and located on the LanguageCert scale
In the analysis of the expert ratings, the CSE-familiar raters exhibited higher levels of agreement with the empirically-derived score levels for the CET items than they did with the equivalent LTE items. This supports the proposition that expert judgement may be used in the calibration process where the experts in question have a strong knowledge of both the test material and the standards against which the test material is to be judged.
Copyright
© David Coniam, Tony Lee, Michael Milanovic, Nigel Pike, Wen Zhao
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Suggested citation
Coniam, D., Lee, T., Milanovic, M., Pike, N., & Zhao, W. (2022). Role of expert judgement in language test validation. Language Education and Assessment, 5(1), 18–33. https://doi.org/10.29140/lea.v5n1.769
Related Articles:
Promoting acts of kindness on campus: Views of Chinese international students in the UK
Ming Cheng, Olalekan Adekola
Intercultural Communication Education Published: 30 April, 2022, Volume 5(1), 17–32. 
Tasks, self-efficacy, and L2 motivational self system in an online emergency EFL speaking class: A mixed-methods study
Nguyễn Nhật Quang, Phạm Nhật Linh, Nguyễn Thị Thu Hiền
The JALT CALL Journal Published: 2 April, 2022, Volume 18(1), 1–33.
Implementing backward design to foster intercultural communicative competence in textbook-based curricula: A proposed framework for English language practitioners
Hiba B. Ibrahim
Intercultural Communication Education Published: 30 April, 2022, Volume 5(1), 1–16.
Challenges and responses: A Complex Dynamic Systems approach to exploring language teacher agency in a blended classroom
Grace Yue Qi, Yuping Wang
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 54–82.
The effects of corpus use on learning L2 collocations
Yoshiho Satake
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 34–53.
Captions and learnability factors in learning grammar from audio-visual input
Anastasia Pattemore, Carmen Muñoz
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 83–109.
Chinese language learners evaluating machine translation accuracy
Li-Ching Chang
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 110–136.
Statistical analysis of the impact of the e-learning platform Furago on French learners’ listening skills
Albéric Derible, Éric Wiel
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 137–161.
Work integrated language learning: Boundary crossing, connectivity, and L2 affordances
Linda Febring, Alastair Henry
Migration and Language Education Published: 29 April, 2022, Volume 3(1), 1–22.
The role of literature in intercultural language education
Melina Porto, Michalinos Zembylas
Intercultural Communication Education Published: 10 December, 2022, Volume 5(3), 86–104.