Volume 5, Issue 1 December 2022, pp. 18–33 Download PDF

Regular Articles

Role of expert judgement in language test validation

David Coniam https://orcid.org/0000-0003-4480-17421, Tony Lee https://orcid.org/0000-0003-1222-00382, Michael Milanovic https://orcid.org/0000-0002-5722-18113, Nigel Pike https://orcid.org/0000-0002-6260-012X4, & Wen Zhao https://orcid.org/0000-0003-4965-01465

1 PeopleCert, UK david.coniam@peoplecert.org
2 LanguageCert, UK Tony.Lee@PeopleCert.org
3 LanguageCert, UK Michael.Milanovic@PeopleCert.org
4 LanguageCert, UK Nigel.Pike@PeopleCert.org
5 School of Foreign Studies, Jinan University, CHINA dianawen@hotmail.com

DOI: https://doi.org/10.29140/lea.v5n1.769


Abstract

The calibration of test materials generally involves the interaction between empirical anlysis and expert judgement. This paper explores the extent to which scale familiarity might affect expert judgement as a component of test validation in the calibration process. It forms part of a larger study that investigates the alignment of the LanguageCert suite of tests, Common European Framework of Reference (CEFR), the China Standards of English (CSE) and China’s College English Test (CET).

In the larger study, Year 1 students at a prestigious university in China were administered two tests – one with items based on China’s College English Test (CET), and the other a CEFR-aligned test developed by LanguageCert. Comparable sections of the CET and the LTE involved sets of discrete items targeting lexico-grammatical competence.

In order to ascertain whether expert judges were equally comfortable placing test items on either scale (CET or CEFR), a group of professors from the university in China who set the CET-based test, were asked to expert judge the CET items against the nine CSE levels with which they were very familiar. They were then asked to judge the LTE items against the six CEFR levels, with which they were less familiar. Both sets of expert ratings and the test taker responses on both tests were then calibrated within a single frame of reference and located on the LanguageCert scale

In the analysis of the expert ratings, the CSE-familiar raters exhibited higher levels of agreement with the empirically-derived score levels for the CET items than they did with the equivalent LTE items. This supports the proposition that expert judgement may be used in the calibration process where the experts in question have a strong knowledge of both the test material and the standards against which the test material is to be judged.



Copyright

© David Coniam, Tony Lee, Michael Milanovic, Nigel Pike, Wen Zhao

CC  4.0
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.


Suggested citation

Coniam, D., Lee, T., Milanovic, M., Pike, N., & Zhao, W. (2022). Role of expert judgement in language test validation. Language Education and Assessment, 5(1), 18–33. https://doi.org/10.29140/lea.v5n1.769


Related Articles:

Promoting acts of kindness on campus: Views of Chinese international students in the UK
Ming Cheng, Olalekan Adekola
Intercultural Communication Education Published: 30 April, 2022, Volume 5(1), 17–32. Download PDF


Tasks, self-efficacy, and L2 motivational self system in an online emergency EFL speaking class: A mixed-methods study
Nguyễn Nhật Quang, Phạm Nhật Linh, Nguyễn Thị Thu Hiền
The JALT CALL Journal Published: 2 April, 2022, Volume 18(1), 1–33. Download PDF


Implementing backward design to foster intercultural communicative competence in textbook-based curricula: A proposed framework for English language practitioners
Hiba B. Ibrahim
Intercultural Communication Education Published: 30 April, 2022, Volume 5(1), 1–16. Download PDF


Challenges and responses: A Complex Dynamic Systems approach to exploring language teacher agency in a blended classroom
Grace Yue Qi, Yuping Wang
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 54–82. Download PDF


The effects of corpus use on learning L2 collocations
Yoshiho Satake
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 34–53. Download PDF


Captions and learnability factors in learning grammar from audio-visual input
Anastasia Pattemore, Carmen Muñoz
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 83–109. Download PDF


Chinese language learners evaluating machine translation accuracy
Li-Ching Chang
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 110–136. Download PDF


Statistical analysis of the impact of the e-learning platform Furago on French learners’ listening skills
Albéric Derible, Éric Wiel
The JALT CALL Journal Published: 9 April, 2022, Volume 18(1), 137–161. Download PDF


Work integrated language learning: Boundary crossing, connectivity, and L2 affordances
Linda Febring, Alastair Henry
Migration and Language Education Published: 29 April, 2022, Volume 3(1), 1–22. Download PDF


The role of literature in intercultural language education
Melina Porto, Michalinos Zembylas
Intercultural Communication Education Published: 10 December, 2022, Volume 5(3), 86–104. Download PDF