An empirical examination of the effect of guessing on vocabulary size test scores

Abstract

The Vocabulary Size Test (VST) was created to provide a reliable estimate of a second language learner’s written receptive vocabulary size, measuring from the most frequent fourteen 1,000 word families of the spoken subsection of the British National Corpus. While some have recommended that users should limit the amount of the test taken to only slightly above a student’s level, others argue that learners should take every level of the test. However, this raises concerns that correct responses on lower frequency levels could largely be attributed to guesses rather than vocabulary knowledge. In this paper we analyze a data set of 3,373 Japanese university students’ responses to the first eight levels of the original VST under the 3PL model, in order to determine the minimum expected score on the test for learners of low ability, examine the proportion of low-level students’ scores on the lowest frequency level tested that can be attributed to guessing under the 3PL model, and conduct a model fit comparison to determine whether the 3PL model offers a significantly better description of the data than the Rasch model. The results indicate that a substantial portion of lower level learners’ scores on items testing low-frequency words can be attributed to guessing and support the position that students should not sit every level of the test. The authors recommend using the results of the 3PL analysis in order to determine which sections of the test learners of different proficiency levels should sit.