Estimating the impact of local item dependency in a test of second language reading comprehension

Abstract

Local item dependency (LID) occurs when test-takers’ responses to one test item are affected by their responses to another. It can be problematic if it causes inflated reliability estimates or distorted person and item measures. The cued-recall reading comprehension test in Hu and Nation’s (2000) well-known and influential coverage–comprehension study may have been susceptible to LID because of its very high sampling rate of idea units from the test’s reading passage. To explore this possibility, we examined Yen’s Q3 statistics in a dataset comprising responses from 63 learners who took Hu and Nation’s test. Using Q3 values of ≥ .30 to flag possible instances of LID, 14 item pairs were identified as being locally dependent. To remove LID, items with high Q3 values were combined into polytomous super-items. Using Rasch measurement, a comparison of the data with and without the effects of LID indicated that LID only modestly inflated reliability estimates and that there was a negligible effect on the ordering of persons by ability. Thus, while LID was present, it had only a small impact on how test scores might be interpreted in a study like Hu and Nation’s.