Effect of Imprecise Parameter Estimates on Ability Estimates in a Multistage Test in an Automatic Item Generation Context

Abstract

In adaptive testing, the availability of large banks of accurately calibrated items is critical for estimating examinees’ ability and for effectively routing them through the test. The development and calibration of the item banks is thus a necessary step before adaptive testing can be implemented. Recent advances in automated item generation, in particular item cloning, might improve upon the traditional item bank development process. One approach to automated item cloning is to construct an item bank comprised of a small number of calibrated item parents. When the test is delivered, the items are then generated on the fly from the parent item that has the desired measurement parameters. The cloned item parameters, however, are inherited with some amount of error. This simulation study evaluated the accuracy with which examinees’ abilities can be estimated when items cloned from a parent are used in multistage tests (MSTs) and the psychometric properties of the clones are assumed to be the same as those of the parent item. The behavior of the clones’ item statistics in this study was modeled based on the results of Sinharay and Johnson’s (2008) investigation into item clones that were administered in an experimental section of the Graduate Record Examination (GRE). The results of the current study indicated that the MST is relatively robust to considerable deviations between the clone’s item statistics used for routing and scoring and the properties, or difficulty, of the clone as seen by the examinee.

Keywords: automatic item generation, multistage testing, adaptive testing, item clones, errors in item parameters