The Impact of Item Bank Transition Rules on Student Ability Estimates and Achievement Level Classifications

Abstract

This paper describes a hybrid interim-summative computerized adaptive assessment design administered across three academic terms (fall, winter, and spring). Each test event had two phases that either stayed on-grade or moved off-grade dynamically. Using three different transition rule conditions, this design was compared to an on-grade hybrid interim-summative computerized adaptive assessment design using a series of simulations for a Grade 4 and Grade 6 mathematics assessment. A 500- and 800-item bank in each grade was simulated with a normal distribution of items that ranged across achievement levels. Simulees were randomly drawn from a normal distribution. Simulees whose ability estimates met the transition rule requirements were routed to off-grade item banks and test blueprints dynamically. During Phase 1, the algorithm was programmed to administer only on-grade items. At the end of Phase 1, a transition rule was used to determine if the student should be routed to off-grade items and blueprints for Phase 2. If the student ability estimate did not meet the transition rule requirements, the adaptive test continued to administer on-grade items. If the ability estimate met the transition rule requirements, the student was routed to item banks and blueprints at the adjacent lower or higher grade, as appropriate. The results were evaluated based on measurement precision (i.e., RMSE), measurement accuracy (i.e., bias), item exposure rates, and classification accuracy. Results indicated that while more students were routed off-grade when more lenient routing rules were used, most of the resulting evaluation indexes remained similar across all conditions, except for item exposure rates that varied across conditions. More lenient transition rules led to fewer overexposed items. More stringent transition rules maintained continuity with the on-grade achievement level designations. This preliminary evidence indicates that it might be feasible to transition students at the tails of the distributions to an off-grade bank and meet federal requirements. Such a transition integrates a fundamental aspect of interim assessments—going off-grade—with the summative assessment ESSA requirement that proficiency determinations be derived from on-grade items and constraints.