How Do Trait Change Patterns Affect the Performance of Adaptive Measurement of Change?

Abstract

Adaptive measurement of change (AMC) is a psychometric procedure to detect intra-individual change in trait levels across multiple testing occasions. However, in studying how AMC performs as a function of change, most previous studies did not specify change patterns systematically. Inspired by Cronbach and Gleser (1953), a quantitative framework was proposed that systematically decomposes a change pattern into three components: magnitude, scatter, and shape. Shape was further decomposed into direction and order. Using monte-carlo simulations, a series of ANOVAs were performed to investigate how each of these components affected the false positive rates (FPRs) and true positive rates (TPRs) for detecting true change, and a change recovery index (CRI). Results showed that FPRs were between 0.05 and 0.075 under all conditions. For TPRs, magnitude had the largest effect among all design factors. With an ideal item bank, TPRs reached 0.8 when magnitude was 1.0. Scatter and shape had some effects when the directions of change were mixed (non-monotone) across testing occasions. In addition, Time 1 true theta and its interactions had some effects under a practical item bank that had low test information at extreme theta values. CRIs were generally under 0.1 except at extreme Time 1 true thetas, indicating good change recovery. The results showed that when the magnitude of change is large, AMC has sufficient power to detect and recover individual change, regardless of the scatter and the shape of that change. When the magnitude of change is small, significance testing results should be interpreted cautiously due to the lack of power.