How Do Trait Change Patterns Affect the Performance of Adaptive Measurement of Change?
Adaptive measurement of change (AMC) is a psychometric procedure to detect intra-individual change in trait levels across multiple testing occasions. However, in studying how AMC performs as a function of change, most previous studies did not specify change patterns systematically. Inspired by Cronbach and Gleser (1953), a quantitative framework was proposed that systematically decomposes a change pattern into three components: magnitude, scatter, and shape. Shape was further decomposed into direction and order. Using monte-carlo simulations, a series of ANOVAs were performed to investigate how each of these components affected the false positive rates (FPRs) and true positive rates (TPRs) for detecting true change, and a change recovery index (CRI). Results showed that FPRs were between 0.05 and 0.075 under all conditions. For TPRs, magnitude had the largest effect among all design factors. With an ideal item bank, TPRs reached 0.8 when magnitude was 1.0. Scatter and shape had some effects when the directions of change were mixed (non-monotone) across testing occasions. In addition, Time 1 true theta and its interactions had some effects under a practical item bank that had low test information at extreme theta values. CRIs were generally under 0.1 except at extreme Time 1 true thetas, indicating good change recovery. The results showed that when the magnitude of change is large, AMC has sufficient power to detect and recover individual change, regardless of the scatter and the shape of that change. When the magnitude of change is small, significance testing results should be interpreted cautiously due to the lack of power.
Authors who publish in JCAT agree to the following terms:
- Authors retain copyright and grant the journal right of first publication.
- Articles may be copied and reproduced for academic and research purposes with no cost.
- All other reproduction requires permission of the author(s).
- If the authors cannot be contacted, permission can be requested from IACAT.
- Authors may enter into separate contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., publish it in a book), with an acknowledgement of its initial publication in JCAT.