Optimizing Test Completion in Time-Constrained Adaptive Tests: A Post-Hoc Simulation Study

Abstract

In operational computerized adaptive tests (CATs), time limits are often imposed to manage logistical constraints, maintain test security, or preserve classroom instructional time. However, time constraints in CATs can lead to inaccurate ability estimates as some students may struggle to complete enough items within the allotted time due to differences in working speed and time intensities of individual items. This study proposes a simple dynamic approach to incorporate item response time (RT) data into the item selection algorithm, with the goal of optimizing test completion in a time-constrained setting. Linear optimization techniques were employed to balance information and RT when selecting subsequent items when needed. A post-hoc simulation study was conducted using real data from a 120-item reading assessment taken by 27,652 students. The results indicate that the optimized algorithm improved ability estimation and test completion rates across the ability spectrum, with the greatest impact on low-ability students. The optimal performance was achieved with a 0.6/0.4 information-time weighting combination, showing the most significant benefits under the shortest time constraints. Practical implications for implementations of time-constrained CATs are discussed.