Detecting Item Preknowledge in Computerized Adaptive Testing Using Information Theory and Combinatorial Optimization

Abstract

Item preknowledge occurs when some examinees (called aberrant examinees) have had access to a subset of items (called a compromised subset) from an administered test prior to an exam. As a result, aberrant examinees might perform better on compromised items as compared to uncompromised items. When the number of aberrant examinees is large, the corresponding testing program and its users might be negatively affected because the aberrant examinees might be given invalid scores. There are numerous item preknowledge detection methods exploiting the difference in an examinee’s performance between compromised items and uncompromised items.  These methods are based on an incorrect assumption that the compromised subset is known and that it does not vary across subgroups of aberrant examinees. Computer simulations demonstrated that when this assumption is slightly violated the detection rate drops dramatically. This paper introduces a new algorithm—the 3D algorithm—merging information theory and combinatorial optimization for detecting subgroups of aberrant examinees and their corresponding compromised subsets of items.