Factors Affecting the Classification Accuracy and Average Length of a Variable-Length Cognitive Diagnostic Computerized Test

Abstract

The aim of cognitive diagnosis models (CDMs) is to provide students and educators with individually tailored diagnostic results for students’ mastery levels of a group of fine-grained skills, or attributes. The field of variable-length cognitive diagnostic computerized adaptive testing (CD-CAT) aims to deliver diagnostic assessments that accurately classify students using the fewest number of items possible. A crucial element of a CD-CAT is the Q-matrix, a 1-0 matrix mapping the skills required by each item. This paper describes a simulation study that systematically explored factors affecting the accuracy and average length of a variable-length CD-CAT, including composition of the Q-matrix, correlation among skills, item selection rule, and CDM. It was found that higher density Q-matrices (i.e., Q-matrices in which individual items tap many skills) yield longer and less accurate tests than lower density Q-matrices (i.e., Q-matrices in which individual items tap fewer skills). The two item selection rules examined—mutual information and modified posterior Kullback-Leibler information—performed very similarly. Higher correlation among skills tended to increase average test lengths and decrease accuracy noticeably when the Q-matrices were high density.