Journal of Computerized Adaptive Testing
JCAT is a peer-reviewed electronic journal designed to advance the science and practice of computerized adaptive testing (CAT). JCAT publishes two types of manuscripts: Empirical research reports, theoretical papers, and integrative critical reviews on topics directly related to CAT (e.g., item selection algorithms, security algorithms, multistage designs, examinee reactions to CAT, DIF in CAT, item bank development, the psychometrics of CAT) and on important ancillary topics (e.g., innovative item types, assessment engineering, psychometric models, issues surrounding the technology of adaptive testing, validity studies). Applications and implementations of CAT. These articles include descriptions of specific decisions made for a particular purpose, required by the nature of the adaptive test being developed, including (but not limited to) the nature of the testing population, the type of decisions being made with the information from the test, the size of the available item bank, the changing nature of item styles, approaches to field testing, and complex item selection procedures. JCAT is the official journal of the International Association for Computerized Adaptive Testing. To submit a manuscript, select the green "Information -- For Authors" link on the bottom right, and follow the instructions. To subscribe to JCAT, select the "Information -- For Readers" link on the bottom right. Subscriptions are free. To access articles published in the current year, select CURRENT on the top line of any page. To access articles from previous years, select ARCHIVES. Editor Dr. Duanli Yan, Director of Data Analysis and Computational Research , Educational Testing Service, U.S.A. Production Editor Matthew D, Finkelman, Tufts University Consulting Editors John Barnard, EPEC, Australia Kirk A. Becker, Pearson VUE, United States Matthew D. Finkelman, Tufts University School of Dental Medicine, United States Andreas Frey, Friedrich Schiller University Jena, Germany Kyung T. Han, Graduate Management Admission Council, United States G. Gage Kingsbury, Psychometric Consultant, United States Alan D Mead, Talent Algorithms Inc., United States Mark D Reckase, Michigan State University, United States Daniel O. Segall, PMC, United States Bernard P Veldkamp, University of Twente, Netherlands Wim van der Linden Alina von Davier, Duolingo Steven L. Wise, Northwest Evaluation Association, United States
Authors who publish in JCAT agree to the following terms: Authors retain copyright and grant the journal right of first publication. Articles may be copied and reproduced for academic and research purposes with no cost. All other reproduction requires permission of the author(s). If the authors cannot be contacted, permission can be requested from IACAT. Authors may enter into separate contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., publish it in a book), with an acknowledgement of its initial publication in JCAT.
Stochastic Curtailment: A New Approach to Improve Efficiency of Variable-Length Computerized Adaptive Tests
Stochastic curtailment (SC) is a statistical procedure that was originally developed to enhance the efficiency of clinical trials. It has been applied to psychological testing, but to sequential mastery testing only (Finkelman, 2008, 2010). This study adapted the method to detect low-precision examinees in computerized adaptive tests (CATs) (i.e., examinees whose final standard error of measurement (FSEM) at the end of a full-length CAT could not reach the pre-specified SEM termination level). Using central limit approximations, the study developed a method to estimate the distribution of test information at maximum test length and the corresponding FSEM. The study also developed a hypothesis testing procedure to implement SC. Using monte-carlo simulations, the study found that (1) the FSEM estimation procedure performed well in the middle range of theta values but less so at extreme theta values; (2) the SC procedure had good predictive accuracy, with excellent performance on positive predictive values and good performance on true positive rates and false positive rates; (3) the potential reduction in test length was substantial. Overall, the study showed that SC is a promising procedure to identify low-precision examinees and enhance efficiency in measurement CATs. A brief guide to implementing SC is provided.
Copyright (c) 2025 Ming Him Tai, Joseph N. DeWeese, David Weiss
Estimating the Joint Item-Score Density Using an Unrestricted Latent Class Model: Advancing Flexibility in Computerized Adaptive Testing
Copyright (c) 2025 Anastasios Psychogyiopoulos, Niels Smits, L. Andries van der Ark
A Score-Based Method for Detecting Item Compromise and Preknowledge in Computerized Adaptive Testing
In recent years, several methods have been proposed to detect compromised items (CI) and examinees with preknowledge (EWP) simultaneously. However, most of these methods are limited in one of two ways: (1) The method was specifically designed for non-adaptive tests and might therefore not be suitable for adaptive tests, or (2) the method involves the analysis of more than just the item scores. In this paper, a new method is developed for computerized adaptive tests that requires only an analysis of the item scores. The perfor-mance of the method is evaluated using simulations in which several factors are manip-ulated, such as test length, the percentage of CI, the percentage of EWP, and EWP ability distribution. Across most conditions, the method is shown to produce reasonable false positive rates and favorable true positive rates for both items and examinees.
Copyright (c) 2025 Kylie Gorney, Chansoon Lee, Jianshen Chen
A Literature Review of Through-Course Summative Assessment Models: The Case for an Adaptive Through-Year Assessment
This review describes various approaches and expected benefits of through-course summative assessment (TCSA) and many of the challenges associated with TCSA models, concluding with a case for why a through-year computerized adaptive test (TY-CAT) would solve many of the challenges. The central feature of TCSA models is that they combine scores from tests administered at different time points of the school year (U.S. Department of Education, 2010). The expected benefits of TCSAs are numerous, including finer-grained feedback due to an increase in the cumulative number of items (Preston & Moore, 2010); increased time to include and score performance tasks, which is expected to increase the content validity of summative scores (Bennett et al., 2011); increased curricular and assessment coherence (Wilson & Sloane, 2000); timely feedback (Wise, 2011); and potentially reduced measurement error (Wise, 2011). The question of whether a set of assessments could be administered throughout the school year and combined to replace a single end-of-year summative test used for accountability has been considered before. The Partnership for Assessment of Readiness for College and Career (PARCC) was considered a through-course summative design (Jerald et al., 2011). Although PARCC’s proposed design created much interest initially, it brought technical challenges, and the design was changed to a more traditional summative assessment. This literature review aims to evaluate different TCSAs in the literature to learn if alternative designs, especially CAT designs, might overcome some of the technical challenges. Three blueprint designs are discussed: distributed, cumulative, and repeated comprehensive. The advantages and limitations of each blueprint and associated score aggregation methods are considered, and both technical challenges and possible solutions are reviewed. The paper concludes by considering how an interim-summative hybrid CAT addresses many of the technical challenges of TCSAs.
Evaluating Effectiveness of Standard Error of Score Estimation as a CAT Termination Criterion
In computerized adaptive testing (CAT), the standard error of score estimation (SEE) value is sometimes used as a test termination criterion since test forms with the same level of SEE are believed to exhibit equivalent measurement precision even when they vary in the number of items. This study aimed to evaluate the appropriateness and effectiveness of using SEE as an approximation for actual standard error of measurement (SEM) under various conditions. First, this study illustrated a potential issue with the inconsistent relationship between the SEE and actual SEM at different lengths in equivalent tests. A series of simulation studies were then conducted to investigate the relationship among SEE, SEM, and test lengths in a variable-length CAT administration. For this purpose, various combinations of score estimation methods and item selection methods were considered both with and without item exposure control. The study found that using the SEE as a sole criterion to terminate a CAT administration might result in a less than ideal level of control in measurement precision when the test was either extremely short or administered in a variable-length CAT. Also, the choice of the score estimation methods and item selection methods with or without item exposure control affected the relationship between SEE and actual SEM.
Copyright (c) 2024 Chansoon Lee, Kyung (Chris) T. Han
The Impact of Item Pool Size and Item Pool Distribution on Student Ability Estimates for a Hybrid Interim-Summative CAT
This paper investigates the impact of a uniform versus a bell-shaped distribution of items in a computer adaptive pool within and across administrations for a hybrid interim-summative assessment. Item pool sizes of 500, 800, and 1500 were simulated for both distributions. One-hundred simulations were conducted for two grades (Grade 4 and Grade 6) in mathematics. The item pools were generated under a Rasch model for dichotomous items and a partial credit model for three-category items. Each item pool was simulated to be vertically scaled and vertically articulated across grades. The items in the pools were generated to align to the blueprints for a state test, and the targeted distribution of items across the four performance levels based on the notion engineered score interpretations were implemented. For the two item pools under the normal distribution, the difficulties for Grades 4 and 6 were drawn from a normal distribution with means of -0.40 and 0.40 and standard deviation of 1.1. For the two item pools under the uniform distribution, the difficulties were drawn from a uniform distribution with differing minimums and maximums for each grade: -3.6 to -2.8 for the minimums and 2.4 to 3.2 for the maximums. The outcome variables investigated were measurement precision (i.e., root mean square error (RMSE)), measurement accuracy (i.e. bias), item pool adaptivity, classification accuracy (true positives, true negatives, false positives, and false negatives), and item exposure rate. Either item distribution largely worked well with slightly improved results for larger banks. Item pool sizes of 800 did not perform materially differently than pool sizes of 1500. In general, while all three administrations had robust findings with the outcome variables, the measurement quality degraded slightly in the 500 item pool. Implications and trade-offs in item pool composition are discussed from a measurement and financial perspective.
Copyright (c) 2025 Garron Gianopulos, Jonghwan Lee, Sangdon Lim, Luping Niu, Sooyong Lee, Seung W. Choi
The Impact of Item Bank Transition Rules on Student Ability Estimates and Achievement Level Classifications
This paper describes a hybrid interim-summative computerized adaptive assessment design administered across three academic terms (fall, winter, and spring). Each test event had two phases that either stayed on-grade or moved off-grade dynamically. Using three different transition rule conditions, this design was compared to an on-grade hybrid interim-summative computerized adaptive assessment design using a series of simulations for a Grade 4 and Grade 6 mathematics assessment. A 500- and 800-item bank in each grade was simulated with a normal distribution of items that ranged across achievement levels. Simulees were randomly drawn from a normal distribution. Simulees whose ability estimates met the transition rule requirements were routed to off-grade item banks and test blueprints dynamically. During Phase 1, the algorithm was programmed to administer only on-grade items. At the end of Phase 1, a transition rule was used to determine if the student should be routed to off-grade items and blueprints for Phase 2. If the student ability estimate did not meet the transition rule requirements, the adaptive test continued to administer on-grade items. If the ability estimate met the transition rule requirements, the student was routed to item banks and blueprints at the adjacent lower or higher grade, as appropriate. The results were evaluated based on measurement precision (i.e., RMSE), measurement accuracy (i.e., bias), item exposure rates, and classification accuracy. Results indicated that while more students were routed off-grade when more lenient routing rules were used, most of the resulting evaluation indexes remained similar across all conditions, except for item exposure rates that varied across conditions. More lenient transition rules led to fewer overexposed items. More stringent transition rules maintained continuity with the on-grade achievement level designations. This preliminary evidence indicates that it might be feasible to transition students at the tails of the distributions to an off-grade bank and meet federal requirements. Such a transition integrates a fundamental aspect of interim assessments—going off-grade—with the summative assessment ESSA requirement that proficiency determinations be derived from on-grade items and constraints.
Copyright (c) 2025 Jonghwan Lee, Sangdon Lim, M. Christina Schneider, Garron Gianopulos, Luping Niu, Sooyong Lee, Seung W. Choi
Design Considerations and Reporting Solutions for a Multiple Administrations Adaptive Testing System
In this paper we overview the policy, test development, psychometric, and reporting deliberations that stakeholders engaging in the design of multiple administration adaptive test, otherwise known as a through-year assessment, will have to consider. We use the development of a design prototype for simulations coupled with newly designed score reports to serve as exemplars of the work to be done by states and vendors to a develop a through-year assessment with the intended purpose of merging two different assessment systems into a singular system that supports teachers in better understanding where students are in their learning and states in meeting accountability requirements.
Copyright (c) 2025 M Christina Schneider, Seung W. Choi, Daniel Lewis
The Influence of Computerized Adaptive Testing (CAT) on Psychometric Theory and Practice
This article presents a review of some of the issues that were prominent in the early days of large-scale assessment and presents an argument that the development of computerized adaptive testing (CAT) was influential in changing the way that these issues were addressed. It is argued that work on CAT increased the rate of adoption of item response theory (IRT) as the basis for scaling and reporting for large-scale assessments. It also shifted the emphasis of test analysis from the number-correct score to characteristics of test items and statistics related to those item characteristics. Further, the emphasis on interval scaling has declined because specifying the form of an item characteristic curve subsumes the assumption of an interval scale. These changes are the result of a paradigm shift in educational and psychological testing that resulted from the shift from classical test theory approaches to IRT-based approaches.
How Do Trait Change Patterns Affect the Performance of Adaptive Measurement of Change?
Adaptive measurement of change (AMC) is a psychometric procedure to detect intra-individual change in trait levels across multiple testing occasions. However, in studying how AMC performs as a function of change, most previous studies did not specify change patterns systematically. Inspired by Cronbach and Gleser (1953), a quantitative framework was proposed that systematically decomposes a change pattern into three components: magnitude, scatter, and shape. Shape was further decomposed into direction and order. Using monte-carlo simulations, a series of ANOVAs were performed to investigate how each of these components affected the false positive rates (FPRs) and true positive rates (TPRs) for detecting true change, and a change recovery index (CRI). Results showed that FPRs were between 0.05 and 0.075 under all conditions. For TPRs, magnitude had the largest effect among all design factors. With an ideal item bank, TPRs reached 0.8 when magnitude was 1.0. Scatter and shape had some effects when the directions of change were mixed (non-monotone) across testing occasions. In addition, Time 1 true theta and its interactions had some effects under a practical item bank that had low test information at extreme theta values. CRIs were generally under 0.1 except at extreme Time 1 true thetas, indicating good change recovery. The results showed that when the magnitude of change is large, AMC has sufficient power to detect and recover individual change, regardless of the scatter and the shape of that change. When the magnitude of change is small, significance testing results should be interpreted cautiously due to the lack of power.
Copyright (c) 2023 Ming Him Tai, Allison W. Cooperman, Joseph N. DeWeese, David J. Weiss
Expanding the Meaning of Adaptive Testing to Enhance Validity
Abstract The computerized adaptive test (CAT), which adjusts the difficulty levels of administered items to match the ability levels of the test takers, has a long history of providing efficient testing. After 50 years of research, however, CATs have evolved little beyond the initial idea of adapting item difficulty. In this paper I suggest that we begin viewing adaptation in a more expanded manner, arguing that measurement will be further improved by a CAT’s being able to detect and adapt to the presence of construct-irrelevant factors that threaten the validity of individual test scores. Several examples of innovative adaptations currently provided by an operational CAT program are discussed.
An Extended Taxonomy of Variants of Computerized Adaptive Testing
This paper builds on foundational work on probabilistic frames of reference and principled assessment design to explore the role of adaptation in assessment. Assessments are characterized in terms of their claim status, observation status, and locus of control. The relevant claims and observations constitute a frame of discernment for the assessment. Adaptation occurs when the frame is permitted to evolve with respect to the claims or observations (or both); adaptive features may be controlled by the examiner or the examinee. In describing the various combinations of these characteristics, it is argued that an online format is preeminent for supporting common and emerging assessment practices in light of adaptation.
Copyright (c) 2023 Robert Mislevy, Roy Levy, John T. Behrens
The basic idea of adaptive testing is quite simple and has been implemented for over a century. Dichotomously scored multiple-choice questions are commonly used to obtain response vectors in computerized adaptive tests (CATs). A response is scored as either correct or as incorrect. However, a correct response does not necessarily mean that the examinee knew the answer. Although the standard error increasingly decreases as the provisional ability is estimated, the question is whether the process can be improved at the item response level. In other words, can more information be extracted from a response than a simple 0 or 1? Implementation of option probability theory holds promise to address this question.
dichotomously scored items, option probability theory, scoring methods, subjective proba-bility
The (non)Impact of Misfitting Items in Computerized Adaptive Testing
To assess the potential impact of misfitting items, simulated examinees received varying percentages of misfitting items. The fit was manipulated to be poor near what would otherwise be the point of maximum information. With 30% misfitting items, ability estimates tended to have more positive bias in the middle ability range, and more negative bias in the high and low ability ranges, than they did with 0% or 10% misfitting items. However, the magnitude of this effect was small. For most abilities and test lengths, the empirical standard error did not vary with the percentage of misfitting items. The standard error estimated from the information function tended to underestimate the empirical standard error for the shortest test length and overestimate it for the longer test lengths, regardless of the percentage of misfitting items. Overall, the misfit had little practical impact.
Three Measures of Test Adaptation Based on Optimal Test Information
This study extends the work of Reckase, Zu, and Kim (2019) by introducing three new measures of test adaptation. These measures are based on the amount of information that the test provides to individual examinees and are designed to provide theoretical, engineering, and operational views of the amount of adaptation that a particular test provides. The three measures are applied to simulations of adaptive, multistage, and fixed-form tests to describe how the measures could be used in an operational setting to identify difficulties with adaptation and to suggest processes for improving adaptivity.
Time-Efficient Adaptive Measurement of Change
The adaptive measurement of change (AMC) refers to the use of computerized adaptive testing (CAT) at multiple occasions to efficiently assess a respondent’s improvement, decline, or sameness from occasion to occasion. Whereas previous AMC research focused on administering the most informative item to a respondent at each stage of testing, the current research proposes the use of Fisher information per time unit as an item selection procedure for AMC. The latter procedure incorporates not only the amount of information provided by a given item but also the expected amount of time required to complete it. In a simulation study, the use of Fisher information per time unit item selection resulted in a lower false positive rate in the majority of conditions studied, and a higher true positive rate in all conditions studied, compared to item selection via Fisher information without accounting for the expected time taken. Future directions of research are suggested.
How Adaptive Is an Adaptive Test: Are All Adaptive Tests Adaptive?
The primary distinguishing feature of computerized adaptive testing (CAT) is that sets of items that are administered to examinees are specifically selected for each examinee during the test process. This implies that examinees who respond differently to items on a test will get different sets of test items. However, if an item bank is small, if there are strict content constraints, or if there is strong exposure control, there might not be much adaptation during the test. Examinees who perform differently might get many of the same test items. The research reported here recommends some statistical indicators of how much adaptation is taking place and shows how these indicators vary for different kinds of adaptive test designs. Guidelines are provided for the values of the statistics indicating that a CAT is strongly adaptive.
Implementing Three CATs Within Eighteen Months
The development of a computerized adaptive test is considered a labor-intensive and time-consuming endeavor. This paper illustrates that this does not have to be the case—by demonstrating the steps taken, the decisions made, and the empirical results obtained during the development of three computerized adaptive tests (CATs) designed to measure student competencies in reading, mathematics, and science. The three tests had to be developed and piloted within an 18-month period, and they were used directly afterward in six research projects of a large nationwide research initiative. To ensure the sound psychometric quality of the CATs developed, the item calibration (N = 1,632) followed several quality control procedures, including item fit analysis, differential item functioning analysis, and preoperational simulation studies. A CAT pilot study (N = 1,093) and an additional CAT simulation confirmed the general usefulness of the constructed instru-ments. It is concluded that the development of CATs—including item calibration, simulations, and piloting within 18 months—is quite possible, even for comparably small development teams. This necessitates an available theoretical framework for the assessment and a sufficient number of items, specific plans for the item calibration, simulations, and a pilot study, as well as an information technology infrastructure for administering the tests.
Adaptive Item Selection Under Matroid Constraints
The shadow testing approach (STA; van der Linden & Reese, 1998) is considered the state of the art in constrained item selection for computerized adaptive tests. The present paper shows that certain types of constraints (e.g., bounds on categorical item attributes) induce a matroid on the item bank. This observation is used to devise item selection algorithms that are based on matroid optimization and lead to optimal tests, as the STA does. In particular, a single matroid constraint can be treated optimally by an efficient greedy algorithm that selects the most informative item preserving the integrity of the constraints. A simulation study shows that for applicable constraints, the optimal algorithms realize a decrease in standard error (SE) corresponding to a reduction in test length of up to 10% compared to the maximum priority index (Cheng & Chang, 2009) and up to 30% compared to Kingsbury and Zara's (1991) constrained computerized adaptive testing.
Factors Affecting the Classification Accuracy and Average Length of a Variable-Length Cognitive Diagnostic Computerized Test
The aim of cognitive diagnosis models (CDMs) is to provide students and educators with individually tailored diagnostic results for students’ mastery levels of a group of fine-grained skills, or attributes. The field of variable-length cognitive diagnostic computerized adaptive testing (CD-CAT) aims to deliver diagnostic assessments that accurately classify students using the fewest number of items possible. A crucial element of a CD-CAT is the Q-matrix, a 1-0 matrix mapping the skills required by each item. This paper describes a simulation study that systematically explored factors affecting the accuracy and average length of a variable-length CD-CAT, including composition of the Q-matrix, correlation among skills, item selection rule, and CDM. It was found that higher density Q-matrices (i.e., Q-matrices in which individual items tap many skills) yield longer and less accurate tests than lower density Q-matrices (i.e., Q-matrices in which individual items tap fewer skills). The two item selection rules examined—mutual information and modified posterior Kullback-Leibler information—performed very similarly. Higher correlation among skills tended to increase average test lengths and decrease accuracy noticeably when the Q-matrices were high density.
Latent-Class-Based Item Selection for Computerized Adaptive Progress Tests
Standard computerized adaptive testing (CAT) methods require an underlying item response theory (IRT) model. An item bank can be constructed from the IRT model, and subsequent items can be selected with maximum information at the examinee’s estimated ability level. IRT models, however, do not always fit test data exactly. In such situations, it is not possible to employ standard CAT methods without violating assumptions. To extend the scope of adaptive testing, this research shows how latent class analysis (LCA) can be used in item bank construction. In addition, the research investigates suitable item selection algorithms using Kullback-Leibler (KL) information for item banks based on LCA. The KL information values can be used to select items and to construct an adaptive test. Simulations show that item selection based on KL information outperformed random selection of items in progress testing. The effectiveness of the selection algorithm is evaluated, and a possible scoring for the new adaptive item selection with two classes is proposed. The applicability of the methods is illustrated by constructing a computerized adaptive progress test (CAPT) on an example data set drawn from the Dutch Medical Progress Test.
Uniform Test Assembly: Concepts, Problems, Solvers, and Applications for Adaptive Testing
This paper presents the latest developments since the publication of the seminal book by van der Linden (2005) on general types of test assembly (TA) problems, major automated test assembly (ATA) methods, and various practical situations in which a TA problem arises. With the power of modern combinatorial optimization (CO) methods, multiple practical tasks in test development and design that were previously intractable can now be solved. The TA problem is, therefore, no longer a central issue for test development but rather a subproblem embedded in different practical tasks, where two major approaches are currently exploited: mixed-integer programming (MIP) and uniform test assembly (UTA). In the world of ATA, the MIP approach is dominant. However, UTA has multiple advantages over MIP. This paper concentrates on UTA and enumerates its multiple applications for adaptive testing.
Effect of Imprecise Parameter Estimates on Ability Estimates in a Multistage Test in an Automatic Item Generation Context
In adaptive testing, the availability of large banks of accurately calibrated items is critical for estimating examinees’ ability and for effectively routing them through the test. The development and calibration of the item banks is thus a necessary step before adaptive testing can be implemented. Recent advances in automated item generation, in particular item cloning, might improve upon the traditional item bank development process. One approach to automated item cloning is to construct an item bank comprised of a small number of calibrated item parents. When the test is delivered, the items are then generated on the fly from the parent item that has the desired measurement parameters. The cloned item parameters, however, are inherited with some amount of error. This simulation study evaluated the accuracy with which examinees’ abilities can be estimated when items cloned from a parent are used in multistage tests (MSTs) and the psychometric properties of the clones are assumed to be the same as those of the parent item. The behavior of the clones’ item statistics in this study was modeled based on the results of Sinharay and Johnson’s (2008) investigation into item clones that were administered in an experimental section of the Graduate Record Examination (GRE). The results of the current study indicated that the MST is relatively robust to considerable deviations between the clone’s item statistics used for routing and scoring and the properties, or difficulty, of the clone as seen by the examinee. Keywords: automatic item generation, multistage testing, adaptive testing, item clones, errors in item parameters
This paper gives an overview of an assessment program with overseas trained medical graduates who wish to practice medicine in Australia. A non-technical discussion of how the program was migrated from a paper-and-pencil format through online administration to computerized adaptive testing (CAT) over approximately 15 years includes various practical aspects encountered and how they were dealt with. Although theoretical CAT aspects are briefly mentioned throughout, the focus is on some guidelines and experiences that might be useful to practitioners.
Cognitive Diagnostic Models and Computerized Adaptive Testing: Two New Item-Selection Methods That Incorporate Response Times
A recent paper proposed an item-selection approach for computerized adaptive testing (CAT) in which the psychometric information per time unit is maximized. The current research extended this methodology to adaptive tests combined with use of a cognitive diagnostic model (CDM). Two new item-selection methods are introduced for the combination of CDMs and CAT: posterior-weighted Kullback-Leibler information per-time-unit, and mutual information per-time-unit. These methods were compared with standard procedures in which the amount of time required to complete an item is not considered. Simulation conditions with and without attribute-balancing constraints indicated that, on average, the new methods required more items but took less time than the standard procedures, while achieving comparable classification accuracy.
Detecting Item Preknowledge in Computerized Adaptive Testing Using Information Theory and Combinatorial Optimization
Item preknowledge occurs when some examinees (called aberrant examinees) have had access to a subset of items (called a compromised subset) from an administered test prior to an exam. As a result, aberrant examinees might perform better on compromised items as compared to uncompromised items. When the number of aberrant examinees is large, the corresponding testing program and its users might be negatively affected because the aberrant examinees might be given invalid scores. There are numerous item preknowledge detection methods exploiting the difference in an examinee’s performance between compromised items and uncompromised items. These methods are based on an incorrect assumption that the compromised subset is known and that it does not vary across subgroups of aberrant examinees. Computer simulations demonstrated that when this assumption is slightly violated the detection rate drops dramatically. This paper introduces a new algorithm—the 3D algorithm—merging information theory and combinatorial optimization for detecting subgroups of aberrant examinees and their corresponding compromised subsets of items.
A Comparison of Multi-Stage and Linear Test Designs for Medium-Size Licensure and Certification Examinations
Many large-scale testing organizations currently use multi-stage testing (MST) for their examinations. The MST design implements the test in several stages, where one module is administered per stage. Successive stages in MST might vary in difficulty depending on the estimated ability level of the examinee. Although several studies have been conducted to compare the performance of MST to the traditional linear test design, all of the investigations known to date have incorporated simulation studies that capitalize on large sample size requirements in order to reduce the number of replicated datasets. As a result, although the statistics under investigation have been estimated with reasonable stability, these studies have been better suited to investigate the performance of MST for large-scale examinations as opposed to small- or medium-size examinations. The purpose of this research was to conduct a series of studies based on simulated datasets for medium-size medical certification examinations. The results confirmed that more accurate ability estimates and more accurate and consistent pass-fail decisions are obtained under the MST design compared to the traditional linear design for these examinations.
The Utility of Adaptive Testing in Addressing the Problem of Unmotivated Examinees
In low-stakes testing programs, test developers have the dual responsibilities of developing and administering a high-quality test that can yield valid scores, and motivating examinees to put forth their best effort to perform well on that test. However, unmotivated examinees present a major threat to validity of scores from these types of testing programs. This integrative review examines the motivational benefits of computerized adaptive tests (CATs), and demonstrates that they can have important advantages over conventional tests in both identifying instances when examinees are exhibiting low effort, and effectively addressing the validity threat posed by unmotivated examinees.
A Comparison of Four Methods for Obtaining Information Functions for Scores From Computerized Adaptive Tests With Normally Distributed Item Difficulties and Discriminations
A simulation study compared four methods of estimating information for maximum-likelihood estimates (MLE) from fixed-length computerized adaptive tests (CAT): those by (1) Lord (1980), (2) Segall, Moreno, and Hetter (1997), (3) Ito, Pommerich, and Segall (2009a), and (4) Ito and Segall (2010), referred to, respectively, as “local,” “quasi-local,” “global-slope,” and “conditional-averaging-on-q” (CA-theta ). A 900-item bank was constructed using operational three-parameter logistic item parameter estimates [i.e., a (discrimination), b (difficulty), and c (pseudo-guessing)] such that the a and b parameters were as normally distributed as possible. Test length and the number of simulees at each theta point were varied. Generally, information functions for the quasi-local and global-slope methods were very close to those for the reference local method. However, those for the CA-theta method were distinctly different, which might be explained conceptually. These findings were considerably different than those from previous studies that had used a skewed item bank, and illustrated how bank characteristics, including distributions of item difficulties, discriminations, and ability-bank (mis)matches, could influence which methods would yield similar score information functions.