Implementing Three CATs Within Eighteen Months

Authors

  • Christian Spoden Department of Research Methods in Education, Friedrich Schiller University Jena, Am Planetarium 4, 07743 Jena, Germany.
  • Andreas Frey Department of Research Methods in Education, Friedrich Schiller University Jena, Am Planetarium 4, D-07743 Jena, Germany; Centre for Educational Measurement (CEMO) University of Oslo, Norway.
  • Raphael Bernhardt Department of Psychology, Friedrich Schiller University Jena, Am Steiger 3, 07743 Jena.

Abstract

The development of a computerized adaptive test is considered a labor-intensive and time-consuming endeavor. This paper illustrates that this does not have to be the case—by demonstrating the steps taken, the decisions made, and the empirical results obtained during the development of three computerized adaptive tests (CATs) designed to measure student competencies in reading, mathematics, and science. The three tests had to be developed and piloted within an 18-month period, and they were used directly afterward in six research projects of a large nationwide research initiative. To ensure the sound psychometric quality of the CATs developed, the item calibration (N = 1,632) followed several quality control procedures, including item fit analysis, differential item functioning analysis, and preoperational simulation studies. A CAT pilot study (N = 1,093) and an additional CAT simulation confirmed the general usefulness of the constructed instru-ments. It is concluded that the development of CATs—including item calibration, simulations, and piloting within 18 months—is quite possible, even for comparably small development teams. This necessitates an available theoretical framework for the assessment and a sufficient number of items, specific plans for the item calibration, simulations, and a pilot study, as well as an information technology infrastructure for administering the tests.

 

Downloads

Published

2018-09-28

Issue

Vol. 6 No. 3 (2018): Implementing Three CATs Within Eighteen Months

Section

Articles

License

Authors who publish in JCAT agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication.
  2. Articles may be copied and reproduced for academic and research purposes with no cost.
  3. All other reproduction requires permission of the author(s).
  4. If the authors cannot be contacted, permission can be requested from IACAT.
  5. Authors may enter into separate contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., publish it in a book), with an acknowledgement of its initial publication in JCAT.