The criterion reference project and the complexity analysis project both provide useful concepts and analyses for the Examinations Institute. Nonetheless, in order to translate the basic ideas into a practical tool, an additional project must be undertaken. This project is to do alignment studies. In an alignment study, items from exams are analyzed in terms of how they connect (align) to variable of interest – in this case the content map from the criterion referencing project and the complexity rating.
We are undertaking this project in collaboration with experts in psychometrics from the Center for Educational Assessment at the University of Massachusetts. The initial goal is to align current ACS Exams to the grid established by the criterion and complexity projects. Once this has been done, we will be able to see where there are “holes” in the way exams test student understanding, and thereby improve test design by making sure that coverage is more complete.