ORCID
Yiqin Pan: https://orcid.org/0009-0002-2045-4117
Sandip Sinharay: https://orcid.org/0000-0003-4491-8510
Abstract
Item compromise is a significant concern in the testing industry. We propose a novel approach for detecting compromised items in computerized linear testing by analyzing response times and scores. Our method consists of three primary components. First, we employ autoencoders, a type of neural network, to model response times and scores, capturing the unique characteristics of each item and examinee. Second, we develop a BERT-based model to analyze examinees’ response patterns, predict expected response times. The characteristic vectors generated by the autoencoders are integrated into the BERT-based model, enhancing its ability to recognize response patterns. Third, an algorithm is designed to flag compromised items using the predicted expected response times and observed response data. Simulation studies demonstrate that the proposed method effectively detects compromised items, controlling the false positive rate at a low level, as long as the extent of preknowledge is not extremely small, maintaining the power at a relatively high level.
Recommended Citation
Pan, Yiqin and Sinharay, Sandip
(2024)
"Detecting Compromised Items in Computerized Linear Testing: A Novel Approach Using Autoencoders and BERT,"
Chinese/English Journal of Educational Measurement and Evaluation | 教育测量与评估双语期刊: Vol. 5:
Iss.
3, Article 7.
DOI: https://doi.org/10.59863/XSSZ8498
Available at:
https://www.ce-jeme.org/journal/vol5/iss3/7
DOI
https://doi.org/10.59863/XSSZ8498