Abstract
This study proposes to apply the Theil-index ratios for the interrater reliability. We discuss the theoretical foundations and examine its function empirically using real data. Our analyses show that Theil-index rations and intraclass correlation (ICC) estimates are highly correlated. However, ICC may underestimate the interrater reliability by some extreme disagreement among raters and be more likely to be influenced by the extreme disagreement. As Theil-index ratios overcome the limitations of ICC to some degree, it seems that Theil-index ratios provide an alternative to evaluating interrater reliability, at least under certain conditions, e.g., when outliers exist in the data, it is difficult to obtain the variance-component estimates, or ICC underestimates interrater reliability.
Recommended Citation
Pan, Tianshu and Yin, Yue
(2023)
"An Application of Theil Indexes for the Interrater Reliability: A Comparison with Intraclass Correlations,"
Chinese/English Journal of Educational Measurement and Evaluation | 教育测量与评估双语期刊: Vol. 4:
Iss.
2, Article 1.
DOI: https://doi.org/10.59863/WDDK7257
Available at:
https://www.ce-jeme.org/journal/vol4/iss2/1
DOI
https://doi.org/10.59863/WDDK7257