A Bayesian many-facet Rasch model with Markov modeling for rater severity drift
Authors: Masaki Uto
Venue / Source: Behavior Research Methods (2022)
Accounting for how grader standards shift over time using Markov chains significantly improves the accuracy and reliability of performance assessment scores.
Key takeaways
- Incorporating Markov chains into Rasch models improves the accuracy of parameter estimation by accounting for temporal dependencies in rater severity.
- The model effectively quantifies individual rater drift through specific standard deviation parameters, offering clearer insights into evaluator consistency over time.
- Robust estimation remains achievable under sparse data conditions, suggesting utility for practical assessment designs with limited rater overlap.
- Computational requirements for MCMC estimation are notably higher than traditional methods, and performance depends on careful hyperparameter selection.
Executive Summary (TL;DR)
Authors' abstract (original)
Fair performance assessment requires consideration of the effects of rater severity on scoring. The many-facet Rasch model (MFRM), an item response theory model that incorporates rater severity parameters, has been widely used for this purpose. Although a typical MFRM assumes that rater severity does not change during the rating process, in actuality rater severity is known to change over time, a phenomenon called rater severity drift. To investigate this drift, several extensions of the MFRM have been proposed that incorporate time-specific rater severity parameters. However, these previous models estimate the severity parameters under the assumption of temporal independence. This introduces inefficiency into the parameter estimation because severities between adjacent time points tend to have temporal dependency in practice. To resolve this problem, we propose a Bayesian extension of the MFRM that incorporates time dependency for the rater severity parameters, based on a Markov modeling approach. The proposed model can improve the estimation accuracy of the time-specific rater severity parameters, resulting in improved estimation accuracy for the other rater parameters and for model fitting. We demonstrate the effectiveness of the proposed model through simulation experiments and application to actual data.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a quantitative method for detecting and modeling rater drift in educational assessment.
- Offers a Bayesian psychometric framework suitable for correcting distortion in assessment data.
- Contributes to the validity argument by providing mechanisms to isolate rater bias from true student ability.
Methodology and Dataset
The study evaluates the proposed model through simulation experiments (varying examinee N=100-500, raters R=5-10, and time points T=3-5) and an empirical application. The empirical dataset comprises 134 Japanese university students completing an essay task evaluated by 15 raters (10 standard raters and 5 intentionally biased 'control' raters over four rating days). The model is implemented in Stan using NUTS-based HMC.
Why Worth Reading
This paper provides a rigorous psychometric defense against 'rater drift'—a common source of assessment invalidity that is amplified when human evaluators are stressed or inconsistent. By providing an open-source Bayesian framework to model time-dependent grading behavior, it offers assessment designers a tool to maintain score integrity in high-stakes environments.
Key Claims & Verified Quotes
Limitations & Study Design Flags
-
Small sample:
found
“We recruited 134 Japanese university students” (p. 18)
-
Monolingual:
found
“We recruited 134 Japanese university students as participants.” (p. 18)
- The simulation and empirical experiments focus on single-item tests, lacking evaluation for complex multi-item assessments.
- The model is restricted to first-order Markov chains, which may not capture longer-term trends in rater behavior.
- Parameter estimation requires high computational cost due to MCMC compared to traditional maximum likelihood methods.
- The model performance relies on hyperparameter tuning (mu_sigma) which requires empirical validation via information criteria.
- supports Previous extensions of MFRMs for rater drift assume temporal independence of severity parameters. «However, these previous models estimate the severity parameters under the assumption of temporal independence.», p. 1, para 5
- supports Rater severity tends to show temporal dependency in practice. «In practice, however, severities between adjacent time points are known to have temporal dependency.», p. 3, para 2
- [method / theoretical]: The current model uses a first-order Markov assumption; however, the author acknowledges it lacks capability for long-term dependency which may exist in real-world professional rater training.