From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment
Authors: Ivo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer, Ulrich Trautwein, Dorottya Demszky, Heather Hill, Enkelejda Kasneci
Venue / Source: arXiv (2026)
Statistical SHAP attributions provide more faithful sentence-level score explanations than large language model rationales across classroom evaluation tasks.
Key takeaways
- Fine-tuned pretrained language models achieve higher accuracy on rubric scoring than prompted large language models, though fine-tuned models suffer from score compression near the scale mean.
- Feature attribution via sentence-level SHAP consistently identifies influential text that alters model predictions when removed, demonstrating higher explanation faithfulness than LLM-generated evidence selections.
- Sentence explanations generated by SHAP transfer effectively across different model architectures, whereas LLM rationales exert weak and non-monotonic impact when transferred to other scoring models.
- Prompted large language models show variability in following output constraints and exhibit low sentence-level overlap with SHAP feature attributions when selecting evidence from transcripts.
Executive Summary (TL;DR)
Authors' abstract (original)
Automated scoring models are increasingly used to assign rubric-based quality ratings to complex language performances, including classroom transcripts, yet they typically provide little insight into why a particular score is produced. We propose a general framework for sentence-level interpretability of rubric-based scoring that combines model-agnostic Shapley-value attributions with rationales generated by large language models (LLMs). Instantiated on the Quality of Feedback dimension of the CLASS framework using the NCTE corpus, the framework enables systematic comparison of fine-tuned pretrained language models (PLMs) and prompted LLMs on both scoring performance and explanation faithfulness. Across 6k annotated transcript segments, fine-tuned PLMs outperform LLMs in prediction accuracy but exhibit label compression toward mid-scale scores. Deletion-based tests show that SHAP identifies sentences that reliably drive model predictions, producing typically larger and more coherent prediction shifts than LLM-generated rationales. Cross-model analyses further reveal that SHAP attributions transfer robustly across architectures, whereas LLM rationales exert limited and inconsistent influence. Overall, the findings demonstrate that SHAP provides more faithful and transferable explanations for rubric-based scoring, and that the proposed framework offers a principled basis for evaluating both scoring models and their explanations in high-stakes educational settings and other rubric-based language assessment tasks.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Evaluates the validity of using LLMs and PLMs as raters on complex, multi-level educational rubrics.
- Provides a quantitative method (deletion-based perturbation testing) for auditing explanation faithfulness in automated scoring systems.
- Highlights systemic label compression issues in supervised educational scoring models when trained on skewed rubric data.
Methodology and Dataset
The study analyzed 6,005 15-minute classroom transcript segments from 1,600 elementary mathematics lessons in the NCTE corpus (4,775 train, 1,230 test), annotated on the 1–7 CLASS Quality of Feedback dimension. Fine-tuned encoder models (BERT, ALBERT, RoBERTa, DeBERTaV3 base/large) were compared against 4-bit quantized open-source instruction-tuned LLMs (Llama 3.1 8B/70B, Mixtral 8x7B/8x22B, Qwen 3 4B/30B/235B, Mistral Small/24B). Baselines included constant mean/median score baselines and length-matched random sentence deletion.
Why Worth Reading
This paper provides a rigorous method for testing whether AI explanations for rubric-based scoring are genuinely faithful or merely plausible-sounding hallucinations. By demonstrating that SHAP feature attributions outperform LLM rationales in causal deletion tests and cross-architecture transfer, it offers clear guidance for building valid, transparent automated assessment and teacher feedback tools.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Monolingual: not mentioned in the text
- Dataset contains severe label imbalance with 81% of ratings falling in the mid-range (3–5), causing fine-tuned models to collapse toward the mean and miss extreme rubric categories.
- Evaluations are restricted to a single text-only CLASS dimension (Quality of Feedback) based on single-rater expert annotations without inter-rater reliability metrics or multimodal cues.
- Sentence deletion perturbs natural classroom discourse flow and pragmatic structure, which may disproportionately affect LLM context processing.
- No human expert evaluations were conducted to verify whether SHAP-selected or LLM-selected sentences align with construct-relevant pedagogical rubrics.
- supports SHAP feature attribution enables local and global interpretability in automated rubric-based scoring models. «Boulanger and Kumar (2020) apply SHAP to automated essay scoring to quantify the contribution of linguistic features to rubric-level predictions, enabling both local and global interpretability of model behavior.», p. 3, para 3
- [explanation_faithfulness / methodological gap]: While LLM generated free-form text rationales appear persuasive to human readers, deletion-based perturbation shows they lack causal faithfulness compared to sentence-embedding SHAP attributions.
- [scoring_accuracy_vs_dispersion / conceptual gap]: Fine-tuned language models optimize error metrics (MAE/MSE) at the cost of severe score compression (failing to predict extreme rubric values), whereas prompted LLMs preserve score variance at the cost of higher error.