Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items
Peng Xie, Amr Alanwar · arXiv (2026) · doi:10.48550/arxiv.2610.40241 · gemini-3.8-flash
TL;DR
Grading multiple student answers together in one prompt to save money causes the AI's grade to depend heavily on which other answers are included and their order. Standard overall test accuracy conceals this problem because errors cancel out in the aggregate. Even taking majority votes over multiple shuffled batches fails to eliminate these arbitrary grade shifts.
Method and sample: Sample size N = 2,000 items per task across three tasks: short-answer grading (SciEntsBank corpus, student responses evaluated against reference answers), social media harm detection (Measuring Hate Speech), and Yelp review ratings. Primary evaluations used five open-weight models: Qwen3-4B-Instruct-2507, Qwen3-8B (thinking disabled), Gemma-4-E4B-it, Granite-3.3-8B-Instruct, and GLM-4-9B-0414. Extended evaluations tested four larger models (Qwen3-14B, Qwen3-32B, Gemma-4-31B-it, GLM-4-32B-0414). Baselines included single-item grading, renamed keys, reworded instructions, and single-item prompt generation.
#LLMs as raters#automated grading#construct-irrelevant variance#batch prompting#measurement validity#rater drift#short-answer scoring