Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items
Authors: Peng Xie, Amr Alanwar
Venue / Source: arXiv (2026)
Batching student answers in one prompt to save cost causes models to change grades depending purely on neighboring submissions.
Key takeaways
- Grading student short answers in batches of 25 changes 11% to 22% of individual verdicts compared to grading them alone.
- Aggregate accuracy metrics like AUROC completely mask batch-induced grade instability because individual score shifts cancel each other out.
- The sequence of items, assigned prompt slots, and batch co-membership all drive grade instability through both text and generated-verdict channels.
- Voting over permutations of a batch fails to resolve the problem because co-membership effects persistently survive majority voting.
- Educators should enforce isolated single-item grading using prompt-caching infrastructure to prevent arbitrary cross-item interference without inflating inference costs.
Executive Summary (TL;DR)
Authors' abstract (original)
To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly speaks to automated grading validity under Kane's argument-based framework: construct-irrelevant variance is introduced by prompt peer-context.
- Provides a formal mathematical framework (variance decomposition) for modeling rater drift and non-separability in LLM-as-a-rater setups.
- Demonstrates engineering countermeasures (block attention masks, prompt prefix caching) to achieve valid automated scoring without cost inflation.
Methodology and Dataset
Sample size N = 2,000 items per task across three tasks: short-answer grading (SciEntsBank corpus, student responses evaluated against reference answers), social media harm detection (Measuring Hate Speech), and Yelp review ratings. Primary evaluations used five open-weight models: Qwen3-4B-Instruct-2507, Qwen3-8B (thinking disabled), Gemma-4-E4B-it, Granite-3.3-8B-Instruct, and GLM-4-9B-0414. Extended evaluations tested four larger models (Qwen3-14B, Qwen3-32B, Gemma-4-31B-it, GLM-4-32B-0414). Baselines included single-item grading, renamed keys, reworded instructions, and single-item prompt generation.
Why Worth Reading
This paper rigorously dismantles the common industry and psychometric practice of in-context batching for automated essay/short-answer scoring. It provides a formal variance decomposition showing Kane-validity degradation: an individual student's score becomes a function of peer submissions rather than construct competence alone.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Primary evaluations were conducted solely on open-weight models between 4B and 32B parameters rather than proprietary frontier models.
- Evaluations were restricted to English-language corpora with structured JSON prompt formats and greedy decoding.
- The Gaussian threshold model assumed uniform context variance across all items, slightly underestimating flips across large numbers of batches.
- supports Batch prompting saves substantial token costs with minimal aggregate accuracy loss in social science annotation. «they save more than 80% of token costs and lose less than 2 points of accuracy», p. 1, para 8
- supports Batched screening decisions shift substantially between single-item processing and batching without aggregate metric changes. «Hida et al. (2026) report that 1–27% of screening decisions change between single-item processing and batches of 20, and that aggregate metrics do not always reflect these changes.», p. 2, para 15
- [metric / conceptual]: Standard psychometric and NLP benchmark evaluations rely on aggregate ROC/accuracy, completely concealing individual-level measurement error caused by item co-occurrence in prompts.
- [method / empirical]: Permutation voting schemes (such as BatchPrompt) assume position independence can be achieved by reordering, ignoring irreducible latent membership variance.