Prompt Sensitivity in LLM-Based Essay Scoring is Model-Specific
Authors: Giulio Corsi, Alexandru Marcoci, Maryam Abo-Tabik, Georgiana Thorpe Apreutesei, Lyba Razzaq, Roni Tibon, Yael Benn, Deborah Talmi
Venue / Source: CLOK (University of Central Lancashire) (2026)
Prompt sensitivity in automated essay grading is model-specific, meaning universal prompt guidelines do not exist and can actively degrade performance.
Key takeaways
- No single prompt component consistently optimizes automated essay grading across frontier language models.
- Poor prompt configurations can perform worse than predicting the cohort mean score, destroying evaluative signal.
- The optimal prompt configuration changes across institutions, reflecting interactions with local marking conventions.
- Prompt engineering for automated grading should be treated as an empirical calibration procedure rather than intuitive instruction design.
Executive Summary (TL;DR)
Authors' abstract (original)
The emergence of Large Language Models (LLMs) has renewed interest in automated essay scoring, yet a barrier to adoption persists: LLM-generated scores are sensitive to prompt formulation, and it is unclear which aspects of a prompt drive this effect or whether findings from one model or institution transfer to another. For educators and institutions considering deployment, this uncertainty is consequential, as without understanding the structure of prompt sensitivity, there is no principled basis for deciding whether to trust an LLM-generated score. This study evaluates three frontier LLMs (GPT-5.4, Claude Opus 4.6, Gemini 3 Flash) across a 3×3×3 factorial design varying criteria specificity, calibration intervention, and scoring strategy on a calibration sample of 153 undergraduate essays from three UK universities with distinct marking conventions. A main-effects ANOVA on the 27 per-model conditions reveals that prompt sensitivity is substantial but strikingly model-specific: criteria specificity accounts for the largest share of condition-level variance for Gemini (57\%, p < .001), scoring strategy for GPT (35\%, p < .01), and scoring strategy and calibration together for Claude (32\% and 24\% respectively), with no single dimension dominating across models. The optimal configuration also differs across institutions, even within the same model. At each institution, the gap between best and worst configurations spans 4–5 RMSE points on the calibration set, with the worst performing worse than a naïve baseline. For the models tested, these findings indicate that no universal prompt recipe exists, and that responsible deployment requires empirical prompt calibration as a prerequisite rather than an optional refinement - a process more similar to hyperparameter tuning than to simple instruction design.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly informs the validity argument (scoring and generalization inferences) of deploying LLMs as automated essay raters in higher education.
- Shows empirical rater drift and measurement distortion introduced purely by prompt structure across different assessment environments.
- Demonstrates that institutional grading norms interact with LLM evaluation behaviors, affecting score comparability.
Methodology and Dataset
Calibration subset of N = 153 undergraduate psychology essays (from a larger corpus of 764 essays across 50 modules) from three UK universities using standard UK percentage marking scales. Three frontier LLMs were evaluated: GPT-5.4, Claude Opus 4.6, and Gemini 3 Flash. A fully crossed 3 x 3 x 3 factorial design tested 27 prompt configurations per model (81 total conditions) crossing criteria specificity (none, summary, full analytic), calibration intervention (none, distributional, debiased), and scoring strategy (direct, analytic-decomposed, deliberative). Performance was benchmarked against a naive cohort-mean baseline using RMSE and band accuracy.
Why Worth Reading
It provides a rigorous factorial decomposition refuting the notion of universal 'best-practice' prompt engineering for automated essay grading, demonstrating that prompt optimization operates like hyperparameter tuning tied to specific models and institutional marking rubrics.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- No adversarial testing: not mentioned in the text
-
Monolingual:
found
“The data is drawn from a corpus of 764 long-form undergraduate psychology essays from three UK universities, spanning 50 modules.” (p. 2)
- Evaluations are strictly in-sample on the calibration set (N = 153), meaning reported best-worst gaps reflect an optimistic upper bound.
- The calibration sample contains approximately 50 essays per institution, yielding wide bootstrap confidence intervals for variance components.
- Analysis is limited to undergraduate psychology essays marked under UK university conventions and three specific model versions at one point in time.
- supports LLM-generated essay scores are sensitive to prompt formulation, with minor variations shifting outputs substantially. «LLM-generated scores are sensitive to prompt formulation, with minor variations in rubric framing or role instructions capable of shifting outputs by a full grade band», p. 1, para 18
- supports LLMs exhibit a tendency to compress assigned scores toward the middle of the rating scale. «addresses the well-documented tendency of LLMs to compress scores towards the centre of the scale [2], which can in principle be mitigated by providing distributional anchoring.», p. 2, para 20
- [method / methodological]: Sequential prompt additions used in earlier studies conflate model architecture and prompt dimension interactions, which this factorial design resolves.
- [sample / empirical]: Cross-institutional transferability was previously assumed, but institutional marking scale quirks create distinct prompt interactions.