quasi_experiment

The Role of Implicit and Explicit Demographic Signals in Large Language Model-based Student Assessment

Authors: Donya Rooein, Luca Benedetto, Dirk Hovy
Venue / Source: arXiv (2026)

Large Language Models consistently adapt their feedback and scoring behavior based on implicit and explicit student demographics, raising significant fairness concerns in automated assessment.

Key takeaways

Language models exhibit systematic bias when processing student demographics, frequently altering feedback sentiment and readability based on both explicit profiles and subtle conversational history.
The study employed a controlled experimental design across six instruction-tuned models to evaluate performance variations in automated essay scoring, formative feedback, and question answering tasks.
Implicit demographic cues from conversation histories tend to trigger more significant and unpredictable behavioral shifts in model performance than direct persona prompting.
The research is limited by a demographic dataset skewed toward majority groups and a lack of linguistic diversity, which may constrain the generalizability of the findings.
The findings suggest that achieving fairness in automated assessment requires addressing latent stylistic biases rather than relying solely on the anonymization of user profiles.

Executive Summary (TL;DR)

In plain words In the paper's terms

Researchers tested whether six common AI models change how they grade or provide feedback based on a student's personal information like age, gender, or socioeconomic status. They found that AI models often adjust their responses based on these details, sometimes in ways that favor certain groups or lead to inconsistent treatment of students. This means that AI-based grading might unintentionally treat students differently depending on their personal background, rather than focusing purely on their work.

This study evaluates how explicit and implicit demographic signals influence LLM performance in automated essay scoring, formative feedback, and metalinguistic question answering. Using a controlled experimental framework with 200 real user profiles, the authors find that LLMs demonstrate systematic demographic sensitivity, particularly in feedback readability, sentiment, and response length. Results indicate that while explicit persona conditioning leads to predictable behavioral shifts, implicit cues from conversational history create larger, more unpredictable biases.

Authors' abstract (original)

Large Language Models are now common in student assessment, but we know little about how student demographics affect their use. Sometimes, considering student demographics may be necessary -- for example, to improve readability for users with lower educational levels. However, it also risks being a cause of discrimination, e.g., when assigning lower scores to students from lower socioeconomic backgrounds. We set up controlled prompts to test 1) explicit demographic effects, where we mention demographic details directly, and 2) implicit effects, where we use conversation history as a demographic signal. We test these settings in three tasks: Automated Essay Scoring, Formative Feedback, and Metalinguistic Question Answering. We test six state-of-the-art LLMs on these tasks. In both explicit and implicit cases, the models pick up on demographic cues and can change their scoring, feedback, and answers accordingly. We find that LLMs frequently adjust the readability of feedback to education levels when these are explicitly mentioned. On the other hand, implicit conditions produce unpredictable biases, such as in question answering, where responses from lower-education levels receive lower sentiment scores. Our results provide clear evidence of demographic sensitivity in LLMs for educational assessment tasks.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

5.0/10

Consensus Challenge

5.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a methodology for testing demographic bias using controlled counterfactuals in LLM assessment.
Demonstrates the difference between explicit and implicit demographic triggers, highly relevant to AI-resistant assessment design.
Highlights the task-specific nature of AI biases, cautioning against universal 'AI-fairness' claims.

Methodology and Dataset

The study evaluates six instruction-tuned LLMs (GPT-mini, GPT-nano, Llama 3.3 70B, Llama 3.2 3B, Qwen3 4B, and Qwen3 30B) across three tasks: Automated Essay Scoring (AES), Formative Feedback (FF), and Metalinguistic Question Answering (QA). The dataset includes 192,480 inference calls generated from 200 user profiles (subsampled from the AI Gap dataset, N=200) and 40 items per task (Persuade 2.0 for AES/FF; ELQA for QA).

Why Worth Reading

This paper provides a rigorous, controlled assessment of how latent demographic signals in LLM interaction histories—not just explicit profiles—can skew educational outcomes. It demonstrates that fairness in AI-based assessment is not just a matter of 'anonymizing' data, but a complex challenge of unintended stylistic and sentiment-based bias.

Key Claims & Verified Quotes

LLMs exhibit systematic demographic sensitivity in educational assessment tasks.

Quote: “Our results provide clear evidence of demographic sensitivity in LLMs for educational assessment tasks” — 10.48550/arxiv.2609.16993, p. 1, para. 7 (exact)

Implicit demographic conditioning has stronger, less predictable effects on LLM response length than explicit conditioning.

Quote: “Notably, our findings were quite different when measuring differences in the response length (number of words) of the LLM responses: indeed, we find more significant results for Imp than for Exp” — 10.48550/arxiv.2609.16993, p. 7, para. 6 (exact)

Limitations & Study Design Flags

The study uses a limited sample of 120 educational items and 200 user profiles, which may impact generalizability.
Most demographic attributes in the source dataset are heavily skewed toward majority groups (e.g., 96% English L1), limiting analysis of certain group contrasts.
Implicit cues are derived from prompt histories unrelated to the educational domain, potentially introducing noise.
Analysis is restricted to English only, preventing evaluation of demographic sensitivity in languages with grammatical gender markers.
Bonferroni p-value corrections, while ensuring robustness, are noted as potentially too conservative.

From the article text (quote verified)

supports Human evaluators are often influenced by demographic factors, which undermines the ideal of score invariance. «In practice, though, this invariance does not hold, as extensive evidence shows that human evaluations are often influenced by demographic factors.», p. 1, para 12

Model inference (gaps & contradictions)

[Method / Data drift]: The paper acknowledges that the prompt history dataset is not educationally representative, which may lead to biased evaluations compared to real student-tutor interactions.

#automated-essay-scoring #algorithmic-bias #fairness #LLM-sensitivity #assessment-validity #AI-in-education