Identifying Items on Which Humans and Chatbots Diverge Using Differential Item Functioning
Authors: Licol Zeinfeld, Alona Strugatski, Ziva Bar-Dov, Ron Blonder, Shelley Rap, Giora Alexandron
Venue / Source: Zenodo (CERN European Organization for Nuclear Research) (2026)
Applying Differential Item Functioning with Item-Total Correlation filtering reliably identifies test items where LLMs and human students diverge.
Key takeaways
- Logistic Regression DIF combined with Item-Total Correlation filtering reliably isolates items where chatbots and humans perform differently at equal ability levels.
- Mantel-Haenszel DIF creates high false-positive rates when comparing humans and LLMs, whereas LR-DIF demonstrates strong stability under placebo testing.
- LLMs outperform ability-matched students on rule-based canonical concepts by avoiding common student misconceptions, but underperform on items requiring visual diagrams or multi-step reasoning.
- Assessment specialists can use human-LLM DIF analytics to pinpoint task dimensions vulnerable to AI assistance and adapt exam designs accordingly.
Executive Summary (TL;DR)
Authors' abstract (original)
The rapid adoption of large language models (LLMs) in education raises profound challenges for assessment design. To adapt assessments to the presence of LLM-based tools, it is crucial to characterize the strengths and weaknesses of LLMs in a generalizable, valid and reliable manner. However, current LLM evaluations often rely on descriptive statistics derived from benchmarks, and little research applies theory-grounded measurement methods to characterize LLM capabilities relative to human learners in ways that directly support assessment design. Here, by combining educational data mining and psychometric theory, we introduce a statistically principled approach for identifying items on which humans and LLMs show systematic response differences, pinpointing where assessments may be most vulnerable to AI misuse, and which task dimensions make problems particularly easy or difficult for generative AI. The method is based on Differential Item Functioning (DIF) analysis – traditionally used to detect bias across demographic groups – together with negative control analysis and item-total correlation discrimination analysis. It is evaluated on responses from human learners and six leading chatbots (ChatGPT-4o \& 5.2, Gemini 1.5 \& 3 Pro, Claude 3.5 \& 4.5 Sonnet) to two instruments: a high school chemistry diagnostic test and a university entrance exam. Subject-matter experts then analyzed DIF-flagged items to characterize task dimensions associated with chatbot over- or under-performance. Results show that DIF-informed analytics may provide a theory-grounded framework for understanding where LLM and human capabilities diverge, and highlight their value for improving the design of valid, reliable, and fair assessment in the AI era.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Extremely relevant for research on educational assessment validity under generative AI.
- Provides a concrete psychometric methodology (LR-DIF + ITC filtering) to model item-level distortion and human vs LLM performance gaps.
- Offers empirical evidence on task dimensions that make items AI-resistant versus AI-vulnerable.
Methodology and Dataset
The study evaluated responses from human examinees across two instruments: a 22-item high school chemistry diagnostic test (N=931 students) and a 40-item quantitative section of a university psychometric entrance exam (N>4,800 students). Chatbot data comprised 120 response sets generated across 20 independent runs per model for six LLMs: OpenAI GPT-4o and GPT-5.2, Google Gemini 1.5 Pro and Gemini 3 Pro, and Anthropic Claude 3.5 Sonnet and Claude 4.5. Methods evaluated included Mantel-Haenszel DIF (MH-DIF) and Logistic Regression DIF (LR-DIF), validated using a 50-run placebo negative control test (replacing chatbots with random human sub-samples) and diagnosed via Item-Total Correlation (ITC). Qualitative expert reviews characterized DIF-flagged chemistry items.
Why Worth Reading
This paper brings mature psychometric theory into the debate on AI validity in education. Rather than relying on simple benchmark accuracy rates, it introduces a principled method (LR-DIF with ITC filtering) to control for overall examinee ability and isolate item-level features that make test items AI-vulnerable or AI-resistant. It offers assessment developers an actionable, statistically validated framework to audit and redesign multiple-choice exams for unproctored or AI-pervasive environments.
Key Claims & Verified Quotes
Limitations & Study Design Flags
-
Unrepresentative sample:
found
“The main limitation to external validity is that the results are based on a small number of instruments and specific GenAI tools.” (p. 7)
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Monolingual: not mentioned in the text
- Bimodal chatbot ability distributions between older and newer model generations reduced effective data size during ability stratification.
- Item evaluation used binary scoring (correct/incorrect), which cannot capture partial credit or intermediate reasoning steps.
- Expert qualitative analysis of DIF task features was conducted on the chemistry test but not yet replicated on the quantitative entrance exam.
- External validity is limited by testing only two assessment instruments and six LLM configurations.
- contradicts Chatbots circumvented student misconceptions on conceptual chemistry items, contradicting prior findings of alignment between LLMs and student error patterns. «the fact that the chatbots dodged these designated 'traps' is interesting, and is in disagreement with previous work that found moderate alignment between LLMs and student misconceptions [12].», p. 7, para 2
- supports Chatbot underperformance on visual and multi-step items aligns with previous findings regarding GenAI limitations in STEM problem solving. «These findings reinforce previous work about GenAI problem solving in STEM domains [24, 26].», p. 7, para 2
- [method / psychometric assumption]: Standard DIF assumes unidimensionality of the construct across groups, but humans and LLMs use fundamentally different cognitive and algorithmic mechanisms, which may induce artificial DIF due to multidimensionality rather than item bias.