Leveraging Measurement Theory for Natural Language Processing Research
Authors: Qixiang Fang
Venue / Source: Utrecht University Repository (Utrecht University) (2024)
Applying Item Response Theory to LLM benchmarking alters model capability rankings compared to raw percentage-correct scores.
Key takeaways
- Raw accuracy scores on AI benchmarks obscure item difficulty and guessing probability, making standard LLM rankings psychometrically unreliable.
- Item Response Theory enables direct, fair comparisons between large language models and human student populations on standardized test items.
- Transformer embeddings demonstrate substantially higher construct validity for representing complex social science constructs than static word embeddings.
- Word embedding gender bias measures exhibit high test-retest reliability but poor inter-rater consistency across different scoring rules.
Executive Summary (TL;DR)
Authors' abstract (original)
This dissertation explores the intersection of natural language processing (NLP) and measurement theory. NLP is a field aimed at enabling machines to process and generate human languages, such as English, German, and Mandarin. These languages are complex, diverse, and full of irregularities, making them challenging for machines to handle compared to structured artificial languages, like programming languages. NLP research ranges from simple tasks, like word frequency analysis, to complex ones involving the understanding and generation of human language. Measurement theory, traditionally applied in social sciences, addresses how we measure various properties scientifically. Key concepts include construct validity, which examines whether a measure accurately represents what it intends to measure, and reliability, which focuses on the consistency of a measure across different conditions. This dissertation argues that many challenges in NLP relate to measurement issues and suggests that principles from measurement theory can help address these challenges, particularly by providing tools to evaluate and improve the quality of NLP models. The structure of the dissertation is as follows: Chapter 2 offers background on NLP and measurement theory, covering essential text representation techniques in NLP, the history of measurement theory, and recent discussions on its unification across fields. Chapters 3-5 apply measurement theory to evaluate NLP models: Chapter 3 explores the reliability of gender bias measures in NLP by using classical reliability estimators from social sciences. Chapter 4 adapts a construct validity testing framework to assess the quality of text representations for social science constructs. Chapter 5 introduces a psychometric-based benchmarking approach to evaluate large language models, demonstrated through a case study on eighth-grade math proficiency. Chapters 6-7 focus on using measurement theory to improve NLP model performance: Chapter 6 presents a framework for designing user models based on measurement principles, achieving better-quality user representations than current methods. Chapter 7 examines how integrating human values into model training can enhance models’ ability to recognize values in human arguments. Chapters 8 and 9 reflect on current NLP research challenges and propose future directions: Chapter 8 identifies challenges in text-based personality computing, offering potential solutions and avenues for research. Chapter 9 concludes with a summary of the dissertation’s findings and suggests future work at the intersection of measurement theory and NLP. This work underscores the potential of measurement theory to enhance NLP research by offering frameworks for evaluating and designing more reliable and valid models. By integrating these approaches, the dissertation aims to bridge NLP and measurement theory, advancing NLP's capability to address complex measurement challenges.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Demonstrates how to apply Item Response Theory (IRT) to evaluate LLMs as test takers.
- Analyzes construct validity and measurement error when using LLMs and embeddings for educational and psychometric measurement.
Methodology and Dataset
The dissertation encompasses multiple empirical studies: Chapter 3 evaluates gender bias measure reliability across Word2Vec and GloVe embeddings; Chapter 4 assesses construct validity using a synthetic dataset of survey questions; Chapter 5 benchmarks LLM cognitive capabilities (including GPT-family and open models) on 8th-grade mathematics using standardized TIMSS datasets (2008 and 2011 cycles) evaluated via item response theory (IRT) latent variable models against human student populations.
Why Worth Reading
This thesis bridges psychometrics and NLP, offering educators and evaluation researchers a mathematically grounded framework to replace uncalibrated raw test scores with Item Response Theory when assessing LLM capabilities.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Conventional human baseline evaluations in existing AI benchmarks rely on small convenience samples of graduate students or benchmark authors rather than representative populations.
- Word embedding gender bias metrics remain highly sensitive to the choice of embedding algorithms despite showing high internal consistency.
- [metric / methodological gap]: Standard AI benchmarks rely on aggregate percentage accuracy, ignoring item difficulty and pseudo-guessing parameters modeled by Item Response Theory.