Leveraging Measurement Theory for Natural Language Processing Research
Qixiang Fang · Utrecht University Repository (Utrecht University) (2024) · doi:10.33540/473 · gemini-3.6-flash
TL;DR
Standard tests used to evaluate AI models often treat all questions as equally difficult and ignore measurement error. By applying established educational measurement techniques, this research shows how to properly test AI models' math and cognitive skills. The findings reveal that conventional scoring methods can misrank AI models when compared to human students.
Method and sample: The dissertation encompasses multiple empirical studies: Chapter 3 evaluates gender bias measure reliability across Word2Vec and GloVe embeddings; Chapter 4 assesses construct validity using a synthetic dataset of survey questions; Chapter 5 benchmarks LLM cognitive capabilities (including GPT-family and open models) on 8th-grade mathematics using standardized TIMSS datasets (2008 and 2011 cycles) evaluated via item response theory (IRT) latent variable models against human student populations.
#Item Response Theory#LLM Benchmarking#Construct Validity#Measurement Theory#Psychometrics#Text Embeddings