The Phantom Hierarchy: How Scientifically Invalid AI IQ Scores Acquire Authority and Cause Real Harm
Authors: Franny Philos Sophia
Venue / Source: Zenodo (CERN European Organization for Nuclear Research) (2026)
Claiming an AI model has an IQ score is mathematically invalid because standardized tests offer zero measurement precision at high ability levels.
Key takeaways
- Item Response Theory shows intelligence tests provide negligible measurement information at extreme ability levels where AI is claimed to operate.
- Standard N-gram filtering fails to prevent structural contamination because models learn the deep logical isomorphism of puzzle families during training.
- The positive manifold assumption fails for LLMs, whose jagged performance profiles invalidate using single scores to infer general cognitive capacity.
- AI evaluation should shift from norm-referenced human IQ rankings to criterion-referenced Service Level Agreements with falsifiable objectives.
Executive Summary (TL;DR)
Authors' abstract (original)
Large language models are routinely reported as achieving "IQ 120" or "above-averagehuman intelligence," and these claims are widely received as evidence of progress towardartificial general intelligence. This paper argues that such claims are scientifically groundless onthree independent levels and that their circulation produces concrete harm.At the level of instrumentation, Item Response Theory demonstrates that standard IQtests provide negligible measurement information in the ability range where AI is claimed tooperate. At the level of ontology, cloud-based AI systems lack the temporal stability, individualboundedness, and trait-like constancy required for psychometric attribution. At the level ofinstitutional purpose, IQ tests were designed as clinical doorway diagnostics for supportallocation, not as competitive benchmarks; their appropriation by the AI industry inverts aprotective tool into a ranking device.Despite this triple failure, AI IQ scores command epistemic authority. This paperidentifies the responsible mechanisms: anthropomorphic interface design generates a PhantomSubject to whom traits can be attributed; the cultural prestige of intelligence amplifies the scores'significance; and a normative drift transforms the statistical median into a minimum standard forcompetence. Together, these mechanisms produce the Phantom Hierarchy: a socially operativebut scientifically unfounded ranking of minds.The paper documents harms already produced by this hierarchy, including the conversionof clinical thresholds into social ceilings, the creation of hidden cognitive entrance exams inAI-mediated services, and the Logic Trap—a failure mode in which AI optimization forhelpfulness amplifies despair. The paper concludes by proposing criterion-referenced ServiceLevel Agreements as an alternative evaluation framework.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly addresses validity arguments (Kane/Messick) regarding what test scores mean when generative AI takes or corrupts assessments.
- Applies Item Response Theory (IRT) information functions to highlight limitations in modeling machine ability versus human ability.
- Demonstrates how structural isomorphism undermines construct validity and item contamination procedures in AI evaluation.
Methodology and Dataset
Theoretical psychometric modeling under the two-parameter logistic (2PL) Item Response Theory model, conceptual ontological synthesis, and qualitative analysis of AI failure modes across four integrated studies examining systems such as OpenAI GPT-4 and ChatGPT.
Why Worth Reading
This paper provides a rigorous psychometric critique of LLM evaluation, showing how construct validity, process validity, and Item Response Theory assumptions collapse when standardized human tests are applied to generative AI. Assessment specialists and university researchers will gain a formal vocabulary to refute ungrounded AI capability claims and adopt proper criterion-referenced evaluation frameworks.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not mentioned in the text
- Unrepresentative sample: not mentioned in the text
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Monolingual: not mentioned in the text
- The study is primarily a theoretical and conceptual synthesis rather than an empirical psychometric benchmark across a large multi-model cohort.
- The proposed Service Level Objectives represent conceptual engineering guidelines rather than an open-source software implementation.
- supports The foundational distinction between trait and state dissolves for systems that sample output stochastically. «The distinction between "trait" and "state" (Borsboom, 2005), which is foundational to psychometric theory, dissolves for a system whose outputs are stochastic by design.», p. 11, para 13
- [metric / conceptual mismatch]: Applying human norm-referenced IQ tests to LLMs violates IRT information bounds and construct validity assumptions because LLM outputs reflect pattern retrieval rather than human fluid reasoning.
- [method / evaluation protocol gap]: Standard N-gram data decontamination filters only surface text matches, leaving benchmarks blind to deep logical isomorphism in model training data.