assessment_design

Automated Generation of Complexity-Validated Decision Scenarios Using Large Language Models

Authors: Abdalla Doleh, Toni Somers, Ratna Babu Chinnam
Venue / Source: arXiv (2026)

Automated scenario generation with LLMs can achieve psychometric rigor comparable to manual expert design, but quality and reliability vary significantly between models.

Key takeaways

An automated, multi-framework scoring system can successfully classify generated test stimuli into complexity tiers with high psychometric reliability.
Different LLMs demonstrate distinct performance profiles; high-throughput models may trade off structural quality for speed.
The instrument shows a strong correlation between scenario complexity and text length, limiting pure construct separation.
Automated stimulus generation requires rigorous validation of both model-specific failure patterns and domain coverage.

Executive Summary (TL;DR)

In plain words In the paper's terms

Researchers created a system that uses AI to automatically write test scenarios for measuring decision-making skills. The study confirmed that this AI system creates tests of consistent difficulty and quality across different models. It also found that while some AI models are much faster at writing these tests, they do not always produce better or more accurate results, emphasizing that researchers must choose and test their AI tools carefully.

This study introduces an automated pipeline using large language models to generate and complexity-validate decision scenarios for cognitive assessment. The system achieved near-perfect inter-LLM agreement (ICC = 0.997) and large tier-based separation (η² = 0.587) across 4,238 generated scenarios. The results reveal significant model-specific differences in generation quality, throughput, and failure patterns, underscoring the need for empirical validation in automated stimulus creation.

Authors' abstract (original)

Cognitive decision-making research depends on diverse scenarios with carefully controlled complexity, yet manual production is slow, inconsistent, and biased. We developed an automated pipeline that uses LLms to generate structured decision scenarios and validates their complexity through a composite framework rooted in established task-complexity theory. We evaluated 4,238 scenarios across multiple domains and complexity tiers. Measurement validation met rigorous psychometric standards. Agreement among five independent model families was nearly perfect, with an intraclass correlation coefficient of 0.997 and a kappa of 0.971. Known-groups validity demonstrated large separation between tiers, with an eta-squared of 0.587 and all pairwise comparisons significant at p less than .001. Factor analysis revealed a dominant complexity construct, with loadings between 0.87 and 0.96 across three frameworks, while interactivity formed a weaker secondary dimension at 0.34. Discriminant validity was limited by a strong relationship between complexity and text length that persisted after controlling for tier, yielding a partial correlation of 0.86. This constrains construct purity but does not undermine the instrument's tier-grading function. Model analyses showed a negative association between throughput and schema pass rate (r = -0.967, p = .007, n = 5), suggesting a speed-quality trade-off, though largely driven by one high-throughput model. Llama 4 Maverick generated scenarios fastest at 134 per minute versus 25 for DeepSeek Chat V3.2, but underproduced complex-tier scenarios, whereas DeepSeek Chat V3.2 balanced domain coverage with high schema compliance. The system demonstrated strong psychometric properties, enabling reliable classification into Simple, Moderate, and Complex tiers and providing the measurement infrastructure needed for downstream cognitive assessment of AI systems

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

7.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a validated methodology for quantifying the 'difficulty' or 'complexity' of AI-generated assessment content.
Offers a replicable framework for auditing LLM-based assessment generators for bias and failure modes.
Supports the development of AI-resilient assessments by ensuring stimulus difficulty is controlled and psychometrically sound.

Methodology and Dataset

The study generated 4,238 valid scenarios across four domains using five LLMs (Grok-4, Llama 4 Maverick, DeepSeek Chat V3, GPT-4o, GPT-5.2) at temperature 0. Validation involved convergent, discriminant, known-groups, and nomological analyses alongside inter-rater agreement checks (N=220 subset) across independent model families.

Why Worth Reading

It provides a rare, rigorous quantitative evaluation of LLMs as generators of standardized test stimuli, offering a blueprint for creating AI-resilient assessment infrastructure using psychometric principles.

Key Claims & Verified Quotes

No verified quotes available for this paper.

Limitations & Study Design Flags

Monolingual: not mentioned in the text

The instrument exhibits a persistent correlation between complexity and text length (partial r=0.86).
The speed-quality trade-off observation is based on a small aggregate panel (n=5).
The study lacks concurrent validation against human expert difficulty ratings or performance outcomes.
The assessment-compatible design of the prompt prevents strictly assessment-blind measurement.

Model inference (gaps & contradictions)

[method / Measurement Bias]: The paper admits to using 'assessment-compatible' scaffolding, which likely biases the complexity scores toward the intended assessment dimensions, potentially inflating the reported convergent validity.
[method / Generalizability]: The use of temperature T=0 for generation and scoring provides high deterministic reliability but masks how these assessment instruments might drift or distort under higher-variability 'creative' settings.

#assessment_design #psychometric_modeling #automated_test_assembly #LLM_evaluation #task_complexity