Towards Scalable Measurement of Durable Skills
Authors: Amir Globerson, Amy Keeling, Anisha Choudhury, Anna Iurchenko, Aviad Segal, Avinatan Hassidim, Ayça Çakmakli, Ben Gomes, Benn Witt, Cathy Cheunga, Cristine Legare, Diana Akrong, Eliad Carmi, Elisabeth Bauer, Gal Elidan, Hadas Gelbart, Hairong Mu, Katherine Chou, Lev Borovoi, Nir Kerem, Niv Efron, Noa Kerrem Gilo, Preeti Singh, Rajvi Kapadia, Rena Levitt, Roni Rabin, Ronit Levavi Morad, Rotem Yulzary, Shashank Agarwal, Sophie Allweis, Tracey Lee-Joe, Tzvika Stein, Yael Bar Moshe, Yael Haramaty, Yaniv Carmel, Yishay Mor, Yoav Bar Sinai, Yoav Bergner, Yossi Matias, Yuri Lev
Venue / Source: arXiv (2026)
AI-steered conversations bridge the gap between natural group interaction and rigorous, standardized measurement of student soft skills.
Key takeaways
- An AI-steered framework improves the density of observable evidence for durable skills during naturalistic human-AI collaborative interactions.
- Automated scoring models demonstrate alignment with human expert ratings, offering a potential bridge between conversational assessment and psychometric rigor.
- Active AI facilitation is more effective than simple instruction-based prompts for eliciting student performance data on soft skills.
- The reliance on specific cultural and linguistic participant groups suggests that further validation is necessary to ensure broader generalizability across diverse populations.
- Disparities in evidence rates across different skill sets indicate that some competencies may be more challenging to measure consistently within open-ended conversational environments.
Executive Summary (TL;DR)
Authors' abstract (original)
Durable skills, such as collaboration, creativity and critical thinking, are instrumental to success in the modern workforce. Yet, measuring these skills remains a persistent challenge. Moreover, because what is not measured is often not taught, these skills are often overlooked in mainstream educational curricula. Designing effective assessments for these skills necessitates balancing two often-conflicting requirements: ecological validity and psychometric rigor. On the one hand, the assessment environment should emulate natural real-world human interaction between humans. On the other hand, it should be scalable, controllable and reproducible. Here we argue that LLMs can be used to better capture both of these aims. Concretely, we develop a framework where the subject converses with AI teammates in a way that resembles human-human interaction for authenticity, while also offering the psychometric control required for informative and robust assessment. Importantly, the AI participants not only act as teammates but also, in an "Executive LLM" setup, steer the conversation towards eliciting a high density of observable evidence for skill proficiency. We complement this with an AI evaluator that can be used to measure skill proficiency in such interactions. We evaluate our assessment protocol based on transcripts of interactions of human participants with our AI framework, for multiple durable skills. For the skill of creativity, we further demonstrate the efficacy of an autorater for evaluating complex tasks performed by real students. Our analysis shows that the use of the Executive LLM significantly increases elicited evidence and that LLM-automated scoring of conversations largely agrees with that of expert annotators. This research demonstrates the utility of orchestrated LLMs approaches for measuring complex social and cognitive constructs in a scalable and controllable manner.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a scalable, psychometrically grounded alternative to traditional unproctored or unstructured assessment.
- Demonstrates the utility of simulated agents (using LLMs) as a sandbox for test assembly and validation.
- Illustrates how to integrate pedagogical rubrics into the AI 'prompt' to achieve measurable evidence elicitation.
Methodology and Dataset
The study analyzed 373 conversations from 188 human participants (ages 18-25, recruited via Prolific, U.S.-based native English speakers) interacting with AI teammates in the 'Vantage' system. Additional data included 100 simulated conversations per skill level for recovery evaluation and 180 student submissions for creativity assessment. The models used were Gemini 2.5 Pro and Gemini 3.
Why Worth Reading
This paper offers a concrete engineering solution to the 'ecological validity vs. psychometric rigor' trade-off in assessment. It demonstrates how generative AI can move beyond simple text analysis to become an active, adaptive participant in testing environments—effectively creating a bridge between open-ended conversational tasks and reliable, standardized psychometric measurement.
Key Claims & Verified Quotes
Limitations & Study Design Flags
-
Unrepresentative sample:
found
“Participants ages 18-25 were recruited via the Prolific platform. They were English native speakers based in the United States.” (p. 7)
- Small sample: not found
- The primary evaluation focused on English-speaking participants in the United States, potentially limiting cultural generalizability.
- Automated ratings for critical thinking and creativity in the human-participant context require further validation against expert human benchmarks.
- The study assumes that simulated interactions using LLMs successfully model the behavior of human subjects without direct verification of this model's alignment with human cognitive processes.
- Inter-rater agreement between humans, though improved, remained in a moderate range (Kappa 0.45-0.64), indicating that coding these interactions is inherently challenging.