Towards Scalable Measurement of Durable Skills
Amir Globerson, Amy Keeling, Anisha Choudhury, Anna Iurchenko, Aviad Segal, Avinatan Hassidim, Ayça Çakmakli, Ben Gomes, Benn Witt, Cathy Cheunga, Cristine Legare, Diana Akrong, Eliad Carmi, Elisabeth Bauer, Gal Elidan, Hadas Gelbart, Hairong Mu, Katherine Chou, Lev Borovoi, Nir Kerem, Niv Efron, Noa Kerrem Gilo, Preeti Singh, Rajvi Kapadia, Rena Levitt, Roni Rabin, Ronit Levavi Morad, Rotem Yulzary, Shashank Agarwal, Sophie Allweis, Tracey Lee-Joe, Tzvika Stein, Yael Bar Moshe, Yael Haramaty, Yaniv Carmel, Yishay Mor, Yoav Bar Sinai, Yoav Bergner, Yossi Matias, Yuri Lev · arXiv (2026) · doi:10.48550/arxiv.2609.15864 · gemini-3.1-flash-lite
TL;DR
Measuring soft skills like collaboration and creativity is difficult because it requires balancing realistic group interaction with the structure needed to grade performance. Researchers created a new system where AI teammates actively steer conversations to prompt users to demonstrate these skills during natural-sounding discussions. They found this method effectively captures enough behavior to allow AI to grade students with accuracy comparable to human experts.
Method and sample: The study analyzed 373 conversations from 188 human participants (ages 18-25, recruited via Prolific, U.S.-based native English speakers) interacting with AI teammates in the 'Vantage' system. Additional data included 100 simulated conversations per skill level for recovery evaluation and 180 student submissions for creativity assessment. The models used were Gemini 2.5 Pro and Gemini 3.
#Psychometrics#Generative AI#Assessment Design#Durable Skills#Collaboration Assessment#Automated Scoring