SimulacraBench Tests AI as Data Surrogate

Overview

A new benchmark named SimulacraBench is set to test whether large language models can serve as viable surrogates for hard-to-reach populations in high-stakes policy research. Announced as part of the Competition Track at NeurIPS 2026, the initiative aims to resolve a critical uncertainty in the humanitarian and development sectors: how well algorithmic representations can stand in for human respondents.

Organized by the Behavioral Sciences & AI Exchange, the competition invites researchers to submit code that predicts responses from unpublished microdata held by three major institutions: the UNHCR, UNICEF, and the World Bank. With approximately 40,000 respondents spanning 14 countries, the dataset represents some of the most sensitive and logistically difficult survey data in existence.

The central premise is pragmatic. Field data collection for vulnerable groups is often slow, expensive, and incomplete. Consequently, agencies are increasingly turning to LLMs to pre-test survey instruments, impute missing responses, and simulate subgroup outcomes. SimulacraBench seeks to quantify exactly how accurate these synthetic surrogates can become.

The Problem with Data Gaps

In classical survey practice, participants answer questions directly, and sample means are used to estimate population-wide outcomes. This method remains the gold standard for accuracy, but it is frequently unfeasible in conflict zones or remote regions where access is restricted.

To bridge this gap, humanitarian programs are exploring the use of AI to create simulacra—algorithmic copies of specific demographic populations. These models can potentially augment sparse human data, allowing agencies to achieve classical-level accuracy across full samples at a fraction of the traditional time and cost.

However, the validity of such an approach is unknown. If an AI model is used to replace human labels entirely, its predictions inherit any biases present in its training data. Because the true responses are never observed, these biases remain hidden from the design, potentially leading to flawed policy decisions.

"Nobody knows how well" AI can represent these populations, according to organizers. The competition is designed to move beyond speculation and provide empirical evidence on the limits of algorithmic substitution.

How the Competition Works

SimulacraBench employs a closed evaluation framework to protect the privacy of respondents. Unlike typical benchmarks where data is released to participants, here the code travels to the data. Participants never see the microdata; only aggregate scores travel back to the competitors.

The process follows three strict steps:

Submission: Developers submit a zip file containing main.py and requirements.txt. This code must implement a predict() function and may list Hugging Face models to be downloaded.
Execution: Organizers install the requirements and run the code on held-out respondents with the network removed, ensuring no external API calls skew the results.
Return: Participants receive a single skill score and the runtime duration. No per-item breakdowns or cell counts are provided, preventing overfitting to the test set.

This "apples-to-apples" structure ensures that all teams are evaluated on identical held-out respondents. The scoring metric is calibration-first, utilizing a normalized log-score that rewards truthful probability estimates rather than simple accuracy on the modal class. A uniform random guess scores zero, while perfect calibration scores one.

Datasets and Baselines

The benchmark features three unpublished instruments, none of which are included in any pretraining corpus. This ensures that any performance is derived from generalization ability rather than memory of specific text.

ERPIS (UNHCR): A household survey of Syrian refugees across four countries of asylum, comprising 10,300 respondents and 285 items.
Faith & Immunisation (UNICEF): A survey covering adults in ten countries regarding religious influence and vaccine uptake, with 19,800 respondents.
Skills Assessment (World Bank): An online instrument for job seekers, primarily in South Africa, with smaller cohorts in Lesotho and Nigeria, totaling 6,900 respondents.

Participants are provided with a schema-only specification. For each item, the schema details the question wording, class, allowed answers, and gating logic. A sandbox generator creates practice data with invented marginals to help developers test their pipelines locally.

Two published baselines are included for comparison: a trivial crowd marginal predictor and a basic LLM prior. The starter kit, licensed under MIT, includes these baselines, the grader, and a tutorial notebook to guide users through the submission format.

Impact and Timeline

The potential impact of this research extends beyond academic interest. If AI can reliably augment survey data with minimal human input, it could revolutionize how global institutions respond to crises. Faster, cheaper data collection would allow for more responsive programming and policy adjustments in real-time.

The competition is scheduled to launch publicly on August 18, 2026. Development submissions will close on November 13, 2026. By validating whether algorithmic predictions can recover classical survey accuracy, SimulacraBench hopes to establish a rigorous standard for the ethical and effective use of AI in humanitarian data science.