LLM-Generated Survey Data Flattens Real Population Diversity

Image via Wikimedia Commons (public domain).
Can large language models stand in for survey respondents? A 2026 PNAS study introduces SSDataBench, a benchmark that judges synthetic data the way survey researchers judge samples: by whether they reproduce population-level statistics. Across seven real datasets, current LLMs tended to compress real-world heterogeneity into simplified types when given sparse information.
Judging synthetic respondents like a survey sample
Much of the excitement about "silicon samples" rests on a simple test: does a model, prompted with a persona, give answers a real person like that would plausibly give? According to Xie et al. (PNAS, 2026), that individual-level framing misses what quantitative social science actually needs. A representative survey is valuable because its statistics approximate those of the target population, so synthetic data should be held to the same standard.
To operationalize that standard, the authors built SSDataBench, a benchmark for population-level statistical realism. It checks five kinds of patterns that social researchers routinely rely on: univariate distributions, bivariate associations, multivariate outcome predictions, distributions of life event sequences, and associations between those sequences and covariates.
The team illustrated the benchmark with four longitudinal and three cross-sectional datasets covering six major social domains: demographics, socioeconomic status, marriage, health, abilities and attitudes. This mix matters, because it moves the evaluation beyond single attitude items toward the life-course and inequality questions at the core of sociology, demography and economics.
The headline finding is a representational limit. According to Xie et al. (PNAS, 2026), current LLMs working under sparse conditioning settings showed a pronounced tendency to compress real-world heterogeneity into simplified typological structures. Put plainly, the synthetic population looks tidier than the real one: fewer unusual combinations, less spread, more stereotyped profiles.
The authors frame their results as a starting point rather than a verdict. The limitation they report is specific to sparse conditioning, where the model receives little information about each simulated person, and they outline a roadmap toward better realism. They also report preliminary results suggesting that domain-specific training can improve population-level realism, which signals that the gap is not necessarily permanent.
Why it matters: If you are considering LLM-generated respondents for piloting, power analysis or hypothesis generation, a plausible individual answer is not enough evidence. Check whether the synthetic data recover the marginal distributions, associations and variance you would expect in the real population, and be especially wary of findings that hinge on tails, subgroups or heterogeneity, since those are exactly what compression erases. SSDataBench offers a concrete template for that kind of validation.
Xie, Y., Liang, L., Li, S., Lu, Y., Xiao, Z., Shi, M., Huang, J., Wang, M., & Xie, Y. (2026). Evaluating the statistical realism of LLM-generated social science data. *Proceedings of the National Academy of Sciences*. Read the original paper. Independent editorial summary of third-party research.





Comments