AI in Research: Who Is Really Answering Your Survey?
Image: "Questionnaire 0001" by roger_mommaerts (CC BY-SA 2.0) via Wikimedia Commons, recoloured.
Welcome to this week's scan of how AI is reshaping research, curated from Nature, Science, PNAS and the leading business journals.
Human or machine? Two PNAS studies put your data to the test
For anyone who collects data online, this is the study to read this week. Grace Zhang, Oleg Urminsky, Kianté Fernandez and colleagues report in PNAS that they first validated a battery of AI-detection tests that reliably separated verified human participants from both prompted commercial AI agents and purpose-built ones. They then fielded surveys on seven online platforms. The share of respondents failing the AI tests ranged from 6% to 41% depending on the platform, against a 2.4% false-positive rate among in-person human participants. Crucially, they also show that undetected AI agents can shift the results of online surveys. The fear that agents are posing as survey-takers is real, but unevenly spread: the authors note that some platforms currently appear to deliver data with a low rate of AI agents.
A companion problem sits on the other side of the ledger: researchers who deliberately use LLMs to *simulate* people. Nicolò Pagan, Petter Törnberg, Christopher Bail and Christopher Barrie, also in PNAS, introduce the Conversational Human Authenticity Test (CHAT), a validation framework that combines detectability, semantic similarity and interpretable linguistic features. Benchmarking nine open-weight LLMs across five calibration strategies (including fine-tuning, stylistic prompting and context retrieval) on real posts from X, Bluesky and Reddit, they find that model output remains clearly distinguishable from human text even after calibration, especially in affective tone and emotional expression. Instruction-tuned models generally did worse than base models, bigger models did not obviously help, and there was a trade-off: making text *seem* more human often pulled it further from what humans actually said.
Why it matters: Together these papers mark a turn from worry to measurement. If you run online surveys, platform choice and built-in AI checks are now part of your method, and worth reporting in your paper. If you use LLMs as synthetic respondents or agents in social simulations, "it looks plausible" is no longer a defensible validation standard; frameworks like CHAT give reviewers, and you, something concrete to test against. The common lesson: the boundary between human and machine data is now an empirical question you have to answer in every study, not an assumption.
More from this week
Nearly 30% of US STEM dissertations now contain AI-written text
An NBER working paper (not yet peer reviewed), reported by Nature, ran the abstracts of more than 226,000 US STEM dissertations from 2019 to May 2026 through the Pangram AI detector. AI-generated text first appeared in 2023 at 1.7%, rose to 18.8% in 2025 and reached 29.4% in early 2026, with the highest rate in mathematical and computer sciences (36%). Linking graduates to LinkedIn profiles, the authors found that those with AI-generated abstracts were 17% less likely to be employed in academia and 16% more likely to be in industry. The data are correlational, but they raise pointed questions for doctoral training and supervision.
ChatGPT hit creative work that Google Translate spared
Using two natural experiments in a major online labour market, Yilmaz, Naumovska and Aggarwal compare Google Neural Machine Translation with ChatGPT. Google's task-specific AI reduced demand for regular translations but had no detectable effect on 'transcreation', the creative, culturally adapted work. ChatGPT produced substantially larger and more uniform reductions across both kinds of task. The lesson for anyone theorising about automation: which human work is 'safe' under predictive AI does not straightforwardly carry over to generative AI.
Will generative AI make all content look the same? Competition says no
Lab and field evidence suggests generative AI reduces the diversity of what people produce. Manish Raghavan points out that producers compete for customers and attention, and shows formally that competition selects for diverse AI models, mitigating homogenisation. A model that tops a benchmark in isolation may add little value in a competitive market, which argues for evaluating models across the breadth of their output distributions. Experiments with text and image models back up the theory.
A principled recipe for measuring constructs in big text with BERT
Tudoran, Richter, Higuera Reina and Verma propose a two-stage framework for organisational researchers: qualitative interpretation and Structural Topic Modeling are used to develop coding schemes, and BERT is then fine-tuned to apply those schemes to every document. They set out two routes, construct-guided and pattern-discovery, both ending in validated, auditable per-document measures, and illustrate the approach on more than 20,000 employee reviews. It is a useful template for anyone turning large text corpora into theory-ready variables.
That's this week. Forward it to a colleague who's about to launch an online survey, and explore AI tools for your own research at gaiforresearch.com.
Comments