top of page

Four Roles LLMs Now Play in Social Science, From Subject to Auditor

57 minutes ago
3 min read

Image: "Street crowd reflecting in the polyhedral mirrors of the station Tokyu" by Basile Morin (CC BY-SA 4.0) via Wikimedia Commons, recoloured.

How are large language models changing social science research? A new Review in Nature Human Behaviour sorts their uses into four roles: simulated subjects, analysis assistants, hypothesis collaborators and quality evaluators. It finds real acceleration, but warns that causal inference, ethics and fidelity to social complexity remain unresolved.

Subject, assistant, collaborator, evaluator

Most methods debates about generative AI in social science still treat it as one thing: a chatbot that either can or cannot be trusted. A Review led by Fengli Xu (Tsinghua University) and James A. Evans (University of Chicago and Santa Fe Institute), with co-authors from Tsinghua, HKUST and Microsoft Research Asia, argues that this framing is too coarse. Published in Nature Human Behaviour, the article surveys recent work across sociology, political science, psychology, anthropology, economics and related fields and organises it by the job the model does in the research process.

According to Xu, Evans and colleagues (Nature Human Behaviour, 2026), LLMs now occupy four distinct research roles. As subjects, they simulate human behaviour, standing in for respondents or participants. As assistants, they automate analysis, such as coding and classifying large volumes of text. As collaborators, they help generate hypotheses. As evaluators, they audit and improve the quality of research itself. The authors argue that together these uses do more than speed up existing work: they are creating new methodological paradigms.

The Review also makes a conceptual point that matters for anyone studying markets, organisations or consumers. Because society is becoming saturated with AI agents, the authors write, LLMs are both research tools and authentic research subjects. A model that simulates a survey respondent is a method; a model that negotiates, recommends or persuades in the real economy is part of the phenomenon. The same system can sit on both sides of a study, which the authors describe as a recursive challenge for the field.

The caveats are stated plainly. The authors report that LLMs struggle with causal inference, raise urgent ethical concerns about privacy and misinformation, and that their ability to reproduce social complexity remains uneven and unproven. The Review closes by identifying research directions for using the models' potential while addressing these limits. Note that this is a synthesis of existing studies rather than a new empirical test, so its value lies in the map it draws, not in fresh effect sizes.

Why it matters: The four-role framework gives researchers a practical way to decide how much validation a given use of an LLM needs. Using a model as an assistant to pre-code open-ended responses carries different risks from using it as a subject that replaces human participants, and both differ from asking it to propose hypotheses or review a manuscript. Naming the role explicitly in a methods section, and matching the validation to that role (human benchmarks for simulated subjects, agreement checks for automated coding, independent review for AI-generated hypotheses), is a straightforward way to make LLM-assisted social science easier for editors and reviewers to evaluate.

Source: Xu, F., Shao, C., Chen, L., Zeng, Q., Chen, Z., Sukiennik, N., Wang, J., Gao, C., Wang, H., Lian, J., Meng, Q., Xie, X., Li, Y. & Evans, J. A. (2026). Large language models for computational social science. *Nature Human Behaviour*, 10(11). Read the original. Independent editorial summary of third-party research.

Comments


This initiative is supported by the following organizations:

  • Twitter
  • LinkedIn
  • YouTube
logo_edited.png
bottom of page