Check an AI-generated data analysis before using its results

An AI-generated analysis is ready for substantive review when you can inspect its code, trace the data transformations, and reproduce its outputs from a clean start. Begin with a bounded analysis and test the points where an unnoticed choice could change the result.

Download the research workflow templates, a ZIP containing editable CSV logs for evidence extraction, citation checking, qualitative coding review, and analysis checks. Open the files in your spreadsheet or research workspace.

This intermediate guide is for researchers using an assistant to write R or Python. You need an analysis question, a data dictionary, an approved dataset, and enough subject knowledge to judge the model specification. For the initial setup, use our coding-agent workflow.

Specify the analysis before asking for code

State the unit of observation, outcome, explanatory variables, inclusion criteria, treatment of missing values, and planned analysis. Ask the assistant to list ambiguities before implementing anything. Distinguish planned analyses from later exploration in your project record.

For example: “One row represents a participant. Estimate the association between training hours and the score, adjusting for the prespecified baseline variables. Describe missingness before fitting the model. Ask before changing exclusions, transformations, or the specification.”

If you are still developing the analysis, request candidate approaches and their assumptions, then make and record the choice. A runnable script alone cannot establish that the statistical approach fits the research question.

Protect inputs and separate outputs

Keep the raw data unchanged. Use separate locations for source data, scripts, derived data, figures, and logs. Record package versions and the instructions for running the project. The Turing Way's research-compendium guide explains why separating inputs, methods, and generated outputs makes research easier to inspect and reuse.

Start the assistant on a synthetic or approved sample when you are developing code. Review the code before giving it access to sensitive data or allowing it to modify files. Use the data environment approved for your study.

Test transformations with small examples

Ask for a data audit that reports row counts, variable types, missing values, duplicate identifiers, and values outside expected ranges. Inspect the results yourself. Then test key transformations on a few hand-checkable rows.

Fictional example: a score of 1 on a five-point item should become 5 after reverse coding, and 5 should become 1. A missing score should remain missing under a rule that preserves missingness. Those checks are more informative than asking the assistant whether its formula looks correct.

For a merge, count rows before and after and inspect unmatched keys. A many-to-many join can duplicate observations. For a recode, show the mapping and confirm that all observed values are covered. For a group mean, hand-calculate one small group and compare it with the code's output.

Review the statistical specification

Compare the code with the recorded plan. Check the reference category, transformation, estimation sample, weights, clustering, and uncertainty calculation where relevant. Inspect diagnostics appropriate to the model. Ask the assistant to explain a line of code in terms of the research design, then verify that explanation.

Check whether conclusions describe association or causal identification appropriately. Keep effect sizes and uncertainty visible. Avoid selecting among specifications solely because one produces a desired significance level; document sensitivity analyses and the reason for each.

Run the analysis from a clean session

Restart the language session or runtime, then execute the documented entry point. Confirm that the script reads only declared inputs and recreates the reported tables and figures. Where randomness is involved, record seeds and relevant environment details; exact reproducibility may also depend on software and hardware.

The Turing Way's reproducibility definitions distinguish repeating a computation with the same data and code from testing a finding with new data. Keep those aims separate in your reporting.

Keep a review record

Use this prompt after the first implementation:

“Compare this script with the analysis plan. List deviations, undeclared inputs, destructive file operations, silent row exclusions, and assumptions that need a researcher decision. Propose small checks with known expected outputs. Do not alter the specification or raw data. Explain what remains unverified.”

Reviewing with another model can suggest additional checks, but agreement between models does not validate the analysis. A colleague's review, suitable diagnostics, and hand-checked examples provide different kinds of evidence.

Finished output: unchanged raw inputs, readable code, a recorded environment, passing checks, reproducible figures and tables, and an account of the decisions made by the research team.

Explore the research learning paths. Original tutorial and fictional test example, published 6 November 2026. Turing Way sources checked on that date.