Check AI-generated data analysis with a worked CSV example
Use this small exercise to check whether an AI-generated analysis follows your data rules. You will identify a duplicated export row, report a missing value, calculate group means, and compare the result with a reproducible answer. Allow about 25 minutes. You need a spreadsheet viewer; running the supplied check also requires Python 3.9 or newer.
Updated 6 November 2026. The data are synthetic teaching observations. The numbers illustrate validation and support no conclusion about real AI performance.
Define the question before asking for code
Question: what is the difference in average task-completion time between the fictional assisted and manual groups, using observed times only?
The unit is one participant. The CSV contains participant_id, condition and minutes. The assisted rows are P01, 12 minutes; P02, 15; P03, 18; P04, missing. The manual rows are P05, 20; P06, 25; P07, 30; P08, 35. A ninth row repeats P03 with 18 minutes.
For this exercise, the repeated P03 row is a known export duplicate. Retain one copy. Keep P04 in the participant count, report its missing time, and exclude it from the time mean. In a real study, establish duplicate provenance first: repeated measurements may be legitimate observations.
Give the assistant a precise analysis request
Copy this prompt and attach observations.csv:
“Audit this synthetic CSV before analysis. The unit is one participant. Report row count, unique participant count, duplicate IDs, conflicting duplicate records and missing times by condition. A repeated identical row is an export duplicate in this exercise. Keep one copy; stop for conflicting values. Exclude missing times only from the time means and report their counts separately. Calculate mean minutes by condition and assisted minus manual. Return Python standard-library code and a short interpretation. Do not fill missing values with zero, infer causal effects or claim statistical significance. State whether you executed the code or only wrote it.”
Before running generated code, inspect its imports and file operations. This task needs local CSV reading and arithmetic. Ask for an explanation of any network access, extra package or modification to the input file.
Check the answer by hand
Input: 9 rows, 8 unique participants, 1 identical duplicated row removed.
Assisted: 4 participants, 3 observed times, 1 missing time. Mean = (12 + 15 + 18) / 3 = 15 minutes.
Manual: 4 participants, 4 observed times, 0 missing times. Mean = (20 + 25 + 30 + 35) / 4 = 27.5 minutes.
Difference, assisted minus manual: 15 − 27.5 = −12.5 minutes.
A suitable interpretation is: “In these synthetic observations, the assisted group's observed mean is 12.5 minutes lower. One assisted observation has missing time. These descriptive numbers do not establish an effect in a population.”
Run the independent check
Unzip the pack, open a terminal in its folder, and run python3 audit_csv.py. Compare its output with expected.json. Then run python3 test_audit.py. On Windows, py -3 can replace python3 if that is your Python launcher.
The reference script uses Python's standard library and needs no account, API key or internet connection. We executed it against the supplied CSV and checked the expected answer plus nine invalid-input cases. It rejects conflicting IDs, unknown groups, malformed rows, and invalid times. Read the code before adapting it to another study.
Diagnose a plausible wrong answer
If the assisted mean is 15.75, the duplicated P03 row was probably retained: (12 + 15 + 18 + 18) / 4. If it is 11.25 after deduplication, the missing P04 value was probably treated as zero: 45 / 4. A fluent explanation can accompany either mistake.
Change the last P03 time from 18 to 19 in a copy of the CSV. The reference script should stop with a conflicting-record error. That interruption is useful: the script cannot determine which measurement is correct without the source record.
Next, ask the assistant to explain why the participant count and the mean's denominator differ in the assisted group. Keep that explanation beside the code and output as part of the analysis record.
Transfer the checks to your research
Write down the observation unit, ID rules, allowed groups, units, missing-value rules and exclusions before generating an analysis. Preserve the raw file. Save the exact script, data version, software version and output. With joins, also check that the number of rows and unique IDs changes only as intended.
This exercise covers descriptive arithmetic. Statistical modelling additionally requires a justified design, assumptions, uncertainty estimates and a review of how missingness affects the research question.
Sources and next steps
Python CSV documentation explains the reader used in the reference script. The data, prompt, arithmetic and tests are original GAI for Research teaching materials.
Watch the curated data-analytics webinar with our practice checklist. For another reproducible research task, try the Inciteful literature-search exercise.
Comments