Language Models Trained on A/B Tests Lift Email Clicks by 32%
Image: "Canberra (AU), Letter Boxes at Kendall Lane -- 2019 -- 1829" by Dietmar Rabich (CC BY-SA 4.0) via Wikimedia Commons,_Letter_Boxes_at_Kendall_Lane_--_2019_--_1829.jpg), recoloured.
Can randomized experiments teach a language model to write better marketing content? A new study in the Journal of Marketing Research fine-tunes a model on within-test comparisons from A/B tests. In a field experiment with 283 million email impressions, subject lines it improved earned 32% higher click-through rates than those written by unassisted human experts.
Turning experiments into training signal
Firms run A/B tests constantly, but the results are usually read once to pick a winner and then archived. Panagiotis Angelopoulos, Kevin Lee and Sanjog Misra propose a different use. Their argument is that a randomized experiment reveals more than which variant won: comparing variants within the same test points to a direction of improvement, and that direction can serve as supervision for a generative model.
The authors call the approach causal alignment. Using pairs of higher- and lower-performing content drawn from the same experiment, they fine-tune a language model into what they describe as an improver: given a piece of content, it rewrites it into a version expected to perform better. Because the model edits rather than generates from scratch, it can sit on top of any existing decision-maker, such as a human copywriter, and propose improvements to that person's draft.
According to Angelopoulos, Lee and Misra (Journal of Marketing Research, 2026), the framework was validated in a large-scale email marketing field experiment covering 36 new campaigns and 283 million impressions. Subject lines improved by the aligned model achieved click-through rates 32% higher than those written by unassisted human experts. A general-purpose language model with 30 times as many parameters as the fine-tuned model did not improve outcomes, which the authors read as evidence that alignment to experimental data adds value beyond raw model capability.
The scope of the evidence is worth keeping in view. The field test concerns email subject lines and click-through rate in one marketing setting, and the article is currently an advance (EXPRESS) version. Whether the same gains hold for longer content, other channels, or downstream outcomes such as purchases is not established by the results summarized in the abstract, so readers should consult the full paper for the authors' own discussion of boundary conditions.
Why it matters: For marketing and social science researchers, the study reframes the archive of past experiments as a training asset rather than a record of decisions. It also offers a clean comparison that many AI evaluations lack: a smaller model aligned to causal evidence against a much larger general model, tested in the field rather than on benchmarks. Researchers who run repeated experiments, from survey wording to message framing, may find the design a useful template for linking generative AI to identified effects.
Angelopoulos, P., Lee, K., & Misra, S. (2026). Causal Alignment: Augmenting Language Models with A/B Tests. Journal of Marketing Research. Read the original paper. Independent editorial summary of third-party research.
Comments