Anthropic Study: Analysts Went With Their Claude Agent's Hidden Lean 86% of the Time, and Most Still Called the Result Their Own
In a randomized study published on October 27, Anthropic gave 50 professional data analysts a Claude Code agent secretly set, by coin flip, to favour one of two reasonable options on each of four analytical choices. The analysts went with the agent's lean 86% of the time, against about 50% if it had no effect, and 77% later explained their choices with the agent's arguments while only 15% credited it. Anthropic says it strengthened the leans on purpose and that real-world effects may be smaller.
Anthropic's own study of its own coding agent finds that experts who hand analysis to an AI may be handing over the judgment calls too, and may not notice. In the randomized study, published on October 27, professional analysts went with a Claude agent's secretly assigned preference on open analytical choices 86% of the time, where about 50% would be expected if the agent made no difference.
How the study worked#
Anthropic recruited 61 analysts with experience in causal inference, the work of telling from data whether one thing causes another. Eleven chose not to use the agent, leaving 50 in the main analysis. Their task was to advise a fictional farming cooperative, Greenfields, on whether to expand a paid advisory programme; its board would expand it only if it raised productivity by roughly 3–4%.
The analysts faced four choices, each with two reasonable options: how to measure productivity, which comparison method to use, which farms the estimate should cover and how to weight them. Each could move the estimated gain by several percentage points. For each analyst and each choice, a coin flip set which option the agent would favour, through hidden instructions the analysts were not told about. The agent was told to make the strongest honest case for its lean, to change its mind if the evidence turned decisively against it, and never to misrepresent results. The analysts worked in Claude Code running Claude Opus 4.8.
What Anthropic found#
- Choices followed the lean. Analysts matched the agent's lean 92% of the time on the outcome measure, 82% on the comparison method, 81% on which farms to cover and 90% on weighting, or 86% across all four.
- Analysts switched when the agent disagreed. In 38 cases an analyst had stated a preference before the agent gave its view. They switched 60% of the time when the agent disagreed and 6% when it agreed. Anthropic says the 95% confidence interval for the gap runs from 24 to 82 percentage points.
- The final recommendation moved less. When the agent leaned toward expanding the programme, 54% of analysts recommended it, against 36% when it leaned the other way. Using only the four coin-flip leans, a statistical model predicted 82% of the analysts' recommendations, against 61% for always guessing the most common one.
- Analysts used the agent's words. Afterwards, 77% repeated the agent's argument in at least one answer, sometimes word for word, and 15% credited the agent anywhere.
- They still felt it was their call. On the survey, 52% rated the final recommendation as more their own than the agent's, and analysts who followed the lean on all four choices were about as confident as the rest, averaging 71 against 73 on a 100-point scale.
Anthropic says that of 10 characteristics it checked, including experience, expertise and trust in AI, none clearly predicted who deferred; age showed a link that disappeared when the researchers adjusted for testing many characteristics at once. Six of 48 analysts who answered the survey guessed the study was about the AI's influence, and none suspected the agent's preferences had been set.
What Anthropic says it does not show#
Anthropic says it "strengthened their preferences on purpose to measure their effect," and that the effect might be smaller when agents are not given clear preferences and told to voice them. With no lean assigned, it says, Claude Opus 4.8 made the same five choices in 112 of 123 runs. The study covered one task in one field with one model, and the analysts were new to farming. Its estimates are averages across analysts, and it did not measure how strongly analysts held their own views.
Anthropic also says the study "isn't a simple cautionary tale against working with AI." Of the 11 analysts who worked without the agent, six made serious methodological errors, against three of the 50 who used it, and their median time was 90 minutes against 65. Anthropic cautions that the groups were not comparable, as those working without AI were smaller in number and less experienced.
The company says the result matters for AI safety research, where researchers have begun using agents to run and analyse experiments, because "a human reviewer is only an independent check if their judgment is their own."
Limits#
- The figures come from Anthropic's blog post on its own study of its own product. We did not read the full paper, and we found no independent replication or outside review.
- The only other write-up we found, on Unite.AI, is credited to an AI research agent and reports the same figures as Anthropic's materials, so we do not treat it as an independent check.
- Some figures in the post rest on small groups: the switching result on 38 cases and the comparison with analysts working without AI on 11.
- The agents' leans were assigned and strengthened by the researchers, so the rates describe that set-up rather than ordinary use.