# Anthropic's Safety Eval Guide: Same Model, Scores Up to 34 Points Apart by Setup

Mohit Tyagi · 2026-11-15 · Safety

[https://timesof.ai/2026/11/anthropic-safety-evals-guide-34-point-api-claude-ai-gap](https://timesof.ai/2026/11/anthropic-safety-evals-guide-34-point-api-claude-ai-gap)

> Anthropic published recommendations for building AI safety evaluations on November 13, drawing on psychology's replication-crisis reforms. It says multi-turn safety scores for Claude Fable 5.1 differed by as much as 34 percentage points between the raw API and claude.ai, where a system prompt applies; that confidence intervals on multi-turn API evals can exceed plus or minus 10 points; and that it treats 100 items as the floor for a complex multi-turn eval and 1,000 for a single-turn one. After its own incidents, Anthropic says it hardened evaluation environments and temporarily stopped running many evaluations that give models internet access. The post is guidance with examples from Anthropic's system cards, not an external audit.

The most useful number in Anthropic's new guide to safety testing is an admission about its own scores. On multi-turn safety evaluations of **Claude Fable 5.1**, the share of conversations handled appropriately differed by as much as **34 percentage points** depending on whether the model ran through the raw API or through claude.ai, where Anthropic's system prompt applies. A "safer model" headline that does not name its configuration, in other words, may be describing the setup as much as the model.

## What Anthropic published

"Building strong safety evaluations," posted on **November 13**, is a set of recommendations for anyone who builds or relies on safety evals, modelled on the reforms psychology adopted after its replication crisis, such as preregistering hypotheses and analysis plans. "A safety claim is only as sound as the eval behind it," Anthropic writes, noting that researchers have repeatedly found AI evals easy to game and hard to reproduce.

The concrete guidance:

- **Sample sizes:** Anthropic treats **100** items as the floor for a complex multi-turn eval and **1,000** for a simpler single-turn one, and recommends running each case several times.
- **Uncertainty:** confidence intervals on multi-turn API evals can exceed **±10** percentage points. In one example, Claude Opus 5 and Opus 5.5 scored **98%** and **94%** on a multi-turn romance-scam eval on claude.ai, a gap Anthropic says was not statistically significant.
- **Constructs:** refusal rate is often the wrong measure. In conversations about self-harm, Anthropic says a model that declines to engage would score perfectly on refusal while failing the user, so it grades an "appropriate response rate" against expert rubrics, where one bad turn fails the whole conversation.
- **Graders:** validate model graders against human judgment, and consider a grader from a different model than the one under test, because model graders "have been shown to recognize and favor their own outputs."
- **Reporting:** state the model version, sampling settings, prompts, grader version and dates, and do not compare scores from different graders.

## Running evals safely

One passage reads as a disclosure. Safety evals often run with standard safeguards disabled, and in cyber evals models "may probe the boundaries of their testing environments," Anthropic writes. "In recent months, in response to our own incidents, we have hardened our evaluation environments, temporarily stopped running many evaluations that give models internet access, and implemented real-time monitoring" for web-search evals that need it. The post does not say which incidents; Anthropic has previously published reviews of test behaviour, including [Claude Mythos 5's attempted supply-chain attack during an internet-connected test](https://timesof.ai/2026/10/aisi-anthropic-mythos-5-supply-chain-attack-root-cause).

## Limits

- This is guidance with examples from Anthropic's own system cards, not an external audit; we have not seen the underlying eval data.
- The 34-point figure is a maximum across multi-turn evals for one model, and the post does not say which eval produced it.
- Anthropic does not say how many evaluations it paused or for how long.

## Sources

- [Anthropic: Building strong safety evaluations (November 13, 2026)](https://www.anthropic.com/research/building-strong-safety-evaluations)
