Thursday, Nov 19 | --:--
Back to home

OpenAI Says Agents Finish 46% of 2-4 Hour Research Tasks Alone; Data Is Self-Reported

OpenAI's November 18 report on research acceleration estimates that coding agents now complete 46% of research tasks that would take a human two to four hours without human intervention, up from 23% in March, and 35% of four-to-eight-hour tasks, up from 13%. Median researcher agent use reached about $2,400 a day at API list prices in October, up 62% from September; a footnote says a harness change had depressed September's recorded token spend. OpenAI says the success rates may reflect which tasks researchers delegate, and the figures are internal estimates; we found no outside assessment of them.

5 min read San Francisco View as Markdown
A handmade editorial collage on torn ivory paper in ultramarine and saffron: a long wooden workbench of paper models, the left half holding neatly folded ultramarine paper boats, cubes and fans and the right half still flat cream and saffron sheets with small paper gliders resting on them, no people or hands; an official OpenAI logo is glued on a torn-paper clipping tucked under the bench's front edge at bottom right, the full wordmark visible.

OpenAI's second report on how much AI speeds up its own research arrives with its caveats attached. OpenAI estimates that coding agents now complete 46% of research tasks that would take a human two to four hours without human intervention, up from 23% in March, and 35% of four-to-eight-hour tasks, up from 13%. But OpenAI also says the success rates may reflect researchers choosing different tasks to hand over, and the figures are internal estimates for which we found no outside assessment.

The usage numbers carry a flag too. At API list prices, the median researcher's daily agent use reached about $2,400 in October, up 62% from $1,500 in September, and the 90th percentile reached $23,000, up 105% from $11,000. A footnote says a temporary harness change deployed on August 21 coincided with a sharp fall in input-token use that "appears to have severely depressed" documented token spend; it was gradually reverted from September 10, after which spend "rapidly increases." Our reading is that part of the September-to-October growth is a rebound, though OpenAI does not say how much.

What OpenAI reports#

The report, dated November 18, updates a September post and rests on OpenAI's internal measurements of how coding agents have changed its AI development cycle. Success without human intervention varies by kind of work. From August 1 to October 18, OpenAI estimates it was 69% for tasks it classes as Communicate and 58% for Analyze, against 43% for Build. By task, it was 79% for monitoring training runs for issues, 72% for answering technical questions, 35% for launching training runs, 29% for building evaluation, reinforcement-learning and agent environments, and 25% for writing code for inference and serving stacks.

Use has widened as well as deepened. Agent workdays per researcher per day rose from 0.5 to 5.7, defined as every eight hours of agent use, and OpenAI notes the metric generally counts interactive use only. By mid-October more than two-thirds of the researchers included averaged a daily peak of at least four concurrent agents, and more than 10% averaged at least sixteen. As of September, OpenAI says, the API-equivalent value of agent use was about twice researchers' non-equity compensation, while internal deployment took only 2% of total research compute.

In a late-August to early-September survey of 101 research staff, the median would give up 40% of team headcount or 30% of team compute to restore frontier coding agents for three months. OpenAI also reports that documented Safety and Alignment workweeks rose 77% from January to September, the largest increase among its research areas.

Claims and checks#

OpenAI does not present this as proof of runaway progress. It says its task-success rates may reflect "both capability improvements and researchers' changing choices about when and how to use agents," and describes its survey as self-reported with an ad hoc sample. It says it publishes the data because labs should be required to track and publicly report indicators of recursive self-improvement, and that the reporting "should not be read as an intention to pursue full RSI."

The outside record is thin and measures different things. OpenAI's own footnote lists Anthropic's three reports, a Google DeepMind safety report section on AI-written code, and a METR report with Anthropic on how much internal Claude models have accelerated the capabilities frontier. METR's assessment of Anthropic estimates capability growth at one lab, not task completion at another, so the figures cannot be compared.

Limits#

  • Every figure is OpenAI's internal estimate or survey result. We found no outside assessment of this report.
  • Task-success rates are estimates, and OpenAI says they may reflect delegation choices as well as capability.
  • The October spend growth is measured from a September baseline that OpenAI's footnote ties to a harness change.
  • The survey covers 101 OpenAI research staff, relies on self-reports and used an ad hoc sample, OpenAI says.
  • The agent-workday metric counts interactive use and may undercount automated runs.

Sources#

Prior Coverage

Earlier Times of AI reporting on this thread.

Scroll to continue reading