Cloudflare and OpenAI Say Pilot Helped OpenAI Index Hundreds of Millions of New Pages
In a co-authored Oct. 15 post, Cloudflare and OpenAI said a pilot begun in July, using a participating cohort of about 1% of Cloudflare's network, has let OpenAI discover and index hundreds of millions of new web pages. They say it uses metadata for discovery, not page content for training. The results are the companies' own.
The companies' own early results make a case that a network operator can steer an AI crawler. In a post co-authored by staff from both companies on Thursday, October 15, Cloudflare and OpenAI said a research pilot launched in July, which "used a participating cohort of ~1% of Cloudflare's total network, has enabled OpenAI to discover and index hundreds of millions of new web pages."
Cloudflare's network signals help OpenAI's search crawler find pages, and OpenAI then independently evaluates and ranks them, the post says. The signals also filter out non-existent and redirecting addresses, tell crawlers when pages have changed, and are being tested to flag individual harmful pages such as planted fake login forms.
"A signal about a page doesn't grant permission to access it," the companies write, adding that the research uses web metadata "for search and discovery, rather than sharing page content for model training." The next phase will measure what happens after content is discovered.
Limits
- All results are the companies' own; no independent coverage of these results was found. How sites joined the participating cohort was not described.