Cloudflare & OpenAI pilot indexes millions of pages

Cloudflare and OpenAI have reported early results from a research pilot using network signals to improve how AI systems discover web content. The project has enabled OpenAI to index hundreds of millions of previously unknown webpages.

The pilot used signals from about 1% of Cloudflare's global network to help OpenAI's search crawler find pages that had not appeared in its index. Those additional pages give ChatGPT more sources to draw on when answering queries and directing users to websites.

The work addresses a growing problem as more people turn to AI tools for information instead of relying solely on traditional search engines. In that environment, useful pages may be overlooked if they have few inbound links or limited prominence elsewhere on the web.

Cloudflare said its network can provide signals that help identify pages worth discovering, while OpenAI evaluates and ranks those pages for relevance and usefulness. The companies described the process as a way to improve both discovery and the quality of information shown to users.

Discovery signals

Early findings suggest metadata and traffic patterns can help an AI crawler decide which pages to visit and which to ignore. That matters because a crawler that tries to fetch everything on the web can waste resources on duplicate pages, redirects and non-existent addresses.

The blog outlined several patterns that emerged during the pilot. Automated traffic can make dead pages appear active. Some websites redirect large numbers of addresses to a homepage, and multiple URLs can point to near-identical content. Filtering those cases before a page is fetched can reduce unnecessary crawling and ease the load on site owners.

Cloudflare and OpenAI also said they are testing signals that can flag potentially harmful pages, including cases where an otherwise legitimate site contains a compromised page such as a fake login form. Early results indicate those signals can help identify pages that should be excluded from an index.

Indexing shift

The pilot highlights how AI-driven discovery is changing the economics of web indexing. In a conventional search model, a page's visibility often depended on ranking highly in search results before a user judged its value. With AI assistants, systems must first find the right source, assess whether it is useful, and then decide how to present it, whether through a direct answer, a citation or a link back to the original site.

That change puts greater emphasis on signals that improve the first stage of discovery. It also increases pressure on AI companies to keep their indexes current without sharply increasing the volume of requests sent to websites.

Cloudflare said the aim is not simply to crawl more pages more often. Instead, the system tries to determine when a page is worth revisiting and when repeated fetching would add little value. For website operators, that could mean less crawler traffic requesting unchanged or invalid pages. For OpenAI, it means a larger share of search resources can be directed towards pages that may help answer questions.

Control for publishers

The companies also stressed that discovery signals do not override a domain owner's choices about access. Cloudflare said a signal about a page does not grant permission for a crawler to retrieve it, and distinguished between using web metadata for search and discovery and using page content for model training.

That point is likely to matter to publishers and website owners who want visibility in AI products but also tighter control over how bots access their sites. The debate over AI crawlers has sharpened as media groups, online publishers and web platforms seek compensation, attribution or technical controls over the use of their material.

Cloudflare said customers can use its AI Crawl Control tools to manage how AI systems access their websites. The pilot suggests the company sees discovery infrastructure as an area where network operators can play a larger role in shaping how AI systems interact with the open web.

For OpenAI, the broader significance is that a larger, fresher index could improve the range of sources available to ChatGPT, particularly for queries that depend on niche, local or recently updated information. Examples cited in the research included product manuals, passport requirements, restaurant menus and opening hours, where a specific page can determine whether an answer is useful or wrong.

The next phase of the work will focus on improving page selection, testing faster change signals and measuring what happens after content is discovered. The companies said the pilot "has enabled OpenAI to discover and index hundreds of millions of new web pages, helping people discover more useful content and get more questions answered through ChatGPT."