Build1 publisher2 min readPublished
Cloudflare routes agent web search through AI Gateway with three crawler-vetted providers
Cloudflare now sells web search from Ceramic.ai, Exa and Linkup through AI Gateway at partner list prices, billed from gateway credits with no markup. Teams already on the gateway skip a separate search vendor account, though their agents still have to issue each search call.
The Engineer · Build desk
What happened
- Cloudflare says agents usually guess a page's URL and curl it, and wrong guesses come back as 404 Not Found errors.
- The three partners committed to Cloudflare's Verified bots requirements, and every search response must link to where the content was crawled.
- Cloudflare says it will identify which partners support Zero Data Retention, so customers know their query data is not kept.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Each result carries a crawled source link, so an agent's follow-up fetch can target a page a crawler actually reached instead of a guessed address.
- decision Because the gateway adds no markup, choosing between gateway credits and your own key comes down to existing contracts and where the team wants the invoice.
- exposure Until Cloudflare names its zero-retention partners, a team sending customer data in search queries has no basis in Cloudflare's material for picking a provider on retention.
- precedent Crawler compliance is now an entry condition for gateway search, so any further provider must identify its crawler and cite sources to get the same distribution.
Anyone who has watched an agent try three spellings of a docs path will recognise the pattern Cloudflare describes [1]. Each wrong guess costs a tool call and returns a 404 with nothing usable in it.
The replacement is a search query. The provider returns structured snippets from the live web, and Cloudflare says those go straight into the model's context [3]. The rule I like best is the attribution requirement. Every search response must include a link to the location of the crawled content [4]. The agent gets a real URL for its next fetch, and a site owner gets a trail from the answer back to the page.
The billing works like the rest of the gateway. Search queries draw down the AI Gateway credit balance [8]. They appear in the same observability logs as other gateway traffic, and admins decide who can reach which provider [7]. Cloudflare charges the partners' list API prices with no markup [10], so a query through the gateway costs the same as buying direct at list [1]. Teams with existing provider contracts can bring their own keys, as they already can for model inference providers [11].
The claim that search grounding becomes a gateway setting holds for the operational side and stops at the client. The direct path in the post is a standard REST endpoint, authenticated with the AI Gateway token [12]. The agent still needs a tool definition or a code path that sends the query and handles the results. What the gateway takes over is billing, logging, key handling and access policy [6][7][11]. On the credits path, a team already on AI Gateway does without a separate provider account and a second place to read request logs [7][8]. Cloudflare points to its web search developer docs for the rest [13].
The partner terms matter beyond this product. Each provider's crawler must meet the "Verified bots" requirements in Cloudflare's developer documentation [4]. The post describes this as identifying the crawler, respecting robots.txt and giving the source of search results [5]. Ceramic.ai, Exa and Linkup accepted those terms to get distribution through the gateway [2][4].
Data retention is still open. Cloudflare says it will identify partners that support Zero Data Retention [9], and the post does not yet name them. In my view, queries that carry customer data should stay off all three providers until Cloudflare publishes that list.
What to watch
- Cloudflare naming which of Ceramic.ai, Exa and Linkup support Zero Data Retention.
- Whether the web search developer docs show results injected into an inference call without a separate agent tool call.
- Further search providers joining under the Verified bots and source-link conditions, or a launch partner leaving.