Webinar: 9 AI Workflows For Better Content. Nov 18, 2 PM ET.

Save Your Spot
  1. SEJ
  2.  ⋅ 
  3. AI Search

Cloudflare CEO Likens AI Agent Optimization To A New Version Of SEO

  • Cloudflare's CEO says products for a new web business model brought first revenue.
  • He says OpenAI pays Cloudflare for hints that tell its crawler when pages change.
  • He says sites may charge agents less when the agents explain why they're visiting.

Cloudflare's CEO says OpenAI pays for hints that tell its crawler when pages change, and sites may charge agents less when they explain their visits.

Cloudflare CEO Likens AI Agent Optimization To A New Version Of SEO

Cloudflare recorded its first revenue from what it calls Act 4 products in the third quarter, co-founder and CEO Matthew Prince said on the company’s earnings call. Act 4 covers the products meant to help create a new business model for the internet.

He also said OpenAI is paying for hints that tell its crawler when web pages change. For customers that want more traffic from AI agents, Cloudflare is helping with what he described as “sort of the new version of SEO.”

Prince gave no figure for Act 4 revenue on the call. Cloudflare’s third-quarter results release, which reported revenue of $782.5 million, up 39% year over year, doesn’t break it out either.

OpenAI Pays For Page-Change Hints

A Goldman Sachs analyst asked when it will be time for Act 4 products to generate meaningful revenue. Prince replied that Cloudflare is already “recognizing significant amounts of revenue” due to the efficiency it brings to AI agents’ visits to websites. As an example, he pointed to the partnership with OpenAI that was explained during Cloudflare’s August earnings call.

He also described how under the partnership, Cloudflare provides OpenAI with information as to when a webpage has been updated (for example, a restaurant updates their pricing), so OpenAI’s crawler doesn’t have to keep going back and re-fetching the same webpage.

“OpenAI is paying for that,” he said, adding that he expects other AI companies to do the same.

Cloudflare made this public on July 8, in a research pilot using signals from the websites participating (for example, content freshness and page changes). In its Oct. 15 results posting, the company said it was still working with OpenAI to determine how better discovery could lead to business outcomes (I wrote about this on Oct. 20).

Prince did not indicate during his call what OpenAI is paying for this relationship or when the payments started.

Optimize For Agents Or Block Them

Asked by a Wells Fargo analyst whether Cloudflare customers want more traffic from AI agents, Prince said the answer depends on the customer, and that some want as many agents as possible.

“Others see them as a bane to their business and want to keep them away,” he said.

Cloudflare helps the customers who want agents draw more of them and handle the traffic efficiently, according to Prince. Customers who see agents as a threat get them blocked, and they aren’t charged for that traffic. He split customers the same way on the August call.

An agent choosing where its user eats dinner might pull menus from 1,000 restaurant websites, he said, and the user goes to one. The other 999 restaurants carry a cost with no benefit. A person might look at two or three pages, and the scale with agents is far higher.

On the August call, Prince said that if current trends continued, nonhuman traffic would be as much as 1,000 times human traffic within five years. This time, he said he expects it to happen sooner.

Traffic on that scale would need more bandwidth and more servers. “It can’t be a free ride for all the agents that are out there,” Prince said.

Discounts For Agents That Explain Themselves

Beyond efficiency deals like OpenAI’s, Prince expects some form of micropayments when agents visit sites. The more an agent tells a site about why it’s there, the more a site may be willing to “charge it less to access the page,” he said.

In his example, an agent tells a travel site it’s planning a Hawaii vacation for a family of four. The site might let that agent in for free because the request opens up advertising opportunities.

He isn’t sure how long it will be before that becomes a major source of revenue. Looking 10 years out, he said, “I would imagine that this becomes the largest part of Cloudflare’s business.”

Cloudflare moved two payment products, the Monetization Gateway and Pay Per Use, into beta on Sept. 30. The Gateway lets sellers charge AI agents per request, and Pay Per Use pays publishers who accept an AI company’s offer for reported uses.

Why This Matters

The one Act 4 payment Prince described in detail goes to Cloudflare for making OpenAI’s crawling more efficient. Whether the websites in the OpenAI pilot get a share wasn’t addressed on the call.

Prince laid out a choice, to welcome agents or block them, that is currently possible within Cloudflare’s dashboard. Cloudflare offers AI Crawl Control on every plan. This allows users to permit or deny access to individual AI crawlers, and its documentation lists the Pay Per Crawl option as being in a private beta. The CEO didn’t mention a single Cloudflare service to optimize websites for agent traffic during the call.

Looking Ahead

Prince expects Cloudflare to talk a lot more about its Act 4 products “in the quarters to come.”

Keep an eye on whether other AI companies follow OpenAI’s lead and pay for crawl hints, as Prince predicted. It’ll also be worth seeing whether Cloudflare’s next disclosures explain how much of that money reaches the sites whose pages the agents use.

Featured Image: Algi Febri Sugita/Shutterstock

Category News AI Search
SEJ STAFF Matt G. Southern Senior News Writer at Search Engine Journal

Follow on YouTube. Matt G. Southern is the Senior News Writer at Search Engine Journal, where he’s covered Google, SEO, ...