Cloudflare Says The Internet Has a Second Audience, but Agents Do Not Pay Like People

Cloudflare says The Internet has a second audience after daily requests from AI agents on its network grew more than 1,700% in one year. More than half of the traffic it observes is now automated. That shift breaks a basic economic assumption behind the open web.

People visit pages, view advertising, buy subscriptions, and sometimes become customers. AI agents often extract the useful information without delivering any of those outcomes. They still consume bandwidth, computing capacity, and publisher content.

Cloudflare now wants websites to treat agents as a distinct audience. Its tools identify automated visitors, separate their purposes, set access rules, and test new payment models. The primary conflict is no longer publishers against all bots. It is open discovery against uncompensated extraction.

The company occupies an unusually influential position in that conflict. Cloudflare says more than 20% of the web uses its network. It also says nearly 80% of leading AI companies rely on its infrastructure.

That reach gives Cloudflare visibility across both sides of the emerging market. It also gives the company considerable influence over how publishers and AI services negotiate access.

The announcement is not simply another update to bot management. Cloudflare is proposing an economic and technical layer for an internet increasingly visited by software acting for people.

Cloudflare’s Traffic Numbers Mark a Structural Change

Cloudflare’s central claim is that automated traffic has crossed from background activity into a primary internet audience.

At the end of 2024, Cloudflare handled an average of 63 million HTTP requests each second. The company now reports an average of 115 million, with peaks above 150 million.

Cloudflare attributes part of that increase to AI agents. Daily agent requests across its network reportedly grew by more than 1,700% during the past year.

Those figures appear in Cloudflare’s September 30 agentic web announcement. They describe traffic passing through Cloudflare, not a complete census of every internet request.

That distinction matters. Cloudflare has an enormous network, but its customer mix and classification methods shape the sample. Its numbers remain company-reported measurements rather than an independent estimate of all global activity.

Even with that limitation, the direction is difficult to dismiss. Automated systems are fetching, comparing, indexing, and acting on web content at a growing scale.

Cloudflare says the fastest-growing category is not the traditional crawler. It is the agent, meaning software that retrieves information or completes a task for a person.

A training crawler collects pages for model development. A search crawler builds an index. An agent typically visits because someone asked a chatbot to answer a question or complete an action.

That agent might compare insurance policies, find a restaurant, check product availability, or buy access to a dataset. The human remains behind the request, but never necessarily visits the underlying website.

This behavior makes agent traffic economically ambiguous. It can represent customer intent while bypassing the interfaces that websites built for customers.

Cloudflare says agent activity follows human patterns, including weekly cycles and seasonal changes. That suggests at least some traffic reflects real demand instead of indiscriminate collection.

However, a request tied to human intent does not automatically create value for the website. An agent can retrieve an answer without generating an advertisement, subscription, lead, or direct sale.

The result is a new accounting problem. Page views once provided a rough signal of audience attention. Automated requests can increase that number while reducing its commercial meaning.

The shift also changes infrastructure planning. A publisher must serve requests even when those requests deliver little measurable return.

Cloudflare says heavily crawled sectors have experienced human traffic declines of up to 40% in less than one year. It identifies retail, software, IT services, and financial services among those categories.

That is a consequential claim, but it should not be read as proof that AI caused every decline. Search changes, market conditions, and measurement differences can also reduce direct visits.

Cloudflare’s strongest evidence is the simultaneous change in traffic composition. Software requests are rising while the commercial value of many website visits is weakening.

That combination creates the article’s central pressure point. Publishers cannot assume more traffic means a larger paying audience.

The Internet Has a Second Audience That Does Not Click Ads

The Internet has a second audience, but its economic behavior is fundamentally different from the human audience that financed publishing.

For roughly three decades, search engines offered websites an implicit exchange. A crawler received permission to index content, then search results sent people back to the source.

Those visitors could read advertisements, register for accounts, purchase products, or subscribe. Discovery and monetization were separate steps, but they usually occurred in the same commercial loop.

Answer engines alter that exchange. They retrieve information from several sources and assemble a response inside an AI product.

A useful answer can satisfy the user before any source page opens. The publisher provides part of the value, yet the AI interface controls the audience relationship.

This does not mean every AI interaction eliminates a visit. Some responses produce citations, referrals, purchases, or follow-up research.

The problem is that publishers cannot assume those outcomes. They need data showing which systems crawl their work, which answers cite it, and which interactions return people.

AI agents add another complication because they can act instead of merely answering. A shopping agent might compare products and complete a purchase without loading the seller’s normal storefront.

That agent can still be a valuable customer. Yet the merchant needs a way to recognize it, provide reliable data, enforce terms, and attribute the transaction.

The same issue applies to research services. An agent might collect a specialist passage for a report while the original reader never sees the publication.

This is where The Internet has a second audience becomes more than a memorable phrase. Websites must decide whether an automated visitor is a threat, a distribution channel, or a customer.

Cloudflare says the answer depends on intent. Its data showed training represented 22% of declared crawler requests in spring 2025. That share reached 52% by June 2026.

Training traffic and user-directed traffic create different value for publishers. A model developer might reuse content across future products, while a live agent responds to a current customer request.

Blocking both categories protects content but reduces discovery. Allowing both preserves reach but can surrender valuable material without compensation.

That is why the old distinction between human and bot is no longer sufficient. Publishers need to distinguish search indexing, model training, and agent activity.

This pressure reaches beyond media companies. Product documentation, comparison pages, research archives, and public knowledge bases can all become inputs for automated answers.

Companies already building an AI knowledge base face a related governance question. They must decide which information systems can retrieve and how that information remains attributable.

The commercial stakes differ by business model. An advertisement-supported page loses value when an agent bypasses the advertisement.

A subscription publication risks losing control over premium reporting. An online store might welcome an agent that completes a purchase through an approved transaction path.

An API provider already expects software customers. Its challenge is letting unfamiliar agents buy limited access without negotiating an account beforehand.

Cloudflare’s argument is that these cases need different policies. A universal block or universal permission cannot reflect their distinct risks and benefits.

That is the reversal at the center of the announcement. Automated traffic was once treated mainly as an operational or security concern.

Now some bots resemble customers, while others resemble extractive intermediaries. Website owners need to know which one is knocking before they answer.

Cloudflare Separates Search, Agents, and Training

Cloudflare’s response begins with classification because publishers cannot set meaningful terms for an audience they cannot identify.

The company replaced its single AI bot switch with separate controls for search, agent, and training activity. Cloudflare says these controls are available across its plans.

The separation addresses a basic consent problem. Fewer than 1% of Cloudflare sites block search crawlers, while 17% block training crawlers.

Those numbers suggest many operators want discovery without unrestricted model training. A combined crawler can make that preference difficult to enforce.

Mixed-use crawlers perform several tasks under one identity. A website that blocks training might also lose search visibility when both activities share the same crawler.

Cloudflare introduced Disallow AI Training on September 15 to address this conflict. The mechanism lets a site remain searchable while signaling that its content cannot be used for training.

Apple, Google, and Microsoft have committed to honor the signal, according to Cloudflare. Compliance still depends on crawler operators respecting the declared rule.

Identity therefore becomes the next layer. Web Bot Auth allows an automated operator to cryptographically sign requests.

A cryptographic signature links a request to a verifiable operator. It reduces reliance on user-agent strings, which can be copied by impersonators.

Cloudflare says OpenAI, Google, and AWS are among the operators signing requests. The company reports more than 500 billion verified bot requests each week.

Verified identity does not guarantee acceptable behavior. It tells a publisher who made the request, which makes policy enforcement and accountability more practical.

Cloudflare also offers AI Crawl Control, Business Insights, and BotBase. Together, these products aim to show who visited, which pages they requested, and what referral traffic returned.

The broader content controls include recommended settings based on a website’s business model. Ad-supported pages can disallow training and block agents where human views generate revenue.

Website owners can change those settings. That flexibility matters because a publisher might welcome search indexing while charging for research-agent access.

The controls also create pressure on AI companies. An operator using clearly separated and authenticated crawlers can receive access that a mixed or unidentified crawler loses.

That incentive might encourage companies to label search, training, and user-directed retrieval more precisely. It could also fragment crawler behavior across several identities.

Cloudflare is positioning itself as the policy enforcement point. Its network can inspect requests before they reach a website’s origin server.

This arrangement reduces work for individual publishers. It also concentrates important classification decisions inside one infrastructure provider.

Cloudflare says its standards remain open. Web Bot Auth can support different identity providers, while website owners can choose other partners.

Still, adoption will determine whether openness exists in practice. A standard becomes useful only when major agents sign requests and major websites enforce the resulting identities.

Classification errors present another risk. A legitimate agent might be blocked, while a determined scraper might disguise its intent or avoid declared crawler infrastructure.

No technical control completely resolves bad-faith access. The immediate value lies in making compliant operators easier to identify and manage.

Cloudflare’s approach therefore targets the cooperative majority rather than every hostile scraper. It creates a structured negotiation between publishers and AI companies willing to identify themselves.

The next question is whether that negotiation produces enough money to support the websites supplying the information.

Payment Turns Bot Management Into a Market Bet

Cloudflare is moving beyond access control by testing whether individual agent interactions can support a functioning content market.

Large publishers and AI companies have already signed licensing agreements. Cloudflare counts more than 50 publisher and AI deals since 2023.

Most are bilateral arrangements between large organizations. They do not offer a practical route for a specialist blog, independent publication, or small data provider.

Cloudflare is testing two broader mechanisms. Pay Per Use compensates a publisher when content contributes to a defined AI product outcome.

Monetization Gateway charges an agent when it accesses a page, dataset, API, model, or tool. The products share identity, measurement, pricing, settlement, and analytics components.

Pay Per Use attempts to connect payment with downstream value. A buyer defines the event it will pay for, such as a citation or recommendation.

Publishers review the offer and decide whether to participate. The AI company then reports qualifying uses through Cloudflare.

Cloudflare checks that each reported use belongs to an enrolled publisher. It handles billing and distributes the resulting payments.

The company’s Pay Per Use beta addresses a weakness in charging for every crawl. An AI service often retrieves more content than it ultimately uses.

Charging only for access can make exploration expensive. Paying for actual use aligns the fee with a buyer’s declared value.

That model introduces a different weakness. Usage reporting is supplied by the buyer, so publishers must trust both the commercial terms and the reporting process.

Cloudflare says publishers can see what was used, when it was used, and what it earned. Some buyers can also report the questions that surfaced the content.

That feedback could influence editorial decisions. A specialist publisher might update material that agents regularly cite or make high-demand information easier to retrieve.

Monetization Gateway addresses services where the request itself creates value. APIs and tools already operate this way, but agent buyers may lack established accounts.

When a request matches a seller’s rule, the gateway returns HTTP 402 Payment Required. The agent can then pay through x402, an open payment protocol.

The closed beta is available to eligible United States Cloudflare customers. Sellers can define prices per request, query, or token, including caps.

Cloudflare offers the example of a sports data service charging when an agent requests a league statistic. The human receives the answer without visiting an advertisement-supported page.

Cloudflare also uses the gateway for its AI Gateway inference service. That internal deployment can expose operational problems before external customers depend on it.

Pay Per Crawl remains a related access model. Cloudflare documentation says website owners can block, allow, or charge individual crawler operators.

Its crawler payment rules charge each successful access, including repeated requests for the same page. Standard files such as robots.txt and sitemap.xml remain free.

Advanced configuration lets operators exclude selected paths or set dynamic pricing. A website can therefore keep general information open while charging for high-value sections.

These mechanisms replace one binary decision with several options. A publisher can allow search, reject training, accept paid uses, and charge transactional agents.

That flexibility is strategically important. It acknowledges that a citation, a model-training copy, and a completed purchase do not create equivalent value.

However, a menu of controls does not guarantee demand. AI companies must decide that licensed access improves their products enough to justify payment.

Publishers must also find payments meaningful after infrastructure expenses and administrative complexity. A technically elegant market can still fail if transaction volume or pricing remains weak.

Cloudflare explicitly describes Pay Per Use and Monetization Gateway as bets. The company also acknowledges that pricing and discovery remain unresolved.

That caution is appropriate. The technical rails exist before anyone knows whether a stable market will run across them.

What Cloudflare’s Numbers Do Not Prove

The rise of automated traffic is clear, but the business case for treating every agent as a customer remains unproven.

The first uncertainty concerns measurement. Cloudflare sees a large and significant portion of web traffic, but not the entire internet.

Its customers also span different industries and technical profiles. Traffic flowing through the network might not represent websites outside that footprint.

Bot classification adds another source of uncertainty. Cloudflare distinguishes declared crawler purposes, verified identities, and behavioral patterns.

Operators can mislabel requests, change infrastructure, or route traffic through services that obscure origin. New agent designs can also challenge existing categories.

A second uncertainty concerns causation. Cloudflare reports steep human traffic declines in heavily crawled sectors, but correlation cannot establish the full reason.

Search algorithms, changing consumer habits, economic cycles, and website quality can all affect traffic. AI answer engines are one factor within a larger distribution shift.

The third uncertainty is commercial adoption. Publishers have strong reasons to want compensation, but AI companies also have alternatives.

They can license a smaller set of premium sources, rely on public data, or reduce crawling. Some may direct agents toward structured APIs instead of general websites.

A fragmented payment landscape could create another barrier. Agents need predictable ways to discover terms, authenticate, estimate costs, and record purchases.

Cloudflare’s use of x402 and Web Bot Auth aims to standardize those steps. Competing providers and protocols could still divide adoption.

Market power is another concern. Cloudflare says the underlying standards are open and that customers can choose other providers.

Yet Cloudflare operates the visibility, identity, access, and settlement layers in its own product stack. That creates convenience alongside concentration risk.

Publishers should examine who controls transaction records, disputed classifications, and payment relationships. They should also consider how easily those records can move elsewhere.

The Pay Per Use model contains a specific trust problem. Buyers report which content generated payable uses.

Cloudflare validates reported events against participating publishers, but it cannot independently observe every internal contribution to an AI response. Attribution inside generative systems remains technically difficult.

A model can synthesize information from several sources. Determining which source caused a sentence, ranking, or recommendation is rarely straightforward.

The definition of a payable use will therefore depend on contract terms and instrumentation. Different buyers can measure similar outcomes differently.

Payments might also favor content that is easy to measure rather than content that creates the most value. A visible citation is simpler to track than background knowledge shaping an answer.

Website operators face their own operational risks. Aggressive blocking can reduce search visibility or prevent beneficial agents from reaching current information.

Permissive access can increase infrastructure expense and weaken control over proprietary material. The best policy depends on the site’s revenue model and content type.

Cloudflare’s controls improve the decision surface, but they do not make the decision automatic. Publishers still need evidence connecting crawler activity with referrals, sales, citations, and payments.

They should avoid treating all automated traffic as either harmless demand or theft. Both labels erase the differences Cloudflare’s new system is designed to expose.

The fairest assessment is that Cloudflare has built a market experiment around a verified trend. Automated audiences are growing, while their sustainable economic role remains unsettled.

Three Signals Will Show Whether the Agentic Web Pays

The next test is whether verified agent traffic produces measurable value rather than another layer of dashboards and protocols.

The first signal is adoption by major AI operators. More companies must separate search, training, and agent crawlers while signing requests through Web Bot Auth.

Broad adoption would strengthen Cloudflare’s classification model. Continued mixed-use crawling would weaken it by keeping access decisions unnecessarily binary.

The second signal is publisher revenue from Pay Per Use and Monetization Gateway. Cloudflare has not yet shown that these systems generate sustainable returns across smaller websites.

The key metric is not enrollment or waitlist size. It is recurring payment volume compared with the cost of serving and managing agent traffic.

Cloudflare’s own announcement says thousands of sellers joined the Monetization Gateway waitlist. Demand from sellers matters, but demand from paying agent operators matters more.

The third signal is crawling efficiency. Cloudflare says more than half of AI crawler activity can involve retrieving unchanged pages.

The company is working with OpenAI on signals that identify fresh or relevant content before another complete crawl. Initial findings were expected within weeks of the announcement.

Tools such as Markdown for Agents can remove presentation code that software does not need. WebMCP can expose structured actions instead of making agents interpret visual interfaces.

These changes could reduce wasted bandwidth and improve reliability. They might also make Cloudflare’s infrastructure more central to agent access.

The Internet has a second audience, but the phrase will matter only if websites can distinguish valuable requests from extraction. Identity, policy, and payment must work together.

For site owners, the immediate action is measurement. Separate human referrals, search crawling, training access, and user-directed agent activity before changing broad rules.

Then test policies against actual business outcomes. Does an authenticated agent produce a sale, citation, qualified lead, or useful payment?

For AI builders, the question is equally direct. Will your agents identify themselves, respect declared purposes, and pay when access creates commercial value?

The old web exchange relied on referrals that were imperfect but understandable. Its replacement needs records, enforceable terms, and economics that work for both sides.

Watch the authenticated traffic share, the payments publishers actually receive, and the reduction in wasteful crawling. Those three outcomes will reveal whether Cloudflare is building a market or merely describing one.