Kernel.org’s Creepy Crawlies Are Turning Open Access Into a CPU Tax

Konstantin Ryabitsev says creepy crawlies now keep 14 to 16 CPU cores busy across git.kernel.org, despite defenses designed to make automated scraping expensive. The crawlers request millions of individual commit pages instead of cloning the underlying Git repositories. That choice turns an efficient public archive into a permanent HTML rendering service for unidentified machines.

Ryabitsev, a Linux Foundation systems administrator involved with kernel.org infrastructure, published the figures on August 29, 2026. Simon Willison highlighted them on September 7 because Datasette presents a similar risk profile. It can expose a database through an enormous collection of useful, crawlable pages.

The immediate story concerns Linux infrastructure, but the conflict reaches much further. Open technical archives were designed to help people inspect, reference, and preserve information. Machine-scale clients can convert that openness into an unpriced computing obligation.

The reversal is especially sharp at git.kernel.org. Its maintainers already provide the complete repository history through Git, a protocol built to transfer that history efficiently. Scrapers are reportedly taking the computationally expensive route anyway, requesting separately rendered pages for commits, patches, and diffs.

That behavior pressures maintainers to restrict interfaces that legitimate developers use. The service remains responsive, according to Ryabitsev, but it is already disabling features and placing more actions behind access controls. The cost of the crawler surge is therefore measured in lost openness as well as CPU time.

Creepy Crawlies Generate Six Million Daily Requests

The scale is large enough to consume infrastructure continuously, even when the site appears healthy to human visitors.

Ryabitsev’s crawler measurements describe about 6 million daily requests for apparently random commits. Anubis, a browser challenge placed before the main site, immediately rejects about 66% of those requests. Another 33% solve the challenge and continue to git.kernel.org.

The numbers do not prove that every challenged request comes from an AI company. Ryabitsev explicitly says operators cannot identify every client with certainty. A person can open an old commit, while a bot can imitate a current browser.

Request patterns provide the strongest evidence. A client moving through unrelated commits in old, inactive forks does not resemble a developer following a patch series. Ryabitsev estimates that legitimate activity represents about 2% of total traffic, even after using assumptions favorable to users.

The resulting workload occupies 14 to 16 of 90 CPU cores across five geographically distributed nodes. That equals roughly 20% of the service’s total processing capacity on average. Actual demand arrives in waves, so the load is less orderly than a fixed 20% allocation.

Those cores perform a narrow task. They turn Git objects into HTML pages for clients that then parse the rendered output. Ryabitsev says this activity consumes more CPU time than every form of legitimate access combined, including Git clones.

This distinction matters because a clone transfers a repository through Git’s native object model. The client receives the history and can traverse commits locally. The server avoids rebuilding a presentation page for every object the client wants to inspect.

HTML requests reverse that efficiency. Each request asks the server to locate repository data, run cgit, and construct a human-readable response. The client discards much of the surrounding interface after extracting the text it wants.

Repeated across millions of URLs, an individually reasonable page becomes an expensive machine endpoint. Traditional capacity planning often treats page views as units of comparable work. The git.kernel.org case shows why that model breaks when one URL can trigger much more computation than another.

The service has not collapsed under the current background load. Ryabitsev says poorly designed continuous integration systems can still cause more acute outages, especially when many nodes perform simultaneous shallow clones. Crawler traffic creates a different problem because it permanently removes operating headroom.

That lost headroom reduces tolerance for traffic spikes, maintenance events, and legitimate automation. It also forces operators to spend time studying adversarial behavior instead of improving services for kernel developers. An apparently stable website can therefore carry a substantial hidden cost.

The crucial change is not that public source code can be downloaded. Kernel.org intentionally supports that use. The change is that unidentified clients are choosing a costly representation and demanding it at industrial scale.

Git Makes the Data Cheap, but HTML Makes It Expensive

The central conflict is not access versus secrecy. It is efficient bulk access versus wasteful page-by-page extraction.

Linux development has always benefited from replication. Git repositories can be cloned, mailing-list archives can be copied, and mirrors can preserve material beyond the life of any single server. Kernel.org encourages that redundancy rather than treating every download as a threat.

Git makes this model economical by transferring objects according to their relationships. A repository stores commits, trees, and file contents as addressable objects. Client and server negotiate which objects the client lacks, then move the necessary data without recreating a web page around each commit.

The web interface serves another purpose. It helps a developer inspect one change, share a stable link, compare revisions, or download a patch without cloning a large repository. cgit, the interface used by git.kernel.org, offers several views because each supports a legitimate human workflow.

Those options also multiply the crawlable surface. The main Linux repository contains about 1.48 million commits, according to Ryabitsev’s account. Git.kernel.org hosts about 922 forks that largely share the same underlying objects.

A crawler can therefore discover many URLs that lead to duplicate commit content. It can also request patch views, plain-text renderings, repository trees, and arbitrary comparisons. The content is finite, but the possible URL space becomes much larger.

This is a familiar failure mode for database-backed websites. A database may contain a manageable number of records while its interface permits countless filter, sort, pagination, and comparison combinations. Every generated route looks like another document to an indiscriminate crawler.

Willison connected that pattern to Datasette, his open source tool for publishing searchable databases on the web. A Datasette instance can turn structured information into browsable pages and query results. That accessibility is valuable to people, search engines, researchers, and assistive tools.

It can also expose expensive combinations of parameters. A bot does not need malicious code to create harmful load. It only needs to enumerate links or construct valid URLs faster than the application can serve them cheaply.

The same risk applies to documentation systems, issue trackers, code browsers, public records portals, and personal archives. Sites that transform structured data on demand are more vulnerable than static pages because every fetch can trigger database work or server-side rendering.

Caching helps when many clients request the same resource. It helps less when crawlers deliberately or accidentally spread requests across unique URLs. Query parameters, arbitrary diffs, and old forks can defeat the reuse that makes a cache valuable.

Operators can pre-render popular pages, but pre-rendering every possible comparison is impossible. They can also provide bulk exports, yet a crawler designed around web links may never discover or select the efficient route. Technical availability does not guarantee sensible client behavior.

For developers building searchable archives, this is an architectural warning. Machine access should use bounded APIs, feeds, exports, or repository protocols wherever possible. Human interfaces need limits that reflect the cost of generating each response.

Teams documenting those choices should also preserve operational decisions alongside the code. A searchable engineering knowledge base can connect crawler policies, costly routes, and incident evidence before another traffic wave arrives.

The kernel.org incident demonstrates that bandwidth is only part of the bill. CPU time, cache churn, observability costs, and operator attention can dominate when automated clients repeatedly request dynamic representations.

Anubis Raised the Price Until Crawlers Paid It

Proof of work reduced abusive traffic temporarily, but persistent crawlers adapted because the underlying data remained valuable.

Kernel.org first used familiar defenses. Operators identified suspicious user-agent strings, inspected logs, and blocked addresses associated with obvious automated collection. That approach worked while crawlers identified themselves or originated from a manageable set of networks.

The traffic then became harder to classify. Ryabitsev describes clients claiming to be ordinary browsers and distributing requests across cloud subnets. Blocking an entire network could stop abuse, but it could also deny legitimate automated checks hosted by the same provider.

The next shift weakened address-based controls further. Requests began arriving from large numbers of residential and mobile addresses. Each address generated only four or five requests before disappearing from the logs, according to Ryabitsev.

This pattern resembles proxy networks that route traffic through consumer devices. A site sees what appears to be a broad population of unrelated users instead of a concentrated scraping operation. By the time an address looks suspicious, that particular source may never return.

Kernel.org responded with Anubis, an open source web firewall that challenges clients before letting them reach an upstream service. Its central mechanism is proof of work, a small mathematical task that is relatively costly for the client but inexpensive for the server to verify.

The Anubis project describes the software as a defense for smaller internet services facing sustained AI crawler traffic. Its maintainers also call it a severe response because challenges can block small scrapers and legitimate archival bots.

At git.kernel.org, the challenge initially changed the economics. Automated clients stopped attempting access, while human visitors accepted a modest delay. That period lasted several months.

The bots later began solving difficulty level four. Operators increased the challenge to level five, which demanded more client computation. Ryabitsev says that setting can take several seconds on a mobile device and make the phone noticeably warm.

Higher difficulty again bought several months. It also imposed a larger cost on every legitimate visitor who received the challenge. Accessibility suffers when older hardware, privacy-focused browsers, disabled JavaScript, or unstable connections cannot complete the expected flow.

The crawlers eventually solved level five too. At the time of Ryabitsev’s report, approximately one-third of the 6 million daily commit requests passed the challenge. Proof of work had not become useless, since it still stopped the other two-thirds, but it no longer restored the earlier balance.

This outcome exposes the central weakness of economic deterrence. A challenge works only while the cost exceeds the scraper’s expected value. Valuable, clean technical history gives collectors a reason to spend more resources.

Ryabitsev argues that Linux commit history is particularly attractive because much of it predates the proliferation of generated text. Researchers worry that repeated training on synthetic material can degrade model quality or amplify artifacts. That makes well-structured, human-produced technical records desirable training material.

The exact identities and purposes of the clients remain unverified. Some may support model training, while others may build search indexes, code datasets, security products, or commercial archives. Their shared behavior matters more operationally than the label.

Anubis developer Xe Iaso has also acknowledged that proof of work is not a complete answer. In a technical discussion, Iaso explained that the mechanism targets the economics of mass scraping but expressed skepticism about its benefits against capable distributed clients.

A client with browser automation can execute JavaScript, store cookies, and solve the same public challenge as a person. Distributed clients can divide the work across many devices. Raising difficulty then risks punishing legitimate visitors faster than it deters well-resourced collectors.

Kernel.org’s experience confirms that tradeoff with production data. The defense succeeded, the adversarial clients adapted, and operators increased the cost. The contest did not end because the attacker and defender both had another setting to change.

Open Archives Are Being Forced to Remove Useful Features

The most damaging result is not a higher server bill. It is the gradual retreat of anonymous, human-friendly access.

Kernel.org plans to reduce the number of crawlable URLs and gate operations that are expensive to execute. Ryabitsev warns that anonymous users should expect some functionality to disappear. The underlying data will remain available for download, but obtaining it may require additional steps.

That response is rational for an infrastructure operator. If one interface generates disproportionate load, restricting it protects the rest of the service. Git clones and developer workflows are more important than unlimited anonymous rendering of arbitrary comparisons.

Yet every restriction changes who can use the archive. A developer with Git installed can clone a repository and inspect it locally. A student following a link, a journalist checking one commit, or a person using a constrained device may depend on the browser interface.

Automated research tools can also have legitimate purposes. Search indexing helps users find old fixes. Archival systems preserve evidence. Security services correlate commits with vulnerabilities. Accessibility tools may fetch pages in ways that do not resemble conventional browsing.

Broad restrictions cannot cleanly separate these clients from abusive collectors. Authentication creates accountability but adds administrative work. Rate limits reduce spikes but can fail when requests arrive from distributed addresses.

Blocking residential networks would exclude real households. Blocking cloud providers would interfere with developer automation. Requiring proof of work makes every visitor spend electricity and time before the server knows whether the request is useful.

Robots.txt provides a policy signal, but it is not an access-control mechanism. The official robots standard states that its rules are not authorization. Cooperative crawlers follow the declared preferences, while an unidentified client can ignore them or impersonate a browser.

The result is an asymmetry. Responsible organizations identify their crawlers, publish documentation, obey exclusions, and become easy to block. Less responsible collectors hide their identity and distribute traffic, making them harder to stop.

This can produce a perverse outcome where compliant crawlers lose access while evasive ones continue. It also makes traffic attribution unreliable. Operators may reasonably describe a behavioral class as AI scraping without being able to connect requests to a named model developer.

That uncertainty is the main skeptical issue in this story. Ryabitsev’s traffic measurements are direct operational evidence, but the purpose behind every request has not been independently established. The 98% estimate concerns apparent scraping behavior, not a verified list of AI companies.

The distinction should shape both policy and reporting. It would be inaccurate to assign all load to a specific vendor without network evidence. It would be equally inaccurate to dismiss the problem because every client lacks a public identity.

The measurable harm exists at the application layer. Millions of requests select old commits, duplicate forks, and expensive renderings. A defensive challenge stops many requests, but substantial volumes pay the computational toll and continue.

Cloudflare has approached the broader problem by giving site owners more granular controls for search, agent, and training crawlers. Its AI traffic controls reflect an important distinction because a model-training crawler and a user-requested assistant do not serve the same purpose.

Large edge networks can combine address intelligence, browser signals, traffic history, and customer-wide observations. Small open source services rarely have that visibility. They must make decisions with local logs and imperfect identifiers.

That gap concentrates the damage on organizations least able to absorb it. A large commercial platform can purchase more capacity and deploy specialized bot management. A volunteer project, academic archive, or independent publisher may simply close expensive routes.

The open web then loses more than a few interface features. It loses the assumption that publishing useful, linkable information is economically safe. Pages remain technically public, but access becomes conditional, challenged, or available only through bulk data workflows.

The Real Conflict Is Openness Versus Unpriced Computation

Open data does not require every possible representation to remain free, anonymous, and unlimited.

The ethical argument around crawling often focuses on permission to copy content. Kernel.org adds a second question: who should pay to transform that content into the format a collector prefers?

The Linux repositories are already available through an efficient transfer mechanism. The operators are not withholding history or demanding exclusive control over it. They are objecting to clients that make the public service repeatedly compute a more expensive representation.

That difference separates the case from a simple debate about restricting knowledge. An efficient clone lets a collector bear much of the processing cost after transfer. Commit-by-commit HTML scraping pushes repeated work onto the source.

A responsible collector should first look for bulk exports, repository access, feeds, sitemaps, or documented APIs. It should cache retrieved material, deduplicate forks, and avoid arbitrary parameter combinations. It should also identify itself and offer a working contact channel.

Rate negotiation matters too. A crawler that needs unusual volume can ask an operator for a mirror or scheduled export. Kernel.org says it intends to continue providing data to people who request it, even as anonymous interfaces become more restrictive.

These practices sound elementary because mature search engines developed them over decades. AI demand has expanded the number of organizations collecting web-scale datasets. Not every team appears to have inherited the same operational norms.

Incentives also differ. A collector racing to acquire scarce pre-AI material benefits from moving quickly. The cost of inefficient collection lands on thousands of unrelated site operators. Without contracts, enforcement, or reliable identity, the crawler does not automatically pay that external cost.

Proof of work attempts to move some expense back to the requester. The git.kernel.org results show both the attraction and the limit of that model. It raises the marginal cost, but it cannot distinguish a valuable human request from a valuable automated one.

Charging per crawl creates another possible route, but payment alone does not solve system design. A crawler can still overload a dynamic endpoint if pricing, quotas, and capacity controls are poorly aligned. Small sites also lack the billing systems needed to negotiate machine access.

Better protocol discovery would help. A machine-readable page could point collectors toward a repository clone, compressed archive, or bounded data export. Crawlers would still need incentives or requirements to respect that direction.

Application design can reduce exposure before traffic arrives. Developers can cap result sizes, reject unreasonable comparisons, canonicalize equivalent URLs, and assign separate limits to expensive routes. They can precompute common views and require authentication for unusual queries.

Observability must measure work rather than requests alone. A million cached static responses can cost less than a few thousand uncached database queries. Operators need route-level CPU time, cache effectiveness, challenge completion rates, and client behavior grouped across addresses.

The deeper policy question concerns enforceable identity. A crawler that declares its operator and purpose can receive tailored access. A distributed client pretending to be millions of browsers prevents negotiation and turns every request into a trust decision.

Until identity improves, defensive systems will rely on behavioral inference. That means false positives will remain unavoidable. The human cost should be tracked as seriously as the blocked traffic because a protection that excludes real users can undermine the service it preserves.

Creepy crawlies therefore represent a governance failure as much as a traffic problem. The web has norms for voluntary crawler behavior, but it lacks a dependable framework for machine-scale clients that ignore those norms.

Three Signals Will Show Whether the Pressure Is Easing

The next phase will be measured through traffic behavior, lost functionality, and better crawler accountability.

The first signal is git.kernel.org’s challenge pass rate. About 33% of random commit requests were clearing Anubis when Ryabitsev published his figures. A sustained decline would suggest that new controls restored economic pressure or improved client classification.

A stable or rising pass rate would point in the opposite direction. It would show that collectors continue valuing the data enough to absorb each defensive increase. Another rise in challenge difficulty would also reveal that the contest remains focused on cost rather than identity.

The second signal is the amount of anonymous functionality kernel.org removes. Restrictions limited to unusually expensive comparisons would support a targeted response. Broader losses across ordinary commit, patch, or browsing views would show that crawler pressure is changing the public experience.

Operators should document what disappears and why. That record would help other projects identify dangerous URL patterns before they reach the same point. It would also reveal whether defenses preserve the common human workflows they were meant to protect.

The third signal is whether major crawler operators adopt verifiable identities and efficient acquisition routes. Published address ranges, purpose-specific user agents, contact information, and enforceable rate policies would let sites distinguish cooperative automation from evasive scraping.

Without that accountability, the defensive market will keep moving toward browser fingerprinting, managed edge controls, authentication, and paid access. Those tools can protect capacity, but they also make independent publishing more complicated.

For developers, the immediate action is to inspect which public routes consume the most CPU and how many distinct URLs expose the same underlying record. Test whether a bulk client can retrieve the same information through a cheaper interface.

Publish that route clearly, then place strict budgets around dynamic HTML generation. Monitor proof-of-work completion rates and legitimate-user failures separately. A defense should reduce abusive computation without turning every reader into collateral damage.

For AI builders, the responsible choice is simpler. Use the lowest-cost authorized representation, identify the crawler, honor site policy, and contact operators before scaling. Open access is an invitation to use shared knowledge, not an unlimited claim on someone else’s processors.

The creepy crawlies at git.kernel.org have made that boundary visible. The open web can support machine readers, but only if those machines stop treating every public URL as free computing capacity.