OpenAI Web Scraping Exposed a Conflict Microsoft Could Not Keep Private

OpenAI web scraping now faces a damaging contradiction from inside its closest corporate partner. A Microsoft executive reportedly called the practice the “largest theft of labor in human history.”

The statement did not come from a critic protesting outside the AI industry. Microsoft Director of Applied Science Brent Hecht reportedly wrote it in an internal January 2023 memo. Microsoft has invested in OpenAI and integrated its models across products, including Copilot.

Unsealed material from a copyright lawsuit has now brought that private warning into public view. The documents suggest employees at Microsoft and OpenAI understood that AI products could weaken the publishers supplying their training material.

That evidence does not settle whether model training qualifies as fair use under US copyright law. It does sharpen the central conflict facing OpenAI and Microsoft. They describe training as transformative while their own employees reportedly worried that the resulting products would replace journalism and undermine its economics.

What the Unsealed Filings Say Happened

The new development is not simply another allegation that OpenAI copied articles. It is evidence that people inside the companies recognized the resulting conflict.

The material emerged from the consolidated copyright litigation involving The New York Times and other publishers. The Times originally sued Microsoft and OpenAI in federal court in December 2023.

Publishers allege that the companies copied copyrighted articles to create training datasets and commercial AI products. OpenAI and Microsoft maintain that their uses are transformative and protected by fair use.

The newly visible documents add internal communications, deposition testimony, and reported dataset figures to that disagreement. However, an important limitation remains. Much of the information appears through the publishers’ summary judgment brief, while some underlying exhibits remain sealed.

That means readers are seeing quotations selected and presented by one side in active litigation. The statements are consequential, but their full surrounding discussions are not always available.

According to an OpenAI scraping report, Hecht warned that large models were absorbing human work on an extraordinary scale. Other coverage identifies his language as “an astonishing theft of unprecedented proportions.”

Hecht reportedly wrote that millions of people would view models “hoovering up” their work as theft. He then described the practice as possibly the “largest theft of labor in human history.”

The phrase is intentionally sweeping and historically contestable. Its importance here comes from who reportedly used it, not from accepting the comparison as a measured historical judgment.

Hecht worked inside Microsoft, the company serving as OpenAI’s most important technology and commercial partner. His warning suggests that the ethical and economic objections were not confined to publishers, authors, or outside researchers.

Microsoft has sought to distance itself from that conclusion. A company spokesperson told AFP that Hecht’s comments represented one employee’s individual perspective, not Microsoft’s position.

The filings reportedly describe more than ten million scraped news articles. Nearly one-third allegedly came from The New York Times, according to an article-copying account based on AFP reporting.

Another figure concerns mid-training datasets, which are collections used to continue training a model after its initial development. Those datasets reportedly contained 91,692 copies of works from the Times, Daily News, and Center for Investigative Reporting.

A dataset derived from Common Crawl allegedly contained more than two million documents from nytimes.com. Common Crawl is a nonprofit archive that periodically collects large portions of the public web.

The publishers also claim Microsoft supplied material to OpenAI through initiatives called Project Taxi and Project Mango. Project Mango allegedly produced a dataset containing at least 160,903 unique works owned by news organizations involved in the litigation.

The allegations go beyond collecting publicly accessible pages. The filing reportedly cites an OpenAI employee who described a method for getting around the Times paywall. OpenAI President Greg Brockman allegedly answered, “ah nice.”

That exchange will receive close scrutiny because access restrictions can affect both legal and factual analysis. Copying an openly available page and bypassing a paywall do not create identical questions.

The filing also alleges that some dataset preparation removed copyright notices before information reached a model. OpenAI has not conceded that these practices establish infringement, and the quoted exchanges require their full context.

Still, these details change the character of the dispute. The case is no longer framed only as publishers inferring how their work reached ChatGPT.

The filings offer a reported view of internal knowledge, technical acquisition methods, and commercial concerns. Those subjects can influence how a court assesses purpose, market effects, and the credibility of each side’s narrative.

Why OpenAI Web Scraping Became a Fair-Use Test

OpenAI web scraping matters legally because the same content allegedly helped build products that can answer users without sending them back to publishers.

Fair use permits some unlicensed uses of copyrighted works. Courts generally examine the use’s purpose, the nature of the original work, the amount copied, and the effect on its potential market.

No single factor decides every case. The analysis depends on specific works, copying methods, outputs, and markets.

OpenAI argues that model training transforms source material into statistical representations used to generate new responses. It also argues that facts contained in news coverage are not protected by copyright.

Publishers answer that their lawsuits concern protected expression, not ownership of facts. They argue that OpenAI copied complete articles at scale and built products that compete for the same readers.

Recent AI copyright decisions have given developers meaningful support. Courts in California have treated some uses of books for model training as transformative, although the facts and remaining claims varied.

OpenAI has cited those decisions in its defense. The Times argues that its dispute presents a stronger market-substitution record because ChatGPT and Copilot can deliver timely information resembling a publisher’s output.

That distinction makes the internal statements especially important. OpenAI’s head of ChatGPT, Nick Turley, reportedly wrote that the company’s products were “largely substitutive.”

He allegedly predicted that they would become more substitutive as their quality improved. Turley also reportedly described AI products as an “existential threat” to publishers.

Microsoft CEO Satya Nadella reportedly acknowledged during a deposition that chatbot conversations had replaced some visits to underlying websites. That does not automatically establish legally cognizable market harm, but it supports the publishers’ factual theory.

Microsoft’s internal measurements could prove more significant. The filings reportedly say its Copilot answer engine reduced click-through rates to the Times domain by as much as 93 percent compared with traditional Bing search.

Click-through rate measures how often users follow a displayed link. A decline can reduce advertising impressions, subscriptions, brand discovery, and the ability to build a direct reader relationship.

The figure needs careful handling. It comes from litigation materials, and its methodology has not been fully tested in public. It may reflect particular queries, interface designs, or measurement periods rather than all Copilot traffic.

Even so, it points toward the question at the heart of the case. Does an AI answer create a new tool, or does it replace the market for the reporting used to produce that answer?

An internal Microsoft presentation reportedly described the situation as a “doom loop.” If AI answers reduce publisher traffic, publishers receive less revenue. Lower revenue then supports fewer reporters and fewer original articles.

That leaves future models with less reliable material to absorb. The AI service can therefore weaken the information supply on which its quality depends.

The concern is particularly acute for reporting that requires expensive human work. Investigations, court coverage, foreign reporting, and local government journalism cannot be generated from existing text alone.

Someone must attend the hearing, verify the document, interview the source, and assume responsibility for publication. A model can compress that work after it appears, but compression does not finance its creation.

The case entered a decisive phase when the parties asked US District Judge Sidney Stein to resolve issues through summary judgment. Summary judgment lets a court decide claims without a trial when no material factual dispute requires a jury.

An AI copyright case review reported that Stein is expected to rule on whether major portions should proceed toward trial. The final decision is not expected immediately.

The US Justice Department has supported OpenAI’s position on training, emphasizing scientific progress, economic growth, and national security. That intervention shows how far the dispute extends beyond one publisher.

A broad decision against model developers could raise the cost of assembling training corpora. A broad decision for developers could sharply reduce publishers’ control over how commercial models use their archives.

Microsoft and OpenAI Face Their Own Words

The primary conflict is between a public fair-use defense and private warnings that AI systems could replace their essential content suppliers.

OpenAI and Microsoft do not have to prove that their technology leaves every existing market untouched. Transformative products often change industries, and market disruption alone does not establish copyright infringement.

Their harder problem is narrower. Internal statements reportedly anticipated substitution, falling referral traffic, damaged publisher economics, and declining future content quality.

Those concerns resemble elements of the publishers’ case. They also complicate the companies’ effort to portray harmful market effects as speculative or remote.

One Microsoft document reportedly observed that an end product rarely threatens the economic foundations of its essential suppliers. It then said Microsoft had created precisely that situation for the content supply chain supporting large language models.

A large language model, or LLM, predicts and generates text from patterns learned during training. Its “content supply chain” includes the people and organizations producing material incorporated into datasets or retrieval systems.

That framing treats journalism as an input to AI development. It also recognizes that the input does not appear automatically.

Publishers employ reporters, photographers, editors, designers, lawyers, and technical teams. They fund records requests, travel, source development, fact checking, and corrections.

ChatGPT can separate the information from the surrounding publication experience. A user receives a synthesized answer without necessarily seeing the reporting process, advertisements, subscription offer, or source relationship.

For readers, that convenience is the product. For publishers, the same convenience can remove the moment when reporting creates economic value.

The tension becomes sharper when paywalled work enters training. Nadella reportedly testified that anyone using paywalled information for grounding or training should license it.

Grounding means supplying a model with external information to support a response. It differs from training because the information can be retrieved when the user asks a question.

Nadella reportedly said he would have invoked Microsoft’s right to require retraining if he had known OpenAI trained on material behind a paywall. That statement does not prove he knew such copying occurred.

It does reveal a distinction Microsoft’s chief executive considered important. Material protected by payment and access controls carries a clearer expectation of licensed commercial use.

OpenAI has pursued licensing agreements with numerous publishers since ChatGPT launched. The Associated Press, Axel Springer, News Corp, and other organizations have announced various content arrangements.

Those agreements support two opposing interpretations. OpenAI can argue that licensing reflects a forward-looking effort to build sustainable partnerships rather than an admission about past conduct.

Publishers can argue that the agreements demonstrate a functioning licensing market. If comparable content has a licensable value, unauthorized copying can look less like an abstract technical process.

The industry has not converged on one approach. Some publishers license archives, some block AI crawlers, and others sue. Several use all three strategies across different products and periods.

The resulting market favors companies with bargaining power. A large international publisher can negotiate compensation, product placement, and attribution terms.

A small local newsroom has less leverage. Its reporting can still contain unique facts that become valuable to an answer engine, especially during emergencies or local political disputes.

This imbalance helps explain why the litigation has expanded beyond the Times. The broader group includes national, specialist, investigative, and local news organizations.

A federal judge allowed the central copyright claims to continue in 2025. An earlier court ruling dismissed some claims but preserved the main dispute over copying and model development.

OpenAI welcomed the dismissed claims and said it builds models with publicly available data in a manner grounded in fair use. Microsoft declined to comment on that ruling.

The distinction between public availability and permission remains unresolved. A work being readable online does not necessarily make every form of copying lawful.

At the same time, copyright does not grant publishers control over facts or ordinary learning. The court must decide where computational training falls between those established principles.

The Filings Are Damaging, but They Are Not a Verdict

The internal language strengthens the publishers’ narrative, yet it does not independently decide infringement, damages, or fair use.

The most dramatic quotation comes from a Microsoft employee, not a judicial finding. Calling an activity “theft” in an internal memo does not determine whether it meets the legal elements of copyright infringement.

Employees often use moral, strategic, or rhetorical language that differs from legal analysis. A court will examine conduct and evidence rather than treating one person’s vocabulary as controlling law.

Microsoft has already emphasized this point by describing Hecht’s comments as an individual perspective. The company can also argue that internal debate demonstrates responsible risk analysis rather than corporate knowledge of wrongdoing.

Companies routinely ask employees to identify worst-case outcomes. A risk memorandum can contain blunt warnings precisely because decision-makers want potential harms stated clearly.

The full exhibits matter for that reason. Without surrounding messages, recipients, and responses, readers cannot know whether executives accepted, rejected, or investigated each warning.

The publishers’ brief also has an advocacy function. Its lawyers selected evidence that advances their motion and organized that evidence around their legal theory.

OpenAI and Microsoft will interpret the same record differently. They can challenge the meaning of quotations, the representativeness of traffic figures, and the causal connection between AI answers and publisher losses.

They can also argue that training and product output should not be collapsed into one act. A model can be trained on a work without reproducing that work for users.

Conversely, a product can generate a substitute answer using licensed, public-domain, or separately retrieved information. The court may need to evaluate different datasets, models, and features independently.

The reported ten-million-article figure also requires precision. The existence of an article in a dataset does not reveal how often it affected model behavior.

Nor does a copied document necessarily remain recoverable as a stored article. Models generally encode learned statistical relationships rather than functioning as ordinary searchable archives.

Yet memorization can occur. Models have sometimes reproduced recognizable passages, especially when material appeared repeatedly in training data or prompts were designed to elicit it.

The publishers argue that such outputs expose copying and compete with the original work. OpenAI argues that atypical prompting and isolated examples do not represent normal product use.

This factual disagreement matters because market substitution is not simply a question of whether someone prefers ChatGPT. Courts will examine whether the challenged use harms an existing or reasonable potential market protected by copyright.

The reported 93 percent click-through decline appears relevant, but the number alone cannot answer that question. The comparison must account for query types, result presentation, citations, and user intent.

A navigational search for a specific Times article differs from a general request for a weather summary. One seeks the publisher, while the other seeks information that many sources can provide.

The phrase “largest theft of labor” can also distract from the legal issues. Its historical scale invites an emotional argument that neither side needs to win the case.

A more useful inquiry asks what was copied, under which access conditions, for what commercial purpose, and with what measurable effect. Those questions connect the technical process to copyright doctrine.

Readers should also separate OpenAI web scraping from every form of AI-assisted search. Training, live retrieval, indexing, caching, summarization, and verbatim reproduction are different operations.

Each can involve different permissions and product choices. Treating them as one undifferentiated practice makes it harder to identify workable rules.

For knowledge workers, the controversy carries a practical warning. Material placed online can travel through datasets, indexes, and answer systems that are difficult to audit later.

Maintaining a traceable personal knowledge base cannot prevent outside scraping. It can preserve sources, authorship, and context when AI-generated summaries blur where information originated.

The same discipline matters inside companies. Teams adopting AI tools should document which sources support generated claims and which licenses govern those sources.

That requirement is not only about avoiding litigation. Provenance helps users distinguish original evidence from model-generated synthesis.

What Happens Next in the OpenAI Scraping Case

Three signals will show whether these disclosures change the legal outcome, the licensing market, or the design of AI answer products.

The first signal is Judge Stein’s summary judgment decision. The ruling will determine which issues can be decided now and which require a trial.

If the court accepts major portions of the publishers’ argument, internal evidence about substitution and paywalls will become more consequential. A trial could expose additional exhibits and testimony to public scrutiny.

If the court grants broad protection to OpenAI and Microsoft, the disclosures will remain reputationally damaging without producing the legal shift publishers want. That outcome would strengthen the fair-use position of other model developers.

A split decision is also plausible. The judge could distinguish training on accessible pages from paywall circumvention, particular outputs, or specific Microsoft data programs.

Such distinctions would matter more than a simple winner label. They could establish different rules for collection, training, retrieval, and generated responses.

The second signal is how licensing practices change before a final judgment. OpenAI has already shown that it will negotiate with publishers under some circumstances.

Watch whether agreements begin covering historical training, live retrieval, model citations, revenue sharing, or all four. Each term addresses a different part of the conflict.

Historical training licenses would assign value to past dataset use. Retrieval agreements would govern current access when an answer needs recent information.

Citation provisions would determine how visibly users see original sources. Revenue arrangements would address whether publishers participate when their reporting supports commercial answers.

The crucial question is whether smaller outlets gain access to similar markets. A licensing system serving only the largest publishers would leave the underlying supply problem unresolved.

Collective licensing or standardized machine-readable terms could broaden participation. However, those systems would still need transparent accounting and a way to identify which content influenced which service.

The third signal is product behavior. Microsoft and OpenAI can respond without waiting for the courts by changing how answer systems route attention.

Prominent links alone may not solve the problem. One OpenAI engineer reportedly acknowledged that users might not follow links regardless of their placement.

A better test is measurable referral quality. Publishers need readers who engage, subscribe, or recognize the source, not merely a citation beneath a complete substitute answer.

AI companies could limit summaries when a query seeks a specific protected article. They could also distinguish licensed retrieval from general model knowledge in the interface.

Publishers, meanwhile, will keep testing technical barriers and legal claims. Expect more disputes over crawler controls, paywalls, cached pages, and content supplied through search indexes.

The broader industry must decide whether original reporting is just another raw input or a service that requires continuing investment. The internal Microsoft “doom loop” warning makes that choice harder to ignore.

For developers, the case should influence data governance now. A technically accessible document is not automatically free of contractual, copyright, or reputational risk.

For enterprise buyers, provenance should become a product requirement. Ask vendors whether answers come from training memory, live retrieval, licensed databases, or customer-controlled documents.

For publishers, waiting for a final ruling carries its own cost. They need evidence showing how AI interfaces change referrals, subscriptions, and reader behavior.

For users, convenience remains real. Chatbots can combine information quickly and make difficult subjects easier to approach.

The responsible response is not to pretend that synthesis creates its own evidence. Readers should open primary sources when accuracy, payment, or accountability matters.

OpenAI web scraping has become more than a dispute about how one model acquired text. It now tests whether AI companies can depend on journalism while diverting the attention that finances it.

The court will decide the legal claims. Publishers, developers, and users will decide whether the information economy supporting those systems can survive their success.

The next time an AI answer replaces a source visit, ask one practical question: who funded the original reporting, and will that organization still be able to fund the next story?