Dwarkesh Patel Pretraining Study Finds Data Beat Model Improvements

Dwarkesh Patel published a pretraining study with Jerry Han that produced a striking result: data improvements delivered 12-fold compute efficiency gains. Model-recipe improvements delivered 3.7-fold gains under the same experimental framework. The Dwarkesh Patel pretraining study therefore attributes about 3.24 times more efficiency progress to data.

That finding challenges a familiar account of language-model progress. Public discussion often centers on architectures, optimizers, larger clusters, and new attention mechanisms. Patel and Han instead found that better corpora created the larger efficiency advantage within their tested range.

The researchers reconstructed six years of open pretraining progress, from GPT-2-era systems through OLMo-2-era recipes. They combined representative model designs with representative datasets and trained those combinations at several compute budgets. This creates a direct contest between two explanations: better machinery and better material.

The result does not establish what happened inside OpenAI, Anthropic, Google DeepMind, or Meta. Those companies disclose limited information about their current training data. It does, however, change the burden of proof for anyone treating data engineering as a secondary concern.

The Dwarkesh Patel Pretraining Study Rebuilt Six Years of Progress

Patel and Han tried to separate two variables that usually change together: the model recipe and the training corpus.

Their pretraining experiment was published on September 8, 2026. It covers representative open model recipes and public datasets released from 2019 through 2025. The authors trained combinations of those ingredients from scratch.

A model recipe includes the architecture, optimizer, initialization, learning-rate schedule, normalization method, and other training choices. A corpus is the collection of text converted into tokens for pretraining. Most public model releases change both components simultaneously, making their individual contributions difficult to isolate.

The model axis begins with the GPT-2 recipe and ends with OLMo-2. Intermediate changes include rotary position embeddings, RMSNorm, SwiGLU activations, revised normalization placement, QK normalization, and cleaner initialization methods. Each improvement changes how a model represents information or remains stable during training.

The data axis begins with OpenWebText. That 2019 corpus contained about 9 billion tokens from web pages linked by sufficiently popular Reddit posts. Later datasets used broader web crawls, stronger extraction, deduplication, and increasingly selective quality filters.

The 2025 endpoint was UltraFineWeb. Its pipeline applies more sophisticated filtering to a much larger pool of web documents. Some modern filters use trained classifiers to predict which documents will improve downstream model performance.

Patel and Han tested five nominal compute budgets. These ranged from 100 quadrillion FLOPs to 10 quintillion FLOPs, written as 1e17 through 1e19 FLOPs. FLOPs measure the arithmetic operations consumed during training.

At every budget, the researchers varied model size and token count. That process let them estimate the compute-optimal combination for each model-and-corpus pairing. Compute optimal means producing the strongest measured result within a fixed amount of training computation.

They controlled several other choices across the experiments. Every run used the GPT-2 BPE tokenizer with a 50,257-token vocabulary. Each also used a 2,048-token context length and a batch containing 262,144 tokens.

The researchers evaluated end capabilities through OLMES, a collection of ten relatively accessible benchmarks. Most use multiple-choice question answering. They chose capability evaluation because comparing cross-entropy loss across different training datasets would create an uneven test.

That choice matters. A model evaluated against the same corpus used to define its training distribution may receive an artificial advantage. Downstream tasks offer a shared target, although they introduce more statistical noise.

The team ran multiple independent seeds for the compute-scaling curves. A seed changes random initialization and data ordering, helping reveal whether a result persists across training variation. The larger 7-by-7 grid at 3.16e18 FLOPs used one seed per combination.

At the 1e19-FLOP budget, newer data produced a 12-fold compute multiplier over the 2019 data baseline. Newer model recipes produced a 3.7-fold multiplier over the GPT-2 recipe. Their ratio yielded the headline 3.24-fold advantage for data.

A compute multiplier describes how much less computation a newer ingredient needs to reach the older ingredient’s performance. It does not mean the model became twelve times more intelligent. It describes efficiency at a chosen performance level and evaluation suite.

The authors also found that data and model gains were largely independent at 3.16e18 FLOPs. An additive statistical model explained 88 percent of the variation in OLMES scores. Only about 12 percent remained for interactions, higher-order effects, or evaluation noise.

That result suggests better data did not require one special architecture to deliver its benefit. Likewise, architectural improvements generally remained useful across corpora. This independence makes the data contribution harder to dismiss as one fortunate model-and-dataset pairing.

Data Engineering Won the Efficiency Contest

Within this experiment, improving what the model read mattered more than improving the mechanism that read it.

The result fits a growing body of data-centric language-model research. Modern corpora are not simply larger piles of internet text. Their builders extract cleaner content, remove duplication, balance sources, filter low-value documents, and control contamination.

Deduplication removes exact or near-identical material. Without it, frequently copied pages can dominate training and reduce the effective diversity of the corpus. Extraction separates meaningful text from menus, advertisements, boilerplate, and corrupted markup.

Filtering then decides which documents deserve limited training computation. Early filters relied heavily on rules involving language, length, repetition, and punctuation. Newer pipelines increasingly apply classifiers trained against examples of useful and unhelpful text.

DataComp-LM made this approach measurable through controlled dataset competitions. Its researchers released a 240-trillion-token Common Crawl pool and fixed the training recipe for participants. Teams could then compete through filtering and data-mixing strategies rather than changing the model.

The DataComp-LM results found that model-based filtering was central to its strongest baseline. Its 7-billion-parameter model reached 64 percent five-shot accuracy on MMLU after training on 2.6 trillion tokens.

That model performed comparably with larger open models on the reported benchmarks. The authors also reported a 6.6 percentage-point MMLU advantage over MAP-Neo while using 40 percent less training compute. These are separate experiments, but their direction supports Patel and Han’s result.

DataComp-LM also revealed that filter design can matter more than expected. At one tested scale, changing the filtering model moved five-shot MMLU accuracy from 35 percent to 44 percent. The underlying raw web pool remained available to every approach.

Surprisingly, the best tested filter used a relatively simple bigram classifier with carefully selected positive and negative examples. Human judgments about document quality offered limited value in those experiments. What looks polished to a reader does not always create better learning signal.

FineWeb research reached a related conclusion. Its creators compared extraction, language filtering, deduplication, and quality-selection choices across large web corpora. The work treated dataset creation as an empirical engineering discipline rather than a cleanup step.

The FineWeb dataset also produced an education-focused subset selected by a trained classifier. That strategy emphasized pages resembling strong educational material. Such filters aim to increase useful reasoning and factual density per token.

This history explains the Dwarkesh Patel data findings. OpenWebText already filtered the web, but it used Reddit popularity as a broad quality signal. Later pipelines tested more direct signals against actual model performance.

The difference is similar to replacing a general recommendation list with a measured curriculum. Both contain readable material. Only the second optimizes what the learner gains from each hour.

This pretraining data efficiency matters because every token consumes computation. Low-value repetition, broken text, and irrelevant boilerplate compete against useful examples within a fixed budget. Better selection increases the amount of learning produced by the same accelerator time.

The economic consequence extends beyond one training run. A dataset pipeline can support repeated experiments, model generations, and specialized variants. Improvements to that pipeline may compound across an organization’s entire research program.

Data also shapes which capabilities emerge. A corpus rich in code will not behave like one dominated by conversational prose. Scientific papers, legal text, multilingual documents, and synthetic reasoning traces each push learning in different directions.

That creates pressure for frontier laboratories. They cannot assume another hardware order will compensate for weak corpus design. They need stronger systems for provenance, licensing, extraction, filtering, mixture design, evaluation, and continual refresh.

The pressure also reaches smaller model developers. Teams with limited accelerator access cannot win a direct spending contest. They can still improve the information density of every training batch and target their corpus toward specific capabilities.

Yet the study does not say every narrow dataset is better. The Pile performed worse than OpenWebText on OLMES in the reported experiments. The Pile includes diverse material from sources such as papers, code, patents, and legal documents.

OLMES mostly evaluates English web-style multiple-choice tasks. Specialized material may therefore receive little credit, even when it supports valuable capabilities elsewhere. Data quality has no universal definition apart from a target.

Better Models Still Made Bigger Training Runs Possible

The 12-fold data result measures efficiency, but model research has also expanded the scale at which useful training remains possible.

Patel and Han explicitly reject the simplest interpretation of their headline. Their experiment does not show that six years of model research contributed little. It measures the computation required to reach certain downstream scores at relatively small scales.

Many model innovations serve a different purpose. They keep training stable as parameter counts, context windows, data volumes, and accelerator clusters grow. An improvement that prevents a massive run from collapsing may not dominate a small-scale efficiency chart.

A larger training cluster creates numerous failure points. Gradients can explode or vanish. Communication bandwidth can become a bottleneck. Memory pressure can restrict sequence length, batch size, or optimizer state.

Normalization changes can improve stability. Better initialization can prevent early divergence. Kernel-level work can keep accelerators occupied instead of waiting for memory movement.

FlashAttention provides one important historical example. It reorganizes exact attention computation around memory access, reducing expensive transfers between different memory levels. That can expand feasible sequence lengths without changing the basic attention result.

Mixture-of-experts models address another scaling constraint. They activate only part of the network for each token, allowing total parameter capacity to grow without proportional computation per token. Such designs complicate routing and distributed training.

Grouped-query attention can reduce the memory required for cached keys and values during inference. That matters greatly when a model serves millions of prompts. Patel and Han note that this kind of inference improvement does not appear in their pretraining compute multiplier.

Tokenizer improvements also remain outside the experiment. A tokenizer determines how text becomes model-readable units. More efficient tokenization can reduce sequence length or improve representation across languages and domains.

The central opposition is therefore data efficiency versus model-enabled scale, not data workers versus model researchers. Better corpora help a system learn more from a fixed budget. Better model engineering lets laboratories deploy larger budgets without instability or prohibitive overhead.

Google DeepMind’s Chinchilla work illustrates how these variables meet. Earlier scaling practice often enlarged parameter counts without increasing training data proportionally. The resulting models consumed substantial inference resources while remaining undertrained.

DeepMind trained more than 400 models to estimate the optimal allocation between parameters and tokens. Its compute-optimal research concluded that model size and token count should rise together under its tested conditions.

Chinchilla used 70 billion parameters and 1.4 trillion training tokens. Gopher used 280 billion parameters under a comparable compute budget. The smaller, more extensively trained Chinchilla outperformed Gopher across most reported evaluations.

That result shifted attention from parameter count toward training duration and dataset volume. It did not erase architecture research. It showed that model size alone offered a poor description of how effectively computation was being used.

Patel and Han extend that conversation by separating corpus vintage from recipe vintage. Chinchilla asked how many parameters and tokens to purchase with a compute budget. The new study asks whether progress came more from the container or its contents.

The answer, at this scale, favors the contents. Yet the container determines how much can be carried. Patel and Han compare small models with sailboats and frontier systems with container ships.

A carefully chosen load matters greatly on a sailboat because capacity is scarce. A container ship can carry vastly more material and survive rougher conditions. Its value appears through scale and reliability rather than speed alone.

For AI laboratories, both capabilities remain essential. A clean corpus cannot train without stable software and hardware. An efficient architecture still wastes compute when its training batches contain duplicated or low-value material.

The study’s useful reversal is narrower. Architecture should no longer receive automatic credit for every generational performance increase. When both data and recipes improve, controlled ablations must determine which change caused the observed gain.

What the 12-Fold Finding Does Not Establish

The strongest conclusion is also tightly bounded: data dominated one open, small-scale pretraining experiment on one capability suite.

The authors repeatedly emphasize those limits. Their maximum budget was 1e19 FLOPs, far below modern frontier training runs. Scale-dependent techniques may deliver little visible benefit in small models and become essential only much later.

Small models have limited capacity. Low-value tokens can crowd out important material because the model cannot absorb everything. Aggressive selection may therefore help small models more than systems with much larger representational capacity.

Large models may benefit from broad coverage, even when the average document quality falls. More capacity can separate weak signals from noise. Excessive filtering might remove rare facts, unusual styles, minority languages, or niche technical material.

Aggressive filtering also shrinks the unique corpus. Developers must then repeat the surviving material across more epochs, meaning more passes through the same examples. Repetition can produce diminishing returns or overfitting.

Research on data-constrained scaling found that repeated data remains useful but becomes less valuable with additional passes. That creates a tradeoff between average quality, total diversity, and repetition.

Patel and Han raise an even larger uncertainty. Frontier models are often trained beyond classic Chinchilla-optimal token allocations because smaller models can reduce later inference costs. A curated dataset may require extensive repetition under that strategy.

The OLMES evaluation adds another limitation. Its ten tasks are relatively accessible and largely based on multiple-choice question answering. Different evaluations could change the ranking of model recipes, datasets, or both.

A coding benchmark might reward repositories and technical documentation. A scientific benchmark could favor papers and specialized explanations. Multilingual testing would value linguistic coverage that an English-centered suite may overlook.

The authors observed anomalies consistent with this measurement problem. NeoX performed worse than GPT-2 at 1e19 FLOPs on OLMES, despite beating it at lower tested budgets. NeoX also performed better on held-out FineWeb-Edu loss.

The Pile’s broad mixture also underperformed OpenWebText on OLMES. That does not establish that diverse sources are generally inefficient. It may show that the evaluation offers limited credit for their specialized knowledge.

Some reported multipliers required extrapolation. The NeoX and Pile curves did not always reach the necessary reference performance within the measured range. Extending a fitted curve beyond observed results introduces additional uncertainty.

Hyperparameter tuning creates another source of error. End capabilities can change with the peak learning rate, batch size, initialization, and schedule. The researchers swept learning rates at selected anchor points but could not test every possible configuration.

They believe uncertainty for model-recipe multipliers exceeds the displayed statistical error bars. Those bars capture variation from their estimation pipeline. They do not capture every consequence of limited hyperparameter optimization.

The reported yearly efficiency rates reinforce the need for caution. Model changes yielded a 1.24-fold annual multiplier, while data changes yielded 1.51-fold. Jointly, the measured annual gain was 1.57-fold.

Those figures fall below an earlier estimate of roughly threefold annual software efficiency improvement. Patel and Han offer several explanations, including scale-dependent gains, excluded inference optimizations, tokenizer changes, and representative-recipe choices.

The study also excludes major sources of modern capability progress. It focuses on pretraining, the phase when a model learns general patterns from broad corpora. It does not decompose reinforcement learning, post-training, tool use, or test-time computation.

Patel and Han note that reinforcement learning drove many visible gains during the two years before publication. A model can become much better at reasoning or instruction following without changing its base pretraining efficiency.

Synthetic data remains another large gap. Frontier laboratories use models to generate explanations, code, reasoning traces, and task examples. The experiment examined public corpora that mostly curate subsets of Common Crawl.

The authors did not test whether synthetic generation can expand a small supply of high-quality information. They also did not compare synthetic expansion with repeating the original documents across more epochs.

Finally, the study does not reproduce closed frontier pipelines. Major laboratories rarely disclose complete dataset compositions, filtering classifiers, mixture weights, or contamination controls. Their internal systems may differ sharply from public recipes.

The appropriate reading is cautious but meaningful. The experiment provides evidence that corpus progress deserves more causal credit. It does not provide a universal 12-fold law for every model, scale, benchmark, or laboratory.

Data Quality Is Becoming a First-Class Scaling Variable

The industry increasingly treats data quality as a measurable scaling input rather than an informal property of a corpus.

Traditional scaling laws connect performance with model size, training tokens, and compute. They often assume that one token is comparable with another. Real datasets violate that assumption through noise, duplication, domain differences, and uneven information density.

Recent research has started modeling quality directly. An ICLR 2026 paper introduced a dimensionless quality parameter and extended Chinchilla-style scaling laws. Its goal was to predict loss from data volume, model size, and data quality together.

The quality-aware model uses proxies based on corruption and dataset deficiency. Controlled experiments found that higher-quality data could reduce the model size and computation required for a target loss.

That approach does not validate Patel and Han’s precise multiplier. It supports the broader mechanism behind their result. Two corpora with equal token counts can provide very different effective amounts of learning.

This changes how teams should describe scaling. Token count remains necessary, but it cannot reveal whether the tokens are unique, relevant, accurate, or educational. Compute numbers alone cannot reveal whether training batches carry useful signal.

The same principle matters below the frontier. A company adapting a model to its own domain must decide which documents represent trusted knowledge. Feeding every available file into a pipeline can amplify duplication, outdated policies, and conflicting instructions.

Good information infrastructure preserves provenance and makes conflicting material visible. It also helps people find the sources behind model-generated answers. A searchable knowledge base applies similar discipline at organizational scale.

Developers should therefore separate three questions. First, does the corpus contain the knowledge needed for the target tasks? Second, can the pipeline extract and represent that knowledge cleanly? Third, does the evaluation reward the intended capability?

Enterprise buyers should ask related questions when assessing specialized models. A vendor’s parameter count says little about domain coverage or source freshness. Benchmark scores can also conceal narrow data choices that fail under real workflows.

Knowledge workers face a smaller version of the same issue. More saved material does not automatically produce better retrieval or reasoning. The value depends on relevance, organization, freshness, and the system’s ability to connect evidence.

For frontier laboratories, better data pipelines may become proprietary assets comparable with model code. Public architectures are often replicated quickly. High-quality mixtures require continuing access, legal review, experiments, and feedback from training runs.

That creates a less visible competitive divide. Well-funded laboratories can pay for licensed collections, specialist-produced examples, private repositories, and synthetic environments. Smaller teams depend more heavily on public web corpora and reproducible filters.

The shift also raises governance questions. Stronger filtering requires a definition of usefulness. That definition can suppress dialects, minority perspectives, controversial material, or knowledge that differs from a classifier’s preferred examples.

Optimization can also target benchmarks too directly. A filter trained against evaluation performance may select documents that resemble known tests. Without contamination controls, apparent data quality can become hidden benchmark exposure.

Dataset engineering therefore needs the same scrutiny applied to model design. Researchers need ablations, documented sources, held-out evaluations, contamination testing, and performance across multiple domains. A single aggregate score cannot capture all tradeoffs.

Patel and Han’s work helps by making data vintage an independent experimental axis. Future studies can expand that design across languages, domains, model sizes, and synthetic mixtures. They can also measure factuality, coding, reasoning, safety, and long-context use.

The most important operational change is simple. Teams should stop treating training data as a static input selected before the serious work begins. Corpus design is part of the model, even when it lives outside the architecture diagram.

Three Signals Will Test the Data-First Thesis

The next evidence must show whether the result survives frontier scale, synthetic expansion, and broader capability testing.

The first signal is a larger replication. Patel and Han reached 1e19 FLOPs, and they describe their experiment as extremely small. A follow-up must extend the same crossed design across larger models and substantially higher compute budgets.

That replication should keep the core separation intact. Researchers need representative model recipes on multiple corpus vintages, not a comparison where every variable changes together. Multiple seeds and comparable hyperparameter effort will also matter.

If the data multiplier remains larger at higher scale, the central thesis becomes much stronger. Frontier laboratories would have firmer evidence that curation remains a dominant efficiency lever. A shrinking multiplier would support the authors’ container-ship explanation.

The second signal is a controlled synthetic-data study. Researchers should begin with a limited, high-quality corpus and compare two strategies. One repeats the original material, while the other expands it through carefully generated synthetic examples.

The evaluation must check more than benchmark accuracy. It should measure novelty, factual reliability, diversity, memorization, and performance outside the generator’s strongest domains. Otherwise, synthetic expansion may only reproduce familiar patterns.

A strong synthetic advantage would weaken the idea of an approaching fixed-data wall. Laboratories could transform limited human-created material into much larger curricula. Weak results would increase the strategic value of licensed and expert-generated data.

The third signal is evaluation breadth. Future replications should test coding, mathematics, multilingual ability, factual retrieval, scientific reasoning, and long-context tasks. They should also report results separately rather than hiding them inside one average.

Consistent data advantages across those categories would strengthen the Dwarkesh Patel pretraining study. Large differences between categories would show that corpus quality depends strongly on the chosen target. That outcome would favor specialized mixtures over one universal ranking.

Readers should also watch disclosure practices. If open-model builders publish better dataset cards, mixture details, and filtering experiments, independent researchers can test the claim more rigorously. Continued secrecy will preserve a large gap between public evidence and frontier reality.

For developers, the immediate lesson is not to abandon architecture work. It is to measure the data pipeline with equal seriousness. Track duplication, source contribution, mixture weights, freshness, and performance changes from every filtering decision.

For enterprise teams, the lesson is to ask where model knowledge comes from and how it is maintained. A larger model cannot guarantee current, relevant, or traceable answers. Better information inputs often matter before another layer of model complexity.

For knowledge workers, the practical question is similar: what information deserves limited attention? Better selection can improve both human and machine reasoning. Collecting everything without structure merely relocates the filtering problem.

The Dwarkesh Patel data findings make one conclusion difficult to ignore. Public narratives have overemphasized visible model changes because architectures are easier to name than corpus pipelines. Controlled evidence now gives the quieter variable a much larger role.

The next three tests will decide how far that role extends. Look for a higher-compute replication, a synthetic-versus-repetition comparison, and diverse capability evaluations. Together, they can confirm or narrow the claim that pretraining progress mostly came from data.

Until then, treat the 12-fold result as a strong experimental signal, not a universal constant. Audit the information feeding your systems, test changes against real tasks, and preserve the sources behind each conclusion. Better models still matter, but their performance begins with what they are allowed to learn.