Google MIT Legacy Meets the Startups Chasing the Next Big Thing in LLMs

Google MIT research now faces its sharpest architectural challenge since eight Google researchers introduced the transformer nine years ago. Startups are questioning whether the mechanism behind nearly every leading LLM should remain the industry’s default.

The conflict is not about building another chatbot. It concerns the computational structure beneath ChatGPT, Claude, Gemini, and most open models. Subquadratic, Liquid AI, Cartesia, and AI21 Labs are exploring systems that process long sequences without performing dense attention at every layer.

Their proposals range from sparse attention to state-space models and convolution-heavy hybrids. Each route promises lower memory use, faster processing, or longer usable context. None has yet displaced the transformer across the full range of tasks that matters.

That qualification defines the race. The next important LLM architecture will not win by producing an impressive laboratory chart. It must preserve reasoning and recall while working inside the software, chips, and deployment systems companies already use.

The Transformer’s Success Created Its Own Bottleneck

The architecture that unlocked modern generative AI becomes more expensive as the amount of input grows.

Google researchers introduced the transformer in the 2017 paper Attention Is All You Need. The paper replaced recurrent processing with attention, a mechanism that lets a model calculate which parts of an input deserve emphasis.

Attention made training highly parallel. It also helped models connect information across distant positions in a document. Those advantages supported the scaling path that eventually produced today’s large language models.

The standard form of self-attention compares each token with every other token in a sequence. A token is a small unit of text, such as a word fragment. Doubling the sequence length can therefore produce roughly four times as many pairwise comparisons within the attention operation.

Engineers describe this behavior as quadratic scaling. It does not mean the entire model becomes exactly four times more expensive under every deployment condition. Hardware, caching, batching, and implementation choices also affect the result.

The underlying relationship still creates pressure. A chatbot answering a short question can tolerate dense attention more easily than an agent reading several repositories, months of messages, and a long action history.

Context windows have expanded, but a model’s advertised limit does not guarantee affordable or accurate use near that limit. Teams must consider the time required to process the prompt, memory consumption, inference expense, and whether the model finds the right evidence.

Retrieval-augmented generation, or RAG, reduces the burden by selecting relevant passages before asking a model to answer. It remains useful, especially when an organization maintains a large AI knowledge base.

Retrieval also introduces a separate failure point. If the search layer omits a crucial passage, the model never sees it. A system that can inspect far more source material directly could reduce that risk for code analysis, legal review, research, and persistent agents.

That prospect explains why architecture research has regained commercial urgency. More compute can extend the transformer, but increasingly long prompts expose a structural cost that better chips alone do not erase.

The Google MIT connection matters here because Google supplied the foundational paper, while MIT-linked researchers later helped build companies pursuing alternatives. The contest is less a dispute between two institutions than a transition from academic insight to competing production systems.

Startups see an opening between what current models advertise and what customers can practically deploy. If they can make long context cheaper without losing quality, they can pressure model labs and infrastructure providers at the same time.

Google MIT Research Is Spawning More Than One Alternative

No single post-transformer design has emerged, so startups are dividing the problem into different technical bets.

Subquadratic is taking the most direct approach. The Miami-based startup says its SubQ model uses Subquadratic Sparse Attention, or SSA, to avoid examining every possible token relationship.

Sparse attention restricts the set of token pairs a model evaluates. The hard part is choosing which connections matter before performing the expensive work that dense attention would require.

Subquadratic says its mechanism scales linearly with context length and supports a research context of 12 million tokens. Its published examples include complete software repositories and long-running agent histories.

Those are company claims, not a settled industry result. The firm’s technical explanation says SSA identifies important relationships while skipping most possible comparisons.

Liquid AI is making a different bet. Founded by researchers with roots at MIT, the company develops Liquid Foundation Models for devices and other resource-constrained environments.

Its LFM2 architecture does not eliminate attention entirely. It combines gated short convolutions with a smaller number of grouped-query attention blocks.

A convolution processes local patterns through reusable filters. Grouped-query attention reduces memory demands by letting multiple query heads share key and value representations.

Liquid says LFM2 was selected through hardware-aware architecture search. That means the team evaluated designs against actual memory and speed characteristics, rather than optimizing only an abstract operation count.

The company’s LFM2 design contains more convolution blocks than attention blocks. This arrangement targets fast execution on CPUs and embedded processors.

Cartesia is pursuing state-space models, or SSMs, which summarize a sequence through a changing internal state. Instead of retaining an explicit relationship between every token pair, an SSM updates a compact representation as new input arrives.

Cartesia’s founders helped develop Mamba, a selective state-space architecture. Selectivity lets the model decide what information to preserve or forget based on the current input.

The original Mamba research reported linear scaling with sequence length and stronger throughput than comparable transformers in its experiments. It also demonstrated applications across language, audio, and genomics.

Cartesia has focused much of its commercial work on voice. Speech is a natural proving ground for efficient sequence processing because interactive audio systems must respond quickly while tracking an evolving conversation.

AI21 Labs offers a fourth route. Its Jamba architecture interleaves Mamba layers with transformer attention and mixture-of-experts components.

A mixture-of-experts model activates only part of its total network for each token. This increases total capacity without using every parameter during every calculation.

Jamba’s design reflects a pragmatic conclusion: efficient sequence processing and dense attention each solve different problems. AI21 retains some transformer layers for global recall while using Mamba layers to reduce memory pressure.

These approaches turn the Google MIT story into a branching architecture race. Subquadratic is trying to preserve attention while making it sparse. Liquid AI combines attention with convolutions. Cartesia emphasizes state-space processing, while AI21 mixes state-space and transformer layers.

The shared target is efficiency. Their disagreements concern what an LLM can safely discard, compress, or process locally without damaging the relationships that produce useful answers.

Sparse Attention Makes the Boldest Claim

Subquadratic is challenging dense attention directly, but its evidence must catch up with the scale of its promise.

Subquadratic emerged publicly in May 2026 with an unusually large claim. It said SubQ could process context extending to 12 million tokens while using far less attention compute than transformer models.

That capacity would cover several complete repositories or a large collection of contracts in one prompt. It could also let an agent retain a much longer operational record without repeatedly summarizing its past.

The company argues that dense attention wastes computation because most possible token relationships contribute little to the result. SSA attempts to find the small subset that matters.

This is harder than simply dropping random connections. A relevant clue can appear millions of tokens away, and its importance might only become clear after the model reads another passage.

A sparse architecture therefore needs an effective routing mechanism. If it selects the wrong connections, computational savings arrive at the expense of recall or reasoning.

Subquadratic says its architecture performs content-dependent selection. The model chooses attention patterns from the input instead of relying only on fixed local windows.

Its initial launch drew skepticism because public access and technical detail were limited. The company later released additional benchmark information and cited evaluations conducted by Appen.

According to the original MIT analysis, independent researcher Will Depue noted that researchers have attempted many ways to overcome dense attention’s cost.

That history matters. Sparse attention, linear attention, recurrent networks, long convolutions, and compressed memory have all shown benefits. Many also struggle when tasks require precise retrieval from distant positions.

Another concern is SubQ’s lineage. MIT Technology Review reported that the company reused weights from a version of Alibaba’s Qwen model to help initialize SubQ.

Weights are the learned values that determine a neural network’s behavior. Reusing them can accelerate development, but it complicates claims about what the new architecture learned independently.

A converted or bootstrapped model might inherit capabilities from transformer training. That does not invalidate the new mechanism, but it leaves open whether the architecture can train efficiently from the beginning.

The distinction matters commercially. A method that transforms existing models could still become valuable infrastructure. A method that trains competitive foundation models from scratch would challenge the economics of the entire market.

Developers also need more than a context-limit demonstration. They need stable APIs, predictable latency, tool-use accuracy, structured output, security controls, and integration with common inference frameworks.

Long context presents its own evaluation problem. A model can technically accept millions of tokens while still missing relevant facts or combining them incorrectly.

Simple needle tests place a distinctive fact inside a long input and ask the model to retrieve it. They measure basic recall, but they do not capture tasks requiring evidence spread across many files.

Repository maintenance provides a better stress test. A model must identify interfaces, follow dependencies, understand tests, and make a change without breaking distant components.

Legal and financial analysis creates similar demands. The correct conclusion can depend on definitions, exceptions, amendments, and dates scattered across many documents.

Subquadratic must show that its sparse routing preserves these relationships under adversarial and ordinary workloads. Independent testing across multiple domains would carry more weight than one benchmark provider.

Its challenge is clear. The firm has offered a mechanism that directly attacks quadratic attention, but widespread adoption requires repeatable evidence and accessible models.

Hybrid Models Are Betting Against a Clean Break

Liquid AI and AI21 treat attention as an expensive ingredient to ration, not a defective component to remove.

Hybrid architectures reflect an engineering judgment that transformers remain unusually good at flexible, content-based recall. Their weakness is the cost of applying that capability everywhere.

Liquid AI’s LFM2 assigns much of the sequence processing to short convolutions. These blocks examine local patterns efficiently and avoid the expanding cache associated with dense attention.

The model still includes attention blocks for operations that benefit from broader comparisons. This division resembles a system that uses inexpensive local processing most of the time, then invokes global analysis selectively.

Liquid’s hardware-aware method also highlights a weakness in theoretical comparisons. An operation with better asymptotic scaling does not automatically run faster on a phone, laptop, GPU, or accelerator.

Modern AI hardware is heavily optimized for transformer workloads. Dense matrix multiplication benefits from mature kernels, compilers, and serving systems.

A new architecture can perform fewer theoretical operations yet run slowly if software cannot schedule them efficiently. Memory movement, cache behavior, and batch size can matter as much as the equation describing the layer.

Liquid uses architecture search to evaluate these constraints directly. Its 2026 LFM2-24B-A2B model contains 24 billion total parameters, with about 2 billion active for each token, according to the company.

The model combines a sparse mixture-of-experts system with LFM2’s convolution-heavy backbone. Liquid says it can fit within 32 gigabytes of memory, making local deployment on some higher-end consumer systems possible.

These are vendor-reported characteristics. Independent application testing must determine how their quality compares with larger cloud models.

The route remains commercially coherent. Many organizations do not need the largest available model for every task. They need acceptable quality with controllable latency, privacy, and infrastructure use.

An on-device model can support private document classification, command parsing, summarization, or voice interfaces without sending every input to a remote service. It may also keep working when network access is limited.

AI21’s Jamba reaches a similar compromise from the state-space direction. The architecture alternates Mamba and transformer components instead of committing exclusively to either.

AI21 says pure state-space models can struggle with recall. Jamba preserves occasional attention layers to restore access to detailed information across the sequence.

The company’s Jamba architecture also uses mixture-of-experts layers to separate total model capacity from active computation.

This approach gives up the clean story that attention is unnecessary. It gains a more conservative path into production because developers can retain known transformer behavior where it matters most.

Hybrid systems are therefore the strongest near-term pressure on conventional models. They do not need to prove that the transformer was a mistake.

They only need to show that every layer does not require the same expensive operation. If that claim holds, model builders can reduce attention gradually while preserving established training and deployment practices.

This path resembles earlier computing transitions. New designs often enter as specialized components before replacing a general default.

Graphics processors first accelerated narrow workloads before becoming central to AI. Solid-state storage coexisted with hard drives before taking over many performance-sensitive roles.

Attention alternatives can follow the same pattern. Convolutions or state-space layers may first handle local processing, streaming input, and long histories. Attention can remain responsible for difficult global relationships.

The resulting architecture may not carry one memorable label. It could be a configurable mixture shaped by the target hardware and workload.

That possibility makes the next big thing in LLMs less dramatic than another “Attention Is All You Need” moment. It may arrive through a steady reduction in where attention is used.

The Real Contest Is Quality per Unit of Compute

An alternative architecture wins only when its savings survive realistic tests of accuracy, latency, and deployment complexity.

Model comparisons often isolate one dimension. A paper might report lower computational complexity, a longer context window, or higher token throughput.

Enterprise buyers face a combined equation. They care how much useful work a system completes within a latency, memory, and reliability budget.

A fast model that misses contractual exceptions is not efficient. A model with excellent recall but unpredictable response times may also be unsuitable for interactive software.

The primary opponent in this race is therefore not one startup or one incumbent. It is the transformer’s established quality-to-infrastructure balance.

Google, OpenAI, Anthropic, Meta, and other labs can continue improving transformer systems. They can use optimized attention kernels, better caching, quantization, distillation, and faster accelerators.

They can also adopt successful alternative components. A startup that validates a new mechanism might influence the incumbents without replacing their model families.

Compatibility adds another defensive advantage. Transformer inference works across a mature collection of GPU kernels, distributed training systems, optimization libraries, and deployment platforms.

Novel recurrent or sparse operations can require custom kernels and caching logic. Developers may need new monitoring tools because performance failures differ from familiar transformer behavior.

Training stability remains another test. Researchers understand how transformers behave across many model sizes and datasets. Alternative architectures have less evidence at the largest scales.

A model can look efficient at a few billion parameters and encounter optimization problems when scaled much further. It can also lose its advantage if compensating for weaker recall requires more layers or training data.

Benchmark selection can conceal these differences. General knowledge tests may say little about multi-file reasoning. Long-context retrieval tests may not predict conversational quality or tool use.

The strongest evaluations will connect architecture to actual work. Relevant tasks include modifying large repositories, reconciling conflicting reports, tracking facts through extended conversations, and analyzing time-dependent records.

Tests should measure answer quality alongside first-token latency, total completion time, peak memory, energy use, and hardware utilization.

They should also disclose whether comparisons use equivalent parameter counts, training data, context lengths, precision settings, and serving conditions.

Without those controls, a claimed architectural advantage may reflect different engineering choices. Even accurate numbers can create a misleading impression when the test conditions differ.

The Google MIT keyword captures a genuine lineage but not a clean institutional rivalry. Google researchers introduced the transformer, while MIT researchers contributed to several alternative lines of work.

The important contest now runs between architectural maturity and architectural efficiency. Transformers offer a known route to high capability. Challengers promise to reach similar capability with less computational waste.

Neither side has a permanent advantage. Incumbents can copy validated ideas, while startups can optimize their systems without supporting years of legacy infrastructure.

Customers should avoid treating advertised context length as the decisive metric. The better question is how much relevant information a model uses accurately within a fixed budget.

For developers, this means testing models against representative data instead of relying on public leaderboards alone. A legal team, software company, and voice platform will expose different architectural weaknesses.

For knowledge workers, more efficient long context could reduce repeated uploading and summarization. It could also help assistants maintain continuity across larger bodies of work.

Those benefits depend on privacy and access controls. A model capable of reading millions of tokens magnifies the consequences of supplying the wrong documents or retaining information too broadly.

Architecture cannot solve those governance problems. It can only change the amount of information a system can process and the resources required to do so.

What to Watch as the Post-Transformer Race Develops

Three signals will show whether the startups have found a successor or merely useful additions to the transformer stack.

The first signal is independent access to SubQ. Outside researchers need enough availability to reproduce long-context results, test difficult retrieval patterns, and examine performance under changing workloads.

A broad release would strengthen Subquadratic’s argument if users can verify its quality and linear-scaling claims. Continued limited access would preserve uncertainty around the architecture.

The most useful tests will go beyond inserting one fact into a long prompt. They should require the model to connect evidence across files, reject misleading context, and maintain accuracy as the input expands.

Researchers should also compare processing time and memory use on disclosed hardware. Otherwise, claims about computational savings remain difficult to translate into deployment decisions.

The second signal is wider adoption of hybrid layers. Liquid AI and AI21 do not require the industry to abandon attention, so their designs can spread incrementally.

Watch whether model developers begin disclosing a lower ratio of attention layers or adopt convolution and state-space components in new production models.

That shift would support the article’s central judgment. The first post-transformer era is likely to ration attention before eliminating it.

Hardware and software support will reveal the same trend. Native kernels, inference engines, and monitoring tools for hybrid models would make architecture choices less risky.

The third signal is performance in persistent agents. Agents create a demanding sequence problem because they accumulate instructions, tool results, code changes, and failed attempts over time.

Current systems often compress or discard that history. Summaries save context, but they can remove details the agent later needs.

Sparse or state-space models should show a clear advantage here if their memory claims hold. An agent should complete longer projects without losing constraints, repeating work, or allowing its context cost to grow uncontrollably.

This is also where the difference between nominal and usable context becomes visible. A model does not need to recall every prior token equally. It must preserve the right commitments and evidence.

The transformer remains the standard because it combines flexible attention, scalable training, and mature infrastructure. Nine years of refinement have made that combination difficult to beat.

Still, the industry’s workloads are changing. Longer repositories, multimodal records, continuous voice, and persistent agents all intensify the weaknesses of dense attention.

The startups chasing the next big thing in LLMs do not yet have a shared answer. Their disagreement is useful because it exposes the actual design space.

Subquadratic asks whether sparse routing can preserve global reasoning. Cartesia tests whether state can replace explicit token relationships. Liquid AI and AI21 ask how little attention a capable model really needs.

The Google MIT legacy will not disappear if one of these systems succeeds. The transformer established attention as the central mechanism of modern AI, and its descendants will retain many of its ideas.

The more likely change is architectural pluralism. Models will combine different sequence mechanisms based on the device, workload, and required memory.

Developers should watch verified quality per unit of compute, not declarations about the transformer’s death. Enterprise buyers should test complete workflows, including retrieval, tool use, latency, and access control.

Which signal would change your model strategy first: independently verified sparse attention, broader hybrid-model support, or an agent that can use months of history without losing the details?