OpenAI Habitat Storage Passed One Billion Users, but Python Had Reached Its Limit

OpenAI Habitat storage now supports products used by more than one billion people weekly, while processing over 70 million requests every second. That scale forced OpenAI to turn an internal Python library into a globally distributed service, then replace most of its Python implementation with Rust.

The headline number is only part of the story. Habitat serves more than 500 petabytes across almost 40 geographic regions, according to OpenAI's storage architecture. Yet the system began in 2023 as a small library sitting between product code and Azure Cosmos DB.

OpenAI did not reach its current scale by selecting an ideal architecture in advance. It accepted technical debt, constrained what developers could ask the storage layer to do, and repeatedly optimized bottlenecks already in production. That path echoes Meta's earlier work on TAO, but OpenAI faced a compressed timeline driven by three consecutive years of more than tenfold growth.

OpenAI Habitat Storage Became a Global Control Layer

The decisive change was organizational as much as technical: OpenAI moved storage logic out of dozens of applications and placed it behind one managed service.

Habitat first launched for GPTs at OpenAI's DevDay in 2023. Its initial job was to give product engineers a small set of storage operations without exposing the details of the underlying database.

The library determined where data belonged and whether a request was authorized. It also handled schema lookup, routing, encryption, serialization, request shaping, caching, and connection pooling.

That abstraction spread inside OpenAI without a broad mandate. Product teams could use Habitat instead of integrating directly with PostgreSQL or Azure Cosmos DB. They could also contribute shared features such as compression and client-side caching.

The approach worked while Habitat remained small. It became harder to manage as more OpenAI services adopted different library versions and product requirements diverged.

By mid-2025, changing the storage layer required coordinated deployments across dozens of services. A new routing rule could take days to distribute because every participating application needed an updated client.

OpenAI described one migration designed to reduce the damage from a regional outage. Engineers wanted to move critical datasets across several regional Cosmos DB accounts, which required new routing logic and a controlled rollout.

The team first added the routing behavior behind a feature flag. It then coordinated upgrades, added request shadowing to test the proposed routes, and distributed a correction after identifying a problem.

When the change was finally ready, an unrelated application rollback restored an older, faulty Habitat client. The rollback reintroduced the exact outage risk that the migration was meant to reduce.

This incident exposed the weakness of putting infrastructure policy inside application libraries. A library simplifies development, but every deployed copy becomes another version that the platform team must track.

OpenAI responded by turning Habitat into a standalone service. Product applications continued using a small client, but routing, access control, monitoring, and storage decisions moved behind a centrally operated interface.

The service created one deployment point for platform changes. OpenAI could update routing or reliability controls once instead of waiting for every product team to release new code.

Centralization also created a security checkpoint. Habitat could enforce authorization, record audit events, and restrict direct access to databases from one layer.

This matters because storage requests now come from more than human-facing applications. OpenAI says Habitat must also protect resources from unauthorized internal services and agent-driven activity.

The resulting system sits between ChatGPT, Codex, the API platform, internal services, and several storage technologies. Azure Cosmos DB remains a central online store, while Valkey, blob storage, Kafka, Rockset, and change-data-capture services support specialized paths.

Habitat therefore became more than a database wrapper. It became a policy and traffic layer through which OpenAI could manage a fast-changing collection of products.

That control solved the deployment problem, but it introduced a new cost. Every database request now had to cross a network service implemented in Python.

Why One Billion Weekly Users Changed the Engineering Priorities

OpenAI's growth compressed years of infrastructure planning into a sequence of short-term capacity decisions.

OpenAI says Habitat supports products used by over one billion people each week. The company does not provide a product-by-product breakdown, so that figure should not be interpreted as a verified count of distinct ChatGPT accounts.

The system currently handles more than 70 million requests per second and over 500 petabytes of data. It operates across almost 40 geographic regions.

Those figures differ from Habitat's earlier Python-era throughput. OpenAI says the Python implementation peaked above 20 million requests per second before the Rust migration. The larger 70 million figure describes the wider platform at the time of publication.

The distinction matters. Habitat's story is not that one Python program suddenly processed the platform's present load. It is a story of staged architectural changes across service code, proxies, caches, databases, and regional infrastructure.

OpenAI says its demand grew more than tenfold year over year for three years. Traditional capacity planning often prepares a system for a tenfold increase and expects that margin to last several years.

Habitat repeatedly consumed that margin within one year. Engineers had to extract more capacity from the existing design while preparing replacements that could arrive before the next limit.

This schedule changed how the team evaluated technical debt. Moving to a service increased CPU use, memory consumption, and network latency, but it immediately reduced deployment coordination.

OpenAI accepted those costs because operational control had become the more urgent constraint. A theoretically efficient client library was no longer efficient when every platform change required negotiations with dozens of application teams.

That trade also placed pressure on Microsoft Azure. Habitat depends heavily on Cosmos DB, whose model spreads data and throughput across logical and physical partitions.

Microsoft's Cosmos partitioning documentation explains that partition-key selection determines how evenly data and requests can spread. A poor key can concentrate traffic, create hot partitions, or force expensive cross-partition queries.

At Habitat's scale, those concerns become platform-wide risks. OpenAI cannot treat placement, partitioning, and regional capacity as implementation details belonging to a single product team.

The move to a central service gave OpenAI one place to coordinate these decisions. It also concentrated responsibility for failures in a layer used by nearly every major product.

A slow Habitat request can delay authentication, settings access, or the start of a ChatGPT conversation. A failed storage path can prevent the associated product action from completing.

One user action can trigger hundreds of database calls. The user experiences the slowest relevant call, not the average performance of the other requests.

This makes tail latency, the slowest portion of the request distribution, more important than a comfortable average. A small percentage of stalled calls can affect a much larger percentage of user interactions.

The scale also changes the meaning of minor inefficiencies. A little extra memory per worker becomes substantial when the system needs a vast worker fleet. A slightly unbalanced connection pool can direct millions of requests toward already stressed processes.

Habitat placed OpenAI in a difficult position. The company needed centralized control immediately, but the quickest implementation language carried costs that would grow with every new user.

How OpenAI Scaled Python Until the Economics Broke

Python remained viable because OpenAI tightly controlled concurrency, request behavior, and downstream connections, not because the language's overhead disappeared.

OpenAI knew that a networked Python service would add latency compared with a library running inside each application. It would also consume more CPU and memory for the same workload.

The team still delayed a rewrite. Product stability and a consistent storage interface mattered more than early resource optimization, while the system's APIs were still changing.

Habitat used asyncio, Python's framework for running many input and output operations concurrently on an event loop. Concurrency lets one task make progress while another waits for a network response.

It does not automatically provide CPU parallelism. The asyncio event loop schedules callbacks and tasks, but CPU-heavy work can still delay other ready tasks in the same process.

Habitat performed more CPU work than a simple network proxy. Its responsibilities included encryption, compression, checksums, routing, health checks, request shadowing, and hedging.

OpenAI found that downstream storage could return promptly while a Habitat coroutine waited to be scheduled again. At high utilization, event-loop delays reached hundreds of milliseconds and, in edge cases, several seconds.

The team measured this delay directly. It scheduled periodic background tasks and compared their expected execution time with the time when the event loop actually ran them.

That signal exposed pressure invisible in standard CPU or memory averages. A process could appear functional while ready requests waited behind CPU-heavy tasks.

OpenAI responded by limiting concurrent requests per process. It then expanded the number of worker processes, trading infrastructure volume for more predictable tail latency.

The strategy kept individual event loops less crowded. However, it multiplied the number of connections and increased the danger of overwhelming downstream services.

Feature-flag processing created another latency spike. Habitat used Statsig configurations that initially contained production rules from across OpenAI and refreshed every minute without randomized timing.

Each pod could run up to eight Python processes. Those workers sometimes parsed the large configuration at nearly the same moment, temporarily diverting CPU from active requests.

Profiling revealed the pattern. OpenAI reduced the configuration's scope, extended the refresh interval, and added jitter so workers would not perform the task simultaneously.

Connection reuse produced a more subtle failure. Some Habitat processes received five to ten times the average concurrency because client connections did not distribute traffic evenly.

Python's aiohttp connector reused connections in last-in, first-out order. After a traffic burst, slower servers returned their connections later, placing those connections at the front of the reuse queue.

New requests then returned to the slow processes. Those workers became even slower, which reinforced the routing imbalance.

The system had entered a metastable state, meaning it continued degrading after the original traffic burst ended. Meta documented a related metastable failure in its network infrastructure, where imbalance persisted until outside intervention changed the conditions.

OpenAI tested connection lifetime limits and confirmed that reuse behavior was contributing to the problem. It then changed the pool to first-in, first-out reuse, which broke the feedback loop.

Habitat now relies mostly on Istio and Envoy for connection pooling and load-aware balancing. Envoy also converts upstream HTTP/1 traffic into multiplexed HTTP/2 connections, letting many requests share fewer downstream connections.

This fan-in protects databases and network infrastructure from a thundering herd. That condition occurs when many workers simultaneously open connections or retry work against the same dependency.

Rate limits and circuit breakers at the proxy layer provide additional protection. Central enforcement is more effective than expecting every Python process to react independently.

These measures pushed Python past 20 million requests per second. They also made the platform one of OpenAI's largest internal consumers of computing resources.

Habitat became OpenAI's second-largest service by CPU core count. Its Envoy footprint ranked fourth within the company, according to the engineering team.

At that point, scaling out Python workers was no longer a neutral operational choice. The resource overhead competed with other services for capacity while increasing the number of connections the platform needed to manage.

The language had served its strategic purpose. OpenAI had stabilized the interface and centralized storage policy, but continued growth made a more efficient implementation unavoidable.

The Real Mechanism Was a Smaller Storage Contract

Habitat reached its scale by limiting what applications could request, shifting complexity away from the critical online path.

Language optimization alone does not explain Habitat's throughput. OpenAI also restricted the service to simple operations with predictable computational cost.

Habitat exposes a NoSQL interface based on objects and edges. Product teams define object types, relationships, and direct lookups instead of sending arbitrary SQL statements.

The model draws inspiration from Meta's TAO design, a geographically distributed store developed for Facebook's social graph. TAO also used a constrained query interface to serve enormous read workloads.

Habitat colocates an object and its edges within a storage partition. This makes direct relationship queries easier to distribute horizontally.

The service does not attempt to place every connected remote object in the same database region or partition. A traversal across several relationships can therefore require calls to different Cosmos DB accounts.

OpenAI accepts that inefficiency because Habitat does not offer unrestricted graph traversal on its online path. Applications can retrieve the direct edges of an object, but they cannot submit an arbitrary graph query to the service.

The constraint prevents a single request from expanding into unpredictable work. Unbounded fanout makes capacity planning difficult because a cheap client operation can generate a large number of storage calls.

OpenAI had encountered that imbalance with PostgreSQL. When the company and its schema set were smaller, engineers could review queries and confirm that important paths used indexes.

That review process stopped scaling as more teams and products appeared. OpenAI says expensive new queries on busy paths became a recurring cause of database outages.

SQL was not inherently the problem. The operational issue was that a concise query could demand a scan, join, or execution plan whose cost was difficult for the shared platform to contain.

Habitat makes expensive behavior more visible to the application. If a product needs several relationship hops, its code must issue those operations explicitly.

That design sacrifices developer convenience and query flexibility. It also gives the storage platform a predictable unit of work that it can isolate, route, throttle, and measure.

For complex queries, OpenAI uses a separate path. Change data capture, or CDC, streams updates from Habitat into isolated Rockset instances with near-real-time freshness.

Each client team scales its own Rockset resources. Analytical and search-heavy requests therefore do not compete directly with authentication, conversation startup, or other transactional operations.

The split resembles a broader pattern in large online systems. The primary store handles bounded, latency-sensitive operations, while secondary systems support flexible searches and analysis.

This pattern also clarifies who bears the complexity. Habitat simplifies the safest common operations, but product teams must design around its limits or operate separate infrastructure.

The choice can slow feature development when a product needs rich traversal or filtering. It can also produce more application requests than a database capable of executing a server-side join.

OpenAI judged that friction preferable to an unpredictable shared service. At 70 million requests per second, a flexible operation with an unexpected fanout can create a platform incident very quickly.

This is the core mechanism behind OpenAI Habitat storage. The platform scaled by refusing to provide some database capabilities, then offering isolated alternatives outside its most sensitive path.

For engineering teams building internal knowledge systems, the lesson is relevant beyond OpenAI. A searchable knowledge base also benefits when transactional retrieval and heavy analysis have explicit boundaries.

The crucial distinction is between a useful abstraction and an unlimited one. Habitat gave developers a simpler interface, but its scale depended on keeping that interface deliberately narrow.

Rust Reduced the Bill, but the Claims Need Context

OpenAI's Rust rewrite produced striking internal efficiency numbers, yet the company has not published enough methodology for independent comparison.

OpenAI began the full service rewrite in the second quarter of 2026. The company says two engineers completed it with assistance from Codex and GPT-5.5.

At publication, the Rust service handled 95 percent of Habitat's production requests. OpenAI planned to remove the remaining Python implementation within weeks.

The company reports that Rust uses one-sixth as much CPU and one-fifteenth as much memory as the Python version. It also reports lower average latency and lower tail latency.

Those numbers are important because Habitat had become a major consumer of internal compute. A sixfold CPU improvement can release substantial capacity when applied to tens of millions of requests per second.

The rewrite also tests a claim that has circulated around AI-assisted software development. OpenAI deliberately deferred migration partly because it expected future coding models to make a later rewrite easier.

According to the company, that wager paid off. Two engineers and OpenAI's models moved the established service to Rust after its interfaces and operational requirements had matured.

The sequence matters more than a simple Python-versus-Rust comparison. OpenAI did not ask a model to discover Habitat's requirements from scratch.

Years of production work had already defined the APIs, metrics, failure cases, routing rules, deployment patterns, and compatibility boundaries. That accumulated knowledge reduced ambiguity for both engineers and coding agents.

The rewrite therefore supports a narrower conclusion. AI coding tools can reduce the labor needed to port a well-understood service with stable behavior and extensive production signals.

It does not establish that two engineers can safely replace any large Python platform. OpenAI has not published the service's code size, test coverage, review effort, incident record, or total support from adjacent teams.

The efficiency comparison also comes from OpenAI's internal measurements. The company has not disclosed workload normalization, hardware details, compiler configuration, or the precise boundaries included in each measurement.

Readers should treat the sixfold and fifteenfold figures as reported production results, not universally reproducible language benchmarks.

There is another unresolved question around operational complexity. Rust can reduce resource use and provide stronger compile-time checks, but it raises the skills required for some contributors.

Python allowed OpenAI to change the early platform quickly. Product engineers could understand and modify the shared client while the storage contract was still taking shape.

A Rust implementation becomes easier to justify after that contract stabilizes. Performing the rewrite earlier might have slowed experimentation without eliminating the need for later architectural changes.

The migration also does not remove Habitat's other scaling constraints. Database partitioning, regional capacity, load balancing, cache behavior, and downstream connection limits remain independent concerns.

OpenAI's own account makes this clear. The service rewrite is one part of Habitat, while the storage layer and its Cosmos DB partnership carry the larger 500-petabyte and 70-million-request workload.

The report is also unusually candid about Python's temporary role. OpenAI describes the first service implementation as intentional technical debt, accepted to restore control during hypergrowth.

That framing creates a useful challenge for other engineering organizations. Technical debt is defensible when teams identify the constraint it solves, measure its accumulating cost, and define the signal for replacing it.

It becomes harder to defend when "temporary" infrastructure has no migration trigger. Habitat crossed that trigger when its worker count, core consumption, and continuing growth made resource efficiency a strategic concern.

OpenAI's claim that coding models enabled the migration remains particularly difficult to isolate. The company builds Codex and GPT, supplied the engineers, owned the target system, and published the evaluation.

A stronger validation would include detailed migration metrics. Useful data would cover engineer hours, generated-code acceptance rates, defects found during review, production regressions, and maintenance work after launch.

Until those details appear, the rewrite is credible evidence from a major production deployment, but not a controlled assessment of AI-generated systems code.

What OpenAI Habitat Storage Must Prove Next

The next test is whether Habitat can preserve predictable latency and isolation as its Rust migration, regional footprint, and storage volume keep expanding.

The first signal is the final retirement of Python. OpenAI said the remaining five percent of production traffic would move within weeks, making the transition a near-term test of compatibility.

A clean retirement would strengthen the company's account of a low-disruption rewrite. Extended dual operation or repeated rollback would suggest that the remaining traffic contains harder dependencies than the reported percentage implies.

The important measure is not simply whether Rust receives all requests. OpenAI must preserve behavior across authorization, routing, encryption, shadowing, rate limits, and product-specific compatibility.

The second signal is the promised technical account of Habitat's database layer. OpenAI says the next installment will explain multi-tenant reliability, layered read optimization, and its work with Azure Cosmos DB.

That publication should clarify how more than 500 petabytes are placed across almost 40 regions. It should also explain how Habitat isolates noisy tenants without wasting large amounts of reserved capacity.

Database evidence could reinforce the article's central judgment. If OpenAI shows bounded operations, effective partitioning, and regional controls working together, Habitat looks like a durable platform architecture.

If the next account relies mainly on extraordinary capacity allocation, then the current design may be harder for other organizations to reproduce.

The third signal is the behavior of Habitat during continued product growth. The company says its user and traffic expansion is still accelerating, so today's 70-million-request peak will not remain the final target.

Readers should watch for published changes to tail latency, incident containment, and per-request resource use. These indicators reveal whether efficiency gains are translating into user-visible reliability.

They also show whether centralization has created an unacceptable blast radius. One control layer simplifies upgrades, but a defect in that layer can reach ChatGPT, Codex, APIs, and internal services together.

Habitat's evolution offers no single recipe for scaling an AI product. It instead shows a sequence of decisions tied to the constraint that mattered at each stage.

OpenAI first optimized developer access through a library. It then traded local efficiency for centralized control, constrained the request model, tuned Python's failure modes, and rewrote the service after resource costs became dominant.

That sequence is the more transferable result. Teams rarely know their final architecture before demand arrives, especially when products and traffic are changing together.

The practical question is whether they can preserve enough observability and interface discipline to replace temporary components later. OpenAI says Habitat did, with coding models helping accelerate the final port.

For developers and enterprise buyers, OpenAI Habitat storage also reveals what sits behind an apparently simple prompt box. Starting a conversation can depend on hundreds of lookups across a platform whose slowest call shapes the experience.

Watch the Python retirement, the promised Cosmos DB analysis, and Habitat's next latency figures. Together, those signals will show whether OpenAI built a lasting storage foundation or another temporary stage in an unusually fast infrastructure race.