OpenAI published an unusually useful engineering case study for anyone building high-throughput services: how Habitat, its internal online storage platform, evolved from a small Python library connected to Azure Cosmos DB into a distributed layer that now handles more than 70 million requests per second, more than 500 PB of data, and traffic for products used by over one billion people every week across nearly 40 regions.

The scale is eye-catching, but the best lessons are in the intermediate decisions: when to centralize a shared library, how to detect asyncio tail latency, why a LIFO connection pool can create a metastable failure, when deliberately restricting an API makes a system easier to scale, and why accepting technical debt can be the right move if it lets you stabilize contracts first.

Primary source: Rapidly scaling online storage to serve over 1 billion ChatGPT users — OpenAI.

1. From client library to central service

Habitat started in 2023 as a Python library. The idea was simple: product teams should not need to think about routing, authorization, encryption, serialization, connection pooling, or whether a read should come from Cosmos DB, a cache, or another backend.

That model works well while the system is relatively small. The problem appears when infrastructure policy lives inside dozens of independently deployed clients.

OpenAI describes a revealing example. To reduce the blast radius of regional failures, the team needed to route critical datasets across multiple regional Cosmos DB accounts. The routing logic had to be shipped behind a feature flag, deployed to many services, validated with shadow traffic, and then enabled. An unrelated rollback by one consumer eventually brought back an older buggy client and triggered the kind of outage the migration was meant to prevent.

The architectural conclusion was clear: if policy must evolve coherently across many consumers, embedding it in every process creates operational fan-out.

before

service A ─┐
service B ─┼─ Habitat library ─→ storage
service C ─┤
service D ─┘

every change requires many client rollouts
after

service A ─┐
service B ─┤
service C ─┼─→ Habitat service ─→ storage
service D ─┘

routing, policy, and observability evolve once

Turning Habitat into a service created one place for deployments, observability, authorization, auditing, request shaping, and backend access.

A useful rule emerges:

When a shared library starts carrying global operational policy, it may no longer be just a library.

2. asyncio gives concurrency, not CPU parallelism

One of the most interesting parts of the article is the Python analysis.

Habitat was highly I/O-bound, so asyncio looked like a natural fit. But the service also performed CPU work:

  • routing;
  • compression;
  • encryption;
  • checksums;
  • health checks;
  • request shadowing;
  • hedging;
  • configuration parsing.

A coroutine may already have a database response waiting and still be delayed because the event loop is busy doing other CPU work.

That creates a misleading trace:

Cosmos DB responds quickly
        ↓
response is ready
        ↓
[event loop is busy]
        ↓
coroutine finally runs again
        ↓
response reaches the client

The database was not slow. The local scheduler added the latency.

OpenAI measured this by scheduling periodic tasks and comparing expected execution time with actual execution time. That delta becomes a direct measure of event-loop scheduling delay.

At high utilization, the team saw jitter in the hundreds of milliseconds and, in edge cases, seconds.

3. The metric many async services are missing

Many dashboards track CPU, memory, disk, network, throughput, and HTTP latency. For an async service, that may not be enough.

It is also useful to track something like:

event_loop_delay = actual_wakeup_time - expected_wakeup_time

If that value rises, downstream responses may already be available while the process is unable to resume the coroutine that should handle them.

A minimal conceptual detector looks like this:

async def monitor_loop(interval=0.1):
    loop = asyncio.get_running_loop()
    expected = loop.time() + interval

    while True:
        await asyncio.sleep(interval)
        now = loop.time()
        delay = now - expected
        observe(delay)
        expected = now + interval

This is not OpenAI’s implementation; it simply illustrates the signal that matters: how long the loop takes to give execution back when it was expected to do so.

Habitat’s response was counterintuitive: low concurrency per process and many Python processes. According to the article, each process served only a small number of concurrent requests while the system scaled out the worker count aggressively.

4. A periodic job can destroy your p99

Another tail-latency source appeared in feature-flag configuration handling.

Workers periodically refreshed a large JSON configuration. The polling happened every minute without jitter, with multiple processes running inside each pod.

The result was that many workers stopped serving requests at roughly the same time to parse configuration.

00:59  normal traffic
01:00  worker 1 parses JSON
01:00  worker 2 parses JSON
01:00  worker 3 parses JSON
...
01:00  p99 jumps

Once CPU profiling identified the problem, the fixes were straightforward:

  • smaller targeted configuration;
  • less frequent refreshes;
  • jitter on periodic background work.

The broader principle is important:

Background jobs are still part of the latency budget of the request path.

A task being “async” or “in the background” does not mean it is free. If it consumes CPU in the same runtime, it competes with requests.

5. The LIFO bug that kept feeding the slow server

The most elegant failure story in the article is about connection pooling.

OpenAI observed that some processes stayed overloaded after a burst and, instead of recovering, received even more traffic.

The cause was related to the LIFO reuse behavior of aiohttp’s TCPConnector.

Imagine three servers:

A fast
B fast
C slow

During a burst, connections to C finish later. In a LIFO pool, the most recently returned connection is one of the first candidates for reuse.

A finishes ─→ pool
B finishes ─→ pool
C finishes ─→ pool   # last

next request
      ↓
LIFO picks C

C, already slow, gets more work. That work finishes later, the connection returns late again, and it stays near the top of the reuse stack.

The feedback loop becomes:

slow server
   ↓
finishes last
   ↓
LIFO reuses it first
   ↓
receives more traffic
   ↓
gets even slower
   ↺

OpenAI describes this as a metastable failure: even after the initial overload disappears, the system’s own dynamics preserve the degraded state.

Switching connection reuse to FIFO broke the loop and also reduced steady-state load variance.

6. Network infrastructure becomes part of application design

Today Habitat relies heavily on Istio and Envoy for connection pooling and load-aware balancing.

That also helps with a second-order problem created by the “many workers” strategy: too many downstream connections.

Large worker fleets can cause:

  • thundering herds during deployments;
  • heavy connection churn;
  • NAT exhaustion;
  • too many database connections;
  • retry cascades.

Envoy becomes a fan-in layer:

many Python workers
        ↓
      Envoy
        ↓
fewer persistent HTTP/2 connections
        ↓
     downstream

The proxy can multiplex HTTP/2 traffic, maintain more stable pools, and enforce rate limits and circuit breakers centrally.

A circuit breaker implemented independently in thousands of workers can react too late or inconsistently. A shared layer can maintain a more coherent view of the downstream dependency.

7. The most interesting design choice: Habitat does less

Habitat does not try to be a universal database interface.

Its NoSQL API is deliberately constrained. OpenAI avoids arbitrary queries with unpredictable joins, scans, or fan-out.

The operational reason is simple: it is easy to write something that is cheap for the developer but extremely expensive for production.

-- one line for the developer
SELECT ... JOIN ... JOIN ...

-- potentially millions of operations in production

Habitat optimizes for small, predictable units of work.

For more complex queries, changes from the OLTP system are streamed through CDC into secondary views in Rockset. Search and analytics therefore do not directly compete with the online storage hot path.

                 ┌→ OLTP / hot path
writes → Habitat ┤
                 └→ CDC → analytics / complex search

The general rule is excellent:

Not everything that needs to query your data should query the database that keeps your product alive.

8. Technical debt can be strategic

OpenAI knew that moving Habitat into a Python service would increase CPU, memory use, and latency. The team also expected that a rewrite would eventually become necessary.

They still chose not to migrate immediately.

Why?

Because the highest-priority problem was not cost per request. It was stabilizing the platform, defining the right APIs, and removing the operational fan-out created by the shared library.

That separated two questions:

  1. What should the system contract be?
  2. What is the most efficient implementation of that contract?

Trying to solve both at once would have increased risk.

This is a valuable architectural distinction. A temporary implementation is not always accidental debt; sometimes it is a way to buy information.

9. Python reached 20M+ requests per second before Rust

Before the migration, the Python service handled more than 20 million requests per second at peak.

That matters because it prevents the simplistic reading that “Python does not scale.” Python scaled very far. The cost was process count, infrastructure, CPU, memory, and careful engineering around tail latency.

Once the platform and its contracts were mature, changing runtimes made more sense.

10. Two engineers, Codex, and a full Rust rewrite

In Q2 2026, OpenAI says two engineers, supported by Codex and GPT-5.5, rewrote the entire service in Rust.

According to the published measurements:

  • the Rust service is roughly 6× more CPU efficient;
  • roughly 15× more memory efficient;
  • average and tail latencies are significantly lower;
  • it already handles about 95% of production traffic.

There is an important lesson here about coding agents.

OpenAI had made an earlier bet that programming tools would improve enough to reduce the cost of a future migration. That does not mean “AI rewrote the system by itself”; the article explicitly credits two engineers working with Codex and GPT.

But it does change the economics of a rewrite.

Once contracts, invariants, tests, observability, and expected behavior are clear, agents can reduce much of the mechanical cost of porting an implementation.

11. Contracts and invariants first; runtime second

Habitat’s evolution can be summarized as:

1. simple library
       ↓
2. broad adoption
       ↓
3. operational pain
       ↓
4. centralized Python service
       ↓
5. stabilize API + invariants + observability
       ↓
6. push Python and understand bottlenecks
       ↓
7. migrate to Rust

The key is the order.

An earlier rewrite would have frozen an architecture that was still changing. The Python phase helped reveal the real system contract.

12. What smaller systems can reuse

You do not need 70 million requests per second to apply these ideas.

Measure event-loop lag

If you use Python async, Node.js, or another cooperative runtime, add a signal that tells you when the scheduler stops meeting expected wake-up times.

Add jitter to periodic work

Thousands of instances refreshing at exactly :00 can turn harmless maintenance into a self-inflicted synchronized load spike.

Understand connection-reuse policy

LIFO, FIFO, connection lifetime, and balancing strategy are not micro-optimizations if they create or break feedback loops during degradation.

Separate OLTP from unpredictable queries

The hot path should have bounded cost. Search, reporting, and analytics can use secondary projections fed by CDC.

Centralize backpressure

Rate limits, connection fan-in, and circuit breakers should exist before a dependency starts failing.

Design APIs that are hard to misuse

A less powerful API can make a system much more scalable when it prevents unbounded operations.

Do not rewrite before you know what must be preserved

The best time to migrate languages is often when you can clearly express the contracts, invariants, and tests the new implementation must maintain.

Conclusion

The Habitat story is not simply “OpenAI moved from Python to Rust.” It is a story about architectural sequencing.

First, OpenAI converted a library into a service to recover operational control. Then the team learned the real behavior of Python under load: event-loop delay, background CPU work, connection-pool feedback loops, and thundering herds. They intentionally constrained the API to make request cost predictable and isolated complex queries through CDC.

Only when the platform was stable enough did they switch runtimes.

That sequence leaves a rule that applies to systems of almost any scale:

Stabilize contracts and invariants first. Optimize the implementation second.

OpenAI says part two will go deeper into multi-tenancy, read optimization, and how the Azure Cosmos DB layer behind 500+ PB and 70M+ req/s was scaled.

Reference