Sometimes performance does not improve because you buy bigger servers.

Sometimes it improves because someone looks at a data structure and asks: why are we paying for bytes we never use?

Cloudflare has published one of those engineering stories where the code changes look small locally and enormous at production scale.

Its DNS platform Big Pineapple, which powers services including 1.1.1.1, keeps more than 250 billion DNS cache entries at any given time. After a series of Rust optimizations, Cloudflare reduced the typical footprint of one cache entry from 953 bytes to 420 bytes.

The aggregate effect was huge: roughly 100 TB of memory freed across the fleet, without adding servers or replacing hardware.

And the most interesting part is that Cloudflare did not trade memory for speed.

It also achieved:

  • 43% higher insert throughput;
  • 19% lower lookup latency;
  • a drop in p99 resident memory per instance from roughly 9.3 GB to 5.3 GB.

This is a great systems-engineering case study because it exposes something that small applications often hide: the real cost of an abstraction depends on how many times you multiply it.

One byte that costs 250 GB

Scale completely changes the economics of a data structure.

If you have 1,000 objects, wasting eight bytes per object is irrelevant.

If you have 250 billion, it is not.

Cloudflare summarizes the situation with a brutal number: one unnecessary byte per cache entry costs more than 250 GB of RAM across the fleet.

Think of it like this:

1 byte × 250,000,000,000 entries
≈ 250 GB

That is why implementation details we normally dismiss — a capacity field, one extra pointer, alignment padding, or an independent allocation — can become terabytes.

The optimization was not one trick. It was a sequence of changes to how a DNS response is represented once it enters the cache.

First idea: if an object will never grow again, stop paying for capacity

In Rust, Vec<T> stores three major pieces of information:

pointer
length
capacity

String follows a similar pattern.

The capacity field is useful when a container can grow. It lets you append data without reallocating on every operation.

But a cached DNS response at Cloudflare has an important property: after insertion, it does not change.

The extra capacity no longer provides value.

So Cloudflare replaced growable containers with fixed-size forms such as:

Vec<T>    -> Box<[T]>
String    -> Box<str>

A Box<[T]> needs a pointer and a length, but it does not need to reserve future capacity.

Eliminating that overhead, along with reserved heap space that was never used, saved more than 15 TB of RAM across the infrastructure.

The lesson is much broader than Rust:

The best structure for building data is not always the best structure for storing it.

A useful pattern is:

build -> validate -> compact -> freeze -> read many times

During construction, you may need mutability and spare capacity.

On the read hot path, you may not.

Second idea: three lists can become one buffer plus offsets

A DNS response contains multiple sections, including:

  • answer;
  • authority;
  • additional.

A natural implementation is to store each section in a separate list.

The problem is that every list needs metadata: pointers, lengths, and potentially separate allocations.

Cloudflare realized those sections could be stored in one region of memory, with small offsets indicating where each section begins.

Conceptually:

before

answers    -> [ ... ]
authority  -> [ ... ]
additional -> [ ... ]

becomes:

records -> [ answers | authority | additional ]
                      ^         ^
                    offset    offset

Because the record count per section fits in a u16, the offsets can be tiny.

That removes auxiliary structures and reduces both metadata and allocations.

Cloudflare also packed several booleans into bitflags, which improved the struct layout and reduced alignment padding.

That detail matters: in low-level systems, the size of a struct is not always simply the sum of its fields.

A compiler may insert padding to keep values properly aligned in memory.

As a result, removing a seemingly small field can save more bytes than expected.

Third idea: do not store information you already have somewhere else

Each DNS record normally includes an owner: the domain name the record belongs to.

But for a large share of responses, that owner is identical to the domain already present in the cache key.

So the cache was repeatedly storing information it already knew.

Cloudflare changed the model so the owner can be optional.

Conceptually:

owner: Option<Box<Name>>

If the owner matches the queried domain, it is not stored.

When Cloudflare builds the DNS response later, it restores the value from the cache key.

If the owner really is different — for example in some records behind a CNAME — the full name is preserved.

This principle appears everywhere in large systems:

If a value can be derived cheaply and deterministically from information already available on the hot path, storing it again may become memory debt.

That does not mean maximal normalization is always correct.

Sometimes duplication saves CPU, simplifies code, or avoids coupling between structures.

The right decision depends on the balance between memory, latency, and complexity.

Cloudflare could make this trade because the cache key was already available on every lookup.

Fourth idea: the largest enum variant can dictate the cost of the common case

This is one of the most interesting Rust-specific details.

DNS records can carry very different kinds of data.

An A record needs only a 4-byte IPv4 address.

An AAAA record needs 16 bytes.

Other records, such as NAPTR, can be much larger.

If all of those types are variants of one enum, the enum needs enough inline space for its largest variant, plus its discriminant and alignment padding.

The result was surprising: tiny records could end up paying for a structure around 144 bytes in size.

And A plus AAAA represented more than 80% of the traffic mix Cloudflare used in its measurements.

As an intermediate step, the team moved larger enum variants to the heap using Box:

pub enum RecordData {
    A(Ipv4Addr),
    Aaaa(Ipv6Addr),
    Txt(Box<Txt>),
    Naptr(Box<Naptr>),
    Svcb(Box<Svcb>),
}

That lets small, common variants remain inline while large, rare variants are represented by a pointer.

It eliminates much of the waste caused by the enum’s maximum inline size.

But it creates a different problem.

Boxing saves space… but can hurt locality

Moving values to the heap creates additional allocations.

And an allocation is never completely free.

The allocator needs metadata and usually rounds requested sizes into fixed allocation classes.

The objects also end up scattered across different heap regions.

That hurts memory locality.

Modern CPUs work in cache lines. When related data is contiguous, one memory fetch can bring in information useful for several later operations.

When every piece lives behind a different pointer, cache misses become more likely.

So Cloudflare arrived at an interesting conclusion:

boxing was better than wasting roughly 120 bytes on common records, but it was still not the ideal final layout.

Fifth idea: store records almost like network bytes

The final optimization was more aggressive.

Instead of preserving every DNS record as a rich Rust object tree, Cloudflare began serializing much of the record data into a compact byte buffer.

Conceptually:

CacheEntry
   |
   +-- metadata
   |
   +-- Box<[u8]>
       [record][record][record][record]

That has several advantages.

First, the record data needs only one allocation.

Second, the data becomes contiguous in memory.

Third, several record types can be copied directly from the buffer into the outgoing DNS response without rebuilding their wire representation field by field.

Cloudflare can do this direct copy for types such as A, AAAA, TXT, and multiple DNSSEC records.

Only records containing domain names, where DNS name compression must be applied, still require additional parsing.

The change improved both memory usage and speed.

This stage alone increased insert throughput by roughly 13% in Cloudflare’s benchmarks.

That leads to an important high-performance systems idea:

Sometimes the most efficient in-memory representation looks more like the transport format than the domain object model.

Rich models are excellent for working with information.

Compact buffers are excellent for storing and moving enormous amounts of it.

The full result

After the five optimizations, Cloudflare reported:

MetricBeforeAfterChange
Net footprint per entry953 B420 B-56%
Allocations per entry1.1 KB461 B-58%
Insert throughput625,000/s893,000/s+43%
Lookup latency828 ns670 ns-19%

In production, p99 resident memory per instance fell from about 9.3 GB to 5.3 GB.

The fleet-wide aggregate working set ended up roughly 100 TB lower.

Cloudflare estimates that this amount of memory is comparable to the RAM in about 130 of its Gen 13 servers.

But there is an important nuance: this does not mean Cloudflare literally shut down exactly 130 machines.

The freed memory is distributed across a large platform and can be reused for something else.

In fact, Cloudflare says it plans to reinvest the memory into larger caches, which should improve hit rates and reduce queries sent to authoritative DNS servers.

Why less memory also became more speed

Optimization discussions often assume an unavoidable tradeoff:

less memory <-> more CPU

That is not always true.

In this case, using less memory also meant:

  • fewer allocations;
  • fewer pointers to follow;
  • less metadata;
  • smaller objects;
  • more useful data per CPU cache line;
  • better locality;
  • less serialization work during lookups.

Modern CPUs are extremely fast when the required data is already in cache.

They are much slower when they have to wait on main memory.

So shrinking the data footprint can increase the chance that the active working set stays closer to the processor.

You are not just saving RAM.

You are helping the CPU find what it needs sooner.

What this means for .NET

You do not need 250 billion entries to apply the same ideas.

.NET also has structures that are better suited to different lifecycle phases.

During construction you may need:

List<T>
Dictionary<TKey, TValue>
StringBuilder

But after building an object that will be read thousands or millions of times, a more compact representation may make sense:

List<T> -> T[]
mutable dictionary -> read-only/frozen structure
objects -> structs when the layout justifies it
multiple buffers -> one contiguous region

.NET also provides tools such as Span<T>, Memory<T>, ArrayPool<T>, and frozen collections that help build hot paths with fewer allocations.

The rule is not “use arrays for everything.”

The rule is separate the construction model from the execution model when the access profile justifies it.

What this means for Python

Python has significantly more per-object overhead than Rust.

A convenient structure like:

{
    "host": "example.com",
    "ttl": 300,
    "type": "A",
    "address": "192.0.2.1",
}

may be ideal for business logic, but expensive if you need to keep tens of millions of them in memory.

For workloads that are truly memory-sensitive, alternatives include:

  • bytes or bytearray for compact data;
  • array and memoryview;
  • struct for binary layouts;
  • dataclass(slots=True) to reduce attribute overhead;
  • contiguous arrays or columns instead of deep object graphs;
  • compact serialized formats when data is mostly read-only.

Again, the goal is not to sacrifice readability before there is a problem.

It is to recognize when the object model itself has become part of the operating cost.

The bigger lesson: optimize the lifecycle, not just the structure

The most valuable part of the Cloudflare story is not memorizing that Box<[T]> is smaller than Vec<T>.

It is understanding why that change was valid.

The team knew each cache entry had two very different phases:

phase 1: construction
- mutable
- needs working buffers
- may grow

phase 2: cache
- read-only
- read many times
- no longer needs spare capacity

Once you recognize those phases, changing representations becomes possible.

An object can start rich and flexible and end compact and immutable.

That pattern applies to:

  • caches;
  • search indexes;
  • rule engines;
  • in-memory catalogs;
  • routing tables;
  • configuration models;
  • event pipelines;
  • inference systems;
  • highly concurrent servers.

Do not optimize eight bytes when you have 800 users

There is also an opposite lesson.

The Cloudflare case can tempt engineers to start counting every byte in every application.

That would be the wrong lesson.

These optimizations make sense because several things are true at the same time:

  1. the structure is repeated at extreme scale;
  2. it is part of a critical hot path;
  3. there are hundreds of billions of instances;
  4. benchmarks can reproduce the effect;
  5. production measurements confirm it.

Measure first.

Then identify what dominates the cost.

Only after that should you accept the added complexity of a more compact representation.

A useful rule is:

clarity first
profiling second
optimization where multiplication justifies it

The byte is not the point. The multiplication is.

Cloudflare did not free 100 TB because it discovered a magical Rust instruction.

It did so by stacking many small, correct decisions:

  • remove capacity that is never used;
  • reduce lists and pointers;
  • stop storing redundant data;
  • prevent rare cases from dictating the size of common cases;
  • pack data into contiguous buffers;
  • reuse representations close to the wire format;
  • benchmark every change against both memory and latency.

At normal scale, some of those decisions would save kilobytes.

At Cloudflare scale, they saved terabytes.

That is probably the most reusable insight from the whole story:

You do not understand the efficiency of an architecture by looking at the cost of one instance. You understand it by multiplying that cost across the full scale of the system.

Sources