This is the right way to deliver software.

Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.

In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.

This is why system programming still matters.

Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.

This reminds me how you can save a bunch of bytes just by making sure your structs are aligned. In go for example:

  type Wasteful struct {
    a int16
    b int
    c byte
  }

  type Aligned struct {
    b int
    a int16
    c byte
  }

With my own MaraDNS, I aggressively optimized the memory usage of blacklist entries by having a single really big malloc() to allocate the memory for the entries, then traversing that memory block for potentially blacklisted entries.

When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.

https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry...

Not sure what they use to hold the cache key and entry. If a hashmap is used, then a radix tree (adaptive radix tree) would be better in saving memory space. Most of content of the qname field of the CacheKey is hostname, like www.site.com. The reverse version com.site.www fits nicely in navigation path of a radix tree. The common prefixes like "com." are shared and compressed in the parent nodes of the tree.

Even a BTree with compressed prefix keys can save space in the qname.

These seem like some fairly standard approaches for reducing memory usage. I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.

If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.

It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.

One of my proudest professional moments was when me and three others managed to reduce memory load of the game Wavetale from 20+GiB to under 3GiB so we could port it to Nintendo Switch.

The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%

Funny thing about cloudflare. I have a dns warming script that uses their top 1k or 10k addresses. Then when my master starts up it warms the entire cache. Everything else uses memcache so the cluster is nice and toasty. As far as I can tell no one else releases domain statistics like them.

The intermediate level Rust dogma is to try your hardest to avoid the heap, and to tear your hair out at the throne of monomorphization. While both are broadly true, it's articles like this that show that a single pointer (or call) indirection can sometimes be better.

We're finally seeing more appreciation for this kind of engineering. Not everything needs to be solved by throwing more hardware at the problem

I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.

So they optimized from Vec to Box, but they're still using Box all over and spending 16 bytes on it? The things they're boxing need 2 bytes for length, and their memory use is low enough that they could cram the pointers into 4 bytes. Trying to pack that into 6 bytes is probably too much fuss for the benefit, but I see no reason to use more than 8 bytes.

It's weird that it took so long for these trivial optimizations but it might just be that they were working on optimizing other stuff.

Where are their users coming from? Besides the few manually putting 1.1.1.1 in their settings.

Why do people seem to think that optimization is something you only have to deal with once the software scales so much that 100s of TB of memory or disk space (or thousands of hours of processing time) are being wasted.

It is almost like nobody even thought during the design phase about what might happen down the road.

This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).

General theme: A programming language's native in-memory object format is typically optimized for random access, uniformity, and mutability (fields at fixed offsets, etc). Serialization formats for network or disk tend to be designed explicitly to be more compact. But you can design your own in-memory representation too, with the properties you need.

The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....

The Record struct contains rtype and data where RecordData is a tagged union. Aren’t those two always in sync? Not a DNS expert, just wondering if this is redundant or there is a reason both are there. Doesn’t matter anymore if they store it already serialized but I would be interested why it was this way.

Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd

> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.

jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?

> 56% A records, 25% AAAA, and 19% TXT

And they say nobody uses IPV6.

How much is this in euro or do we measure money in ram now?

Obvious question: why wasn’t this done earlier? It looks like all the data was already available. At THAT scale, reducing memory usage is a must-have, not a nice-to-have. Weird.

> we store the records as a single Box<[u8]> containing each record encoded as a 2-byte length prefix followed by its raw bytes.

Interestingly this is exactly how netlink works-ish: https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.ht...

You start, get the type & length, and then that is how many bytes you read.

Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.

In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.

So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.

As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.

I wonder at their scale, why wouldn’t it make sense to store the entries lightly compressed in memory?

I'll buys some spare RAM you now have. I only need 64GB.

The Art of Production.

Now put the 100 terabytes of memory back to the market. Stop hoarding RAM.

> Once we store a DNS response in the cache, however, we never modify it again. The capacity field serves no purpose, but still costs 8 bytes per Vec

Were there no design discussions/reviews when the system was setup to catch trivial things like this?

One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?

What's the speed of service/response time relative to the data source?

At that point it might be enough to replace your multiple caches with fewer in-RAM databases?

It's an interesting problem.