Heh, from the kernel patch: <a href="https:&#x2F;&#x2F;lore.kernel.org&#x2F;all&#x2F;CALCETrXbj__SFQMzPZhES5y6-sh4np-ZHY5T_=4QY5+Fn8BM4A@mail.gmail.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lore.kernel.org&#x2F;all&#x2F;CALCETrXbj__SFQMzPZhES5y6-sh4np-...</a>&gt; I saw a fun bug report in ripgrep and a studious but pretty bad
AI-generated analysisReferring to <a href="https:&#x2F;&#x2F;github.com&#x2F;dfoxfranke&#x2F;ripgrep-3494-analysis" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dfoxfranke&#x2F;ripgrep-3494-analysis</a> which I indeed thought &quot;that&#x27;s an awful lot written to have been written by a human.&quot;Looks like that thread is from...today!

That&#x27;s good! But I wonder why it wasn&#x27;t then enabled for that configuration or why the override wasn&#x27;t &quot;global&quot; enough and the default allocator was still partly used.

&gt; I get why people don&#x27;t bother replacing the default allocator from musl all the time (it&#x27;s there, convenient).ripgrep actually sets jemalloc as global allocator when built for 64b musl: <a href="https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;ripgrep&#x2F;blob&#x2F;435f59fc4b43af3ab32f34d53fa34978f393fe52&#x2F;crates&#x2F;core&#x2F;main.rs#L20-L41" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;ripgrep&#x2F;blob&#x2F;435f59fc4b43af3ab...</a>

If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to.It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.

The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it&#x27;s bad practice for the library to arbitrarily limit how many of those an application has at a time.Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.

The way most programs achieve being fast is by re-using allocations. You don&#x27;t need a fast allocator if you don&#x27;t allocate. Nothing of what ripgrep does inherently requires frequent allocations.

Sure, but the problem isn&#x27;t that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.mallocng should not be used at all.

It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.

There is a real tradeoff:- The musl allocator is only slow with multi-threading.- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl&#x27;s allocator.Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.

I get why people don&#x27;t bother replacing the default allocator from musl all the time (it&#x27;s there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven&#x27;t bothered replacing it with another more performant one.mallocng is bad at dealing with contention during multithreading. I&#x27;ve had applications that usually were I&#x2F;O bound suddenly become &quot;malloc&quot; bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.I get that there&#x27;s a real issue there and it&#x27;s interesting (to some) to address it, but it should have never surfaced this way in the first place.

&gt; All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it&#x27;s hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page&#x27;s set of objects.In theory, that&#x27;s what git&#x27;s packfiles are for: loose objects get packed together into a file, and if the file is well-arranged then objects that tend to get read together will also end up getting paged-in together (see also: <a href="https:&#x2F;&#x2F;www.kernel.org&#x2F;pub&#x2F;software&#x2F;scm&#x2F;git&#x2F;docs&#x2F;technical&#x2F;pack-heuristics.html" rel="nofollow">https:&#x2F;&#x2F;www.kernel.org&#x2F;pub&#x2F;software&#x2F;scm&#x2F;git&#x2F;docs&#x2F;technical&#x2F;p...</a>).GitHub in particular has spent a lot of time and effort on packing things well (e.g. <a href="https:&#x2F;&#x2F;github.blog&#x2F;open-source&#x2F;git&#x2F;scaling-monorepo-maintenance&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.blog&#x2F;open-source&#x2F;git&#x2F;scaling-monorepo-mainten...</a>), so I doubt that the &quot;billions of tiny files&quot; problem is at the root of their recent instability.

Similarly, I&#x27;ve often wondered if this is the root cause of why GitHub has become fragile of late. All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it&#x27;s hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page&#x27;s set of objects.And if <a href="https:&#x2F;&#x2F;isolveproblems.substack.com&#x2F;p&#x2F;how-microsoft-vaporized-a-trillion" rel="nofollow">https:&#x2F;&#x2F;isolveproblems.substack.com&#x2F;p&#x2F;how-microsoft-vaporize...</a> is to be believed even in the slightest, any unoptimized paths in Azure&#x27;s filesystem abstractions could cause usage spikes to have massive splash radius.

Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I&#x2F;O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn&#x27;t take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.

I&#x27;m becoming increasingly allergic to the performative writing of LLMs.The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to &quot;reason&quot; and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.

This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I&#x27;m not a kernel expert but I found it pretty understandable, so I&#x27;ll try to explain it.MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses used by a program map to physical RAM.Page: A 4k region of memory addresses.Anonymous: The page is just regular old memory used by a single process for general purposes, it&#x27;s not part of e.g. a memory-mapped file.Backing: The memory corresponding to a page. The kernel can direct the MMU to map any page to any part of RAM, or delete the mapping entirely.Zero page: The kernel keeps a single 4 KB page filled with zeros (the &quot;zero page&quot;).Page fault: An error that occurs when you try to write a page that&#x27;s not present, or write to a read-only page. Some page faults are normal and expected by the kernel, they support features like swapping, memory-mapped files, etc. Such &quot;expected&quot; page faults are invisible to your program. (On the other hand, an unexpected page fault occurs when your program just straight-up accesses an address it&#x27;s not supposed to, which will end your program with a SIGSEGV signal -- an all-too-familiar experience for a C programmer.)When you allocate say 4 MB of memory, the kernel simply sets it to 1k read-only copies of the 4 KB zero page. Then when your program tries to write to a page, it faults. The kernel handles this expected fault by assigning that page an individualized memory region. If a program allocates a lot of memory, it&#x27;s only assigned memory for what it actually uses, when it first uses it. This is an optimization: Every program that allocates more memory than it needs is wasteful, but the kernel can recover that waste by not backing the allocation with memory -- the program has to prove each page is needed by writing to it.Store to a freshly-faulted anonymous page: The program stored data to its own memory, which caused an expected page fault.&gt; becomes invisible to that same thread&#x27;s reload ~10 instructions laterThe thread fairly quickly tried to read the data from the same place in memory it had just written to. It doesn&#x27;t get the same data back. This is a major problem that causes the program to malfunction and crash.&gt; A pagemap read at the instant of the fault shows the backing is the kernel&#x27;s zero pageThe expected fault occurred because the program tried to write to an address that was mapped to the zero page, probably because it&#x27;s the first write to freshly allocated memory. (This rules out other reasons it might be faulted, e.g. the page had been swapped to disk.)&gt; concurrent munmap&#x27;s TLB shootdownThe page table is big and slow so the CPU caches parts of it in an area called the TLB (translation lookaside buffer). munmap is a kernel function that removes the backing for an address region. Since munmap changes the mapping, it has to inform the MMU the relevant part of the TLB is no longer valid.&gt; per-VMA-lock anonymous-fault fast pathThis is what the kernel does when the program first writes to a zero page. Saying it&#x27;s the &quot;fast path&quot; implies there&#x27;s some (slow) special case handling in the kernel code, but the bug doesn&#x27;t trigger any of the special cases, we&#x27;re in the most common case, which the kernel code tries to handle as quickly as possible because it has measurable impact on performance. I would guess per-VMA-lock has to do with how the kernel synchronizes this code (a &quot;lock&quot; is a basic synchronization primitive, you should be familiar with it if you do any kind of multithreaded programming).&gt; The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap&#x27;s TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interactionThe mechanism: What&#x27;s actually happening to cause the program not to be able to load data it just stored in memory.Localizes: The cause of a problem like this is a needle in a haystack -- it could be caused by the program, the kernel or the hardware. The analysis has narrowed down the haystack to a specific part of the kernel code.Race in that interaction: Two parts of the kernel code, (1) The kernel code for munmap and (2) the kernel code to handle a program&#x27;s first write to the read-only zero page for a fresh allocation. These two parts work fine individually but the problem occurs when they both try to change the page tables &#x2F; TLB at exactly the same time. This is quite a small, specific haystack compared to &quot;somewhere in the program, kernel or hardware&quot; we started with.

Agreed, the article you linked is a great example of clear, understandable, detailed technical writing.

This is a pretty clear analysis if you are a kernel developer.&gt; a thread&#x27;s own store to a freshly- faulted anonymous page becomes invisible to that same thread&#x27;s reload ~10 instructions later, because the page&#x27;s backing is replaced mid-functionA result of a machine instruction to store something in RAM got erased, when the same thread attempted to load it ~10 machine instructions later. The reason is that the physical RAM page backing that particular virutal page got replaced while the thread was running.The fault happens in the fast path for anonymous (i.e. not backed by files on disk) pages, in the granular per-VMA (virtual memory area) locks.&gt; &quot;A pagemap read at the instant of the fault shows the backing is the kernel&#x27;s zero page.&quot; -&gt; Backing?Yes, it&#x27;s typical kernel terminology. &quot;A backing page&quot; or a &quot;backing file&quot;.&gt; This is... words strung together. Nothing more. I wonder how people read and make sense of this.There&#x27;s nothing wrong with this description. It&#x27;s just written for kernel developers. It needs to be expanded and explained for people who are not.

fwiw &quot;backing&quot; is standard jargon in this context, and &quot;freshly faulted&quot; is phrasing you&#x27;ll see in Linux source comments. The writeup as a whole is indeed slop, though.

&quot;backing&quot; is a Virtual-Memory related term. With virtual memory, you can have memory areas (called &quot;pages&quot;) that are not really there but only present in the metadata (the &quot;book-keeping&quot;, so to say). The first time, someone actually tries to access this page, that access is interrupted (&quot;faulted&quot;) and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actual physical memory and fill it with zeroes, or crash the process).The &quot;backing&quot; refers to the physical memory that might or might not be present for every &quot;virtual&quot; piece of memory that your program has allocated.&quot;Freshly faulted&quot; means that a page of (virtual) memory has just received a &quot;backing&quot; by the process above and is, thus, very fresh in physical memory (even though the virtual memory might have been allocated much earlier)&quot;~10 instructions later&quot; refers to the assembly- (machine-) code, which, contrary to a high-level language like C, usually has long(ish) sequences of rather simple &quot;instructions&quot;. 10 instructions is a rather short interval in assembly code.As for the &quot;localize&quot;, the term used is actually &quot;localizes to&quot; which I read as &quot;turns out to be located in&quot; (probably just a bad English translation by the original author)While this whole summary reads a bit weird, I don&#x27;t think it is necessarily the result of an LLM, it&#x27;s probably just that someone who is rather inexperienced at writing up technical summaries did it.

I don&#x27;t care if the cat is black or white, so long as it catches mice.Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.

I tried reading and gave up at the &quot;Headline&quot;...Quoting from the bug analysis:&gt;Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread&#x27;s own store to a freshly- faulted anonymous page becomes invisible to that same thread&#x27;s reload ~10 instructions later, because the page&#x27;s backing is replaced mid-function. A pagemap read at the instant of the fault shows the backing is the kernel&#x27;s zero page. A captured core dump confirms the crash site with matched virtual addresses. The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap&#x27;s TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction, and a git comparison across v6.19&#x2F;v7.0&#x2F;v7.1&#x2F;mainline identifies the v7.0-introduced change on the munmap-teardown side that widens it.&quot;a thread&#x27;s own store to a freshly- faulted anonymous page becomes invisible to that same thread&#x27;s reload ~10 instructions later, because the page&#x27;s backing is replaced mid-function.&quot;
-&gt; ??? What&#x27;s a &quot;backing&quot; of a page? Freshly-faulted? Fresh fries?
&quot;~10 instructions later&quot;?&quot;A pagemap read at the instant of the fault shows the backing is the kernel&#x27;s zero page.&quot;
-&gt; Backing?&quot;The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap&#x27;s TLB shootdown.&quot;
-&gt; How can a mechanism &quot;localize&quot;?And more.This is... words strung together. Nothing more. I wonder how people read and make sense of this.In case people have forgotten what real technical writing looks like, here&#x27;s a sample (I am not the author):
<a href="https:&#x2F;&#x2F;yifan.lu&#x2F;2019&#x2F;01&#x2F;11&#x2F;the-first-f00d-exploit&#x2F;" rel="nofollow">https:&#x2F;&#x2F;yifan.lu&#x2F;2019&#x2F;01&#x2F;11&#x2F;the-first-f00d-exploit&#x2F;</a>

The computer seems to have identified source code to blame though? Unless of course it hallucinated which there’s always a non zero chance of

<pre><code> The overflow would still overflow; the use-after-free would still use after free; a musl mask race would still race. 
</code></pre>
Hilarious. Apart from the computer poetry, the conclusion seems to be “it’s something in Linux 7.0 + musl 1.2.5”, although the only reproduction is still on the same physical Threadripper CPU and only sometimes when heavily exercised, so it hasn’t really ruled out a hardware issue.

Stopped after&gt; Headline. The crash is real and reproducible.

Maybe it&#x27;s not going to get any better. LLMs were trained to do two things:- Mimic human language (and logic, since that&#x27;s part of what we express with language). This includes code written by humans.- Write code to solve problems. The figure of merit here is solving the problem, not reproducing anything human.I&#x27;m not an expert on LLMs, but it&#x27;s not obvious that the human reasoning they are fitting in the first case is going to be anything like the problem solving required in the second case. It was trained to get itself out of a problem, not to do it in a way that a human would relate to.And we&#x27;re already running out of human data to train on, while synthetic data has no limit. Bug reports in the future might amount to &quot;fix this because then I&#x27;ll get a cookie&quot;.

Reading an AI-generated bug report is awful.

The part that doesn&#x27;t make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn&#x27;t)).I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky&#x2F;speculative&#x2F;inconsistent.Alternatively, it could be a hardware bug, since afaict it&#x27;s only been repro&#x27;d on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)

The explanation is obviously written by an AI agent.

Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be that somehow a zero-page PTE was present when it shouldn’t have been.This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs.Also, I don’t think the zero page has pfn zero.If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.

Not really. It&#x27;s typical rambling LLM slop-analysis. It may be a correct analysis (I couldn&#x27;t stomach reading it in detail), but it&#x27;s a pain in the ass to read. A human doing the same analysis would have written something 1&#x2F;5th the length.

I can read it but not sure if worth spending energy on it.It doesn&#x27;t make sense for the reader to spend more energy
 than the writer spent on creating it.

Why is it unreadable? I actually find LLM bug reports&#x2F;breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don&#x27;t do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the &quot;3.3 The self-store tear&quot;, the LLM clearly describes exactly what went wrong, so you can verify it by hand.

I had the opposite reaction. Clear, detailed, well-organized. Pretty close to the ideal writeup.

The analysis of the kernel bug may be a better thing to link to: <a href="https:&#x2F;&#x2F;github.com&#x2F;dfoxfranke&#x2F;ripgrep-3494-analysis" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dfoxfranke&#x2F;ripgrep-3494-analysis</a>.

Coincidence. It also only triggers on one machine.

Probably because musl&#x27;s allocator exposes single newly faulted pages directly to the app. Other allocators tend to pre-allocate multiple pages at a time, so that the race window is narrower.

So, why the bug triggers only with muslc and not other libc ?

I would normally suspect musl&#x27;s thread stack size. Is the kernel bug confirmed?

What does one have to do with the other?

Codex bundles a ripgrep binary that is linked against musl. This is noted in the bug report: &quot;I originally encountered this bug in the rg bundled with OpenAI Codex. That binary is byte-for-byte identical with the one in <a href="https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;ripgrep&#x2F;releases&#x2F;download&#x2F;15.2.0&#x2F;ripgrep-15.2.0-x86_64-unknown-linux-musl.tar.gz" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;ripgrep&#x2F;releases&#x2F;download&#x2F;15.2...</a> [...]&quot;

Thank you for writing virtually all of the intelligent competent comments on this page.

This bug is almost certainly not a bug in ripgrep or Rust or musl or any user code at all. Maybe one could complain that musl is using an inferior allocator that is freeing then immediately reallocating the same address.edit: silly observation: the general Rust practice of using immutable bindings by default and not leaking shadowing bindings out of blocks would have made the bug I think I found less likely.

Shevy is a long time troll. Should have been banned a while back

I would suggest you revisit this in a day or two and reevaluate whether your knee-jerk blaming of rust was appropriate here, then ask yourself what biases led you to do that.

In addition to this likely not being a bug in the code you&#x27;re pointing at, but the kernel (written in C). musl is literally C code, and that most rust users compile against glibc just like most C users do.

Rust wouldn&#x27;t have helped, because this is a logic bug in the kernel, that creates a memory bug in userspace. The relevant kernel code would probably be marked &quot;unsafe&quot; with or without the bug.

The bug is in the C code that Rust is interacting with. If anything, you seem to be making an argument in favor of replacing even more of the C code with Rust so that this doesn&#x27;t happen.What you seem to think is a defense of C is accidentally just providing ammunition to the &quot;rewrite it in Rust&quot; crusade that I&#x27;m guessing you&#x27;re very much not a fan of. There are very solid arguments against rewriting everything in Rust, and you&#x27;re arguably doing more harm than good by making a counterproductive one.

For starters, musl is a C library, not a Rust library.

As they wrote, it&#x27;s &quot;a bit sad&quot; indeed because they cannot blame Rust for this.

A kernel bug and you’re criticizing Rust. Weird take

This is a bit sad. Here we have an example of rust and musl struggling whereas C&#x2F;C++ and glibc does not. That&#x27;s an oversimplification, but we also had not long ago the rewrite of ... I believe it was coreutils, in rust (or was it utillinux ... but I think it was coreutils), also have issues. Rust needs to toughen up here when it really wants to replace all of C in the linux ecosystem. Alternatively it could admit failure, then people can say that C will be - and remain - the forever king.

RipGrep musl binaries occasionally segfault during very-large searches