We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.

LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your LLM-based “AI” gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.

LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence. What bothers me about this entire situation is that presumably the folks that relied on the LLM-based “AI” to generate these vulnerabilities knew (or should have known) enough about their tool to know this would happen, but did not.

Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.

A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.

I think this is expected with the increased usage of AI on both fronts (finding vulns and automating their remediation). Yes, there are and will be companies that help other companies validate CVEs before they raise any alarms, but that noise/signal battle has always been there.

I work as a sec eng at a biotech firm and have a tool I built that pulls our ITIL and runs a search on NVD, GHSA, etc. for enrichment and classification. But the number of CVEs I see for products we house is astronomical compared to the actual version of that product we are running (and that's only one value). But that's also a challenge when working with so many sources and having a small team try to tackle them.

The age-old validate, remediate, verify proves itself useful once again!

The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult.

But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.

Not validating submissions seems like avenue for massive attack. Flood the whole system with endless false reports. Thus making it significantly less reliable.

I almost feel that we have a new generation of the "script-kiddie". People without much (or any?) knowledge of software engineering using outside tooling to do "stuff" they wouldn't be able to do themselves.

Maybe it's a not a perfect analogy - in this case the intentions seemed to be laudable - but we will see much more of this, including from malicious actors.

This is going to be fun for organizations that are mandated to patch all CVEs, isn't it?

> Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.

I have no experience in the realm of security, but why isn't the submission process similar to any normal software shop (big or small) in terms of required bug repro steps?

Somewhat related: https://sqlite.org/cves.html

Damn, dunno why I've always assumed that if something gets a CVE, then some authority has reproduced and verified it.

Shouldn't it be CNA's job? Why would anything get a number if it hadn't been verified?

> The cited code didn't even exist in those versions or referenced unrelated logic. > When testing the PoC payloads they didn’t work (not triggering any crash).

I think the future is pretty obvious, if this isn't being done on projects already: you need to automate these checks and reject automatically

They couldn't even be bothered to use a non-AI image.

Could be just me but I would be quite careful running these PoC's, especially since the CVE's mostly seem nonsensical. I suppose these CVE's could be targeting some kind of PoC execution instead of actual bug reporting? (I didn't look at the code in depth).

Why is the repo even mixing CVE's for "schreibfaul1 ESP32-audioI2S" and "SQLite"? Is mixing CVE's for different products in one repo common practice?

It really is a shame that funding cuts at NIST has led directly to this. I don't know if there is an incentive for a non-profit to come out of the woods to do the validation and enrichment process, but boy do we need it.

Like gortok has said, LLMs are just probabilistic text-prediction engines. They cannot automate the vulnerability validation lifecycle because they cannot account for improbable edge cases. They may not even get the version of the software right...

This isn't great but a negative side effect of an otherwise positive change. I know a lot of comments will try to make this into a gotcha but its really not. Decreased friction leads to misuse. Just like before we'll figure out ways (reputation, light filter pass) to deal with it.

The federal government ought to be funding NIST sufficiently to actually do sufficient analysis. Do we care about funding civil "cyber" defense or not?

(Obviously the answer is not, we only care about funding offensive capabiltiies).

Llm are great at things that are repeatable. Otherwise they are just like us - great at believable bullshit… a sign of true intelligence

> This incident demonstrates a systemic issue with automated vulnerability ingestion

But they make X-Ray which does automated vulnerability indexing and matching dependencies to CVEs.

You're absolutely right...

I think this will give a lot of ammunition for people on both sides, but the thing is we have made software so complex, ofc a lot of it is inherent, that we can't even tell if the vulnerability is hallucinated or not.

I find it getting closer and closer to the nonsensical 50 page mathematical proofs built with computer assited tools, that been driving reason for abstract proof systems so that we can break down such complicated examples into something simpler that we can feed into a system to help us verify.

I feel like for some reason we have built the output machine before we could even build the actual output verification machine. And aren't ready to accept that we should all move towards building some kind of system to better help verify the "truth" produced by these AI systems.

I am now very convinced that human built and proven to work verification systems for LLM outputs are more important than ever, but I just don't see any company or team working towards it.

Trillion dollar systems shipping to production code that is hardly understood by any one without any verification what really could go wrong. Honestly I really truly hope nothing goes wrong.

To note I am not anti-AI but I want to be able to have my LLM outputs be verified by more than just me under a tighter and tighter deadlines, accepting worse and worse slop just because it would less cost to me to accept the PR and then fix it in prod.

LLMs aren't sustainable software development, they are delusional slop factories if we aren't able to actually make sure the outputs have any amount of correctness.

With humans it worked because all 10 of us could in some ways verify each others outputs, and had trust in the fact that our timelines included the time for reviews.

A critical rating on a vuln that doesn't exist is wild. The triage cost still lands on the maintainers.

As a non-expert in this field, I feel we have introduced more “stochasticity”. Before, We had to deal with some human error/ misunderstanding for most part but now we have to deal with power of condensed human expertise along with homogenized flaws.

So a so-called vulnerability has:

- cited code not exist, and

- PoC not work.

How would any serious system allow it become a big deal?

nonexistent functions being referenced is pretty bad, I haven’t had that issue since like opus 4. Shows either complete lack of oversight and/or using a bargain basement model (so no even having financial skin in the game). Most likely both

Wild catch. This highlights just how critical precise dependency vetting is for core database engines.

could this be used to create some panic and make some money through trading or something?

no idea who submitted all of those, but it sounds like malicious intent

Taking security theatre to a whole new level.

Great article!

We need to further emphasize the importance of responsibility when using LLM tools to produce output for others. It's great to use them for refactoring and bug discovery, but keep in mind that it's your responsibility to analyze it and iterate on it with AI. It makes your code better and develops technical expertise.

The "Hey, analyze that codebase, find all dangerous CVEs, and write a README for the PoCs, so I can post it online for others to analyze, and if I'm lucky, I'll get a paycheck or a title to add to my resumé" approach might work in 0.1% of cases, but it will generate a ton of slop for the community to drown in.

I also think GPTZero and other AI detectors have far more false positives than correct guesses. I tried it on several texts & messages I wrote before 2019, and it flagged them as 80% AI-generated.

In that case, it's reasonable to assume that AI also generated the README text for each discovered CVE. In other cases, however, we should be more cautious.

In the mean time, my very real vulnerability reports are not even acknowledged because maintainers are busy handling this kind of nonsense.

> All advisories in this repo seem AI generated when testing them with Gptzero

I pasted this blog post from "Analysis Matrix" to the end in Gptzero, and it also says the blog post was AI-generated (71% chance of AI, 29% chance of AI-Human mix).

The irony of writing an article about slop reports and then defacing it with a giant unrelated slop image at the top.

just ran this article that ran a cve through gptzero.. through gptzero

-GPTZero AI Detection

-Model 4.8b

-We are moderately confident this text is a mix of AI and human

-63/88 Sentences likely AI generated

> Over the past few days, a newly created GitHub repo (programmervuln/cveadvisory-) published a batch of SQLite vulnerability advisories (as part of other 50+ CVEs which we believe are also LLM slop except from one). NVD quickly flagged these as critical, and CISA's ADP agreed.

These two sentences seem incompatible to me. A brand new actor can just say some stuff and cause major disruptions in this way?

Thankfully I’m seeing other people mention that this article itself seems AI-generated. While there's less proof, i suspect that the actual analysis was done by an llm as well. So while i like to dunk on hallucinations as much as the next guy, who's to say this analysis doesnt have them too?

(for the record, i think the cve's are probably bogus, i just think there's an inherent problem with debunking slop with slop)

Wild catch. This highlights just how critical precise dependency vetting is for core database engines."

Sue them for defamation.

Seriously, the hubris on these "AI" sloppy clowns needs consequences. =3

Underrated wisdom here

imagine being a maintainer and having to write "this code does not exist" in a CVE response

can anyone explain what value does the ai slop image at the top add to this blog post?

The duality of LLMs is real. On one hand, they are solving unsolved math problems, on the other hand they get fooled by the strawberry test or the car wash test.

We see this all the time on reddit and HN, where some users are building crazy things with LLM and others are swearing by how useless LLMs are. Some people claim SWE will be dead in a few years, and others are claiming the AI bubble is going to burst anytime now.

There should be some sort of defamation lawsuit possible when people publicly publish trash repos or reports like this. If you were writing a competing product in a safe language you’d love to see such FUD.

Honest take, this is a critical CVE.