I don't like the use of the term "misfolding." Firstly, why overload a term when there's no reason to? Secondly, the analogy barely holds. Protein misfolding occurs by chance, as an "accident" of nature. Weight space programming is a targeted operation, with clear intent. It is not an accidental process.
dimatura
This is totally not what the paper is about, but I immediately thought of the original concept of meme [0]. Since so much content online is LLM generated, if that content is used to train new LLMs, in theory that could serve as a transmission vector of ideas or concepts in a viral fashion. The same way certain memes are more successful in human hosts for various reasons, maybe some are more successful for LLM hosts for whatever reason, like quirks in the RLHF/ finetuning process.
They've only observed weight space editing. They haven't proved any sort of "virus". Possible? Sure. But the headline is very misleading
jrflo
Very misleading post title.
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
show comments
aatd86
All the more reasons for openweights and model hashing.
show comments
kirth_gersen
This definitely seems like a fascinating emerging field of cyber security. Good reason to keep old models around from earlier times. They may be dumb, but they are probably honest, if only because they predate the capabilities to tamper with the weights in these intentionally malicious ways...
fraywing
This title shouldn't even be allowed for this research.
angry_octet
This is a terrible slop paper. No virus like behaviour, absolutely nothing like protein folding diseases like prions. Banish these authors to writing listicle slop.
kinfish
there has to some sort public scanner for those zero weight viruses.
dare944
From the abstract:
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
iaanix
There are many such of vibe-coded "research ideas" just after the weight editing MIT blog appeared. Perhaps, it is because it is so easy to set up experiments for this stuff, and I presume some toy examples may even work from time to time. Also there is nothing virus-like here.
Kevin_Flynn
Interesting anti virus opportunity here perhaps.
RIP John McAfee.
show comments
donsupreme
lol
> We call this weight space misfolding, by analogy with protein misfolding in prion disease
show comments
xbar
I'm sure those Chinese models are fine.
show comments
redanddead
Why are we always trying to find how to make AI useless instead of useful
show comments
shevy-java
What does this mean?
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
show comments
doublerabbit
Page loads then fails? Not sure where to report this fwiw. iPhone 13
Error Details
{message: "Importing a module script failed.", name: "TypeError"}
message: "Importing a module script failed."
name: "TypeError"
Component Stack
<Lazy> at unknown
<Suspense> at unknown
<qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261
...
I don't like the use of the term "misfolding." Firstly, why overload a term when there's no reason to? Secondly, the analogy barely holds. Protein misfolding occurs by chance, as an "accident" of nature. Weight space programming is a targeted operation, with clear intent. It is not an accidental process.
This is totally not what the paper is about, but I immediately thought of the original concept of meme [0]. Since so much content online is LLM generated, if that content is used to train new LLMs, in theory that could serve as a transmission vector of ideas or concepts in a viral fashion. The same way certain memes are more successful in human hosts for various reasons, maybe some are more successful for LLM hosts for whatever reason, like quirks in the RLHF/ finetuning process.
[0] https://en.wikipedia.org/wiki/Meme
They've only observed weight space editing. They haven't proved any sort of "virus". Possible? Sure. But the headline is very misleading
Very misleading post title.
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
All the more reasons for openweights and model hashing.
This definitely seems like a fascinating emerging field of cyber security. Good reason to keep old models around from earlier times. They may be dumb, but they are probably honest, if only because they predate the capabilities to tamper with the weights in these intentionally malicious ways...
This title shouldn't even be allowed for this research.
This is a terrible slop paper. No virus like behaviour, absolutely nothing like protein folding diseases like prions. Banish these authors to writing listicle slop.
there has to some sort public scanner for those zero weight viruses.
From the abstract:
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
There are many such of vibe-coded "research ideas" just after the weight editing MIT blog appeared. Perhaps, it is because it is so easy to set up experiments for this stuff, and I presume some toy examples may even work from time to time. Also there is nothing virus-like here.
Interesting anti virus opportunity here perhaps.
RIP John McAfee.
lol
> We call this weight space misfolding, by analogy with protein misfolding in prion disease
I'm sure those Chinese models are fine.
Why are we always trying to find how to make AI useless instead of useful
What does this mean?
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
Page loads then fails? Not sure where to report this fwiw. iPhone 13
Error Details
{message: "Importing a module script failed.", name: "TypeError"} message: "Importing a module script failed." name: "TypeError" Component Stack <Lazy> at unknown <Suspense> at unknown <qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261 ...