The bug was apparently fixed in open source repo a week earlier, this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week (the commit message even included "Reported by Xint.io" and I'm guessing skilled black hats would know about Xint)
Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
Are we going to discover in a year that every server online this month had a rootkit installed? There’s got to be thousands who are too small time to wipe and reinstall.
show comments
ROllerozxa
Well I guess this was the vuln that was teased about on Mastodon some hours ago:
I got worried for a sec, then remembered all my nginx lives behind HaProxy and it looks like nginx is only vulnerable if its going http2/TLS directly.
Now I sit and wait for an HaProxy announcement...
ps. if you are on FreeBSD and doing TLS directly
sed -i.bak -E 's/^([[:space:]]server[[:space:]]\{)$/\1\
http2 off;/' /usr/local/etc/nginx/nginx.conf
;service nginx restart
show comments
codedokode
Integer underflow again. Programming languages should trap on integer overflows and not ignore them. Don't care about legacy CPUs not having a trap for this - they will go sooner or later.
show comments
yerimyerimie
Fun collision! We (Xint) were one of the teams and actually ended up finding a few more bugs along the way. If anyone wants to check it out our writeup covers those plus the full chain to root: https://nginx2root.com/
Nginx Proxy Manager isn't updated yet, I know a lot of people use this for their home server so just a reminder to disable http/2 here as well.
show comments
hosteur
So work around is to disable http/2 and only allow http/1.1?
show comments
puszczyk
ugh, looks like if you still run ingress-nginx controller (you shouldn't but it was super popular in Kubernetes docs/community and there was a very little time to migrate away) you won't get these patches https://github.com/kubernetes/ingress-NGINX
edit noted it's about a Kubernetes controller
andrelaszlo
I assume OpenResty is affected as well? ("High Performance Web Platform Based on Nginx and LuaJIT")
This sound suspiciously similar to the issue Microsoft said they were dealing with excess traffic on their front door, taking them down.
maybe not an RCE but a DOS in the same http/tls code.
show comments
manishrana
Everyday we are seeing new vulnerabilities. Thanks to AI
show comments
tancop
Everybody here talking about going back to HTTP/1.1, it's a good stop gap solution but we should really think about moving to HTTP3-only as fast as possible. Changing over is easy, if your server doesn't support it just put it behind a reverse proxy that does.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
show comments
jascha_eng
Never turned http2 on in my side project pfew.
But still, this is one is pretty rough...
How many hours before this is exploited in the wild ? Probably not all that long, that blog has a significant amount of details.
show comments
testtestmetest
To all the people here that would like to give AI credit for everything, and dont think it matters that everyone at Wiz is Israel Ex Cyber "Defense", and all you need is AI to find things like this. I have a challenge for you:
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
Hey this is TZubiri, the guy that comments that you should uninstall the dep whenever there is an post about a CVSS 9+ vuln.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
FLeXMurphy
Is this going to end up being another exploit that requires a ridiculous amounts of special conditions so as to not actually be applicable to majority of sites running nginx?
therepanic
What’s the point of these researchers if their autonomous system is now finding these vulnerabilities?
show comments
philipwhiuk
So...what library/tool/language is next folks? Presumably Wiz is sitting on a bunch more.
testtestmetest
Isreael btw everyone at Wiz is a veteran of Unit 8200.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.
K0nserv
Just another set of unfortunate examples of programmers who are bad at C, definitely not a C problem.
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
show comments
sebiw
Uhm, am I missing something or are the patched releases not out yet? Why disclose this level of detail when the patches are not ready yet?
Edit: All good. They released the new versions within 30 mins.
show comments
superkuh
It seems like it always HTTP/2 (except when it's regex) with nginx. Glad I'm http/1.1 only.
show comments
ronfriedhaber
Great work by wiz.
FAYSAL-KA
i dont know what it is this but good work
hollerith
HTTP/2 is so complicated that the code that handles it tends to be vulnerable. What a surprise
show comments
Imustaskforhelp
Okay now I believe that AI is finally burning down the internet and that all the incidents are not just pure coincidences.
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
show comments
johnbellone
The beatings will continue until morale improves.
show comments
gabucino
oh it's just http/2
who the fuck cares
only insane people enable modernity dogshit like this
having low karma has an additional benefit: i can shit wherever i like
The bug was apparently fixed in open source repo a week earlier, this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week (the commit message even included "Reported by Xint.io" and I'm guessing skilled black hats would know about Xint)
Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
Small consolation but HTTP/2 is not enabled by default in Nginx and "only" 12% of servers run it (still huge at Internet scale) https://w3techs.com/technologies/segmentation/ce-http2/web_s...
If you want to check if your host might be vulnerable:
https://xint.io/blog/nginx2root-remote-code-execution#23-com...
Workarounds if you're vulnerable and can't update immediately:
https://xint.io/blog/nginx2root-remote-code-execution#33-eme...
AI is burning down the internet. This is insane.
Are we going to discover in a year that every server online this month had a rootkit installed? There’s got to be thousands who are too small time to wipe and reinstall.
Well I guess this was the vuln that was teased about on Mastodon some hours ago:
https://is-a.cat/@ar/117634569518544368
FreeBSD is unpatched,
I got worried for a sec, then remembered all my nginx lives behind HaProxy and it looks like nginx is only vulnerable if its going http2/TLS directly.
Now I sit and wait for an HaProxy announcement...
ps. if you are on FreeBSD and doing TLS directly
sed -i.bak -E 's/^([[:space:]]server[[:space:]]\{)$/\1\ http2 off;/' /usr/local/etc/nginx/nginx.conf ;service nginx restart
Integer underflow again. Programming languages should trap on integer overflows and not ignore them. Don't care about legacy CPUs not having a trap for this - they will go sooner or later.
Fun collision! We (Xint) were one of the teams and actually ended up finding a few more bugs along the way. If anyone wants to check it out our writeup covers those plus the full chain to root: https://nginx2root.com/
> Wiz Research, Ikotas Labs, Metnir, Rapid7, and Xint independently uncovered
Is this some kind of record for bug collision?
Nginx Proxy Manager isn't updated yet, I know a lot of people use this for their home server so just a reminder to disable http/2 here as well.
So work around is to disable http/2 and only allow http/1.1?
ugh, looks like if you still run ingress-nginx controller (you shouldn't but it was super popular in Kubernetes docs/community and there was a very little time to migrate away) you won't get these patches https://github.com/kubernetes/ingress-NGINX
edit noted it's about a Kubernetes controller
I assume OpenResty is affected as well? ("High Performance Web Platform Based on Nginx and LuaJIT")
OPs article mentions 2 CVEs, however this one claims that there are 5 new ones: https://news.ycombinator.com/item?id=50547423
This sound suspiciously similar to the issue Microsoft said they were dealing with excess traffic on their front door, taking them down. maybe not an RCE but a DOS in the same http/tls code.
Everyday we are seeing new vulnerabilities. Thanks to AI
Everybody here talking about going back to HTTP/1.1, it's a good stop gap solution but we should really think about moving to HTTP3-only as fast as possible. Changing over is easy, if your server doesn't support it just put it behind a reverse proxy that does.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
Never turned http2 on in my side project pfew. But still, this is one is pretty rough...
don't know why the mainline alpine have the empty image sha-256 : https://hub.docker.com/layers/library/nginx/1.31.9-alpine/im...
How many hours before this is exploited in the wild ? Probably not all that long, that blog has a significant amount of details.
To all the people here that would like to give AI credit for everything, and dont think it matters that everyone at Wiz is Israel Ex Cyber "Defense", and all you need is AI to find things like this. I have a challenge for you:
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
Docker images are a bit delayed because of unstable GitHub runners: https://github.com/nginx/docker-nginx/pull/1075#issuecomment...
Cue xkcd
Hey this is TZubiri, the guy that comments that you should uninstall the dep whenever there is an post about a CVSS 9+ vuln.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
Is this going to end up being another exploit that requires a ridiculous amounts of special conditions so as to not actually be applicable to majority of sites running nginx?
What’s the point of these researchers if their autonomous system is now finding these vulnerabilities?
So...what library/tool/language is next folks? Presumably Wiz is sitting on a bunch more.
Isreael btw everyone at Wiz is a veteran of Unit 8200.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.
Just another set of unfortunate examples of programmers who are bad at C, definitely not a C problem.
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
Uhm, am I missing something or are the patched releases not out yet? Why disclose this level of detail when the patches are not ready yet?
Edit: All good. They released the new versions within 30 mins.
It seems like it always HTTP/2 (except when it's regex) with nginx. Glad I'm http/1.1 only.
Great work by wiz.
i dont know what it is this but good work
HTTP/2 is so complicated that the code that handles it tends to be vulnerable. What a surprise
Okay now I believe that AI is finally burning down the internet and that all the incidents are not just pure coincidences.
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
The beatings will continue until morale improves.
oh it's just http/2
who the fuck cares only insane people enable modernity dogshit like this
having low karma has an additional benefit: i can shit wherever i like
fuck dang