> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution.
Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.
show comments
londons_explore
A big class of attacker is nation state attackers who do not want to risk discovery.
A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.
For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.
Suddenly, nation state attackers won't be interested in MITM-ing at all.
Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
show comments
asalahli
It baffles me why VPS providers don't display the fingerprints on their dashboards or expose them through the API.
show comments
INTPenis
The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.
It's a neat little trick if you're often deploying VPS in shared cloud environments.
show comments
washingupliquid
I'm supposed to believe MitM with the same exact keypair is somehow possible? Private keys are never exchanged. Did everybody forget how crypto works?
Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?
Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.
show comments
kayson
I've been fighting against this chicken and egg problem in my homelab. I have a step CA SSH CA set up along with automated deployment, bootstrapping, and cert renewals with an ansible playbook. It sets StrictHostKeyChecking yes in the ssh_config for my domain so I'm happily protected against a very unlikely attacker that has a foothold in my network. Theoretically, it means I should never have to worry about host keys again.
Unfortunately, ansible does everything over.... SSH. So if I spin up a new VM or host, I have to manually trust the certificate for the first connection, which is the whole point of this article. I always have console access so I can log in and check the pub key.
There are various ways around this, including the authors suggestion with cloud init. None are very helpful for new physical hosts, though. I'm leaning towards a feature step CA supports that lets hosts authenticate themselves with an X509 cert, which you can easily get with ACME. It's so easy that you could even do it over console on a new physical host.
What I really wish is that there were something like the ACME TLS-ALPN challenge but for SSH servers. They can already present a self signed certificate, so it would just be a matter of connecting all the plumbing.
show comments
SoftTalker
So you inject a host ssh key pair at VM creation using cloud-init, then connect one time trusting that key and immediately replace the host keys in the VM. So there is still a brief race condition where the attacker could impersonate the VM if he had intercepted your cloud-init file, correct?
Also I don't see anywhere that the script reloads the sshd daemon, which AFAIK is necessary to get it to start using the new host keys and stop using the deleted initial host key.
lvlabguy
MitM is not possible if one uses public key authentication.
show comments
athrowaway3z
So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:
Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.
But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.
You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.
show comments
IgorPartola
There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.
crypt0r84
provision the hosts with an SSH CA, use the CA as a trust root in openssh.
they are various version out there from the big players.
zuzululu
but how real is this threat? you buy a box from hetzner or some VPS provider and its rooted when you try to ssh ?
show comments
TacticalCoder
> inject a temporary SSH host (private) key via cloud-init...
Reading the comments here I'm tempted to believe that if cloud-init is available and if we consider Heztner (and OVH etc.) provides a secure access to cloud-init (i.e. the box running cloud-init is really the box you think it is), then there are many different ways to solve this problem.
show comments
burnt-resistor
I guess they don't teach public key-enabled SPA secure port knocking anymore to defend ports of jumpboxes. And using jumpboxes and an internal vlan rather than putting control ports on the wild internet, right?
riffic
zero touch prod might be something folks should keep as a north star - SSH should be break glass and a last resort if possible.
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).
To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.
It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).
Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.
show comments
PunchyHamster
Seems like way easier way would be sshing for the first time and just typing `sudo reboot`. If VM reboots, it is yours
Or cat-ing some secrets that would be on target machine but not attacker
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution.
Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.
A big class of attacker is nation state attackers who do not want to risk discovery.
A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.
For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.
Suddenly, nation state attackers won't be interested in MITM-ing at all.
Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
It baffles me why VPS providers don't display the fingerprints on their dashboards or expose them through the API.
The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.
It's a neat little trick if you're often deploying VPS in shared cloud environments.
I'm supposed to believe MitM with the same exact keypair is somehow possible? Private keys are never exchanged. Did everybody forget how crypto works?
Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?
Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.
I've been fighting against this chicken and egg problem in my homelab. I have a step CA SSH CA set up along with automated deployment, bootstrapping, and cert renewals with an ansible playbook. It sets StrictHostKeyChecking yes in the ssh_config for my domain so I'm happily protected against a very unlikely attacker that has a foothold in my network. Theoretically, it means I should never have to worry about host keys again.
Unfortunately, ansible does everything over.... SSH. So if I spin up a new VM or host, I have to manually trust the certificate for the first connection, which is the whole point of this article. I always have console access so I can log in and check the pub key.
There are various ways around this, including the authors suggestion with cloud init. None are very helpful for new physical hosts, though. I'm leaning towards a feature step CA supports that lets hosts authenticate themselves with an X509 cert, which you can easily get with ACME. It's so easy that you could even do it over console on a new physical host.
What I really wish is that there were something like the ACME TLS-ALPN challenge but for SSH servers. They can already present a self signed certificate, so it would just be a matter of connecting all the plumbing.
So you inject a host ssh key pair at VM creation using cloud-init, then connect one time trusting that key and immediately replace the host keys in the VM. So there is still a brief race condition where the attacker could impersonate the VM if he had intercepted your cloud-init file, correct?
Also I don't see anywhere that the script reloads the sshd daemon, which AFAIK is necessary to get it to start using the new host keys and stop using the deleted initial host key.
MitM is not possible if one uses public key authentication.
So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:
Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.
But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.
You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.
There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.
provision the hosts with an SSH CA, use the CA as a trust root in openssh. they are various version out there from the big players.
but how real is this threat? you buy a box from hetzner or some VPS provider and its rooted when you try to ssh ?
> inject a temporary SSH host (private) key via cloud-init...
Reading the comments here I'm tempted to believe that if cloud-init is available and if we consider Heztner (and OVH etc.) provides a secure access to cloud-init (i.e. the box running cloud-init is really the box you think it is), then there are many different ways to solve this problem.
I guess they don't teach public key-enabled SPA secure port knocking anymore to defend ports of jumpboxes. And using jumpboxes and an internal vlan rather than putting control ports on the wild internet, right?
zero touch prod might be something folks should keep as a north star - SSH should be break glass and a last resort if possible.
https://www.usenix.org/conference/srecon19emea/presentation/...
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).
To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.
It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).
Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.
Seems like way easier way would be sshing for the first time and just typing `sudo reboot`. If VM reboots, it is yours
Or cat-ing some secrets that would be on target machine but not attacker