While it wouldn't prevent the issue they described, I prefer to pull, rather than push. My thinking is, if you pull, you're still connected. If you push, as soon as the push finishes, you're locked out.
chasil
I have a few observations about this article.
Generally, try not to use SCP. It has been a crufty old program from the Berkeley R-Utilities, but newer OpenSSH releases have rewritten it to use the sftp-server server instead. There will be wildly different behavior between these implementations.
If you need something that SFTP cannot do, then use tar on both sides.
PuTTY has implemented their pscp to prefer the sftp-server for many years, in a long prediction of the eventual abandonment. Their pscp implementation is a better drop-in replacement than the OpenSSH solutions.
The allure of SCP is retry on failure, which is somewhat more difficult with SFTP:
until scp source.txt user@target:dir/
do echo target down; sleep 300
done
Converting that to pscp is much easier than SFTP.
I also have an older rhel5 system where I am running tinysshd to use better SSH crypto. Due to upgrades, NFS is now squashing everything to nobody, so I had to disable precisely these checks to let users login with their authorized_keys. I can post the code if anybody is curious.
show comments
jamiesonbecker
Common and easy footgun with scp/rsync and ~/.ssh perms/authorized_keys drift, even with experienced sysadmins
Not to shill too hard, but this specific failure mode is basically why Userify exists (est. 2011): local accounts on every box + a tiny outbound-only agent that polls and overwrites desired state (keys, perms, sudo role). So if a deploy step clobbers authorized_keys or breaks perms/sudo for managed users, the next poll (~90s) just re-converges it. Also kills sessions when a user is removed. No inbound ports to nodes; no PAM/NSS; designed to be auditable.
Totally sympathetic to the author here - I did something similar last year with an ec2 instance and was getting ready to shut it down, snap and attach so I could repair it, but then userify magically fixed it. :)
but also... who has a dir with 777 permissions? Is that something people do nowadays?
show comments
zahlman
I assume using `./*` rather than `.` in the `scp` command would have worked around the issue?
show comments
tracker1
I accidentally nuked my hosted server's network stack with a config error... my bigger mistake was generating a massive random password for the root account... the remote terminal management console didn't support pasting and the default config only gave you like 30s to login.... not fun at all.
Script all the things. double-check your scripts... always be backing up.
show comments
LoganDark
You did not transfer the files within a directory. You transferred the directory itself, via `.`. That is why scp changed the permissions of your home directory itself; if you instead had transferred via `*` I am sure you would not have had this problem.
impure
Ah, file permissions. My old friend. Good thing this happened on a 'local' server and not a remote VPS.
MomsAVoxell
Done stupid stuff like this enough times that I just use tar, and also make a sandbox directory to receive it, to double-check whats going to happen, before un—tar’ing it again into the destination intended and/or do a manual move.
Too many burned fingers to not do this little dance almost every other time.
Actually, I lied, I just use rsync like an insane person.
crest
It's nice to see people sharing their mistakes too.
sowbug
Related: In my Bash logout script I have a chmod that fixes authorized_keys. It won't help with scp because that's non-interactive, but it has helped the other 999 times I've forgotten to clean up the mess I made during an ssh session.
TZubiri
Getting locked out of a server must be a cannonical experienc in the sysadmin journey, like checking the logs to see you are being attacked as soon as your online, or trying to build your own linux from scratch without bloat.
roelschroeven
tl;dr: I you scp -r to your homedir, expect scp to copy not just files and directories but their permissions as well (which I think isn't all that surprising).
show comments
binaryturtle
When I load the site in my (slightly older) Firefox I just get some random junk and gibberish (markov chain generated nonsense?)
While it wouldn't prevent the issue they described, I prefer to pull, rather than push. My thinking is, if you pull, you're still connected. If you push, as soon as the push finishes, you're locked out.
I have a few observations about this article.
Generally, try not to use SCP. It has been a crufty old program from the Berkeley R-Utilities, but newer OpenSSH releases have rewritten it to use the sftp-server server instead. There will be wildly different behavior between these implementations.
The backend SCP changes are documented here:
https://lwn.net/Articles/835962/
If you need something that SFTP cannot do, then use tar on both sides.
PuTTY has implemented their pscp to prefer the sftp-server for many years, in a long prediction of the eventual abandonment. Their pscp implementation is a better drop-in replacement than the OpenSSH solutions.
The allure of SCP is retry on failure, which is somewhat more difficult with SFTP:
Converting that to pscp is much easier than SFTP.I also have an older rhel5 system where I am running tinysshd to use better SSH crypto. Due to upgrades, NFS is now squashing everything to nobody, so I had to disable precisely these checks to let users login with their authorized_keys. I can post the code if anybody is curious.
Common and easy footgun with scp/rsync and ~/.ssh perms/authorized_keys drift, even with experienced sysadmins
Not to shill too hard, but this specific failure mode is basically why Userify exists (est. 2011): local accounts on every box + a tiny outbound-only agent that polls and overwrites desired state (keys, perms, sudo role). So if a deploy step clobbers authorized_keys or breaks perms/sudo for managed users, the next poll (~90s) just re-converges it. Also kills sessions when a user is removed. No inbound ports to nodes; no PAM/NSS; designed to be auditable.
Totally sympathetic to the author here - I did something similar last year with an ec2 instance and was getting ready to shut it down, snap and attach so I could repair it, but then userify magically fixed it. :)
Shim (old but readable): https://github.com/userify/shim/blob/master/shim.py#L308
(obligatory: https://userify.com)
This is a useful tip!
but also... who has a dir with 777 permissions? Is that something people do nowadays?
I assume using `./*` rather than `.` in the `scp` command would have worked around the issue?
I accidentally nuked my hosted server's network stack with a config error... my bigger mistake was generating a massive random password for the root account... the remote terminal management console didn't support pasting and the default config only gave you like 30s to login.... not fun at all.
Script all the things. double-check your scripts... always be backing up.
You did not transfer the files within a directory. You transferred the directory itself, via `.`. That is why scp changed the permissions of your home directory itself; if you instead had transferred via `*` I am sure you would not have had this problem.
Ah, file permissions. My old friend. Good thing this happened on a 'local' server and not a remote VPS.
Done stupid stuff like this enough times that I just use tar, and also make a sandbox directory to receive it, to double-check whats going to happen, before un—tar’ing it again into the destination intended and/or do a manual move.
Too many burned fingers to not do this little dance almost every other time.
Actually, I lied, I just use rsync like an insane person.
It's nice to see people sharing their mistakes too.
Related: In my Bash logout script I have a chmod that fixes authorized_keys. It won't help with scp because that's non-interactive, but it has helped the other 999 times I've forgotten to clean up the mess I made during an ssh session.
Getting locked out of a server must be a cannonical experienc in the sysadmin journey, like checking the logs to see you are being attacked as soon as your online, or trying to build your own linux from scratch without bloat.
tl;dr: I you scp -r to your homedir, expect scp to copy not just files and directories but their permissions as well (which I think isn't all that surprising).
When I load the site in my (slightly older) Firefox I just get some random junk and gibberish (markov chain generated nonsense?)
<bleep> that nonsense!