I work in the refurb division of an ewaste recycling company[0]. To prepare a machine for sale, the drive needs to be wiped, and (optionally) an OS loaded. Wiping happens in WipeOS[1], which loads when you PXE boot on the internal company network. To install an OS, I have a separate network on my desk that will load iVentoy[2] when PXE booted, where I can further boot from ISOs I have on my server, but I almost always install Linux Mint. With those 2 things, I can largely do my job without fumbling with and losing USB drives.
I have 2 16 port switches on my desk, with over a dozen ethernet cables plugged into each. The yellow cables will PXE boot WipeOS, and the black ones PXE boot iVentoy.
The fun thing about learning to boot from PXE, is that you have to learn it every time you onboard a new type of hardware... or a new VM hypervisor... or new NIC firmware... or new BIOS firmware.
God help you if you actually want to install an operating system.
PXE is such a vital capability for working with on-prem servers. But it's ten different things which all have to play nicely together. Every time I build a PXE system I feel like I'm reinventing the universe in my tiny subnet.
show comments
bradfa
PXE is awesome, especially if you combine it with systemd's UKI mechanism and its EFI stub. You can load a single file via TFTP or HTTP(S) and boot into a read-only (or ramdisk-only) full Linux system. Most off the shelf distributions can be made to work in this way, with a small bit of effort. A very usable Debian system is a few hundred MB.
You can extend this with secure boot (using your own keys) to sign the entire UKI file, so your firmware will authenticate the full "disk" image that it boots into.
starkparker
I've used PXE (not even iPXE, just DHCP/TFTP without HTTP) mainly in environments where a LAN client-server game would need to be launched on many systems at once. Nothing quite like rolling out a hand-tailored distro for a single game to 16 computers and seeing them all boot and load straight into the game, one after the other, entirely unattended, from one broadcast boot-over-Ethernet trigger.
I think at one point we were even using distcc to use the clients to speed up rebuilds while iterating on the game. I should revisit that with iPXE and icecream.
pzmarzly
TFTP is crazy slow, even with RFC 7740 (buffering), but the payloads are usually small so few people care.
Thankfully modern BIOSes tend to implement HTTP boot option, where you can point to any HTTP or HTTPS URL (as long as the URL ends with ".efi", which is a pretty dumb limitation if you ask me).
show comments
anonymousiam
Having done lots of network booting over the years, here are a few of my lessons learned:
PXE is a big improvement over the boot EPROMs that we needed to install on our NICs back in the day. Those would get an address via DHCP and then TFTP the boot image, and boot it.
I've had some trouble with PXE boot that's been caused by STP. If your PXE boot server has, or is behind a bridge with STP turned on, it can prevent the client from booting. I think this has something to do with the STP "learning state", but turning off STP on the bridge can solve the problem, as long as you're sure that you will not be creating any network loops on the affected interfaces.
There's also a new "https boot", which is supposed to be a PXE replacement, but TLS certs have time validity windows, and some clients may not have an RTC, or might have a dead CMOS battery, and those might not boot if the date is wrong.
show comments
ronniefalcon
Lots of fun with this and lots of possibilities.
Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.
show comments
happyPersonR
I’m glad a lot of server stuff has redfish. But something better still needs to be there for non-server stuff for sure. Raspberry pi style bootloaders would be amazing, ones we could configure to use a certain image before powering on for first boot would be even more amazinger.
latchkey
I figured out how to PXE boot 20,000 PS5 APU blades (BC-250) during covid when I couldn't even get to the actual hardware. Great fun.
Oh oh oh I know this!
I work in the refurb division of an ewaste recycling company[0]. To prepare a machine for sale, the drive needs to be wiped, and (optionally) an OS loaded. Wiping happens in WipeOS[1], which loads when you PXE boot on the internal company network. To install an OS, I have a separate network on my desk that will load iVentoy[2] when PXE booted, where I can further boot from ISOs I have on my server, but I almost always install Linux Mint. With those 2 things, I can largely do my job without fumbling with and losing USB drives.
I have 2 16 port switches on my desk, with over a dozen ethernet cables plugged into each. The yellow cables will PXE boot WipeOS, and the black ones PXE boot iVentoy.
[0] https://www.ebay.com/str/evolutionecycling
[1] https://www.wipeos.com/
[2] https://www.iventoy.com/en/index.html
The fun thing about learning to boot from PXE, is that you have to learn it every time you onboard a new type of hardware... or a new VM hypervisor... or new NIC firmware... or new BIOS firmware.
God help you if you actually want to install an operating system.
PXE is such a vital capability for working with on-prem servers. But it's ten different things which all have to play nicely together. Every time I build a PXE system I feel like I'm reinventing the universe in my tiny subnet.
PXE is awesome, especially if you combine it with systemd's UKI mechanism and its EFI stub. You can load a single file via TFTP or HTTP(S) and boot into a read-only (or ramdisk-only) full Linux system. Most off the shelf distributions can be made to work in this way, with a small bit of effort. A very usable Debian system is a few hundred MB.
You can extend this with secure boot (using your own keys) to sign the entire UKI file, so your firmware will authenticate the full "disk" image that it boots into.
I've used PXE (not even iPXE, just DHCP/TFTP without HTTP) mainly in environments where a LAN client-server game would need to be launched on many systems at once. Nothing quite like rolling out a hand-tailored distro for a single game to 16 computers and seeing them all boot and load straight into the game, one after the other, entirely unattended, from one broadcast boot-over-Ethernet trigger.
I think at one point we were even using distcc to use the clients to speed up rebuilds while iterating on the game. I should revisit that with iPXE and icecream.
TFTP is crazy slow, even with RFC 7740 (buffering), but the payloads are usually small so few people care.
Thankfully modern BIOSes tend to implement HTTP boot option, where you can point to any HTTP or HTTPS URL (as long as the URL ends with ".efi", which is a pretty dumb limitation if you ask me).
Having done lots of network booting over the years, here are a few of my lessons learned:
PXE is a big improvement over the boot EPROMs that we needed to install on our NICs back in the day. Those would get an address via DHCP and then TFTP the boot image, and boot it.
I've had some trouble with PXE boot that's been caused by STP. If your PXE boot server has, or is behind a bridge with STP turned on, it can prevent the client from booting. I think this has something to do with the STP "learning state", but turning off STP on the bridge can solve the problem, as long as you're sure that you will not be creating any network loops on the affected interfaces.
There's also a new "https boot", which is supposed to be a PXE replacement, but TLS certs have time validity windows, and some clients may not have an RTC, or might have a dead CMOS battery, and those might not boot if the date is wrong.
Lots of fun with this and lots of possibilities.
Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.
I’m glad a lot of server stuff has redfish. But something better still needs to be there for non-server stuff for sure. Raspberry pi style bootloaders would be amazing, ones we could configure to use a certain image before powering on for first boot would be even more amazinger.
I figured out how to PXE boot 20,000 PS5 APU blades (BC-250) during covid when I couldn't even get to the actual hardware. Great fun.
[flagged]