PXE Booting multiple Raspberry Pi 3s with FreeBSD

To make it easier to upgrade the various Raspberry Pis that are around, lets PXE boot them so they can be upgraded easily.

In this article we will use the following tools to create a shared base system and then individual mounts for each RPi:

  • NFS
  • Poudriere
  • PXE
  • TFTP
  • ZFS

To make cable management easier, each Raspberry Pi is connected to a PoE switch using one of these Adafruit PoE Splitter. Using PoE for power also has the additinal benefit of each Rasperry Pi being powered by the UPS that the switch is connected to.

Creating a Poudriere build environment

Setting up Poudriere to build armv7 packages is pretty easy these days.

First, assuming there is a source checkout in /usr/src, create a build jail:

poudriere jail -c -j armv7 -m src=/usr/src -a arm.armv7 -b -K GENERIC

Note: armv7 is 32-bit and is required to access the RPi Camera Module. If the Camera Module is not being used, then arm64 can be used instead.

Build packages

This RPi will be used to run homebridge and some python scripts to monitor a temperature sensor and report data to a MQTT server.

Create a list of packages to build in /root/arm-packages, for this use case the packages list consists of:

net-mgmt/icinga2
net-mgmt/lldpd
www/npm
net/py-paho-mqtt
lang/python
sysutils/tmux
editors/vim-console
shells/zsh

Finally run the build and be prepared for it to take a long time:

poudriere bulk -f /root/arm-pkgs -j armv7

Setting up the RPi

Setting up the RPi requires booting and enabling USB booting at least once. The official Rasperry Pi website has directions here: Network booting

But the only part we care about is booting once with the following in /boot/config.txt:

program_usb_boot_mode=1

This is important because the NIC is USB attached and so we want to enable booting off of USB.

Setting up DHCP

host garagepi {
	filename "loader.efi";
	option root-path "/usr/pxeroot/garagepi"
	fixed-address 192.168.1.15;
	next-server 192.168.1.31;
}

Setting up the TFTP environment

Create a new ZFS dataset for the tftp root:

zfs create zroot/usr/pxeroot/tftp

Configure inetd to start tftp using the ZFS dataset that was just created, by finding the commented out tftp entry and modifying it to be:

tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/pxeroot/tftp

Enable the inetd service and start it:

sysrc inetd_enable="YES"
service inetd start

The tftp root must be populated with the following:

  • loader.efi
  • bcm2837-rpi-3-b.dtb

Copy the loader.efi out of the arm jail and into the tftproot.

Then build the dtb file

make builddtb FDT_DTS_FILE=/usr/src/sys/contrib/device-tree/src/arm/bcm2837-rpi-3-b.dts

And copy the file into the tftproot dtb/bcm2837-rpi-3-b.dtb

Setting up the filesystem layout

Create the ZFS dataset for this version based on the date:

DATE=`date +%Y%m%d-%H%M%S`
zfs create zroot/usr/pxeroot/pxe-${DATE}

Copy the contents of the jail into the NFS export:

cp -r /usr/local/poudriere/jails/armv7/* /usr/pxeroot/pxe-${DATE}/

Note: This is complete copy so that the poudriere jail is independent of what happens with these Raspberry Pis.

Modify the loader.conf to enable the serial console and load the NIC driver:

console="comconsole"
if_ue_load="YES"

Enable SSH in rc.conf:

sysrc -R /usr/pxeroot/pxe-${DATE}/ sshd_enable="YES"

Configure pkg(8):

mkdir -p /usr/pxeroot/pxe-${DATE}/usr/local/etc/pkg/repos
echo 'FreeBSD: { enabled: no }' >> /usr/pxeroot/pxe-${DATE}/usr/local/etc/pkg/repos/FreeBSD.conf
echo 'local: { url: "http://192.168.1.31/armv7-default" }' >> /usr/pxeroot/pxe-${DATE}/usr/local/etc/pkg/repos/local.conf

Install the packages needed by most or all of the Raspberry Pis:

pkg -r /usr/pxeroot/pxe-${DATE} install zsh lldpd tmux vim-console

Create a snapshot of the dataset so it can be cloned:

zfs snapshot zroot/usr/pxeroot/pxe-${DATE}@initial

Finally clone the dataset for the specific Raspberry Pi, in this case called garagepi:

zfs clone zroot/usr/pxeroot/pxe-${DATE}@initial zroot/usr/pxeroot/garagepi

Create a user in the dataset to ssh in as:

pw -R /usr/pxeroot/pxe-${DATE}/ useradd -n brd -c "Brad Davis" -z /usr/local/bin/zsh -m -h -

Note: The usage of -h - will disable password based login, so make sure to copy in a SSH public key.

Install additional packages or edit the contents of the files in the clone as needed before starting the Raspberry Pi.

Repeat the cloning process as needed for as many devices as will be using this setup.

Upgrading

Once it is time to upgrade, update the source code found in /usr/src via git or some other method and then have poudriere upgrade the jail by running:

poudriere jail -j armv7 -u

Next rebuild all the packages:

poudriere bulk -f /root/arm-pkgs -j armv7

Once all the packages are rebuilt, repeat the previous section called ‘Setting up the filesystem layout’ to recreate and repopulate each NFS mountpoint. Once complete the old datasets can be kept around as long as needed and later destroyed once they are not useful any longer.

Additional thoughts

One idea to further improve this setup, is to use remote syslog so that the Raspberry Pis are not logging to NFS.