PXE Booting multiple Raspberry Pi 3s with FreeBSD

    FreeBSD Raspberry Pi ZFS

To make it easier to upgrade the various Raspberry Pis that are around, lets PXE boot them so they can be upgraded easily.

In this article we will use the following tools to create a shared base system and then individual mounts for each RPi:

  • NFS
  • Poudriere
  • PXE
  • TFTP
  • ZFS

To make cable management easier, each Raspberry Pi is connected to a PoE switch using one of these Adafruit PoE Splitter. Using PoE for power also has the additinal benefit of each Rasperry Pi being powered by the UPS that the switch is connected to.

Creating a Poudriere build environment

Setting up Poudriere to build armv7 packages is pretty easy these days.

First, assuming there is a source checkout in /usr/src, create a build jail:

poudriere jail -c -j armv7 -m src=/usr/src -a arm.armv7 -b -K GENERIC

Note: armv7 is 32-bit and is required to access the RPi Camera Module. If the Camera Module is not being used, then arm64 can be used instead.

Build packages

This RPi will be used to run homebridge and some python scripts to monitor a temperature sensor and report data to a MQTT server.

Create a list of packages to build in /root/arm-packages, for this use case the packages list consists of:

net-mgmt/icinga2
net-mgmt/lldpd
www/npm
net/py-paho-mqtt
lang/python
sysutils/tmux
editors/vim-console
shells/zsh

Finally run the build and be prepared for it to take a long time:

poudriere bulk -f /root/arm-pkgs -j armv7

Setting up the RPi

Unfortunatley the official docs do not seem to work on my Raspberry Pi 3s. So instead we can use a small SD Card to configure U-boot to boot off the network.

Add the following line to the bottom of sysutils/u-boot-rpi3/files/rpi3_fragment:

For 32-bit, 0x200000 and for 64-bit use 0x1000000

CONFIG_BOOTCOMMAND="dhcp; bootefi 0x200000"

Rebuild sysutils/u-boot-rpi3.

Create a small msdos partiton, like 64M.

Place the following files from the rpi-firmware & u-boot-rpi3 packages in the partition:

bcm2710-rpi-3-b.dtb
overlays/*
u-boot.bin
bootcode.bin

Create a config.txt on the SD Card that contains the following:

init_uart_clock=3000000
enable_uart=1
kernel=u-boot.bin
kernel7=u-boot.bin
dtoverlay=mmc
dtoverlay=pi3-disable-bt

Place the SD Card into the RPi.

Setting up DHCP

host garagepi {
	filename "loader.efi";
	option root-path "/usr/pxeroot/garagepi"
	fixed-address 192.168.1.15;
	next-server 192.168.1.31;
}

Setting up the TFTP environment

Create a new ZFS dataset for the tftp root:

zfs create zroot/usr/pxeroot/tftp

Configure inetd to start tftp using the ZFS dataset that was just created, by finding the commented out tftp entry and modifying it to be:

tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/pxeroot/tftp

Enable the inetd service and start it:

sysrc inetd_enable="YES"
service inetd start

The tftp root must be populated with the loader.efi out of the arm jail and into the tftproot.

Setting up the filesystem layout

Create the ZFS dataset for this version based on the date:

DATE=`date +%Y%m%d-%H%M%S`
zfs create zroot/usr/pxeroot/pxe-${DATE}

Copy the contents of the jail into the NFS export:

cp -r /usr/local/poudriere/jails/armv7/* /usr/pxeroot/pxe-${DATE}/

Note: This is complete copy so that the poudriere jail is independent of what happens with these Raspberry Pis.

Modify the loader.conf to enable the serial console and load the NIC driver:

console="comconsole"

Enable SSH in rc.conf:

sysrc -R /usr/pxeroot/pxe-${DATE}/ sshd_enable="YES"

Configure pkg(8):

mkdir -p /usr/pxeroot/pxe-${DATE}/usr/local/etc/pkg/repos
echo 'FreeBSD: { enabled: no }' >> /usr/pxeroot/pxe-${DATE}/usr/local/etc/pkg/repos/FreeBSD.conf
echo 'local: { url: "http://192.168.1.31/armv7-default" }' >> /usr/pxeroot/pxe-${DATE}/usr/local/etc/pkg/repos/local.conf

Install the packages needed by most or all of the Raspberry Pis:

pkg -r /usr/pxeroot/pxe-${DATE} \
	-R /usr/pxeroot/pxe-${DATE}/etc/pkg \
	-o ABI_FILE=/usr/pxeroot/pxe-${DATE}/usr/lib/crt1.o \
	install zsh lldpd tmux vim

Create a user in the dataset to ssh in as:

pw -R /usr/pxeroot/pxe-${DATE}/ useradd -n brd -c "Brad Davis" -z /usr/local/bin/zsh -m -h -

Note: The usage of -h - will disable password based login, so make sure to copy in a SSH public key.

Create a snapshot of the dataset so it can be cloned:

zfs snapshot zroot/usr/pxeroot/pxe-${DATE}@initial

Finally clone the dataset for the specific Raspberry Pi, in this case called garagepi:

zfs clone zroot/usr/pxeroot/pxe-${DATE}@initial zroot/usr/pxeroot/garagepi

Install additional packages or edit the contents of the files in the clone as needed before starting the Raspberry Pi.

Repeat the cloning process as needed for as many devices as will be using this setup.

Upgrading

Once it is time to upgrade, update the source code found in /usr/src via git or some other method and then have poudriere upgrade the jail by running:

poudriere jail -j armv7 -u

Next rebuild all the packages:

poudriere bulk -f /root/arm-pkgs -j armv7

Once all the packages are rebuilt, repeat the previous section called ‘Setting up the filesystem layout’ to recreate and repopulate each NFS mountpoint. Once complete the old datasets can be kept around as long as needed and later destroyed once they are not useful any longer.

Additional thoughts

One idea to further improve this setup, is to use remote syslog so that the Raspberry Pis are not logging to NFS.

brd