One of my customers needs to have builds done on arm64 hosts, but they lack the necessary hardware to do so in house and procuring such hardware can be very difficult due to vendors EoL some hardware without the replacement hardware being ready to ship. Currently they already have Jenkins configured with a couple of amd64 build agents that do most of the building, so we wanted something we could just drop in to their existing infrastructure. So we started evaluating different options like Equinix Metal (formerly Packet.net) and AWS ec2. Both provide nice options for being able to fire up a host and do a build and then shut it off when the build is complete.
Equinix Metal
Equinix Metal has some decent ARM Server options, but their FreeBSD support is way behind. At minimum we need FreeBSD 13.0 support and while their sales folk say that it is coming, it has been a few months now and nothing yet. Granted in the future we would probably roll our own image, that is not something we wanted to do right away as it would delay getting things working.
EC2
AWS EC2 is the heavy-weight in the room and while I would rather not use them for various reasons, we still have to evaluate them. The c6g instance type offers many different configuration sizes for almost any workload. The pricing is a bit more than what Equinix Metal offers, but since we are just using this for random builds and the machines do not need to be run 24x7 we can leverage the spot pricing which is quite attractive often less than 1/3rd the cost of normal. So with a hefty discount in pricing in hand, lets see what it takes to setup Jenkins to use FreeBSD on EC2.
Jenkins Setup
The first step is to install the EC2 Plugin into the Jenkins instance. Then add the AWS Secret Key and AWS Access Key credentials to the Jenkins credentials store. Then navigate to ‘Manage Jenkins’ -> ‘Manage Nodes and Clouds’ -> ‘Configure Clouds’ and select ‘Add a new cloud’ and choose ‘Amazon EC2’. Give the cloud a name and select the credentials from the dropdown. In most cases using an ‘Alternate EC2 Endpoint’ will not be needed. Pick an AWS Region that is close to you, in this case ‘us-east-2’ is a good choice.
AMI Setup
Select add an AMI and in this case the FreeBSD 13.0 AMI id can be found
in the
13.0 Announce Email.
Make sure to choose the AMI id for the right region and architecture.
In this case for arm64 in us-east-2 the AMI id is:
ami-0426d56d6b3d8a432.
The Instance Type for this build will need to be a pretty powerful
machine, so choose c6g8xlarge from the dropdown menu.
And so we can achieve the cheapest price, check the box for ‘Spot
configuration’ and set the ‘Spot Max bid price’ to something like .5 to
allow a little bit of fluctuation, but not let it go too high.
The AMI Type is a bit obvious that this is a unix type image, but
what is not is the other fields can be left blank.
The Init script is where some interesting things need to happen.
For this case an extra disk is being provided to the VM for running ZFS
on which will be created in a following segment and some required tools
need to be installed such as Java so the Jenkins agent process can start:
pkg install -y openjdk8 git-lite
zpool create -O compression=zstd scratch /dev/nda1
zfs create -o mountpoint=/usr/src scratch/src
zfs create -o mountpoint=/usr/obj scratch/obj
zfs create -o mountpoint=/var/jenkins scratch/jenkins
Click the Advanced button to show the User Data so that the ssh pubic
key can be installed for the root user.
Normally this is bad practice but the FreeBSD builds require root so lets
just go ahead and make it the default.
The User Data script should look something like this:
#!/bin/sh
mkdir /root/.ssh
fetch -o /root/.ssh/authorized_keys http://169.254.169.254/1.0/meta-data/public-keys/0/openssh-key
chmod 600 /root/.ssh/authorized_keys
echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
service sshd restart
Arguably it could be improved to support multiple SSH keys, but for now this works well enough.
In the future we will talk about creating custom AMI images that are preseeded with the tools we need such as the git client and Java to prevent networking or other issues from making our instances come up properly.