Creating Custom FreeBSD AMIs for Jenkins - Part 1

    FreeBSD Jenkins ec2

Currently one of my customers is using the FreeBSD 13.0-RELEASE AMIs that are provided by the FreeBSD Release Engineering team. While at first these AMIs made our lives easier, as we have grown there are a couple of problems and opportunities for optimization that have become apparent:

  • First, FreeBSD Update on first boot. While I am very supportive of updating the VM with any updates before it is used, I would rather do this on my own schedule where I can test things out and verify everything works as expected and avoid the time it takes to install the updates and reboot
  • Second, Preseeding the images with the tools that are needed for the build. This makes life easier because it prevents the time hit to fetch and install those packages and we know that a network blip or even a dependency failing to build will prevent our builds from working

So lets dive into how such a task can be accompished.

Jenkins node configuration

With the EC2 plugin for Jenkins installed and configured, we will create two new AMIs in the Configure Clouds section of the Jenkins Node configuration page. The reason for having two, is one will be the active build image that is used for normal builds and one will be the test image that will be used when we go rebuild the image to make sure it still works.

To avoid the chicken and egg problem, go look at the FreeBSD Snapshot mailing list archives and find the latest email for the main branch snapshots. At the time of this writing this email is the most recent and for aarch64 in us-east-2 the latest AMI is ami-0d08e33c697335cd0. So both AMIs in Jenkins will be using this AMI to begin with as a starting point.

The first one will be called FreeBSD-main-aarch64 and note the name is very important since we will reference that in the other build job.

Here is a screenshot of what the AMI configuration should look like:

FreeBSD-main-aarch64 AMI Configuration

And the second, which is almost exactly the same as the first, with only minor tweaks to the name and labels:

FreeBSD-main-aarch64-test AMI Configuration

Note: The Security Groups will need to be changed for your environment.

The labels for these are also important as they control which one will be used for each job role.

Jenkins job configuration

Now this is where things get a bit more interesting. In these jobs we will do a build and then use a bit of groovy code to modify the Jenkins configuration itself. Why you ask, well to update the AMI ID so that after the build job is successful the test AMI can be updated and our tests can validate that it works as expected.

The job will be done in multiple phases:

  1. Clone the repo
  2. Inject environment variables from the clone phase
  3. Execute system Groovy script to update the build description
  4. Build the FreeBSD world, kernel and packages and upload the new AMI
  5. Inject environment variables with the new AMI ID
  6. Update the FreeBSD-main-test-aarch64 AMI with the new AMI ID

So create a new Freestyle Project job in Jenkins and name it something like new-base-AMI-build.

Cloning the repo

In this job I want to be able to leverage a copy of the git repo stored in S3, so it will not use Jenkins git configuration setup, but as that is not relevant to this example we will skip over that for now. Just make sure to clone the repo to a sub directory called src.

Inject environment variables from the clone phase

Under the Build section choose Add build step and select the type to be Execute shell.

cd src
echo "GIT_COMMIT=$( git rev-parse HEAD )" >> ../gitenv.txt

Execute system Groovy script to update the build description

Then choose Add build step and select the type to be Inject environment variables. Set the Properties File Path to match the line above gitenv.txt and leave the Properties Content section blank.

Then choose Add build step and select the type to be Execute system Groovy script. Select Groovy command from the first drop down, and set the Groovy Script to:

build.setDescription(build.envVars.containsKey('GIT_COMMIT') ? build.envVars.GIT_COMMIT : "")

This will populate the description field of each build with the Git hash to make it easier to track down breakages.

Building the FreeBSD image

This section is where the real work is done.

Create an S3 bucket with a name like engineering-ami-upload-temp-storage and give it a policy to delete any files after a day so that bsdec2-image-upload has a temporary place to put the multipart uploaded pieces of the AMI.

So under the Build section choose Add build step and select the type to be Execute shell and add the following:

JOBS=$( sysctl -n hw.ncpu )
if [ ${ARCH} = "amd64" ]; then
	T=amd64.amd64
else
	T=arm64.aarch64
	EXTRA_ARGS="--arm64"
fi
KERNCONF=GENERIC-NODEBUG

BOOTSTRAP_PKGS="jq bsdec2-image-upload poudriere-devel"
for i in ${BOOTSTRAP_PKGS}; do
	if ! pkg info ${i}; then
		pkg install -y ${i}
	fi
done
pkg info

echo "KERNCONF=${KERNCONF}" > /etc/make.conf
echo 'WITHOUT_DEBUG_FILES=yes' >> /etc/make.conf
echo 'WITHOUT_DEBUG_FILES=yes' >> /usr/local/etc/poudriere.d/src.conf
echo "ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" > keyfile
echo "ACCESS_KEY_SECRET=${AWS_SECRET_ACCESS_KEY}" >> keyfile
export AWS_DEFAULT_REGION=us-east-2

# setup poudriere
echo 'ZPOOL=scratch' >> /usr/local/etc/poudriere.conf
echo 'ALLOW_MAKE_JOBS=yes' >> /usr/local/etc/poudriere.conf
echo 'ALLOW_MAKE_JOBS_PACKAGES="pkg ccache cmake gcc* llvm* openjdk* py*"' >> /usr/local/etc/poudriere.conf
mkdir -p /usr/ports/distfiles

poudriere jail -c -v main -j ${ARCH} -K ${KERNCONF} -m src=${WORKSPACE}/src

# populate ports repo from copy stashed in S3
aws s3 cp s3://engineering-build-freebsd-obj-tarballs/FreeBSD-ports.tz . --no-progress
poudriere ports -c -m null
zfs create scratch/poudriere/ports/default
zfs list
tar -C /usr/local/poudriere/ports/default --strip-components 1 -xf FreeBSD-ports.tz
ls -al /usr/local/poudriere/ports/default/
( cd /usr/local/poudriere/ports/default; git branch --set-upstream-to=origin/main main; git pull )
#poudriere ports -c -m git+https -U https://git.freebsd.org/ports.git -B main

# Base pkgs
PKGS="${PKGS} sysutils/ec2-scripts"
PKGS="${PKGS} net/isc-dhcp44-client"
[ ${ARCH} = "amd64" ] && PKGS="${PKGS} sysutils/amazon-ssm-agent"

# packages we need most or all of the time, so inject them in
PKGS="${PKGS} devel/awscli"
PKGS="${PKGS} net/bsdec2-image-upload"
PKGS="${PKGS} net-mgmt/collectd5"  # For gathering storage usage
PKGS="${PKGS} devel/git"
PKGS="${PKGS} textproc/jq"
PKGS="${PKGS} www/nginx"
PKGS="${PKGS} java/openjdk12"
PKGS="${PKGS} ports-mgmt/poudriere-devel"
PKGS="${PKGS} net/rsync"
PKGS="${PKGS} security/sudo"
PKGS="${PKGS} sysutils/tmux"
PKGS="${PKGS} editors/vim"
PKGS="${PKGS} shells/zsh"

echo 'OPTIONS_UNSET=  ALSA AVAHI FONTCONFIG X11 DBUS GNUTLS IPPTOOL LIBPAPER NLS' >> /usr/local/etc/poudriere.d/make.conf
echo 'net-mgmt_collectd5_SET=  RRDTOOL CGI' >> /usr/local/etc/poudriere.d/make.conf
echo 'net-mgmt_collectd5_UNSET=  GCRYPT' >> /usr/local/etc/poudriere.d/make.conf

# build packages
poudriere bulk -j ${ARCH} -b latest ${PKGS}

ls -al /usr/local/poudriere/data/packages/${ARCH}-default

# build image
(
cd src
env WITH_CLOUDWARE=yes PACKAGESITE="file:///usr/local/poudriere/data/packages/${ARCH}-default" PKGS="${PKGS}" make -C release cw-ec2
)

echo
echo 'disk usage'
df -h
zfs list
echo

# Upload
/usr/local/bin/bsdec2-image-upload \
	--sriov \
	--ena \
	${EXTRA_ARGS} \
	/usr/obj/${WORKSPACE}/src/${T}/release/ec2.img \
	"FreeBSD main-${ARCH}-${BUILD_NUMBER}" \
	"FreeBSD/${ARCH} ${GIT_BRANCH}@${GIT_COMMIT}" \
	us-east-2 \
	engineering-ami-upload-temp-storage \
	${WORKSPACE}/keyfile > ${WORKSPACE}/ami

NEWAMI=$( cat ami | awk '{print $6}' )
echo "NEWAMI=${NEWAMI}" > env.txt
echo "ARCH=${ARCH}" >> env.txt
(
	cd src
	echo "GIT_COMMIT=$( git rev-parse HEAD )" >> ../env.txt
)

This should end with a new AMI uploaded EC2 with the name FreeBSD main-${ARCH}-${BUILD_NUMBER}

Inject environment variables with the new AMI ID

Under the Build section select Add build step and choose Inject environment variables.

Set the Properties File Path to match the last line above env.txt and leave the Properties Content section blank.

Update the FreeBSD-main-test-aarch64 AMI with the new AMI ID

Under the Build section select Add build step and choose Execute system Groovy script.

This will modify the test AMI configured in the cloud section of Jenkins so make sure the names line up:

import jenkins.model.*;
import hudson.plugins.ec2.*;

def config = new HashMap()
config.putAll(binding.variables)
def logger = config['out']

def envvars = new HashMap()
envvars.putAll(build.getEnvironment(listener))
def newami = envvars['NEWAMI']
println("AMI from env: " + newami)
def arch = envvars['ARCH']


Jenkins.instance.clouds.each {
  println('cloud: ' + it.displayName)
  if (it.displayName == 'engineering-aws') {
    it.getTemplates().each {
      if (it.description == 'FreeBSD-main-test-' + arch) {
        println('description: ' + it.description)
        println("Current AMI id: " + it.getAmi())
        it.setAmi(newami)
        println("New AMI id: " + it.getAmi())
      }
    }
  }
}

Jenkins.instance.save()

Now once this step has been run, the test AMI is pointing at the new AMI ID that was just uploaded.

Post-build Actions

In the next job the env.txt file will be used to know which AMI ID to use later in the build.

In the Part 2 of this series we will create the test job and if it passes this AMI will be promoted to be the new base AMI ID all other jobs are built on.

brd