Tips to install RPM packages in bootc image builds

Some RPM packages are difficult to install into a Container. This can be a big problem, especially when building a bootc image.

Currently only RPM packages can be installed into bootc images (because only Red Hat based Linux distributions offer a bootc image which include the Linux Kernel).

Often when installing a package into a bootc image, it appears to work. Until you apply the bootc image to a host and attempt to use the new package. I’ve found this is usually because the pre and post RPM install scripts (scriptlets) don’t actually run an important step when installing the RPM.

RPM Scriptlets

I’ve never looked into RPM packaging before now, but from what I’ve learned the quick summary is: Get the RPM file and extract the scripts used before and after the install. Usually this will be a shell (sh) or bash script. Read through it to understand what is supposed to happen. Only the install scripts are important (%pre and %post).

There’s some examples below, but here’s a sample “one-liner” to help extract the RPM scriptlets:

wget https://example.com/package.rpm && rpm -qp --scripts ${_##*/} > ${_##*/}.sh && cat $_.sh

Common Issues when installing RPM packages into bootc images

Linux Penguin with RPM packages boxes in a (bootc) Container

I’ve had a few common issues when installing RPM packages into bootc images. These include:

  1. Missing user or group
  2. Missing application directory when writing to /var
  3. Missing systemd related services or tasks

More details below, along with specific examples and workarounds.

Adding user or group

Be sure to read the bootc documentation for Users and groups. The documentation mentions using systemd-sysusers, sysuser.d, and that’s what I like to use.

Missing application directory when writing to /var

I often have missing directories if the package being installed is trying to create a directory or file in /var. And the Fedora docs explain why under: Semantics of /var and container images
The key point I’ll highlight from the documentation:

Important:

In a container build, you can write to /var. However, this will have a semantic similar to a Dockerfile VOLUME instruction; the content from the container image is only copied at initial install time. Any subsequent system updates will not by default see new changes.

It’s recommended instead to use e.g. systemd tmpfiles.d as a way to ensure that newly added state “reconciles” across upgrades as desired.

So, if you see ANY attempts in the RPM scriptlets to create files or directories under /var it will not work. Manually create them in the Containerfile at build time.

systemd related services or tasks

I’ve also had problems when a package does some kind of check to see if systemd is running. Since systemd won’t be running when building the bootc image, the systemd related service or timer, etc. will not be copied over.

Workaround for this is to copy the related systemd files during the bootc image build.

Examples

Figure out how to get the RPM file. Read the install documentation for the app being installed and find the repository information. For example, the current Zabbix Agent LTS version on RHEL 9 is here.

Extract the RPM Scriptlets

Download the RPM, extract the scriptlets and show the script(s):

wget https://repo.zabbix.com/zabbix/7.0/rhel/9/x86_64/zabbix-agent-7.0.9-release1.el9.x86_64.rpm && rpm -qp --scripts ${_##*/} > ${_##*/}.sh && cat $_.sh

Example: Missing user or group

When viewing the output from the above command, a user and group are trying to be created. Unfortunately this does not work when building the bootc image:

preinstall scriptlet (using /bin/sh):
getent group zabbix > /dev/null || groupadd -r zabbix
getent passwd zabbix > /dev/null || \
        useradd -r -g zabbix -d /var/lib/zabbix -s /sbin/nologin \
        -c "Zabbix Monitoring System" zabbix

postinstall scriptlet (using /bin/sh):
...

To fix, manually do these steps in the bootc image build. Sample Containerfile to build the bootc image:

# other stuff here

RUN \
    <<EOT bash
    set -xeuo pipefail
    cat > /usr/lib/sysusers.d/zabbix-agent.conf << EOF
u zabbix 911 "Zabbix Monitoring System" /var/lib/zabbix /sbin/nologin
EOF
    # Fix zabbix-agent missing folders
    cat > /usr/lib/tmpfiles.d/zabbix-agent-dir-log.conf << EOF
d /var/log/zabbix/ 0755 zabbix zabbix -
EOF
EOT

If you don’t use heredocs in your Containerfiles, you should check it out.

The above is using systemd sysusers.d to create a user at boot time.

Also above is adding a missing folder (another example below with details). Unfortunately in this case, this can only be found by trying to start the zabbix-agent systemd service on the bootc host and looking at the errors from systemctl status zabbix-agent.

Example: Missing application directory when writing to /var

Another example is telegraf, from the RPM scriptlet:

postinstall scriptlet (using /bin/sh):
...
# Set up log directories
LOG_DIR=/var/log/telegraf
mkdir -p $LOG_DIR
chown -R -L telegraf:telegraf $LOG_DIR
chmod 755 $LOG_DIR

STATE_DIR=/var/lib/telegraf
test -d "$STATE_DIR" || {
    mkdir -p "$STATE_DIR"
    chmod 770 "$STATE_DIR"
    chown root:telegraf "$STATE_DIR"
}

STATE_FILE="$STATE_DIR/statefile"
test -f "$STATE_FILE" || {
    touch "$STATE_FILE"
    echo {} > "$STATE_FILE"
    chown root:telegraf "$STATE_FILE"
    chmod 660 "$STATE_FILE"
}

...

This will not work at all. No directories or files are created and definitely not with the desired permissions. This time systemd tmpfiles.d to the rescue. In the Containerfile:

RUN \
    <<EOT bash
    set -xeuo pipefail
    cat > /usr/lib/tmpfiles.d/telegraf-dir.conf << EOF
d /var/log/telegraf 0755 telegraf telegraf -
d /var/lib/telegraf 0700 root telegraf -
f /var/lib/telegraf/statefile 0660 root telegraf -
EOF
EOT

Example: Missing systemd related services or tasks

Also check the files in the RPM package. Is there a systemd service file?

rpm -qlp zabbix-agent-7.0.9-release1.el9.x86_64.rpm

# Output
...
/usr/lib/systemd/system/zabbix-agent.service
...

Yes, there is for zabbix-agent.

For the next example though, the systemd service file is missing when applying the bootc image to the host.

When checking the postinstall for telegraf we see:

...
# Set up systemd service - check if the systemd directory exists per:
# https://www.freedesktop.org/software/systemd/man/sd_booted.html
if [[ -d /run/systemd/system ]]; then
    cp -f /usr/lib/telegraf/scripts/telegraf.service /usr/lib/systemd/system/telegraf.service
    systemctl enable telegraf
    systemctl daemon-reload
fi
...

And that won’t work since /run/systemd/system doesn’t exist when building the bootc image, since systemd isn’t running!
To fix, in the Containerfile add the line containing the command cp.

Further Reading

Another good article is from Red Hat around best practices which also mentions some of the same things.

Conclusion

If you have survived this far, hopefully you now have a working bootc image with all the desired applications working. This is definitely the most painful part I’ve encountered when building bootc images. But once the bootc image works, it just works.. on every server that has this image installed and with automatic updates. If you aren’t convinced this is worth the hassle, check out my first post to get an overview of bootc.

More Posts about bootc