Earlier in 2024 Red Hat announced “Image Mode” for Red Hat Enterprise Linux. Currently Image Mode is in “Technology Preview” for RHEL. This is powered by Bootable Containers (or bootc for short). Fedora is also using this new technology, and they have great documentation.

To summarize what bootc does,
from the Fedora Documentation:

The bootc documentation summarizes bootable containers as “transactional, in-place operating system updates using OCI/Docker container images”. In other words, updates to the operating system (OS) are shipped by using container images. That implies that the Linux kernel, the bootloader, drivers, etc. are all part of the container image which renders the container image “bootable”.

These deployments are atomic, meaning they are (mostly) read-only by default.

Updates are easy, just re-build the Container Image and by default the bootc host will update (and reboot) on it’s own.

What makes bootc so exciting?

One Container Image to rule them all.
Sorry, I meant to say: One Container Image to keep every server up to date, automatically.

Every server automatically checks for a new container image. When there’s a new image, the server automatically updates. If the update or reboot fails, no problem. It will “auto-heal”. (there’s a buzz word I haven’t heard in a while!)

There is also no chance of a different package version being installed during a rolling update of many servers (when external repositories are not in your control).

If you need more than one image (if you have many different workloads on your servers then don’t put everything into one image!), it’s pretty simple to have a single “base image” and many specialized images built from that.

What could be better?

• Updates require a reboot.. for any update, any package install.
• There is still possibility for configuration drift in /etc
• Initial install can be difficult.
• Can’t easily add extra (non-root) disk (via the Dockerfile/Containerfile).
• Some packages can’t be installed into a Container.
• No release notes or changelog to see what packages change.
• Day 2 configuration changes are somewhat painful. Redeploying the entire OS isn’t great if you want to be truly immutable.
• Ansible doesn’t detect a bootc server any different from a normal RHEL or Fedora installation. But trying to install a package on a bootc server will obviously fail the Ansible Playbook. This makes re-using Ansible Roles difficult.
• Simple things like user management are not simple considering issues with drift. This means using Ansible for User Management doesn’t really make sense anymore.
• Advanced automatic update mechanism not included, like rolling updates of a 5 node cluster.

Conclusion

I think bootc has the potential for amazing things in 2025 (and beyond). An Atomic server OS that scales. Even with the above somewhat long list of needed improvements, I’m still very excited. Is it too early to start using Bootable Containers in production? I don’t think so.

The post bootc (Bootable Containers): One Container Image to rule them all appeared first on Ryan Daniels.

Problem with running a newer OS (like RHEL 9)

Wow, RHEL 9 is using a modern kernel and toolchain. Ok, now I can continue.

Some newer Linux Operating Systems are moving away from iptables and changing to nftables. This can be a problem when an app is expecting iptables. Normally the OS can load the older iptables, but you can run into problems when running containers.

Error from GitLab Runner

It’s even more fun trying to figure out what’s going on when running on a GitLab Runner.

This is the error from a GitLab Runner job:

ERROR: error during connect: Get https://docker:2376/v1.40/info: dial tcp: lookup docker on 8.8.8.8:53: server misbehaving

Error when running Docker container

This is the error when running a container using Docker:

# docker run --rm -it --privileged --name dind docker:19.03-dind

...
INFO[2023-03-18T21:03:46.764203869Z] Loading containers: start.                   
WARN[2023-03-18T21:03:46.774269934Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2023-03-18T21:03:46.801647539Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
...
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

So here we can see the problem. Our container is trying to use iptables and it’s not working.

But, if you run a newer Docker in Docker (DinD) container, it works! And then the old container also starts to work (this is, until a reboot of the host). The culprit? iptables, and a clever trick in newer DinD containers to workaround this issue..

Enable iptables from inside a Container

If you run a container as privileged, you can actually trigger the host OS to load the iptables kernel modules!

This was found from Docker in Docker (DinD) adding a custom modprobe script (called from the entrypoint script), which is essentially just running this command:

# ip link show ip_tables

The Docker in Docker modprobe script gives credit to:

Enable iptables on Red Hat Enterprise Linux 9

But if your container isn’t using this ‘ip link show ip_tables’ trick, the container will have problems. You need to enable the iptables legacy module on your host OS.

To have iptables loaded and ready to go, you can also run the above trick directly on the host. But the “proper” way is to use modprobe when the OS boots.

# modprobe ip_tables

That will dynamically enable the older iptables. But after a reboot the change is gone, so to make a persistent change:

echo ip_tables > /etc/modules-load.d/ip_tables.conf

Reboot and check:

# lsmod|grep -E "^ip_tables|^iptable_filter|^iptable_nat"
ip_tables              28672  0

Now the older containers will also work (that need iptables (legacy).

Conclusion

Red Hat doesn’t recommend running Docker (instead they recommend Podman). Probably for these reasons. And I wonder if this problem also has something to do with the underlying and undocumented way that Docker uses iptables INPUT chain.

The post Docker and Trouble with Red Hat Enterprise Linux 9: iptables appeared first on Ryan Daniels.