Earlier in 2024 Red Hat announced “Image Mode” for Red Hat Enterprise Linux. Currently Image Mode is in “Technology Preview” for RHEL. This is powered by Bootable Containers (or bootc for short). Fedora is also using this new technology, and they have great documentation.

To summarize what bootc does,
from the Fedora Documentation:

The bootc documentation summarizes bootable containers as “transactional, in-place operating system updates using OCI/Docker container images”. In other words, updates to the operating system (OS) are shipped by using container images. That implies that the Linux kernel, the bootloader, drivers, etc. are all part of the container image which renders the container image “bootable”.

These deployments are atomic, meaning they are (mostly) read-only by default.

Updates are easy, just re-build the Container Image and by default the bootc host will update (and reboot) on it’s own.

What makes bootc so exciting?

One Container Image to rule them all.
Sorry, I meant to say: One Container Image to keep every server up to date, automatically.

Every server automatically checks for a new container image. When there’s a new image, the server automatically updates. If the update or reboot fails, no problem. It will “auto-heal”. (there’s a buzz word I haven’t heard in a while!)

There is also no chance of a different package version being installed during a rolling update of many servers (when external repositories are not in your control).

If you need more than one image (if you have many different workloads on your servers then don’t put everything into one image!), it’s pretty simple to have a single “base image” and many specialized images built from that.

What could be better?

• Updates require a reboot.. for any update, any package install.
• There is still possibility for configuration drift in /etc
• Initial install can be difficult.
• Can’t easily add extra (non-root) disk (via the Dockerfile/Containerfile).
• Some packages can’t be installed into a Container.
• No release notes or changelog to see what packages change.
• Day 2 configuration changes are somewhat painful. Redeploying the entire OS isn’t great if you want to be truly immutable.
• Ansible doesn’t detect a bootc server any different from a normal RHEL or Fedora installation. But trying to install a package on a bootc server will obviously fail the Ansible Playbook. This makes re-using Ansible Roles difficult.
• Simple things like user management are not simple considering issues with drift. This means using Ansible for User Management doesn’t really make sense anymore.
• Advanced automatic update mechanism not included, like rolling updates of a 5 node cluster.

Conclusion

I think bootc has the potential for amazing things in 2025 (and beyond). An Atomic server OS that scales. Even with the above somewhat long list of needed improvements, I’m still very excited. Is it too early to start using Bootable Containers in production? I don’t think so.

The post bootc (Bootable Containers): One Container Image to rule them all appeared first on Ryan Daniels.

Side note: I still use Ansible today, including the first Role I ever created to Manage Users with Ansible.

Fast forward to today, and as a user of Ansible it’s just annoying. Take a minute to think about the ansible-core Version Support Matrix.

And then you need to think about:

• Certain Ansible versions only work on certain Control nodes, because they need to use a certain version of Python.
• Certain Ansible versions only work on certain Managed nodes, because they need to use a certain version of Python.
• Make sure you have the specific version of Python on your Control node to work with the version of Ansible you need. Is that version of Python available on your OS?
• Install and update all the Ansible Collections you need. Which ones?
• Install the dependencies for all your Ansible Collections using Python pip (or whatever you use), because those are separate! Which ones?
• If you are upgrading from an old version of Ansible: fix all your Ansible Playbooks and Ansible Roles to work with the new version of Ansible.

What happened to Ansible?

If you want to manage a Red Hat Enterprise Linux (RHEL) 8 host? Good luck with that, you are basically stuck on Ansible 2.16. At least it appears Ansible 2.16 will stick around for a long time (Until the end of RHEL 8 lifecycle). Ansible 2.16 is the new Ansible 2.9 I guess.. At least until RHEL 11 comes out which will probably use an unsupported version of Python.

I’ve continued using Ansible 2.9 for as long as possible but it is inevitable. Newer hosts will use newer versions of Python, so the upgrade from Ansible 2.9 must begin because Ansible 2.9 doesn’t work with Managed hosts using newer versions of Python. RHEL 10 will be using Python 3.12. So now it is time to start moving to a new version of Ansible.

Ansible 2.16 to the rescue

The only option at this point (for me) is Ansible 2.16. You *just* need to install it, do all the other install requirements, upgrade all your playbooks & roles, etc. Easier said than done, but let’s go. Clean up some technical debt this holiday season!

Cheat sheet to setup Ansible 2.16 on a RHEL 9 Control node

On a RHEL 9 Control node, you need to install a newer version of Python. Then proceed to install all the rest of Ansible’s requirements.
For example:

# On Control node as Ansible user (not root):
sudo dnf install python3.11 python3.11-pip

ln -s /usr/bin/python3.11 ~/.local/bin/python

python3.11 -m pip install -U --user ansible-core<2.17 ansible-lint netaddr jmespath jc

ansible-galaxy collection install ansible.posix
ansible-galaxy collection install ansible.netcommon
ansible-galaxy collection install community.general
ansible-galaxy collection list

You will now be using Ansible 2.16 on a RHEL 9 Control node which should work across all “modern” Linux hosts, even RHEL 7 (which is not modern, but Python 2.7 still works in Ansible 2.16).

Next up, change every single Ansible Playbook and Role and hope for the best. These Porting Guides do not look fun!! Don’t forget to Ansible Lint & syntax check everything.

ansible-lint *.yml

ansible-playbook --syntax-check *.yml

Conclusion – RIP Ansible 2.9

RIP Ansible 2.9
Long live Ansible 2.16

The post Goodbye Ansible 2.9 – Hello Ansible 2.16 appeared first on Ryan Daniels.

Getting an OS package version using Ansible seems like a simple thing to do, except it’s not simple when there’s a dash in the package name. This approach will always work, even for packages with a dash in the name.

Get the OS package version using Ansible

This is using the builtin Ansible module: package_facts.

The example will get the version of the OS package “haproxy”, even if it has a dash. And the template file has logic to use certain configuration if the version of the OS package is greater or equal to version “2.0”. If the version is less than 2.0, then the older configuration will be used.

Example Ansible task:

# tasks/main.yml

- name: Gather the OS package facts
  package_facts:
    manager: auto
  tags:
    - always

- name: show package version (RHEL 7)
  debug:
    # This doesn't work. Doesn't work with a dash in the name
    # var: ansible_facts.packages.rh-haproxy18-haproxy.version
    var: ansible_facts.packages | json_query('"rh-haproxy18-haproxy"[*].version') | join(' ')
  when:
    - ansible_os_family == "RedHat"
    - ansible_distribution_major_version | int == 7
  tags:
    - always

- name: Set fact for haproxy version (RHEL 7)
  set_fact:
    stat_install_haproxy_haproxy_version: "{{ ansible_facts.packages | json_query('\"rh-haproxy18-haproxy\"[*].version') | join(' ') }}"
  when:
    - ansible_os_family == "RedHat"
    - ansible_distribution_major_version | int == 7
    - ansible_facts.packages | json_query('"rh-haproxy18-haproxy"[*].version') | join(' ') is defined
  tags:
    - always

- name: install-haproxy | show haproxy version
  debug: var=stat_install_haproxy_haproxy_version
  tags:
    - always

- name: Generate haproxy config (RHEL7)
  template:
    src: templates/haproxy.cfg.j2
    dest: "{{ install_haproxy_config_dir }}/haproxy.cfg"
    validate: haproxy -f %s -c -q
  environment:
    PATH: "/usr/sbin:/usr/local/sbin:/sbin:{{ install_haproxy_bin_dir }}"
  notify:
    - reload haproxy
  when:
    - ansible_os_family == "RedHat"
    - ansible_distribution_major_version | int == 7
  tags:
    - config

And then a template file which applies different configuration options depending on the package version.

Example Ansible template file:

# templates/haproxy.cfg.j2

# {{ ansible_managed | comment }}

{% if stat_install_haproxy_haproxy_version is version('2.0', '>=') %}
    config here only for version greater or equal to 2.0
{% else %}
    config here for version less than 2.0
{% endif %}

# {{ ansible_managed | comment }}

The above example isn’t complete, some variables need to be defined. But you get the idea.

Conclusion

That’s it. Not exactly straight forward, but the example above should be enough to understand how to accomplish getting an OS package version using Ansible. This even works when there is a dash in the package name.

Want another interesting Ansible example? Check out How to add an Inline Comment using Ansible.

The post Use Ansible to get OS package version (even with a dash) appeared first on Ryan Daniels.

I find this useful for my infrastructure as code configuration with terraform. Each directory is a different environment or site, and I can use different versions of terraform based on the name of the directory.

This is for bash shell on Linux. Tested on bash version 4.3 or newer. Might work on older version too.

Add this code to your .bashrc file:

function terraform () {
  case "${PWD##*/}" in
    *env-2023*) ~/terraform_1.6.2 "$@";;
    *) ~/terraform_0.11.15 "$@";; # default case
  esac
}

Then log out and log back in.

What is this doing?

When you run the command “terraform”, your current directory is checked. If you are in a directory that has “env-2023” anywhere in it, then terraform_1.6.2 will run. Otherwise the (really) old version will run.

Conclusion

With this in your .bashrc file, you can have multiple directory patterns in the case syntax, each running a specific version of an application. I find this helpful when in the process of upgrading terraform config for all of my different environments for my infrastructure as code.

Here are some more Linux related posts.

The post Run different Linux program based on current directory (using Bash shell) appeared first on Ryan Daniels.

Problem with running a newer OS (like RHEL 9)

Wow, RHEL 9 is using a modern kernel and toolchain. Ok, now I can continue.

Some newer Linux Operating Systems are moving away from iptables and changing to nftables. This can be a problem when an app is expecting iptables. Normally the OS can load the older iptables, but you can run into problems when running containers.

Error from GitLab Runner

It’s even more fun trying to figure out what’s going on when running on a GitLab Runner.

This is the error from a GitLab Runner job:

ERROR: error during connect: Get https://docker:2376/v1.40/info: dial tcp: lookup docker on 8.8.8.8:53: server misbehaving

Error when running Docker container

This is the error when running a container using Docker:

# docker run --rm -it --privileged --name dind docker:19.03-dind

...
INFO[2023-03-18T21:03:46.764203869Z] Loading containers: start.                   
WARN[2023-03-18T21:03:46.774269934Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2023-03-18T21:03:46.801647539Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
...
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

So here we can see the problem. Our container is trying to use iptables and it’s not working.

But, if you run a newer Docker in Docker (DinD) container, it works! And then the old container also starts to work (this is, until a reboot of the host). The culprit? iptables, and a clever trick in newer DinD containers to workaround this issue..

Enable iptables from inside a Container

If you run a container as privileged, you can actually trigger the host OS to load the iptables kernel modules!

This was found from Docker in Docker (DinD) adding a custom modprobe script (called from the entrypoint script), which is essentially just running this command:

# ip link show ip_tables

The Docker in Docker modprobe script gives credit to:

Enable iptables on Red Hat Enterprise Linux 9

But if your container isn’t using this ‘ip link show ip_tables’ trick, the container will have problems. You need to enable the iptables legacy module on your host OS.

To have iptables loaded and ready to go, you can also run the above trick directly on the host. But the “proper” way is to use modprobe when the OS boots.

# modprobe ip_tables

That will dynamically enable the older iptables. But after a reboot the change is gone, so to make a persistent change:

echo ip_tables > /etc/modules-load.d/ip_tables.conf

Reboot and check:

# lsmod|grep -E "^ip_tables|^iptable_filter|^iptable_nat"
ip_tables              28672  0

Now the older containers will also work (that need iptables (legacy).

Conclusion

Red Hat doesn’t recommend running Docker (instead they recommend Podman). Probably for these reasons. And I wonder if this problem also has something to do with the underlying and undocumented way that Docker uses iptables INPUT chain.

The post Docker and Trouble with Red Hat Enterprise Linux 9: iptables appeared first on Ryan Daniels.

For example, in a Linux config file like /etc/hosts you can have Ansible add a comment block:

#
# This is for my NAS
#
192.168.1.100 nas

But, what if you want to have the comment at the end of the line (an inline comment)?

192.168.1.100 nas # This is for my NAS

Ansible Comment

Adding a comment with Ansible is easy.
And when adding a comment in a Linux config file, you need to use the # character.
This can be accomplished using Ansible’s “comment filter“.

ansible_variable1: "192.168.1.100 nas {{ 'This is for my NAS' | comment }}"

The result will be:

TASK [Print normal comment] ********************************************************************************************
ok: [localhost] => {
    "msg": "#\n# This is for my NAS\n#\n192.168.1.100 nas"
}

And when using something like the Ansible Module lineinfile, you would get:

#
# This is for my NAS
#
192.168.1.100 nas

Ansible Inline Comment

Adding an inline comment with Ansible isn’t obvious. You can’t just add the # character as part of the text in the Ansible yaml. Because that’s actually a comment in the yaml file.
You need to use Ansible’s “comment filter“, like a normal comment, however you need to change a few options.

Modify the prefix and postfix options for the comment filter:

ansible_variable2: "192.168.1.100 nas {{ 'This is for my NAS' | comment('plain', prefix='', postfix='') }}"

The result will be:

TASK [Print inline comment] ********************************************************************************************
ok: [localhost] => {
    "msg": "192.168.1.100 nas # This is for my NAS\n"
}

And when using something like the Ansible Module lineinfile, you would get:

192.168.1.100 nas # This is for my NAS

Demo

Here is a demo Ansible Playbook to test with.

---
- hosts: localhost
  name: Ansible Inline Comment Demo
  vars:
    my_other_var: "nas"
    ansible_variable1: "{{ 'This is for my NAS' | comment }}\n192.168.1.100 {{ my_other_var }}"
    ansible_variable2: "192.168.1.100 {{ my_other_var }} {{ 'This is for my NAS' | comment('plain', prefix='', postfix='') }}"

  tasks:
    - name: Print normal comment
      debug:
        msg: "{{ ansible_variable1 }}"

    - name: Print inline comment
      debug:
        msg: "{{ ansible_variable2 }}"

If you want to get started with Ansible, check out my post: Getting Started with Ansible. It has some Ansible basics that can help you get started.

The post Ansible Quick Tip: How to add an Inline Comment appeared first on Ryan Daniels.

What is Helm?

From the Helm website: “Helm helps you manage Kubernetes applications”.
If you use Kubernetes, there’s a good chance you will be using at least some portion of Helm. Helm has many features, and is most well known for Helm Charts. You can (and should) also use Helm to deploy your application to Kubernetes.

Helm Post-Renderer

The Helm Post-Renderer feature allows Helm to deploy an application using advance configurations using tools like Kustomize.

Windows Batch Script with Helm Post-Renderer and WSL

I want my Kubernetes deployment (using Helm and Kustomize) to be able to work on my laptop, and also in a GitLab CI/CD pipeline. And because my Windows laptop has WSL (no WSL2!) I can do a few cool Linux things with it, like directly run Helm commands (Maybe WSL2 also has this problem). If only I could use a real Linux laptop at work, but anyways..

The Helm Post-Renderer gives an error when running a Linux Bash script in WSL. So much to my amazement, in 2021 I needed to write a Windows Batch Script!

Error message in WSL

Running a Helm install/upgrade gives the error:

helm upgrade --install release123 my-charts/app123-charts --version=0.1.0 --namespace=dev1 --values=../values.yaml --post-renderer ./kustomize.sh --debug --dry-run

## Error: error while running post render on files: error while running command \\C\Users\username\helm\my-charts\app123-charts\app-kustomize\kustomize-post-render\kustomize.sh. error output:
## : fork/exec \\C\Users\username\helm\my-charts\app123-charts\app-kustomize\kustomize-post-render\kustomize.sh: %1 is not a valid Win32 application.

Just the error:

%1 is not a valid Win32 application.

Okay, that’s interesting. Running a helm command gives me a Windows related error. So now we need to do it the Windows way.

Helm Post-Renderer Windows Batch Script

This Windows Batch Script will work from Windows (kustomize.bat):

@echo off

REM fix mapped drive in TS: https://stackoverflow.com/questions/9013941/how-to-run-batch-file-from-network-share-without-unc-path-are-not-supported-me/34182234#34182234
@pushd %~dp0

REM findstr is pure windows command: https://superuser.com/questions/853580/real-windows-equivalent-to-cat-stdin/1425671#1425671
FINDSTR . > all.yaml

kustomize build . && del all.yaml || exit 1

@popd

The above Batch Script also works from a Windows Terminal Server that is mapping a drive.

Helm Post-Renderer Linux Bash script

For reference, the same in Linux would be (kustomize.sh):

#!/bin/bash

cat <&0 > all.yaml

kustomize build . && rm all.yaml

Above Linux bash script source: Taylor Thomas.

Conclusion

With this Windows Batch Script, you can run Helm install/upgrade commands directly on your Windows laptop so you can deploy to your local or remote Kubernetes cluster. This can help to debug your true pipeline deployments, if using something like GitLab CI/CD since it’s now possible to deploy using the same method.

The post Helm Post-Renderer and an ugly Windows Batch Script appeared first on Ryan Daniels.

Let’s discuss the background of firewall issues with Docker, and a working solution for my use case (either setup manually or using Ansible). By the end we will use a firewall on the server to lock down everything by default, only allowing my trusted IPs! With the option to open specified ports publicly (like SSH).

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Background

Firstly, even if you were using a firewall like iptables, Docker makes that useless. Docker punches a whole right through your firewall!

And if you try another firewall, like firewalld? Docker Swarm (and even regular Docker) with firewalld is a complete mess. Restart the firewalld service, or change the firewalld config, and you lost all the config that Docker needed. Now you have to restart Docker! What happens if the firewalld service failed and restarted? That Docker Swarm node is out of service.

Keep in mind, when I mention Docker, it means the regular Docker Engine. Docker Swarm means SwarmKit, which is the newer way of using Swarm. (The old way is the standalone solution which is old and not referenced at all here).

There are many attempts to solve this problem by users of Docker. Unfortunately the Docker team has been pretty quiet about these issues. They recommend a manual user solution, and to disable Docker’s use of iptables. I’m speculating here, but it seems like any future change from Docker will likely be a breaking change since this is a complicated issue to fix.

The attempted solutions (from users) are not very straight forward for a normal Docker user. And that’s the real issue. On top of that, out of the box (after you start one service with a published port), there is no security at the network layer. Anyone can connect to an exposed container’s port. Most importantly, even if you think you have a firewall protecting you.. Wrong, you don’t! With normal Docker, you can bind your service to your localhost which helps. But what about Docker Swarm? Nope, that doesn’t work.

There’s a great article about “The problem of forcing users to make choices (in security).” Definitely worth the read!

Roll your own solution

Until this is actually addressed in Docker, our only hope is to find the simplest solution possible. And turning off iptables integration in Docker is unacceptable (which is constantly recommended by the Docker developers). The other option is to move on from Docker and/or Docker Swarm. I hear this thing called Kubernetes is pretty great. Anyways, back to Docker.
Many people have spent hours trying to learn, and figure out iptables as a solution to this. You need to roll your own solution apparently! So iptables is probably the best approach, since that is what Docker needs to use to do it’s magic (and most other firewalls are just a wrapper for iptables anyway depending on your OS).

Something fun I found out while testing this, Docker Swarm uses iptables in an undocumented way. Docker Swarm uses the iptables INPUT chain! It’s only for encrypted overlay networks. But it’s not very fun realizing that! All of a sudden rules are being appended to the INPUT chain.

Okay, enough backstory. On with my futile attempt to roll my own solution. This took way longer than I thought it would! But it does work! Currently it works at least.. (That’s why I use the word futile!)

Problems I need to solve

1. Only allow traffic from multiple “trusted” IP addresses to my servers. Not all of these IPs will be in the same “IP block/range” either. This will be to all services running directly on the server, and also all of the Docker containers.
2. Let only specific ports be publicly accessible, like SSH.
3. I’m not managing which containers are accessible through the firewall. Meaning, I’m not manually adding ports into my firewall solution. That kind of manual work is not happening. I need a dynamic, and flexible solution that blocks by default except to my trusted IPs.
4. The firewall solution must be simple. More complex means more room for error.
5. The firewall solution must not impact performance significantly.
6. Restarting the firewall won’t break Docker.
7. Restarting Docker won’t break the firewall.
8. No impact to running server processes or Docker services when making a change. Things need to keep working!
   Firewall changes need to happen online and not impact Docker. Meaning I can’t be restarting Docker because I made a firewall change.
   Docker “changes” need to happen online. Meaning I can’t be restarting the firewall because I made a Docker change. (A Docker “change” means starting/stopping a container).

That sounds very simple! Unfortunately, it is not with Docker (and Docker Swarm).

Solution – Docker firewall with iptables and ipset

If you don’t know much about iptables, or ipset, that’s okay. You don’t really need to know. You should have some basic understandings though, so you don’t break your servers! The Arch Linux wiki has great information about iptables. Including this helpful visual about the iptables flow.

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

High level summary

iptables with ipset will handle all of this for us. And keep our servers, and Docker locked down from the network level.

In this solution, we will use the iptables INPUT chain to jump to another chain (let’s call our custom chain FILTERS), but return if there’s some legitimate looking traffic, so the Swarm overlay can do whatever it wants in INPUT with IPSEC, or whatever it is appending to INPUT.
Inside our custom chain FILTERS, we drop everything that doesn’t match our trusted list of IPs. We also allow our SSH port and the basic default iptables stuff.. You can also add any OS port to be publicly accessible.
The DOCKER-USER chain only needs a few entries. Any internal Docker traffic is returned, and it will drop any other traffic that’s not in our allowed IP list. You can also add any container port to be publicly accessible.

One of the dangers with this approach is if Docker changes it’s behaviour our firewall could break, or our Docker services could stop working. Since Docker doesn’t offer any solution for their users, we need our own solution. So keep in mind that you need to test this when upgrading to a new version of Docker. That is the trade-off with a “roll your own” solution. But what choice do we have?

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

Warnings

Warning: Be sure you have everything needed in your configuration. Once the iptables firewall is started it blocks anything that wasn’t added! Don’t lock yourself out of your server. Be sure to have another way to connect, like a console.

Disclaimer: Keep in mind, you should test all of this in your lab or staging environments. I can’t guarantee this will be 100% safe and can’t be held responsible for anything going wrong!

SELinux Bug: If using SELinux, currently there’s a bug with SELinux which prevents saving the iptables rules to the iptables.save file.
Impact: Saving the iptables rules a 2nd time will silently fail. Workaround has been added so SELinux allows chmod to interact with the iptables.save file. See notes on GitHub for SELinux workaround steps. Alternatively you could disable SELinux, but that’s not recommended. Bug report: https://bugs.centos.org/view.php?id=12648

The manual way

Run the commands below. These commands are only for CentOS/RHEL 7. If you don’t want to do this manually, jump to the Automatic section, using Ansible (which also works with Ubuntu).

Prep

Make note of what you already have in iptables (if you are already using it). Be sure you have some background with iptables, since you could break things!

# iptables -nvL --line-numbers

Install the required packages for CentOS / RHEL:

# yum install iptables iptables-services ipset ipset-service

Configure ipset

ipset allows you to add a list of IPs that you can use with iptables. In our case, we will add a list of IPs we want to be able to connect to our servers.

Configure ipset with a setname of ip_allow.
Add IPs you want to allow. Change the IPs below to your actual trusted/allowed IP ranges. Be sure to include your Docker server IPs here, because if you don’t they can’t communicate with eachother:

# mkdir -p /etc/sysconfig/ipset.d
# vi /etc/sysconfig/ipset.d/ip_allow.set

create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
add ip_allow 192.168.1.123
add ip_allow 192.168.100.0/24
add ip_allow 192.168.101.0/24
add ip_allow 192.168.102.0/24

Start, and Enable the ipset service:

# systemctl status ipset
# systemctl start ipset
# systemctl enable ipset

See what ipset has in it’s loaded configuration:

# ipset list | head

Important: Make note of the size of “Number of entries”. If that number is close to the maxelem size (65536), then you need to delete the ipset and re-create it with a larger max size. If you only use a few IP ranges like above, you don’t need to worry and will be well below the limit.

Configure iptables

Next up, iptables. iptables is our solution for a firewall. We will create a file with our rules and then add those rules into iptables. The important part is to not flush the existing rules if you are already using Docker (or Docker Swarm) on your server.

Create an iptables file to use with iptables-restore, to add the rules into iptables:

# vi iptables-rules.txt

Add below to the file. There is a lot going on here..

*filter
:DOCKER-USER - [0:0]
:FILTERS - [0:0]
#Can't flush INPUT. wipes out docker swarm encrypted overlay rules
#-F INPUT
#Use ansible or run manually once instead to add -I INPUT -j FILTERS
#-I INPUT -j FILTERS
-F DOCKER-USER
-A DOCKER-USER -m state --state RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -i docker_gwbridge -j RETURN
-A DOCKER-USER -s 172.18.0.0/16 -j RETURN
-A DOCKER-USER -i docker0 -j RETURN
-A DOCKER-USER -s 172.17.0.0/16 -j RETURN
#Below Docker ports open to everyone if uncommented
#-A DOCKER-USER -p tcp -m tcp -m multiport --dports 8000,8001 -j RETURN
#-A DOCKER-USER -p udp -m udp -m multiport --dports 9000,9001 -j RETURN
-A DOCKER-USER -m set ! --match-set ip_allow src -j DROP
-A DOCKER-USER -j RETURN
-F FILTERS
#Because Docker Swarm encrypted overlay network just appends rules to INPUT. Has to be at top unfortunately
-A FILTERS -p udp -m policy --dir in --pol ipsec -m udp -m set --match-set ip_allow src --dport 4789 -j RETURN
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -p icmp -j ACCEPT
-A FILTERS -i lo -j ACCEPT
#Below OS ports open to everyone if uncommented
-A FILTERS -p tcp -m state --state NEW -m tcp -m multiport --dports 22 -j ACCEPT
#-A FILTERS -p udp -m udp -m multiport --dports 53,123 -j ACCEPT
-A FILTERS -m set ! --match-set ip_allow src -j DROP
-A FILTERS -j RETURN
COMMIT

Use iptables-restore to add the above rules into iptables. The very important flag is -n. That makes sure we don’t flush the iptables rules if we have rules already in Docker (or Docker Swarm).

# iptables-restore -n < iptables-rules.txt

Next, add a rule to the INPUT chain, so we start using the new rules in FILTERS. It has to be at the top, and only needs to be added once:

# iptables -I INPUT 1 -j FILTERS

Save the iptables rules:

# /usr/libexec/iptables/iptables.init save

That will save any existing and our new iptables rules to the iptables configuration file so it will be persistent after a reboot.

In addition, it was needed to run the above iptables command manually since we want to ensure it’s only inserted once. And we can’t flush the INPUT chain to ensure that since Docker Swarm could have rules there already.

Start and Enable the iptables service:

# systemctl status iptables
# systemctl start iptables
# systemctl enable iptables

If you want to customize the iptables rules to allow more ports to be open to everyone, just add the port to the appropriate rule in the iptables file (tcp or udp), then re-run the same commands from above:

# iptables-restore -n < iptables-rules.txt
# /usr/libexec/iptables/iptables.init save

Don’t miss the Warnings from above! Especially about SELinux.

If you don’t want to do all of that manually, and you use Ansible, then do this instead..

The Automatic way with Ansible

Manually run all of the above, on every Docker server is not ideal. Let’s use Ansible instead!

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

This works on CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Install the Ansible Role using Ansible Galaxy:

$ ansible-galaxy install ryandaniels.iptables_docker

Or, clone the GitHub project:

$ git clone https://github.com/ryandaniels/ansible-role-iptables-docker.git roles/ryandaniels.iptables_docker

Create the Ansible Playbook, called iptables_docker.yml:

---
- hosts: '{{ inventory }}'
  become: yes
  vars:
    # Use this role
    iptables_docker_managed: true
  roles:
  - ryandaniels.iptables_docker

Make configuration changes to add desired IP addresses and ports as needed.

Don’t miss the Warnings from above!

Then run the playbook:

$ ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7" -i hosts-dev

Conclusion

In conclusion, now we have secured our Docker (and Docker Swarm) environments using Ansible to perform the installation and configuration of iptables! None of our Docker published ports are exposed to the world, unless we want them to be! We have created a custom Docker firewall with iptables. Hopefully, some day this will be the default behaviour and shipped with Docker out of the box! Dare to dream. Security is hard.

References

Background for Docker’s undocumented use of iptables INPUT chain

See my previous post about this.

References and links

References, notes, and links about the Docker firewall discussion:

• Docker Documentation – Overlay Networks
• Docker bypasses ufw firewall rules
• unrouted – Solution using iptables. Not for Swarm. It clobbers the INPUT chain, which is used by encrypted overlay with Docker Swarm
• The big thread about Docker and a firewall
• Arch Linux wiki to the rescue to show iptables flow which links to a great visual
  
  

The post Secure Docker with iptables firewall and Ansible appeared first on Ryan Daniels.

I fixed it by running one command:

$ sudo apt install cinnamon-desktop-environment

After it’s installed, reboot your computer. Then, before logging in, click the gear icon and change your session to Cinnamon.

Now everything is so much better!

Cinnamon Desktop

This will install the Cinnamon Desktop Environment for you. Which was initially developed for Linux Mint.

The main downside with this approach is you’ll probably be using this version of Cinnamon Desktop for a while. Packages don’t update very often in Ubuntu. However, in general not being on the newest of the new does bring stability.

If you still use Windows you are missing out. You should think about switching to Linux.

It’s amazing! Give it a try.

The post How to fix Ubuntu 20.04 in 1 step appeared first on Ryan Daniels.

I could find pretty much nothing about this, anywhere.

Wait, let’s take a step back. What am I talking about?

I was testing iptables with Docker. What a nightmare. Anyways, I noticed something strange with my three node Docker Swarm test VMs. Some very strange rules at the top of the iptables INPUT chain.

# iptables -nvL INPUT
Chain INPUT (policy ACCEPT 33477 packets, 8115K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100600"
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100600"
   21  3669 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100100"
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100100"
    1   101 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100200"
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100200"

What is that? Some kind of rules using a Docker port, and something about IPSEC.

Searching the internet has no information about this, that I could find at least. Only some random Docker issues on GitHub. It does have some similar details in the INPUT chain though.

Also found were two references to the Docker documentation.

The Docker documentation does have a pretty good section about iptables. But no mention of the INPUT chain. They very specifically say they only modify the DOCKER-USER and DOCKER chains in iptables.

There’s also some information about the overlay network in the Docker documentation, but unfortunately it’s also very high level. No details about the Docker magic that makes all the networking work seamlessly! However, it does mention using IPSEC. Time to do some testing.

When testing in Docker Swarm, by starting containers which use an encrypted overlay I get those iptables rules in the INPUT chain. Fun!

Conclusion

Not a very exciting mystery. But very unexpected that Docker is modifying the INPUT chain in iptables! That changes a few thing when trying to secure Docker.. Hopefully other people weren’t relying on having the last rule in the INPUT chain.

Only Docker knows how their next version of Docker will behave. Anything a user does to try and secure Docker at the network layer is futile.

That being said, I’ve attempted to secure Docker with an iptables firewall. Check it out!

The post Docker and the iptables INPUT chain appeared first on Ryan Daniels.