Please, no:

RUN dnf upgrade -y && \
    dnf install -y package123 && \
    rm -rf /var/cache/dnf/* && \
    dnf clean all

All those “&& \” over and over.. But the bigger issue for me is figuring out where something broke. When there’s an error the whole block of code is shown and no indication where the problem actually is!

Heredoc inside a Dockerfile / Containerfile to save the day:

RUN <<EOF
    set -ex
    dnf upgrade -y
    dnf install -y package123
    rm -rf /var/cache/dnf/*
    dnf clean all
EOF

Or even better, specify bash shell if the image has it, with pipefail:

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN <<EOF bash
    set -ex
    # Do stuff
EOF

What is a heredoc?

As described by Wikipedia:
here document (here-document, here-text, heredoc, hereis, here-string or here-script) is a file literal or input stream literal: it is a section of a source code file that is treated as if it were a separate file. The term is also used for a form of multiline string literals that use similar syntax, preserving line breaks and other whitespace (including indentation) in the text.
The most common syntax for here documents, originating in Unix shells, is << followed by a delimiting identifier (often the word EOF or END), followed, starting on the next line, by the text to be quoted, and then closed by the same delimiting identifier on its own line. This syntax is because here documents are formally stream literals, and the content of the here document is often redirected to stdin (standard input) of the preceding command or current shell script/executable.
The here document syntax is analogous to the shell syntax for input redirection, which is < followed by the name of the file to be used as input.

More information about heredocs is in Docker’s announcement. Podman also supports heredocs.

Conclusion

Writing long scripts in a Dockerfile / Containerfile is painful. Use heredocs instead to make life easier.

This will also help if you start using bootc!

The post heredoc in Dockerfile appeared first on Ryan Daniels.

Problem with running a newer OS (like RHEL 9)

Wow, RHEL 9 is using a modern kernel and toolchain. Ok, now I can continue.

Some newer Linux Operating Systems are moving away from iptables and changing to nftables. This can be a problem when an app is expecting iptables. Normally the OS can load the older iptables, but you can run into problems when running containers.

Error from GitLab Runner

It’s even more fun trying to figure out what’s going on when running on a GitLab Runner.

This is the error from a GitLab Runner job:

ERROR: error during connect: Get https://docker:2376/v1.40/info: dial tcp: lookup docker on 8.8.8.8:53: server misbehaving

Error when running Docker container

This is the error when running a container using Docker:

# docker run --rm -it --privileged --name dind docker:19.03-dind

...
INFO[2023-03-18T21:03:46.764203869Z] Loading containers: start.                   
WARN[2023-03-18T21:03:46.774269934Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2023-03-18T21:03:46.801647539Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
...
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

So here we can see the problem. Our container is trying to use iptables and it’s not working.

But, if you run a newer Docker in Docker (DinD) container, it works! And then the old container also starts to work (this is, until a reboot of the host). The culprit? iptables, and a clever trick in newer DinD containers to workaround this issue..

Enable iptables from inside a Container

If you run a container as privileged, you can actually trigger the host OS to load the iptables kernel modules!

This was found from Docker in Docker (DinD) adding a custom modprobe script (called from the entrypoint script), which is essentially just running this command:

# ip link show ip_tables

The Docker in Docker modprobe script gives credit to:

Enable iptables on Red Hat Enterprise Linux 9

But if your container isn’t using this ‘ip link show ip_tables’ trick, the container will have problems. You need to enable the iptables legacy module on your host OS.

To have iptables loaded and ready to go, you can also run the above trick directly on the host. But the “proper” way is to use modprobe when the OS boots.

# modprobe ip_tables

That will dynamically enable the older iptables. But after a reboot the change is gone, so to make a persistent change:

echo ip_tables > /etc/modules-load.d/ip_tables.conf

Reboot and check:

# lsmod|grep -E "^ip_tables|^iptable_filter|^iptable_nat"
ip_tables              28672  0

Now the older containers will also work (that need iptables (legacy).

Conclusion

Red Hat doesn’t recommend running Docker (instead they recommend Podman). Probably for these reasons. And I wonder if this problem also has something to do with the underlying and undocumented way that Docker uses iptables INPUT chain.

The post Docker and Trouble with Red Hat Enterprise Linux 9: iptables appeared first on Ryan Daniels.

Let’s discuss the background of firewall issues with Docker, and a working solution for my use case (either setup manually or using Ansible). By the end we will use a firewall on the server to lock down everything by default, only allowing my trusted IPs! With the option to open specified ports publicly (like SSH).

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Background

Firstly, even if you were using a firewall like iptables, Docker makes that useless. Docker punches a whole right through your firewall!

And if you try another firewall, like firewalld? Docker Swarm (and even regular Docker) with firewalld is a complete mess. Restart the firewalld service, or change the firewalld config, and you lost all the config that Docker needed. Now you have to restart Docker! What happens if the firewalld service failed and restarted? That Docker Swarm node is out of service.

Keep in mind, when I mention Docker, it means the regular Docker Engine. Docker Swarm means SwarmKit, which is the newer way of using Swarm. (The old way is the standalone solution which is old and not referenced at all here).

There are many attempts to solve this problem by users of Docker. Unfortunately the Docker team has been pretty quiet about these issues. They recommend a manual user solution, and to disable Docker’s use of iptables. I’m speculating here, but it seems like any future change from Docker will likely be a breaking change since this is a complicated issue to fix.

The attempted solutions (from users) are not very straight forward for a normal Docker user. And that’s the real issue. On top of that, out of the box (after you start one service with a published port), there is no security at the network layer. Anyone can connect to an exposed container’s port. Most importantly, even if you think you have a firewall protecting you.. Wrong, you don’t! With normal Docker, you can bind your service to your localhost which helps. But what about Docker Swarm? Nope, that doesn’t work.

There’s a great article about “The problem of forcing users to make choices (in security).” Definitely worth the read!

Roll your own solution

Until this is actually addressed in Docker, our only hope is to find the simplest solution possible. And turning off iptables integration in Docker is unacceptable (which is constantly recommended by the Docker developers). The other option is to move on from Docker and/or Docker Swarm. I hear this thing called Kubernetes is pretty great. Anyways, back to Docker.
Many people have spent hours trying to learn, and figure out iptables as a solution to this. You need to roll your own solution apparently! So iptables is probably the best approach, since that is what Docker needs to use to do it’s magic (and most other firewalls are just a wrapper for iptables anyway depending on your OS).

Something fun I found out while testing this, Docker Swarm uses iptables in an undocumented way. Docker Swarm uses the iptables INPUT chain! It’s only for encrypted overlay networks. But it’s not very fun realizing that! All of a sudden rules are being appended to the INPUT chain.

Okay, enough backstory. On with my futile attempt to roll my own solution. This took way longer than I thought it would! But it does work! Currently it works at least.. (That’s why I use the word futile!)

Problems I need to solve

1. Only allow traffic from multiple “trusted” IP addresses to my servers. Not all of these IPs will be in the same “IP block/range” either. This will be to all services running directly on the server, and also all of the Docker containers.
2. Let only specific ports be publicly accessible, like SSH.
3. I’m not managing which containers are accessible through the firewall. Meaning, I’m not manually adding ports into my firewall solution. That kind of manual work is not happening. I need a dynamic, and flexible solution that blocks by default except to my trusted IPs.
4. The firewall solution must be simple. More complex means more room for error.
5. The firewall solution must not impact performance significantly.
6. Restarting the firewall won’t break Docker.
7. Restarting Docker won’t break the firewall.
8. No impact to running server processes or Docker services when making a change. Things need to keep working!
   Firewall changes need to happen online and not impact Docker. Meaning I can’t be restarting Docker because I made a firewall change.
   Docker “changes” need to happen online. Meaning I can’t be restarting the firewall because I made a Docker change. (A Docker “change” means starting/stopping a container).

That sounds very simple! Unfortunately, it is not with Docker (and Docker Swarm).

Solution – Docker firewall with iptables and ipset

If you don’t know much about iptables, or ipset, that’s okay. You don’t really need to know. You should have some basic understandings though, so you don’t break your servers! The Arch Linux wiki has great information about iptables. Including this helpful visual about the iptables flow.

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

High level summary

iptables with ipset will handle all of this for us. And keep our servers, and Docker locked down from the network level.

In this solution, we will use the iptables INPUT chain to jump to another chain (let’s call our custom chain FILTERS), but return if there’s some legitimate looking traffic, so the Swarm overlay can do whatever it wants in INPUT with IPSEC, or whatever it is appending to INPUT.
Inside our custom chain FILTERS, we drop everything that doesn’t match our trusted list of IPs. We also allow our SSH port and the basic default iptables stuff.. You can also add any OS port to be publicly accessible.
The DOCKER-USER chain only needs a few entries. Any internal Docker traffic is returned, and it will drop any other traffic that’s not in our allowed IP list. You can also add any container port to be publicly accessible.

One of the dangers with this approach is if Docker changes it’s behaviour our firewall could break, or our Docker services could stop working. Since Docker doesn’t offer any solution for their users, we need our own solution. So keep in mind that you need to test this when upgrading to a new version of Docker. That is the trade-off with a “roll your own” solution. But what choice do we have?

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

Warnings

Warning: Be sure you have everything needed in your configuration. Once the iptables firewall is started it blocks anything that wasn’t added! Don’t lock yourself out of your server. Be sure to have another way to connect, like a console.

Disclaimer: Keep in mind, you should test all of this in your lab or staging environments. I can’t guarantee this will be 100% safe and can’t be held responsible for anything going wrong!

SELinux Bug: If using SELinux, currently there’s a bug with SELinux which prevents saving the iptables rules to the iptables.save file.
Impact: Saving the iptables rules a 2nd time will silently fail. Workaround has been added so SELinux allows chmod to interact with the iptables.save file. See notes on GitHub for SELinux workaround steps. Alternatively you could disable SELinux, but that’s not recommended. Bug report: https://bugs.centos.org/view.php?id=12648

The manual way

Run the commands below. These commands are only for CentOS/RHEL 7. If you don’t want to do this manually, jump to the Automatic section, using Ansible (which also works with Ubuntu).

Prep

Make note of what you already have in iptables (if you are already using it). Be sure you have some background with iptables, since you could break things!

# iptables -nvL --line-numbers

Install the required packages for CentOS / RHEL:

# yum install iptables iptables-services ipset ipset-service

Configure ipset

ipset allows you to add a list of IPs that you can use with iptables. In our case, we will add a list of IPs we want to be able to connect to our servers.

Configure ipset with a setname of ip_allow.
Add IPs you want to allow. Change the IPs below to your actual trusted/allowed IP ranges. Be sure to include your Docker server IPs here, because if you don’t they can’t communicate with eachother:

# mkdir -p /etc/sysconfig/ipset.d
# vi /etc/sysconfig/ipset.d/ip_allow.set

create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
add ip_allow 192.168.1.123
add ip_allow 192.168.100.0/24
add ip_allow 192.168.101.0/24
add ip_allow 192.168.102.0/24

Start, and Enable the ipset service:

# systemctl status ipset
# systemctl start ipset
# systemctl enable ipset

See what ipset has in it’s loaded configuration:

# ipset list | head

Important: Make note of the size of “Number of entries”. If that number is close to the maxelem size (65536), then you need to delete the ipset and re-create it with a larger max size. If you only use a few IP ranges like above, you don’t need to worry and will be well below the limit.

Configure iptables

Next up, iptables. iptables is our solution for a firewall. We will create a file with our rules and then add those rules into iptables. The important part is to not flush the existing rules if you are already using Docker (or Docker Swarm) on your server.

Create an iptables file to use with iptables-restore, to add the rules into iptables:

# vi iptables-rules.txt

Add below to the file. There is a lot going on here..

*filter
:DOCKER-USER - [0:0]
:FILTERS - [0:0]
#Can't flush INPUT. wipes out docker swarm encrypted overlay rules
#-F INPUT
#Use ansible or run manually once instead to add -I INPUT -j FILTERS
#-I INPUT -j FILTERS
-F DOCKER-USER
-A DOCKER-USER -m state --state RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -i docker_gwbridge -j RETURN
-A DOCKER-USER -s 172.18.0.0/16 -j RETURN
-A DOCKER-USER -i docker0 -j RETURN
-A DOCKER-USER -s 172.17.0.0/16 -j RETURN
#Below Docker ports open to everyone if uncommented
#-A DOCKER-USER -p tcp -m tcp -m multiport --dports 8000,8001 -j RETURN
#-A DOCKER-USER -p udp -m udp -m multiport --dports 9000,9001 -j RETURN
-A DOCKER-USER -m set ! --match-set ip_allow src -j DROP
-A DOCKER-USER -j RETURN
-F FILTERS
#Because Docker Swarm encrypted overlay network just appends rules to INPUT. Has to be at top unfortunately
-A FILTERS -p udp -m policy --dir in --pol ipsec -m udp -m set --match-set ip_allow src --dport 4789 -j RETURN
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -p icmp -j ACCEPT
-A FILTERS -i lo -j ACCEPT
#Below OS ports open to everyone if uncommented
-A FILTERS -p tcp -m state --state NEW -m tcp -m multiport --dports 22 -j ACCEPT
#-A FILTERS -p udp -m udp -m multiport --dports 53,123 -j ACCEPT
-A FILTERS -m set ! --match-set ip_allow src -j DROP
-A FILTERS -j RETURN
COMMIT

Use iptables-restore to add the above rules into iptables. The very important flag is -n. That makes sure we don’t flush the iptables rules if we have rules already in Docker (or Docker Swarm).

# iptables-restore -n < iptables-rules.txt

Next, add a rule to the INPUT chain, so we start using the new rules in FILTERS. It has to be at the top, and only needs to be added once:

# iptables -I INPUT 1 -j FILTERS

Save the iptables rules:

# /usr/libexec/iptables/iptables.init save

That will save any existing and our new iptables rules to the iptables configuration file so it will be persistent after a reboot.

In addition, it was needed to run the above iptables command manually since we want to ensure it’s only inserted once. And we can’t flush the INPUT chain to ensure that since Docker Swarm could have rules there already.

Start and Enable the iptables service:

# systemctl status iptables
# systemctl start iptables
# systemctl enable iptables

If you want to customize the iptables rules to allow more ports to be open to everyone, just add the port to the appropriate rule in the iptables file (tcp or udp), then re-run the same commands from above:

# iptables-restore -n < iptables-rules.txt
# /usr/libexec/iptables/iptables.init save

Don’t miss the Warnings from above! Especially about SELinux.

If you don’t want to do all of that manually, and you use Ansible, then do this instead..

The Automatic way with Ansible

Manually run all of the above, on every Docker server is not ideal. Let’s use Ansible instead!

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

This works on CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Install the Ansible Role using Ansible Galaxy:

$ ansible-galaxy install ryandaniels.iptables_docker

Or, clone the GitHub project:

$ git clone https://github.com/ryandaniels/ansible-role-iptables-docker.git roles/ryandaniels.iptables_docker

Create the Ansible Playbook, called iptables_docker.yml:

---
- hosts: '{{ inventory }}'
  become: yes
  vars:
    # Use this role
    iptables_docker_managed: true
  roles:
  - ryandaniels.iptables_docker

Make configuration changes to add desired IP addresses and ports as needed.

Don’t miss the Warnings from above!

Then run the playbook:

$ ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7" -i hosts-dev

Conclusion

In conclusion, now we have secured our Docker (and Docker Swarm) environments using Ansible to perform the installation and configuration of iptables! None of our Docker published ports are exposed to the world, unless we want them to be! We have created a custom Docker firewall with iptables. Hopefully, some day this will be the default behaviour and shipped with Docker out of the box! Dare to dream. Security is hard.

References

Background for Docker’s undocumented use of iptables INPUT chain

See my previous post about this.

References and links

References, notes, and links about the Docker firewall discussion:

• Docker Documentation – Overlay Networks
• Docker bypasses ufw firewall rules
• unrouted – Solution using iptables. Not for Swarm. It clobbers the INPUT chain, which is used by encrypted overlay with Docker Swarm
• The big thread about Docker and a firewall
• Arch Linux wiki to the rescue to show iptables flow which links to a great visual
  
  

The post Secure Docker with iptables firewall and Ansible appeared first on Ryan Daniels.

I could find pretty much nothing about this, anywhere.

Wait, let’s take a step back. What am I talking about?

I was testing iptables with Docker. What a nightmare. Anyways, I noticed something strange with my three node Docker Swarm test VMs. Some very strange rules at the top of the iptables INPUT chain.

# iptables -nvL INPUT
Chain INPUT (policy ACCEPT 33477 packets, 8115K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100600"
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100600"
   21  3669 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100100"
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100100"
    1   101 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir in pol ipsec udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100200"
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 u32 "0x0>>0x16&0x3c@0xc&0xffffff00=0x100200"

What is that? Some kind of rules using a Docker port, and something about IPSEC.

Searching the internet has no information about this, that I could find at least. Only some random Docker issues on GitHub. It does have some similar details in the INPUT chain though.

Also found were two references to the Docker documentation.

The Docker documentation does have a pretty good section about iptables. But no mention of the INPUT chain. They very specifically say they only modify the DOCKER-USER and DOCKER chains in iptables.

There’s also some information about the overlay network in the Docker documentation, but unfortunately it’s also very high level. No details about the Docker magic that makes all the networking work seamlessly! However, it does mention using IPSEC. Time to do some testing.

When testing in Docker Swarm, by starting containers which use an encrypted overlay I get those iptables rules in the INPUT chain. Fun!

Conclusion

Not a very exciting mystery. But very unexpected that Docker is modifying the INPUT chain in iptables! That changes a few thing when trying to secure Docker.. Hopefully other people weren’t relying on having the last rule in the INPUT chain.

Only Docker knows how their next version of Docker will behave. Anything a user does to try and secure Docker at the network layer is futile.

That being said, I’ve attempted to secure Docker with an iptables firewall. Check it out!

The post Docker and the iptables INPUT chain appeared first on Ryan Daniels.

Every time you build a new image, use a new version for your image tags.
Every time you pull an image, use a specific image tag version.

This article describes why latest is bad.

For me, in this theoretical example.. An image I was using introduced breaking changes. This is fine, since they tagged the new releases as version 2. But, only if I was actually using a specific image version and not latest would this be okay. Latest is latest. So the latest that was version 1, now turned into version 2. And everything broke.

Now that you will never use latest again, you could still have a problem. All those docker containers you have running are using images with the :latest tag. How do you find what version latest was?

I hope you’re Lucky.. Trick #1

Check the labels on your image. If you’re lucky, the developer added a label to the image with the version. I’m using the utility jq below. It’s very helpful, be sure to install it.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
traefik             latest              96c63a7d3e50        2 months ago        85.7MB

$ IMAGE_ID=96c63a7d3e50

$ docker image inspect --format '{{json .}}' "$IMAGE_ID" | jq -r '. | {Id: .Id, Digest: .Digest, RepoDigests: .RepoDigests, Labels: .Config.Labels}'

Output:

{
  "Id": "sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834",
  "Digest": null,
  "RepoDigests": [
    "traefik@sha256:5ec34caf19d114f8f0ed76f9bc3dad6ba8cf6d13a1575c4294b59b77709def39"
  ],
  "Labels": {
    "org.opencontainers.image.description": "A modern reverse-proxy",
    "org.opencontainers.image.documentation": "https://docs.traefik.io/",
    "org.opencontainers.image.title": "Traefik",
    "org.opencontainers.image.url": "https://traefik.io/",
    "org.opencontainers.image.vendor": "Containous",
    "org.opencontainers.image.version": "v1.7.20"
  }
}

Bingo! That’s version 1.7.20 of Traefik!

Lucky Trick #2

Pull a bunch of images and hope the Image ID matches. Not much to this trick..

Lucky Trick #3

If your image is from Docker Hub, they have a new experimental tool you can use called “Docker Hub Tool“.

But what if you are unlucky? What if there’s a way to check all version tags of an image?

Find Version Tag for Latest Docker image

There’s a way to check all version tags on Docker Hub (for example), against the local docker image’s “Image ID”.

You can get every tag from a Docker Registry (like Docker Hub), then use every tag you found, to get the image ID information from the manifest of every image.

Docker Hub has some quirks compared to a proper Docker Registry, and the API isn’t well documented. But let’s focus on Docker Hub, since that’s a huge public image repository.

If you don’t want to do all of this manually, you can use my script (on GitHub). Here’s the script in action:

# ./docker_image_find_tag.sh -n traefik -i 96c63a7d3e50 -f 1.7. -l 10 -v

Using IMAGE_NAME: traefik
Using REGISTRY: https://index.docker.io/v2
Found Image ID Source: sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834
Found Total Tags: 610
Found Tags (after filtering): 178
Limiting Tags to: 10
Limit reached, consider increasing limit (-l [number]) or use more specific filter (-f [text])

Found Tags:
v1.7.21-alpine
v1.7.21
v1.7.20-alpine
v1.7.20
v1.7.19-alpine
v1.7.19
v1.7.18-alpine
v1.7.18
v1.7.17-alpine
v1.7.17

Checking for image match..
Found match. tag: v1.7.20
Image ID Target: sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834
Image ID Source: sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834

This example is searching for an image named traefik, with an image ID of 96c63a7d3e50 (which we got from running docker images.

The total number of tags for traefik images is 610! You could use this script to check every one, but that will take a minute or two. Instead, you can filter the results a bit. I *think* I’m running something in version 1.7.x. I’m also limiting the search to 10 tags in this example to keep the output small.

And the find version tag script found it! The local image ID matches an image ID on the Docker Registry with a tag of 1.7.20!

You can find the script on GitHub: https://github.com/ryandaniels/docker-script-find-latest-image-tag

Some issues to note:
Issue #1: If the image no longer exists on the Docker Registry. You obviously won’t find it.
Issue #2: The script doesn’t work for Windows images. There are no manifests available for those.

Side note: In this example I’m using the traefik image. What is traefik? It’s “The Cloud Native Edge Router” which means it’s a reverse proxy and load balancer for HTTP and TCP-based applications. It works really well from my experience! Just be careful when upgrading from version 1 to version 2.

Conclusion

In conclusion, we ran the find docker image version script to match all image ID’s on Docker Hub against the local docker image that’s only tagged as latest. And we found a match, so we know what version we are running!

Now that we have a versioned image tag, we will update our running containers to use that specific version. And never use latest again!

Check out other interesting articles about Docker.

The post Find Version Tag for Latest Docker Image appeared first on Ryan Daniels.

Using a dynamic Dockerfile can have great benefits when used in your CI/CD pipeline. You can use the ARG statement in your Dockerfile to pass in a variable at build time. Even use a variable in the FROM statement! Dockerfile ARG FROM ARG. That will make more sense later.

This post is about Container virtualization technology, and problems when building a container image using Docker Engine.
For some background on a dynamic Dockerfile, check out Jeff Geerling’s post.

One problem I’ve come across with this, is the Dockerfile ARG variable usage isn’t intuitive. It is actually kind of weird. And I’m not alone on this! There’s lots of feedback in GitHub Issue #34129.

What’s the Problem?

A variable declared at the top of the Dockerfile using ARG doesn’t work if you try to use it after the FROM statement. Not by default at least. It only works if used in the FROM statement.

ARG TAG=latest
FROM myimage:$TAG # <------ This works!
LABEL BASE_IMAGE="myimage:$TAG" # <------ Does not work!

The FROM above will use the TAG passed in from a build-arg, or default to use the string “latest”.
But the LABEL won’t work. The TAG portion will be empty!

This was tested with the most recent version of Docker Engine (version: 19.03.5).

How Does it Work?

To use a variable in the FROM section, you need to put the ARG before the FROM. So far so good. But, now let’s use that variable again after the FROM? No you can’t. It’s not there. Unless you use another ARG statement after the FROM.
Wait, it gets more weird. Any ARG before the FIRST FROM, can be used in any FROM (for multi-stage builds). In fact the ARG must be above the first FROM even if you only want to use it in a later FROM. Wow, that’s a lot of ARGs and FROMs. The ARG is kind of “global” but only if you declare it again inside the build stage.

Dockerfile ARG FROM ARG

Let’s look at an example to make it easier. Below is not a useful Dockerfile. It’s only an example to illustrate this.

#These ARGs are used in the FROM statements
#Or as global variables if declared again (without the default value)
ARG BUILDER_TAG=latest
ARG BASE_TAG=latest
ARG APP="app.go"

#First build stage
FROM mybuildapp:$BUILDER_TAG AS builder
ARG APP
RUN compile_my_app $APP

#Second build stage
FROM registry.access.redhat.com/ubi8/ubi-minimal:$BASE_TAG
ARG BASE_TAG
ARG APP
LABEL BASE_IMAGE="registry.access.redhat.com/ubi8/ubi-minimal:$BASE_TAG"
COPY --from=builder $APP .

Now, let’s run the example build:

docker build . \
  --pull
  --build-arg BUILDER_TAG="2.0" \
  --build-arg BASE_TAG="8.1" \
  --build-arg APP="the_best_app.go" \
  -t image_tag

Once the Docker build finishes, this is the result of your command line argument usage inside the docker image:

BUILDER_TAG="2.0"
BASE_TAG="8.1"
APP="the_best_app.go"

Results

The BUILDER_TAG and BASE_TAG will be used in the FROM statements. BASE_TAG is also used inside the second build stage. And APP will be used in both build stages. ARG APP="app.go" only needs to be declared if you want to set a default value. In my example we don’t really need that since we are passing in a build argument.

Conclusion

Remember that ARGs at the top of the Dockerfile can be used in the FROM statements. They are “global” only if you declare the ARG again in each build stage (after FROM (and before the next FROM if using a multi-stage build)).
You can remember it this way: Dockerfile ARG FROM ARG
But, after wasting an hour on this, it was more like: Dockerfile Arrggh! FROM Arrggh!

You can use something like this in your CI/CD, for automatic docker image building.

Side note: Building an image using Podman does not have the same issue as Docker Engine (tested with Podman version 1.4.4, 1.6.3, and 1.7.0). You do not need to specify the ARG again inside the build stage when Podman creates the container image!

Further reading:
Dockerfile Reference: Understand how ARG and FROM interact
Dockerfile Reference: Scope
Multi-Stage Builds

The post Dockerfile ARG FROM ARG trouble with Docker appeared first on Ryan Daniels.