Let’s discuss the background of firewall issues with Docker, and a working solution for my use case (either setup manually or using Ansible). By the end we will use a firewall on the server to lock down everything by default, only allowing my trusted IPs! With the option to open specified ports publicly (like SSH).

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Background

Firstly, even if you were using a firewall like iptables, Docker makes that useless. Docker punches a whole right through your firewall!

And if you try another firewall, like firewalld? Docker Swarm (and even regular Docker) with firewalld is a complete mess. Restart the firewalld service, or change the firewalld config, and you lost all the config that Docker needed. Now you have to restart Docker! What happens if the firewalld service failed and restarted? That Docker Swarm node is out of service.

Keep in mind, when I mention Docker, it means the regular Docker Engine. Docker Swarm means SwarmKit, which is the newer way of using Swarm. (The old way is the standalone solution which is old and not referenced at all here).

There are many attempts to solve this problem by users of Docker. Unfortunately the Docker team has been pretty quiet about these issues. They recommend a manual user solution, and to disable Docker’s use of iptables. I’m speculating here, but it seems like any future change from Docker will likely be a breaking change since this is a complicated issue to fix.

The attempted solutions (from users) are not very straight forward for a normal Docker user. And that’s the real issue. On top of that, out of the box (after you start one service with a published port), there is no security at the network layer. Anyone can connect to an exposed container’s port. Most importantly, even if you think you have a firewall protecting you.. Wrong, you don’t! With normal Docker, you can bind your service to your localhost which helps. But what about Docker Swarm? Nope, that doesn’t work.

There’s a great article about “The problem of forcing users to make choices (in security).” Definitely worth the read!

Roll your own solution

Until this is actually addressed in Docker, our only hope is to find the simplest solution possible. And turning off iptables integration in Docker is unacceptable (which is constantly recommended by the Docker developers). The other option is to move on from Docker and/or Docker Swarm. I hear this thing called Kubernetes is pretty great. Anyways, back to Docker.
Many people have spent hours trying to learn, and figure out iptables as a solution to this. You need to roll your own solution apparently! So iptables is probably the best approach, since that is what Docker needs to use to do it’s magic (and most other firewalls are just a wrapper for iptables anyway depending on your OS).

Something fun I found out while testing this, Docker Swarm uses iptables in an undocumented way. Docker Swarm uses the iptables INPUT chain! It’s only for encrypted overlay networks. But it’s not very fun realizing that! All of a sudden rules are being appended to the INPUT chain.

Okay, enough backstory. On with my futile attempt to roll my own solution. This took way longer than I thought it would! But it does work! Currently it works at least.. (That’s why I use the word futile!)

Problems I need to solve

1. Only allow traffic from multiple “trusted” IP addresses to my servers. Not all of these IPs will be in the same “IP block/range” either. This will be to all services running directly on the server, and also all of the Docker containers.
2. Let only specific ports be publicly accessible, like SSH.
3. I’m not managing which containers are accessible through the firewall. Meaning, I’m not manually adding ports into my firewall solution. That kind of manual work is not happening. I need a dynamic, and flexible solution that blocks by default except to my trusted IPs.
4. The firewall solution must be simple. More complex means more room for error.
5. The firewall solution must not impact performance significantly.
6. Restarting the firewall won’t break Docker.
7. Restarting Docker won’t break the firewall.
8. No impact to running server processes or Docker services when making a change. Things need to keep working!
   Firewall changes need to happen online and not impact Docker. Meaning I can’t be restarting Docker because I made a firewall change.
   Docker “changes” need to happen online. Meaning I can’t be restarting the firewall because I made a Docker change. (A Docker “change” means starting/stopping a container).

That sounds very simple! Unfortunately, it is not with Docker (and Docker Swarm).

Solution – Docker firewall with iptables and ipset

If you don’t know much about iptables, or ipset, that’s okay. You don’t really need to know. You should have some basic understandings though, so you don’t break your servers! The Arch Linux wiki has great information about iptables. Including this helpful visual about the iptables flow.

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

High level summary

iptables with ipset will handle all of this for us. And keep our servers, and Docker locked down from the network level.

In this solution, we will use the iptables INPUT chain to jump to another chain (let’s call our custom chain FILTERS), but return if there’s some legitimate looking traffic, so the Swarm overlay can do whatever it wants in INPUT with IPSEC, or whatever it is appending to INPUT.
Inside our custom chain FILTERS, we drop everything that doesn’t match our trusted list of IPs. We also allow our SSH port and the basic default iptables stuff.. You can also add any OS port to be publicly accessible.
The DOCKER-USER chain only needs a few entries. Any internal Docker traffic is returned, and it will drop any other traffic that’s not in our allowed IP list. You can also add any container port to be publicly accessible.

One of the dangers with this approach is if Docker changes it’s behaviour our firewall could break, or our Docker services could stop working. Since Docker doesn’t offer any solution for their users, we need our own solution. So keep in mind that you need to test this when upgrading to a new version of Docker. That is the trade-off with a “roll your own” solution. But what choice do we have?

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

Warnings

Warning: Be sure you have everything needed in your configuration. Once the iptables firewall is started it blocks anything that wasn’t added! Don’t lock yourself out of your server. Be sure to have another way to connect, like a console.

Disclaimer: Keep in mind, you should test all of this in your lab or staging environments. I can’t guarantee this will be 100% safe and can’t be held responsible for anything going wrong!

SELinux Bug: If using SELinux, currently there’s a bug with SELinux which prevents saving the iptables rules to the iptables.save file.
Impact: Saving the iptables rules a 2nd time will silently fail. Workaround has been added so SELinux allows chmod to interact with the iptables.save file. See notes on GitHub for SELinux workaround steps. Alternatively you could disable SELinux, but that’s not recommended. Bug report: https://bugs.centos.org/view.php?id=12648

The manual way

Run the commands below. These commands are only for CentOS/RHEL 7. If you don’t want to do this manually, jump to the Automatic section, using Ansible (which also works with Ubuntu).

Prep

Make note of what you already have in iptables (if you are already using it). Be sure you have some background with iptables, since you could break things!

# iptables -nvL --line-numbers

Install the required packages for CentOS / RHEL:

# yum install iptables iptables-services ipset ipset-service

Configure ipset

ipset allows you to add a list of IPs that you can use with iptables. In our case, we will add a list of IPs we want to be able to connect to our servers.

Configure ipset with a setname of ip_allow.
Add IPs you want to allow. Change the IPs below to your actual trusted/allowed IP ranges. Be sure to include your Docker server IPs here, because if you don’t they can’t communicate with eachother:

# mkdir -p /etc/sysconfig/ipset.d
# vi /etc/sysconfig/ipset.d/ip_allow.set

create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
add ip_allow 192.168.1.123
add ip_allow 192.168.100.0/24
add ip_allow 192.168.101.0/24
add ip_allow 192.168.102.0/24

Start, and Enable the ipset service:

# systemctl status ipset
# systemctl start ipset
# systemctl enable ipset

See what ipset has in it’s loaded configuration:

# ipset list | head

Important: Make note of the size of “Number of entries”. If that number is close to the maxelem size (65536), then you need to delete the ipset and re-create it with a larger max size. If you only use a few IP ranges like above, you don’t need to worry and will be well below the limit.

Configure iptables

Next up, iptables. iptables is our solution for a firewall. We will create a file with our rules and then add those rules into iptables. The important part is to not flush the existing rules if you are already using Docker (or Docker Swarm) on your server.

Create an iptables file to use with iptables-restore, to add the rules into iptables:

# vi iptables-rules.txt

Add below to the file. There is a lot going on here..

*filter
:DOCKER-USER - [0:0]
:FILTERS - [0:0]
#Can't flush INPUT. wipes out docker swarm encrypted overlay rules
#-F INPUT
#Use ansible or run manually once instead to add -I INPUT -j FILTERS
#-I INPUT -j FILTERS
-F DOCKER-USER
-A DOCKER-USER -m state --state RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -i docker_gwbridge -j RETURN
-A DOCKER-USER -s 172.18.0.0/16 -j RETURN
-A DOCKER-USER -i docker0 -j RETURN
-A DOCKER-USER -s 172.17.0.0/16 -j RETURN
#Below Docker ports open to everyone if uncommented
#-A DOCKER-USER -p tcp -m tcp -m multiport --dports 8000,8001 -j RETURN
#-A DOCKER-USER -p udp -m udp -m multiport --dports 9000,9001 -j RETURN
-A DOCKER-USER -m set ! --match-set ip_allow src -j DROP
-A DOCKER-USER -j RETURN
-F FILTERS
#Because Docker Swarm encrypted overlay network just appends rules to INPUT. Has to be at top unfortunately
-A FILTERS -p udp -m policy --dir in --pol ipsec -m udp -m set --match-set ip_allow src --dport 4789 -j RETURN
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -p icmp -j ACCEPT
-A FILTERS -i lo -j ACCEPT
#Below OS ports open to everyone if uncommented
-A FILTERS -p tcp -m state --state NEW -m tcp -m multiport --dports 22 -j ACCEPT
#-A FILTERS -p udp -m udp -m multiport --dports 53,123 -j ACCEPT
-A FILTERS -m set ! --match-set ip_allow src -j DROP
-A FILTERS -j RETURN
COMMIT

Use iptables-restore to add the above rules into iptables. The very important flag is -n. That makes sure we don’t flush the iptables rules if we have rules already in Docker (or Docker Swarm).

# iptables-restore -n < iptables-rules.txt

Next, add a rule to the INPUT chain, so we start using the new rules in FILTERS. It has to be at the top, and only needs to be added once:

# iptables -I INPUT 1 -j FILTERS

Save the iptables rules:

# /usr/libexec/iptables/iptables.init save

That will save any existing and our new iptables rules to the iptables configuration file so it will be persistent after a reboot.

In addition, it was needed to run the above iptables command manually since we want to ensure it’s only inserted once. And we can’t flush the INPUT chain to ensure that since Docker Swarm could have rules there already.

Start and Enable the iptables service:

# systemctl status iptables
# systemctl start iptables
# systemctl enable iptables

If you want to customize the iptables rules to allow more ports to be open to everyone, just add the port to the appropriate rule in the iptables file (tcp or udp), then re-run the same commands from above:

# iptables-restore -n < iptables-rules.txt
# /usr/libexec/iptables/iptables.init save

Don’t miss the Warnings from above! Especially about SELinux.

If you don’t want to do all of that manually, and you use Ansible, then do this instead..

The Automatic way with Ansible

Manually run all of the above, on every Docker server is not ideal. Let’s use Ansible instead!

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

This works on CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Install the Ansible Role using Ansible Galaxy:

$ ansible-galaxy install ryandaniels.iptables_docker

Or, clone the GitHub project:

$ git clone https://github.com/ryandaniels/ansible-role-iptables-docker.git roles/ryandaniels.iptables_docker

Create the Ansible Playbook, called iptables_docker.yml:

---
- hosts: '{{ inventory }}'
  become: yes
  vars:
    # Use this role
    iptables_docker_managed: true
  roles:
  - ryandaniels.iptables_docker

Make configuration changes to add desired IP addresses and ports as needed.

Don’t miss the Warnings from above!

Then run the playbook:

$ ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7" -i hosts-dev

Conclusion

In conclusion, now we have secured our Docker (and Docker Swarm) environments using Ansible to perform the installation and configuration of iptables! None of our Docker published ports are exposed to the world, unless we want them to be! We have created a custom Docker firewall with iptables. Hopefully, some day this will be the default behaviour and shipped with Docker out of the box! Dare to dream. Security is hard.

References

Background for Docker’s undocumented use of iptables INPUT chain

See my previous post about this.

References and links

References, notes, and links about the Docker firewall discussion:

• Docker Documentation – Overlay Networks
• Docker bypasses ufw firewall rules
• unrouted – Solution using iptables. Not for Swarm. It clobbers the INPUT chain, which is used by encrypted overlay with Docker Swarm
• The big thread about Docker and a firewall
• Arch Linux wiki to the rescue to show iptables flow which links to a great visual
  
  

The post Secure Docker with iptables firewall and Ansible appeared first on Ryan Daniels.

Background on firewalld

Firewalld is a firewall which runs on many Linux distributions. It is software which runs on a server, and is used to locally secure the server. For example, it can restrict which ports are available for all source IPs, or it can further restrict access by allowing only certain IPs to a specific port (and zone). Check out the documentation to read more about the features of firewalld.

Everything is blocked when firewalld starts! The only exception is for ssh on port 22. This is normally what you want on a new server.

Problem

However, firewalld isn’t with out it’s limitations. To run commands to configure which ports you want to allow through firewalld, you need to have the firewalld service running. But, what if you are adding firewalld to a server that has an application which is already in use? As soon as you start firewalld, it will be blocked!

There are two ways around this.

1. Copy your firewalld config files in the right place, with all the right ports, zones, etc. And then start firewalld. This can work. But having Ansible copy files around and deal with all that is a challenge since firewalld behaves a certain way.
2. Use the “offline” firewalld commands!

Let’s explore options 2. And lucky for us, we want to use Ansible to help us manage this configuration. And, Ansible can make use of “offline” mode.
But, unlucky for us, the Ansible module which handles firewalld can’t do everything. (At least not everything that I want it to do).

I want the configuration for firewalld to be as simple as possible. I don’t want the configuration for services in one place, and the configuration for ports in another.
But more importantly, I don’t want any service interruption.

Solution

Create my own Ansible role that does what I want.

If you’re new to Ansible, check out my Ansible Getting Started Guide.

I want firewalld to behave in a certain way. Meaning if I have an application already running, and I want firewalld to start restricting access, I don’t want an outage on that application as soon as I start up the firewalld service on the server.

Firewalld Limitations

Firewalld does have some limitations. You should be aware of this before using it.
Anything firewalld is not aware of in the back-end (for example if using iptables), will be removed since firewalld wants to be the single source of truth for firewall rules.
This is why firewalld and Docker do not get along.
See details from: here, here, and here.

In case you missed that..
Do not use firewalld and Docker.
Update: Instead, if you need a solution to work with Docker try using iptables directly. See my post about securing Docker with iptables and Ansible.

Firewalld Ansible Role

You can find my firewalld Ansible Role on GitHub or in Ansible Galaxy.

This Ansible Role manages firewalld in offline and persistent mode only. I don’t want to mess around with non-persistent changes. And using online mode is not an option.

It uses the Ansible module for firewalld for a few things, but most changes have to be done using the command module. Specifically: “firewall-offline-cmd”.

Why? What’s the benefit doing it this way? It let’s you create your own custom service in firewalld. Even built-in firewalld services can be used, and easily modified. For example, moving the ssh port from 22 to something else. (Warning, be careful changing your SSH port. You could lock yourself out!)

Here’s a short example of the configuration:

rhel_firewalld_zone_source:
  - zone: internal
    state: enabled
    source:
      - "192.168.22.64/26"
      - "192.168.23.64/26"

rhel_firewalld_custom_service:
  - name: zabbix-agent
    zone: public
    state: enabled
    port_protocol:
    # - 10050/tcp
      - 3333/tcp
  - name: openvpn
    zone: public
    state: enabled
  - name: app123-public
    zone: public
    state: enabled
    description: app123 firewall rules for public zone
    port_protocol:
      - 5000/tcp
  - name: app123-internal
    zone: internal
    state: enabled
    description: app123 firewall rules for internal zone
    port_protocol:
      - 8080/tcp
      - 9000/tcp

In the variable rhel_firewalld_zone_source:
This will configure the Zone named internal to only allow IP ranges that are in source.

In the variable rhel_firewalld_custom_service:
This will configure the zabbix-agent service (which is actually built-in) for the zone public, but not use the default port (of 10050, commented out for demonstration). Instead it’s using port 3333.
The openvpn service is added to the zone public, and will just use the default configuration (since this is also a built-in firewalld service). You need to know it’s built-in, if you don’t specify a port. So it’s usually better to just specify a port all the time!
Last, there are two custom services, app123-public and app123-internal, which are added to their defined zones. They are also adding port 5000 using tcp (attached to the public zone), and adding ports 8080 and 9000 both using tcp (attached to the internal zone).

If you decide you want to change a port, just remove it from the configuration.

More details are on GitHub. Be sure to read the README!

Getting Started – Ansible firewalld

Install from Ansible Galaxy:

ansible-galaxy install ryandaniels.firewalld

Or, clone the GitHub project:

git clone https://github.com/ryandaniels/ansible-role-firewalld.git roles/firewalld

Create the Ansible Playbook, called firewalld.yml:

---
- hosts: '{{ inventory }}'
  become: yes
  vars:
    # Use this role
    rhel_firewalld_managed: true
  roles:
  - firewalld

Make configuration changes add desired firewalld services and ports.

Warning: Be sure you have everything needed in your configuration. Once firewalld is started it blocks anything that wasn’t added!

Disclaimer: Keep in mind, you should test all of this in your lab or staging environments. I can’t guarantee this will be 100% safe and can’t be held responsible for anything going wrong!

Then run the playbook:

ansible-playbook firewalld.yml --extra-vars "inventory=centos7" -i hosts-dev

Conclusion

In conclusion, Ansible is now being used to manage firewalld rules! We have a custom firewalld service created using two different zones. The built-in firewalld services are also enabled, but we don’t really care they are built-in, since we can change the port they use if we want.
We have a nice and simple configuration, and this was all done in “offline” mode so there was no impact to any existing services using the ports.

Again, your mileage may vary. Always test in your lab or staging environment first! I’m not responsible for anything you break.

Stay safe!

The post Manage firewalld with Ansible appeared first on Ryan Daniels.

Every time you build a new image, use a new version for your image tags.
Every time you pull an image, use a specific image tag version.

This article describes why latest is bad.

For me, in this theoretical example.. An image I was using introduced breaking changes. This is fine, since they tagged the new releases as version 2. But, only if I was actually using a specific image version and not latest would this be okay. Latest is latest. So the latest that was version 1, now turned into version 2. And everything broke.

Now that you will never use latest again, you could still have a problem. All those docker containers you have running are using images with the :latest tag. How do you find what version latest was?

I hope you’re Lucky.. Trick #1

Check the labels on your image. If you’re lucky, the developer added a label to the image with the version. I’m using the utility jq below. It’s very helpful, be sure to install it.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
traefik             latest              96c63a7d3e50        2 months ago        85.7MB

$ IMAGE_ID=96c63a7d3e50

$ docker image inspect --format '{{json .}}' "$IMAGE_ID" | jq -r '. | {Id: .Id, Digest: .Digest, RepoDigests: .RepoDigests, Labels: .Config.Labels}'

Output:

{
  "Id": "sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834",
  "Digest": null,
  "RepoDigests": [
    "traefik@sha256:5ec34caf19d114f8f0ed76f9bc3dad6ba8cf6d13a1575c4294b59b77709def39"
  ],
  "Labels": {
    "org.opencontainers.image.description": "A modern reverse-proxy",
    "org.opencontainers.image.documentation": "https://docs.traefik.io/",
    "org.opencontainers.image.title": "Traefik",
    "org.opencontainers.image.url": "https://traefik.io/",
    "org.opencontainers.image.vendor": "Containous",
    "org.opencontainers.image.version": "v1.7.20"
  }
}

Bingo! That’s version 1.7.20 of Traefik!

Lucky Trick #2

Pull a bunch of images and hope the Image ID matches. Not much to this trick..

Lucky Trick #3

If your image is from Docker Hub, they have a new experimental tool you can use called “Docker Hub Tool“.

But what if you are unlucky? What if there’s a way to check all version tags of an image?

Find Version Tag for Latest Docker image

There’s a way to check all version tags on Docker Hub (for example), against the local docker image’s “Image ID”.

You can get every tag from a Docker Registry (like Docker Hub), then use every tag you found, to get the image ID information from the manifest of every image.

Docker Hub has some quirks compared to a proper Docker Registry, and the API isn’t well documented. But let’s focus on Docker Hub, since that’s a huge public image repository.

If you don’t want to do all of this manually, you can use my script (on GitHub). Here’s the script in action:

# ./docker_image_find_tag.sh -n traefik -i 96c63a7d3e50 -f 1.7. -l 10 -v

Using IMAGE_NAME: traefik
Using REGISTRY: https://index.docker.io/v2
Found Image ID Source: sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834
Found Total Tags: 610
Found Tags (after filtering): 178
Limiting Tags to: 10
Limit reached, consider increasing limit (-l [number]) or use more specific filter (-f [text])

Found Tags:
v1.7.21-alpine
v1.7.21
v1.7.20-alpine
v1.7.20
v1.7.19-alpine
v1.7.19
v1.7.18-alpine
v1.7.18
v1.7.17-alpine
v1.7.17

Checking for image match..
Found match. tag: v1.7.20
Image ID Target: sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834
Image ID Source: sha256:96c63a7d3e502fcbbd8937a2523368c22d0edd1788b8389af095f64038318834

This example is searching for an image named traefik, with an image ID of 96c63a7d3e50 (which we got from running docker images.

The total number of tags for traefik images is 610! You could use this script to check every one, but that will take a minute or two. Instead, you can filter the results a bit. I *think* I’m running something in version 1.7.x. I’m also limiting the search to 10 tags in this example to keep the output small.

And the find version tag script found it! The local image ID matches an image ID on the Docker Registry with a tag of 1.7.20!

You can find the script on GitHub: https://github.com/ryandaniels/docker-script-find-latest-image-tag

Some issues to note:
Issue #1: If the image no longer exists on the Docker Registry. You obviously won’t find it.
Issue #2: The script doesn’t work for Windows images. There are no manifests available for those.

Side note: In this example I’m using the traefik image. What is traefik? It’s “The Cloud Native Edge Router” which means it’s a reverse proxy and load balancer for HTTP and TCP-based applications. It works really well from my experience! Just be careful when upgrading from version 1 to version 2.

Conclusion

In conclusion, we ran the find docker image version script to match all image ID’s on Docker Hub against the local docker image that’s only tagged as latest. And we found a match, so we know what version we are running!

Now that we have a versioned image tag, we will update our running containers to use that specific version. And never use latest again!

Check out other interesting articles about Docker.

The post Find Version Tag for Latest Docker Image appeared first on Ryan Daniels.

What if you need to test a bunch of IPs and ports from several different servers. You can actually use Ansible to test the connectivity from multiple servers (which could be in different regions), to multiple tools or services (using their IP and port). Ansible is such a flexible tool for automation!

Not sure what Ansible is all about? First, check out the Ansible getting started guide.

Problem

Sometimes you need to do a quick test from one server to a certain IP and port to make sure there is network connectivity. Or maybe from multiple servers to multiple IPs and ports.

Sometimes you need to do this only once (hopefully) to get a new firewall rule implemented. Or perhaps you want to run this every day to ensure network connectivity from multiple servers to a certain tool.

Ansible can help us with this!

How does it work?

The Ansible Role uses a few Linux utilities, mainly netcat (nc), to test the source server(s) can reach the destination IP(s) and port(s).
This works with TCP and UDP protocols.

In addition, if your Ansible host (source server) has multiple IPs, it uses the ip route get command to determine which IP to use. Note, if that is wrong and you need a custom route added to your host, that is something you will need to fix.

The Connectivity Test Ansible Role is on Ansible Galaxy and GitHub.

Ansible Connectivity Test Role

Setup

Add this Ansible Role to your Ansible control node (or Ansible Tower, or Jenkins, etc).

Configuration

Add the variables to your Ansible setup. For example, if you want to test a group of Ansible hosts add this to group_vars where connectivity_test_grp is your group:
group_vars/connectivity_test_grp/connectivity-test-vars.yml

---
connectivity_test_destinations:
  - { ip: 192.168.56.10, port: 22 }
  - { ip: 192.168.56.10, port: 5000 }
  - { ip: 192.168.1.2, port: 53, protocol: udp }

The TCP protocol is used by default so it doesn’t need to be specified. (But you can if you want).

The above configuration will test 3 IPs and ports from every Ansible host in the group connectivity_test_destinations.
The first two lines test the IP 192.168.56.10 on ports 22 and 5000 using the TCP protocol.
The last line will test the IP 192.168.1.2 on port 53 using the UDP protocol.

Next, add your Ansible Playbook connectivity-test.yml:

---
- hosts: '{{inventory}}'
  become: yes
  roles:
  - connectivity-test

Usage

Run the Ansible Playbook (where connectivity_test_grp is a group in your inventory (hosts) file:

ansible-playbook connectivity-test.yml --extra-vars "inventory=connectivity_test_grp" -i hosts

If the requirements are already installed on the Ansible hosts, you can speed up the playbook by skipping that part:

ansible-playbook connectivity-test.yml --extra-vars "inventory=connectivity_test_grp" -i hosts --skip-tags connectivity_install_pkg

To clarify, more details are in the GitHub readme.

Conclusion – Test Network Connectivity

In conclusion, you can now use Ansible to test network connectivity from one or many Ansible hosts, to one or many different IPs and ports!

This can be helpful for testing firewall issues. Or to ensure a certain tool is reachable from a certain location. If you want you could add this to Ansible Tower or Jenkins as a scheduled task. For example, if you want to know if your lab environment can’t reach an important tool.

What other interesting things can you do with Ansible? Manage users with Ansible, keep your servers up to date, and use Jenkins as a front-end with Ansible Vault.

The post Ansible Role to Test Network Connectivity appeared first on Ryan Daniels.

Using a dynamic Dockerfile can have great benefits when used in your CI/CD pipeline. You can use the ARG statement in your Dockerfile to pass in a variable at build time. Even use a variable in the FROM statement! Dockerfile ARG FROM ARG. That will make more sense later.

This post is about Container virtualization technology, and problems when building a container image using Docker Engine.
For some background on a dynamic Dockerfile, check out Jeff Geerling’s post.

One problem I’ve come across with this, is the Dockerfile ARG variable usage isn’t intuitive. It is actually kind of weird. And I’m not alone on this! There’s lots of feedback in GitHub Issue #34129.

What’s the Problem?

A variable declared at the top of the Dockerfile using ARG doesn’t work if you try to use it after the FROM statement. Not by default at least. It only works if used in the FROM statement.

ARG TAG=latest
FROM myimage:$TAG # <------ This works!
LABEL BASE_IMAGE="myimage:$TAG" # <------ Does not work!

The FROM above will use the TAG passed in from a build-arg, or default to use the string “latest”.
But the LABEL won’t work. The TAG portion will be empty!

This was tested with the most recent version of Docker Engine (version: 19.03.5).

How Does it Work?

To use a variable in the FROM section, you need to put the ARG before the FROM. So far so good. But, now let’s use that variable again after the FROM? No you can’t. It’s not there. Unless you use another ARG statement after the FROM.
Wait, it gets more weird. Any ARG before the FIRST FROM, can be used in any FROM (for multi-stage builds). In fact the ARG must be above the first FROM even if you only want to use it in a later FROM. Wow, that’s a lot of ARGs and FROMs. The ARG is kind of “global” but only if you declare it again inside the build stage.

Dockerfile ARG FROM ARG

Let’s look at an example to make it easier. Below is not a useful Dockerfile. It’s only an example to illustrate this.

#These ARGs are used in the FROM statements
#Or as global variables if declared again (without the default value)
ARG BUILDER_TAG=latest
ARG BASE_TAG=latest
ARG APP="app.go"

#First build stage
FROM mybuildapp:$BUILDER_TAG AS builder
ARG APP
RUN compile_my_app $APP

#Second build stage
FROM registry.access.redhat.com/ubi8/ubi-minimal:$BASE_TAG
ARG BASE_TAG
ARG APP
LABEL BASE_IMAGE="registry.access.redhat.com/ubi8/ubi-minimal:$BASE_TAG"
COPY --from=builder $APP .

Now, let’s run the example build:

docker build . \
  --pull
  --build-arg BUILDER_TAG="2.0" \
  --build-arg BASE_TAG="8.1" \
  --build-arg APP="the_best_app.go" \
  -t image_tag

Once the Docker build finishes, this is the result of your command line argument usage inside the docker image:

BUILDER_TAG="2.0"
BASE_TAG="8.1"
APP="the_best_app.go"

Results

The BUILDER_TAG and BASE_TAG will be used in the FROM statements. BASE_TAG is also used inside the second build stage. And APP will be used in both build stages. ARG APP="app.go" only needs to be declared if you want to set a default value. In my example we don’t really need that since we are passing in a build argument.

Conclusion

Remember that ARGs at the top of the Dockerfile can be used in the FROM statements. They are “global” only if you declare the ARG again in each build stage (after FROM (and before the next FROM if using a multi-stage build)).
You can remember it this way: Dockerfile ARG FROM ARG
But, after wasting an hour on this, it was more like: Dockerfile Arrggh! FROM Arrggh!

You can use something like this in your CI/CD, for automatic docker image building.

Side note: Building an image using Podman does not have the same issue as Docker Engine (tested with Podman version 1.4.4, 1.6.3, and 1.7.0). You do not need to specify the ARG again inside the build stage when Podman creates the container image!

Further reading:
Dockerfile Reference: Understand how ARG and FROM interact
Dockerfile Reference: Scope
Multi-Stage Builds

The post Dockerfile ARG FROM ARG trouble with Docker appeared first on Ryan Daniels.

What’s the problem?

If you have multiple servers to manage, it can be a pain to manually add a new user, change a password, or lock an old account. Manually logging into all of your servers and performing these tasks is a real pain, and a huge waste of time.

Luckily there are several solutions.

Solution: Red Hat IDM

There are many different solutions for user management. Red Hat Identity Management (IDM) is a very good solution that works with many Linux servers. IDM is a central location that holds all the user configuration. And it can even connect to Windows Active Directory.

However, if you don’t have a subscription to Red Hat Enterprise Linux there are other options.

Solution: FreeIPA

FreeIPA is the upstream open source project for Red Hat Identify Management. So you can download the FreeIPA package and follow one of many available tutorials for setting up a server and client.

Solution: Ansible for user management

If the above solutions seem too complicated there’s another option. However, you should really take some time to setup user management properly. Use FreeIPA. It’s free!

But, if you want to use another solution while you are setting up FreeIPA, and you already have Ansible as part of your environment then you have another option. Ansible can do this of course!

The reason Ansible is a decent option for user management is because it is idempotent. This means, Ansible will only make a change if the change is needed to get to the desired state. So you can run the same Ansible Playbook multiple times, and if the Ansible tasks are written properly Ansible will only make a change the first time.

Just keep in mind, I warned you to use FreeIPA. This solution is not “enterprise” grade. But it certainly works for certain scenarios, like in a lab environment. And if you want to use Ansible for something new, there is a user management role available on GitHub or on Ansible Galaxy.

If you aren’t familiar with Ansible, check out my Getting Started with Ansible guide.

This Ansible role is tested on:

• Ubuntu
• Red Hat Enterprise Linux (RHEL) 7.x, 6.5, 5.9
• CentOS 7.x, 6.5, 5.9

What this user management Ansible role does

This role will manage users in the user list configuration file (list is in the file vars/lab_users.secret in the example below).
It can add users, change passwords, lock/unlock user accounts, manage sudo access (per user), add ssh key(s) for ssh key based authentication.
This is done on a per “group” basis (using Ansible group variables), as set in the configuration file. The group comes from the Ansible group as set for a server in the inventory file.

Note: Deleting users is not done on purpose.

The great thing about Ansible Playbooks, if you don’t like something it’s easy enough to modify.

Install the Ansible user management role

First let’s install the Ansible role to manage users into your Ansible control server:

$ su - ansible
$ cd ~/ansible
$ git clone https://github.com/ryandaniels/ansible-role-create-users.git roles/create-users

Or, you can use Ansible Galaxy to install this role:

$ su - ansible
$ ansible-galaxy install ryandaniels.create_users

Setup Ansible Vault

If you don’t already have Ansible Vault configured, you will need to set it up now. Vault uses AES encryption to store your sensitive information. And we need Ansible Vault to encrypt our user configuration file since we don’t want to be exposing even a hashed password into source control.

First, (if using git), make sure to update your .gitignore file so your Vault password isn’t saved to your source control. And also add the secret file we will be creating later:

$ vi .gitignore
.vaultpass
secret
*.secret

Next, create a password for your Ansible Vault and save it in your Password Manager (like KeePassXC). Then create the .vaultpass file, add the Vault password, and fix the permissions:

$ vi .vaultpass
#Enter password here

$ chmod 600 .vaultpass

Also you need to update the Ansible config file to reference where the Vault file is located:

$ vi ansible.cfg
[defaults]
vault_password_file = ./.vaultpass

Create Ansible Playbook

Next, create the Playbook file:

$ vi create-users.yml
---
- hosts: '{{inventory}}'
  vars_files:
    - vars/lab_users.secret
  become: yes
  roles:
  - create-users

Create configuration for user management

Next, add your users into a configuration file. The below is only an example, don’t use it in your user management configuration file. Also, make sure the filename matches a .gitignore entry. In this case it will match “*.secret”.
Use the special Ansible command to create the encrypted Ansible Vault file:

$ mkdir -p vars
$ ansible-vault create vars/lab_users.secret
---
users:
  - username: alice
    password: $6$/y5RGZnFaD3f$96xVdOAnldEtSxivDY02h.DwPTrJgGQl8/MTRRrFAwKTYbFymeKH/1Rxd3k.RQfpgebM6amLK3xAaycybdc.60
    update_password: on_create
    comment: Test User 100
    shell: /bin/bash
    ssh_key: |
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8crAHG/a9QBD4zO0ZHIjdRXy+ySKviXVCMIJ3/NMIAAzDyIsPKToUJmIApHHHF1/hBllqzBSkPEMwgFbXjyqTeVPHF8V0iq41n0kgbulJG alice@laptop
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8crAHG/a9QBD4zO0ZHIjdRXy+ySKviXVCMIxxxxxxxxxxxxxxxxxxJmIApHHHF1/hBllqzBSkPEMwgFbXjyqTeVPHF8V0iq41n0kgbulJG alice@server1
    exclusive_ssh_key: yes
    use_sudo: no
    use_sudo_nopass: no
    user_state: present
    servers:
      - webserver
      - database
      - monitoring

  - username: bob
    password: $6$XEnyI5UYSw$Rlc6tXtECtqdJ3uFitrbBlec1/8Fx2obfgFST419ntJqaX8sfPQ9xR7vj7dGhQsfX8zcSX3tumzR7/vwlIH6p/
    ssh_key: AAAAB3NzaC1yc2EAAAADAQABAAACAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8crAHG/a9QBD4zO0ZHIjdRXy+ySKviXVCMIxxxxxxxxxxxxxxxxxxJmIApHHHF1/hBllqbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbulJG bob@laptop
    use_sudo: no
    user_state: lock
    servers:
      - webserver
      - database

In the above example, there are two users. Users: alice and bob. If you are familiar with user management on Linux, then most of these settings are self explanatory. For example, both Alice and Bob have a hashed password and ssh keys. Alice actually has two ssh keys configured. But Bob’s account is locked so he can’t log in.

The interesting part is the list of “servers” in the configuration. These are groups defined in your Ansible hosts file. So you can give certain users access to a specific group of servers, depending on the user’s role.

Important note: Be careful with the update_password setting. When set to always, the user’s password will be changed to what is defined in password. This might not be what they wanted if they’ve manually changed their password so it’s usually safer to use on_create.

For details about all the different settings, see the README on GitHub.

Going forward, to update the file, instead of the “create” command line option, use “edit”:

$ ansible-vault edit vars/lab_users.secret

And don’t forget to save this into your source control (git).

Run the Ansible Playbook

$ ansible-playbook create-users.yml --extra-vars "inventory=all-dev" -i hosts-dev

Marvel at the output generated by Ansible. All these users are created, updated, or locked, but only if something needed to change since Ansible is idempotent.

Bonus: Add this in Jenkins

If you also use Jenkins for your CI/CD pipeline, you can add this Ansible Playbook into Jenkins.

You just need to setup Jenkins to use Ansible and Ansible Vault. And there is a great Jenkins Plugin to help. Just follow this guide to use Ansible Vault with Jenkins. Keep in mind that updating the user configuration file is only possible from the command line, since it’s encrypted using Ansible Vault.

Conclusion

In conclusion, there are many options for user management on Linux servers. From enterprise solutions like Red Hat IDM. Or, using the upstream open source software IDM uses directly, FreeIPA. If you decide to use Ansible, then the above Ansible role will get the job done. And you can also add it into Jenkins.

The post User Management with Ansible appeared first on Ryan Daniels.

If you are new to Ansible, read the getting started with Ansible post first.

Why use Ansible Vault?

You want to use Ansible Vault when you need to use sensitive data with Ansible, like a login and password, or SSH keys. Instead of storing this sensitive information in plain text, where anyone can read it when you save your Ansible Playbooks to your source control (like git), Vault uses an AES cypher to encrypt your data.

For more details, read the documentation.

Ansible Vault with Jenkins

Jenkins is software for Continuous Integration and Continuous Delivery. You can add plugins to Jenkins to extend its base features. And one handy plugin allows you to use Ansible with Jenkins.

Recently, the Ansible Plugin for Jenkins was updated to version 0.8.0, and as a result it now works seamlessly with Ansible Vault.

Install Ansible plugins for Jenkins

The first thing you need to install is the Ansible plugin for Jenkins.
Also, install the dependent plugin called Credentials Binding.

One thing to note is you can only read from encrypted Ansible Vault encrypted files. You cannot create or modify them using the Jenkins Ansible plugin. So you will still need to use the CLI to modify your encrypted files.

Add Vault password to Jenkins

The simplest way to use Ansible Vault with Jenkins is to add your Vault password into a Jenkins Credential. Then you can bind the credential to an environment variable, or use it directly as a “Vault Credential”.

Configure your Jenkins freestyle project that uses an Ansible Playbook. If you have the newest version of the Ansible plugin, under the “Build” section you will now see an option for “Vault Credentials” since this is a new feature.

Add a new Jenkins Credential as “Secret text”, and enter your Ansible Vault encryption password.

Next, select the new Vault Credentials and you are ready to use your encrypted files in your Jenkins build job from your Ansible Playbooks.

Conclusion

In conclusion, you are now using Ansible Vault encrypted files automatically inside of your Jenkins build jobs. This helps you to keep your sensitive data secured in your source control. And Jenkins is helping you to deploy your Ansible Playbooks, while seamlessly decrypting your files.

Next, you can add a job to Jenkins that requires Ansible Vault. If you want to use Ansible for user management on Linux servers, check out the guide for User Management with Ansible.

The post Ansible Vault with Jenkins appeared first on Ryan Daniels.

The below commands are based on the post Upgrade your SSH keys. Check that out for more details.

Identify and rename old ssh keys

First, identify your old ssh keys:

$ cd ~/.ssh
$ for sshkey in $(ls *.pub);do echo $sshkey;ssh-keygen -lf $sshkey;done
/home/ryan/.ssh/id_rsa.pub
2048 SHA256:gIDkzI98pEna3j9+2Ja5di0+k2dCaXtCtx6k71dskA1 ryan@home (RSA)

From the output, this is showing it is an RSA 2048 key.

Next rename your old public and private ssh keys:

$ mv id_rsa.pub id_rsa_legacy.pub
$ mv id_rsa id_rsa_legacy

Stop using insecure keys

Use only keys that are greater than RSA 2048, or Ed25519.
Specifically, remove:

• DSA
• RSA 1024 and 2048
• ECDSA

Just be sure you aren’t using these ssh keys. Only remove them when you no longer need them and have the new keys setup and working.

Create or Change your existing password

If you old RSA ssh key didn’t use a password, now is the time to set a password. You can upgrade ssh keys that previously existed and didn’t use a password, without breaking anything. Also adding 100 rounds helps make your old key more secure when at rest if you need to continue using it.

$ ssh-keygen -f ~/.ssh/id_rsa_legacy -p -o -a 100

Upgrade ssh keys – Generate RSA 4096 ssh keys

RSA 4096 is good to use for legacy systems which do not yet support the new Ed25519 key.

Generate the RSA 4096 keys, and make sure to use a strong password:

$ ssh-keygen -t rsa -b 4096 -o -a 100
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ryan/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ryan/.ssh/id_rsa.
Your public key has been saved in /home/ryan/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:732MQCPPaMS815Y7MFXl4n2WXuawR1Zsat+PEG1TnPA ryan@home
The key's randomart image is:
+---[RSA 4096]----+
|         +. o+.oo|
|        o * .+*..|
|         B oo=Eo.|
|          =o=..oo|
|        S .BoBo+=|
|         .  *.Ooo|
|          .  B =.|
|        .   B * .|
|          .... . |
+----[SHA256]-----+

The “-o -a 100” means it is harder to crack the private key’s password using brute-force attacks. If your legacy system doesn’t support this, remove this option.

Upgrade ssh keys – Generate Ed25519 ssh keys

Ed25519 ssh keys work on modern systems (OpenSSH 6.7+) and are much shorter than RSA keys. Note, the “-o -a 100” option is implied with Ed25519 key generation.
Generate your new Ed25519 key and use a strong password:

$ ssh-keygen -t ed25519
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ryan/.ssh/id_ed25519.
Your public key has been saved in /home/ryan/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:oU44/TftLl2/d4IKQla36Z1TMqrcdK1xBcerye78fxM ryan@home
The key's randomart image is:
+--[ED25519 256]--+
|        . .      |
|     . . . . .   |
|    . o .   o o .|
|   .   + . o o = |
|    . + S * + +  |
|     = . = B  .E |
|      + B.=.oo ..|
|       =.=o.o ..=|
|      ..oo=+o..+=|
+----[SHA256]-----+

Now you have both RSA 4096 and Ed25519 ssh keys ready to go.

Next, add the keys to your ssh agent so it will remember the keys for you.

Install and configure ssh agent

Using an ssh agent allows you to type in a password once, and then the agent remembers the ssh keys.

On Ubuntu 16.04 there is one problem though. The built-in Gnome-keyring doesn’t support Ed25519. To work-around this, you could use the normal ssh-agent. But, I suggest instead to use gpg-agent and disable the gnome-keyring.

The below commands are based on the Arch wiki and an answer from the Ask Ubuntu Forum.

Disable Gnome ssh keyring daemon (reference):

$ cp -rp /etc/xdg/autostart/gnome-keyring-ssh.desktop ~/.config/autostart
$ echo "Hidden=true" >> ~/.config/autostart/gnome-keyring-ssh.desktop

Install gpg-agent:

$ sudo apt-get install gpa gnupg-curl

Enable ssh support in gpg-agent, and set a timeout to remember the key and password for 1 hour, since this adds some convenience:

$ echo "enable-ssh-support" >> ~/.gnupg/gpg-agent.conf
$ echo "default-cache-ttl-ssh 3600" >> ~/.gnupg/gpg-agent.conf
$ echo "max-cache-ttl-ssh 3600" >> ~/.gnupg/gpg-agent.conf

Edit your .bashrc to automatically load gpg-agent when you log in.

$ vi ~/.bashrc
# Set SSH to use gpg-agent
unset SSH_AGENT_PID
if [ "${gnupg_SSH_AUTH_SOCK_by:-0}" -ne $$ ]; then
export SSH_AUTH_SOCK="~/.gnupg/S.gpg-agent.ssh"
fi

Reload gpg-agent:

$ gpg-connect-agent reloadagent /bye

Next, add your new ssh keys to the gpg-agent:

$ ssh-add ~/.ssh/id_ed25519 ~/.ssh/id_rsa ~/.ssh/id_rsa_legacy

You will be prompted to enter your ssh key password. And after you enter that, another prompt will pop-up, from the gpg-agent. It will be used to unlock the gpg-agent’s key storage. This should be a different password than the password for your ssh keys. You will be prompted to enter this password every time you log in, or after the 1 hour timeout.

Enter the password that you want to use to unlock the gpg-agent key storage. And if you use the same password here for all your ssh keys, you only have to unlock it once:

And now you can list your keys stored in the gpg-agent:

$ ssh-add -l -E md5 # list fingerprint using md5 hash
$ ssh-add -l        # list fingerprint using sha256 hash
$ ssh-add -L        # list public key parameters

Re-deploy the new public keys

Lastly, you need to add your new public keys to your servers. This command will ssh to your server and add your new public keys to the authorized_keys file.
Change 1.2.3.4 to your server’s IP:

$ ssh 1.2.3.4 "mkdir -p .ssh;echo $(cat ~/.ssh/id_ed25519.pub) >> .ssh/authorized_keys;echo $(cat ~/.ssh/id_rsa.pub) >> .ssh/authorized_keys;chmod 700 .ssh;chmod 640 .ssh/authorized_keys"

Be sure to do this for ever server you connect to.

Important: Be sure you don’t lock yourself out of your servers. Have another session open and test to make sure you can log in again after making this change.

Now you can remove your old ssh key from the ssh agent and only use the secure ssh keys:

$ ssh-add -d 2>/dev/null;ssh-add ~/.ssh/id_ed25519;ssh-add ~/.ssh/id_rsa

Be sure to add the Ed25519 key first, like above. Since it seems to be a feature that prioritizes RSA keys first. So add the Ed25519 key first.

SSH to your servers to test the new key is working. You can also see in the logs the type of ssh key being used.

Confirm using the new Ed25519 ssh key, on Ubuntu 16.04:

$ grep " sshd\[" /var/log/auth.log|grep "Accepted publickey"|tail
Feb 11 10:00:00 home sshd[1802]: Accepted publickey for ryan from 192.168.1.100 port 60708 ssh2: ED25519 SHA256:oU44/TftLl2/d4IKQla36Z1TMqrcdK1xBcerye78fxM

Conclusion

Now you have more secure ssh keys since you are using Ed25519 and RSA 4096 keys. You have been able to upgrade ssh keys that are insecure! Next, you should look into updating your ssh server and client settings to use better encryption settings, using these references.

And if you don’t want to manually deploy the new ssh keys to all of your servers, instead you can use Ansible for user management.

The post Upgrade SSH keys and use gpg-agent with Ed25519 keys appeared first on Ryan Daniels.

What is Ansible?

First of all, Ansible is amazing at IT automation. Ansible is a command line IT automation solution that can deploy configuration changes, software, and perform many other tasks all automatically. To get more information about what Ansible is, check out the Ansible documentation.

Ansible is modular, so you can create groups of tasks, called roles. And then you can group the roles together to all run sequentially inside of a playbook.
An Ansible Playbook can run against one or multiple servers, depending how you reference the servers in your Playbook, and also depending on how you group the servers in your Ansible inventory file.

Setup Ansible

Ansible uses ssh to connect to all of your remote servers to perform the tasks you define. After you install Ansible, it’s a good idea to create a user for Ansible to use. You need to do this on all servers that Ansible will be managing. Instead of doing that manually, use Ansible!

Install Ansible

First, let’s install Ansible on the local machine.

On Ubuntu 16.04 install Ansible using apt-get:

$ sudo apt-get update && sudo apt-get install ansible

On CentOS/RHEL 7 install Ansible using yum:

$ sudo yum install ansible

Ansible local user and Ansible directory

On the local (control) server, all of the below Ansible commands are executed as the local Ansible user.

Create the user if it doesn’t exist and then run every command from now on as the Ansible user:

$ sudo useradd ansible

$ su - ansible

Create and go into the Ansible base directory:

$ mkdir ~/ansible
$ cd ~/ansible

Ansible Inventory

Ansible uses an inventory file which contains all the remote servers you will be connecting to. This is the first thing you need do in order to setup Ansible.

An example of a simple inventory file using a group name “ubuntu-dev” with the hostname of two remote servers:

$ cat > ~/ansible/hosts << 'EOF'
[ubuntu-dev]
ubuntu-web
ubuntu-db
EOF

Create SSH Key

Create an ssh key for the local Ansible user using RSA keys, since that is still more common than ed25519:

$ ssh-keygen -t rsa -b 4096

SSH to all remote servers

You need to ssh to the servers beforehand, since you need to have the ssh host keys in your known_hosts file for security reasons. Therefore, make sure you are connecting to the expected servers.

You could do this manually, but instead use Ansible to ssh to all the servers in your Ansible inventory file to populate the known_hosts file. Do not run this in a production environment and make sure you are connecting to the expected servers by verifying the ssh key fingerprints.

$ ansible all -m ping --extra-vars "ansible_ssh_common_args='-o StrictHostKeyChecking=no'" -i hosts

This command is using the Ansible ping module (-m ping) to ssh to all servers in your inventory file (-i hosts), and this will create entries in the ~/.ssh/known_hosts file.

Install Ansible required packages

Next, you need to install some packages that Ansible requires on the remote servers. Currently, Ansible uses Python 2 by default. So your remote servers will need to have Python 2 installed.

Use Ansible to install Python 2 on a remote Ubuntu server:

$ ansible ubuntu-dev -m raw -a "apt-get update && apt-get install -y python-minimal python-simplejson" -i hosts

If using CentOS 7 or RHEL 7 you don’t need to install Python 2 since it is already installed by default.

But, if using an old version like CentOS 5 or RHEL 5, you will need to run these commands. And if using SELinux with CentOS 5 or RHEL 5, run these commands to allow the Ansible copy/file/template modules.

Create Ansible user on remote servers

You can use my Ansible role on GitHub called create-user-ansible to have Ansible create its own user.

This new user will then use password-less sudo by default. But you can change this with a variable if you want to be prompted for a password every time you run Ansible.

Clone the git repository:

$ git clone https://github.com/ryandaniels/ansible-role-create-user-ansible.git ~/ansible/roles/create-user-ansible

Create the Ansible Playbook file:

$ cat > ~/ansible/create-user-ansible.yml << 'EOF'
---
- hosts: '{{inventory}}'
  become: yes
  roles:
  - create-user-ansible
EOF

Now the most important step, run the Ansible playbook.

This assumes you have a way to log in via ssh to your servers. (How else would you remotely connect?)
Change “remote_existing_user” to the username you already have on your servers.
Change “username” to your local Ansible username. This should be the same user as the local Ansible user since this makes things easier.

As a result, this will create a new user on all your servers named: ansible
More options are available, see the README on GitHub.

If your existing user uses a password for ssh authentication and does not use password-less sudo:

$ ansible-playbook create-user-ansible.yml --ask-pass --become --become-method=su --ask-become-pass --extra-vars "inventory=all ansible_ssh_user=remote_existing_user username=ansible" -i hosts

If your existing user uses password-less ssh using ssh keys and no password for sudo:

$ ansible-playbook create-user-ansible.yml --become --become-method=sudo --extra-vars "inventory=all ansible_ssh_user=remote_existing_user username=ansible" -i hosts

Conclusion

In conclusion, now you have an Ansible user created on all of your servers. You have installed and setup Ansible, and this is the first step! Next, write your own Playbook. Or use one of mine to update all of your Ubuntu or CentOS servers and automatically reboot them.

This guide is a quick way to get started with Ansible. Keep in mind, Ansible can be complex. You should plan out how you want to setup Ansible’s directory structure by following their best practices. And take some time to read their documentation!

The post Getting Started with Ansible on Ubuntu or CentOS appeared first on Ryan Daniels.

Using a VPN (Virtual Private Network) can help to protect your privacy. You can use a VPN to appear like you are in another country so you can reach a website that was previously blocked. You can also use a VPN to stop your ISP from restricting you to certain websites, and when travelling to protect your data while using insecure WiFi. There are many many more reason. However, there can be an extra benefit to using a VPN that people don’t usually think about: Automatic content blocking (also known as ad blocking).

Why?

Ad blocking is now a necessity because of dangerous ads. Some ad networks have had problems with ads containing malicious JavaScript which is actually malware. Even recently YouTube was showing ads that contained a cryptocurrency miner. An add-on like uBlock Origin for Firefox or Chrome can help. And customizing your Firefox privacy settings will also help. But what if apps on your phone are showing dangerous ads? Or, what if you have an old phone that’s no longer getting updates and you want better protection from the recent Meltdown vulnerability? A browser add-on alone will not help you.

Now, combine the benefits of a VPN, and ad blocking into one solution. This is where OpenVPN with ad blocking using dnsmasq comes in. For this to work you need control of the OpenVPN server configuration. So using a VPN provider is not an option. But, this way is cheaper! Plus you get the satisfaction of setting up your own VPN!

Setup a Raspberry Pi with OpenVPN

You can create your own VPN with a Raspberry Pi. If you live in Canada, a good way to get started is with the Raspberry Pi 3 CanaKit. There are many guides for setting up a Raspberry Pi with OpenVPN.

However, this method won’t help if a website restricts your country, or if your ISP blocks you from certain content, since you will have the Raspberry Pi at home. But, you can still use this method to protect you when using open WiFi while travelling, or when at home to get the benefit of ad blocking on your mobile devices.

Setup a VPS with OpenVPN

If using a Raspberry Pi isn’t your thing, you can buy an inexpensive VPS (Virtual Private Server) and host your VPN from there. This may seem like more work, but it’s pretty easy. And with a VPS you don’t need to worry about exposing your home network to the public internet. Also, when using a VPS that is only for OpenVPN, you can use common ports to have a better chance of being able to connect to your VPN.

Linode is a great VPS provider for $5 per month. Or, Vultr offers a VPS for $2.50 per month! (For IPv6 only, IPv4 is $3.50). Whichever VPS provider you choose, make sure to create your VPS in a country that’s right for you. For example, if you need to access a website usually only available to a certain country. Also make sure you obey their TOS.  I’m not responsible for anything you do from following this guide!

When using your own VPS, I suggest setting it up with OpenVPN using the Streisand project. It has many options to bypass blocking if you live in an oppressive country. Note that the commands below will only install OpenVPN (with stunnel) if using Streisand. Plus, Streisand does a good job at protecting your VPS with security tweaks and automatically updating packages.

Install OpenVPN

You can install OpenVPN many different ways. Here are two ways I recommend:

To install OpenVPN via PiVPN, follow these steps.

To install (only) OpenVPN (also with stunnel) via Streisand, follow the prerequisites steps, and then run:

$ remote_server_IP=1.2.3.4  # Change IP to your VPS/remote server IP
$ git clone https://github.com/ryandaniels/ansible-role-dnsmasq-adblock.git ~/ansible/roles/dnsmasq-adblock
$ git clone https://github.com/StreisandEffect/streisand.git && cd streisand
$ ~/streisand/deploy/streisand-existing-cloud-server.sh --ip-address $remote_server_IP --ssh-user username123 --site-config ~/ansible/roles/dnsmasq-adblock/files/streisand-local-site.yml

Change “remote_server_IP” to your server’s IP.
This will use the user “username123” with ssh keys for a passwordless ssh connection. Be sure to use a use that you’ve setup with ssh keys.

Setup ad blocking for OpenVPN using dnsmasq

Now that you have OpenVPN setup on a RaspberryPi or your own VPS you are ready for the last step, ad blocking. For this you can use a program called dnsmasq. It performs DNS lookups and you can modify the dnsmasq behaviour to block certain domains.

Manually install and configure dnsmasq on Ubuntu

These steps were modified from this github project by Bob Nisco and also uses Steve Black’s host project.

Install and configure dnsmasq using the DNS servers from OpenDNS:

$ sudo apt-get -y install dnsmasq
$ sudo wget https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -O /etc/hosts_blocked

$ sudo cat > /etc/dnsmasq.conf << 'EOF'
domain-needed
bogus-priv
no-resolv
server=208.67.222.222
server=208.67.220.220
listen-address=127.0.0.1
listen-address=10.8.0.1
addn-hosts=/etc/hosts_blocked
EOF

Configure OpenVPN:

$ sudo vi /etc/openvpn/server.conf

Add below, and make sure no other lines have “dhcp-option DNS”:

push "dhcp-option DNS 10.8.0.1"

Create a script to update the domains being blocked. And have it run every week:

$ sudo cat > /etc/cron.weekly/adblock_dl_hosts << 'EOF'
#!/bin/bash
wget https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -O /etc/hosts_blocked && systemctl restart dnsmasq.service
EOF

$ sudo chmod +x /etc/cron.weekly/adblock_dl_hosts

Restart dnsmasq and OpenVPN:

$ sudo systemctl restart dnsmasq.service 
$ sudo systemctl restart openvpn.service

Using Ansible to automatically install and configure dnsmasq on Ubuntu

If you are familiar with Ansible, then you can add a new role to handle the installation and configuration of dnsmasq for ad blocking automatically. If you are not familiar with Ansible but want to know more, check out my post explaining what is Ansible.

I’ve created this Ansible role to configure dnsmasq on Ubuntu or Raspbian.

Create the Ansible Playbook:

$ cat > ~/ansible/install-dnsmasq-adblock.yml << 'EOF'
---
- hosts: '{{inventory}}'
  become: yes
  roles:
  - dnsmasq-adblock
EOF

If using PiVPN or a normal OpenVPN installation (and not Streisand), clone the git repository and run the Ansible Playbook:

$ git clone https://github.com/ryandaniels/ansible-role-dnsmasq-adblock.git ~/ansible/roles/dnsmasq-adblock
$ ansible-playbook install-dnsmasq-adblock.yml --extra-vars "inventory=openvpn-server" -i hosts

If you are using Streisand, then you already cloned the git repositories. Just run the Ansible Playbook:

$ ansible-playbook install-dnsmasq-adblock.yml --extra-vars "inventory=streisand-host adblock_manage_openvpn=false" -i ~/streisand/inventories/inventory-existing

Conclusion

In conclusion, now you are more secure by using OpenVPN with ad blocking on your own Raspberry Pi or VPS. And if you are using your own VPS you can use the Streisand project to secure your installation and also to expose common ports to have a higher chance of being able to connect to your VPN from anywhere.

The post How to setup OpenVPN with ad blocking on Raspberry Pi or a VPS appeared first on Ryan Daniels.