Let’s discuss the background of firewall issues with Docker, and a working solution for my use case (either setup manually or using Ansible). By the end we will use a firewall on the server to lock down everything by default, only allowing my trusted IPs! With the option to open specified ports publicly (like SSH).

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Background

Firstly, even if you were using a firewall like iptables, Docker makes that useless. Docker punches a whole right through your firewall!

And if you try another firewall, like firewalld? Docker Swarm (and even regular Docker) with firewalld is a complete mess. Restart the firewalld service, or change the firewalld config, and you lost all the config that Docker needed. Now you have to restart Docker! What happens if the firewalld service failed and restarted? That Docker Swarm node is out of service.

Keep in mind, when I mention Docker, it means the regular Docker Engine. Docker Swarm means SwarmKit, which is the newer way of using Swarm. (The old way is the standalone solution which is old and not referenced at all here).

There are many attempts to solve this problem by users of Docker. Unfortunately the Docker team has been pretty quiet about these issues. They recommend a manual user solution, and to disable Docker’s use of iptables. I’m speculating here, but it seems like any future change from Docker will likely be a breaking change since this is a complicated issue to fix.

The attempted solutions (from users) are not very straight forward for a normal Docker user. And that’s the real issue. On top of that, out of the box (after you start one service with a published port), there is no security at the network layer. Anyone can connect to an exposed container’s port. Most importantly, even if you think you have a firewall protecting you.. Wrong, you don’t! With normal Docker, you can bind your service to your localhost which helps. But what about Docker Swarm? Nope, that doesn’t work.

There’s a great article about “The problem of forcing users to make choices (in security).” Definitely worth the read!

Roll your own solution

Until this is actually addressed in Docker, our only hope is to find the simplest solution possible. And turning off iptables integration in Docker is unacceptable (which is constantly recommended by the Docker developers). The other option is to move on from Docker and/or Docker Swarm. I hear this thing called Kubernetes is pretty great. Anyways, back to Docker.
Many people have spent hours trying to learn, and figure out iptables as a solution to this. You need to roll your own solution apparently! So iptables is probably the best approach, since that is what Docker needs to use to do it’s magic (and most other firewalls are just a wrapper for iptables anyway depending on your OS).

Something fun I found out while testing this, Docker Swarm uses iptables in an undocumented way. Docker Swarm uses the iptables INPUT chain! It’s only for encrypted overlay networks. But it’s not very fun realizing that! All of a sudden rules are being appended to the INPUT chain.

Okay, enough backstory. On with my futile attempt to roll my own solution. This took way longer than I thought it would! But it does work! Currently it works at least.. (That’s why I use the word futile!)

Problems I need to solve

1. Only allow traffic from multiple “trusted” IP addresses to my servers. Not all of these IPs will be in the same “IP block/range” either. This will be to all services running directly on the server, and also all of the Docker containers.
2. Let only specific ports be publicly accessible, like SSH.
3. I’m not managing which containers are accessible through the firewall. Meaning, I’m not manually adding ports into my firewall solution. That kind of manual work is not happening. I need a dynamic, and flexible solution that blocks by default except to my trusted IPs.
4. The firewall solution must be simple. More complex means more room for error.
5. The firewall solution must not impact performance significantly.
6. Restarting the firewall won’t break Docker.
7. Restarting Docker won’t break the firewall.
8. No impact to running server processes or Docker services when making a change. Things need to keep working!
   Firewall changes need to happen online and not impact Docker. Meaning I can’t be restarting Docker because I made a firewall change.
   Docker “changes” need to happen online. Meaning I can’t be restarting the firewall because I made a Docker change. (A Docker “change” means starting/stopping a container).

That sounds very simple! Unfortunately, it is not with Docker (and Docker Swarm).

Solution – Docker firewall with iptables and ipset

If you don’t know much about iptables, or ipset, that’s okay. You don’t really need to know. You should have some basic understandings though, so you don’t break your servers! The Arch Linux wiki has great information about iptables. Including this helpful visual about the iptables flow.

Note: This solution works with CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

High level summary

iptables with ipset will handle all of this for us. And keep our servers, and Docker locked down from the network level.

In this solution, we will use the iptables INPUT chain to jump to another chain (let’s call our custom chain FILTERS), but return if there’s some legitimate looking traffic, so the Swarm overlay can do whatever it wants in INPUT with IPSEC, or whatever it is appending to INPUT.
Inside our custom chain FILTERS, we drop everything that doesn’t match our trusted list of IPs. We also allow our SSH port and the basic default iptables stuff.. You can also add any OS port to be publicly accessible.
The DOCKER-USER chain only needs a few entries. Any internal Docker traffic is returned, and it will drop any other traffic that’s not in our allowed IP list. You can also add any container port to be publicly accessible.

One of the dangers with this approach is if Docker changes it’s behaviour our firewall could break, or our Docker services could stop working. Since Docker doesn’t offer any solution for their users, we need our own solution. So keep in mind that you need to test this when upgrading to a new version of Docker. That is the trade-off with a “roll your own” solution. But what choice do we have?

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

Warnings

Warning: Be sure you have everything needed in your configuration. Once the iptables firewall is started it blocks anything that wasn’t added! Don’t lock yourself out of your server. Be sure to have another way to connect, like a console.

Disclaimer: Keep in mind, you should test all of this in your lab or staging environments. I can’t guarantee this will be 100% safe and can’t be held responsible for anything going wrong!

SELinux Bug: If using SELinux, currently there’s a bug with SELinux which prevents saving the iptables rules to the iptables.save file.
Impact: Saving the iptables rules a 2nd time will silently fail. Workaround has been added so SELinux allows chmod to interact with the iptables.save file. See notes on GitHub for SELinux workaround steps. Alternatively you could disable SELinux, but that’s not recommended. Bug report: https://bugs.centos.org/view.php?id=12648

The manual way

Run the commands below. These commands are only for CentOS/RHEL 7. If you don’t want to do this manually, jump to the Automatic section, using Ansible (which also works with Ubuntu).

Prep

Make note of what you already have in iptables (if you are already using it). Be sure you have some background with iptables, since you could break things!

# iptables -nvL --line-numbers

Install the required packages for CentOS / RHEL:

# yum install iptables iptables-services ipset ipset-service

Configure ipset

ipset allows you to add a list of IPs that you can use with iptables. In our case, we will add a list of IPs we want to be able to connect to our servers.

Configure ipset with a setname of ip_allow.
Add IPs you want to allow. Change the IPs below to your actual trusted/allowed IP ranges. Be sure to include your Docker server IPs here, because if you don’t they can’t communicate with eachother:

# mkdir -p /etc/sysconfig/ipset.d
# vi /etc/sysconfig/ipset.d/ip_allow.set

create -exist ip_allow hash:ip family inet hashsize 1024 maxelem 65536
add ip_allow 192.168.1.123
add ip_allow 192.168.100.0/24
add ip_allow 192.168.101.0/24
add ip_allow 192.168.102.0/24

Start, and Enable the ipset service:

# systemctl status ipset
# systemctl start ipset
# systemctl enable ipset

See what ipset has in it’s loaded configuration:

# ipset list | head

Important: Make note of the size of “Number of entries”. If that number is close to the maxelem size (65536), then you need to delete the ipset and re-create it with a larger max size. If you only use a few IP ranges like above, you don’t need to worry and will be well below the limit.

Configure iptables

Next up, iptables. iptables is our solution for a firewall. We will create a file with our rules and then add those rules into iptables. The important part is to not flush the existing rules if you are already using Docker (or Docker Swarm) on your server.

Create an iptables file to use with iptables-restore, to add the rules into iptables:

# vi iptables-rules.txt

Add below to the file. There is a lot going on here..

*filter
:DOCKER-USER - [0:0]
:FILTERS - [0:0]
#Can't flush INPUT. wipes out docker swarm encrypted overlay rules
#-F INPUT
#Use ansible or run manually once instead to add -I INPUT -j FILTERS
#-I INPUT -j FILTERS
-F DOCKER-USER
-A DOCKER-USER -m state --state RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -i docker_gwbridge -j RETURN
-A DOCKER-USER -s 172.18.0.0/16 -j RETURN
-A DOCKER-USER -i docker0 -j RETURN
-A DOCKER-USER -s 172.17.0.0/16 -j RETURN
#Below Docker ports open to everyone if uncommented
#-A DOCKER-USER -p tcp -m tcp -m multiport --dports 8000,8001 -j RETURN
#-A DOCKER-USER -p udp -m udp -m multiport --dports 9000,9001 -j RETURN
-A DOCKER-USER -m set ! --match-set ip_allow src -j DROP
-A DOCKER-USER -j RETURN
-F FILTERS
#Because Docker Swarm encrypted overlay network just appends rules to INPUT. Has to be at top unfortunately
-A FILTERS -p udp -m policy --dir in --pol ipsec -m udp -m set --match-set ip_allow src --dport 4789 -j RETURN
-A FILTERS -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FILTERS -p icmp -j ACCEPT
-A FILTERS -i lo -j ACCEPT
#Below OS ports open to everyone if uncommented
-A FILTERS -p tcp -m state --state NEW -m tcp -m multiport --dports 22 -j ACCEPT
#-A FILTERS -p udp -m udp -m multiport --dports 53,123 -j ACCEPT
-A FILTERS -m set ! --match-set ip_allow src -j DROP
-A FILTERS -j RETURN
COMMIT

Use iptables-restore to add the above rules into iptables. The very important flag is -n. That makes sure we don’t flush the iptables rules if we have rules already in Docker (or Docker Swarm).

# iptables-restore -n < iptables-rules.txt

Next, add a rule to the INPUT chain, so we start using the new rules in FILTERS. It has to be at the top, and only needs to be added once:

# iptables -I INPUT 1 -j FILTERS

Save the iptables rules:

# /usr/libexec/iptables/iptables.init save

That will save any existing and our new iptables rules to the iptables configuration file so it will be persistent after a reboot.

In addition, it was needed to run the above iptables command manually since we want to ensure it’s only inserted once. And we can’t flush the INPUT chain to ensure that since Docker Swarm could have rules there already.

Start and Enable the iptables service:

# systemctl status iptables
# systemctl start iptables
# systemctl enable iptables

If you want to customize the iptables rules to allow more ports to be open to everyone, just add the port to the appropriate rule in the iptables file (tcp or udp), then re-run the same commands from above:

# iptables-restore -n < iptables-rules.txt
# /usr/libexec/iptables/iptables.init save

Don’t miss the Warnings from above! Especially about SELinux.

If you don’t want to do all of that manually, and you use Ansible, then do this instead..

The Automatic way with Ansible

Manually run all of the above, on every Docker server is not ideal. Let’s use Ansible instead!

I’ve created an Ansible Role: iptables for Docker, on GitHub and Ansible Galaxy.

This works on CentOS 7, RHEL 7, Ubuntu 18.04, and Ubuntu 20.04.

Install the Ansible Role using Ansible Galaxy:

$ ansible-galaxy install ryandaniels.iptables_docker

Or, clone the GitHub project:

$ git clone https://github.com/ryandaniels/ansible-role-iptables-docker.git roles/ryandaniels.iptables_docker

Create the Ansible Playbook, called iptables_docker.yml:

---
- hosts: '{{ inventory }}'
  become: yes
  vars:
    # Use this role
    iptables_docker_managed: true
  roles:
  - ryandaniels.iptables_docker

Make configuration changes to add desired IP addresses and ports as needed.

Don’t miss the Warnings from above!

Then run the playbook:

$ ansible-playbook iptables_docker.yml --extra-vars "inventory=centos7" -i hosts-dev

Conclusion

In conclusion, now we have secured our Docker (and Docker Swarm) environments using Ansible to perform the installation and configuration of iptables! None of our Docker published ports are exposed to the world, unless we want them to be! We have created a custom Docker firewall with iptables. Hopefully, some day this will be the default behaviour and shipped with Docker out of the box! Dare to dream. Security is hard.

References

Background for Docker’s undocumented use of iptables INPUT chain

See my previous post about this.

References and links

References, notes, and links about the Docker firewall discussion:

• Docker Documentation – Overlay Networks
• Docker bypasses ufw firewall rules
• unrouted – Solution using iptables. Not for Swarm. It clobbers the INPUT chain, which is used by encrypted overlay with Docker Swarm
• The big thread about Docker and a firewall
• Arch Linux wiki to the rescue to show iptables flow which links to a great visual
  
  

The post Secure Docker with iptables firewall and Ansible appeared first on Ryan Daniels.

Background on firewalld

Firewalld is a firewall which runs on many Linux distributions. It is software which runs on a server, and is used to locally secure the server. For example, it can restrict which ports are available for all source IPs, or it can further restrict access by allowing only certain IPs to a specific port (and zone). Check out the documentation to read more about the features of firewalld.

Everything is blocked when firewalld starts! The only exception is for ssh on port 22. This is normally what you want on a new server.

Problem

However, firewalld isn’t with out it’s limitations. To run commands to configure which ports you want to allow through firewalld, you need to have the firewalld service running. But, what if you are adding firewalld to a server that has an application which is already in use? As soon as you start firewalld, it will be blocked!

There are two ways around this.

1. Copy your firewalld config files in the right place, with all the right ports, zones, etc. And then start firewalld. This can work. But having Ansible copy files around and deal with all that is a challenge since firewalld behaves a certain way.
2. Use the “offline” firewalld commands!

Let’s explore options 2. And lucky for us, we want to use Ansible to help us manage this configuration. And, Ansible can make use of “offline” mode.
But, unlucky for us, the Ansible module which handles firewalld can’t do everything. (At least not everything that I want it to do).

I want the configuration for firewalld to be as simple as possible. I don’t want the configuration for services in one place, and the configuration for ports in another.
But more importantly, I don’t want any service interruption.

Solution

Create my own Ansible role that does what I want.

If you’re new to Ansible, check out my Ansible Getting Started Guide.

I want firewalld to behave in a certain way. Meaning if I have an application already running, and I want firewalld to start restricting access, I don’t want an outage on that application as soon as I start up the firewalld service on the server.

Firewalld Limitations

Firewalld does have some limitations. You should be aware of this before using it.
Anything firewalld is not aware of in the back-end (for example if using iptables), will be removed since firewalld wants to be the single source of truth for firewall rules.
This is why firewalld and Docker do not get along.
See details from: here, here, and here.

In case you missed that..
Do not use firewalld and Docker.
Update: Instead, if you need a solution to work with Docker try using iptables directly. See my post about securing Docker with iptables and Ansible.

Firewalld Ansible Role

You can find my firewalld Ansible Role on GitHub or in Ansible Galaxy.

This Ansible Role manages firewalld in offline and persistent mode only. I don’t want to mess around with non-persistent changes. And using online mode is not an option.

It uses the Ansible module for firewalld for a few things, but most changes have to be done using the command module. Specifically: “firewall-offline-cmd”.

Why? What’s the benefit doing it this way? It let’s you create your own custom service in firewalld. Even built-in firewalld services can be used, and easily modified. For example, moving the ssh port from 22 to something else. (Warning, be careful changing your SSH port. You could lock yourself out!)

Here’s a short example of the configuration:

rhel_firewalld_zone_source:
  - zone: internal
    state: enabled
    source:
      - "192.168.22.64/26"
      - "192.168.23.64/26"

rhel_firewalld_custom_service:
  - name: zabbix-agent
    zone: public
    state: enabled
    port_protocol:
    # - 10050/tcp
      - 3333/tcp
  - name: openvpn
    zone: public
    state: enabled
  - name: app123-public
    zone: public
    state: enabled
    description: app123 firewall rules for public zone
    port_protocol:
      - 5000/tcp
  - name: app123-internal
    zone: internal
    state: enabled
    description: app123 firewall rules for internal zone
    port_protocol:
      - 8080/tcp
      - 9000/tcp

In the variable rhel_firewalld_zone_source:
This will configure the Zone named internal to only allow IP ranges that are in source.

In the variable rhel_firewalld_custom_service:
This will configure the zabbix-agent service (which is actually built-in) for the zone public, but not use the default port (of 10050, commented out for demonstration). Instead it’s using port 3333.
The openvpn service is added to the zone public, and will just use the default configuration (since this is also a built-in firewalld service). You need to know it’s built-in, if you don’t specify a port. So it’s usually better to just specify a port all the time!
Last, there are two custom services, app123-public and app123-internal, which are added to their defined zones. They are also adding port 5000 using tcp (attached to the public zone), and adding ports 8080 and 9000 both using tcp (attached to the internal zone).

If you decide you want to change a port, just remove it from the configuration.

More details are on GitHub. Be sure to read the README!

Getting Started – Ansible firewalld

Install from Ansible Galaxy:

ansible-galaxy install ryandaniels.firewalld

Or, clone the GitHub project:

git clone https://github.com/ryandaniels/ansible-role-firewalld.git roles/firewalld

Create the Ansible Playbook, called firewalld.yml:

---
- hosts: '{{ inventory }}'
  become: yes
  vars:
    # Use this role
    rhel_firewalld_managed: true
  roles:
  - firewalld

Make configuration changes add desired firewalld services and ports.

Warning: Be sure you have everything needed in your configuration. Once firewalld is started it blocks anything that wasn’t added!

Disclaimer: Keep in mind, you should test all of this in your lab or staging environments. I can’t guarantee this will be 100% safe and can’t be held responsible for anything going wrong!

Then run the playbook:

ansible-playbook firewalld.yml --extra-vars "inventory=centos7" -i hosts-dev

Conclusion

In conclusion, Ansible is now being used to manage firewalld rules! We have a custom firewalld service created using two different zones. The built-in firewalld services are also enabled, but we don’t really care they are built-in, since we can change the port they use if we want.
We have a nice and simple configuration, and this was all done in “offline” mode so there was no impact to any existing services using the ports.

Again, your mileage may vary. Always test in your lab or staging environment first! I’m not responsible for anything you break.

Stay safe!

The post Manage firewalld with Ansible appeared first on Ryan Daniels.

What’s the problem?

If you have multiple servers to manage, it can be a pain to manually add a new user, change a password, or lock an old account. Manually logging into all of your servers and performing these tasks is a real pain, and a huge waste of time.

Luckily there are several solutions.

Solution: Red Hat IDM

There are many different solutions for user management. Red Hat Identity Management (IDM) is a very good solution that works with many Linux servers. IDM is a central location that holds all the user configuration. And it can even connect to Windows Active Directory.

However, if you don’t have a subscription to Red Hat Enterprise Linux there are other options.

Solution: FreeIPA

FreeIPA is the upstream open source project for Red Hat Identify Management. So you can download the FreeIPA package and follow one of many available tutorials for setting up a server and client.

Solution: Ansible for user management

If the above solutions seem too complicated there’s another option. However, you should really take some time to setup user management properly. Use FreeIPA. It’s free!

But, if you want to use another solution while you are setting up FreeIPA, and you already have Ansible as part of your environment then you have another option. Ansible can do this of course!

The reason Ansible is a decent option for user management is because it is idempotent. This means, Ansible will only make a change if the change is needed to get to the desired state. So you can run the same Ansible Playbook multiple times, and if the Ansible tasks are written properly Ansible will only make a change the first time.

Just keep in mind, I warned you to use FreeIPA. This solution is not “enterprise” grade. But it certainly works for certain scenarios, like in a lab environment. And if you want to use Ansible for something new, there is a user management role available on GitHub or on Ansible Galaxy.

If you aren’t familiar with Ansible, check out my Getting Started with Ansible guide.

This Ansible role is tested on:

• Ubuntu
• Red Hat Enterprise Linux (RHEL) 7.x, 6.5, 5.9
• CentOS 7.x, 6.5, 5.9

What this user management Ansible role does

This role will manage users in the user list configuration file (list is in the file vars/lab_users.secret in the example below).
It can add users, change passwords, lock/unlock user accounts, manage sudo access (per user), add ssh key(s) for ssh key based authentication.
This is done on a per “group” basis (using Ansible group variables), as set in the configuration file. The group comes from the Ansible group as set for a server in the inventory file.

Note: Deleting users is not done on purpose.

The great thing about Ansible Playbooks, if you don’t like something it’s easy enough to modify.

Install the Ansible user management role

First let’s install the Ansible role to manage users into your Ansible control server:

$ su - ansible
$ cd ~/ansible
$ git clone https://github.com/ryandaniels/ansible-role-create-users.git roles/create-users

Or, you can use Ansible Galaxy to install this role:

$ su - ansible
$ ansible-galaxy install ryandaniels.create_users

Setup Ansible Vault

If you don’t already have Ansible Vault configured, you will need to set it up now. Vault uses AES encryption to store your sensitive information. And we need Ansible Vault to encrypt our user configuration file since we don’t want to be exposing even a hashed password into source control.

First, (if using git), make sure to update your .gitignore file so your Vault password isn’t saved to your source control. And also add the secret file we will be creating later:

$ vi .gitignore
.vaultpass
secret
*.secret

Next, create a password for your Ansible Vault and save it in your Password Manager (like KeePassXC). Then create the .vaultpass file, add the Vault password, and fix the permissions:

$ vi .vaultpass
#Enter password here

$ chmod 600 .vaultpass

Also you need to update the Ansible config file to reference where the Vault file is located:

$ vi ansible.cfg
[defaults]
vault_password_file = ./.vaultpass

Create Ansible Playbook

Next, create the Playbook file:

$ vi create-users.yml
---
- hosts: '{{inventory}}'
  vars_files:
    - vars/lab_users.secret
  become: yes
  roles:
  - create-users

Create configuration for user management

Next, add your users into a configuration file. The below is only an example, don’t use it in your user management configuration file. Also, make sure the filename matches a .gitignore entry. In this case it will match “*.secret”.
Use the special Ansible command to create the encrypted Ansible Vault file:

$ mkdir -p vars
$ ansible-vault create vars/lab_users.secret
---
users:
  - username: alice
    password: $6$/y5RGZnFaD3f$96xVdOAnldEtSxivDY02h.DwPTrJgGQl8/MTRRrFAwKTYbFymeKH/1Rxd3k.RQfpgebM6amLK3xAaycybdc.60
    update_password: on_create
    comment: Test User 100
    shell: /bin/bash
    ssh_key: |
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8crAHG/a9QBD4zO0ZHIjdRXy+ySKviXVCMIJ3/NMIAAzDyIsPKToUJmIApHHHF1/hBllqzBSkPEMwgFbXjyqTeVPHF8V0iq41n0kgbulJG alice@laptop
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8crAHG/a9QBD4zO0ZHIjdRXy+ySKviXVCMIxxxxxxxxxxxxxxxxxxJmIApHHHF1/hBllqzBSkPEMwgFbXjyqTeVPHF8V0iq41n0kgbulJG alice@server1
    exclusive_ssh_key: yes
    use_sudo: no
    use_sudo_nopass: no
    user_state: present
    servers:
      - webserver
      - database
      - monitoring

  - username: bob
    password: $6$XEnyI5UYSw$Rlc6tXtECtqdJ3uFitrbBlec1/8Fx2obfgFST419ntJqaX8sfPQ9xR7vj7dGhQsfX8zcSX3tumzR7/vwlIH6p/
    ssh_key: AAAAB3NzaC1yc2EAAAADAQABAAACAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8crAHG/a9QBD4zO0ZHIjdRXy+ySKviXVCMIxxxxxxxxxxxxxxxxxxJmIApHHHF1/hBllqbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbulJG bob@laptop
    use_sudo: no
    user_state: lock
    servers:
      - webserver
      - database

In the above example, there are two users. Users: alice and bob. If you are familiar with user management on Linux, then most of these settings are self explanatory. For example, both Alice and Bob have a hashed password and ssh keys. Alice actually has two ssh keys configured. But Bob’s account is locked so he can’t log in.

The interesting part is the list of “servers” in the configuration. These are groups defined in your Ansible hosts file. So you can give certain users access to a specific group of servers, depending on the user’s role.

Important note: Be careful with the update_password setting. When set to always, the user’s password will be changed to what is defined in password. This might not be what they wanted if they’ve manually changed their password so it’s usually safer to use on_create.

For details about all the different settings, see the README on GitHub.

Going forward, to update the file, instead of the “create” command line option, use “edit”:

$ ansible-vault edit vars/lab_users.secret

And don’t forget to save this into your source control (git).

Run the Ansible Playbook

$ ansible-playbook create-users.yml --extra-vars "inventory=all-dev" -i hosts-dev

Marvel at the output generated by Ansible. All these users are created, updated, or locked, but only if something needed to change since Ansible is idempotent.

Bonus: Add this in Jenkins

If you also use Jenkins for your CI/CD pipeline, you can add this Ansible Playbook into Jenkins.

You just need to setup Jenkins to use Ansible and Ansible Vault. And there is a great Jenkins Plugin to help. Just follow this guide to use Ansible Vault with Jenkins. Keep in mind that updating the user configuration file is only possible from the command line, since it’s encrypted using Ansible Vault.

Conclusion

In conclusion, there are many options for user management on Linux servers. From enterprise solutions like Red Hat IDM. Or, using the upstream open source software IDM uses directly, FreeIPA. If you decide to use Ansible, then the above Ansible role will get the job done. And you can also add it into Jenkins.

The post User Management with Ansible appeared first on Ryan Daniels.

What is Ansible?

First of all, Ansible is amazing at IT automation. Ansible is a command line IT automation solution that can deploy configuration changes, software, and perform many other tasks all automatically. To get more information about what Ansible is, check out the Ansible documentation.

Ansible is modular, so you can create groups of tasks, called roles. And then you can group the roles together to all run sequentially inside of a playbook.
An Ansible Playbook can run against one or multiple servers, depending how you reference the servers in your Playbook, and also depending on how you group the servers in your Ansible inventory file.

Setup Ansible

Ansible uses ssh to connect to all of your remote servers to perform the tasks you define. After you install Ansible, it’s a good idea to create a user for Ansible to use. You need to do this on all servers that Ansible will be managing. Instead of doing that manually, use Ansible!

Install Ansible

First, let’s install Ansible on the local machine.

On Ubuntu 16.04 install Ansible using apt-get:

$ sudo apt-get update && sudo apt-get install ansible

On CentOS/RHEL 7 install Ansible using yum:

$ sudo yum install ansible

Ansible local user and Ansible directory

On the local (control) server, all of the below Ansible commands are executed as the local Ansible user.

Create the user if it doesn’t exist and then run every command from now on as the Ansible user:

$ sudo useradd ansible

$ su - ansible

Create and go into the Ansible base directory:

$ mkdir ~/ansible
$ cd ~/ansible

Ansible Inventory

Ansible uses an inventory file which contains all the remote servers you will be connecting to. This is the first thing you need do in order to setup Ansible.

An example of a simple inventory file using a group name “ubuntu-dev” with the hostname of two remote servers:

$ cat > ~/ansible/hosts << 'EOF'
[ubuntu-dev]
ubuntu-web
ubuntu-db
EOF

Create SSH Key

Create an ssh key for the local Ansible user using RSA keys, since that is still more common than ed25519:

$ ssh-keygen -t rsa -b 4096

SSH to all remote servers

You need to ssh to the servers beforehand, since you need to have the ssh host keys in your known_hosts file for security reasons. Therefore, make sure you are connecting to the expected servers.

You could do this manually, but instead use Ansible to ssh to all the servers in your Ansible inventory file to populate the known_hosts file. Do not run this in a production environment and make sure you are connecting to the expected servers by verifying the ssh key fingerprints.

$ ansible all -m ping --extra-vars "ansible_ssh_common_args='-o StrictHostKeyChecking=no'" -i hosts

This command is using the Ansible ping module (-m ping) to ssh to all servers in your inventory file (-i hosts), and this will create entries in the ~/.ssh/known_hosts file.

Install Ansible required packages

Next, you need to install some packages that Ansible requires on the remote servers. Currently, Ansible uses Python 2 by default. So your remote servers will need to have Python 2 installed.

Use Ansible to install Python 2 on a remote Ubuntu server:

$ ansible ubuntu-dev -m raw -a "apt-get update && apt-get install -y python-minimal python-simplejson" -i hosts

If using CentOS 7 or RHEL 7 you don’t need to install Python 2 since it is already installed by default.

But, if using an old version like CentOS 5 or RHEL 5, you will need to run these commands. And if using SELinux with CentOS 5 or RHEL 5, run these commands to allow the Ansible copy/file/template modules.

Create Ansible user on remote servers

You can use my Ansible role on GitHub called create-user-ansible to have Ansible create its own user.

This new user will then use password-less sudo by default. But you can change this with a variable if you want to be prompted for a password every time you run Ansible.

Clone the git repository:

$ git clone https://github.com/ryandaniels/ansible-role-create-user-ansible.git ~/ansible/roles/create-user-ansible

Create the Ansible Playbook file:

$ cat > ~/ansible/create-user-ansible.yml << 'EOF'
---
- hosts: '{{inventory}}'
  become: yes
  roles:
  - create-user-ansible
EOF

Now the most important step, run the Ansible playbook.

This assumes you have a way to log in via ssh to your servers. (How else would you remotely connect?)
Change “remote_existing_user” to the username you already have on your servers.
Change “username” to your local Ansible username. This should be the same user as the local Ansible user since this makes things easier.

As a result, this will create a new user on all your servers named: ansible
More options are available, see the README on GitHub.

If your existing user uses a password for ssh authentication and does not use password-less sudo:

$ ansible-playbook create-user-ansible.yml --ask-pass --become --become-method=su --ask-become-pass --extra-vars "inventory=all ansible_ssh_user=remote_existing_user username=ansible" -i hosts

If your existing user uses password-less ssh using ssh keys and no password for sudo:

$ ansible-playbook create-user-ansible.yml --become --become-method=sudo --extra-vars "inventory=all ansible_ssh_user=remote_existing_user username=ansible" -i hosts

Conclusion

In conclusion, now you have an Ansible user created on all of your servers. You have installed and setup Ansible, and this is the first step! Next, write your own Playbook. Or use one of mine to update all of your Ubuntu or CentOS servers and automatically reboot them.

This guide is a quick way to get started with Ansible. Keep in mind, Ansible can be complex. You should plan out how you want to setup Ansible’s directory structure by following their best practices. And take some time to read their documentation!

The post Getting Started with Ansible on Ubuntu or CentOS appeared first on Ryan Daniels.

With all the latest news about the Spectre and Meltdown security vulnerabilities it is important to keep the software updated on all your devices. If you are a desktop user, update your OS now*. If you maintain Linux servers, update those now* as well.
The below information can help you use Ansible to update all of your Ubuntu, CentOS, and Redhat servers.

* Disclaimer: Test any updates thoroughly before upgrading your production environments since an update could introduce performance impacts or issues. I am not responsible if you break anything!

What is Ansible?

First of all, Ansible is amazing at IT automation. Ansible is a command line IT automation solution that can deploy configuration changes, software, and perform many other tasks all automatically. However, you should be extra cautious when running any type of automation for server updates. To get more information about what Ansible is, check out the Ansible documentation.

Ansible is modular, so you can create groups of tasks, called roles. And then you can group the roles together to all run sequentially inside of a playbook.
An Ansible playbook can run against one or multiple servers, depending how you reference the servers in your playbook, and also depending on how you group the servers in your Ansible inventory file.

What the Ansible server update role does

This is an Ansible role to update your servers with the latest packages, reboot the server if needed, and wait for the server to start up.
You can also exclude packages from the update, update only specified packages, or install only specified packages.

A reboot of the server is only be performed if the reboot flag is set (which is enabled by default), and if the kernel was updated or another package indicates to the OS that a reboot is needed.

Also, it seems obvious, but just in case.. Be sure to stop your applications before updating and rebooting your server, and then start your applications again after.
You can add more roles after this role to continue installing and configuring your server.

Tested on Ubuntu 16.04, Ubuntu 18.04, CentOS 7.x, and Redhat Enterprise Linux (RHEL) 7.x servers.

You can find this Ansible role on Github or Ansible Galaxy. Check back for any updates on Github, and also for more information about the variables used in this Ansible role.

Bonus – Spectre and Meltdown mitigation

As an extra bonus, this Ansible role can also help you mitigate the recent security vulnerabilities called Spectre and Meltdown (once updates are available for your operating system of course).

More information about how to mitigate Spectre and Meltdown (CVE-2017-5754, CVE-2017-5753, CVE-2017-5715) is available for Ubuntu and CentOS/Redhat. When new mitigations become available you will need to patch your servers again. And with Ansible it can be less painful.

To begin updating, continue reading. You can also jump directly to the example commands which can be used to mitigate Spectre and Meltdown if that is all you want to do. But, keep in mind it is usually better to keep all of the packages updated*.

Using Ansible to update Ubuntu and CentOS/Redhat 7

This assumes you have Ansible and git installed.
In addition, you also need to follow a few steps before running the playbook. If you have already done this, you can skip ahead to installing the role.

Prerequisites

Before you begin, make sure you have your Ansible “group_vars” setup for the Ansible hosts you are running this playbook against. “centos-dev” is the host group in the example below.

$ vi ./group_vars/centos-dev/proxy.yml

If you are behind a corporate firewall and use a proxy, add:

proxy_env:
  http_proxy: http://my.internal.proxy:80/
  https_proxy: https://my.internal.proxy:80/

If not using a proxy, add:

proxy_env: []

Install Ansible role to update the server

To install the Ansible role that handles updating your server(s), go into the directory you have your Ansible playbooks. Or, if you don’t have any existing playbooks, then create a new directory for the Ansible role and playbook.

Next, clone my Github repository with the Ansible role.

$ cd ~/ansible
$ git clone https://github.com/ryandaniels/ansible-role-server-update-reboot.git roles/server-update-reboot

Or, you can use Ansible Galaxy to install this role:

$ cd ~/ansible
$ ansible-galaxy install ryandaniels.server_update_reboot

Create your Ansible Playbook

Now, create your Ansible Playbook file for the server update role. You can also add other roles to run before or after.

$ vi server-update-reboot.yml

---
- hosts: '{{inventory}}'
max_fail_percentage: 0
serial: 1
become: yes
roles:
# - stop-applications
- server-update-reboot
# - server-config-xyz
#  - start-applications

Tip: The above will update one server at a time (using max_fail_percentage and serial). If you want to update everything at once you can comment those two lines out. Be careful with this since you could update and reboot all your servers at once!

Update your servers – Run the Playbook

Finally, the last step is to run the Ansible Playbook. Below are examples for various scenarios.

Note: It is important to understand what will happen. This will reboot your server by default!

Example for Ubuntu & Redhat/CentOS to update all packages

Use all defaults for the role to: update packages, reboot server if needed, and wait for the server to start up.

ansible-playbook server-update-reboot.yml --extra-vars "inventory=all-dev" -i hosts-dev

Below is the same as above, but now the server is not rebooted, even when a reboot is needed. In this example the extra variable “reboot_default” is used on the command line to change the reboot variable to false.

ansible-playbook server-update-reboot.yml --extra-vars "inventory=all-dev reboot_default=false" -i hosts-dev

Keep in mind that a server reboot may be necessary to complete the updates. For example, if a kernel update was applied.

Example for Redhat/CentOS to limit packages being updated

Update all packages except package(s) specified:

ansible-playbook server-update-reboot.yml --extra-vars 'inventory=centos-dev server_update_yum_exclude_pkgs="mysql*, bash, openssh*"' -i hosts-dev

Update (or install) only specific package(s):

ansible-playbook server-update-reboot.yml --extra-vars "inventory=centos-dev server_update_yum_install_pkgs='kernel-*, iwl*firmware, microcode_ctl, dracut'" -i hosts-dev

Example for Ubuntu to limit packages being updated

Update all packages except package(s) specified:

ansible-playbook server-update-reboot.yml --extra-vars 'inventory=ubuntu-dev server_update_apt_exclude_default=true' --extra-vars '{"server_update_apt_exclude_pkgs": [bash, openssl, ^mysql*, ^openssh*]}' -i hosts-dev

Update only specific package(s):

ansible-playbook server-update-reboot.yml --extra-vars "inventory=ubuntu-dev server_update_apt_default=update_specific" --extra-vars "{'server_update_apt_install_pkgs': [linux-firmware, linux-generic, linux-headers-generic, linux-image-generic, intel-microcode, openssh*]}" -i hosts-dev

Install only specific package(s):

ansible-playbook server-update-reboot.yml --extra-vars "inventory=ubuntu-dev server_update_apt_default=install" --extra-vars "{'server_update_apt_install_pkgs': [bash, openssh-server]}" -i hosts-dev

Be careful with wildcards since they can install more than you might want.

Examples for Spectre and Meltdown Mitigation

To update Ubuntu 16.04, Redhat 7, and CentOS 7 with only the available Spectre and Meltdown mitigations use the below examples.

Keep in mind it’s usually better to keep all of the packages up to date.

For Redhat/CentOS 7 (Spectre/Meltdown Mitigation)

ansible-playbook server-update-reboot.yml --extra-vars "inventory=centos-dev server_update_yum_install_pkgs='kernel-*, iwl*firmware, microcode_ctl, dracut'" -i hosts-dev

For Ubuntu 16.04 (Spectre/Meltdown Mitigation)

ansible-playbook server-update-reboot.yml --extra-vars "inventory=ubuntu-dev server_update_apt_default=update_specific" --extra-vars "{'server_update_apt_install_pkgs': [linux-firmware, linux-generic, linux-headers-generic, linux-image-generic, intel-microcode]}" -i hosts-dev

Conclusion

Automation using a tool like Ansible is very powerful. With Ansible’s help you can update all of your Ubuntu, CentOS, and Redhat servers quickly. Finally, with all of the recent issues surrounding Spectre and Meltdown, the Ansible server update role can help you keep everything updated and more secure.

Now that your servers are patched, you can look at other Ansible roles to help you install and configure software. If you have your own VPS, check out the Ansible role to setup OpenVPN with ad blocking.

The post Using Ansible to Update Ubuntu, CentOS, and Redhat appeared first on Ryan Daniels.