I needed to deploy an OpenShift Origin instance for testing purposes. This article describes how I used openshift-ansible to deploy the software.

Existing tools

There are several solutions to do this:

These solutions work fine but provide a limited set of features by default.


I used a x86 physical server for the deployment:

  • 8 cores
  • 32G RAM
  • 2 x 1T disks

OpenShift and the ansible playbook only support RedHat-like distributions. I used a minimal CentOS 7.4 installation without SELinux, and without firewalld.

The machine DNS name is op1.pocentek.net.

Docker setup

The OpenShift playbook requires a working docker-engine installation on the target host. For better performance OpenShift recommends to use the overlay2 storage driver. This driver requires an XFS filesystem.

Docker installation steps:

# mkfs.xfs /dev/sdb1  # dedicated disk for docker in this setup
# mkdir /var/lib/docker
# echo '/dev/sdb1 /var/lib/docker xfs defaults 0 0' >> /etc/fstab
# mount -a
# yum install -y docker
# echo '{"storage-driver": "overlay2"}' > /etc/docker/daemon.json
# systemctl enable docker.service
# systemctl start docker.service
# docker ps  # make sure you can talk to the docker daemon

DNS setup

To benefit from OpenShift routing feature I defined a wildcard A entry in the pocentek.net DNS zone:

*.oc.pocentek.net. IN A

This allows dynamic resolution for all the application deployed on OpenShift, as long as they are routed using a matching domain name.

Playbook configuration

The OpenShift playbook requires only a few variables to be set to perform the installation. But a single node setup requires a few tweaks.

You first need to retrieve the code. I used the 3.6 version of OpenShift if this example:

$ git clone https://github.com/openshift/openshift-ansible.git
$ cd openshift-ansible
$ git checkout --track origin/release-3.6

All the settings are defined in an inventory file. I used the following inventory/hosts file:


op1.pocentek.net openshift_public_hostname="{{ inventory_hostname }}" openshift_hostname="{{ ansible_default_ipv4.address }}"


op1.pocentek.net openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_schedulable=true




openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_master_htpasswd_users={'gpocentek': 'some_htpasswd_encrypted_passwd'}



Some variables require a bit of explanation:

By default a master node will be configured to be ignored by the OpenShift scheduler. Application containers will not be created on masters. Since we only have one node, the master should be configured to host application containers.
openshift_router_selector and openshift_registry_selector

Routers (which expose services to the outside world) and the docker registry both run as containers on one or several nodes of the OpenShift cluster. By default they run on infrastructure nodes: dedicated nodes hosting internal services. To make sure that these services are properly scheduled and started on the single node deployment we explicitly label the node (region: primay) and configure the router and registry selector to match this node.

We also make sure that only 1 container is scheduled for each service (openshift_hosted_{router,registry}_replicas).

In this setup htpasswd authentication is used, and a gpocentek user is created by the playbook. You can generate the encrypted password using the htpasswd tool.

The node can be deployed using:

$ ansible-playbook -i inventory/hosts playbooks/byo/config.yml


Note: you can find sample inventories in inventory/byo/.


One feature I couldn't manage to deploy is persistent storage support. Since the deployment isn't meant for production, I used a NFS server deployed on the OpenShift machine to provide PVs:

for i in {1..9}; do
    mkdir -p /exports/volumes/vol0$i
    chown nfsnobody:nfsnobody /exports/volumes/vol0$i
    chmod 775 /exports/volumes/vol0$i
    echo "/exports/volumes/vol0$i *(rw,root_squash,no_wdelay)" >> /etc/exports

    cat | oc create -f - << EOF
    apiVersion: v1
    kind: PersistentVolume
      name: pv0$i
        storage: 5Gi
        - ReadWriteOnce
        path: /exports/volumes/vol0$i
      persistentVolumeReclaimPolicy: Recycle

Containers using PVCs created using these PVs most define a custom securityContext:

  supplementalGroups: [65534]

References: https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_nfs.html#nfs-supplemental-groups

Ansible playbooks often contain sensitive information that need to be kept private: passwords, private keys, DNS transfer keys and so on. It becomes a real problem when you have to share the playbooks and their sensitive data with coworkers in a git repository.

To solve this problem ansible provides the ansible-vault tool. It encrypts files using a password:

$ ansible-vault create group_vars/host
New Vault password:
Confirm New Vault password:
$ ansible-vault edit group_vars/host
Vault password:

What you commit in your git repository is something that looks like this (only longer):


You then need to use the --ask-vault-pass or --vault-password-file options to unlock the encrypted file when you run your playbook. Nothing complicated, but:

  • what happens if you don't manually run ansible, but instead use an orchestration tool like Jenkins or Ansible Tower?
  • how do you share and store the password with your coworkers in a secure manner?

What to do?

A solution is to use an external tool to store and retrieve the password, for instance pass or HashiCorp Vault.

To do this you need to use a script instead a file with the --vault-password-file option. You also need to tell ansible to always use this file:

  1. Write a script in a vault_pass file. This script should print the ansible-vault password on the standard output:

    # using pass
    pass pocentek.net/ansible/vault
    # or using vault
    vault read -field=password secret/pocentek.net/ansible_vault
  2. Make the script executable:

    $ chmod +x vault_pass
  3. Add the following in your ansible.cfg file:

    vault_password_file = ./vault_pass
  4. Run your playbook:

    ansible-playbook your-playbook.yml

Pass or Vault as external tool?

pass is really easy to setup and is my tool of choice for personal projects. When working with several persons it becomes more complicated to use:

  • every user must store the shared password at a predefined path on their local machine
  • if the password must be changed every user must update it locally

vault is more complex to setup but offers some nice advantages:

  • no need for everyone to store the password locally
  • vault supports ACLs. If a user leaves the project, her permissions are revoked and the password updated only once on the vault server
  • password changes are easier to handle and can be done more often

What we want to do

Ansible provides an lxc_container module to manage LXC containers on a remote host. It is very handy but once you've deployed a container you need to manage applications deployed inside this container. Also with Ansible obviously!

A simple approach could be to write a playbook to deploy the LXC containers, then generate a static inventory, and finally use this inventory with another playbook to deploy your final application.

An other approach is to have a single playbook. The first play will deploy the LXC containers and generate an in-memory inventory using the add_host module for each container. The lxc_container module returns the IP addresses for the container (once it's started).

If your containers are connected to an internal bridge on the remote host, you also need to configure your SSH client to help ansible access them.

An example of how it can be done

The example uses the following setup:

  • the LXC hosts are listed in the [lxc_hosts] group in the inventory

  • for each host a list of containers to manage is defined in the containers variable in a host_vars/{{inventory_hostname}} file, with content similar to this:

      - name: memcached-1
        service: memcache
      - name: mysql-1
        service: mysql
  • containers are connected to an lxcbr0 bridge, on a network

  • containers are deployed using a custom ubuntu-ansible template, based on the original ubuntu template. The template provides some extra configuration steps to ease ansible integration:

    • installation of python2.7
    • passwordless sudo configuration
    • injection of an SSH public key

    You can use the container_command argument of the lxc_container module instead if using a custom template.

Sample playbook

The first play creates the containers (if needed), and retrieves the dynamically assigned IP addresses of all managed containers:

- hosts: lxc_hosts
  become: true
  - name: Create the containers
      template: ubuntu-ansible
      name: "{{ item.name }}"
      state: started
    with_items: "{{ containers }}"
    register: containers_info

  - name: Wait for the network to be setup in the containers
    when: containers_info|changed
    pause: seconds=10

  - name: Get containers info now that IPs are available
      name: "{{ item.name }}"
    with_items: "{{ containers }}"
    register: containers_info

  - name: Register the hosts in the inventory
      name: "{{ item.lxc_container.ips.0 }}"
      group: "{{ item.item.service }}"
    with_items: "{{ containers_info.results }}"

The following plays use the newly added groups and hosts:

- hosts: memcache
  become: true
  - debug: msg="memcached deployment"

- hosts: mysql
  become: true
  - debug: msg="mysql deployment"

SSH client configuration

In the example setup Ansible can't reach the created containers because they are connected on an isolated network. This can be dealt with an ssh configuration in ~/.ssh/config:

Host lxc1
    Hostname lxc1.domain.com
    User localadmin

Host 10.0.100.*
    User ubuntu
    ForwardAgent yes
    ProxyCommand ssh -q lxc1 nc %h %p
    StrictHostKeyChecking no

Final word

Although this example deploys LXC containers, the same process can be used for any type of VM/container deployment: EC2, OpenStack, GCE, Azure or any other platform.

The problem

Ansible playbooks that can deploy applications in multiple contexts (for example a 1-node setup for tests, and multi-node setup with HA for production) might have to deal with rather complex variable definitions. The templating system provided by Ansible is a great help, but it can be difficult and very verbose to use it sometimes.

I recently had to solve a simple problem. Depending on the installation node of the HAProxy load balancer, the 15 balanced services had to listen on different ports to avoid conflicts.

One solution

Instead of computing the selected port using the template system, I chose to develop a module that would set the chosen ports as a fact on every target. The module is called once at the beginning of the playbook, and the ports are available as a variable in all the tasks.

The module

The module takes one mandatory boolean argument, with_haproxy.

The implementation looks like this (library/get_ports.py):

    'service1': 11001,
    'service2': 11002
    'service1': 1001,
    'service2': 1002

def main():
    module = AnsibleModule(
        argument_spec = dict(
            with_haproxy = dict(type='bool', required=True)

    with_haproxy = module.params['with_haproxy']
    ports = {'public': PUBLIC}
    ports['internal'] = INTERNAL if with_haproxy else PUBLIC

    module.exit_json(changed=False, result="success",
                     ansible_facts={'ports': ports})

    from ansible.module_utils.basic import *

The ansible_facts argument name is important, it will tell ansible to register this variable as a fact, so you don't need to use the register attribute in your task.

The ports dict holds the ports information for public and internal access. The ports.internal dict is used to configure the services ports. If HAProxy is used, they have custom (INTERNAL) ports to avoid conflicting with HAProxy. Otherwise they use the official (PUBLIC) port.

The playbook

To register the facts, the first task of the playbook looks like this:

- name: Register ports
    module: get_ports
    with_haproxy: true|false

- debug: var=ports.internal.service1
- debug: var=ports.internal.service2

The local_action module avoids a useless connection to the targets.


If you have not written modules for Ansible yet, have a look at the tutorial. They can be written in any language, although using Python makes things a lot easier.