Posted on Fri 02 October 2015

Filtering items with results from stat module with Ansible

I want to use Ansible to replace an existing Bash script which allows me to rsync files (artifacts) to a remote host and also skip non-existent ones.

The script is similar to this one:

#!/bin/sh
src=$1
dest=$1

if [ -e "${src}" ]; then
  echo "Deploying '${src}' to '${dest}'..."
  rsync --archive --delete ${src} ${dest}
fi

The main challenge I encountered when trying to translate this script to Ansible is that the synchronize module does not allow you to skip non-existent sources, it fails. I tried several solutions before finding one that answers all my needs.

First, I settled on using the following vars to define the artifacts I wish to synchronize. Note the optional attribute which makes this particular artifact optional. This means that if the source (src) does not exist, it should be skipped without failing the play. By default, an artifact is not optional.

artifacts:
  - src: "./repository/"
    dest: "/target/repository"
  - src: "./manifests/"
    dest: "/target/manifests"
  - src: "./target/modules/"
    dest: "/target/modules"
  - src: "./hieradata/hieradata/"
    dest: "/target/hieradata"
    optional: true

Failed attempt #1

I tried to write my own synchronize module. It didn't go well. It just so happens that Ansible has a synchronize action plugin which does all sort of magic to make the synchronize module work like it does. Although I know Python, I felt this wasn't the path to go due to the added complexity to what was at first a simple Bash script.

Failed attempt #2

I tried to register a variable with results from the stat module for all artifacts and then use with_together to filter non-existent sources using a when statement. While this works, it makes the output unintelligible because both the artifact definition and stat module results are dumped at the console for each iteration. Here is the playbook so you have a better understanding:

---

- name: Deploy project
  hosts: all
  gather_facts: false
  vars:
    artifacts:
      - src: "./repository/"
        dest: "/target/repository"
      - src: "./manifests/"
        dest: "/target/manifests"
      - src: "./target/modules/"
        dest: "/target/modules"
      - src: "./hieradata/hieradata/"
        dest: "/target/hieradata"
        optional: true
  tasks:
    - name: Register artifacts existence
      stat:
        path: "{{ item.src }}"
      with_items: artifacts
      register: artifacts_stat
      delegate_to: 127.0.0.1

    - name: Synchronize artifacts
      synchronize:
        archive: true
        delete: true
        compress: true
        src: "{{ item.0.src }}"
        dest: "{{ item.0.dest }}"
      with_together:
        - artifacts
        - artifacts_stat.results
      when: >
        not item.0.optional|default(False) or
        item.1.stat.exists

Failed attempt #3

I tried to create a separated task file and include it in my playbook for each artifact using with_items. It didn't work as the combined use of include and with_items statements fails with this message:

include + with_items is a removed deprecated feature

Note that this feature will be restored in Ansible 2.0:

Dynamic include statements, which bring back the ability to use include + with_* loops. In 1.9.x and before, includes function simply as a pre-processor macro, in which tasks are expanded before any task execution starts. Now, in 2.0 and beyond, includes are executed as any other task is and expanded only at the point it is executed.

Here is the playbook:

---

- name: Deploy project
  hosts: all
  gather_facts: false
  vars:
    artifacts:
      - src: "./repository/"
        dest: "/target/repository"
      - src: "./manifests/"
        dest: "/target/manifests"
      - src: "./target/modules/"
        dest: "/target/modules"
      - src: "./hieradata/hieradata/"
        dest: "/target/hieradata"
        optional: true
  tasks:
    - include: synchronize-artifact.yaml
      vars:
        artifact_source: "{{ item.src }}"
        artifact_source: "{{ item.dest }}"
        artifact_optional: "{{ item.optional|default(False)|bool }}"
      with_items: artifacts
# synchronize-artifact.yaml
---

- name: Register artifact existence
  stat:
    path: "{{ artifact_source }}"
  register: artifact_stat
  delegate_to: 127.0.0.1

- name: Synchronize artifact
  synchronize:
    archive: true
    delete: true
    compress: true
    src: "{{ artifact_source }}"
    dest: "{{ artifact_destination }}"
  when: >
    not artifact_optional|default(False) or
    artifact_stat.results.stat.exists

Failed attempt #4

I then tried to manually include the task file as many times as needed instead of using with_items. While this worked, it's far from ideal as as you can't use variable in task name. This means it's near impossible to debug your play when all tasks have the exact same name for each deployed artifact.

The solution

Here is the solution I came up with. It uses the with_indexed_items statement which gives you access to the index (item.0) of the item (item.1). This means it can be used to access item from this other register variable containing stat results. It also means you will have a much cleaner console output containg the src, dest and optional values found in the artifact definition, making debugging much easier.

---

- name: Deploy project
  hosts: all
  gather_facts: false
  vars:
    artifacts:
      - src: "./repository/"
        dest: "/target/repository"
      - src: "./manifests/"
        dest: "/target/manifests"
      - src: "./target/modules/"
        dest: "/target/modules"
      - src: "./hieradata/hieradata/"
        dest: "/target/hieradata"
        optional: true
  tasks:
    - name: Register artifacts existence
      stat:
        path: "{{ item.src }}"
      with_items: artifacts
      register: artifacts_stat
      delegate_to: 127.0.0.1

    - name: Synchronize artifacts
      synchronize:
        archive: true
        delete: true
        compress: true
        src: "{{ item.1.src }}"
        dest: "{{ item.1.dest }}"
      with_indexed_items: artifacts
      when: >
        not item.1.optional|default(False) or
        artifacts_stat.results[item.0].stat.exists

© Mathieu Gagné. Built using Pelican. Theme by Giulio Fidente on github.