Data Munging with Ansible and Jinja2

From Building Network Automation Solutions

Building Network Automation Solutions
6 week advanced interactive online course Button-click-here.png
Course starting in
September 2017

At a Glance

  • High-intensity interactive online course;
  • Jump-start your network automation career;
  • Hands-on experience working on a solution to your own problem;
  • 6 week course spread across ~2 months;
  • Live discussion and guest speaker sessions;
  • Design and coding assignments and group work;
  • Final course completion certificate.

Ansible playbooks are a great task sequencing tool, but extraordinary clumsy when it comes to selecting data from suboptimal data structures or transforming one data structure into another. It’s often much better to transform data first and then use a more optimal data structure in an Ansible playbook.

Example

Imagine a data model that describes a network by listing the links between the nodes instead of the more traditional nodes/interfaces/addresses approach. A simple three-node network might be described with this data structure:

nodes:
  - name: E1
    mgmt: 172.16.1.110
  - name: E2
    mgmt: 172.16.1.111
  - name: PE1
    mgmt: 172.16.1.112

fabric:
  - {left: E1, left_ip: 10.0.0.21, left_port: GigabitEthernet0/2, 
     right: E2, right_ip: 10.0.0.22, right_port: GigabitEthernet0/2,
     cost: 5 }
  - {left: E1, left_ip: 10.0.0.13, left_port: GigabitEthernet0/1, 
     right: PE1, right_ip: 10.0.0.14, right_port: GigabitEthernet0/1,
     cost: 10 }
  - {left: E2, left_ip: 10.0.0.17, left_port: GigabitEthernet0/1, 
     right: PE1, right_ip: 10.0.0.18, right_port: GigabitEthernet0/2,
     cost: 1 }
The nodes structure is a mandatory part of the data model. While it’s po{ssible to extract the node names from the fabric data, the fabric elements (links) cannot be feasibly tied to node management IP addresses. Explicit listing of node names can also serve as a consistency checking mechanism: you can verify that links listed in fabric data contain valid node names and that each node participates in at least one link.

While it’s possible to use this data model to generate device configurations, it’s much easier to transform the data model into another one that is more suited to the automation tasks that have to be performed like configuring devices or checking OSPF neighbors.

David Barroso explains the benefits of the above data model in the Abstract Everything section of Network Automation Use Cases webinar.

A sample target data structure is shown below:

nodes:
  E1:
    internal: 
      GigabitEthernet0/2: { ip: 10.0.0.21 , cost: 5 }
      GigabitEthernet0/1: { ip: 10.0.0.13 , cost: 10 }

  E2:
    internal: 
      GigabitEthernet0/2: { ip: 10.0.0.22 , cost: 5 }
      GigabitEthernet0/1: { ip: 10.0.0.17 , cost: 1 }
  PE1:
    internal: 
      GigabitEthernet0/1: { ip: 10.0.0.14 , cost: 10 }
      GigabitEthernet0/2: { ip: 10.0.0.18 , cost: 1 }

Transforming Data with Jinja2

Jinja2 templating language is expressive enough to traverse complex data structures (which is extremely hard to do with the more limited looping mechanisms in Ansible playbooks). With somewhat careful formatting it’s possible to generate valid YAML files with Jinja2; JSON data is even easier to create as it’s not indent-sensitive.

Writing a Python module (Jinja2 filter or Ansible plugin) that transforms data is often easier than using a Jinja2 template.

The following Jinja2 template can be used to generate the optimized data model:

{% macro interface(name,ip,cost) %}
{{ name }}: { ip: {{ip}}, cost: {{cost}} }{% endmacro %}
---
nodes:
{% for node in nodes %}
  {{ node.name }}:
    mgmt: {{ node.mgmt }}
    internal: 
{%   for link in fabric %}
{%     if link.left == node.name %}
      {{ interface(link.left_port,link.left_ip,link.cost) }}
{%     elif link.right == node.name %}
      {{ interface(link.right_port,link.right_ip,link.cost }}
{%     endif %}
{%   endfor %}
{% endfor %}
For a more comprehensive example explore the Routing-Deployment example in the Ansible Examples Github repository.

Let’s analyze the individual parts of the template:

{% macro internal_link(name,ip,cost) %}
{{ name }}: { ip: {{ip}}, cost: {{cost}} }{% endmacro %}

The interface macro generates the description of a single interface. It will be called from a loop that generates YAML dictionary – proper indentation and line breaks are thus vital. The output of the macro is thus left-aligned and the endmacro directive is in the same line as the generated output to prevent an extra line break. The macro is also using JSON-in-YAML syntax to avoid further indentation problems.

---
nodes:
{% for node in nodes %}
  {{ node.name }}:
    mgmt: {{ node.mgmt }}

This part of the Jinja2 template starts the YAML document and creates the dictionary-within-dictionary data structure. nodes is a key in the outer dictionary; value of the nodes element is a dictionary with keys equal to node names. The inner dictionary data structure is generated by looping through the nodes data and generating YAML elements for every node.

Indentation is extremely important in a YAML document:

  • nodes key is not indented; it’s at the root level of the YAML document.
  • Indent of node.name is 2 spaces representing keys for inner dictionary under the nodes key.
  • Indent of keys of individual node values (example: mgmt key) is 4 spaces.
    internal: 
{%   for link in fabric %}
{%     if link.left == node.name %}
      {{ interface(link.left_port,link.left_ip,link.cost) }}
{%     elif link.right == node.name %}
      {{ interface(link.right_port,link.right_ip,link.cost }}
{%     endif %}
{%   endfor %}

This part of the template generates the interface data. It starts with another key within the node dictionary (indent: 4 spaces) and generates interface data by looping through the whole fabric data structure and generating an interface element every time the link.left or link.right node name equals the current node name.

Interface macro call is indented 6 spaces to generate a dictionary within the internal key.

Using Data Transformations in an Ansible Playbook

You can use data-transforming Jinja2 templates in various ways:

  • Standalone playbook that creates new YAML files (or even host variables) with template module;
  • Store transformed data model into a YAML file with the template module and read the new data model later in the Ansible playbook with the vars_files option or include_vars module, for example:
#
# Deploy OSPF routing in a WAN fabric
#
---
- name: Create and deploy OSPF configurations
  hosts: all
  vars:
    configs: "{{inventory_dir}}/configs"
  tasks:
  - include_vars: "{{ item }}"
    with_first_found:
      - nodes.yml
      - "{{ inventory_dir }}/nodes.yml"
The playbook uses with_first_found construct to find the data model either within the current directory or in the Ansible inventory directory.
  • Set an Ansible fact to the transformed data model with lookup filter, for example:
#
# Deploy OSPF routing in a WAN fabric
#
---
- name: Create and deploy OSPF configurations
  hosts: all
  tasks:
  - include_vars: "{{ item }}"
    with_first_found:
      - fabric.yml
      - "{{ inventory_dir }}/fabric.yml"
  - set_fact: data={{ lookup('template','../model/nodes.j2') }}

Next Steps

If this trick was all you were looking for I hope you’ll find it useful. If you’re looking for a bigger picture, start with Ansible for Networking Engineers webinar, explore other network automation webinars and register for the Building Network Automation Solutions online course.