Breaking ipaddr or ipmath filter changes with version 2.6

[ClausHolbechArista]

SUMMARY

With the release of ansible.netcommon v2.6 our roles using the ansible.netcommon.ipaddr combined with ansible.netcommon.ipmath filters fail.

It is not clear if it is the output of ansible.netcommon.ipaddr that changed, or the input validation for ansible.netcommon.ipmath.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.netcommon.ipmath

ANSIBLE VERSION

ansible [core 2.11.3] 
  config file = /home/holbech/ipaddr/ansible.cfg
  configured module search path = ['/home/holbech/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/holbech/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/holbech/ansible-avd:/home/holbech/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/holbech/.local/bin/ansible
  python version = 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

# /home/holbech/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
ansible.netcommon 2.6.0  
ansible.posix     1.3.0  
ansible.utils     2.5.1  
arista.avd        2.2.1  
arista.cvp        3.2.0  
arista.eos        4.1.1  
netbox.netbox     3.1.1  

# /home/holbech/ansible-avd/ansible_collections
Collection Version
---------- -------
arista.avd 3.3.2  

CONFIGURATION

COLLECTIONS_PATHS(/home/holbech/ipaddr/ansible.cfg) = ['/home/holbech/ansible-avd', '/home/holbech/.ansible/collections', '/usr/share/ansible/collections']
DEFAULT_JINJA2_EXTENSIONS(/home/holbech/ipaddr/ansible.cfg) = jinja2.ext.loopcontrols,jinja2.ext.do,jinja2.ext.i18n

OS / ENVIRONMENT

N/A

STEPS TO REPRODUCE

Simple playbook to only

---

- name: Test
  hosts: localhost
  gather_facts: false
  connection: local
  tasks:
    - debug:
        msg: "{{ '192.168.255.0' | ansible.netcommon.ipaddr('network') | ansible.netcommon.ipmath(123) }}"

EXPECTED RESULTS

Testing with ansible.netcommon v.2.5.1:

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Test] *****************************************************************************************************************

TASK [debug] ****************************************************************************************************************
ok: [localhost] => {
    "msg": "192.168.255.123"
}

PLAY RECAP ******************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

ACTUAL RESULTS

Fatal error during play

ansible-playbook [core 2.11.3] 
  config file = /home/holbech/ipaddr/ansible.cfg
  configured module search path = ['/home/holbech/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/holbech/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/holbech/ansible-avd:/home/holbech/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/holbech/.local/bin/ansible-playbook
  python version = 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0]
  jinja version = 3.0.3
  libyaml = True
Using /home/holbech/ipaddr/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
Loading callback plugin default of type stdout, v2.0 from /home/holbech/.local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: test.yml **********************************************************************************************************
Positional arguments: test.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in test.yml

PLAY [Test] *****************************************************************************************************************
META: ran handlers

TASK [debug] ****************************************************************************************************************
task path: /home/holbech/ipaddr/test.yml:8
Loading collection ansible.netcommon from /home/holbech/.ansible/collections/ansible_collections/ansible/netcommon
[DEPRECATION WARNING]: Use 'ansible.utils.ipmath' module instead. This feature will be removed from ansible.netcommon in a 
release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
redirecting filter ansible.netcommon.ipmath to ansible.utils.ipmath
Loading collection ansible.utils from /home/holbech/.ansible/collections/ansible_collections/ansible/utils
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature will be removed from ansible.netcommon in a 
release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
redirecting filter ansible.netcommon.ipaddr to ansible.utils.ipaddr
redirecting filter ansible.netcommon.ipaddr to ansible.utils.ipaddr
redirecting filter ansible.netcommon.ipmath to ansible.utils.ipmath
redirecting filter ansible.netcommon.ipmath to ansible.utils.ipmath
redirecting filter ansible.netcommon.ipaddr to ansible.utils.ipaddr
redirecting filter ansible.netcommon.ipmath to ansible.utils.ipmath
fatal: [localhost]: FAILED! => {
    "msg": "You must pass a valid IP address; ['192.168.255.0'] is invalid"
}

PLAY RECAP ******************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[jantari]

We just ran into the exact same issue.

Reposting my commrnt from #359 (comment) here where it fits better:

---

Also wanted to chime in and say this is a breaking change (and affected us), because the old ansible.netcommon.ipaddr("network/prefix") would return a single string but the new ansible.utils.ipaddr("network/prefix") returns a list containing a string.

Easy to verify by running the following task with netcommon 2.5.1 and the latest utils collection:

- name: DEBUG NEW IPADDR FILTER
  debug:
    msg:
      - 'SUBNET VAR           : {{ item.subnet }}'
      - 'OLD NETCOMMON FILTER : {{ item.subnet | ansible.netcommon.ipaddr("network/prefix") }}'
      - 'NEW UTILS FILTER     : {{ item.subnet | ansible.utils.ipaddr("network/prefix") }}'
  loop:
    - subnet: 1.1.1.1/32
    - subnet: 10.10.10.0/24
    - subnet: 192.168.0.0/23

Output looks like this:

TASK [DEBUG IPADDR FILTER] *******************************************************
ok: [localhost] => (item={'subnet': '1.1.1.1/32'}) =>
  msg:
  - 'SUBNET VAR           : 1.1.1.1/32'
  - 'OLD NETCOMMON FILTER : 1.1.1.1/32'
  - 'NEW UTILS FILTER     : [''1.1.1.1/32'']'
ok: [localhost] => (item={'subnet': '10.10.10.0/24'}) =>
  msg:
  - 'SUBNET VAR           : 10.10.10.0/24'
  - 'OLD NETCOMMON FILTER : 10.10.10.0/24'
  - 'NEW UTILS FILTER     : [''10.10.10.0/24'']'
ok: [localhost] => (item={'subnet': '192.168.0.0/23'}) =>
  msg:
  - 'SUBNET VAR           : 192.168.0.0/23'
  - 'OLD NETCOMMON FILTER : 192.168.0.0/23'
  - 'NEW UTILS FILTER     : [''192.168.0.0/23'']'

---

I have pinned ansible.netcommon to 2.5.1 for now and it works around the issue.

But we would need clarification on whether this is an unintended bug ( I would think so because the PR doesn't read like it was perceived as a breaking change ) or whether it is intentionally breaking and we have to update all our ansible code?

[usrme]

This also has downstream effects on other collections, such as community.zabbix that use the ipaddr filter: https://github.com/ansible-collections/community.zabbix/search?q=%7C+ipaddr.

[ashwini-mhatre]

@jantari @ClausHolbechArista @usrme we recently moved ipaddr filters from netcommon to utils. its unintentional bug in utils collection. I am working on the fix and that we will release utils in end of march or 1st week of April. let me know if you need any information or anything from my side. For now you can use netcommon 2.5.1 as workaround.

[bluikko]

What did I just read?
2 major breakages were introduced on very commonly used filters. We are expected to go through all the code and fix all non-FQCN ipaddr filters and the cases where the unexpected lists are returned, and then 1 month later go back and undo all those changes that now don't return a list again?

Revert!

[ashwini-mhatre]

Opening this bug again as we have not released netcommon #378

[ashwini-mhatre]

We have released utils 2.5.2 and netcommon 2.6.1 which fixed #375 and ansible-collections/ansible.utils#148 issues.

For FQDN issue ansible-core team already merged ansible/ansible#77210. This fix will be available in 28th march release.

which means,
For non fqdn ipaddr filters users can use old netcommon collection version 2.5.1 as workaround until core-team will release ansible code(28th march). For fqdn ipaddr filters users can use latest versions of any of netcommon(2.6.1) or utils(2.5.2) cc: @trishnaguha

[blakedot]

Do you folks honestly expect everyone to rewrite the thousands of existing playbooks that use the ipaddr() filter?
Do you folks ever test your code on production configurations?
Do you folks realize you're forcing everyone with existing playbooks to pin utils to 2.5, so your development on new versions of that collection is essentially wasted effort?