pylibssh takes surprisingly longer time than paramiko to receive the same data

[NilashishC]

SUMMARY

• Related to nxos_vlans results in command_timeout with libssh ansible-collections/cisco.nxos#145
• The above issue is caused because pylibssh takes ~300 times longer than paramiko to read the output of show vlan | json from the same target device. This resulted in the command_timeout value to work for paramiko but not libssh.
• This was realised by adding time.perf_counter() in relevant places within network_cli code as defined in this patch.
• Based on multiple passes, paramiko takes ~3 secs while pylibssh takes ~1000 secs to receive the same data.
• Data from one such pass is added here - https://gist.github.com/NilashishC/62cf8e945ac9a49e743fb2b553b8716d.
• On bumping the buffer size in read_non_blocking() to 8096 bytes and removing the upper cap of 1024 bytes, this delay was reduced to ~280 secs.

ISSUE TYPE

• Bug Report

PYLISSH and LIBSSH VERSION

$ pip show ansible-pylibssh
Name: ansible-pylibssh
Version: 0.2.0
Summary: Python bindings for libssh client specific to Ansible use case
Home-page: https://github.com/ansible/pylibssh
Author: Ansible, Inc.
Author-email: info+github/ansible/[email protected]
License: LGPLv2+
Location: /Users/nchakrab/venvs/dev/lib/python3.6/site-packages
Requires:
Required-by:

>>> import pylibsshext
>>> pylibsshext.__libssh_version__
'0.9.5'

OS / ENVIRONMENT

ProductName: Mac OS X
ProductVersion: 10.15.7

STEPS TO REPRODUCE

• Configure 500+ vlans on a Cisco Nexus appliance.
• Run the following playbook with first paramiko and then libssh.

---
- hosts: nxos
  gather_facts: no
  tasks:
    - cisco.nxos.nxos_vlans:
        config:
          - vlan_id: 10
            enabled: True
        state: merged

EXPECTED RESULTS

The playbook run doesn't take an exorbitantly high time to complete.

ACTUAL RESULTS

When command_timeout < 1100 it results in a timeout. With a higher command_timeout set, the playbook run completes as expected.

[webknjaz]

FTR I've got a similar report via email and it seems like ssh_channel.write() is what's blocking (this is the core of the reproducer).

[NilashishC]

In this particular case, strangely enough it seems that if the json output is pretty-printed, it works fine. For e.g., this works fine, but this triggers this issue, although both of them contain the exact same data.

[webknjaz]

It aligns with my observations of sending multiple lines of data that may even end up with a SEGFAULT. Not sure why though.

[SpiderD555]

I have the same issue, here is the workaround for anyone wondering how to deal with it quickly:
Add following variable to either job template or host in inventory and it will work
ansible_network_cli_ssh_type: paramiko