AWS Code Build fails with error message

[ckunki]

AWS CodeBuild Log

In setup of test_jupyter_with_ec2_based_on_new_ami

    @pytest.fixture(scope="session")
    def new_ec2_from_ami():

Line 97

change_password(host=ec2_instance_description.public_dns_name, user='ubuntu',
                                curr_pass=default_password, new_password=new_password)

Error message

/root/.cache/pypoetry/virtualenvs/exasol-ai-lab-X0J5PNhP-py3.10/lib/python3.10/site-packages/
paramiko/auth_handler.py:263: BadAuthenticationType

[ckunki]

Currently, the AI-Lab code uses lock_passwd from module users-and-groups, potentially, module set-passwords could also play a role.

AI Lab already contains Ansible task Enable SSH password authentication in file reset_password_tasks.yml, but the current assumption is that later this is disabled by cloud init, again.

Observation: Inside the Docker Container the file /etc/ssh/sshd_config doesn't exist anymore.

[ckunki]

Proposal:

- name: Enable SSH password authentication
  become: "{{need_sudo}}"
  lineinfile:
    dest: /etc/cloud/cloud.cfg
    regexp: '^(\s*)ssh_pwauth: false'
    line: '\1ssh_pwauth: true'
    state: present
    backrefs: yes

[ckunki]

Proposal for investigation:

• Start EC2 instance via ai-lab CLI and run Ansible
  • setup-ec2-and-install-dependencies
  • or install-dependencies on existing
• inspect (e.g. via temporary ssh key generated by ai-lab)
  • /etc/ssh/sshd_config
  • /etc/cloud/cloud.cfg
• export AMI, CLI export-vm
• launch EC2 from AMI
• inspect per SSH key, Note: This requires to put my ssh key to AWS, "Key pairs"

[ckunki]

It seems already Ansible installation reports an error:

/usr/bin/apt-get -y
  -o "Dpkg::Options::=--force-confdef"
  -o "Dpkg::Options::=--force-confold"
  install 'network-manager=1.36.4-2ubuntu1'
  -o APT::Install-Recommends=no'

failed: E: Unable to correct problems, you have held broken packages.
stdout:

Reading package lists...
Building dependency tree...
Reading state information...

Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: network-manager : Depends: libnm0 (= 1.36.4-2ubuntu1) but 1.36.6-0ubuntu2 is to be installed

[ckunki]

I will update network-manager as described above

[ckunki]

I could retrieve files /etc/cloud/cloud.cfg and /etc/ssh/sshd_config now.

/etc/cloud/cloud.cfg contains

system_info:
  default_user:
    name: ubuntu
    lock_passwd: True

/etc/ssh/sshd_config contains

#PasswordAuthentication yes

[ckunki]

Calling poetry run python exasol/ds/sandbox/main.py reset-password now displays

Unable to encrypt nor hash, passlib must be installed. No module named 'passlib'. Unable to encrypt nor hash, passlib must be installed. No module named 'passlib'"

See https://stackoverflow.com/questions/78525098/ansible-unable-to-encrypt-nor-hash-passlib-must-be-installed

[ckunki]

Added dependency to passlib: poetry add passlib, unfortunatly caused some downgrades of other dependencies:

  • Downgrading urllib3 (2.2.2 -> 2.2.1)
  • Downgrading botocore (1.34.137 -> 1.34.125)
  • Installing commonmark (0.9.1)
  • Downgrading fastjsonschema (2.20.0 -> 2.19.1)
  • Downgrading plux (1.11.0 -> 1.10.0)
  • Downgrading psutil (6.0.0 -> 5.9.8)
  • Downgrading pydantic-core (2.20.0 -> 2.18.4)
  • Downgrading rich (13.7.1 -> 12.6.0)
  • Downgrading s3transfer (0.10.2 -> 0.10.1)
  • Downgrading boto3 (1.34.137 -> 1.34.125)
  • Installing jsonpickle (3.2.1)
  • Installing pbr (6.0.0)
  • Downgrading pydantic (2.8.0 -> 2.7.4)
  • Downgrading setuptools (70.2.0 -> 70.0.0)
  • Downgrading ansible-core (2.17.1 -> 2.16.7)
  • Downgrading invoke (2.2.0 -> 1.7.3)
  • Installing jschema-to-python (1.2.3)
  • Installing junit-xml (1.9)
  • Downgrading networkx (3.3 -> 2.8.8)
  • Installing pathlib2 (2.3.7.post1)
  • Downgrading pytest (8.2.2 -> 7.4.4)
  • Installing sarif-om (1.0.4)
  • Downgrading ansible (10.1.0 -> 9.6.1)
  • Downgrading cfn-lint (1.4.2 -> 0.65.1)
  • Downgrading fabric (3.2.2 -> 2.7.1)
  • Downgrading importlib-metadata (7.2.1 -> 7.1.0)
  • Installing passlib (1.7.4)
  • Downgrading pytest-check-links (0.10.1 -> 0.9.3)
  • Downgrading tenacity (8.4.2 -> 8.3.0)

[ckunki]

I was now able to run poetry run python exasol/ds/sandbox/main.py reset-password without obvious errors.

After commenting out some parts of file reset_password_tasks.yml, I was still able to log in via ssh key file but not with password.

File contents:

$ grep -H lock /etc/cloud/cloud.cfg ; grep -H Passwo /etc/ssh/sshd_config
/etc/cloud/cloud.cfg:    lock_passwd: False
/etc/ssh/sshd_config:PasswordAuthentication yes

[ckunki]

Thanks to @tkilias we identified to additionally need sshd option KbdInteractiveAuthentication yes.
Additionally, we were able to remove dependency passlib and replace it by python standard module crypt.
This enables to revert the downgrades of the other packages mentioned above.

[ckunki]

Python test based on fabric / paramiko still failed as with

paramiko.ssh_exception.BadAuthenticationType:
Bad authentication type; allowed types: ['publickey', 'keyboard-interactive']

It could be possible to use keyboard-interactive authentication with fabric, see

• https://docs.fabfile.org/en/latest/api/connection.html#fabric.connection.Connection.open
• https://docs.paramiko.org/en/latest/api/transport.html#paramiko.transport.ServiceRequestingTransport
• https://docs.paramiko.org/en/latest/api/transport.html#paramiko.transport.ServiceRequestingTransport.auth_password
• transport_factory=paramiko.transport.ServiceRequestingTransport.__init__

On the other hand, I found the following at https://superuser.com/a/1828947, which turned out to be true in our case, too:

sneaky file /etc/ssh/sshd_config.d/60-cloudimg-settings.conf that had a single line PasswordAuthentication no

So I updated ansible task to modify all files in folder /etc/ssh/sshd_config.d, too,
using file globbing for remote files, as described here: https://stackoverflow.com/questions/33543551/

[ckunki]

I manually verified login with KbdInteractiveAuthentication no in /etc/ssh/sshd_config to be successful.