AWX: Unable to SSH as root into target host only when using a specific Ansible role

In our AWX environment we have an Ansible role "Nessus Applicaiton" for RHEL 7/RHEL 8. It deploys the Nessus agent application just fine for new RHEL 7 servers, but not for RHEL 8.5 servers as it seems it cannot connect via SSH port 22.

We have several other templates that allows AWX to connect as root to the RHEL 8 target hosts, and deploy other roles and templates without issues.

The root public key is baked into the authorized_keys of the RHEL 7/RHEL 8 images.

• firewalld is not enabled. No firewall rules are blocking AWX from target host.
• I tested SSH from the AWX server to the target host with root username and password as well as ssh keys without issue to target host. Each connection method works. I have tried removing the public key from the authorized_keys on target host and re-adding.
• I have attempted to disable GSS authentication on the AWX host and target host, restarted sshd, and reran AWX Nessus Applicaiton role deployment.
• In ansible.cfg in Git repo tried changing ssh_args to IdentitiesOnly=no
• Tried adding [ssh_connection] ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=no

fatal: [devops11422-n]: UNREACHABLE! => { "changed": false, "msg": "Failed to connect to the host via ssh: OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS 28 May 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for 'final all' host 204.67.211.123 originally 204.67.211.123\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2… task path: /tmp/awx_10051_46nhtsu2/project/tenable-nessus-agent-test.yml:4 <204.67.211.106> ESTABLISH SSH CONNECTION FOR USER: root <204.67.211.106> SSH: EXEC sshpass -d10 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_10051_46nhtsu2/cp/860fdfa239 204.67.211.106 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"'' <204.67.211.106> (255, b'', b'OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS 28 May 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for \'final all\' host 204.67.211.106 originally 204.67.211.106\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched \'final\'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 204.67.211.106 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf\r\ndebug2: checking match for \'final all\' host 204.67.211.106 originally 204.67.211.106\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched \'final\'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]\r\ndebug3: kex names ok: [curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/tmp/awx_10051_46nhtsu2/cp/860fdfa239" does not exist\r\ndebug2: ssh_connect_direct\r\ndebug1: Connecting to 204.67.211.106 [204.67.211.106] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9990 ms remain after connect\r\ndebug1: identity file /root/.ssh/id_rsa type -1\r\ndebug1: identity file /root/.ssh/id_rsa-cert type -1\r\ndebug1: identity file /root/.ssh/id_dsa type -1\r\ndebug1: identity file /root/.ssh/id_dsa-cert type -1\r\ndebug1: identity file /root/.ssh/id_ecdsa type -1\r\ndebug1: identity file /root/.ssh/id_ecdsa-cert type -1\r\ndebug1: identity file /root/.ssh/id_ed25519 type -1\r\ndebug1: identity file /root/.ssh/id_ed25519-cert type -1\r\ndebug1: identity file /root/.ssh/id_xmss type -1\r\ndebug1: identity file /root/.ssh/id_xmss-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_8.0\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_8.0\r\ndebug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 204.67.211.106:22 as \'root\'\r\ndebug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:120\r\ndebug3: load_hostkeys: loaded 1 keys from 204.67.211.106\r\ndebug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521\r\ndebug3: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-31\r\ndebug1: Server host key: ssh-ed25519 SHA256:0K2ICbjw/u+cp4lDsJ9BUbNelTV+G+WjHjDilpLspAY\r\ndebug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:120\r\ndebug3: load_hostkeys: loaded 1 keys from 204.67.211.106\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host key has just been changed.\r\nThe fingerprint for the ED25519 key sent by the remote host is\nSHA256:0K2ICbjw/u+cp4lDsJ9BUbNelTV+G+WjHjDilpLspAY.\r\nPlease contact your system administrator.\r\nAdd correct host key in /root/.ssh/known_hosts to get rid of this message.\r\nOffending ECDSA key in /root/.ssh/known_hosts:120\r\nPassword authentication is disabled to avoid man-in-the-middle attacks.\r\nKeyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.\r\ndebug3: send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey out after 4294967296 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug3: receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1: rekey in after 4294967296 blocks\r\ndebug1: Will attempt key: /root/.ssh/id_rsa \r\ndebug1: Will attempt key: /root/.ssh/id_dsa \r\ndebug1: Will attempt key: /root/.ssh/id_ecdsa \r\ndebug1: Will attempt key: /root/.ssh/id_ed25519 \r\ndebug1: Will attempt key: /root/.ssh/id_xmss \r\ndebug2: pubkey_prepare: done\r\ndebug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>\r\ndebug3: receive packet: type 6\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r\ndebug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password\r\ndebug3: preferred gssapi-with-mic,publickey\r\ndebug3: authmethod_lookup gssapi-with-mic\r\ndebug3: remaining preferred: publickey\r\ndebug3: authmethod_is_enabled gssapi-with-mic\r\ndebug1: Next authentication method: gssapi-with-mic\r\ndebug1: Unspecified GSS failure. Minor code may provide more information\nNo Kerberos credentials available: Invalid UID in persistent keyring name\n\n\r\ndebug1: Unspecified GSS failure. Minor code may provide more information\nNo Kerberos credentials available: Invalid UID in persistent keyring name\n\n\r\ndebug2: we did not send a packet, disable method\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: \r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Trying private key: /root/.ssh/id_rsa\r\ndebug3: no such identity: /root/.ssh/id_rsa: No such file or directory\r\ndebug1: Trying private key: /root/.ssh/id_dsa\r\ndebug3: no such identity: /root/.ssh/id_dsa: No such file or directory\r\ndebug1: Trying private key: /root/.ssh/id_ecdsa\r\ndebug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory\r\ndebug1: Trying private key: /root/.ssh/id_ed25519\r\ndebug3: no such identity: /root/.ssh/id_ed25519: No such file or directory\r\ndebug1: Trying private key: /root/.ssh/id_xmss\r\ndebug3: no such identity: /root/.ssh/id_xmss: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n')

Can anyone point me in the right direction from here?

ในกรณีที่มีใครพบเจอสิ่งนี้...ฉันได้ตรวจสอบไฟล์known_hostsแล้ว แต่ไม่ได้มองหาที่ที่เหมาะสม

ไฟล์known_hosts ที่เป็นปัญหาอยู่ภายในคอนเทนเนอร์ AWX_TASK /root/.ssh/known_hosts บรรทัดที่ 120 มีคีย์ ed25519 ที่ขัดแย้งกันอยู่ที่นั่น ฉันลบมันออก นำบทบาทไปใช้ใหม่และปรับใช้บทบาทสำเร็จแล้ว เนื่องจากเราใช้ IP ซ้ำ เราคาดว่าเมื่อเวลาผ่านไปจะพบปัญหานี้อีกครั้ง เราประหลาดใจที่เราไม่ได้ก่อนหน้านี้ เราอาจจะสร้างสคริปต์เพื่อกำจัด .ssh/known_hosts ในอนาคต