Infiniband fabric with 3 nodes - newbie

I am trying to connect 3 HP z840 workstations using:

Mellanox ConnectX-3 VPI 40 / 56GbE Dual-Port QSFP Adapter MCX354A-FCBT
Mellanox SX6005 12-port Non-blocking Unmanaged 56Gb/s

Description of machines to be connected: oak-rd0-linux (main node where I will run things from and where opensm is running) oak-rd1-linux oak-rd2-linux

I have installed the latest fw on the cards and installed the latest mlnx ofed driver that supports my cards (MLNX_OFED_LINUX-4.9-4.1.7.0-ubuntu20.04-x86_64). Running ubuntu 20.04 (Linux 5.4.0-26-generic kernel as required by the mlnx_ofed driver).

how I installed the MLNX OFED:

sudo touch /etc/apt/sources.list.d/mlnx_ofed.list
sudo nano /etc/apt/sources.list.d/mlnx_ofed.list
deb file:/home/user/infiniband/MLNX_OFED_LINUX-4.9-4.1.7.0-ubuntu20.04-x86_64/DEBS/UPSTREAM_LIBS ./
wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -
apt-key list
sudo apt-get update
sudo apt-get install mlnx-ofed-all

I also got the hpcx-v2.6.0-gcc-MLNX_OFED_LINUX-4.7-1.0.0.1

I start opensm in deamon mode with:

/etc/init.d/opensmd start

I run sudo ibdiagnet and it yields a clean summary (NOTE: I cannot run ibdiagnet without sudo )

Running: ibdiagnet -r
----------
Load Plugins from:
/usr/share/ibdiagnet2.1.1/plugins/
(You can specify more paths to be looked in with "IBDIAGNET_PLUGINS_PATH" env variable)

Plugin Name Result Comment
libibdiagnet_cable_diag_plugin-2.1.1 Succeeded Plugin loaded
libibdiagnet_phy_diag_plugin-2.1.1 Succeeded Plugin loaded

---------------------------------------------
Discovery
-I- Discovering ... 4 nodes (1 Switches & 3 CA-s) discovered.
-I- Fabric Discover finished successfully

-I- Discovered 4 nodes (1 Switches & 3 CA-s).

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- VS Capability GMP finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- VS Capability SMP finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- VS ExtendedPortInfo finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Port Info Extended finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Switch Info retrieving finished successfully

-I- Duplicated GUIDs detection finished successfully

-I- Duplicated Node Description detection finished successfully

---------------------------------------------
Lids Check
-I- Lids Check finished successfully

---------------------------------------------
Links Check
-I- Links Check finished successfully

---------------------------------------------
Subnet Manager
-I- SM Info retrieving finished successfully

-I- Subnet Manager Check finished successfully

---------------------------------------------
Port Counters
-I- Retrieving PMClassPortInfo ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Retrieving PMPortSampleControl ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Ports counters retrieving finished successfully

-I- Going to sleep for 1 seconds until next counters sample
-I- Time left to sleep ... 1 seconds.

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Ports counters retrieving (second time) finished successfully

-I- Ports counters value Check finished successfully

-I- Ports counters Difference Check (during run) finished successfully

---------------------------------------------
Nodes Information
-I- Devid: 4099(0x1003), PSID: MT_1090120019, Latest FW Version:2.42.5000
-I- Devid: 51000(0xc738), PSID: EMC1260110021, Latest FW Version:9.3.8000
-I- FW Check finished successfully

---------------------------------------------
Speed / Width checks
-I- Link Speed Check (Compare to supported link speed)
-I- Links Speed Check finished successfully

-I- Link Width Check (Compare to supported link width)
-I- Links Width Check finished successfully

---------------------------------------------
Alias GUIDs
-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Alias GUIDs retrieving finished successfully

-I- Alias GUIDs finished successfully

---------------------------------------------
Virtualization
-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Virtualization finished successfully

-I- Virtual ports retrieving finished successfully

-I- Virtual ports retrieving finished successfully

---------------------------------------------
Partition Keys
-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Partition Keys retrieving finished successfully

-I- Partition Keys finished successfully

---------------------------------------------
Temperature Sensing
-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Temperature Sensing finished successfully

---------------------------------------------
Routing

-I- EXT switch info retrieving finished successfully

-I- PLFT is enabled on 0 switches.
-I- PLFT data retrieving finished successfully

-I- Adaptive Routing is enabled on 0 switches.
-I- AR data retrieving finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Unicast FDBS Info retrieving finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Multicast FDBS Info retrieving finished successfully

-I- Retrieving ... 4/4 nodes (1/1 Switches & 3/3 CA-s) retrieved.
-I- Dump SLVL Table finished successfully

-I- Load SLVL file.
---------------------------------------------
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 0 0
-I- Port Counters 0 0
-I- Nodes Information 0 0
-I- Speed / Width checks 0 0
-I- Alias GUIDs 0 0
-I- Virtualization 0 0
-I- Partition Keys 0 0
-I- Temperature Sensing 0 0
-I- Routing 0 0

-I- You can find detailed errors/warnings in: /var/tmp/ibdiagnet2/ibdiagnet2.log


-I- ibdiagnet database file : /var/tmp/ibdiagnet2/ibdiagnet2.db_csv
-I- LST file : /var/tmp/ibdiagnet2/ibdiagnet2.lst
-I- Network dump file : /var/tmp/ibdiagnet2/ibdiagnet2.net_dump
-I- Subnet Manager file : /var/tmp/ibdiagnet2/ibdiagnet2.sm
-I- Ports Counters file : /var/tmp/ibdiagnet2/ibdiagnet2.pm
-I- Nodes Information file : /var/tmp/ibdiagnet2/ibdiagnet2.nodes_info
-I- Alias guids file : /var/tmp/ibdiagnet2/ibdiagnet2.aguid
-I- VPorts file : /var/tmp/ibdiagnet2/ibdiagnet2.vports
-I- VPorts Pkey file : /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey
-I- Partition keys file : /var/tmp/ibdiagnet2/ibdiagnet2.pkey
-I- VL2VL file : /var/tmp/ibdiagnet2/ibdiagnet2.vl2vl
-I- PLFT file : /var/tmp/ibdiagnet2/ibdiagnet2.plft
-I- AR file : /var/tmp/ibdiagnet2/ibdiagnet2.ar
-I- Full AR file : /var/tmp/ibdiagnet2/ibdiagnet2.far
-I- Unicast FDBS file : /var/tmp/ibdiagnet2/ibdiagnet2.fdbs
-I- Multicast FDBS file : /var/tmp/ibdiagnet2/ibdiagnet2.mcfdbs
-I- SLVL Table file : /var/tmp/ibdiagnet2/ibdiagnet2.slvl

ibping seems to be running fine although I am not sure if these are good performance values

ibstat | egrep "Port|Base"

(base) baird@oak-rd0-linux:~$ ibstat | egrep "Port|Base"
Port 1:
Base lid: 0
Port GUID: 0x0010e00001885689
Port 2:
Base lid: 1
Port GUID: 0x0010e0000188568a

server ( oak-rd0-linux )
ibping -S -P 2 -d (I know that Port2 is the active one)

I can then ibping from host1 and host2 with:
ibping -P 1 1

Host1 ( oak-rd1-linux )
baird@oak-rd1-linux:~$ sudo ibping -P 1 1
Pong from oak-rd0-linux.(none) (Lid 1): time 0.027 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.037 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.042 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.044 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.038 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.029 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.042 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.029 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.038 ms
^C
--- oak-rd0-linux.(none) (Lid 1) ibping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8028 ms
rtt min/avg/max = 0.027/0.036/0.044 ms

Host2 ( oak-rd2-linux )
(base) baird@oak-rd2-linux:~$ sudo ibping -P 1 1
Pong from oak-rd0-linux.(none) (Lid 1): time 0.029 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.015 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.041 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.043 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.044 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.037 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.042 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.039 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.038 ms
Pong from oak-rd0-linux.(none) (Lid 1): time 0.040 ms
^C
--- oak-rd0-linux.(none) (Lid 1) ibping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9055 ms
rtt min/avg/max = 0.015/0.036/0.044 ms

seems to be working fine assuming that all is good on the infiniband side, here's my problem:

I can run the tests in the ompi that comes with hpcx

mpicc $HPCX_MPI_TESTS_DIR/examples/hello_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_c
mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c

Hello, world, I am 1 of 2, (Open MPI v4.0.3rc4, package: Open MPI root@0e5a40994726 Distribution, ident: 4.0.3rc4, repo rev: v4.0.3rc4-6-g8b4a8cd34c, Unreleased developer copy, 148)
Hello, world, I am 0 of 2, (Open MPI v4.0.3rc4, package: Open MPI root@0e5a40994726 Distribution, ident: 4.0.3rc4, repo rev: v4.0.3rc4-6-g8b4a8cd34c, Unreleased developer copy, 148)

However, when I try to run with:

mpirun -x LD_LIBRARY_PATH -np 2 -H oak-rd0-linux,oak-rd1-linux $HPCX_MPI_TESTS_DIR/examples/hello_c

I don't get any feedback, no errors no output, it seems to be hanging.

-Can someone please guide me on how to connect/use my other hosts' CPU? -What are the utils I need to use to debug the issue?

I am a complete newbie in this and I would greatly appreciate any help/suggestion etc. I am ready to provide any additional information, test suggestions out etc. Cheers!