Description

Phone call from Maxs stating they were having problems with JAOPOST020 not being able to run CASA when connected to sciops lustre.

Who

JAO - Maxs Simmonds

NRAO - Jessica Otey, Mike Hatz

What

jaopost020.sco.alma.cl was not maintaning connectivity to sciops lustre

Where

JAO - sciops lustre

JAOPOST020 - 192.168.0.32
LST-OSS-01 - 192.168.0.213
LST-MDS(?) - 192.168.0.212

When

2016-11-02

Why

Please see attached logs too.

One of the suggested changes was due to a difference we spotted between our /etc/modrpobe.d/nrao-lustre.conf files:

ON JAOPOST020:

bash-4.1$ ibping 192.168.0.212
ibwarn: [35854] mad_rpc_open_port: can't open UMAD port ((null):0)
ibping: iberror: failed: Failed to open '(null)' port '0'
-bash-4.1$ ibping 192.168.0.213
ibwarn: [35866] mad_rpc_open_port: can't open UMAD port ((null):0)
ibping: iberror: failed: Failed to open '(null)' port '0'

AND

-bash-4.1$ ibstat | grep "Port GUID"
                Port GUID: 0xf452140300ca72f7
-bash-4.1$ ibping -S
ibwarn: [35947] mad_rpc_open_port: can't open UMAD port ((null):0)
ibping: iberror: failed: Failed to open '(null)' port '0'


We have an alias parameter in modprobe.d/lustre.conf that you are missing, and it corresponds to that UMAD error.

Here's a sample config (from our LARD):

alias ib0 ib_ipoib
alias ib0 ib_umad
alias net-pf-27 ib_sdp
options lnet networks="o2ib0(ib0)"

Here's yours on JAOPOST020:

alias ib0 mlx4_ib
alias ib0 ib_ipoib
options lnet networks="o2ib0(ib0)"
#routes="tcp 10.7.17.[11-12]@o2ib0; tcp2 10.7.17.[11-12]@o2ib0" live_router_check_interval=60 dead_router_check_interval=60

You are missing the alias ib0 ib_umad line. That *may* be making the difference!

I would:

unmount lustre on jaopost020
unload the lustre modules (lustre_rmmod, I believe)
make the change in that file
just for the heck of it, bounce the interface (ifdown ib0; ifup ib0)
reload the modules (modprobe lustre)
remount lustre

THEN, you should test the IB connection between jaopost020 and the MDT or OSS (or both). 

Unfortunately, this did not resolve the issue.

Additional information from Maxs:

Jessica, got ibping working.

Works in client/server mode.

Server (lst-oss-01.sco.alma.cl):

[root@lst-oss-01 ~]# ibping -S -C mlx4_0 -P 0

Client (jaopost020.sco.alma.cl):

[root@jaopost020 ~]# ibping -c 10000 -f -C mlx4_0 -P 0 -L 11

--- lst-oss-01.(none) (Lid 11) ibping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 838 ms
rtt min/avg/max = 0.028/0.083/0.245 ms

Use "--debug" to see internals on each command.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Testing_Early_InfiniBand_RDMA_operation.html

-- MikeHatz - 2016-11-03