Description

NAASC-MDS freezes due to being out of memory.

More RAM is installed taking it from 12 GB to 48 GB.

On a subsequent reboot, the MDS cannot mount the MDT

Who

NRAO - Jessica Otey / Mike Hatz, Wolfgang Baudler

Aeon - Jeff Johnson

What

MDT will not mount.

asimov kernel: LustreError:
2871:0:(obd_config.c:1591:class_config_llog_handler())
MGC10.7.17.8@o2ib: cfg command failed: rc = -22

And a detailed, verbose mount fails like this:

[root@asimov ~]# mount -vvvv /export/lustre/mdt
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "/dev/md127"
mount: node:  "/export/lustre/mdt"
mount: types: "lustre"
mount: opts:  "defaults,_netdev"
final mount options: '(null)'
mount: external mount: argv[0] = "/sbin/mount.lustre"
mount: external mount: argv[1] = "/dev/md127"
mount: external mount: argv[2] = "/export/lustre/mdt"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,_netdev"
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw,_netdev
arg[4] = /dev/md127
arg[5] = /export/lustre/mdt
source = /dev/md127 (/dev/md127), target = /export/lustre/mdt
options = rw,_netdev
checking for existing Lustre data: found
Reading CONFIGS/mountdata
mounting device /dev/md127 at /export/lustre/mdt, flags=0x1000000 options=osd=osd-ldiskfs,iopen_nopriv,user_xattr,errors=remount-ro,mgs,param=mdt.group_upcall=/usr/sbin/l_getgroups,svname=naaschpc-MDT0000,device=/dev/md127
warning: opening /sys/block/md127/md/stripe_cache_size: No such file or directory
mount.lustre: mount /dev/md127 at /export/lustre/mdt failed: Invalid argument retries left: 0
mount.lustre: mount /dev/md127 at /export/lustre/mdt failed: Invalid argument
This may have multiple causes.
Are the mount options correct?
Check the syslog for more info.

Where

NAASC Lustre --> NAASC-MDS

When

2016-11-22:09:00 US ET - 19:00 US ET

Why

Ultimately we think we had log corruption on the MDT as well as vestigial 1.8.9 options in our Lustre configurations. They were resolved by mimicing Wolfgang's Lustre in Green Bank with writeconfs.

On each OST:
tunefs.lustre --ost --writeconf <device>

On the MDS:
tunefs.lustre --mdt --writeconf --erase-params --mountfsoptions=errors=remount-ro,user_xattr /dev/md127 

This resolved the outage.

-- MikeHatz - 2016-11-22