Changes

• Migrated to scripted form of patching
• New version is 6.7.1.1 on all systems
• Migrated Lustre servers from Centos 5.7 to RHEL 6.7
• Migrated Lustre servers from Lustre 1.8.9 to Lustre 2.4.3

Problems

• Problems with force_over_24tb flag on OSTs
  • FIXED with tunefs.lustre and adding 'mountfsoptions='errors=remount-ro,extents,mballoc' to re-add original default options and remove the force_over_24tb flag.
• Node-16 is down, it had problems prior to the upgrade (unreported). Powered off for now
• The Mark5s changed their interface mapping. Eth0 became eth2, eth2 became eth0.
  • This actually fixed the long standing Lustre mounting problem
  • It broke Mark5fx01 mounting of san01 since eth3 (old 10.1.33.0 interface) became eth0. FIXED by tweaking /opt/bin/netconfig.sh
• Node-01 doesn't start difx and calcserver properly on boot, seems to be a race condition
• Mar. 24, 2016 krowe: Control-C still doesn't work. I'm surprised how often I need it.
• Mar. 24, 2016 krowe: usno-serv-2 still doesn't rsync it's data partition from usno-serv-1

Resolved

• Mar. 24, 2016 krowe: (V-38643) Turn off virt-who on servers because it makes a world writable file /var/run/virt-who.pid

  chkconfig virt-who off ; /etc/init.d/virt-who stop

• Mar. 24, 2016 krowe: (V-38643) chmod files on data partition so that they are not world writable

  chmod 755 /opt/services/ganglia-web/ganglia/dwoo/compiled/templates/default/*

  chmod go-w /opt/services/tftpboot/*

  chmod -R o-w /opt/services/ganglia-local/var/rrds

• Mar. 28, 2016 krowe : (V-38580) Does not apply to us. All our machines pass.
• Mar. 28, 2016 jrobnett: applied the following changes (note the 3 kernel RPMs commands were not run on the Lustre servers:

  rpm -Uvh /home/usno-serv/updates/yum/RHEL-6.7.1.2/kernel/kernel-abi-whitelists-2.6.32-573.18.1.el6.noarch.rpm 
  rpm -e kernel-debug-devel-2.6.32-573.el6.i686
  rpm -e kernel-debug-devel-2.6.32-573.el6.x86_64
  sed -i -e 's/PASS_MIN_LEN *14/PASS_MIN_LEN    15/g' /etc/login.defs
  chmod 600 /var/log/boot.log
  chmod 644 /usr/lib64/librpfits.a