Overview

Lustre OSSes were purchased from Aeon computing and come pre-installed with Centos 5.9 and Lustre 1.8.9. The following are modifications to the received OS.

Network

• Modify on each OSS node /etc/sysconfig/network-scripts/ifcfg-eth0 (usno-oss-1 used as example)

  # Intel Corporation I350 Gigabit Network Connection
  DEVICE=eth0
  BOOTPROTO=static
  HWADDR=00:25:90:E0:DC:86
  ONBOOT=yes
  HOSTNAME=usno-oss-1
  IPADDR=10.1.34.101
  NETMASK=255.255.255.0
  GATEWAY=10.1.34.1
  TYPE=ethernet
  

• Modify on each OSS node for each 10gbit interface /etc/sysconfig/network-scripts/ifcfg-eth{2,3} (usno-oss-1-data used as example)

  # Solarflare Communications SFC9020 [Solarstorm]
  DEVICE=eth2
  HWADDR=00:0f:53:0e:25:e4
  BOOTPROTO=none
  ONBOOT=yes
  USERCTL=no
  MASTER=bond0
  SLAVE=yes
  MTU=9000
  

• Create on each OSS node a /etc/sysconfig/network-scripts/ifcfg-bond0 interface

  DEVICE=bond0
  IPADDR=10.1.35.102
  NETWORK=10.1.35.0
  NETMASK=255.255.255.0
  USERCTL=no
  BOOTPROTO=none
  ONBOOT=yes
  MTU=9000
  

• Append the following on each OSS node to /etc/modprobe.conf

  alias bond0 bonding
  options bond0 mode=balance-alb miimon=100
  

Setup ganglia

Since the OSS use a different OS version (centos 5.9) we can't share the ganglia install with the compute nodes, need to re-install from source. The steps essentially follow the DiFX clients instructions here: https://safe.nrao.edu/wiki/bin/view/HPC/UsnoDifxGanglia . Install on one of the OSSes and then tar up/copy to the other OSSes.

• Create an /opt area on usno-serv-1:/export/home/usno-serv/client_fs/opt-lustre
• On one OSS node copy /opt/MegaRaid to usno-serv-1:/export/home/usno-serv/client_fs/opt-lustre
• On each OSS add following /opt mount to /etc/fstab

  usno-serv-1:/export/home/usno-serv/client_fs/opt-lustre /opt nfs defaults       0 0
  

• On each OSS Install apr-devel-1.2.7-11.el5_6.5.x86_64.rpm, needed by ganglia and not installed with Aeon Centos (copied into /opt/services/src)

  rpm -Uvh /opt/services/src/apr-devel-1.2.7-11.el5_6.5.x86_64.rpm 
  

• On each OSS Install pcre-devel-6.6-6.el5_6.1.x86_64.rpm, needed by ganglia and not installed with Aeon Centos (copied into /opt/services/src)

   
  rpm -Uvh /opt/services/src/pcre-devel-6.6-6.el5_6.1.x86_64.rpm
  

• If building a new version from scratch then on one OSS follow instructions here: https://safe.nrao.edu/wiki/bin/view/HPC/UsnoDifxGanglia stop at the end of Compile Ganglia section, otherwise skip this step:

• Install Disk Metrics

Get diskstats.py script

Dwonload diskstats.py from https://github.com/ganglia/gmond_python_modules/pull/1/files

cp /home/src/ganglia/scripts/diskstats.py /opt/services/ganglia-local/lib64/ganglia/python_modules

Get disk_gmetric.sh script

Download disk_gmetric.sh from http://ben.hartshorne.net/ganglia/, then from download location copy to Lustre opt area on usno-serv-1:/export/home/usno-serv/client_fs/opt-lustre/services/ganglia-local/bin $ Copy startup script from NRAO-NM /home/src/ganglia/init.d/nrao-disk_gmetric to usno-serv-1:/export/home/usno-serv/client_fs/opt-lustre/services/ganglia-local/init.d/nrao-disk_gmetric

On each Create disk_gemtric startup script

It runs /opt/services/ganglia-local/bin/disk_gmetric.sh every 30 seconds

ln -s /opt/services/ganglia-local/init.d/nrao-disk_gmetric /etc/init.d
chkconfig --level 345 nrao-disk_gmetric on
/etc/init.d/nrao-disk_gmetric start

• On each node setup gmond start script

  ln -s /opt/services/ganglia-local/init.d/nrao-gmond /etc/init.d
  chkconfig --add nrao-gmond
  

$ On each node setup lustre for each device. When making mount directories like /export/lustre/ostX-Y X is the number of the OSS server and Y is the enumerated OST on that server. For instance, /export/lustre/ost1-1 is the first OST on the 1st server, /export/lustre/ost3-2 is the 2nd OST on the 3rd server.:

• Setup LNET configuration, add following line to /etc/modprobe.conf:

  echo 'options lnet networks="tcp0(bond0)"' >> /etc/modprobe.conf
  

• Make filesystems on /dev/sdc and /dev/sdd

  mkfs.lustre --fsname lustre -o force_over_24tb --mkfsoptions="-E stride=32,stripe-width=256 -m 0" --ost --mgsnode=10.1.35.100@tcp0 /dev/sdc
  mkfs.lustre --fsname lustre -o force_over_24tb --mkfsoptions="-E stride=32,stripe-width=256 -m 0" --ost --mgsnode=10.1.35.100@tcp0 /dev/sdd
  

• Mount one filesystem and server at a time. Do both FSes on one node, then the next node etc

  mkdir -p /export/lustre/ost1-1   *remember to change ost to ost2 or ost3 for other nodes*
  mkdir -p /export/lustre/ost1-2
  mount -t lustre /dev/sdc /export/lustre/ost1-1
  mount -t lustre /dev/sdd /export/lustre/ost1-2
  

• Add following lines to /etc/fstab, again change ost1 to ost2 or ost3 for 2nd, 3rd OSSes.

  /dev/sdc                /export/lustre/ost1-1   lustre  defaults        0 0
  /dev/sdd                /export/lustre/ost1-2   lustre  defaults        0 0
  

-- JamesRobnett - 2014-07-30