Installing RedHat-Enterprise 6 Workstation on the USNO Lustre MDS

Pre Installation

Enable Virtualization

This is necessary for 64bit versions of VMware. While we don't plan to use VMware on these hosts it's a good habit to turn this on.

Disable Hyper Threading

Hyper Threading is a very cheap trick to simulate dual CPUs. All it does is create a second entry point into the pipeline which should reduce bubbles in the pipeline but it can be detrimental for some mathematical functions of which we do many. Turn it off in the BIOS if you can.

Disable C-States

If the computer is to do significant computation or bandwidth (e.g. cluster node, lustre server, etc) then disable both C1E and C-State in the BIOS.

Install image

Install OS as a normal AOC Linux system, use usno-mds kickstart file.

Post Installation

Update /etc/files

Replace /etc/passwd from usno-serv-1-ext:/etc/passwd (must include root password and difxmgr,admin,oper, etc accounts.: Replace /etc/shadow from usno-serv-1-ext:/etc/shadow (must include root password and difxmgr,admin,oper, etc accounts.: Replace /etc/group from usno-serv-1-ext:/etc/group

Configure it to run off our network

rm -f /etc/cron.daily/autoyum.cron*
chkconfig ypbind off
chkconfig autofs off
chkconfig update_os off
chkconfig autoyum off
service autofs stop

Remove the os_update cron job

sed -i '$d' /etc/crontab

Configure ntpd from usno node /etc/ntp.conf

driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6::1
server usno-serv-1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys

Configure X not to start

sed -i -e 's/id:5:initdefault:/id:3:initdefault:/' /etc/inittab

The servers don't need /usr/local and since we don't rsync /usr/local to them, it will only get out of date. So just remove it

rm -f /usr/local
rm -f /opt/local
mv /usr/local.bak /usr/local

Configure the internal 1Gb network

edit /etc/sysconfig/network-scripts/ifcfg-em1:

DEVICE="em1"
BOOTPROTO="static"
HWADDR="<insert addr>"
IPV6INIT="no"
MTU=1500
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
IPADDR=10.1.34.100
NETMASK=255.255.255.0
NETWORK=10.1.34.0

Update fstab and /home/usno mount

mkdir -p /home/usno
echo "usno-serv-1:/export/home/usno-serv/usno /home/usno nfs  defaults        0 0" >> /etc/fstab

Wait for network

netfs runs too soon after network so for the above to work need to add the following line to /etc/sysconfig/network

NETWORKDELAY=10

Update resolv.conf

search usno.nrao.edu.
nameserver 10.1.34.2

Syslog

Replace /etc/rsyslog.conf @loghost reference line with:

*.info;mail.none;authpriv.none;cron.none                @usno-serv-1

Install e2fsprogs

rpm -Uvh --nodeps e2fsprogs-*

Note this is nearly irreversable. you can not easily return to the stock version, you'll need to boot off of CD to do it

Install Lustre RPMS

rpm -Uvh lustre*

Install Lustre kernel

rpm -Uvh --force kernel*

Fixes to pam from the above script

Delete the 'nodefgroup' reference in etc/pam.d/system-auth

Fixes to /etc/security/access.conf from the above script

change the users and wheel line to the one below

+:users:ALL
+:wheel:ALL

Setup LNET configuration, add following line to /etc/modprobe.conf

echo 'options lnet networks="tcp0(bond0)"' >> /etc/modprobe.conf

Setup raid device for Lustre MDT filesystem

mdadm --create /dev/md0 -l 1 -n 2 -x 0 /dev/sdb /dev/sdc

Create the MDT filesystem

mkfs.lustre --fsname lustre --mdt --mgs /dev/md0

Create the mount point and mount the filesystem

mkdir -p /export/lustre/mdt
mount -t lustre /dev/md0 /export/lustre/mdt 

Add mount to /etc/fstab

echo "/dev/md0 /export/lustre/mdt lustre defaults 0 0" >> /etc/fstab