-- DanielKlopp - 2011-12-15

Comments on Content

This information is provided as is in the hope that it will be helpful to some people. Be careful utilizing some of the scripts in this document as they may cause data loss if used incorrectly. BE WARNED, some of these scripts will forcibly elliminate disk partitions and remake them in a destructive manner. Use at your own risk.

I've removed a few items from this documentation which should be set by each individual site utilizing this method. These mostly have to do with internal IP addresses, NIS and some miscellaneous odds and ends. If you find a variable pointing to what should be, for example, the NISDOMAIN, it will likely be left unset in this document. Please set it to what is appropriate to your installation. I will point out the missing parts when you reach them.

This document is not wholly accurate at this time

Operation of the Diskless NFS Root Nodes

First, the "why".

In Charlottesville we will be charged with up to 64 compute nodes of the same physical hardware. Each one is to be part of a cluster scheduler and used in a pipeline. These nodes will be maintained for many years to come. In order to ease their installation, replacement, configuration normalization, and general maintenance, I decided it would be best to attach all nodes into a diskless NFS Client installation. One central server provides a read-only copy of the root filesystem. Each compute node mounts the same read-only copy of the root filesystem with an early boot init script "shared_nfs_root" which makes some minor tweaks to the filesystem for read-write areas and configures host specific items (such as IP address, hostname, etc). A central management script, called multivac_maintainer, is used to handle system updates, kernel updates, and the like. The management scripts enables updates to be made but not affect the compute nodes. When acceptable changes have been made, the updates may be synchronized into the running system as appropriate. The central management script, with this document, is also capable of regenerating an entire root filesystem from an already existing installation.

Each compute node, though "diskless" expects a small hard drive to be available. Due to our computational needs, we require a significant /tmp and swap space. The script shared_nfs_root will forcibly recreate the hard drive with a 32GB swap and 10G miscellaneous space for /var, /tmp and a few other directories. This was done so that the maximum amount of RAM possible could be used for scientific computing rather than /tmp space.

Prerequisite Steps

Before starting I assume you have a standard template installation of a compute node and a reserved computer for serving out the NFS root filesystem. For this example I will use multivac08.cv.nrao.edu as the template compute node and foundation.cv.nrao.edu as the NFS root server and multivac13.cv.nrao.edu as an NFS Root client (sometimes called 'diskless' node though strictly speaking we don't have that).

Furthermore, I assume that all compute nodes have static IP addresses with an associated DNS entry (not dynamic DNS). This is necessary for some of the scripts to work during booting. A proper DHCP server capable of serving PXE requests is required as well. I also assume you have already configured your tftp server with a common layout of /tftpboot/{kernels here} and /tftpboot/pxelinux.cfg/{boot entries here} per the instructions at https://wiki.archlinux.org/index.php/Diskless_network_boot_NFS_root

Initial Setup

To start, downloaded multivac_maintainer.sh and place it in /bin/ on foundation. Load your ssh root keys or be prepared to type in root passwords several times.

Use your favorite editor and open /bin/multivac_maintainer.sh . Change the beginning line variables to whatever suits your environment. It is very important some of these values are correctly entered, as multivac_maintainer will take ownership of several of the directories, chroot into them, delete them, move them, and synchronize them back and forth. For reference, here are the variables you should ensure are set correctly:

BASEDIR=/export/data_1/croot
ACTIVEROOTDIR=$BASEDIR/root
LATESTROOTDIR=$BASEDIR/root.latest
TFTPDIR=/tftpboot
BOOTFILE=$TFTPDIR/pxelinux.cfg/default
SSHOPTS="ConnectTimeout=2s"
KERNELSOURCE=$BASEDIR/kernels

# This corresponds to multivac01 to multivac64
# Note how here you do not start with a leading
# 0 even if appropriate
FIRSTNODE=9
LASTNODE=64

NODEBASENAME="multivac"
NODEDOMAINNAME="cv.nrao.edu"
NISDOMAIN=""
LUSTRESOURCE=$KERNELSOURCE/lustre-1.8.5.tar.gz

Set NISDOMAIN above to what is appropriate for your system.

$KERNELSOURCE is where you place the src rpms from redhat when building a new kernel. $LUSTRESOURCE is where the lustre source code tar file is located. The other variables are self explanatory.

Look at the section with:

if [ -z "$FOUNDATION" ]; then
   FOUNDATION=`ifconfig|head -2|grep "inet addr"|awk -F: '{print $2}'|awk '{print $1}'`
fi

FOUNDATION is the box where NFS root is located. By default multivac_maintainer.sh is run on that box so those lines will grab the correct IP address.

Edit the file where it reads:

cat > dhclient.conf <<-EOF
send domain-name "$NODEDOMAINNAME";
append domain-name " $SEARCHPATH";
EOF

And update the $SEARCHPATH variable to whatever your normal /etc/resolv.conf search path is.

Now run the script with the command

multivac_maintainer create new root

Here is some sample output.

Creating a new root directory
Note that this will not work well unless you have root ssh keys
loaded, otherwise expect alot of password prompts

Have you updated this file with the proper variable
updates? Currently the ones of interest are
BASEDIR : /export/data_1/newroot
ACTIVEROOTDIR : /export/data_1/newroot/root
LATESTROOTDIR : /export/data_1/newroot/root.latest
TFTPDIR : /tftpboot
BOOTFILE : /tftpboot/pxelinux.cfg/default
KERNELSOURCE : /export/data_1/newroot/kernels
NODEBASENAME : multivac
NODEDOMAINNAME : cv.nrao.edu
FIRSTNODE : 9
LASTNODE : 64
LUSTRESOURCE : /export/data_1/newroot/kernels/lustre-1.8.5.tar.gz
NISDOMAIN : --

Please note that FIRSTNODE and LASTNODE imply the compute nodes
have a naming scheme of name01, name02, name03, ..., name31, etcetera
If not, this script won't work as intended

Set properly [yes/no]?yes

Enter FQDN of the computer we are to mirror the root
filesystem from
FQDN:multivac08.cv.nrao.edu

After which it will proceed to synchronize the root directory of multivac08.cv.nrao.edu (or whatever you specify to the script). For full reference it will pull the filesystem likeso:

rsync -av --exclude="lost+found" --exclude='/net/*' --exclude='/users/*' --exclude='/proc/*' --exclude='/sys/*' --exclude='/export/*' --exclude='/home/*' multivac08.cv.nrao.edu:/ $ACTIVEROOTDIR

Upon exiting the system add the following entries to $ACTIVEROOTDIR/etc/rc.local (change $IPADDR to the ip address of your Lustre MDS)

modprobe ib_umad
modprobe lustre
sleep 15 # The IB driver is slow to wake up
modprobe lustre && mount -t lustre $IPADDR@o2ib:/naaschpc -o user_xattr,flock /export/lustre
# because I do not have a drift file the system will not be
# stable for the clock
# but I do not see that as a problem given the current implementation
# Still, a quick update is a good thing
ntpdate -u ntp1.cv.nrao.edu
/opt/services/torque/sbin/pbs_mom

The above loads the Infiniband modules, lustre, mounts lustre, updates the clock and enables the PBS monitoring daemon. You may need to tweak it for your specific installation.

A word on a booting machine. multivac_maintainer modifies /etc/rc.sysinit to run /etc/init.d/shared_nfs_root . This script first erases the partition table, then creates a new partition table with appropriate filesystems. After this it will create a series of symlinks, directories, mounts and network changes. Then the box continues booting. The reason for using a local disk is to handle /var, /root, /tmp and swap as some of our processes require immense swap areas.

Download shared_nfs_root (link at bottom of wiki) and place it in $ACTIVEROOTDIR/etc/init.d/ and set it as executable. Double check the variables at the top of the script to ensure they are appropriate for your installation. Ensure that the functions configureetc() and configurenet() have proper settings as well, otherwise it won't work. Activate the script (change ACTIVEROOTDIR as you have it in multivac_maintainer):

cat > $ACTIVEROOTDIR/runthis.sh <<EOF
chmod +x /etc/init.d/shared_nfs_root
chkconfig yum-updatesd off
chkconfig autoyum off
chkconfig shared_nfs_root on
/sbin/chkconfig autoyum off
/sbin/chkconfig update_os off
EOF

chmod +x $ACTIVEROOTDIR/runthis.sh
chroot $ACTIVEROOTDIR /runthis.sh
rm -f $ACTIVEROOTDIR/runthis.sh

Update Foundation's /etc/exports to point to $ACTIVEROOTDIR. My setup is:

/export/data_1/root *(ro,no_root_squash,no_all_squash)

Then run the command "exportfs -r"

Edit Foundation's tftp server likeso:

service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot
        disable                 = no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

Now restart xinetd "service xinetd restart"

Configure an rsync daemon by editing /etc/rsyncd.conf and adding something along the following lines (/export/data_1/root should be ACTIVEROOTDIR)

[var]
        path = /export/data_1/root/var.pristine
        read only = true
        uid = root
        gid = root
        comment = var directory

[etc]
        path = /export/data_1/root/etc_template
        read only = true
        uid = root
        gid = root
        comment = modified /etc components directory
[root]
        path = /export/data_1/root/root.pristine
        read only = true
        uid = root
        gid = root
        comment = /root copy so it is writable, state is lost
[services]
        path = /export/data_1/root/opt/services.pristine
        read only = true
        uid = root
        gid = root
        comment = /opt/services copy so it is writable, state is lost

Now open $ACTIVEROOTDIR/etc/init.d/shared_nfs_root and edit the rsyncdisks function to match what you have set in the rsyncd.conf file above.

You are not yet ready to boot a node. Before doing that you have to generate the kernel and modules. The easiest way to do this is with the multivac_maintainer in the next section.

Creating the Latest Root

There is active root and latest root. Active root is where running compute nodes mount their root filesystem. Latest root is where Foundation handles updates, patches and configuration tweaks. You can roll back changes to latest, or you can synchronize the updates in latest with active. This was done to avoid interrupting end users.

Creating the latest root is easy if you have active root properly set up (see prior section). If so, all you do is run multivac_maintainer like this:

multivac_maintainer sync active to latest

That will take a bit. While it is doing that ensure that the $KERNELSOURCE has at least one up to date Red Hat 5 kernel source rpm (eg kernel-2.6.18-274.12.1.el5.src.rpm ) as well as the lustre source (eg lustre-1.8.5.tar.gz). Next download kernel-config from this wiki and copy that as kernel-config to $KERNELSOURCE.

For multivac_maintainer to function properly please see that newer files have newer timestamps. A bunch of the update scripts depend upon accurate timestamps otherwise you won't get the latest kernel.

When the sync is complete proceed to update the system (you may need to tweak the files in $LATESTROOTDIR/etc/yum.repos.d/)

multivac_maintainer maint packages system

Next update casa

multivac_maintainer maint packages casa

For the above two steps you could have used "maint package" or "maint package all". The kernel is never updated unless explicitly called for.

build the kernel

multivac_maintainer maint kernel update

Migrate changes in latest to active.

multivac_maintainer sync latest to active

Set the kernel to boot from

multivac_maintainer maint kernel set default $version

Where $version is the filename of the linux kernel in /tftpboot (eg, vmlinuz-2.6.18-274.12.1.el5)

Boot multivac13.cv.nrao.edu

• kernel-nfsroot-working-config: Kernel config

• shared_nfs_root: shared_nfs_root boot control script

• multivac_maintainer.sh: NFS Root management script

• multivac_maintainer.sh: NFS Root management script