You are here: NRAO Public Wiki>HPC Web>USNODifxCorrelator>UsnoCorrelatorLegacy>UsnoProjectBook>UsnoDifxDisklessBoot (2014-08-05, KScottRowe) Edit wiki text Edit Attach Print version

Diskless Booting of the Compute Nodes

This topic discusses the configuration of the diskless network boot system proposed to be used with the USNO DifX software correlator. This system has completely diskless nodes that mount the root filesystem to a tmpfs filesystem. The /usr filesystem is mounted from the infrastructure node to keep the image size small. /tmp is mounted on it's own tmpfs file system with a limit on it's size. An additional mount to /opt from the infrastructure node provides more shared disk area.

How it works (overview)
Building the ramdisk image
Add init.d scripts to image
Creating a new image with a new version number
Enable remote logging to infrastructure node.
Configure email
USNO Security
Add a node

How it works (overview)

When a node powers up, it sends a PXE boot DHCP request which the infrastructure node answers and provides the pxelinux.0 file through TFTP. This file provides the routines necessary to continue a network boot. A second DHCP request is made and the node's IP address is obtained. This is used by the node to find it's configuration file in the pxelinux.cfg directory in the tftpboot area of the server. This file specifies the kernel and initial ramdisk to use. Both are provided by TFTP.

The node now boots the kernel and loads the initial ramdisk image and makes one more DHCP request to configure the node's network interface(s). The node then NFS mounts /usr, /opt, and /home/usno from the infrastructure node and is ready.

The following services should be configured first:

Building the ramdisk image

The first steps are performed on usno-node-1t, then the rest is done on usno-serv-1t. Install OS on usno-node-1t and perform steps to create a copy of the OS. The scripts referred to below, USNO_OSCopy.sh and USNO_BuildImage.sh, are kept in /opt/services/build/bin on the test cluster infrastructure server and use RCS for version control.

Install RHEL6.3 workstation in usno-node-1t:

Add DHCP entry on zia and restart DHCP

# static 10.64.1.44
   host usno-node-1t {
      hardware ethernet 24:B6:FD:F5:AA:85;
      server-name "146.88.1.8";
      filename "/os/install/Linux/kickstart/profiles/usno-node-1t";
      fixed-address usno-node-1t-ext.aoc.nrao.edu;
   }

Add to DNS on zia

edit /opt/services/named/updates/nraohosts.aoc

update add usno-node-1t-ext.aoc.nrao.edu 86400 A 10.64.1.44
send

edit /opt/services/named/updates/nrao-nm.rev

update add 44.1.64.10.in-addr.arpa 86400 PTR usno-node-1t-ext.aoc.nrao.edu
send

Kickstart file on zia

/home/os/install/Linux/kickstart/profiles/usno-node-1t

Perform kickstart

$ Run the attached usno_security.sh script to comply with USNO security policies:

Make a copy of the OS for image building on usno-node-1t-ext

On usno-node-1t, set environment variable BUILDROOT (as root)

mkdir -p /export/home/usno-node-1/build
export BUILDROOT=/export/home/usno-node-1/build

Run the script USNO_OSCopy.sh

This creates the directory /export/home/usno-node-1/build/0_OSimage/ $ Copy (rsync) this to usno-serv-1:/opt/services/build:

ssh root@usno-serv-1 mkdir -p /opt/services/build
cd /export/home/usno-node-1/build
rsync -av 0_OSimage root@usno-serv-1:/opt/services/build/

Now done with work on usno-node-1-ext

Node image creation steps performed on usno-serv-1-ext

The following steps are performed on usno-serv-1-ext as root to create the nodes' ramdisk image. Some of the changes are made to the 0_OSimage using a chroot environment and most are made by the script USNO_BuildImage.sh as it builds the image.

Set the environment variable BUILDROOT

export BUILDROOT=/opt/services/build

Make a backup copy in case recovery is necessary

cd $BUILDROOT
tar czvf 0_OSimage.0.tgz 0_OSimage

Install pgplot RPMs from /home/os/install/Linux/updates/yum/rhel5.8/x86_64/nrao_casa-x86_64/

These were requested by Walter for testing the cluster and need to be copied into the 0_OSimage/root directory for installation in the chroot later. The RPMs are for RHEL5.8, but seem to work on RHEL6.3. Copy these over from zia. On zia:

cd /home/os/install/Linux/updates/yum/rhel5.8/x86_64/nrao_casa-x86_64
scp pgplot-5.3.1-19.el5.x86_64.rpm pgplot-devel-5.3.1-19.el5.x86_64.rpm root@usno-serv-1-ext:/opt/services/build/0_OSimage/root/

Chroot into the OS copy directory

cd $BUILDROOT/0_OSimage
chroot .

Mount the proc file system

mount -t proc none /proc

Perform some setup tasks in the chroot:

Turn off some services

for i in NetworkManager nfs acpid autofs cpuspeed edac fcoe-target ksm \
ksmtuned libvirt-qmf lvm2-monitor nfslock portreserve smartd \
stap-server systemtap virt-who firstboot autoyum ypbind; do \
chkconfig $i off; done

Turn on some services

for i in mcelogd netfs network ntpd;do chkconfig $i on;done

Install pgplot RPMs copied earlier in to /root

cd /root
rpm -Uvh  pgplot-5.3.1-19.el5.x86_64.rpm pgplot-devel-5.3.1-19.el5.x86_64.rpm

Perform any other tasks in the chroot that need doing

Exit the chroot and umount proc

exit
umount $BUILDROOT/0_OSimage/proc

Add init.d scripts to image

These are init.d scripts that we want to run. They are installed as actual files in the 0_OSimage area. Then chroot and run chkconfig --add . The image creation script then removes the actual files and creates symlinks to copies of the links that live in the /opt/services tree.

nrao-umount_nfs

Lazily unmounts all NFS mounts thus allowing the machine to reboot without hanging on NFS.
Save the attached nrao-umount_nfs in /opt/services/build/0_OSimage/etc/init.d

chroot /opt/services/build/0_OSimage
    chmod 755 /etc/init.d/nrao-umount_nfs
    chkconfig --add nrao-umount_nfs
    exit

Run USNO_BuildImage.sh os-image: This creates a cpio gzipped file in $BUILDROOT with a name using a date and time stamp string. e.g., sysimage.cpio.201207110907.gz.

Creating a new image with a new version number

Any time changes are made to the image in 0_OSimage using chroot, the new image created from these changes must synchronize with this changed $BUILDROOT/0_OSimage/usr. To do this, version numbers are used to match the image with the mounted /usr.

For example: After changes are made in 0_OSimage using chroot (these should be the only way 0_OSimage is changed), 0_OSimage/usr must be copied (rsync -av or cp -a) to /export/home/usno-serv/client_fs/, e.g. /export/home/usno-serv/client_fs/usr-6.3.1 and the image creation script, USNO_BuildImage.sh, must be changed to reflect the changed mount of /usr. This is accomplished by editing the here document in the section commented with Setup /etc/fstab.

The version numbering scheme is to increment the minor number each time a new image is created from a changed chrooted 0_OSimage. It's not necessary to increment to version or copy /usr if the changes to the image are only made in the USNO_BuildImage.sh script.

The image version must also be updated in /opt/services/tftpboot/pxelinux.cfg/, can be default or a config file for a specific node. If a new kernel was installed in the chrooted 0_OSimage. The new kernel must be copied to /opt/services/tftpboot/linux and the config file updated to reflect it. Each node has a symbolic link in /opt/services/tftpboot/pxelinux.cfg/ representing the hex version of it's IP address. This link can be pointed to the desired config file. e.g.,

[root@usno-serv-1 pxelinux.cfg]# pwd
/opt/services/tftpboot/pxelinux.cfg
[root@usno-serv-1 pxelinux.cfg]# ls -l
total 12
lrwxrwxrwx 1 root root   5 Jul 11 09:37 0A012265 -> 6.3.2
lrwxrwxrwx 1 root root   5 Jul 11 09:37 0A012266 -> 6.3.2
lrwxrwxrwx 1 root root   5 Jul 11 09:37 0A012267 -> 6.3.2
lrwxrwxrwx 1 root root   5 Jul 11 09:37 0A012268 -> 6.3.2
lrwxrwxrwx 1 root root   5 Jul 11 09:13 0A012269 -> 6.3.2
-rw-r--r-- 1 root root 144 Jul 10 13:15 6.3.2
-rw-r--r-- 1 root root 144 Jul 17 14:01 6.3.3
-rw-r--r-- 1 root root 144 Jul  9 16:08 default
[root@usno-serv-1 pxelinux.cfg]#

Enable remote logging to infrastructure node.

Add the following line to /etc/rsyslog.conf

*.info;mail.none;authpriv.none;cron.none                @usno-serv-1

and build a new image.

Configure email

The node needs to know to what machine to send email and as what domain to masquerade so it will get through.

edit /etc/mail/submit.mc

FEATURE(`msp', `[usno-serv-1]')dnl
MASQUERADE_AS(`usno.nrao.edu')dnl

Make the submit.cf file

cd /etc/mail ; make

and build a new image.

USNO Security

cd $BUILDROOT/0_OSimage chroot .

Mount the proc file system

mount -t proc none /proc

$ Run the attached usno_security.sh script to comply with USNO security policies:

Turn off X11 Forwarding

sed -i -e 's/^\(X11Forwarding.*$\)/#\1/g' /etc/ssh/sshd_config

Ensure users can only ssh in with passphrase keys

echo "PasswordAuthentication no" >> /etc/ssh/sshd_config

Change root passwd to match USNO requirements: passwds are a minimum of 10 characters and must satisfy all four character classes. Set the passwd on the machine after running the USNO security script so that the hash is created the way USNO wants it.

Exit the chroot and umount proc

exit
umount $BUILDROOT/0_OSimage/proc

Add a node

Enable PXE boot in bios
Add a dhcp entry on the server
Set the BIOS clock to current, UTC time. If not, the DHCP lease may be so old that when dhclient runs next, it will do a DHCPDISCOVER which unplumbs the network.

Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
EXT	nrao-umount_nfs	manage	1 K	2013-03-27 - 11:50	KScottRowe	allows node to reboot
sh	usno_security.sh	manage	9 K	2012-08-29 - 17:04	KScottRowe	USNO security changes

Topic revision: r32 - 2014-08-05, KScottRowe

HPC

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback