Lustre Build and Install notes

Lustre build notes for building and installing Lustre servers and clients against Redhat (RHEL5.5) kernel.

Get Quilt patch manager

• RHEL 5 RPM variant can be obtained from rpm.pbone.net

Download, install kernel source

• Download kernel source from RHN current: version is kernel-2.6.18-164.15.1.el5.src.rpm
  wget ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/SRPMS/kernel-2.6.18-164.15.1.el5.src.rpm
• Install RPM
  rpm -ivh kernel-2.6.18-128.1.1.el5.src.rpm
• edit /usr/src/redhat/SPECS/kernel-2.6.spec and modify buildid: %define buildid .lustre
• Build RPM
  rpmbuild -bp --target=x86_64 /usr/src/redhat/SPECS/kernel-2.6.spec

Download, configure Lustre source

• Download lustre source from http://wiki.lustre.org/index.php/Download
  • Select OS version, select source rather than OS version
  • Download latest version current: lustre-1.8.5.tar.gz
  • Unpack lustre, currently /users/jrobnett/Projects/Lustre/source
    • Will probably have to put it somewhere local for root write permissions ${LUSTRE_SRC}

Patching kernel

• Clean up any old patch or series files, link to new patch, series files
  cd /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
  rm -f patches series
  ln -s ${LUSTRE_SRC}/lustre/kernel_patches/series/2.6-rhel5.series series
  ln -s ${LUSTRE_SRC}/lustre/kernel_patches/patches patches
  
• Apply patches (will likely complain about jbd/journal.c, ignore)
  quilt push -av

Configure kernel

• cd /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
• cp /boot/config-'uname -r' .config
• make oldconfig (converts original .config to new version appropriate .config)
• make menuconfig (make any changes necessary, likely not needed)
• edit .config add # x86_64 to first line
• edit Makefile, make EXTRAVERSIONS match version (likely -164.15.1.el5.lustre)
• Update versions in various files
  • make include/asm
  • make include/linux/version.h
  • make SUBDIRS=scripts
  • make include/linux/utsrelease.h
• Do not install the kernel yet

Build Lustre RPMS

• cd to lustre source
• ./configure --with-linux={path to linux source}, eg ./configure --with-linux=/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
• make rpms (creates RPMS in /usr/src/redhat/RPMS)
• Should result in the following RPMS

lustre-1.8.3-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm
lustre-debuginfo-1.8.3-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm
lustre-ldiskfs-3.0.9-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm
lustre-ldiskfs-debuginfo-3.0.9-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm
lustre-modules-1.8.3-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm
lustre-source-1.8.3-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm
lustre-tests-1.8.3-2.6.18_164.15.1.el5_lustre_201006281617.x86_64.rpm

Build Kernel RPM and install

• build kernel rpm: rpmbuild -bb --with baseonly --without debug --without debuginfo --target=`arch` /usr/src/redhat/SPECS/kernel-2.6.spec
• rpm -ivh /usr/src/redhat/RPMS/x86_64/kernel-*-<VERSION> (make sure kernel, kernel-devel and kernel-headers)
• Reboot using new kernel (may have to change /boot/grub/grub.conf default entry).

e2fsprogs

• Install all e2fsprogs RPM from http://wiki.lustre.org/index.php/Download (NRAO users can find this in /home/src/lustre)
  • Click on the download link
  • For Platform select Red Hat Enterprise Linux 5, x86_64
  • Download e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm
  • e2fsprogs has to be forced (rpm -Uvh --force e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm)

Lustre OS setup

• Modify /etc/modprobe.conf, add lnet and infiniband entries.
  • echo 'alias ib0 ib_ipoib' >> /etc/modprobe.conf
  • echo 'alias net-pf-27 ib_sdp' >> /etc/modprobe.conf
  • echo 'options lnet networks="o2ib0(ib0),tcp0(eth0)"' >> /etc/modprobe.conf
• Modify rc.local
  • echo 'blockdev --setra 8192 /dev/sdb' >> /etc/rc.local (repeat for each device, sdb, sdc, sdd, sde etc)
  • echo 'modprobe ib_umad' >> /etc/rc.local (needed for ganglia monitoring)
  • echo '/opt/services/ganglia/bin/infin.py' >> /etc/rc.local (needed for ganglia monitoring, should be elsewhere)
• Setup ib0 interface
  • copy from existing OSS, modify IP addr to be next in series
• Change default kernel to new lustre kernel and reboot

Lustre filesystem setup and mount

• Setup partitions on all raid groups, need to use parted due to fdisk size limitation
  • parted -s /dev/sdX mklabel gpt (Make label for each device: sdb, sdc, sdd, sde etc)
  • parted -s /dev/sdX mkpart ext3 0 100% (Make partition for each disk that spans entire disk)
  • Result should look like below for each disk

                       
                                    [root@aoc-oss-3 ~]# parted -s /dev/sdb  print
                                    Model: AMCC 9650SE-8LP DISK (scsi)
                                    Disk /dev/sdb: 8000GB
                                    Sector size (logical/physical): 512B/512B
                                    Partition Table: gpt

                                    Number  Start   End     Size    File system  Name  Flags
                                    1      17.4kB  8000GB  8000GB               ext3   

WARNING
The next steps create OST's and contact an active MDS server. Steps are not reversible. Stop if you're not certain of what you're doing.
WARNING

• mkfs.lustre --fsname lustre --mkfsoptions="-E stride=16,stripe-width=64 -m 0" --ost --mgsnode=192.168.1.10@o2ib0 /dev/sdX make lustre filesystem for each device: sdb, sdc, sdd, sde etc
  • stride= Raid chunk size (64k) / filesystem block size (4k) = 16
  • stripe-width= stride (16) * number of data disks (4) = 64
  • mgsnode is address of MDS server @XXXX is link type (tcp, infiniband over IP etc).
• Add fstab entry of the form /dev/sdX /export/lustre/ostY-Z lustre defaults 0 0
  • Where sdX is the device, sdb, sdc etc
  • ostY-Z is the oss and device so ost3-2 would be the 2nd disk on the 3rd OSS server.
• Make mount points, and mount

-- JamesRobnett - 22 Jun 2010 -- KScottRowe - 2011-07-11