Lustre Build and Install notes

Lustre build notes for building and installing Lustre servers and clients against Redhat (RHEL5.5) kernel.

Get Quilt patch manager

Download, install kernel source

  • Download kernel source from RHN current: version is kernel-2.6.18-164.15.1.el5.src.rpm
  • Install RPM
    rpm -ivh kernel-2.6.18-128.1.1.el5.src.rpm
  • edit /usr/src/redhat/SPECS/kernel-2.6.spec and modify buildid: %define buildid .lustre
  • Build RPM
    rpmbuild -bp --target=x86_64 /usr/src/redhat/SPECS/kernel-2.6.spec

Download, configure Lustre source

  • Download lustre source from
    • Select OS version, select source rather than OS version
    • Download latest version current: lustre-1.8.5.tar.gz
    • Unpack lustre, currently /users/jrobnett/Projects/Lustre/source
      • Will probably have to put it somewhere local for root write permissions ${LUSTRE_SRC}

Patching kernel

  • Clean up any old patch or series files, link to new patch, series files
    cd /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
    rm -f patches series
    ln -s ${LUSTRE_SRC}/lustre/kernel_patches/series/2.6-rhel5.series series
    ln -s ${LUSTRE_SRC}/lustre/kernel_patches/patches patches
  • Apply patches (will likely complain about jbd/journal.c, ignore)
    quilt push -av

Configure kernel

  • cd /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
  • cp /boot/config-'uname -r' .config
  • make oldconfig (converts original .config to new version appropriate .config)
  • make menuconfig (make any changes necessary, likely not needed)
  • edit .config add # x86_64 to first line
  • edit Makefile, make EXTRAVERSIONS match version (likely -164.15.1.el5.lustre)
  • Update versions in various files
    • make include/asm
    • make include/linux/version.h
    • make SUBDIRS=scripts
    • make include/linux/utsrelease.h
  • Do not install the kernel yet

Build Lustre RPMS

  • cd to lustre source
  • ./configure --with-linux={path to linux source}, eg ./configure --with-linux=/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
  • make rpms (creates RPMS in /usr/src/redhat/RPMS)
  • Should result in the following RPMS

Build Kernel RPM and install

  • build kernel rpm: rpmbuild -bb --with baseonly --without debug --without debuginfo --target=`arch` /usr/src/redhat/SPECS/kernel-2.6.spec
  • rpm -ivh /usr/src/redhat/RPMS/x86_64/kernel-*-<VERSION> (make sure kernel, kernel-devel and kernel-headers)
  • Reboot using new kernel (may have to change /boot/grub/grub.conf default entry).


  • Install all e2fsprogs RPM from (NRAO users can find this in /home/src/lustre)
    • Click on the download link
    • For Platform select Red Hat Enterprise Linux 5, x86_64
    • Download e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm
    • e2fsprogs has to be forced (rpm -Uvh --force e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm)

Lustre OS setup

  • Modify /etc/modprobe.conf, add lnet and infiniband entries.
    • echo 'alias ib0 ib_ipoib' >> /etc/modprobe.conf
    • echo 'alias net-pf-27 ib_sdp' >> /etc/modprobe.conf
    • echo 'options lnet networks="o2ib0(ib0),tcp0(eth0)"' >> /etc/modprobe.conf
  • Modify rc.local
    • echo 'blockdev --setra 8192 /dev/sdb' >> /etc/rc.local (repeat for each device, sdb, sdc, sdd, sde etc)
    • echo 'modprobe ib_umad' >> /etc/rc.local (needed for ganglia monitoring)
    • echo '/opt/services/ganglia/bin/' >> /etc/rc.local (needed for ganglia monitoring, should be elsewhere)
  • Setup ib0 interface
    • copy from existing OSS, modify IP addr to be next in series
  • Change default kernel to new lustre kernel and reboot

Lustre filesystem setup and mount

  • Setup partitions on all raid groups, need to use parted due to fdisk size limitation
    • parted -s /dev/sdX mklabel gpt (Make label for each device: sdb, sdc, sdd, sde etc)
    • parted -s /dev/sdX mkpart ext3 0 100% (Make partition for each disk that spans entire disk)
    • Result should look like below for each disk
                                    [root@aoc-oss-3 ~]# parted -s /dev/sdb  print
                                    Model: AMCC 9650SE-8LP DISK (scsi)
                                    Disk /dev/sdb: 8000GB
                                    Sector size (logical/physical): 512B/512B
                                    Partition Table: gpt

                                    Number  Start   End     Size    File system  Name  Flags
                                    1      17.4kB  8000GB  8000GB               ext3   

The next steps create OST's and contact an active MDS server. Steps are not reversible. Stop if you're not certain of what you're doing.

  • mkfs.lustre --fsname lustre --mkfsoptions="-E stride=16,stripe-width=64 -m 0" --ost --mgsnode= /dev/sdX make lustre filesystem for each device: sdb, sdc, sdd, sde etc
    • stride= Raid chunk size (64k) / filesystem block size (4k) = 16
    • stripe-width= stride (16) * number of data disks (4) = 64
    • mgsnode is address of MDS server @XXXX is link type (tcp, infiniband over IP etc).
  • Add fstab entry of the form /dev/sdX /export/lustre/ostY-Z lustre defaults 0 0
    • Where sdX is the device, sdb, sdc etc
    • ostY-Z is the oss and device so ost3-2 would be the 2nd disk on the 3rd OSS server.
  • Make mount points, and mount

-- JamesRobnett - 22 Jun 2010 -- KScottRowe - 2011-07-11
Topic revision: r3 - 2011-07-12, KScottRowe
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback