Install and Configure Lustre Software

Lustre Server RPM install

Download the following RPMs, matching your lustre version, OS, and kernel. Source
Intel's public repo
  • e2fsprogs
  • lustre; lustre-ldisksfs; lustre-modules
  • kernel; kernel-devel; kernel-headers

Example list
    • e2fsprogs-1.41.90.wc4-0redhat.x86_64.rpm
    • lustre-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
    • lustre-ldiskfs-3.1.51-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
    • lustre-modules-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
    • kernel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
    • kernel-devel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
    • kernel-headers-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm

Install e2fsprogs on MDS and all OSSes
rpm -Uvh --nodeps e2fsprogs-*
Note this is nearly irreversable. You can not easily return to the stock version, you'll need to boot off of CD to do it

Install Lustre kernel on MDS and all OSSes
rpm -Uvh --force kernel*

Install Lustre RPMS on MDS and all OSSes (ignore WARNINGs)
rpm -Uvh lustre*

Lustre Server config

Currently, our OSSes touch all three networks (1GB, 10GB, infiniband). This can be inefficient and someday we hope to install lnet routers that touch all three networks and only have infiniband on the OSSes.

If on the 1GB ethernet
echo 'options lnet networks="tcp0(eth0)"' >> /etc/modprobe.conf

If on the 10GB ethernet
echo 'options lnet networks="tcp1(eth2)"' >> /etc/modprobe.conf

If on infiniband
echo 'options lnet networks="o2ib0(ib0)"' >> /etc/modprobe.conf
Assign IP's within a private network, we use to ib0 interfaces for IP over IB. NRAO/AOC uses a scheme where:
  • The MDS is
  • The OSSes are -
  • and clients are -

Create ifconfig file
edit /etc/sysconfig/network-scripts/ifcfg-ib0
IPADDR=<IP Address>

Install IB modules
echo "alias ib0 ib_ipoib" >> /etc/modprobe.conf
echo "alias ib0 ib_umad" >> /etc/modprobe.conf

If on infiniband and 1GB ethernet
echo 'options lnet networks="o2ib0(ib0),tcp0(eth0)"' >> /etc/modprobe.conf

If on infiniband, 10GB and 1GB ethernet
echo 'options lnet networks="o2ib0(ib0),tcp0(eth0),tcp1(eth2)"' >> /etc/modprobe.conf

NOTE registration of the MDT and OSTs with the Lustre message server (MGS) occurs at creation and includes information about available networks. The LNET line should be correct and include all networks before creating filesystem.


-- JamesRobnett - 2011-07-12
Topic revision: r9 - 2016-03-18, JessicaOtey
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback