Install and Configure Lustre Software

---

Lustre Server RPM install

Download the following RPMs, matching your lustre version, OS, and kernel. Source

Intel's public repo

• e2fsprogs
• lustre; lustre-ldisksfs; lustre-modules
• kernel; kernel-devel; kernel-headers

Example list

• • e2fsprogs-1.41.90.wc4-0redhat.x86_64.rpm
  • lustre-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
  • lustre-ldiskfs-3.1.51-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
  • lustre-modules-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
  • kernel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
  • kernel-devel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
  • kernel-headers-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm

Install e2fsprogs on MDS and all OSSes

rpm -Uvh --nodeps e2fsprogs-*

Note this is nearly irreversable. You can not easily return to the stock version, you'll need to boot off of CD to do it

Install Lustre kernel on MDS and all OSSes

rpm -Uvh --force kernel*

Install Lustre RPMS on MDS and all OSSes (ignore WARNINGs)

rpm -Uvh lustre*

---

Lustre Server config

Currently, our OSSes touch all three networks (1GB, 10GB, infiniband). This can be inefficient and someday we hope to install lnet routers that touch all three networks and only have infiniband on the OSSes.

If on the 1GB ethernet

echo 'options lnet networks="tcp0(eth0)"' >> /etc/modprobe.conf

If on the 10GB ethernet

echo 'options lnet networks="tcp1(eth2)"' >> /etc/modprobe.conf

If on infiniband

echo 'options lnet networks="o2ib0(ib0)"' >> /etc/modprobe.conf

Assign IP's within a private network, we use 192.168.1.xxx to ib0 interfaces for IP over IB. NRAO/AOC uses a scheme where:

• The MDS is 192.168.1.10
• The OSSes are 192.168.1.11 - 192.168.1.99
• and clients are 192.168.1.101 - 192.168.1.255

Create ifconfig file

edit /etc/sysconfig/network-scripts/ifcfg-ib0

DEVICE=ib0
BOOTPROTO=static
DHCPCLASS=
ONBOOT=yes
IPADDR=<IP Address>
NETMASK=255.255.255.0
 

Install IB modules

echo "alias ib0 ib_ipoib" >> /etc/modprobe.conf
echo "alias ib0 ib_umad" >> /etc/modprobe.conf

If on infiniband and 1GB ethernet

echo 'options lnet networks="o2ib0(ib0),tcp0(eth0)"' >> /etc/modprobe.conf

If on infiniband, 10GB and 1GB ethernet

echo 'options lnet networks="o2ib0(ib0),tcp0(eth0),tcp1(eth2)"' >> /etc/modprobe.conf

NOTE registration of the MDT and OSTs with the Lustre message server (MGS) occurs at creation and includes information about available networks. The LNET line should be correct and include all networks before creating filesystem.

reboot

-- JamesRobnett - 2011-07-12