Lustre Filesystem Creation

Documentation for creating MDS and OSS filesystems. Ensure the following steps have been done before proceeding
  • Operating systems installed on MDS and OSSes
  • Network, including Infiniband is installed and tested
  • Destructive raw performance tests have been completed
  • Lustre RPMS have been installed and modules loaded (modprobe lustre)


MDS Filesystem

Following operations should be performed on the MDS

  • The MDT (metadata target) stores all metadata for the lustre array.
    • Since it's a single point of failure consider making it a software raid 1.
    • mdadm --create /dev/md0 -l 1 -n 2 -x 0 /dev/sdb /dev/sdc
  • Consider selecting a name other than the default of 'lustre' for the filesystem, especially if you plan on having multiple filesystems. The name has an 8 character limit.
  • Make and Mount Volumes
    • Make MDT Lustre filesytem:
      • mkfs.lustre --fsname lustre --mdt --mgs /dev/md0
      • --fsname name is the filesystem name
      • --mdt creates the metadata target
      • --mgs makes the MDS act as message server as well
  • Create mount point and mount filesystem. (servers mount everything at /export/lustre)
    • mkdir -p /export/lustre/mdt
    • mount -t lustre /dev/md0 /export/lustre/mdt
  • Add mount to /etc/fstab
    • echo "/dev/md0 /export/lustre/mdt lustre defaults 0 0" >> /etc/fstab

Usefull mdadm (software raid) commands
  • To fail a disk
    mdadm --fail /dev/md0 /dev/sdb
  • To add a disk, for example after failing one
    mdadm --add /dev/md0 /dev/sdb
  • To monitor RAID rebuild process and current status
    cat /proc/mdstat


OST filesystems

The following operations should be performed for each OST on each OSS. OSTs are registered in the order first mounted, so create all the filesystems and mount one at a time on OSS-1 and then all the filesystems on OSS-2, etc. I'd suggest doing them in order from lowest SCSI target to highest such that sdb maps to OST0001, sdc to OOST0002 etc.

Create Filesystems
For each OST on each OSS, execute something like the following

mkfs.lustre --fsname name --mkfsoptions="-E stride=16,stripe-width=64 -m 0" --ost --mgsnode=MDSIP@o2ib0 /dev/device

  • name is the name of the filesystem, preferrably not 'lustre'.
  • stride= Raid chunk size (64k) / filesystem block size (4k) = 16
  • stripe-width= stride (16) * number of data disks (4) = 64
  • MDSIP is the IP address of the MDS server's Infiniband interface
  • device is each raid array on the OSS: sdb, sdc, etc.

Create mount points
Make directories like /export/lustre/ostX-Y where X is the number of the OSS server and Y is the enumerated OST on that server. For instance, /export/lustre/ost1-1 is the first OST on the 1st server, /export/lustre/ost2-3 is the 3rd OST on the 2nd server.

Add mount points to /etc/fstab
for example on first OSS
/dev/sdb                /export/lustre/ost1-1   lustre  defaults        0 0
/dev/sdc                /export/lustre/ost1-2   lustre  defaults        0 0
/dev/sdd                /export/lustre/ost1-3   lustre  defaults        0 0
/dev/sde                /export/lustre/ost1-4   lustre  defaults        0 0


Topic revision: r9 - 2012-10-08, DanielKlopp
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback