Lustre Disk Performance Tests

Performance tests listed here should be done prior to creating actual Lustre filesystems. The tests are either destructive, dd and its variants, or must be used with raw filesystems rather than Lustre filesystems.

The tests assume raid arrays have already been created.

---

System config options

These options can be set and tests below re-performed as needed.

Change read ahead

Use the blockdev command to set the read ahead for each device to 8MB (these settings are not persistent on reboot, add to /etc/rc.local to keep)
blockdev --setra 16384 /dev/sdb

• Setting proper alignment when creating the filesystem can improve performance
  • Need to set the stride and stripe-width values to match filesystem to raid geometry where:
    • stride= Raid chunk size (64k) / filesystem block size (4k) = 16
    • stripe-width= stride (16) * number of data disks (4) = 6
  • mke4fs -j -m0 -E stride=16 -E stripe-width=64 /dev/sd{b.c.d.e}

---

Raw disk tests

xdd was available from http://www.ioperformance.com/ but no longer. The NRAO has a copy in /home/src/xdd. Xdd is a destructive test, so please bear this in mind before running it.

Compile xdd

mkdir /tmp/root
cd /tmp/root
tar xfvz /home/src/xdd/xdd65.013007.tgz
cd xdd65.013007
make -f linux.makefile xdd
cd bin

Run xdd against target arrays

A typical run would be
xdd.linux -op [read|write] -targets 1 /dev/<device> -queuedepth 1 -blocksize 512 -reqsize 2048 -mbytes 2*Memory-passes 2 -verbose

• -reqsize set to 2048 to mimic 1MB Lustre I/O's
• -passes set to 2 to verify repeatability
• -mbytes set to 2*Memory to avoid cache. 16384 in our case with 8GB of memory.
• Run with read and then with write for each decive
• Example:

  xdd.linux -op read -targets 1 /dev/sdb -queuedepth 1 -blocksize 512 -reqsize 2048 -mbytes 16384 -passes 2 -verbose

• Results should be in the vicinity of 400MB/s per device
  • MSIMMOND 2: compiled the source on JAO-OSS-1
  • MSIMMOND 3: run xdd as:

    xdd -op write -targets 1 /dev/sdb -queuedepth 1 -blocksize 512 -reqsize 2048 -mbytes 16384 -passes 2 -verbose

  • MSIMMOND 4.1: results from JAO-OSS-1
  • MSIMMOND 4.2: results from JAO-OSS-2

---

Filesystem tests

Create ext4 filesystems (see proper alignment note System Config Options above if you want to imitate Lustre's base I/O)

For each device (e.g. sdb, sdc, sdd, sde), make a filesystem. There is a minor performance gain from alignment by making the entire disk into a filesystm, ie don't partition the disk, simply write to the raw /dev/sd{b,c,d,e} device.
mke4fs -m0 -j /dev/<device>
Then mount the device someplace temporary
mkdir /tmp/<device>
mount -t ext4 /dev/<device> /mnt/<device>

Perform Tests

Install and run iozone (http://www.iozone.org/) and/or bonnie++ (http://www.coker.com.au/bonnie++) against the created filesystems. Prebuilt binaries can be found at
/home/apathy/iozone3_311/src/current/iozone
/users/krowe/bin/x86_64/Linux/bonnie++

iozone

Run something like
cd /mnt/<device> ; iozone -ec -t 1 -r 1M -s <2*Mem>g -+n -i 0 -i 1

• -ec clears cache and buffers between tests
• -t 1 runs 1 thread
• -r 1M does 1MB I/Os
• -s <2*Mem>g is 2 times memory, 16g in our case.
• -+n for no retests
• -i 0 -i 1 to do write and read tests only
• By default, iozone writes to the current working directory
• Example:

  cd /mnt/sdb ; iozone -ec -t 1 -r 1M -s 16g -+n -i 0 -i 1

bonnie++

run something like
=bonnie++ {-u 101} -f -s<2*Mem>g -d <path> =

• -u 101 switches to a different user if root, directory must be world writable
• -f does fast tests, no per char I/O
• -s<2*Mem>g is 2 times memory, 16g in our case to avoid cache hits
• -d {path} is path to filesystem, could be '.'
• Example:

  bonnie++ -u 101 -f -s16g -d /mnt/sdb

• Results should be in the 350-400MB/s range

-- JamesRobnett - 2011-07-12