You are here: NRAO Public Wiki>HPC Web>LustreFilesystem>ZfsExperiment (2016-04-14, JessicaOtey) Edit wiki text Edit Attach Print version

Experiment with ZFS

1 Good intro doc
2 Install ZFS rpms (source: http://zfsonlinux.org/epel.html)
3 Set storcli disks as JBODS
4 Make a zpool
5 What about IOPS?

1 Good intro doc

I found this write-up helpful

2 Install ZFS rpms (source: http://zfsonlinux.org/epel.html)

# yum localinstall --nogpgcheck https://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
# yum install kernel-devel zfs
### activate kernel module ###
# modprobe zfs

3 Set storcli disks as JBODS

If needed, delete RAID set with storcli /cX/vX del
Enable JBOD mode with storcli /cX set jbod=on
Now you should see your JBODs with storcli /c0 show all | grep -i jbod

4 Make a zpool

4.1 Use parted to show the dev names of the disks

[root@lard-oss-1 ~]# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print devices                                                    
/dev/sda (80.0GB)
/dev/sdb (80.0GB)
/dev/sde (8002GB)
/dev/sdf (8002GB)
/dev/sdg (8002GB)
/dev/sdh (8002GB)
/dev/sdi (8002GB)
/dev/sdj (8002GB)
/dev/sdk (8002GB)
/dev/sdl (8002GB)
/dev/sdm (8002GB)
/dev/sdn (8002GB)
/dev/sdo (8002GB)
/dev/sdp (8002GB)
/dev/sdq (8002GB)
/dev/sdr (8002GB)
/dev/sds (8002GB)
/dev/sdt (8002GB)
/dev/sdu (8002GB)
/dev/sdv (8002GB)
/dev/sdw (8002GB)
/dev/sdx (8002GB)
/dev/sdy (8002GB)
/dev/sdz (8002GB)
/dev/sdaa (8002GB)
/dev/sdab (8002GB)
/dev/sdac (8002GB)
/dev/sdad (8002GB)
/dev/sdae (8002GB)
/dev/sdaf (8002GB)
/dev/sdag (8002GB)
/dev/sdah (8002GB)
(parted) exit

4.2 Create the zpool using those device letters

[root@lard-oss-1 ~]# zpool create -f z2pool raidz2 /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm spare /dev/sdn

the -f flag means force; it overcomes the "/dev/sde does not contain an EFI label but it may contain partition information in the MBR" error
z2pool is the name (and will be the mount point for the pool)
raidz2 is equivalent to RAID6

4.3 See your pool

Now zpool list will show your pool. It has been automatically mounted!

[root@lard-oss-1 ~]# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
z2pool    65T   708K  65.0T         -     0%     0%  1.00x  ONLINE  -

Cooler still, try zpool status

[root@lard-oss-1 ~]# zpool status
  pool: z2pool
 state: ONLINE
  scan: none requested
config:

   NAME        STATE     READ WRITE CKSUM
   z2pool      ONLINE       0     0     0
     raidz2-0  ONLINE       0     0     0
       sde     ONLINE       0     0     0
       sdf     ONLINE       0     0     0
       sdg     ONLINE       0     0     0
       sdh     ONLINE       0     0     0
       sdi     ONLINE       0     0     0
       sdj     ONLINE       0     0     0
       sdk     ONLINE       0     0     0
       sdl     ONLINE       0     0     0
       sdm     ONLINE       0     0     0
   spares
     sdn       AVAIL   

errors: No known data errors

Finally, you can also see your pool with zfs list

[root@lard-oss-1 ~]# zfs list
NAME     USED  AVAIL  REFER  MOUNTPOINT
z2pool   576K  48.0T   219K  /z2pool

4.4 Consistent device IDs via export/import

Taken from: http://kbdone.com/zfs-basics/
The goal is to have the device names persist through reboots and hot-plugging drives.

4.4.1 Solution 1: Import/Export Trick

[root@lard-oss-1 ~]# zpool export z2pool
[root@lard-oss-1 ~]# zpool import z2pool -d /dev/disk/by-id
[root@lard-oss-1 ~]# zpool status
  pool: z2pool
 state: ONLINE
  scan: none requested
config:

   NAME                                  STATE     READ WRITE CKSUM
   z2pool                                ONLINE       0     0     0
     raidz2-0                            ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BEXX  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840C3LP  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BFJW  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BER9  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z8408VK9  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BFMH  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BGYC  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BFV5  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840B55G  ONLINE       0     0     0
   spares
     ata-ST8000AS0002-1NA17Z_Z840BGQG    AVAIL   

errors: No known data errors

This solves the problem but there is a prettier way

4.4.2 Solution 2: Use the vdev_id.conf file

4.4.2.1 Look at the import we just did

[root@lard-oss-1 ~]# ls -al /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 3280 Apr 12 10:09 .
drwxr-xr-x 5 root root  100 Mar 24 10:21 ..
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-INTEL_SSDSC2BB080G6_BTWA519509AF080BGN -> ../../sdb
lrwxrwxrwx 1 root root   10 Mar 24 14:21 ata-INTEL_SSDSC2BB080G6_BTWA519509AF080BGN-part1 -> ../../sdb1
lrwxrwxrwx 1 root root    9 Apr 12 09:37 ata-INTEL_SSDSC2BB080G6_BTWA51950AMD080BGN -> ../../sda
lrwxrwxrwx 1 root root   10 Mar 24 14:21 ata-INTEL_SSDSC2BB080G6_BTWA51950AMD080BGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9 -> ../../sdi
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9-part1 -> ../../sdi1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9-part9 -> ../../sdi9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G -> ../../sdm
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G-part1 -> ../../sdm1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G-part9 -> ../../sdm9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9 -> ../../sdh
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9-part1 -> ../../sdh1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9-part9 -> ../../sdh9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX -> ../../sde
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX-part1 -> ../../sde1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX-part9 -> ../../sde9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW -> ../../sdg
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW-part1 -> ../../sdg1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW-part9 -> ../../sdg9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH -> ../../sdj
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH-part1 -> ../../sdj1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH-part9 -> ../../sdj9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5 -> ../../sdl
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5-part1 -> ../../sdl1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5-part9 -> ../../sdl9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG -> ../../sdn
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG-part1 -> ../../sdn1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG-part9 -> ../../sdn9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC -> ../../sdk
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC-part1 -> ../../sdk1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC-part9 -> ../../sdk9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP -> ../../sdf
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP-part9 -> ../../sdf9
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FAT2 -> ../../sdp
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FGWF -> ../../sdv
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FVR6 -> ../../sdo
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FW08 -> ../../sdah
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FWKJ -> ../../sdab
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FX8E -> ../../sdx
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840G5AR -> ../../sdy
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GNTM -> ../../sdaa
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GNX5 -> ../../sdq
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GPDA -> ../../sds
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GSSX -> ../../sdaf
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GTYL -> ../../sdt
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H1CT -> ../../sdu
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H1HB -> ../../sdae
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H2CY -> ../../sdad
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H2F3 -> ../../sdag
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H2ND -> ../../sdr
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H3DC -> ../../sdw
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H482 -> ../../sdac
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H8J0 -> ../../sdz
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_INTEL_SSDSC2BB0BTWA519509AF080BGN -> ../../sdb
lrwxrwxrwx 1 root root   10 Mar 24 14:21 scsi-SATA_INTEL_SSDSC2BB0BTWA519509AF080BGN-part1 -> ../../sdb1
lrwxrwxrwx 1 root root    9 Apr 12 09:37 scsi-SATA_INTEL_SSDSC2BB0BTWA51950AMD080BGN -> ../../sda
lrwxrwxrwx 1 root root   10 Mar 24 14:21 scsi-SATA_INTEL_SSDSC2BB0BTWA51950AMD080BGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z8408VK9 -> ../../sdi
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z8408VK9-part1 -> ../../sdi1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z8408VK9-part9 -> ../../sdi9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840B55G -> ../../sdm
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840B55G-part1 -> ../../sdm1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840B55G-part9 -> ../../sdm9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BER9 -> ../../sdh
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BER9-part1 -> ../../sdh1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BER9-part9 -> ../../sdh9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BEXX -> ../../sde
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BEXX-part1 -> ../../sde1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BEXX-part9 -> ../../sde9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFJW -> ../../sdg
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFJW-part1 -> ../../sdg1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFJW-part9 -> ../../sdg9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFMH -> ../../sdj
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFMH-part1 -> ../../sdj1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFMH-part9 -> ../../sdj9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFV5 -> ../../sdl
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFV5-part1 -> ../../sdl1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFV5-part9 -> ../../sdl9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGQG -> ../../sdn
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGQG-part1 -> ../../sdn1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGQG-part9 -> ../../sdn9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGYC -> ../../sdk
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGYC-part1 -> ../../sdk1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGYC-part9 -> ../../sdk9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840C3LP -> ../../sdf
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840C3LP-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840C3LP-part9 -> ../../sdf9
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FAT2 -> ../../sdp
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FGWF -> ../../sdv
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FVR6 -> ../../sdo
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FW08 -> ../../sdah
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FWKJ -> ../../sdab
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FX8E -> ../../sdx
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840G5AR -> ../../sdy
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GNTM -> ../../sdaa
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GNX5 -> ../../sdq
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GPDA -> ../../sds
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GSSX -> ../../sdaf
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GTYL -> ../../sdt
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H1CT -> ../../sdu
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H1HB -> ../../sdae
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H2CY -> ../../sdad
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H2F3 -> ../../sdag
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H2ND -> ../../sdr
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H3DC -> ../../sdw
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H482 -> ../../sdac
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H8J0 -> ../../sdz
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50086fb8934 -> ../../sdi
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50086fb8934-part1 -> ../../sdi1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50086fb8934-part9 -> ../../sdi9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087701758 -> ../../sdm
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087701758-part1 -> ../../sdm1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087701758-part9 -> ../../sdm9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008770b199 -> ../../sdk
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b199-part1 -> ../../sdk1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b199-part9 -> ../../sdk9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008770b8ec -> ../../sdn
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b8ec-part1 -> ../../sdn1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b8ec-part9 -> ../../sdn9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087710dd5 -> ../../sdl
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087710dd5-part1 -> ../../sdl1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087710dd5-part9 -> ../../sdl9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008771495b -> ../../sdj
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771495b-part1 -> ../../sdj1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771495b-part9 -> ../../sdj9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087714f65 -> ../../sdg
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087714f65-part1 -> ../../sdg1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087714f65-part9 -> ../../sdg9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008771a369 -> ../../sde
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771a369-part1 -> ../../sde1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771a369-part9 -> ../../sde9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008771b61c -> ../../sdh
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771b61c-part1 -> ../../sdh1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771b61c-part9 -> ../../sdh9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087b48cec -> ../../sdf
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087b48cec-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087b48cec-part9 -> ../../sdf9
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c500909e7db6 -> ../../sdp
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090a8b603 -> ../../sdv
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090d065dd -> ../../sdx
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090d0c6b0 -> ../../sdab
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090d0d857 -> ../../sdah
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090d10772 -> ../../sdo
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090e2e11d -> ../../sdy
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090eb2611 -> ../../sdt
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090eb9c38 -> ../../sdaf
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ed4a43 -> ../../sds
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ed7ae1 -> ../../sdq
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ed7cd0 -> ../../sdaa
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090edb598 -> ../../sdac
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090edea06 -> ../../sdw
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ee2186 -> ../../sdag
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ee24de -> ../../sdad
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ee2a56 -> ../../sdr
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ee792e -> ../../sdu
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ee7b7e -> ../../sdae
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ef49d4 -> ../../sdz
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x55cd2e404c0236eb -> ../../sdb
lrwxrwxrwx 1 root root   10 Mar 24 14:21 wwn-0x55cd2e404c0236eb-part1 -> ../../sdb1
lrwxrwxrwx 1 root root    9 Apr 12 09:37 wwn-0x55cd2e404c023c70 -> ../../sda
lrwxrwxrwx 1 root root   10 Mar 24 14:21 wwn-0x55cd2e404c023c70-part1 -> ../../sda1

Messy, ain't it?

4.4.2.2 Pull out the ids of the relevant drives

[root@lard-oss-1 ~]# ls -al /dev/disk/by-id | grep ata-ST | grep -v part | sort -k11
# Only relevant drives shown here 
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX -> ../../sde
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP -> ../../sdf
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW -> ../../sdg
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9 -> ../../sdh
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9 -> ../../sdi
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH -> ../../sdj
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC -> ../../sdk
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5 -> ../../sdl
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G -> ../../sdm
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG -> ../../sdn

4.4.2.3 You can alias those drive-ids in /etc/zfs/vdev_id.conf

[root@lard-oss-1 ~]# cat /etc/zfs/vdev_id.conf
# /etc/zfs/vdev_id.conf
# make sure to run "udevadm trigger" to update the /dev/disk/by-vdev/ list once each time you change this file!
alias 00    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BEXX
alias 01    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840C3LP
alias 02    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BFJW
alias 03    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BER9
alias 04    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408VK9
alias 05   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BFMH
alias 06   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BGYC
alias 07   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BFV5
alias 08   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840B55G
alias 09   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BGQG

Run udevadm trigger to update the by-vdev list
You can see them now by ls -al /dev/disk/by-vdev

4.4.3 Recreate your pool leveraging your aliased drives

[root@lard-oss-1 ~]# zpool destroy z2pool
[root@lard-oss-1 ~]# zpool create z2pool raidz2 00 01 02 03 04 05 06 07 08 spare 09
[root@lard-oss-1 ~]# zpool status
  pool: z2pool
 state: ONLINE
  scan: none requested
config:

   NAME        STATE     READ WRITE CKSUM
   z2pool      ONLINE       0     0     0
     raidz2-0  ONLINE       0     0     0
       00      ONLINE       0     0     0
       01      ONLINE       0     0     0
       02      ONLINE       0     0     0
       03      ONLINE       0     0     0
       04      ONLINE       0     0     0
       05      ONLINE       0     0     0
       06      ONLINE       0     0     0
       07      ONLINE       0     0     0
       08      ONLINE       0     0     0
   spares
     09        AVAIL   

errors: No known data errors

Neat, ain't it?

4.4.4 Create a filesystem

Creating filesystems is very easy

[root@lard-oss-1 ~]# zfs create z2pool/skylar
[root@lard-oss-1 ~]# zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
z2pool          795K  48.0T   219K  /z2pool
z2pool/skylar   219K  48.0T   219K  /z2pool/skylar
[root@lard-oss-1 ~]# cd /z2pool/skylar/
[root@lard-oss-1 skylar]# ls -al
total 1
drwxr-xr-x 2 root root 2 Apr 12 14:40 .
drwxr-xr-x 3 root root 3 Apr 12 14:40 ..
[root@lard-oss-1 skylar]# mount
/dev/sda1 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/dev/sdd on /mnt/sdd type ext4 (rw)
cvfiler:/vol/vol0/local/redhat on /home/rhlocal type nfs (rw,nosuid,hard,intr,sec=sys,sloppy,addr=10.2.96.95)
z2pool on /z2pool type zfs (rw)   // here is our mounted pool
z2pool/skylar on /z2pool/skylar type zfs (rw) // here is our filesystem

4.4.5 Look at your filesystem's properties

The default record size (the strip on a traditional RAID) is 128 K
You can see properties with zfs get all z2pool/skylar

[root@lard-oss-1 skylar]# zfs get all z2pool/skylar
NAME           PROPERTY              VALUE                  SOURCE
z2pool/skylar  type                  filesystem             -
z2pool/skylar  creation              Tue Apr 12 14:40 2016  -
z2pool/skylar  used                  219K                   -
z2pool/skylar  available             48.0T                  -
z2pool/skylar  referenced            219K                   -
z2pool/skylar  compressratio         1.00x                  -
z2pool/skylar  mounted               yes                    -
z2pool/skylar  quota                 none                   default
z2pool/skylar  reservation           none                   default
z2pool/skylar  recordsize            128K                   default
z2pool/skylar  mountpoint            /z2pool/skylar         default
z2pool/skylar  sharenfs              off                    default
z2pool/skylar  checksum              on                     default
z2pool/skylar  compression           off                    default
z2pool/skylar  atime                 on                     default
z2pool/skylar  devices               on                     default
z2pool/skylar  exec                  on                     default
z2pool/skylar  setuid                on                     default
z2pool/skylar  readonly              off                    default
z2pool/skylar  zoned                 off                    default
z2pool/skylar  snapdir               hidden                 default
z2pool/skylar  aclinherit            restricted             default
z2pool/skylar  canmount              on                     default
z2pool/skylar  xattr                 on                     default
z2pool/skylar  copies                1                      default
z2pool/skylar  version               5                      -
z2pool/skylar  utf8only              off                    -
z2pool/skylar  normalization         none                   -
z2pool/skylar  casesensitivity       sensitive              -
z2pool/skylar  vscan                 off                    default
z2pool/skylar  nbmand                off                    default
z2pool/skylar  sharesmb              off                    default
z2pool/skylar  refquota              none                   default
z2pool/skylar  refreservation        none                   default
z2pool/skylar  primarycache          all                    default
z2pool/skylar  secondarycache        all                    default
z2pool/skylar  usedbysnapshots       0                      -
z2pool/skylar  usedbydataset         219K                   -
z2pool/skylar  usedbychildren        0                      -
z2pool/skylar  usedbyrefreservation  0                      -
z2pool/skylar  logbias               latency                default
z2pool/skylar  dedup                 off                    default
z2pool/skylar  mlslabel              none                   default
z2pool/skylar  sync                  standard               default
z2pool/skylar  refcompressratio      1.00x                  -
z2pool/skylar  written               219K                   -
z2pool/skylar  logicalused           40K                    -
z2pool/skylar  logicalreferenced     40K                    -
z2pool/skylar  filesystem_limit      none                   default
z2pool/skylar  snapshot_limit        none                   default
z2pool/skylar  filesystem_count      none                   default
z2pool/skylar  snapshot_count        none                   default
z2pool/skylar  snapdev               hidden                 default
z2pool/skylar  acltype               off                    default
z2pool/skylar  context               none                   default
z2pool/skylar  fscontext             none                   default
z2pool/skylar  defcontext            none                   default
z2pool/skylar  rootcontext           none                   default
z2pool/skylar  relatime              off                    default
z2pool/skylar  redundant_metadata    all                    default
z2pool/skylar  overlay               off                    default

4.5 Run the iozone test

[root@lard-oss-1 skylar]# /tmp/root/iozone3_434/src/current/iozone -ec -t 1 -r 1M -s 128g -+n -i 0 -i 1
# # # RESULTS # # #
   Run began: Tue Apr 12 14:53:45 2016

   Include fsync in write timing
   Include close in write timing
   Record Size 1024 kB
   File size set to 134217728 kB
   No retest option selected
   Command line used: /tmp/root/iozone3_434/src/current/iozone -ec -t 1 -r 1M -s 128g -+n -i 0 -i 1
   Output is in kBytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 kBytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.
   Throughput test with 1 process
   Each process writes a 134217728 kByte file in 1024 kByte records

   Children see throughput for  1 initial writers    =  894253.75 kB/sec
   Parent sees throughput for  1 initial writers    =  894249.98 kB/sec
   Min throughput per process          =  894253.75 kB/sec 
   Max throughput per process          =  894253.75 kB/sec
   Avg throughput per process          =  894253.75 kB/sec
   Min xfer                = 134217728.00 kB

   Children see throughput for  1 readers       =  779242.50 kB/sec
   Parent sees throughput for  1 readers       =  779239.32 kB/sec
   Min throughput per process          =  779242.50 kB/sec 
   Max throughput per process          =  779242.50 kB/sec
   Avg throughput per process          =  779242.50 kB/sec
   Min xfer                = 134217728.00 kB

iozone test complete.

5 What about IOPS?

z2 on ZFS has IOPS penalties.
We can achieve similar performance only by using mirrored vdevs, but that throws away 50% of our disk space, which we hadn't planned on
It is worthwhile to investigate our IOPS requirements as part of the LARD project

5.1 Bonnie++ on ZFS z2pool (equivalent of HW RAID6)

Bonnie++ Benchmark results

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

lard-oss-1

130000M

453192

109770

555042

175.8

19139

+++++

+++

19591

18125

+++++

+++

19649

Latency

11682us

835ms

282ms

315ms

Latency

24123us

329us

378us

46921us

15us

257us

5.2 Bonnie++ on single-disk ZFS file system

Bonnie++ Benchmark results

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

lard-oss-1

126G

155287

24888

135671

158.6

17729

+++++

+++

18734

17153

+++++

+++

19752

Latency

48146ms

45853ms

640ms

303ms

Latency

23939us

807us

420us

49821us

17us

99us

5.3 Seagate archive disk benchmarks:

seagate archive 8tb sata main 4kwrite throughput.png

5.4 Bonnie++ on HW RAID6

Bonnie++ Benchmark results

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

lard-oss-2

126G

624397

107059

172076

284.9

16549

+++++

+++

+++++

+++

+++++

+++

+++++

+++

+++++

+++

Latency

526ms

3371ms

3259ms

113ms

Latency

104us

673us

295us

84us

15us

29us

5.5 Bonnie++ on single-disk ext4 file system

Bonnie++ Benchmark results

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

lard-oss-1

126G

166882

60675

180482

356.8

543

+++++

+++

14577

21885

+++++

+++

24754

Latency

526ms

562ms

702ms

256ms

Latency

70us

1044us

533us

184us

12us

51us

Topic revision: r5 - 2016-04-14, JessicaOtey

HPC

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback