Experiment with ZFS

1 Good intro doc

2 Install ZFS rpms (source: http://zfsonlinux.org/epel.html)

# yum localinstall --nogpgcheck https://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
# yum install kernel-devel zfs
### activate kernel module ###
# modprobe zfs

3 Set storcli disks as JBODS

  • If needed, delete RAID set with storcli /cX/vX del
  • Enable JBOD mode with storcli /cX set jbod=on
  • Now you should see your JBODs with storcli /c0 show all | grep -i jbod

4 Make a zpool

4.1 Use parted to show the dev names of the disks

[root@lard-oss-1 ~]# parted
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print devices                                                    
/dev/sda (80.0GB)
/dev/sdb (80.0GB)
/dev/sde (8002GB)
/dev/sdf (8002GB)
/dev/sdg (8002GB)
/dev/sdh (8002GB)
/dev/sdi (8002GB)
/dev/sdj (8002GB)
/dev/sdk (8002GB)
/dev/sdl (8002GB)
/dev/sdm (8002GB)
/dev/sdn (8002GB)
/dev/sdo (8002GB)
/dev/sdp (8002GB)
/dev/sdq (8002GB)
/dev/sdr (8002GB)
/dev/sds (8002GB)
/dev/sdt (8002GB)
/dev/sdu (8002GB)
/dev/sdv (8002GB)
/dev/sdw (8002GB)
/dev/sdx (8002GB)
/dev/sdy (8002GB)
/dev/sdz (8002GB)
/dev/sdaa (8002GB)
/dev/sdab (8002GB)
/dev/sdac (8002GB)
/dev/sdad (8002GB)
/dev/sdae (8002GB)
/dev/sdaf (8002GB)
/dev/sdag (8002GB)
/dev/sdah (8002GB)
(parted) exit    

4.2 Create the zpool using those device letters

[root@lard-oss-1 ~]# zpool create -f z2pool raidz2 /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm spare /dev/sdn
  • the -f flag means force; it overcomes the "/dev/sde does not contain an EFI label but it may contain partition information in the MBR" error
  • z2pool is the name (and will be the mount point for the pool)
  • raidz2 is equivalent to RAID6

4.3 See your pool

  • Now zpool list will show your pool. It has been automatically mounted!
[root@lard-oss-1 ~]# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
z2pool    65T   708K  65.0T         -     0%     0%  1.00x  ONLINE  -

  • Cooler still, try zpool status
[root@lard-oss-1 ~]# zpool status
  pool: z2pool
 state: ONLINE
  scan: none requested
config:

   NAME        STATE     READ WRITE CKSUM
   z2pool      ONLINE       0     0     0
     raidz2-0  ONLINE       0     0     0
       sde     ONLINE       0     0     0
       sdf     ONLINE       0     0     0
       sdg     ONLINE       0     0     0
       sdh     ONLINE       0     0     0
       sdi     ONLINE       0     0     0
       sdj     ONLINE       0     0     0
       sdk     ONLINE       0     0     0
       sdl     ONLINE       0     0     0
       sdm     ONLINE       0     0     0
   spares
     sdn       AVAIL   

errors: No known data errors

  • Finally, you can also see your pool with zfs list
[root@lard-oss-1 ~]# zfs list
NAME     USED  AVAIL  REFER  MOUNTPOINT
z2pool   576K  48.0T   219K  /z2pool

4.4 Consistent device IDs via export/import

4.4.1 Solution 1: Import/Export Trick

[root@lard-oss-1 ~]# zpool export z2pool
[root@lard-oss-1 ~]# zpool import z2pool -d /dev/disk/by-id
[root@lard-oss-1 ~]# zpool status
  pool: z2pool
 state: ONLINE
  scan: none requested
config:

   NAME                                  STATE     READ WRITE CKSUM
   z2pool                                ONLINE       0     0     0
     raidz2-0                            ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BEXX  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840C3LP  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BFJW  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BER9  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z8408VK9  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BFMH  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BGYC  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840BFV5  ONLINE       0     0     0
       ata-ST8000AS0002-1NA17Z_Z840B55G  ONLINE       0     0     0
   spares
     ata-ST8000AS0002-1NA17Z_Z840BGQG    AVAIL   

errors: No known data errors
  • This solves the problem but there is a prettier way

4.4.2 Solution 2: Use the vdev_id.conf file

4.4.2.1 Look at the import we just did

[root@lard-oss-1 ~]# ls -al /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 3280 Apr 12 10:09 .
drwxr-xr-x 5 root root  100 Mar 24 10:21 ..
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-INTEL_SSDSC2BB080G6_BTWA519509AF080BGN -> ../../sdb
lrwxrwxrwx 1 root root   10 Mar 24 14:21 ata-INTEL_SSDSC2BB080G6_BTWA519509AF080BGN-part1 -> ../../sdb1
lrwxrwxrwx 1 root root    9 Apr 12 09:37 ata-INTEL_SSDSC2BB080G6_BTWA51950AMD080BGN -> ../../sda
lrwxrwxrwx 1 root root   10 Mar 24 14:21 ata-INTEL_SSDSC2BB080G6_BTWA51950AMD080BGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9 -> ../../sdi
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9-part1 -> ../../sdi1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9-part9 -> ../../sdi9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G -> ../../sdm
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G-part1 -> ../../sdm1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G-part9 -> ../../sdm9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9 -> ../../sdh
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9-part1 -> ../../sdh1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9-part9 -> ../../sdh9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX -> ../../sde
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX-part1 -> ../../sde1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX-part9 -> ../../sde9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW -> ../../sdg
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW-part1 -> ../../sdg1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW-part9 -> ../../sdg9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH -> ../../sdj
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH-part1 -> ../../sdj1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH-part9 -> ../../sdj9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5 -> ../../sdl
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5-part1 -> ../../sdl1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5-part9 -> ../../sdl9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG -> ../../sdn
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG-part1 -> ../../sdn1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG-part9 -> ../../sdn9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC -> ../../sdk
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC-part1 -> ../../sdk1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC-part9 -> ../../sdk9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP -> ../../sdf
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP-part9 -> ../../sdf9
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FAT2 -> ../../sdp
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FGWF -> ../../sdv
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FVR6 -> ../../sdo
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FW08 -> ../../sdah
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FWKJ -> ../../sdab
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840FX8E -> ../../sdx
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840G5AR -> ../../sdy
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GNTM -> ../../sdaa
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GNX5 -> ../../sdq
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GPDA -> ../../sds
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GSSX -> ../../sdaf
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840GTYL -> ../../sdt
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H1CT -> ../../sdu
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H1HB -> ../../sdae
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H2CY -> ../../sdad
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H2F3 -> ../../sdag
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H2ND -> ../../sdr
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H3DC -> ../../sdw
lrwxrwxrwx 1 root root   10 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H482 -> ../../sdac
lrwxrwxrwx 1 root root    9 Apr 12 09:36 ata-ST8000AS0002-1NA17Z_Z840H8J0 -> ../../sdz
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_INTEL_SSDSC2BB0BTWA519509AF080BGN -> ../../sdb
lrwxrwxrwx 1 root root   10 Mar 24 14:21 scsi-SATA_INTEL_SSDSC2BB0BTWA519509AF080BGN-part1 -> ../../sdb1
lrwxrwxrwx 1 root root    9 Apr 12 09:37 scsi-SATA_INTEL_SSDSC2BB0BTWA51950AMD080BGN -> ../../sda
lrwxrwxrwx 1 root root   10 Mar 24 14:21 scsi-SATA_INTEL_SSDSC2BB0BTWA51950AMD080BGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z8408VK9 -> ../../sdi
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z8408VK9-part1 -> ../../sdi1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z8408VK9-part9 -> ../../sdi9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840B55G -> ../../sdm
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840B55G-part1 -> ../../sdm1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840B55G-part9 -> ../../sdm9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BER9 -> ../../sdh
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BER9-part1 -> ../../sdh1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BER9-part9 -> ../../sdh9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BEXX -> ../../sde
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BEXX-part1 -> ../../sde1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BEXX-part9 -> ../../sde9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFJW -> ../../sdg
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFJW-part1 -> ../../sdg1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFJW-part9 -> ../../sdg9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFMH -> ../../sdj
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFMH-part1 -> ../../sdj1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFMH-part9 -> ../../sdj9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFV5 -> ../../sdl
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFV5-part1 -> ../../sdl1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BFV5-part9 -> ../../sdl9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGQG -> ../../sdn
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGQG-part1 -> ../../sdn1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGQG-part9 -> ../../sdn9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGYC -> ../../sdk
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGYC-part1 -> ../../sdk1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840BGYC-part9 -> ../../sdk9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840C3LP -> ../../sdf
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840C3LP-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 scsi-SATA_ST8000AS0002-1N_Z840C3LP-part9 -> ../../sdf9
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FAT2 -> ../../sdp
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FGWF -> ../../sdv
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FVR6 -> ../../sdo
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FW08 -> ../../sdah
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FWKJ -> ../../sdab
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840FX8E -> ../../sdx
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840G5AR -> ../../sdy
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GNTM -> ../../sdaa
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GNX5 -> ../../sdq
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GPDA -> ../../sds
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GSSX -> ../../sdaf
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840GTYL -> ../../sdt
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H1CT -> ../../sdu
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H1HB -> ../../sdae
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H2CY -> ../../sdad
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H2F3 -> ../../sdag
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H2ND -> ../../sdr
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H3DC -> ../../sdw
lrwxrwxrwx 1 root root   10 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H482 -> ../../sdac
lrwxrwxrwx 1 root root    9 Apr 12 09:36 scsi-SATA_ST8000AS0002-1N_Z840H8J0 -> ../../sdz
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50086fb8934 -> ../../sdi
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50086fb8934-part1 -> ../../sdi1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50086fb8934-part9 -> ../../sdi9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087701758 -> ../../sdm
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087701758-part1 -> ../../sdm1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087701758-part9 -> ../../sdm9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008770b199 -> ../../sdk
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b199-part1 -> ../../sdk1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b199-part9 -> ../../sdk9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008770b8ec -> ../../sdn
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b8ec-part1 -> ../../sdn1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008770b8ec-part9 -> ../../sdn9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087710dd5 -> ../../sdl
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087710dd5-part1 -> ../../sdl1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087710dd5-part9 -> ../../sdl9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008771495b -> ../../sdj
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771495b-part1 -> ../../sdj1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771495b-part9 -> ../../sdj9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087714f65 -> ../../sdg
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087714f65-part1 -> ../../sdg1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087714f65-part9 -> ../../sdg9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008771a369 -> ../../sde
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771a369-part1 -> ../../sde1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771a369-part9 -> ../../sde9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c5008771b61c -> ../../sdh
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771b61c-part1 -> ../../sdh1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c5008771b61c-part9 -> ../../sdh9
lrwxrwxrwx 1 root root    9 Apr 12 10:09 wwn-0x5000c50087b48cec -> ../../sdf
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087b48cec-part1 -> ../../sdf1
lrwxrwxrwx 1 root root   10 Apr 12 10:09 wwn-0x5000c50087b48cec-part9 -> ../../sdf9
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c500909e7db6 -> ../../sdp
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090a8b603 -> ../../sdv
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090d065dd -> ../../sdx
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090d0c6b0 -> ../../sdab
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090d0d857 -> ../../sdah
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090d10772 -> ../../sdo
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090e2e11d -> ../../sdy
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090eb2611 -> ../../sdt
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090eb9c38 -> ../../sdaf
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ed4a43 -> ../../sds
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ed7ae1 -> ../../sdq
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ed7cd0 -> ../../sdaa
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090edb598 -> ../../sdac
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090edea06 -> ../../sdw
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ee2186 -> ../../sdag
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ee24de -> ../../sdad
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ee2a56 -> ../../sdr
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ee792e -> ../../sdu
lrwxrwxrwx 1 root root   10 Apr 12 09:36 wwn-0x5000c50090ee7b7e -> ../../sdae
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x5000c50090ef49d4 -> ../../sdz
lrwxrwxrwx 1 root root    9 Apr 12 09:36 wwn-0x55cd2e404c0236eb -> ../../sdb
lrwxrwxrwx 1 root root   10 Mar 24 14:21 wwn-0x55cd2e404c0236eb-part1 -> ../../sdb1
lrwxrwxrwx 1 root root    9 Apr 12 09:37 wwn-0x55cd2e404c023c70 -> ../../sda
lrwxrwxrwx 1 root root   10 Mar 24 14:21 wwn-0x55cd2e404c023c70-part1 -> ../../sda1

  • Messy, ain't it?

4.4.2.2 Pull out the ids of the relevant drives

[root@lard-oss-1 ~]# ls -al /dev/disk/by-id | grep ata-ST | grep -v part | sort -k11
# Only relevant drives shown here 
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BEXX -> ../../sde
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840C3LP -> ../../sdf
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFJW -> ../../sdg
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BER9 -> ../../sdh
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z8408VK9 -> ../../sdi
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFMH -> ../../sdj
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGYC -> ../../sdk
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BFV5 -> ../../sdl
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840B55G -> ../../sdm
lrwxrwxrwx 1 root root    9 Apr 12 10:09 ata-ST8000AS0002-1NA17Z_Z840BGQG -> ../../sdn

4.4.2.3 You can alias those drive-ids in /etc/zfs/vdev_id.conf

[root@lard-oss-1 ~]# cat /etc/zfs/vdev_id.conf
# /etc/zfs/vdev_id.conf
# make sure to run "udevadm trigger" to update the /dev/disk/by-vdev/ list once each time you change this file!
alias 00    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BEXX
alias 01    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840C3LP
alias 02    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BFJW
alias 03    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BER9
alias 04    /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z8408VK9
alias 05   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BFMH
alias 06   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BGYC
alias 07   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BFV5
alias 08   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840B55G
alias 09   /dev/disk/by-id/ata-ST8000AS0002-1NA17Z_Z840BGQG

  • Run udevadm trigger to update the by-vdev list
  • You can see them now by ls -al /dev/disk/by-vdev

4.4.3 Recreate your pool leveraging your aliased drives

[root@lard-oss-1 ~]# zpool destroy z2pool
[root@lard-oss-1 ~]# zpool create z2pool raidz2 00 01 02 03 04 05 06 07 08 spare 09
[root@lard-oss-1 ~]# zpool status
  pool: z2pool
 state: ONLINE
  scan: none requested
config:

   NAME        STATE     READ WRITE CKSUM
   z2pool      ONLINE       0     0     0
     raidz2-0  ONLINE       0     0     0
       00      ONLINE       0     0     0
       01      ONLINE       0     0     0
       02      ONLINE       0     0     0
       03      ONLINE       0     0     0
       04      ONLINE       0     0     0
       05      ONLINE       0     0     0
       06      ONLINE       0     0     0
       07      ONLINE       0     0     0
       08      ONLINE       0     0     0
   spares
     09        AVAIL   

errors: No known data errors
  • Neat, ain't it?

4.4.4 Create a filesystem

  • Creating filesystems is very easy
[root@lard-oss-1 ~]# zfs create z2pool/skylar
[root@lard-oss-1 ~]# zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
z2pool          795K  48.0T   219K  /z2pool
z2pool/skylar   219K  48.0T   219K  /z2pool/skylar
[root@lard-oss-1 ~]# cd /z2pool/skylar/
[root@lard-oss-1 skylar]# ls -al
total 1
drwxr-xr-x 2 root root 2 Apr 12 14:40 .
drwxr-xr-x 3 root root 3 Apr 12 14:40 ..
[root@lard-oss-1 skylar]# mount
/dev/sda1 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/dev/sdd on /mnt/sdd type ext4 (rw)
cvfiler:/vol/vol0/local/redhat on /home/rhlocal type nfs (rw,nosuid,hard,intr,sec=sys,sloppy,addr=10.2.96.95)
z2pool on /z2pool type zfs (rw)   // here is our mounted pool
z2pool/skylar on /z2pool/skylar type zfs (rw) // here is our filesystem

4.4.5 Look at your filesystem's properties

  • The default record size (the strip on a traditional RAID) is 128 K
  • You can see properties with zfs get all z2pool/skylar

[root@lard-oss-1 skylar]# zfs get all z2pool/skylar
NAME           PROPERTY              VALUE                  SOURCE
z2pool/skylar  type                  filesystem             -
z2pool/skylar  creation              Tue Apr 12 14:40 2016  -
z2pool/skylar  used                  219K                   -
z2pool/skylar  available             48.0T                  -
z2pool/skylar  referenced            219K                   -
z2pool/skylar  compressratio         1.00x                  -
z2pool/skylar  mounted               yes                    -
z2pool/skylar  quota                 none                   default
z2pool/skylar  reservation           none                   default
z2pool/skylar  recordsize            128K                   default
z2pool/skylar  mountpoint            /z2pool/skylar         default
z2pool/skylar  sharenfs              off                    default
z2pool/skylar  checksum              on                     default
z2pool/skylar  compression           off                    default
z2pool/skylar  atime                 on                     default
z2pool/skylar  devices               on                     default
z2pool/skylar  exec                  on                     default
z2pool/skylar  setuid                on                     default
z2pool/skylar  readonly              off                    default
z2pool/skylar  zoned                 off                    default
z2pool/skylar  snapdir               hidden                 default
z2pool/skylar  aclinherit            restricted             default
z2pool/skylar  canmount              on                     default
z2pool/skylar  xattr                 on                     default
z2pool/skylar  copies                1                      default
z2pool/skylar  version               5                      -
z2pool/skylar  utf8only              off                    -
z2pool/skylar  normalization         none                   -
z2pool/skylar  casesensitivity       sensitive              -
z2pool/skylar  vscan                 off                    default
z2pool/skylar  nbmand                off                    default
z2pool/skylar  sharesmb              off                    default
z2pool/skylar  refquota              none                   default
z2pool/skylar  refreservation        none                   default
z2pool/skylar  primarycache          all                    default
z2pool/skylar  secondarycache        all                    default
z2pool/skylar  usedbysnapshots       0                      -
z2pool/skylar  usedbydataset         219K                   -
z2pool/skylar  usedbychildren        0                      -
z2pool/skylar  usedbyrefreservation  0                      -
z2pool/skylar  logbias               latency                default
z2pool/skylar  dedup                 off                    default
z2pool/skylar  mlslabel              none                   default
z2pool/skylar  sync                  standard               default
z2pool/skylar  refcompressratio      1.00x                  -
z2pool/skylar  written               219K                   -
z2pool/skylar  logicalused           40K                    -
z2pool/skylar  logicalreferenced     40K                    -
z2pool/skylar  filesystem_limit      none                   default
z2pool/skylar  snapshot_limit        none                   default
z2pool/skylar  filesystem_count      none                   default
z2pool/skylar  snapshot_count        none                   default
z2pool/skylar  snapdev               hidden                 default
z2pool/skylar  acltype               off                    default
z2pool/skylar  context               none                   default
z2pool/skylar  fscontext             none                   default
z2pool/skylar  defcontext            none                   default
z2pool/skylar  rootcontext           none                   default
z2pool/skylar  relatime              off                    default
z2pool/skylar  redundant_metadata    all                    default
z2pool/skylar  overlay               off                    default

4.5 Run the iozone test

[root@lard-oss-1 skylar]# /tmp/root/iozone3_434/src/current/iozone -ec -t 1 -r 1M -s 128g -+n -i 0 -i 1
# # # RESULTS # # #
   Run began: Tue Apr 12 14:53:45 2016

   Include fsync in write timing
   Include close in write timing
   Record Size 1024 kB
   File size set to 134217728 kB
   No retest option selected
   Command line used: /tmp/root/iozone3_434/src/current/iozone -ec -t 1 -r 1M -s 128g -+n -i 0 -i 1
   Output is in kBytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 kBytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.
   Throughput test with 1 process
   Each process writes a 134217728 kByte file in 1024 kByte records

   Children see throughput for  1 initial writers    =  894253.75 kB/sec
   Parent sees throughput for  1 initial writers    =  894249.98 kB/sec
   Min throughput per process          =  894253.75 kB/sec 
   Max throughput per process          =  894253.75 kB/sec
   Avg throughput per process          =  894253.75 kB/sec
   Min xfer                = 134217728.00 kB

   Children see throughput for  1 readers       =  779242.50 kB/sec
   Parent sees throughput for  1 readers       =  779239.32 kB/sec
   Min throughput per process          =  779242.50 kB/sec 
   Max throughput per process          =  779242.50 kB/sec
   Avg throughput per process          =  779242.50 kB/sec
   Min xfer                = 134217728.00 kB

iozone test complete.

5 What about IOPS?

  • z2 on ZFS has IOPS penalties.
  • We can achieve similar performance only by using mirrored vdevs, but that throws away 50% of our disk space, which we hadn't planned on
  • It is worthwhile to investigate our IOPS requirements as part of the LARD project

5.1 Bonnie++ on ZFS z2pool (equivalent of HW RAID6)

Bonnie++ Benchmark results
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
lard-oss-1130000M453192861097702355504244175.811161913993++++++++19591991812598++++++++1964998
Latency11682us835ms282ms315msLatency24123us329us378us46921us15us257us

5.2 Bonnie++ on single-disk ZFS file system

Bonnie++ Benchmark results
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
lard-oss-1126G155287402488851356719158.69161772991++++++++18734991715399++++++++1975299
Latency48146ms45853ms640ms303msLatency23939us807us420us49821us17us99us

5.3 Seagate archive disk benchmarks:

seagate archive 8tb sata main 4kwrite throughput.png

5.4 Bonnie++ on HW RAID6

Bonnie++ Benchmark results
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
lard-oss-2126G62439771107059101720767284.940161654916++++++++++++++++++++++++++++++++++++++++
Latency526ms3371ms3259ms113msLatency104us673us295us84us15us29us

5.5 Bonnie++ on single-disk ext4 file system

Bonnie++ Benchmark results
Version 1.96Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
lard-oss-1126G1668821860675518048210356.861654321++++++++14577242188536++++++++2475434
Latency526ms562ms702ms256msLatency70us1044us533us184us12us51us

Topic revision: r5 - 2016-04-14, JessicaOtey
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback