Naasc Lustre Upgrade 'Toaster' Notes

1 What is the toaster?

  • Our 'toaster' is a rebuilt Supermicro with an OS 6.7 OS drive

2 How to clone drives

2.1 Drive insertion

  • Your cloned drive is inserted and you know which drive letter it's been assigned.
  • alias lssd='lsscsi | grep sd'
  • This knowledge comes from watching what letter appears when you insert it, e.g.:
[root@naasc-toaster ~]# lsscsi | grep -v sda | grep -v sr0 | grep sd*
[6:0:17:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdd 
[6:0:19:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sde 
[6:0:20:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdb 
[root@naasc-toaster ~]# lsscsi | grep -v sda | grep -v sr0 | grep sd*
[6:0:17:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdd 
[6:0:19:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sde 
[6:0:20:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdb 
[7:0:18:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdc 
# new drive is sdc

2.2 What to mount

  • You do NOT need to mount the source to clone it
  • If you want to mount a source drive to look around, mount it read only.
  • Alias for mount read only on naasc-toaster is mountro
  • You DO need to mount the destination drive (rw) to clone to it

2.3 Variables are good for preventing fat finger errors (or reusing commands)

  • pick a DEST_NAME for your new disk, e.g., DEST_NAME==clonefortracy
  • Set variables for the device letters of both the source and destination disks
  • DEST=$clone e.g., /dev/sdc
  • SOURCE=$original e.g., /dev/sdd

2.4 Look at your original with parted

  • parted $SOURCE --script print
  • Note how many data partitions exist on the original disk, as you will need to dump and restore each one
  • Note the file system type of data partition(s)

2.5 Make partitions

  • Copy the part_table of the original disk into file
    • sfdisk -d $SOURCE > ${DEST_NAME}_part_table
  • Clone the part_table onto the new disk
    • sfdisk --force $DEST < ${DEST_NAME}_part_table
  • Take a look: parted $DEST --script print
  • Should match the original, although the file systems won't be there

2.6 Make filesystems

  • Zero out the boot sector on the relevant partition (usually 1)
    • dd if=/dev/zero of=${DEST}1 bs=512 count=1
  • make your filesystem(s):
    • mkfs -t ext4 ${DEST}1
    • mkswap ${DEST}2
    • mkfs -t ext4 ${DEST}3
  • Copy the labels of any data partitions
    • tune2fs -L "/" /${DEST}1
    • tune2fs -L "/export/data_1" /${DEST}3

2.7 Dump command -- use once per data partition

  • Make a mount point for your destination drive
    • mkdir -p /mnt/${DEST_NAME}
  • Mount the first (if more than one) data partition
    • mount ${DEST}1 /mnt/${DEST_NAME}
  • cd into the mount point
    • cd /mnt/${DEST_NAME}
  • Dump each data drive
    • dump -a0f - ${SOURCE}1 | restore -rf -
    • a=autosize; 0 (zero)=start at block 0; f = file, - =stdout; $SOURCE is the SOURCE PARTITION; r=rebuild; f=file; -=stdout
  • Unmount the drive
    • cd
    • umount /mnt/$DEST_NAME
  • Repeat from "Mount the partition" if more than one data drive

2.8 Rebuild the grub

  • The grub utility allows you to install a boot loader onto the cloned drive (which you will need because it is an OS drive)
  • You need to address the disk by its hd number, which follows (slavishly) its lettered order mount point. Meaning:
    • hd0 = sda
    • hd1 = sdb
    • hd2 = sdc
    • hd3 = sdd ... [ETC]
[root@naasc-toaster /]# grub
grub> root (hd1,0)    
root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd1)
setup (hd1)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd1)"...  27 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd1) (hd1)1+27 p (hd1,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.
grub> quit

3 Writing a udev rule to make the internal OS drive come up as /dev/sda every time

  • grub and etc/fstab use the UUID of the internal drive so rebooting is possible without ejecting all drives
  • I made that change after realizing that reboots are very often needed to erase a gap in /dev/sd* letters
  • The icing on the cake would be to ensure that the OS drive is always /dev/sda

3.1 udevinfo on RHEL

  • I've aliased the RHEL command 'udevadm info' to udevinfo, because that's in all the docs
  • Great trick to get the skinny on any device (obviates knowing /sys path): udevinfo -a -p $(udevinfo -q path -n /dev/sd$letter)

3.2 Choose some attributes for your rule. Note that:

  • Udevadm info starts with the device specified by the devpath and then walks up the chain of parent devices.
  • For every device found, it prints all possible attributes in the udev rules key format.
  • A rule to match a device can be composed by the attributes of the device and the attributes from one single parent device.
  • You want to choose attributes which identify your device in a persistent AND human-recognizable way.
  • Disk size and model number are examples of good choices.

[root@naasc-toaster rules.d]# udevinfo -a -p $(udevinfo -q path -n /dev/sde)

  looking at device '/devices/pci0000:00/0000:00:1f.2/host4/target4:0:0/4:0:0:0/block/sde':
    KERNEL=="sde"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{range}=="16"
    ATTR{ext_range}=="256"
    ATTR{removable}=="0"
    ATTR{ro}=="0"
    ATTR{size}=="976773168"
    ATTR{alignment_offset}=="0"
    ATTR{discard_alignment}=="0"
    ATTR{capability}=="52"
    ATTR{stat}=="  167472    48163  6610866   541427   200124   542642  5933320  3148106        0  1472981  3689474"
    ATTR{inflight}=="       0        0"

  looking at parent device '/devices/pci0000:00/0000:00:1f.2/host4/target4:0:0/4:0:0:0':
    KERNELS=="4:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    ATTRS{device_blocked}=="0"
    ATTRS{type}=="0"
    ATTRS{scsi_level}=="6"
    ATTRS{vendor}=="ATA     "
    ATTRS{model}=="WDC WD5000AAKX-0"
    ATTRS{rev}=="1H15"
    ATTRS{state}=="running"
    ATTRS{timeout}=="30"
    ATTRS{eh_timeout}=="10"
    ATTRS{iocounterbits}=="32"
    ATTRS{iorequest_cnt}=="0x66e7b"
    ATTRS{iodone_cnt}=="0x66bd3"
    ATTRS{ioerr_cnt}=="0x24c"
    ATTRS{modalias}=="scsi:t-0x00"
    ATTRS{evt_media_change}=="0"
    ATTRS{evt_inquiry_change_reported}=="0"
    ATTRS{evt_capacity_change_reported}=="0"
    ATTRS{evt_soft_threshold_reached}=="0"
    ATTRS{evt_mode_parameter_change_reported}=="0"
    ATTRS{evt_lun_change_reported}=="0"
    ATTRS{dh_state}=="detached"
    ATTRS{queue_depth}=="31"
    ATTRS{queue_ramp_up_period}=="120000"
    ATTRS{queue_type}=="simple"

  looking at parent device '/devices/pci0000:00/0000:00:1f.2/host4/target4:0:0':
    KERNELS=="target4:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:1f.2/host4':
    KERNELS=="host4"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:1f.2':
    KERNELS=="0000:00:1f.2"
    SUBSYSTEMS=="pci"
    DRIVERS=="ahci"
    ATTRS{vendor}=="0x8086"
    ATTRS{device}=="0x1d02"
    ATTRS{subsystem_vendor}=="0x15d9"
    ATTRS{subsystem_device}=="0x0664"
    ATTRS{class}=="0x010601"
    ATTRS{irq}=="129"
    ATTRS{local_cpus}=="0f"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{modalias}=="pci:v00008086d00001D02sv000015D9sd00000664bc01sc06i01"
    ATTRS{numa_node}=="0"
    ATTRS{enable}=="1"
    ATTRS{broken_parity_status}=="0"
    ATTRS{msi_bus}==""

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

3.3 Once you match your device, then you can specify various attributes (assignment keys/values)

  • In this case, I want to assign the NAME, which is what comes after /dev/
  • Here's my file, which I named 10-early.rules
SUBSYSTEM=="block", SUBSYSTEMS=="scsi", ATTR{size}=="976773168", ATTRS{model}=="WDC WD5000AAKX-0", NAME="sda"

3.4 applying your rule

  • udevadm trigger will apply your rule

3.5 Nice try, but...

  • You can name your internal drive sda till the cows come home, but it is just clobbering the drive letter it was originally assigned
  • This wouldn't be so bad IF that other letter could be reassigned, but it cannot be -- any such rule will just give you an unlabeled device--fun times!
Topic revision: r6 - 2017-01-28, MikeHatz
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback