Naasc Lustre Upgrade 'Toaster' Notes

Show TOCHide TOC

1 What is the toaster?

• Our 'toaster' is a rebuilt Supermicro with an OS 6.7 OS drive

2 How to clone drives

2.1 Drive insertion

• Your cloned drive is inserted and you know which drive letter it's been assigned.
• alias lssd='lsscsi | grep sd'
• This knowledge comes from watching what letter appears when you insert it, e.g.:

[root@naasc-toaster ~]# lsscsi | grep -v sda | grep -v sr0 | grep sd*
[6:0:17:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdd 
[6:0:19:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sde 
[6:0:20:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdb 
[root@naasc-toaster ~]# lsscsi | grep -v sda | grep -v sr0 | grep sd*
[6:0:17:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdd 
[6:0:19:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sde 
[6:0:20:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdb 
[7:0:18:0]   disk    ATA      WDC WD5003ABYZ-0 1S03  /dev/sdc 
# new drive is sdc

2.2 What to mount

• You do NOT need to mount the source to clone it
• If you want to mount a source drive to look around, mount it read only.
• Alias for mount read only on naasc-toaster is mountro
• You DO need to mount the destination drive (rw) to clone to it

2.3 Variables are good for preventing fat finger errors (or reusing commands)

• pick a DEST_NAME for your new disk, e.g., DEST_NAME==clonefortracy
• Set variables for the device letters of both the source and destination disks
• DEST=$clone e.g., /dev/sdc
• SOURCE=$original e.g., /dev/sdd

2.4 Look at your original with parted

• parted $SOURCE --script print
• Note how many data partitions exist on the original disk, as you will need to dump and restore each one
• Note the file system type of data partition(s)

2.5 Make partitions

• Copy the part_table of the original disk into file
  • sfdisk -d $SOURCE > ${DEST_NAME}_part_table
• Clone the part_table onto the new disk
  • sfdisk --force $DEST < ${DEST_NAME}_part_table
• Take a look: parted $DEST --script print
• Should match the original, although the file systems won't be there

2.6 Make filesystems

• Zero out the boot sector on the relevant partition (usually 1)
  • dd if=/dev/zero of=${DEST}1 bs=512 count=1
• make your filesystem(s):
  • mkfs -t ext4 ${DEST}1
  • mkswap ${DEST}2
  • mkfs -t ext4 ${DEST}3
• Copy the labels of any data partitions
  • tune2fs -L "/" /${DEST}1
  • tune2fs -L "/export/data_1" /${DEST}3

2.7 Dump command -- use once per data partition

• Make a mount point for your destination drive
  • mkdir -p /mnt/${DEST_NAME}
• Mount the first (if more than one) data partition
  • mount ${DEST}1 /mnt/${DEST_NAME}
• cd into the mount point
  • cd /mnt/${DEST_NAME}
• Dump each data drive
  • dump -a0f - ${SOURCE}1 | restore -rf -
  • a=autosize; 0 (zero)=start at block 0; f = file, - =stdout; $SOURCE is the SOURCE PARTITION; r=rebuild; f=file; -=stdout
• Unmount the drive
  • cd
  • umount /mnt/$DEST_NAME
• Repeat from "Mount the partition" if more than one data drive

2.8 Rebuild the grub

• The grub utility allows you to install a boot loader onto the cloned drive (which you will need because it is an OS drive)
• You need to address the disk by its hd number, which follows (slavishly) its lettered order mount point. Meaning:
  • hd0 = sda
  • hd1 = sdb
  • hd2 = sdc
  • hd3 = sdd ... [ETC]

[root@naasc-toaster /]# grub
grub> root (hd1,0)    
root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd1)
setup (hd1)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd1)"...  27 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd1) (hd1)1+27 p (hd1,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.
grub> quit

3 Writing a udev rule to make the internal OS drive come up as /dev/sda every time

• grub and etc/fstab use the UUID of the internal drive so rebooting is possible without ejecting all drives
• I made that change after realizing that reboots are very often needed to erase a gap in /dev/sd* letters
• The icing on the cake would be to ensure that the OS drive is always /dev/sda

3.1 udevinfo on RHEL

• I've aliased the RHEL command 'udevadm info' to udevinfo, because that's in all the docs
• Great trick to get the skinny on any device (obviates knowing /sys path): udevinfo -a -p $(udevinfo -q path -n /dev/sd$letter)

3.2 Choose some attributes for your rule. Note that:

• Udevadm info starts with the device specified by the devpath and then walks up the chain of parent devices.
• For every device found, it prints all possible attributes in the udev rules key format.
• A rule to match a device can be composed by the attributes of the device and the attributes from one single parent device.
• You want to choose attributes which identify your device in a persistent AND human-recognizable way.
• Disk size and model number are examples of good choices.

Show udevinfo outputHide udevinfo output

[root@naasc-toaster rules.d]# udevinfo -a -p $(udevinfo -q path -n /dev/sde)

  looking at device '/devices/pci0000:00/0000:00:1f.2/host4/target4:0:0/4:0:0:0/block/sde':
    KERNEL=="sde"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{range}=="16"
    ATTR{ext_range}=="256"
    ATTR{removable}=="0"
    ATTR{ro}=="0"
    ATTR{size}=="976773168"
    ATTR{alignment_offset}=="0"
    ATTR{discard_alignment}=="0"
    ATTR{capability}=="52"
    ATTR{stat}=="  167472    48163  6610866   541427   200124   542642  5933320  3148106        0  1472981  3689474"
    ATTR{inflight}=="       0        0"

  looking at parent device '/devices/pci0000:00/0000:00:1f.2/host4/target4:0:0/4:0:0:0':
    KERNELS=="4:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    ATTRS{device_blocked}=="0"
    ATTRS{type}=="0"
    ATTRS{scsi_level}=="6"
    ATTRS{vendor}=="ATA     "
    ATTRS{model}=="WDC WD5000AAKX-0"
    ATTRS{rev}=="1H15"
    ATTRS{state}=="running"
    ATTRS{timeout}=="30"
    ATTRS{eh_timeout}=="10"
    ATTRS{iocounterbits}=="32"
    ATTRS{iorequest_cnt}=="0x66e7b"
    ATTRS{iodone_cnt}=="0x66bd3"
    ATTRS{ioerr_cnt}=="0x24c"
    ATTRS{modalias}=="scsi:t-0x00"
    ATTRS{evt_media_change}=="0"
    ATTRS{evt_inquiry_change_reported}=="0"
    ATTRS{evt_capacity_change_reported}=="0"
    ATTRS{evt_soft_threshold_reached}=="0"
    ATTRS{evt_mode_parameter_change_reported}=="0"
    ATTRS{evt_lun_change_reported}=="0"
    ATTRS{dh_state}=="detached"
    ATTRS{queue_depth}=="31"
    ATTRS{queue_ramp_up_period}=="120000"
    ATTRS{queue_type}=="simple"

  looking at parent device '/devices/pci0000:00/0000:00:1f.2/host4/target4:0:0':
    KERNELS=="target4:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:1f.2/host4':
    KERNELS=="host4"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:1f.2':
    KERNELS=="0000:00:1f.2"
    SUBSYSTEMS=="pci"
    DRIVERS=="ahci"
    ATTRS{vendor}=="0x8086"
    ATTRS{device}=="0x1d02"
    ATTRS{subsystem_vendor}=="0x15d9"
    ATTRS{subsystem_device}=="0x0664"
    ATTRS{class}=="0x010601"
    ATTRS{irq}=="129"
    ATTRS{local_cpus}=="0f"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{modalias}=="pci:v00008086d00001D02sv000015D9sd00000664bc01sc06i01"
    ATTRS{numa_node}=="0"
    ATTRS{enable}=="1"
    ATTRS{broken_parity_status}=="0"
    ATTRS{msi_bus}==""

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

3.3 Once you match your device, then you can specify various attributes (assignment keys/values)

• In this case, I want to assign the NAME, which is what comes after /dev/
• Here's my file, which I named 10-early.rules

SUBSYSTEM=="block", SUBSYSTEMS=="scsi", ATTR{size}=="976773168", ATTRS{model}=="WDC WD5000AAKX-0", NAME="sda"

3.4 applying your rule

• udevadm trigger will apply your rule

3.5 Nice try, but...

• You can name your internal drive sda till the cows come home, but it is just clobbering the drive letter it was originally assigned
• This wouldn't be so bad IF that other letter could be reassigned, but it cannot be -- any such rule will just give you an unlabeled device--fun times!