July Lustre Installation steps

Prior to Tuesday July 22nd

• USNO Tasks
  • Identify Lustre MDS server: separately ordered Dell with 3 Hard Drives otherwise similar to 32 nodes DONE
  • Identify Lustre OSS servers: 3 4U systems shipped from Aeon Computing DONE
  • Locate 2U APC AP7930 110V 20A zeroU PDU DONE
  • If possible move rack to it's temporary position
  • Locate box of 10Gbit PCI-E Solarflare cards need QTY 4 DONE
  • Locate box of Y power cords need QTY 4 3foot cords DONE
  • Locate KVM ATEN KL1516AI (ALTUSCN) DONE
  • Locate KVM VGA/USB to UTP dongles DONE
  • Locate QTY 8 10Gbit MMF SFPs DONE

• NRAO tasks
  • Order new network cables DONE
  • Pre-configure Lustre MDS boot disk and ship DONE
  • Configure 1Gbit/10Gbit switch ports for Lustre servers DONE
  • Print following pages for reference
    • https://safe.nrao.edu/wiki/bin/view/HPC/UsnoDifxLustreInstall
    • https://safe.nrao.edu/wiki/bin/view/HPC/UsnoDifxKVM#ALTUSCN_40usno_45kvm_452_41
  • Email Mike and Brittany port numbers for PDU DONE
  • usno-pdu-lustre 10.1.34.97
    • usno-kvm-1 Outlet #17
    • usno-oss-3 Outlet #15
    • usno-oss-2 Outlet #13
    • usno-oss-1 Outlet #3
    • usno-mds-1 Outlet #1

July 22nd

• NRAO tasks physical installation
  • Quickly check that fiber/UTP cables will reach from existing rack to bottom of Lustre rack.
  • Install drawer in bottom of rack (examine unit in existing rack for height)
  • Install 10gbit PCI-e cards in MDS and OSSes (probably easier prior to rack installation)
  • Physically Install Lustre MDS above drawer
    • Replace system disk with preloaded boot disk shipped to USNO
  • Physically install 3 Lustre OSSes above MDS
  • Physically install KVM near center of rack (compare to existing rack for height)
  • Physically install USNO supplied UPS above OSSes, where ever it will fit (may already done)
  • Install APC PDU (Jeffmay need to help with explanation for installing)
    • Can be installed on left or right, place on side nearest OSS power supplies
  • Plug PDU into UPS via L5-30P/R 30 Amp connectors
  • Connect gbit UTP cables from MDS, OSSes and PDU to Catalyst 2960s gbit switch in correlator rack as follows
    • MDS port 0/36
    • OSS 1 through 3 ports 0/37 through 0/39 sequentially
    • PDU port 0/41
    • KVM port 0/42
    • note port 40 intentionally left blank, currently ports 36 through 42 are currently unused
  • Insert 8 MMF SFPs in Arista 7050 in main correlator switch ports 34 through 41
  • Connect 10Gbit fibers, 2 per server as follows
    • MDS to 10Gbit Arista ports 34 & 35
    • OSS to 10Gbit Arista ports 36 & 37, 38 & 39, 40 & 41, two ports per OSS.
  • Connect MDS to PDU via Y cords
  • Connect OSSes to PDU via Y cords
  • Connect MDS and OSSes to KVM
    • Connect VGA/USB to UTP dongle to each node
    • Connect 5' cable from dongle to KVM, MDS first port then OSS1 through 3

• NRAO tasks logical installation (requires, power, network and KVM above completed)
  • Power on PDU configure IP address (10.1.34.97) per Jeff Long instructions
    • Test pinging from master node
  • Power on KVM configure IP address per K.Scott instruction https://safe.nrao.edu/wiki/bin/view/HPC/UsnoDifxKVM#ALTUSCN_40usno_45kvm_452_41
  • Power on MDS, enter BIOS
    • Disable C1/E states
    • Disable Hyperthreading
    • Write down lowest order Gbit Mac Address
  • Boot MDS, enter single user mode
    • edit /etc/sysconfig/network-scripts/ifcfg-eth0 and insert MAC Addr
    • Test pinging usno-serv-1
    • See that /home/usno mounted
  • Boot OSS, enter BIOS
    • Disable C1/E states
    • Disable Hyperthreading
  • Boot OSS
    • Change root password to agreed temp password
    • Edit /etc/sysconfig/network-scripts/ifcfg-
      • change to static
      • IP: 10.1.34.{101,102,103} for oss-{1,2,3} respectively
      • Gateway: 10.1.34.1
      • Netmask: 255.255.255.0
    • Testing pinging usno-serv-1 by address 10.1.34.2 (no name resolution at this point)

• NRAO tasks testing
  • Demonstrate remote SSH to MDS and OSS
  • Demonstrate remote control of PDU
  • Demonstrate remote access through KVM

CV Packing List / Miscellaneous

• VA Drivers License AND US Passport
• Tie wraps - vinyl and velcro
• Leathernan & granny glasses (mhatz)
• 2 x 4-in-1 screw drivers
• Spare rack screws, nuts, rack fasteners
• Super thin regular tip screwdriver or rack fastener removal tool
• Snips
• Pliers
• Sharpies
• Clipboards
• Dymo label maker
• Spare ethernet cables / patch length cables
• Cable tags - see Derek or Dale
• All kinds of random length power cords, just in case...
• [X] USB <--> RS-232 - tested with Derek. Laptop with a Serial R232 port? You'll need one for the configuration of the switched pdu. If not, we have been using a usb serial adapter from TrippLite model:USA-19HS
• If relevant - paper printouts of all relevant wiki pages. Do not assume network connectivity!
• Verify hard drive sizes and pack spares.
• Verify cables have been shipped to USNO from NRAO.

Post Activity Notes

• The MDS / OSS rack's front door would not close. The catch was adjusted and the door closes effortlessly now.
• UPS was not installed prior to our arrival. A custom aluminum shelf was fabricated on site. The UPS was installed in the top of the rack and fully charged itself during the evening of 22 July.
• The length of the power cords to the APC PDU was in sufficient. We had to change the mapping of the ports. The ports were also edited via the CLI.

1. Port 24 - MDS
2. Port 23 - bottom OSS - OSS1
3. Port 22 - middle OSS - OSS2
4. Port 21 - empty
5. Port 20 - KVM - USNO-KVM-1
6. -------------------------------
7. Port 12 - top oss - OSS3

• We had some issues with concurrent access on the KVM. When the console operator took control of the session, the remote user (James) was "blacked out" on his video.
• The rack had to be rocked "candy machine style" to get the main power cord routed into the floor tile cut-out.
• KVM was mounted much lower in the rack to address ergonomic considerations.
• Ethernet and Fiber runs were across the tops of the racks as is consistent with the other temporary infrastructure accommodations.