Disk Layout

Server Disk Partitioning

The disk layout is shown below as displayed using lsblk. The two disks are partitioned somewhat in parallel although the first disk (sda) contains the the actual boot partitions while the second disk just has them to stay in sync. Two of the partitions on each disk are used to form a RAID level one disk. The partitions sda2 and sdb2 are used to form the md0 raid disk that is mounted as the root of the system file system; the RAID system 64GB of storage on each physical disk. The partitions sda4 and sdb5 form the md1 RAID disk; this device contains ~860 GB of storage and is mounted on /export.

# lsblk

sda 8:0 0 931.5G 0 disk    
├─sda1 8:1 0 256M 0 part /boot/efi UEFI boot partition
├─sda2 8:2 0 1G 0 part /boot Boot partition; system images, etc. are here.
├─sda3 8:3 0 64G 0 part    
│ └─md0 9:0 0 64G 0 raid1 / System root
├─sda4 8:4 0 16G 0 part /tmp DoD STIG requires /tmp to have its own partition.
├─sda5 8:5 0 16G 0 part /var DoD STIG requires /var to have its own parition.
├─sda6 8:6 0 8G 0 part [SWAP]  
└─sda7 8:7 0 826.3G 0 part    
└─md1 9:1 0 826.1G 0 raid1 /export DiFX Software and user home directories live here.
sdb 8:16 0 931.5G 0 disk    
├─sdb1 8:17 0 64G 0 part    
│ └─md0 9:0 0 64G 0 raid1 /  
├─sdb2 8:18 0 16G 0 part /tmpSpare Spare of /tmp partition on other disk.
├─sdb3 8:19 0 16G 0 part /varSpare Spare of /var partition on other disk.
├─sdb4 8:20 0 8G 0 part [SWAP]  
├─sdb5 8:21 0 1G 0 part /bootSpare Spare of /boot on other disk.
├─sdb6 8:22 0 256M 0 part /bootSpare/efiSpare Spare of /boot/efi on other disk.
└─sdb7 8:23 0 826.3G 0 part    
└─md1 9:1 0 826.1G 0 raid1 /export  

Server directory layout

TFTP Directory

Bootup of the diskless systems requires the use of the TFTP daemon. This daemon has a restricted root so that it may only access files located in /opt/services/tftpboot. Some miscellaneous files are contained in that directory in addition to the files used to boot up the diskless systems. Those directories are named to correspond to the diskless images (see diskless images section below). Below the directory named after the diskless system image (e.g., RHEL- is one named swc. This directory contains the files needed to start the UEFI bootstrap of the SWCs:

The configuration provided to the GRUB bootloader on the SWC. The one in pxelinux.cfg is not apprently used.

Initial file system used in bootup (the kernel version number is elideed)'

The kernel file used in bootup (the kernel version number is elideed)'

A signed(?) file used to start up the secure UEFI boot process.

Diskless Images and Host-Specific Snapshots

The diskless images are stored under /opt/services/diskless_boot. Several versions of diskless images will be stored there and they are named RHEL-7.a.b.c where a is the RHEL-7 minor version, while b and c are used by NRAO to identify the patch version of the particular image version. The intent is that b will be incremented when a notable modification is made to the image and c is incremented each time a new image is produced which will be whenever the system image is patched.

Below the system image directory there are three subdirectories: etc, swc and var. The most important one is swc; swc has two subdirectories root and snapshot. The root directory is the root, /, of the system image provided to the SWCs. The snapshot directory is used to hold host-specific files.

Host Specific Snapshots

For each host requiring host specific files there will be a directory named using the host's fully DNS named (e.g., swc-001.usno.nrao.edu). Below the host-named directory are the directories required to reach the host-specific files. For example, for swc-001 the layout is:

    |-- firewalld
    |   |-- zones
    |       |--swc-001-external.xml
    |-- sysconfig
        |-- network-scripts
            |-- ifcfg-em4

In addition to the directories leading towards host-specific files, the host-named directory (swc-001t.usno.nrao.edu) also contains a file named files. This file contains the aboslute paths to any files that are host-specific; for swc-001t that file contains:



In addition, the file /etc/sysconfig/readonly-root is used to make the readonly diskless image usable by the SWCs. It causes them to put system files which are normally written to, into a tmpfs type partition. Below are the contents of the file on server-1t as of 1/21/20:

# Set to 'yes' to mount the system filesystems read-only.
# NOTE: It's necessary to append 'ro' to mount options of '/' mount point in
#       /etc/fstab as well, otherwise the READONLY option will not work.
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
# Place to put a tmpfs for temporary scratch writable space
# Label on local filesystem which can be used for temporary scratch space
# Options to use for temporary mount
# Label for partition with persistent data
# Where to mount to the persistent data
# Options to use for persistent mount
# NFS server to use for persistent data?
# Use slave bind-mounts

Important Directories

Server boot files

The server system boot files. These include kernel-versioned files holding: initial RAM filesystem, kernel, various configuration files. Usually there will be serveral kernel versions plus a rescue image.
Root of the efi partition used by UEFI boot.
Grub files to enable UEFI boot. In particular the file grub.cfg is used to generate the GRUB menu transiently displayed during bootup.

System configuration files

Standard system configuration tree root.
Files used to configure bash logins.
Files used to set up some aspects of the networks. Other network configuration is performed as a systemd service (see below).
systemd system configuraiton files are located here. Nonstandard .service files are located here. Standard (unmodified) service files are located in /usr/lib/systemd/system. Services are enabled/disabled using the systemctl command; these often modifies the contents of some of this folders subdirectories by placing links to enabled serves and removing those when services are disabled.

Difx, User home directories and Patches

This directory is the root of the second RAID 1 partition and contains the bulk of the storage on the server's disks. This directory is normally mounted by the diskless systems to enabled DiFX operation as well as access to user home directories.
This rather long path was kept (somewhat mindlessly) from an earlier installation.
The root of the DiFX installation.
Most user home directories are located below this directory; this includes the homes of USNO folks having correlator accounts.
The admin account's home directory lives here.
The difxmgr account's home directory lives here.
The oper account's home directory lives here.

Patch Data Directories

Data related to system updates (patches) for all systems are stored here during the patch process.
Each new patch version is named according to RHEL-7.a.b.c (e.g., RHEL- The files used to update to that version from the previous one are contained below this directory.
Updates for the two servers are stored below here.
Nonkernel update rpms are stored here.
Kernel update rpms are stored here.
The latest NRAO scripts, etc. are stored here as tar files.
Updates for the correlator nodes are stored below here.
Basic updates for the correlator nodes are stored here
Kernel updates for the correlator nodes are stored below here.
Updates related to lustre are stored here. These are usually RPMs created by rebuilding the Lustre client modules against the newly updated kernel.
Updates related to lustre are stored here. These are usually RPMs created by rebuilding the Lustre client modules against the newly updated kernel. This is TBD at the current moment (1/22/20)

System Administration Directories

A GIT repo for NRAO-created scripts and data. The source files typically live in there own repo (e.g., one at /opt/services/bin) but because of the patching scripts have the potential of overwriting these directories, a backup repo is located here. When committed changes are made to the source repo, they should be pushed to this one as well.
The root of RedHat's python3 installation.
The root directory of NRAO managed scripts, data, etc.
System admin scripts live here and below
Patching script auxiliary files live here.
Files needed to remotely log into the KVMs are here (see KVM Access).
This is the root of the directory tree accessible by the TFTP daemon. In particular the files used to boot the diskless hosts are contained under this directory.
The files used to boot diskless image RHEL-7.a.b.c are located here (see above).
The diskless images served up to the SWCs are located here (see above).
Scripts used as part of a NRAO systemd service are located here (if any). There is a counterpart swc-system directory in the diskless images which currently require this feature.
The home directory for the root account.
Location of systemd system-related service files, etc.

Miscellaneous Packages

Unusual application files, etc., are stored below here.
A place for nonstandard executable files (or links to them); root and admin have this directory in their paths. Most importantly, the links to python3 are here.
The root of the nagios installation.

System working area

Standard location for Linux log files.
Root of the named configuration. Zone files are here. Named runs in a chroot located below this directory so things look strange at first glance.
Home directory for the web server content.

-- JimJacobs - 2019-05-13
Topic revision: r5 - 2020-02-25, JimJacobs
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback