Disk Layout

Server Disk Partitioning

The disk layout is shown below as displayed using lsblk. The two disks are partitioned somewhat in parallel although the first disk (sda) contains the the actual boot partitions while the second disk just has them to stay in sync. Two of the partitions on each disk are used to form a RAID level one disk. The partitions sda2 and sdb2 are used to form the md0 raid disk that is mounted as the root of the system file system; the RAID system 64GB of storage on each physical disk. The partitions sda4 and sdb5 form the md1 RAID disk; this device contains ~860 GB of storage and is mounted on /export.

# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT Notes
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT Notes
sda 8:0 0 931.5G 0 disk    
├─sda1 8:1 0 256M 0 part /boot/efi UEFI boot partition
├─sda2 8:2 0 1G 0 part /boot Boot partition; system images, etc. are here.
├─sda3 8:3 0 64G 0 part    
│ └─md0 9:0 0 64G 0 raid1 / System root
├─sda4 8:4 0 16G 0 part /tmp DoD STIG requires /tmp to have its own partition.
├─sda5 8:5 0 16G 0 part /var DoD STIG requires /var to have its own parition.
├─sda6 8:6 0 8G 0 part [SWAP]  
└─sda7 8:7 0 826.3G 0 part    
└─md1 9:1 0 826.1G 0 raid1 /export DiFX Software and user home directories live here.
sdb 8:16 0 931.5G 0 disk    
├─sdb1 8:17 0 64G 0 part    
│ └─md0 9:0 0 64G 0 raid1 /  
├─sdb2 8:18 0 16G 0 part /tmpSpare Spare of /tmp partition on other disk.
├─sdb3 8:19 0 16G 0 part /varSpare Spare of /var partition on other disk.
├─sdb4 8:20 0 8G 0 part [SWAP]  
├─sdb5 8:21 0 1G 0 part /bootSpare Spare of /boot on other disk.
├─sdb6 8:22 0 256M 0 part /bootSpare/efiSpare Spare of /boot/efi on other disk.
└─sdb7 8:23 0 826.3G 0 part    
└─md1 9:1 0 826.1G 0 raid1 /export  

Server directory layout

TFTP Directory

Bootup of the diskless systems requires the use of the TFTP daemon. This daemon has a restricted root so that it may only access files located in /opt/services/tftpboot. Some miscellaneous files are contained in that directory in addition to the files used to boot up the diskless systems. Those directories are named to correspond to the diskless images (see diskless images section below). Below the directory named after the diskless system image (e.g., RHEL-7.6.0.4) is one named swc. This directory contains the files needed to start the UEFI bootstrap of the SWCs:

grub.cfg
The configuration provided to the GRUB bootloader on the SWC. The one in pxelinux.cfg is not apprently used.

initramfs...
Initial file system used in bootup (the kernel version number is elideed)'

vmlinuz...
The kernel file used in bootup (the kernel version number is elideed)'

grubx64.efi
A signed(?) file used to start up the secure UEFI boot process.

Diskless Images and Host-Specific Snapshots

The diskless images are stored under /opt/services/diskless_boot. Several versions of diskless images will be stored there and they are named RHEL-7.a.b.c where a is the RHEL-7 minor version, while b and c are used by NRAO to identify the patch version of the particular image version. The intent is that b will be incremented when a notable modification is made to the image and c is incremented each time a new image is produced which will be whenever the system image is patched.

Below the system image directory there are three subdirectories: etc, swc and var. The most important one is swc; swc has two subdirectories root and snapshot. The root directory is the root, /, of the system image provided to the SWCs. The snapshot directory is used to hold host-specific files.

Host Specific Snapshots

For each host requiring host specific files there will be a directory named using the host's fully DNS named (e.g., swc-001.usno.nrao.edu). Below the host-named directory are the directories required to reach the host-specific files. For example, for swc-001 the layout is:

|---etc
    |-- firewalld
    |   |-- zones
    |       |--swc-001-external.xml
    |-- sysconfig
        |-- network-scripts
            |-- ifcfg-em4

In addition to the directories leading towards host-specific files, the host-named directory (swc-001t.usno.nrao.edu) also contains a file named files. This file contains the aboslute paths to any files that are host-specific; for swc-001t that file contains:

/etc/sysconfig/network-scripts/ifcfg-em4
/etc/firewalld/zones/swc-001-external.xml

Readonly-root

In addition, the file /etc/sysconfig/readonly-root is used to make the readonly diskless image usable by the SWCs. It causes them to put system files which are normally written to, into a tmpfs type partition. Below are the contents of the file on server-1t as of 1/21/20:

# Set to 'yes' to mount the system filesystems read-only.
# NOTE: It's necessary to append 'ro' to mount options of '/' mount point in
#       /etc/fstab as well, otherwise the READONLY option will not work.
READONLY=no
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
TEMPORARY_STATE=no
# Place to put a tmpfs for temporary scratch writable space
RW_MOUNT=/var/lib/stateless/writable
# Label on local filesystem which can be used for temporary scratch space
RW_LABEL=stateless-rw
# Options to use for temporary mount
RW_OPTIONS=
# Label for partition with persistent data
STATE_LABEL=stateless-state
# Where to mount to the persistent data
STATE_MOUNT=/var/lib/stateless/state
# Options to use for persistent mount
STATE_OPTIONS=
# NFS server to use for persistent data?
CLIENTSTATE=
# Use slave bind-mounts
SLAVE_MOUNTS=yes

Important Directories

Server boot files

/boot
The server system boot files. These include kernel-versioned files holding: initial RAM filesystem, kernel, various configuration files. Usually there will be serveral kernel versions plus a rescue image.
/boot/efi
Root of the efi partition used by UEFI boot.
/boot/efi/EFI/redhat
Grub files to enable UEFI boot. In particular the file grub.cfg is used to generate the GRUB menu transiently displayed during bootup.

System configuration files

/etc
Standard system configuration tree root.
/etc/profile.d
Files used to configure bash logins.
/etc/sysconfig/network-scripts
Files used to set up some aspects of the networks. Other network configuration is performed as a systemd service (see below).
/etc/systemd/system
systemd system configuraiton files are located here. Nonstandard .service files are located here. Standard (unmodified) service files are located in /usr/lib/systemd/system. Services are enabled/disabled using the systemctl command; these often modifies the contents of some of this folders subdirectories by placing links to enabled serves and removing those when services are disabled.

Difx, User home directories and Patches

/export
This directory is the root of the second RAID 1 partition and contains the bulk of the storage on the server's disks. This directory is normally mounted by the diskless systems to enabled DiFX operation as well as access to user home directories.
/export/home/usno-serv/usno
This rather long path was kept (somewhat mindlessly) from an earlier installation.
/export/home/usno-serv/usno/difx
The root of the DiFX installation.
/export/home/usno-serv/usno/users
Most user home directories are located below this directory; this includes the homes of USNO folks having correlator accounts.
/export/home/usno-serv/usno/users/admin
The admin account's home directory lives here.
/export/home/usno-serv/usno/users/difxmgr
The difxmgr account's home directory lives here.
/export/home/usno-serv/usno/users/oper
The oper account's home directory lives here.

Patch Data Directories

/export/home/usno-serv/usno/updates
Data related to system updates (patches) for all systems are stored here during the patch process.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c
Each new patch version is named according to RHEL-7.a.b.c (e.g., RHEL-7.7.0.4). The files used to update to that version from the previous one are contained below this directory.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server
Updates for the two servers are stored below here.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server/basic
Nonkernel update rpms are stored here.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server/kernel
Kernel update rpms are stored here.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server/scripts
The latest NRAO scripts, etc. are stored here as tar files.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/swc
Updates for the correlator nodes are stored below here.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/swc/basic
Basic updates for the correlator nodes are stored here
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/swc/kernel
Kernel updates for the correlator nodes are stored below here.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/lustre
Updates related to lustre are stored here. These are usually RPMs created by rebuilding the Lustre client modules against the newly updated kernel.
/export/home/usno-serv/usno/updates/RHEL-7.a.b.c/bgfs
Updates related to lustre are stored here. These are usually RPMs created by rebuilding the Lustre client modules against the newly updated kernel. This is TBD at the current moment (1/22/20)

System Administration Directories

/opt/nrao-git
A GIT repo for NRAO-created scripts and data. The source files typically live in there own repo (e.g., one at /opt/services/bin) but because of the patching scripts have the potential of overwriting these directories, a backup repo is located here. When committed changes are made to the source repo, they should be pushed to this one as well.
/opt/rh/rh-python36/root
The root of RedHat's python3 installation.
/opt/services
The root directory of NRAO managed scripts, data, etc.
/opt/services/bin
System admin scripts live here and below
/opt/services/bin/patching
Patching script auxiliary files live here.
/opt/services/kvm
Files needed to remotely log into the KVMs are here (see KVM Access).
/opt/services/tftpboot
This is the root of the directory tree accessible by the TFTP daemon. In particular the files used to boot the diskless hosts are contained under this directory.
/opt/services/tftpboot/RHEL-7.a.b.c/swc
The files used to boot diskless image RHEL-7.a.b.c are located here (see above).
/opt/services/diskless_boot
The diskless images served up to the SWCs are located here (see above).
/opt/services/system-scripts
Scripts used as part of a NRAO systemd service are located here (if any). There is a counterpart swc-system directory in the diskless images which currently require this feature.
/root
The home directory for the root account.
/usr/lib/systemd/system
Location of systemd system-related service files, etc.

Miscellaneous Packages

/usr/local
Unusual application files, etc., are stored below here.
/usr/local/bin
A place for nonstandard executable files (or links to them); root and admin have this directory in their paths. Most importantly, the links to python3 are here.
/usr/local/nagios
The root of the nagios installation.

System working area

/var/log
Standard location for Linux log files.
/var/named
Root of the named configuration. Zone files are here. Named runs in a chroot located below this directory so things look strange at first glance.
/var/www
Home directory for the web server content.

-- JimJacobs - 2019-05-13
Topic revision: r5 - 2020-02-25, JimJacobs
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback