Disk Layout
Server Disk Partitioning
The disk layout is shown below as displayed using lsblk. The two disks are partitioned somewhat in parallel although the first disk (sda) contains the the actual boot partitions while the second disk just has them to stay in sync. Two of the partitions on each disk are used to form a RAID level one disk. The partitions sda2 and sdb2 are used to form the md0 raid disk that is mounted as the root of the system file system; the RAID system 64GB of storage on each physical disk. The partitions sda4 and sdb5 form the md1 RAID disk; this device contains ~860 GB of storage and is mounted on /export.
# lsblk
NAME |
MAJ:MIN |
RM |
SIZE |
RO |
TYPE |
MOUNTPOINT |
Notes |
NAME |
MAJ:MIN |
RM |
SIZE |
RO |
TYPE |
MOUNTPOINT |
Notes |
sda |
8:0 |
0 |
931.5G |
0 |
disk |
|
|
├─sda1 |
8:1 |
0 |
256M |
0 |
part |
/boot/efi |
UEFI boot partition |
├─sda2 |
8:2 |
0 |
1G |
0 |
part |
/boot |
Boot partition; system images, etc. are here. |
├─sda3 |
8:3 |
0 |
64G |
0 |
part |
|
|
│ └─md0 |
9:0 |
0 |
64G |
0 |
raid1 |
/ |
System root |
├─sda4 |
8:4 |
0 |
16G |
0 |
part |
/tmp |
DoD STIG requires /tmp to have its own partition. |
├─sda5 |
8:5 |
0 |
16G |
0 |
part |
/var |
DoD STIG requires /var to have its own parition. |
├─sda6 |
8:6 |
0 |
8G |
0 |
part |
[SWAP] |
|
└─sda7 |
8:7 |
0 |
826.3G |
0 |
part |
|
|
└─md1 |
9:1 |
0 |
826.1G |
0 |
raid1 |
/export |
DiFX Software and user home directories live here. |
sdb |
8:16 |
0 |
931.5G |
0 |
disk |
|
|
├─sdb1 |
8:17 |
0 |
64G |
0 |
part |
|
|
│ └─md0 |
9:0 |
0 |
64G |
0 |
raid1 |
/ |
|
├─sdb2 |
8:18 |
0 |
16G |
0 |
part |
/tmpSpare |
Spare of /tmp partition on other disk. |
├─sdb3 |
8:19 |
0 |
16G |
0 |
part |
/varSpare |
Spare of /var partition on other disk. |
├─sdb4 |
8:20 |
0 |
8G |
0 |
part |
[SWAP] |
|
├─sdb5 |
8:21 |
0 |
1G |
0 |
part |
/bootSpare |
Spare of /boot on other disk. |
├─sdb6 |
8:22 |
0 |
256M |
0 |
part |
/bootSpare/efiSpare |
Spare of /boot/efi on other disk. |
└─sdb7 |
8:23 |
0 |
826.3G |
0 |
part |
|
|
└─md1 |
9:1 |
0 |
826.1G |
0 |
raid1 |
/export |
|
Server directory layout
TFTP Directory
Bootup of the diskless systems requires the use of the TFTP daemon. This daemon has a restricted root so that it may only access files located in
/opt/services/tftpboot
. Some miscellaneous files are contained in that directory in addition to the files used to boot up the diskless systems. Those directories are named to correspond to the diskless images (see
diskless images section below). Below the directory named after the diskless system image (e.g.,
RHEL-7.6.0.4
) is one named
swc
. This directory contains the files needed to start the UEFI bootstrap of the SWCs:
- grub.cfg
- The configuration provided to the GRUB bootloader on the SWC. The one in
pxelinux.cfg
is not apprently used.
- initramfs...
- Initial file system used in bootup (the kernel version number is elideed)'
- vmlinuz...
- The kernel file used in bootup (the kernel version number is elideed)'
- grubx64.efi
- A signed(?) file used to start up the secure UEFI boot process.
Diskless Images and Host-Specific Snapshots
The diskless images are stored under
/opt/services/diskless_boot
. Several versions of diskless images will be stored there and they are named RHEL-7.a.b.c where
a is the RHEL-7 minor version, while
b and
c are used by NRAO to identify the patch version of the particular image version. The intent is that
b will be incremented when a notable modification is made to the image and
c is incremented each time a new image is produced which will be whenever the system image is patched.
Below the system image directory there are three subdirectories:
etc
,
swc
and
var
. The most important one is
swc
;
swc
has two subdirectories
root
and
snapshot
. The
root
directory is the root,
/
, of the system image provided to the SWCs. The
snapshot
directory is used to hold host-specific files.
Host Specific Snapshots
For each host requiring host specific files there will be a directory named using the host's fully DNS named (e.g.,
swc-001.usno.nrao.edu
). Below the host-named directory are the directories required to reach the host-specific files. For example, for swc-001 the layout is:
|---etc
|-- firewalld
| |-- zones
| |--swc-001-external.xml
|-- sysconfig
|-- network-scripts
|-- ifcfg-em4
In addition to the directories leading towards host-specific files, the host-named directory (
swc-001t.usno.nrao.edu
) also contains a file named
files
. This file contains the aboslute paths to any files that are host-specific; for swc-001t that file contains:
/etc/sysconfig/network-scripts/ifcfg-em4
/etc/firewalld/zones/swc-001-external.xml
Readonly-root
In addition, the file
/etc/sysconfig/readonly-root
is used to make the readonly diskless image usable by the SWCs. It causes them to put system files which are normally written to, into a
tmpfs
type partition. Below are the contents of the file on server-1t as of 1/21/20:
# Set to 'yes' to mount the system filesystems read-only.
# NOTE: It's necessary to append 'ro' to mount options of '/' mount point in
# /etc/fstab as well, otherwise the READONLY option will not work.
READONLY=no
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
TEMPORARY_STATE=no
# Place to put a tmpfs for temporary scratch writable space
RW_MOUNT=/var/lib/stateless/writable
# Label on local filesystem which can be used for temporary scratch space
RW_LABEL=stateless-rw
# Options to use for temporary mount
RW_OPTIONS=
# Label for partition with persistent data
STATE_LABEL=stateless-state
# Where to mount to the persistent data
STATE_MOUNT=/var/lib/stateless/state
# Options to use for persistent mount
STATE_OPTIONS=
# NFS server to use for persistent data?
CLIENTSTATE=
# Use slave bind-mounts
SLAVE_MOUNTS=yes
Important Directories
Server boot files
- /boot
- The server system boot files. These include kernel-versioned files holding: initial RAM filesystem, kernel, various configuration files. Usually there will be serveral kernel versions plus a rescue image.
- /boot/efi
- Root of the efi partition used by UEFI boot.
- /boot/efi/EFI/redhat
- Grub files to enable UEFI boot. In particular the file
grub.cfg
is used to generate the GRUB menu transiently displayed during bootup.
System configuration files
- /etc
- Standard system configuration tree root.
- /etc/profile.d
- Files used to configure bash logins.
- /etc/sysconfig/network-scripts
- Files used to set up some aspects of the networks. Other network configuration is performed as a
systemd
service (see below).
- /etc/systemd/system
-
systemd
system configuraiton files are located here. Nonstandard .service
files are located here. Standard (unmodified) service files are located in /usr/lib/systemd/system. Services are enabled/disabled using the systemctl
command; these often modifies the contents of some of this folders subdirectories by placing links to enabled serves and removing those when services are disabled.
Difx, User home directories and Patches
- /export
- This directory is the root of the second RAID 1 partition and contains the bulk of the storage on the server's disks. This directory is normally mounted by the diskless systems to enabled DiFX operation as well as access to user home directories.
- /export/home/usno-serv/usno
- This rather long path was kept (somewhat mindlessly) from an earlier installation.
- /export/home/usno-serv/usno/difx
- The root of the DiFX installation.
- /export/home/usno-serv/usno/users
- Most user home directories are located below this directory; this includes the homes of USNO folks having correlator accounts.
- /export/home/usno-serv/usno/users/admin
- The
admin
account's home directory lives here.
- /export/home/usno-serv/usno/users/difxmgr
- The
difxmgr
account's home directory lives here.
- /export/home/usno-serv/usno/users/oper
- The
oper
account's home directory lives here.
Patch Data Directories
- /export/home/usno-serv/usno/updates
- Data related to system updates (patches) for all systems are stored here during the patch process.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c
- Each new patch version is named according to RHEL-7.a.b.c (e.g., RHEL-7.7.0.4). The files used to update to that version from the previous one are contained below this directory.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server
- Updates for the two servers are stored below here.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server/basic
- Nonkernel update rpms are stored here.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server/kernel
- Kernel update rpms are stored here.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/server/scripts
- The latest NRAO scripts, etc. are stored here as tar files.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/swc
- Updates for the correlator nodes are stored below here.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/swc/basic
- Basic updates for the correlator nodes are stored here
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/swc/kernel
- Kernel updates for the correlator nodes are stored below here.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/lustre
- Updates related to lustre are stored here. These are usually RPMs created by rebuilding the Lustre client modules against the newly updated kernel.
- /export/home/usno-serv/usno/updates/RHEL-7.a.b.c/bgfs
- Updates related to lustre are stored here. These are usually RPMs created by rebuilding the Lustre client modules against the newly updated kernel. This is TBD at the current moment (1/22/20)
System Administration Directories
- /opt/nrao-git
- A GIT repo for NRAO-created scripts and data. The source files typically live in there own repo (e.g., one at /opt/services/bin) but because of the patching scripts have the potential of overwriting these directories, a backup repo is located here. When committed changes are made to the source repo, they should be pushed to this one as well.
- /opt/rh/rh-python36/root
- The root of RedHat's python3 installation.
- /opt/services
- The root directory of NRAO managed scripts, data, etc.
- /opt/services/bin
- System admin scripts live here and below
- /opt/services/bin/patching
- Patching script auxiliary files live here.
- /opt/services/kvm
- Files needed to remotely log into the KVMs are here (see KVM Access).
- /opt/services/tftpboot
- This is the root of the directory tree accessible by the TFTP daemon. In particular the files used to boot the diskless hosts are contained under this directory.
- /opt/services/tftpboot/RHEL-7.a.b.c/swc
- The files used to boot diskless image RHEL-7.a.b.c are located here (see above).
- /opt/services/diskless_boot
- The diskless images served up to the SWCs are located here (see above).
- /opt/services/system-scripts
- Scripts used as part of a NRAO
systemd
service are located here (if any). There is a counterpart swc-system directory in the diskless images which currently require this feature.
- /root
- The home directory for the root account.
- /usr/lib/systemd/system
- Location of
systemd
system-related service files, etc.
Miscellaneous Packages
- /usr/local
- Unusual application files, etc., are stored below here.
- /usr/local/bin
- A place for nonstandard executable files (or links to them); root and admin have this directory in their paths. Most importantly, the links to python3 are here.
- /usr/local/nagios
- The root of the nagios installation.
System working area
- /var/log
- Standard location for Linux log files.
- /var/named
- Root of the named configuration. Zone files are here. Named runs in a chroot located below this directory so things look strange at first glance.
- /var/www
- Home directory for the web server content.
--
JimJacobs - 2019-05-13