WARNING! THIS TOPIC IS GENERATED BY System.ChecklistPlugin PLUGIN. DO NOT EDIT THIS TOPIC (except table data)! Back to the checklist topic UsnoDifxOsPatch2. *con...
Fringing Host (test cluster only) Overview In addition to the normal correlation work, some other analytical work is done. At the USNO, they have two fringing...
Contents Networking As of this writing the plan is to fold the new correlator into the existing correlator network layout. This will facilitate keeping the two i...
WARNING! THIS TOPIC IS GENERATED BY System.ChecklistPlugin PLUGIN. DO NOT EDIT THIS TOPIC (except table data)! Back to the checklist topic UsnoDifxPatch2Draft. *...
USNO Maintenance Log 2 This log page allows significant maintenance/administration events to be logged for future use. The goal is to record information that migh...
KVM Console/Switchs (jhj 1/15/21) Currently two of the KVM retained from the old correlator are not on the net. These are the KVM in the second rack of SWCs and...
Environment Monitor The environment monitor is positioned to read the temperature and humidity via a sensor mounted above the rack containing the correlator nodes...
Host SSH Aliases In root's .bashrc are defined a number of aliases to make it a bit quicker to do commands to a single host. Basically it aliases the host name wi...
Ping Diskless The custom python3 utility ping diskless was originally designed to ping the SWCs to watch them through the reboot cycle. It has evolved such that ...
Nagios Nagios is a watchdog daemon that can be configured to check on various properties of a computer system and issue warnings and/or alarms if one of the prope...
Client (SWC) Installation The initial client installation was performed using the hard disk drive that came with SWC 001. The host was booted via PXE boot and the...
BGFS Installation The BGFS hosts were installed by vendor. They were then modified by us to meet the most pressing requirements (i.e., "critical" and "high" find...
Logging Rsyslog Server 1 runs rsyslogd via the systemd rsyslog.service; the main configuration for the rsyslod is /etc/rsyslog.conf. DHCP Logging DHCPD messages...
Time Service Overview Time services is provided primarily by server 1 using the chronyd daemon. The configuration file for chronyd is /etc/chrony.conf. Service...
BGFS File Servers Overview The new correlator is using BEEGFS (bgfs) in place of Lustre to provide a high speed, high capacity file system. The bgfs servers are ...
Compute Nodes Basic System Specs The SWC (SoftWare Correlator) hosts are Dell R640 systems. These have two Intel Xeon Gold 6132 CPUs which run at 2.60 GHz; each ...
USNO Correlator Documentation Wiki This Wiki section serves as the primary documentation for the NRAO project to provide a DifX correlator to USNO. It is intentio...
Infiniband Switches Overview The correlators use Infiniband (IB) network connections to pass data between the correlator nodes (SWCs) and also with the file serv...
RADIUS Service RADIUS is an authentication service often supported by simpler devices such as switches. The servers run a systemd service named radiusd that the ...
Admin Servers Introduction Each correlator has two administrative servers, server 1 and server 2. The servers are used to administer the correlator and take no d...
USNO Correlator II Overview Production Correlator The USNO uses a computing cluster built and maintained by NRAO for performing correlations using DiFX. In 2019...
Neo Correlator Patching The patch process for the Neo correlator (installed at USNO on November 2019) is outlined below. A set of "checkboxes" is provided to assi...
AIDE AIDE is a program that scans important files in the system and reports any changes in the files relative to the accepted baseline. The security STIG require...
Shutting off Nagios Alarms When a nagios problem can't be fixed right away it can get tiresome to have the alarm message coming out every hour or so. Almost the s...
Pty Allocation Problem So far, this problem has mostly occurred on server 1 on both clusters, probably because it gets the most login traffic suring maintenance. ...
Debugging SE Linux Both server 1..2 and the BGFS hosts are running SE Linux in targeted mode; the SWCs are not running SE Linux although since the images are host...
Correlator DNS Service Server Configuration Both server 1 and server 2 provide DNS service to the correlator. Server 2 should be considered a backup to server 1....
Mail Service Overview Since the correlator is effectively a compute device, very little email service is required. The only exception is that the system administ...
File Transfer From an NRAO Host The hosts on either correlator can be reached via the NRAO DNS, file transfer can be initiated from an NRAO host in a pretty stra...
Do on hosts This python script is designed to execute a command on a set of cluster hosts. It is invoked via the link do on hosts located in /opt/services/bin wh...
Host Status Tool The host status tool can be used to summarize important attributes for all hosts in the system. It is invoked by doing host status all on one of...
WARNING! THIS TOPIC IS GENERATED BY System.ChecklistPlugin PLUGIN. DO NOT EDIT THIS TOPIC (except table data)! Back to the checklist topic UsnoDifxOsPatch. *cont...
The file ~/.ssh/jjacobs s1t.config This is an instance of a user specific file to create the linkage from server 1p to server 1t. It currently hops to usno serv 1...
Mellanox (Infiniband) Switch Software information * Collected 5/15/20 by Jim Jacobs data m6036 1 standalone: master show version Product name: MLNX OS...
Disk Layout Server Disk Partitioning The disk layout is shown below as displayed using lsblk. The two disks are partitioned somewhat in parallel although the fir...
DHCP Overview Server 1 provides DHCP service on the network admin network. The setting for the daemon are in /etc/dhcp/dhcpd.conf. The IP addresses on all of ne...
VNC and X Window Connections To launch an X application, make an ssh tunnel and then launch the X app an it'll appear locally. * ssh X J admin@usno serv 1t ...
This rule requires the installation and continuous execution of a host based intrusion detection system. The preferred package is McAfee but apparently something ...
WARNING! THIS TOPIC IS GENERATED BY System.ChecklistPlugin PLUGIN. DO NOT EDIT THIS TOPIC (except table data)! Back to the checklist topic UsnoRhel7Stig2Productio...
The STIG requires multifactor access for all privleged accounts; it mentions using two government approved mechanisms (one is CAC) as examples. It's possible that...
The twoservers do not mount any NFS volumes. The SWCs have two mounts to server 1. We'll have to see if this causes any heartburn when it's enabled on the SWCs.
None of the hosts are currently doing any packet forwarding. On the old cluster the firewall had to be modified to allow packet forwarding to support some of the ...