Server-1 runs rsyslogd via the
; the main configuration for the
DHCPD messages are logged to
with older instances of the log archived in the same directory. The reason for this is that there are some rogue devices out there requesting DHCP leases. The MAC addresses indicate they come from SuperMicro
so they're probably related to the BGFS hosts in some way; since unused NICs have not be connected to the switch, it's possibly some sort of virtual NIC (e.g., like a virtual machine) or it could be related to the IP over IB stuff in some unexpected way. The resulting messages seem to occur from several devices at a rate of a few seconds which puts a lot of clutter into the main log file which in turn increases the difficulty in scanning the log.
In addtion, server-1 provides syslog service to all of the hosts on the admin network (10.1.36). These are handled somewhat differently than in the past were the remote messages were folded into rsyslog's log file
. Instead, each remote hosts gets its own log file located in
with a log file bearing the host's named (e.g.,
For whatever reason, the default SE Linux policy file allows unencrypted remote syslog messages to come in on port 20514. To accomodate this, the
files on the remote host must specify this port. It also requires that the server-1 firewall be configured to support this. A new service,
was added to
. The file was based on the stock one
. The new service was added in the allowed services for the
firewall zone (located in
Log rotation is handled by the
utility. This is not a service, but works as a cron job.
as its main configuration file; that file also includes the files located in
. The various log files have different rotation frequencies and retention limits. The default is to rotate weekly, tag old versions with a date stamp, compress all but the most recent log and keep only 12 versions around; however, the various configuration files in
override the defaults for some of the log file types.
The log rotation settings can be somewhat validated by doing
logrotate -d /etc/logrotate.conf
. Because it's a debug mode it will only state what would have been done without actually doing anything which limits its ability for testing post rotation steps and other script-like steps. Full on testing can be done by twiddling the
, which is where it keeps track of the lst time it acted on a file and therefore determins the age of a log file. Testing this way is a bit tedious but feasible.
Remote Log Rotation
A new configuration file,
, was added to
to assist in the rotation of remote host log files. These log files are rotated weekly. A
step was added so that the file ends up being copied over to
so that the risk of overfilling the smallish /var partition is minimized. All but the latest version of a log on the
partition is compressed and all carry a timestamp as part of their file name (e.g., swc-001.log-20200114.gz). A custom python program is used in the post rotation step to ensure that only 12 versions of a particular log file is kept around).
service is set up, per the STIG, to log a huge number of different access types. On the servers the logs go into
. On the SWCs they normally log to /var/log which is a volatile
; the one exception is swc-001 which logs to
which is a mount to
/export/home/usno-serv/usno on server-1. Since swc-001 is the only SWC that allows off admin subnet access, it seemed like a good idea to keep more log history around. This facility required a host-specific =/etc/audit/auditd.conf
file for swc-001. It also required twiddling with the diskless image =/usr/lib/systemd/system/auditd.service file cause it to wait for remote file systems to be established and also to remove a circular dependency that this would produce involving sysint.target.