Logging

Rsyslog

Server-1 runs rsyslogd via the systemd rsyslog.service; the main configuration for the rsyslod is /etc/rsyslog.conf.

DHCP Logging

DHCPD messages are logged to /var/log/dhcpd/dhcpd.log with older instances of the log archived in the same directory. The reason for this is that there are some rogue devices out there requesting DHCP leases. The MAC addresses indicate they come from SuperMicro so they're probably related to the BGFS hosts in some way; since unused NICs have not be connected to the switch, it's possibly some sort of virtual NIC (e.g., like a virtual machine) or it could be related to the IP over IB stuff in some unexpected way. The resulting messages seem to occur from several devices at a rate of a few seconds which puts a lot of clutter into the main log file which in turn increases the difficulty in scanning the log.

SWC Logging

In addtion, server-1 provides syslog service to all of the hosts on the admin network (10.1.36). These are handled somewhat differently than in the past were the remote messages were folded into rsyslog's log file /var/log/messages. Instead, each remote hosts gets its own log file located in /var/log/hosts with a log file bearing the host's named (e.g., swc-023.log).

Firewall Issues

For whatever reason, the default SE Linux policy file allows unencrypted remote syslog messages to come in on port 20514. To accomodate this, the /etc/rsyslog.conf files on the remote host must specify this port. It also requires that the server-1 firewall be configured to support this. A new service, syslogSeLinux was added to /etc/firewalld/services. The file was based on the stock one /usr/lib/firewalld/services/syslog.xml. The new service was added in the allowed services for the server-admin firewall zone (located in /etc/firewalld/zones).

Rotation

Log rotation is handled by the logrotate utility. This is not a service, but works as a cron job. logrotate uses /etc/logrotate.conf as its main configuration file; that file also includes the files located in /etc/logrotate.d. The various log files have different rotation frequencies and retention limits. The default is to rotate weekly, tag old versions with a date stamp, compress all but the most recent log and keep only 12 versions around; however, the various configuration files in logrotate.d override the defaults for some of the log file types.

The log rotation settings can be somewhat validated by doing logrotate -d /etc/logrotate.conf. Because it's a debug mode it will only state what would have been done without actually doing anything which limits its ability for testing post rotation steps and other script-like steps. Full on testing can be done by twiddling the logrotate state file, /var/lib/logrotate/logrotate.status, which is where it keeps track of the lst time it acted on a file and therefore determins the age of a log file. Testing this way is a bit tedious but feasible.

Remote Log Rotation

A new configuration file, hosts, was added to logrotate.d to assist in the rotation of remote host log files. These log files are rotated weekly. A post rotation step was added so that the file ends up being copied over to /export/var/log/hosts so that the risk of overfilling the smallish /var partition is minimized. All but the latest version of a log on the export partition is compressed and all carry a timestamp as part of their file name (e.g., swc-001.log-20200114.gz). A custom python program is used in the post rotation step to ensure that only 12 versions of a particular log file is kept around).

Audit Logging

The auditd service is set up, per the STIG, to log a huge number of different access types. On the servers the logs go into /var/log/audit. On the SWCs they normally log to /var/log which is a volatile tmpfs; the one exception is swc-001 which logs to /home/usno which is a mount to /export/home/usno-serv/usno on server-1. Since swc-001 is the only SWC that allows off admin subnet access, it seemed like a good idea to keep more log history around. This facility required a host-specific =/etc/audit/auditd.conf file for swc-001. It also required twiddling with the diskless image =/usr/lib/systemd/system/auditd.service file cause it to wait for remote file systems to be established and also to remove a circular dependency that this would produce involving sysint.target.

-- JimJacobs - 2020-01-23