Logging
Rsyslog
Server-1 runs rsyslogd via the
systemd
rsyslog.service
; the main configuration for the
rsyslod
is
/etc/rsyslog.conf
.
DHCP Logging
DHCPD messages are logged to
/var/log/dhcpd/dhcpd.log
with older instances of the log archived in the same directory. The reason for this is that there are some rogue devices out there requesting DHCP leases. The MAC addresses indicate they come from
SuperMicro so they're probably related to the BGFS hosts in some way; since unused NICs have not be connected to the switch, it's possibly some sort of virtual NIC (e.g., like a virtual machine) or it could be related to the IP over IB stuff in some unexpected way. The resulting messages seem to occur from several devices at a rate of a few seconds which puts a lot of clutter into the main log file which in turn increases the difficulty in scanning the log.
SWC Logging
In addtion, server-1 provides syslog service to all of the hosts on the admin network (10.1.36). These are handled somewhat differently than in the past were the remote messages were folded into rsyslog's log file
/var/log/messages
. Instead, each remote hosts gets its own log file located in
/var/log/hosts
with a log file bearing the host's named (e.g.,
swc-023.log
).
Firewall Issues
For whatever reason, the default SE Linux policy file allows unencrypted remote syslog messages to come in on port 20514. To accomodate this, the
/etc/rsyslog.conf
files on the remote host must specify this port. It also requires that the server-1 firewall be configured to support this. A new service,
syslogSeLinux
was added to
/etc/firewalld/services
. The file was based on the stock one
/usr/lib/firewalld/services/syslog.xml
. The new service was added in the allowed services for the
server-admin
firewall zone (located in
/etc/firewalld/zones
).
Rotation
Log rotation is handled by the
logrotate
utility. This is not a service, but works as a cron job.
logrotate
uses
/etc/logrotate.conf
as its main configuration file; that file also includes the files located in
/etc/logrotate.d
. The various log files have different rotation frequencies and retention limits. The default is to rotate weekly, tag old versions with a date stamp, compress all but the most recent log and keep only 12 versions around; however, the various configuration files in
logrotate.d
override the defaults for some of the log file types.
The log rotation settings can be somewhat validated by doing
logrotate -d /etc/logrotate.conf
. Because it's a debug mode it will only state what would have been done without actually doing anything which limits its ability for testing post rotation steps and other script-like steps. Full on testing can be done by twiddling the
logrotate
state file,
/var/lib/logrotate/logrotate.status
, which is where it keeps track of the lst time it acted on a file and therefore determins the age of a log file. Testing this way is a bit tedious but feasible.
Remote Log Rotation
A new configuration file,
hosts
, was added to
logrotate.d
to assist in the rotation of remote host log files. These log files are rotated weekly. A
post rotation
step was added so that the file ends up being copied over to
/export/var/log/hosts
so that the risk of overfilling the smallish /var partition is minimized. All but the latest version of a log on the
export
partition is compressed and all carry a timestamp as part of their file name (e.g., swc-001.log-20200114.gz). A custom python program is used in the post rotation step to ensure that only 12 versions of a particular log file is kept around).
Audit Logging
The
auditd
service is set up, per the STIG, to log a huge number of different access types. On the servers the logs go into
/var/log/audit
. On the SWCs they normally log to /var/log which is a volatile
tmpfs
; the one exception is swc-001 which logs to
/home/usno
which is a mount to
/export/home/usno-serv/usno on server-1. Since swc-001 is the only SWC that allows off admin subnet access, it seemed like a good idea to keep more log history around. This facility required a host-specific =/etc/audit/auditd.conf
file for swc-001. It also required twiddling with the diskless image =/usr/lib/systemd/system/auditd.service file cause it to wait for remote file systems to be established and also to remove a circular dependency that this would produce involving sysint.target.
--
JimJacobs - 2020-01-23