---

Nagios

---

We install service software in /opt/services instead of the normal location. This seperates it from the OS install and allows us to re-install and/or upgrade the OS without worry of interfearing with such services. We also keep /opt/services on a seperate disk which allows us to replace the entire OS disk without interfearing with installed services. Finally, we try to seperate the local configurations of a service (nagios-local) from the service itsetlf (nagios). This allows us to easily upgrade the service itself while still keeping the configuration of the service.

nrpe is the remote agent that monitors things locally that the nagios server cannot monitor on its own. Situations where we use nrpe are the 3ware cards and disk usage. Since a remote machine cannot get any such information remotelly through ports or existing services we install nrpe on the remote host and it provides information to the nagios server. For the USNO, the only think I can think of that might need nrpe would be to monitor the 10Gb interface.

---

Install Nagios

Retrieve nagios from http://www.nagios.org/ Use the Nagios Core download (nagios-3.4.1.tar.gz)

Set a variable for the version of nagios to use

NAGIOS_VER="3.4.1"

Ensure installation area

/opt/services probably already exists and is a symlink to some data partition.

mkdir /opt/services

Create nagios user and group

useradd -b /opt/services -N nagios
groupadd nagios
usermod -a -G nagios nagios
usermod -a -G nagios apache
rm -rf /opt/services/nagios

Compile nagios

download Nagios Core from http://www.nagios.org/ and save it in /tmp/root

mkdir /tmp/root
cd /tmp/root
tar xfvz nagios-${NAGIOS_VER}.tar.gz 
cd nagios
mkdir -p /opt/services/nagios-local/etc
mkdir -p /opt/services/nagios-${NAGIOS_VER}
(cd /opt/services ; ln -s nagios-${NAGIOS_VER} nagios)
./configure --prefix=/opt/services/nagios-${NAGIOS_VER} --sharedstatedir=/opt/services/nagios-local/etc --localstatedir=/opt/services/nagios-local/var --with-command-group=nagios
make all
make install
make install-config
make install-commandmode

Create startup script

mkdir -p /opt/services/nagios-local/init.d
cp daemon-init /opt/services/nagios-local/init.d/nrao-nagios
chmod 755 /opt/services/nagios-local/init.d/nrao-nagios

edit /opt/services/nagios-local/init.d/nrao-nagios

prefix=/opt/services/nagios-local
exec_prefix=/opt/services/nagios

configure script to start

ln -s /opt/services/nagios-local/init.d/nrao-nagios /etc/init.d
chkconfig --add nrao-nagios

---

Create Nagios Tar Ball

This tar ball can be used to simplify the installation of clients. Follow the Install Nagios steps above first.

cd /opt/services
tar cfvz nagios_nrao_usno_`uname -i`-${NAGIOS_VER}.tgz nagios* nrpe*

copy this tar ball to someplace your clients can access like /home/src.

---

Install Nagios Client (from tar ball)

You should have already created a tar ball for client installs as part of the server install above.

Install the tarball made in the previous section if not already installed

echo "nagios:x:1103:1103:nagios:/var/log/nagios:/bin/sh" >> /etc/passwd
echo 'nagios:!!:15280::::::' >> /etc/shadow
echo "nagios:x:1103:" >> /etc/group

cd /opt/services
tar xfvz /home/src/nagios/client/magios_nrao_usno_`uname -i`-${NAGIOS_VER}.tgz

If the machine has RAID arrays to monitor

emacs -nw /etc/sudoers.d/nrao-nagios

Defaults !requiretty
nagios ALL = NOPASSWD: /opt/services/nagios-local/plugins/check_3ware.sh
nagios ALL = NOPASSWD: /opt/services/nagios-local/plugins/check_3ware_spare.sh
nagios ALL = NOPASSWD: /opt/services/nagios-local/plugins/check_megaraid_sas
nagios ALL = NOPASSWD: /opt/services/nagios-local/plugins/check_md_raid
nagios ALL = NOPASSWD: /opt/services/nagios-local/plugins/check_lsi_raid

If we want to monitor the free space on the disk disk(s) (This is rare)

emacs -nw /opt/services/nrpe-local/etc/nrpe.cfg and add something like this

command[nrao_check_lando4]=/opt/services/nagios-plugins/libexec/check_disk -w 1% -p /export/home/lando4

If the machine is a Lustre OSS

emacs -nw /opt/services/nrpe-local/etc/nrpe.cfg and add something like this

command[check_mountpoints]=/opt/services/nagios-local/plugins/check_mountpoints.sh /export/lustre/ost1-1 /export/lustre/ost1-2

Start Nagios

ln -s /opt/services/nrpe-local/init.d/nrao-nrpe /etc/init.d
chkconfig --add nrao-nrpe
chkconfig --level 345 nrao-nrpe on
/etc/init.d/nrao-nrpe start

---

Configure Nagios Server

copy configuration file(s)

mkdir /opt/services/nagios-local/etc/nrao
cp /opt/services/nagios/etc/objects/timeperiods.cfg /opt/services/nagios-local/etc/nrao
cp /opt/services/nagios/etc/objects/templates.cfg /opt/services/nagios-local/etc/nrao
cp /opt/services/nagios/etc/objects/commands.cfg /opt/services/nagios-local/etc/nrao

Change workhours

sed -i -e 's/workhours/24x7/g' /opt/services/nagios-local/etc/nrao/templates.cfg

create macros file

cp /opt/services/nagios-3.4.1/etc/resource.cfg /opt/services/nagios-local/etc
sed -i -e "s/nagios-${NAGIOS_VER}/nagios-plugins/g" /opt/services/nagios-local/etc/resource.cfg

configure file

Save the attached nagios.cfg as /opt/services/nagios-local/etc/nagios.cfg

Configure cfg files

Save the attached hosts config file for nagios as /opt/services/nagios-local/etc/nrao/hosts.cfg
Save the attached services config file for nagios as /opt/services/nagios-local/etc/nrao/services.cfg
Save the attached contacts config file for nagios as /opt/services/nagios-local/etc/nrao/contacts.cfg
edit /opt/services/nagios-local/etc/nrao/commands.cfg

define command{
    command_name    check_dns
    command_line    $USER1$/check_dns -H $ARG1$
}

define command{
    command_name    check_dhcp_process
    command_line    $USER1$/check_procs -C dhcpd
}

define command{
    command_name    check_ntp_time
    command_line    $USER1$/check_ntp_time -H $HOSTADDRESS$
}

---

Configure Web Interface

copy sample-config/httpd.conf somplace

cd /tmp/root/nagios
make install-webconf
sed -i -e 's|/opt/services/nagios-3.4.1/etc/htpasswd.users|/opt/services/nagios-local/etc/htpasswd.users|g' /etc/httpd/conf.d/nagios.conf
sed -i -e "s/nagios-${NAGIOS_VER}/nagios/g" /etc/httpd/conf.d/nagios.conf
echo "SetEnv NAGIOS_CGI_CONFIG \"/opt/services/nagios-local/etc/cgi.cfg\"" >> /etc/httpd/conf.d/nagios.conf 

Setup admin account

htpasswd -c /opt/services/nagios-local/etc/htpasswd.users admin

Setup usno guest account

htpasswd /opt/services/nagios-local/etc/htpasswd.users usnogst

Setup CGI config

cp /opt/services/nagios/etc/cgi.cfg /opt/services/nagios-local/etc
sed -i -e "s/nagios-${NAGIOS_VER}/nagios-local/g" /opt/services/nagios-local/etc/cgi.cfg
sed -i -e "s/nagiosadmin/admin/g" /opt/services/nagios-local/etc/cgi.cfg

edit /opt/services/nagios-local/etc/cgi.cfg

authorized_for_system_information=admin,guest

authorized_for_configuration_information=admin,guest

authorized_for_all_services=admin,guest
authorized_for_all_hosts=admin,guest

authorized_for_read_only=guest

---

Install Nagios Plugins

nagios-plugins are available at http://www.nagios.org/download/plugins They are seperate from the nagios install but are required as they are the actual programs that ping, telnet, check disk space, etc. I create a directory in nagios-local for plugins that don't come with the nagios-plugins package. This will keep them seperate and allow upgrading of nagios-plugins just like nagios-local allows for upgrading of nagios without loosing configuration.

Set a variable for the version of ganglia-web to use

PLUGIN_VER="1.4.15"

Compile nagios-plugins

Get nagios-plugins from http://www.nagios.org/download/plugins and save it in /tmp/root:

cd /tmp/root
tar xfvz nagios-plugins-${PLUGIN_VER}.tar.gz
cd nagios-plugins-${PLUGIN_VER}
./configure --prefix=/opt/services/nagios-plugins-${PLUGIN_VER} --with-nagios-user=nagios --with-nagios-group=nagios
make
make install
(cd /opt/services ; ln -s nagios-plugins-${PLUGIN_VER} nagios-plugins)

Create local plugin area

mkdir /opt/services/nagios-local/plugins
echo  >> /opt/services/nagios-local/etc/resource.cfg
echo "# \$USER32\$ is for local plugins not in the standard nagios-plugins" >> /opt/services/nagios-local/etc/resource.cfg
echo "\$USER32\$=/opt/services/nagios-local/plugins" >> /opt/services/nagios-local/etc/resource.cfg

---

hosts.allow

echo "nrpe:10.1.34.2" >> /etc/hosts.allow

---

Start

Start nagios server

/etc/init.d/nrao-nagios start

Start web server

/etc/init.d/httpd restart

---

Access

One can now access nagios via

https://usno-serv-1-ext/nagios/ or https://localhost/nagios/ but only through VNC.

---

Acknowledging Notifications

When a nagios error is triggered, nagios will start and continue sending notification messages to usno-admins@nrao.edu. After the problem has been identified, the nagios issue should be acknowledged so nagios will stop sending the notifications. To acknowledge:

1. VNC to usno-serv-1, start konquerer and access nagios by https as above.

2. Click on Tactical Overview.

3. The issue should be displayed under hosts or services, click on the issue.

4. The issue should be displayed as a line with Hosts and Services columns. If it is a host issue, click on the link under hosts. If it is a service issue, click on the link under services.

5. To the right there should be a link to acknowledge the issue. Click on the acknowledge link.

6. Add a comment to the comment box and click on the Commit button. The issue is now acknowledge and, after a final acknowledged notification, notifications for this issue should stop.

-- KScottRowe - 2012-06-14