Errata and Issues

• The usno-serv host may need to be able to be a mail server and devices like the PDUs can send email to it and it can relay out. Mail from usno-serv-1 works but mail relayed through it doesn't. For example, sending mail to root on node-001 fails.
• Oct. 12, 2015 krowe: with the upgrade to RHEL6.7 at the USNO, Ctrl-C no longer works.

Done

• Done Node-12 had an issue with network interface em1 and could not boot from it. No network traffic was seen at the switch. Another node's cable from the switch was tried and also failed. em2 was tested and worked. Reconfigured em1 and it now works. Similar errors occured later with node-12 and so node-12 was swapped with test node-5. This should fix the problem.
• Done All compute nodes (1 to 33) need to have multicast enabled on the 10gbit interface and an mcast route set up.

  ifconfig p2p1 multicast
  route add -net 224.0.0.0 netmask 240.0.0.0 dev p2p1 

  Solution modified /opt/bin/netconfig.sh to implement these changes.
• Done Node-9 had an issue with network interface em1. Link lights would go out when moving cable around. After wiggling and replugging several times, the problem disappeared.
• Done Node-16 10gbit fiber seems to be bad at the switch end. Link loss can be corrected by wiggling the fiber. Replaced the SFP with a local spare, need to order spare LC-LC cables and spare SFPs
• Done Shutting down or rebooting the nodes requires

  umount -l /usr && reboot

  or the node will hang not being able to unmount /usr. Power cycling seems to be the only way out from there.
  • Solution with the change to the new NFS diskless system, this is no longer an issue. * There is a DNS server available at usno. The compute nodes need to either be in some "made up" domain like usno.aoc.nrao.edu and the nrao-serv dns server will be the master for that "made up" doamin, or a real domain like usno.navy.mil and the nrao-serv dns server will just be a forwarder.
  • Solution We created a usno.nrao.edu and made usno-serv-1 the master of it.
• what does USNO have for NTP servers? We need some servers and peers. I suppose we could use usno-serv-2 as a peer.
  • Solution The nodes use usno-serv-1 as an ntp server which in turn uses two usno ntp servers (192.5.41.40 and 192.5.41.41)
• Done Need instructions for installing Intel Primitives (ipp) software
  • Solution UsnoDifxInstallationPrerequisites
• They may want us to relay email through their server or make sure the usno-serv can relay port 25.
  • Solution The instructions mostly cover this but we also had to add the following to /etc/hosts on our mailservers

    198.116.24.185  usno-serv-1.usno.nrao.edu
    198.116.24.186  usno-serv-2.usno.nrao.edu
    

• Done Feb. 4, 2016 krowe: with the upgrade to RHEL6.7 at the USNO, the new version of logrotate (logrotate-3.7.8-26.el6_7.x86_64) tries to update /var/lib/logrotate.status in an atomic way (no clue why they thought this was necessary). But it means it now tries to write something like /var/lib/logrotate_temp.sqjqAA and then rename it. It cannot do this because /var/lib is read-only on the diskless clients. The is some chatter about making a /var/lib/logrotate/status but I don't understand how that helps.
  • Feb. 18, 2015 krowe: I tried adding "dirs /var/lib" to /etc/rwtab.d/nrao-logrotate but then /var/lib/rpm ended up mostly empty and I was no longer able to use rpm to lookup any packages. Not sure why that happened.
  • SOLUTION Feb. 18, 2015 krowe: I tried making /var/lib/logrotate in the root filesystem, adding "dirs /var/lib/logrotate" to /etc/rwtab.d/nrao-logrotate, changing /etc/cron.daily/logrotate to use --sate /var/lib/logrotate/status. There was some concern in the bugzilla report that this may interfere with some postinstall script. But, I don't see any scripts on the logrotate rpm. This seems to work and I have added it to the os-image-base-6.7.sh script.