edit-patch-variables

1. Increase both versions by one. (RHEL-7.6.0.7 goes to RHEL-7.6.0.8)
2. Correct NRAO user name, if needed (required to get into gygax).
3. Set date to the patch day (e.g., 31-October-2020)

spread-patch-variabes

This copies the patch variables to server-2t, server-1p and server-2p. Since the BGFS and SWC hosts mount the script directory on server-1t and server-1p, they'll get them from there.

usno_create-diskless-image.sh

This will use rsync to create a copy of the current diskless image (using ${OLDVER}) named /opt/services/diskless_boot/${OLDVER}

nohup usno_sync-server-2.sh > /opt/services/bin/log/sync-server2.log & tail -f /opt/services/bin/log/sync-server2.log

This will synchronize important files from the primary server onto the corresponding secondary server. This can be fairly slow, so it's done on Prepatch Monday and then repeated on Patch day to pick up any changes.

check-for-patches

This will indicate the number of patches available for the various system types (server, SWCs, file servers, etc.)

usno_patch.sh get-patches

Complete patch fetch on s1t before starting patch fetch on s1p.

On the test cluster this will pull down the patches from Red Hat for the servers, the SWCs and the bgfs servers. It will also bundle up the relevant parts of the home directory trees for root and admin along with /opt/services/bin (the idea is that server-1t is the reference copy for these files). On the production cluster, the script will copy over the update files prepared on server-1t (with current DC network setup this is especially awkward). See Remote access of server-1t from server-1p

nohup usno_sync-server-2.sh > /opt/services/bin/log/sync-server2.log & tail -f /opt/services/bin/log/sync-server2.log

This will synchronize important files from the primary server onto the corresponding secondary server. Since this only picks up mods since the previous day the operation should be fairly quick.

host-status --all

The output of this command provides a quick status summary for the correlator systems. It can be handy to see how things were before system modifications are performed (e.g., an swc might be down prior to a diskless image page so if it won't come back up it's probably not the diskless image, etc.)

After the patching appears to succeed:

bgfs-restart-or-halt restart

This tries to restart the BGFS hosts in an order that should allow them to come un synchronized. If there is a kernel patch, it's possible that the Infiniband network device will be down after reboot. Test for this by:

do-on-hosts --bgs ip link show dev ib0

It this fails then the drivers need to be installed:

do-on-hosts --timeout=180 --bgs bgfs-install-mellanox-drivers

This should install the drivers that were build on bg-mds-1t and copied over. No reboot should be required. 10/8/20 (JHJ):It's possible that the drivers could be installed as part of the BGFS patching operation.

Repeat the ip command above to double check.

usno_patch.sh apply-diskless

This will apply the patches to the recently created diskless image.

After it's back up check its status:

host-status swc-005

Note:The SWCs become unreachable via pinging after they start shutting down, echo a few pings and then become unreachable again until they finally reboot.

Check correlator status:

host-status --all

Note:The SWCs become unreachable via pinging after they start shutting down, echo a few pings and then become unreachable again until they finally reboot.

usno_patch.sh apply-host

Reboot.

usno_patch.sh apply-host

Reboot.

usno_patch.sh get-patches

Complete patch fetch on s1t before starting patch fetch on s1p.

On the test cluster this will pull down the patches from Red Hat for the servers, the SWCs and the bgfs servers. It will also bundle up the relevant parts of the home directory trees for root and admin along with /opt/services/bin (the idea is that server-1t is the reference copy for these files). On the production cluster, the script will copy over the update files prepared on server-1t (with current DC network setup this is especially awkward). See Remote access of server-1t from server-1p

nohup usno_sync-server-2.sh > /opt/services/bin/log/sync-server2.log & tail -f /opt/services/bin/log/sync-server2.log

This will synchronize important files from the primary server onto the corresponding secondary server. Since this only picks up mods since the previous day the operation should be fairly quick.

usno_patch.sh apply-host

Reboot.

host-status --all

The output of this command provides a quick status summary for the correlator systems. It can be handy to see how things were before system modifications are performed (e.g., an swc might be down prior to a diskless image page so if it won't come back up it's probably not the diskless image, etc.)

After the patching appears to succeed:

bgfs-restart-or-halt restart

This tries to restart the BGFS hosts in an order that should allow them to come un synchronized. If there is a kernel patch, it's possible that the Infiniband network device will be down after reboot. Test for this by:

do-on-hosts --bgs ip link show dev ib0

It this fails then the drivers need to be installed:

do-on-hosts --timeout=180 --bgs bgfs-install-mellanox-drivers

This should install the drivers that were build on bg-mds-1t and copied over. No reboot should be required. 10/8/20 (JHJ):It's possible that the drivers could be installed as part of the BGFS patching operation.

Repeat the ip command above to double check.

After it's back up check its status:

host-status swc-032

swc-016 was chose because it has dual Infiniband NICs.

Note:The SWCs become unreachable via pinging after they start shutting down, echo a few pings and then become unreachable again until they finally reboot.

After the systems stabilize:

host-status --all

Note:The SWCs become unreachable via pinging after they start shutting down, echo a few pings and then become unreachable again until they finally reboot.

usno_patch.sh apply-host

Reboot.

Do one final sanity check:

hostStatus.py --all