Do-on-hosts

This python script is designed to execute a command on a set of cluster hosts. It is invoked via the link do-on-hosts located in /opt/services/bin while the script itself lives as /opt/services/bin/python/do_on_hosts.py.

Help Output

15:56 root@S-1-p python]# do-on-hosts --help
usage:  [-h] [--all] [--servers] [--ss] [--mds] [--bgs] [--hosts HOSTS]                           
        [--swcs] [--timeout TIMEOUT] [--quiet]                                                    
        ...                                                                                       
                                                                                                  
positional arguments:                                                                             
  command            Command to perform on all hosts                                              
                                                                                                  
optional arguments:                                                                               
  -h, --help         show this help message and exit                                              
  --all              Do on all hosts                                                              
  --servers          Do on servers                                                                
  --ss               Do on BG storage servers                                                     
  --mds              Do on BG metadata servers                                                    
  --bgs              Do on all BG servers                                                         
  --hosts HOSTS      Comma separated list of hosts                                                
  --swcs             Do on SWCs                                                                   
  --timeout TIMEOUT  Timeout period for commands in seconds                                       
  --quiet            Minimal output                                                               
                                                                                                  
to pass options to make use --args args...                                                        
[16:34 root@S-1-p python]#                                                                       
                             

After the switches have been supplied, the rest of the command line is take to be the command to be executed on the specified hosts. For common groups of hosts, switches are provided; switches can be combined to cover more hosts (e.g., --ss --swcs to specify the storage servers and the SWCs). Custom sets of hosts can be specified using the --hosts switch followed by a comma-separated list of hosts (e.g., --hosts a,b,c or --hosts=a,b,c).

Some operations may require a larger than default timeout to run to completion: thus the --timeout switch.

Fancier commands should be enclosed in quotes so as to not have them intercepted by the local shell:

• do-on-hosts --swcs "mount | grep opt/services/bin"

The commands will be executed remotely using the current username. Using su aUser before do-on-hosts or including sudo in the command given to do-on-hosts can mitigate this somewhat.

Example

In this example the uptime of the SWCs is desired. The output shows the output for each SWC in order. Unreachable hosts are summarized (these hosts don't really exist on the test cluster but were put there to test the effect of missing hosts).

[16:37 root@S-1-p python]# do-on-hosts --swcs uptime
['swc-001', 'swc-002', 'swc-003', 'swc-004', 'swc-005', 'swc-006', 'swc-007', 'swc-008', 'swc-009\
', 'swc-010']
--------- swc-001
16:41:19 up 45 days, 23:26,  0 users,  load average: 0.00, 0.01, 0.05
--------- swc-002
16:41:19 up 46 days, 55 min,  0 users,  load average: 0.00, 0.01, 0.05
--------- swc-003
16:41:19 up 46 days, 55 min,  0 users,  load average: 0.00, 0.01, 0.05
--------- swc-004
16:41:19 up 46 days, 55 min,  0 users,  load average: 0.00, 0.01, 0.05
--------- swc-005
16:41:19 up 46 days, 55 min,  0 users,  load average: 0.00, 0.01, 0.05
--- Unreachable hosts: swc-006..010
[16:41 root@S-1-T python]# 

-- JimJacobs - 2020-08-04