Cluster Scheduling

Overview

The NRAO has a computer cluster for running CASA. A resource manager, Torque, and a scheduling program, Maui, are required to allow multiple users to use this cluster efficiently.

Torque is a distributed resource manager and can control the execution and termination of batch jobs. It is this last functionality, the control of batch jobs, that causes a problem when using torque to control CASA. Torque expects to start and stop the batch jobs on computers for which it is responsible. Unfortunatly, so does CASA. Rather than retool CASA to work with Torque we decided it was easier to let Torque select the nodes to use and then provide that list of nodes to CASA.

Documentation

Troubleshooting

  • rm: cannot remove directory `./regressionTemp/TDEM0003.data/TDEM0003.0007.ms': No such file or directory
    try rm -rf regressionTemp/*
  • Unable to copy file
    make sure your shell is in a directory to which you can write when you submit your job. Otherwise, you may receive errors.

TODO Items

  • AOC
    • We need a name for the nm-pipe cluster so I can rename nodescheduler and submit to something better.
    • How do we handle multiple queues for fowen, cfa, berkely, caltech, public machines, etc
    • If we use torque to reserve public machines, how can we have multiple users reserve one machine at the same time? We currently allow one local and one remote user per public machine.
    • It would be nice if a user could submit, terminate and check the status of jobs in the cluster from their desktop instead of having to login to some specific machine like nm-pipe-1. However this may not be feasible because it may require such desktop machines to be configured in torque.
      • see Torque docs on this
      • also the -q option to qsub could be helpful. It may allow me to put qsub in /usr/local/bin without /opt/services/torque
      • qstat installed torque in /opt/local. This allows qstat to be run from any machine.
      • qsub
      • qdel
    • Create a vla pipeline queue.
    • If the user submitting the job from a non-host-keyed machine, doesn't have ssh setup to work without a password then an email error is generated when Torque tries to scp the setout/stderr stream.
      • qsub -j doesn't help because it only either combines or doesn't combine the stout and stderr streams
      • qsub -e doesn't help because no matter the path specified Torque always uses scp.
      • qsub -k doesn't help. The -k [eo] tells Torque to write directly to the users home area. -k n tells Torque to write to the torque area and then scp.
    • How do we get qsub -u to work?
      • setup /etc/hosts.equiv with hostnames and usernames
    • We will need a way to reserve like one machine for the "clean" process with a lot of memory (192G) and several for the "gridding" process with moderate memory (24G).
      • Moab has a feature for submitting sub jobs that may solve this.
      • A more simple solution may be to just use Node Properties.
        • e.g. qsub -l nodes=1:bigmem+7 will request one "bigmem" node and seven regular nodes.
    • Need to figure out how to have jobs automaticly start after they have been queued (status Q) because of missing resources when resources become available.

Done Items

  • AOC
    • The email sent by torque from abm@aoc.nrao.edu at the beginning and ending of each job is not very useful to an average user.
      SOLUTION: setup smarter cluster_job_begun.sh and cluster_job_ended.sh scripts to send email
    • a Final email reading that time has expired may be nice.
      SOLUTION: setup smarter cluster_job_begun.sh and cluster_job_ended.sh scripts to send email
    • will the -n option to qsub do the same as specifying all cores?
      SOLUTION: Actually, I created a sepearate "interactive" queue and set the following in maui.cfg
      QOSCFG[interactive_qos] QFLAGS=DEDICATED
      CLASSCFG[interactive] QDEF=interactive_qos
    • using nodescheduler to request multiple interactive nodes only sets up access.conf and the sleep job on one node.
      • this may be because I am only using qsub instead of qsub to start pbsdsh.
      • Solution: use prologue.parallel and epilogue.parallel scripts.
    • Can I setup a special queue with prologue and epiloque scripts to do what submit and yaw does?
      • No need as yaw is no longer used
    • How can we extend the requested time of an interactive job, especially once it is running.
      • SOLUTION: qalter -lwalltime=hours:minutes:seconds Job_ID

References

Topic attachments
I Attachment Action Size Date Who Comment
control_logins.shsh control_logins.sh manage 1 K 2011-12-08 - 08:07 DanielKlopp Run from cron job to control logins
nodeschedulerEXT nodescheduler manage 11 K 2012-08-08 - 13:16 KScottRowe Interactive node scheduler
nodescheduler.pdfpdf nodescheduler.pdf manage 24 K 2012-08-08 - 12:51 KScottRowe Interactive time request process
reservenodeEXT reservenode manage 401 bytes 2011-12-08 - 08:06 DanielKlopp Script to reserve node on interactive scheduler
submitEXT submit manage 3 K 2012-08-07 - 19:00 KScottRowe Script the user executes to submit a casa job.
submit.pdfpdf submit.pdf manage 49 K 2012-08-07 - 18:57 KScottRowe CASA submition process
Topic revision: r54 - 2014-01-24, KScottRowe
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback