Description

Some CASA Pipeline jobs do not complete successfully at JAO on 64 GB cluster machines, but do on machines with more memory at JAO and the NAASC.

This was initially posted to the HPC Mailing List.

Who

JAO - Fernando Morales, and...?

NRAO - Mike Hatz

What

From Mark Lacy:

"Here is the list that Tony sent to us of pipeline runs that failed at JAO that he would like rerun here (on 64GB machines):

I looked at the first 3, the first one should be quite small and quick to run, the next two are larger, but not huge."

MOUS uid:
uid://A001/X2fe/X418
uid://A001/X2fb/Xa11
uid://A001/X5a3/X11e
~~~~~~~~~~~~~~~~~~~~
uid://A001/X2fb/Xdd
uid://A001/X2d8/X101
uid://A001/X87a/X69c
uid://A001/X340/X14a
uid://A001/X2fa/X1d0
uid://A001/X2fb/Xc80
uid://A001/X2fe/X414
uid://A001/X2fb/X332
uid://A001/X2fb/X338

Please post the following diagnostic information (from an e-mail):

o OS on nodes at JAO and NAASC, and particularly is there an OS difference between client nodes at JAO.

o Lustre version on servers and clients, again is there a version difference between client nodes at JAO, e.g. do their older 64GB nodes run a different OS than their newer 128GB nodes

o The actual error as viewed from the shell running CASA (ie stderr and casa log output)

o lfs df -h output on a client that just exhibited the error (I'm guessing you'd see 'not available' messages for specific OSTs

o client kernel log messages that map to the time of the CASA error (I'd predict a raft of Lustre errors then).

Where

CVPOST017.CV.NRAO.EDU has been reserved for Fernando on 2016-11-01

This machine has 64 GB of memory until Saturday November 3 and will need to be rebooted as a "MEM=64GB" machine after that.

This activity will need Tests on 64GB machine at JAO - success or fail? Tests on 64GB machine at NAASC - success or fail? Tests on 128GB+ machine at JAO/NAASC - success or fail?

When

Request logged on 2016-11-01 to HPC Mailing List

Why

To prove or disprove the assertion that there are CASA Pipeline issue running some jobs on HPC machines with 64 GB of RAM.

Results at JAO:

-waiting for test output at 12PM 2016/NOV/08

- results (got from John Carpenter) (tests as of 2016/NOV/07 09AM (DMG group at JAO)):

- calPipeIF

- 64G: 5 success, 1 running

- 128G: 1 success

- imgPipeIF

- 64G: 1 success, 1 running

- 128G: 1 success, 2 running

- redPipeSD

- 64G: no data

- 128G: 1 running

- redPipeIF

- 64G: 1 success, 1 running

- 128G: no data

-- MikeHatz - 2016-11-03