Casa Parallelization meeting minutes

Monday February 28th Room 317, 3:30pm MDT

Attendees: James, Bryan, Jim, Sanjay, Jeff

Targets/tasks for February 28th - March 6th

• Confirm memmap vs cache performance behavior James
• Resolve auxiliary tables issues with listobs Sanjay
• Resolve reference-ms and auxiliary table issues Jeff
• Test/confirm serial performance of non-parallel tasks against multims ???
• Test async performance against parallel continuum case James
• Further refine continuum test timings James
• Identify small read causes in split James/Jim
• Continued discussion of threaded gridding approach Sanjay/Jeff/James

Discussion/Agenda

• Convert default from memmap to cache update. Jeff
  • Done in test and active
• Make listobs multi-ms compatible Sanjay
  • Nearly done
  • Problem with auxiliary and sub-tables (antenna, spectral, feed)
  • Allows non-parallel tasks like plotms to continue to function
  • Need to test serial performance of non-parallel tasks against multims
• Status of Async I/O Jim
  • Some existing imaging problems.
    • Last known problem appears to be corrected
  • Integrated with large continuum test case but relatively untested
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
• Test end script and generate timings James
  • Some initial timings exist for single node 1,2,4 and 8 engines and 2 to 4 node 16 engines
  • Small read performance dominates results
  • Jim will look into small read reasons
  • Refine scripts, create generalized script
  • Major cycle, minor cycle wall clock time
• Cluster scheduler scripting with CV James
  • Dan Klopp here from March 8th to 28th
  • Discuss with Jim semantics for cluster/CASA interaction
• General discussion on what's next
  • Global design for how split, filler, flagging will work
  • Script vs task level
    • Honglin and Jeff to work on parallel script framework
    • Need to consider training local staff
  • Multi-threaded access in gridding

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure to test by friday 2/4/11
    • When is less clear now
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)