Casa Parallelization meeting minutes

Monday March 7th Room 317, 3:30pm MDT

Attendees: Brian, Jeff, Jim, Sanjay, James

Targets/tasks for March 7th - March 13th

• Identify/remove small read causes in split James/Jim
• Examine streaming framework design Jim
• Identify cause of large number of open file descriptors in imaging Jeff
• Confirm memmap vs cache performance behavior James
• Test async performance against parallel continuum case James
• Further refine continuum test timings James
• Further develop parallel script frame work Jeff/Honglin
• Cluster scheduler scripting with CV James

Discussion/Agenda

• Identify small read causes in split James/Jim
  • Feed table look up, small bucket and cache size
  • cachesize defaults to 2
  • bucketsize is 32 * some default column size
  • Can we use slurp to be more effecient
    • Slurp is visiter function
  • Are there existing memory managers we can use for small tables
  • Where are we selecting storage manager when creating sub MSes
  • Using a memory storage manager that has an on disk representation for handling sub tables
• Confirm memmap vs cache performance behavior James
  • Not done
• Resolve auxiliary tables issues with listobs Sanjay *DONE
  • Resolved for 3.2 release, all reference-MSes have same structure
  • Breaks for spectral parallelization because split remumbers spectral windows
    • Stop renumbering subtables
  • Time separation approach for spectral line would obviate renumbering issue but would require scatter/gather
• Resolve reference-ms and auxiliary table issues Jeff DONE
  • Ger has checked in code all is code.
• Test/confirm serial performance of non-parallel tasks against multims ??? DONE
  • Jeff examined no real difference
• Test async performance against parallel continuum case James
  • Not done
• Further refine continuum test timings James
  • Pending new version from Jeff
  • Decouple parallel portion from project specific
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
  • Some discussion about who should be involved in user interface
• Test end script and generate timings James
  • Refine scripts, create generalized script
• Cluster scheduler scripting with CV James
  • Dan Klopp here from March 8th to 28th
  • Discuss with Jim semantics for cluster/CASA interaction
• General discussion on what's next
  • Becoming test bound, need to identify testers *

Older Issues

• Continued discussion of threaded gridding approach Sanjay/Jeff/James defer for now
• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure to test by friday 2/4/11
    • When is less clear now
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)