Casa Parallelization meeting minutes

Monday March 21st Room 317, 3:30pm MDT

Attendees: Bryan, Brian, Jim, James, Sanjay, Dan Klopp

Targets/tasks for March 21st - March 27th

• Resolve pimager.py memory issue Kumar
• Resolve async issues with pcontmultims Jim
• Complete cpu profiling of split Jim
• Further refine continuum test timings James
• First pass of CASA/Torque interface Dan/James

Discussion/Agenda

• Imaging issues
  • Current pimager.py has memory leak Kumar
  • Imaging seems to use more memory than expected
  • Confusing 100% utilization in minor cycle, possibly openmp spinning unused threads
  • filehandle limits a function of time/channel/field selection and reference ms
• Profile split Jim
  • Seems to be in type casting from bit to booleans
  • Kumar email oprofile pointer
• Resolve async issues with pcontmultims
• Examine streaming framework design Jim
• Clarify and document parallel imaging framework Jeff/Kumar/Sanjay
• Test async performance against parallel continuum case James
  • Pending resolution to pcontmultims issues
• Further refine continuum test timings James
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
  • Some discussion about who should be involved in user interface
• Cluster scheduler scripting with CV James
  • Dan Klopp here from March 8th to 28th
  • First pass at CASA/Torque interface nearly done
  • Chris Mattman here from JPL to discuss OODT April 4th/5th
• Confirm memmap vs cache performance behavior James
• General discussion on what's next
  • Defer till next week

Older Issues

• Continued discussion of threaded gridding approach Sanjay/Jeff/James defer for now
• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure to test by friday 2/4/11
    • When is less clear now
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)