Casa Parallelization meeting minutes

Monday March 28th Room 317, 3:30pm MDT

Attendees: Brian, Kumar, Jeff, Jim, Sanjay, James, Dan

Targets/tasks for March 28th - April 3rd

Discussion/Agenda

• Imaging issues
  • Identify pimager.py memory issue as a function of version Kumar
  • Imaging seems to use slightly more memory than expected
  • Confusing 100% utilization in minor cycle, possibly openmp spinning unused threads
  • filehandle limits a function of time/channel/field selection and reference ms
• Discussion of threaded grid cpu utilization Sanjay
  • CPU load correct ? Total number of threads ?
• Clarify and document parallel imaging framework Jeff/Kumar/Sanjay
• Resolve async issues with pcontmultims Jim
  • What is CPU saturation indicative
  • James to run using Jim's code (or test with single engine)
• Profile split Jim
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
• Cluster scheduler scripting with CV James
  • First pass of CASA/Torque interface Dan
  • Chris Mattman here from JPL to discuss OODT April 4th/5th
  • Clean up and make available James
• Test async performance against parallel continuum case James
  • Pending resolution to pcontmultims async issues
• Further refine continuum test timings James
• Examine streaming framework design Jim
• What is flagbackup=True in flagging and applycal ?
• General discussion on what's next
  • Where to draw the line on short term functional/usable vs longer term design.
    • Pending user interface, memory issues, filehandle issues

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)