Casa Parallelization meeting minutes

Monday April 18th Room 317, 3:30pm MDT

Attendees: Bryan, James, Jeff, Kumar, Sanjay, Jim

Targets/tasks for April 18th - April 24th

• Continue to examine memory issues in imaging Kumar
• Parrallel gridder development Sanjay
• Examine async in split Jim
• Advance parallel script frame work Jeff
• Further timing tests *James8

Discussion/Agenda

• Imaging issues
  • Identify pimager.py memory issue(s) Kumar
  • Imaging seems to use slightly more memory than expected
• Parallel gridder status Sanjay
  • Memory scales weakly by number of cores
  • Need a preparation thread for visbuffer
  • Should work with various gridder algorithms
    • A-projection gridder doesn't quite work
• Clarify and document parallel imaging framework Jeff/Kumar/Sanjay
• async status in pcontmultims Jim
  • seems to work as intended
  • consider implementing in split with time averaging or apply cal
• Profile split Jim
  • Overview of Jim's report
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
  • Parallel version of split in scan mode
  • Pass a list of tasks+parameters (dictionary) framework
    • Pclean needs to take a reference ms rather than multims
• Cluster scheduler scripting with CV James
  • Clean up and make available James
• Test async performance against parallel continuum case James
  • Need to test vs various hardware configs
• Further refine continuum test timings James
• Examine streaming framework design Jim
  • Continue to work on in parallel with async split
• General discussion on what's next

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)