Casa Parallelization meeting minutes

Friday August 20

Attendees:

Targets/tasks for August 20-27

• Time domain partitioning in continuum imaging Kumar
• Enable multiple MSes per imaging process Kumar
• Identify labor for End-to-End tests Brian Bryan
• Further Lustre and I/O manager testing James
• Create CASA parallelization priority list James/Bryan/Brian
• Check on multi-ms viewer Sanjay stalled till Ger gets back
• Implement proto-parallel calibration task Sanjay (full version to be assigned)

Discussion/Agenda

• Time domain partitioning in continuum imaging Kumar
  • Delayed for work on multiscale-clean performance issues
• Enable multiple MSes per imaging process Kumar
  • Delayed for work on multiscale-clean performance issues
• Examine Multiscale-clean performance issue in object copy Kumar DONE
  • Improved/fixed
• Prototype I/O manager implemented James DONE
  • Initial tests show expected speed up. ~70 seconds faster across a 7200MB/s data set.
  • Some discussion on relative merits of by file (expediency) vs table (versatility) buffering
    • Currently doing by file
      • There are some implications as to tile size for by file buffering, tile size needs be around 256MB
      • Some advantage to splitting MS on scan bounderies since were limiting scan length anyway
    • Table based has less limitation on MS size and may be more adaptable to other tasks (eg calibration)
• Create CASA parallelization priority list James/Bryan/Brian
  • Need to focus on End-to-end tests try to start next week
  • Initial script for filler -> calibration -> flagging -> split -> imaging (eventually move split post-filler)
    • Who to be decided by Brian and Bryan, potentially Ed Green or Lindsey Davis
    • Target 15 to 20GB interesting data sets to be run on a single node
    • Consider folding in the summer school scripts
• Understand planning for time segregation of filler and MS independent calibration Brian DONE
  • Labor dependencies a little clearer
  • Needs to wait till next cycle, effectively a few weeks
  • Currently low priority but at beginning of next cycle immediately escalates as it blocks other projects.
• Look into status of M. Caillat and parallel filler (Bryan or Brian ?)
  • Currently low priority
  • Needed to reduce memory requirement on filler host and improve general performance
• Check on multi-ms viewer Sanjay stalled till Ger gets back
• Implement proto-parallel calibration task Sanjay (stalled till Sanjay gets back)