Casa Parallelization meeting minutes

Monday April 25th Room 317, 3:30pm MDT

Attendees: Jim, Jeff, James, Kumar, Bryan, Brian

Targets/tasks for April 25th - May 8th.

Discussion/Agenda

• Imaging issues
  • Identify pimager.py memory issue(s) Kumar
  • Imaging seems to use slightly more memory than expected
  • New memory leak in imaging as a function of Nehalem series (56xx vs 54xx or 55xx)
    • Isn't clearing internal to CASA tile cache
    • Affects clean so is a release blocker
• Parallel gridder status Sanjay
  • Memory scales weakly by number of cores
  • Need a preparation thread for visbuffer
  • Should work with various gridder algorithms
    • A-projection gridder doesn't quite work
    • error in distributing grids to cores, results in missing data
• Clarify and document parallel imaging framework Jeff/Kumar/Sanjay DONE
  • Will there be an eventual document
• async status Jim
  • Implement in split with time averaging or apply cal
• Examine streaming framework design Jim
  • Continue to work on in parallel with async split
  • Profile split Jim
    • Overview of Jim's report
    • Main hotspot is ordering of two iteration loops, gaincal and spectral windows triggers
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
  • Parallel version of split in scan mode
  • Pass a list of tasks+parameters (dictionary) framework
    • Pclean needs to take a reference ms rather than multims
• Cluster scheduler scripting with CV James
  • Clean up and make available James
• Test async performance against parallel continuum case James
  • Need to test vs various hardware configs
• Further refine continuum test timings James
• General discussion on what's next
  • Things generally no hold due to meetings and release till May 9th.
  • Ger and Co. will be here May16-27,
  • Dirk will be here 21-27
  • Shinnnosukae having problems building with Intel Compiler

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)