Casa Parallelization meeting minutes

Tuesday May 10th Room 317, 11:00am MDT

Attendees: Jim, Jeff, James, Brian, Sanjay, Kumar and visitors from LOFAR

Targets/tasks for May 10th - May 16th

Discussion/Agenda

• LOFAR visitors
  • Examining W and A projection suitability for LOFAR
• Imaging issues
  • Identify pimager.py memory issue(s) Kumar
  • Imaging seems to use slightly more memory than expected
  • New memory leak in imaging as a function of Nehalem series (56xx vs 54xx or 55xx)
    • Seems to be fixed but not in test version
• async status Jim
  • Thread failure in active and test builds, may simply be a check-in issue
  • serializing/deserializing objects can decouple them such that destruction is safe
  • Ger may have a solution
  • Build against Ger's checkin's this week
• Parallel gridder status Sanjay
  • Memory scales weakly by number of cores
  • Should work with various gridder algorithms
    • A-projection gridder doesn't quite work Fixed
    • error in distributing grids to cores, results in missing data
  • C++ vs Fortran performance, possibly a compiler issue
    • May just be a reflection of GCC optimization flags
  • Need code snippet to show to some external folks and possibly give to nVidia for GPU purposes (memory impact)
    • Calculate A/W terms on the fly in GPU rather than store in memory (month or two to get code to nVidia)
    • Consider DFT's in parallel in GPU
  • LOFAR sees similar memory pressures
• Examine streaming framework design Jim
  • Continue to work on in parallel with async split
  • Profile split Jim
    • First task to obtain new framework is applycal
    • Main hotspot is ordering of two iteration loops, gaincal and spectral windows triggers
  • Week or two for initial tests
• Further develop parallel script frame work Jeff/Honglin
  • Pending reference-MS approach testing, auxiliary table creation problems
  • Parallel version of split in scan mode
  • Pass a list of tasks+parameters (dictionary) framework
    • Pclean needs to take a reference ms rather than multims
• Cluster scheduler scripting with CV James
  • Clean up and make available James
  • Pass on inhouse script and focus on OODT ?
• Test async performance against parallel continuum case James
  • Need to test vs various hardware configs
• Further refine continuum test timings James
• General discussion on what's next

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)