Casa Parallelization meeting minutes

Monday January 30th Room 317, 3:30pm MDT

Attendees: Jim, James, Bryan, Kumar, Sanjay, Jeff

Targets/tasks for January 30th - February 5th

Discussion/Agenda

• Partitioning status (null selections, non-conforming tasks) Jeff
  • Dirk not started yet on design
  • Memory consumption needs to be addressed
• asdm2ms_xl is in test targeted for next stable
  • Does not handle large syspower tables
  • Still serial, eventually multi-threaded across spw, multi-process across scans
• Pcube and pclean as task after model data work Kumar
• Model data is in
  • Reduces data size from 3x to 2x
  • gaincal recreates scratch column should be fixed soon
• Raid tests
  • Async issues have stalled
• Lustre file I/O failure Jim/James
  • New version of lustre being installed January 12th
  • Need to test whether it's actually fixed
• Revisit async iterator for writing Jim/Justo
  • Refactoring seems to be working in active
  • Justo is testing flagging
  • Pointer rewrite in active
• Memory resident subtables Jim
  • Recent tests with storage manager inconclusive, James needs to retest
    • Check that it affects number of FDs as well
  • flagging infrastructure most likely next target (Justo might do)
  • Partition worth looking at
• Flagging being examined for parallelizatin, rflag agent
• Begin re-examination of multithreaded gridder
• Spectral imaging Kumar
  • Understand channel chunk choice effect on memory James
• Refine and distribute cluster scheduler interface James
  • Ready for testing
• Need 400+GB continuum case with complex field
  • Stalled for partitioning

CASA HPC Initiatives for the 3.4 Cycle

• Memory Footprint Issues
  • Deployment of Multi-Field MS-MFS (Urvashi) Deferred
  • Deployment of Parallel Gridder and splitting of the Grid (Jim)
  • Design of MS-MFS Minor Cycle Parallelization (Jim et. al.) using Friday meetings Deferred
• Initial Deployment of ASP Clean (Sanjay)
• Performance Issues:
  • Virtual Model Data Column (Kumar) Done
    • Removes model data column
    • removes need for Setjy
  • Memory Resident Subtables Deployment In progress needs testing
  • VisBuffer Processing Framework
    • Add Write capability (Jim) Need to revisit but seems to work in not imaging
    • Deployment in tasks (Jeff to Assign)
  • Parallel Filler (and partitioning) (Tak) In progress
• User Interface:
  • Parallel Imager in Task (Kumar)
  • Parallelization of other Tasks (assigned to Europe)

Deferred Topics

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels