Casa Parallelization meeting minutes

Monday January 9th Room 317, 3:30pm MDT

Attendees: James, Jeff, Brian, Kumar, Sanjay, Jim

Targets/tasks for January 9th - 16th

Discussion/Agenda

• Status of parallel framework
  • general discussion of current status per Sanjay's email
  • Jeff: good design for parallelization (reference MS, partition, imaging tasks, applycal)
  • Jeff: continuum and cube clean are parallel but not tasks
  • Jeff: efforts on other performance areas: async I/O, single data access model
  • Jeff: fix partitioning, fix tasks that require re-partitioning
  • Sanjay: Make a goal to achieve basic continuum imaging work in a robust way
  • Jeff: need to fix partition to handle null selection (Dirk will be examining)
  • Consensus: make task of pcube/pclean, DPetry to fix partitioning
  • Flagging not a formal part of group but needs to be addressed for end to end imaging
    • Need heuristics developed for EVLA flagging

• Raid tests
  • Async issues have stalled
• Lustre file I/O failure Jim/James
  • New version of lustre being installed January 12th
• Revisit async iterator for writing Jim/Justo
  • Refactoring seems to be working in active
  • Justo is testing flagging
  • Pointer rewrite in active
• Memory resident subtables Jim
  • Recent tests with storage manager inconclusive, James needs to retest
    • Check that it affects number of FDs as well
  • flagging infrastructure most likely next target (Justo might do)
  • Partition worth looking at
• Continuum imaging
  • Partition generates empty selection in certain circumstances
  • Exception needs to be handled
• Spectral imaging Kumar
  • Understand channel chunk choice effect on memory James
• Complete parallel user interface frame work Jeff/Honglin
  • Status
• Refine and distribute cluster scheduler interface James
  • Ready for testing
• Need 400+GB continuum case with complex field
  • Stalled for partitioning
• Version 3 Filler status
  • In testing

CASA HPC Initiatives for the 3.4 Cycle

• Memory Footprint Issues
  • Deployment of Multi-Field MS-MFS (Urvashi)
  • Deployment of Parallel Gridder and splitting of the Grid (Jim)
  • Design of MS-MFS Minor Cycle Parallelization (Jim et. al.) using Friday meetings
• Initial Deployment of ASP Clean (Sanjay)
• Performance Issues:
  • Virtual Model Data Column (Kumar)
    • Removes model data column
    • removes need for Setjy
  • Memory Resident Subtables Deployment (Identify list of tasks)
  • VisBuffer Processing Framework
    • Add Write capability (Jim) IN TESTING
    • Deployment in tasks (Jeff to Assign)
  • Parallel Filler (and partitioning) (Tak)
• User Interface:
  • Parallel Imager in Task (Kumar)
  • Parallelization of other Tasks (Kumar and Jeff)

Deferred Topics

• Test version of auto-flagging deferred

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels