Casa Parallelization meeting minutes

Monday November 14th Room 317, 3:30pm MDT

Attendees: Brian, Kumar, Sanjay, Jeff, Jim, James

Targets/tasks for November 14th - 21st

• James Further RAID tests
• Jim Check with Justo on async write, and memory mgr inclusion
• James Retest pcube with async prior to next active build
• James Re-examine Miriam's case

Discussion/Agenda

• Raid tests
• Lustre file I/O failure Jim/James
  • Seems to be working
  • Consequence for things like lustre
  • New version of lustre
• Revisit async iterator for writing Jim/Justo
  • Refactoring seems to be working in active
  • Justo is testing flagging
  • Pointer rewrite in active possibly this week
• Move subtables to in-memory storage manager Jim
  • Recent tests with storage manager inconclusive, James needs to retest
    • Check that it affects number of FDs as well
  • flagging infrastructure most likely next target (Justo might do)
  • Partition worth looking at
• Continuum imaging
  • Partition generates empty selection in certain circumstances fixed
• Spectral imaging Kumar
  • Understand channel chunk choice effect on memory James
• Complete parallel user interface frame work Jeff/Honglin
  • Status
• Refine and distribute cluster scheduler interface James
  • In progress, Kscott working on implementation and user documentation]
  • Bug in node partitioning case (6 engines one node)
• Need 400+GB continuum case with complex field
  • Formalized reduction status
  • Scripted reduction status
• Version 3 Filler status
  • In test, using Jirwin data sets
  • Michel

CASA HPC Initiatives for the 3.4 Cycle

• Memory Footprint Issues
  • Deployment of Multi-Field MS-MFS (Urvashi)
  • Deployment of Parallel Gridder and splitting of the Grid (Jim)
  • Design of MS-MFS Minor Cycle Parallelization (Jim et. al.) using Friday meetings
• Initial Deployment of ASP Clean (Sanjay)
• Performance Issues:
  • Virtual Model Data Column (Kumar)
    • Removes model data column
    • removes need for Setjy
  • Memory Resident Subtables Deployment (Identify list of tasks)
  • VisBuffer Processing Framework
    • Add Write capability (Jim) IN TESTING
    • Deployment in tasks (Jeff to Assign)
  • Parallel Filler (and partitioning) (Tak)
• User Interface:
  • Parallel Imager in Task (Kumar)
  • Parallelization of other Tasks (Kumar and Jeff)

Deferred Topics

• Test version of auto-flagging deferred

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels