Casa Parallelization meeting minutes

Monday June 27th Room 317, 3:30pm MDT

Attendees: Kumar, James, Jeff, James, Jim, Sanjay

Targets/tasks for June 27th - July 10th

• Give Kumar 3 MS, 16 spw, 16 time and 16spwsx2time domains Jeff
• 3c147 in L band 8 hour observation set up by Sanjay

Discussion/Agenda

• No meeting next week
• Examine async thread failure problem Jim/James
  • Memory resident subtables may help
  • Move subtables to in memory storage rather than storage manager Jim
    • In early testing stage
    • Discussion of whether it should go in casacore portion of MS or in visiterator
    • Suggested by Ger in favor of AIPSio storage manager
• Work toward implementing streaming framework in applycal Jim
  • deferred for subtable storage manager rework
• Spectral imaging Kumar
  • Parallelization status
    • Generates correct results
    • Performance issue putting images back together plane by plane is slow
• Complete parallel user interface frame work Jeff/Honglin
  • In testing: flagging, calibration seem to work
  • Confusion with respect to memory consumption in pcont
  • May get checked in this week
  • Slow scan selection fixed
• Refine and distribute cluster scheduler interface James
  • Re-addressing this week,
• Parallel gridder status Sanjay Defer for now
  • Needs further optimization on hold
• Need 400+GB continuum case with complex field Sanjay
  • 3c147 in L band 8 hour observation set up by Sanjay
  • Michel Brenjens may generate scheduling block
• MS-MFS performance vs image size James Defer for now
  • Appears to be an L1-L2 cache issue on older systems (newer Nehalems have one cache line per core)
  • Consider other parallelization approaches
• Test version of auto-flagging
  • Few weeks from suitable for testing here
• Give Kumar 3 MS, 16 spw, 16 time and 16spwsx2time domains

Priorities based on 3.3 release plans

• Parallel framework
• BDF size independent filler aka streaming filler
• Async I/O
• Multims spectral, continuum imaging
• auto-flagging

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels
• Broader parallelization of minor cycle for MS-MFS