Casa Parallelization meeting minutes

Monday June 13th Room 317, 3:30pm MDT

Attendees: Jim, James, Kumar, Jeff

Targets/tasks for June 13th - June 19th

Discussion/Agenda

• Examine async thread failure problem Jim/James
  • In progress
• Move subtables to in memory storage rather than storage manager Jim
  • Suggested by Ger in favor of AIPSio storage manager
• Work toward implementing streaming framework in applycal Jim
  • deferred for subtable storage manager rework
• Spectral imaging Kumar
  • Serial improvements
  • Parallelization status
    • Discussion on how to split MSes
    • Split on spectral window by default
• Consider implementing Cvel in filler drop for now but possibly a good idea
• Complete parallel user interface frame work Jeff/Honglin
  • Rob is implementing partition so can split by scan, channel and time
  • Need to get with Honglin on cluster specfication
  • In progress this week
• Refine and distribute cluster scheduler interface James
  • Delayed due to vacations
• Parallel gridder status Sanjay
  • Needs further optimization
• Need 300+GB continuum case with complex field Bryan
  • Bryan will look into observing (spectral line and extended emission)
  • Potential one from James Miller Jones
  • Should use if possible proto-pipeline from Miriam
• MS-MFS performance vs image size James
  • Appears to be an L1-L2 cache issue on older systems (newer Nehalems have one cache line per core)
  • Consider other parallelization approaches
• General discussion on what's next
  • Priorities based on 3.3 release plans
    • Parallel framework
    • BDF size independent filler aka streaming filler
    • Async I/O
    • Multims spectral, continuum imaging
    • auto-flagging

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger