Casa Parallelization meeting minutes

Monday June 6th Room 317, 3;30pm MDT

Attendees: Bryan, Brian, Kumar, Jeff, James, Sanjay

Targets/tasks for June 6th - June 12th

Discussion/Agenda

• Examine async thread failure problem Jim/James
  • In progress
• Work toward implementing streaming framework in applycal Jim
  • Initial Jim testing, not ready for general testing
  • deferred for subtable storage manager rework
• Rework subtable storage manager Jim
  • Suggested by Ger in favor of AIPSmgr
• Refine and distribute cluster scheduler interface James
  • Delayed due to vacations
• Spectral imaging Kumar
  • Serial improvements
  • Parallelization status
    • Discussion on how to split MSes
• Consider Cvel in filler
• Parallel gridder status Sanjay
  • Needs further optimization
• Complete parallel user interface frame work Jeff/Honglin
  • In progress this week
• Need 300+GB continuum case with complex field Bryan
  • Bryan will look into observing (spectral line and extended emission)
  • Potential one from James Miller Jones
  • Should use if possible proto-pipeline from Miriam
• MS-MFS performance vs image size James
  • Appears to be an L1-L2 cache issue on older systems (newer Nehalems have one cache line per core)
  • Consider other parallelization approaches (ie not by scale).
• General discussion on what's next
  • Priorities based on 3.3 release plans
    • Parallel framework IN
    • BDF size independent filler aka streaming filler IN
    • Async I/O IN
    • Streaming framework not in
    • ms-mfs not in
    • Multims spectral, continuum IN
    • auto-flagging IN

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger