Casa Parallelization meeting minutes

Monday August 8th Room 317, 3:30pm MDT

Attendees: Brian, Jeff, Kumar, James

Targets/tasks for August 8th - 14th

• Continue on torque scheduler James
• Work toward testable pcube Kumar

Discussion/Agenda

• Justo visiting for 2 weeks drop
  • Working on async flagging, any updates ?
  • Writing async still problematic
• Move subtables to in-memory storage manager Jim
  • In testing
• Examine async thread failure problem Jim/James
  • On hold for in-memory storage manager
  • Seems stable in recent tests
• Spectral imaging Kumar
  • Parallelization status
    • Input output channel mapping is serial and slow, moving to actual processing engines now parallel
  • Nearing testable
• Complete parallel user interface frame work Jeff/Honglin
  • Memory usage increases in pcont as a function of number of cores vs spws due to data slice bug/feature
    • Work around implemented But doesn't seem to work
  • Shoot for end of July to have a proto parallel script
  • Miriam running into a problem where some data doesn't have calibrator and setjy fails
    • Need to differentiate bad selection from partition generating null selection in exception handler
• Refine and distribute cluster scheduler interface James
  • In progress, working with Kscott on multiple queue options
  • Awaiting proto parallel script
• Need 400+GB continuum case with complex field Sanjay
  • Formalized reduction status
  • Scripted reduction status
• External users
  • Josh Marvil using pclean, reports faster but missing options

Deferred Topics

• Test version of auto-flagging deferred
  • Few weeks from suitable for testing
• Parallel gridder testing James deferred
• MS-MFS performance vs image size James Deferred
  • Appears to be an L1-L2 cache issue on older systems (newer Nehalems have one cache line per core)
  • Consider other parallelization approaches
• Work toward implementing streaming framework in applycal Jim deferred
  • deferred for subtable storage manager rework

Priorities based on 3.3 release plans

• Parallel framework
• BDF size independent filler aka streaming filler
• Async I/O
• Multims spectral, continuum imaging
• auto-flagging

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels
• Broader parallelization of minor cycle for MS-MFS