Casa Parallelization meeting minutes

Monday July 11thth Room 317, 3:30pm MDT

Attendees: James, Jeff, Sanjay, Kumar, Bryan, Simon Radcliffe

Targets/tasks for June 11th - July 17th

Discussion/Agenda

• Simon Radcliffe from SKA sitting in
  • Workiing with Sanjay on GPU implementations
  • A/W projection and gridding
  • First target is getting convolution server working
• Examine async thread failure problem Jim/James
  • Memory resident subtables may help
  • Move subtables to in memory storage rather than storage manager Jim
• Work toward implementing streaming framework in applycal Jim
  • deferred for subtable storage manager rework
• Spectral imaging Kumar
  • Parallelization status
    • Generates correct results
    • Performance issue putting images back together plane by plane is slow Fixed
    • Issue with more processes than number of MSes
• Complete parallel user interface frame work Jeff/Honglin
  • In testing: flagging, calibration seem to work
  • Confusion with respect to memory consumption in pcont
  • Shoot for end of month to have a proto parallel script
• Refine and distribute cluster scheduler interface James
  • Re-addressing this week (this time for real), needs to be ready for proto parallel script above
  • Potentially use OODT on order of a month
• Need 400+GB continuum case with complex field Sanjay
  • 3c147 in L band 8 hour observation set up by Sanjay
    • Will still try to observe but may get replaced with a night time observation
  • Michel Brenjens may generate scheduling block
• Test version of auto-flagging
  • Few weeks from suitable for testing
• Parallel gridder testing James
• MS-MFS performance vs image size James Defer for now
  • Appears to be an L1-L2 cache issue on older systems (newer Nehalems have one cache line per core)
  • Consider other parallelization approaches
• Fits to MS to Fits tile shape Drop
• Tsys table in uvexportfits brought up

Priorities based on 3.3 release plans

• Parallel framework
• BDF size independent filler aka streaming filler
• Async I/O
• Multims spectral, continuum imaging
• auto-flagging

ARDG Targets

• What is negative side effect of CVel in filler, non-reversible
• What needs to be done to acceptably implement Auto-boxing with large number of channels
• Broader parallelization of minor cycle for MS-MFS