Casa Parallelization meeting minutes

Monday July 18th Room 317, 3:30pm MDT

Attendees: Kumar, Jim, Jeff, James, Justo

Targets/tasks for June 18th - July 24th

  • Create a local account for Justo James
  • Test CalMS selection in partition James
  • Fill 400GB data set Sanjay

Discussion/Agenda

  • Justo visiting for 2 weeks
    • Working with Jim on current state of affairs
    • Working on async flagging
      • MS selection appears to be slow, wrong sort order most likely
  • Examine async thread failure problem Jim/James
    • Memory resident subtables may help
    • Move subtables to in memory storage rather than storage manager Jim
  • Simon Radcliffe from SKA
    • Do we want to track A/W projection and GPU work here or is it too early
  • Spectral imaging Kumar
    • Parallelization status
      • Issue with more processes than number of MSes fixed
      • Input output channel mapping is serial and slow, moving to children
  • Complete parallel user interface frame work Jeff/Honglin
    • Memory usage increases in pcont as a function of number of cores vs spws due to data slice bug/feature
    • Shoot for end of month to have a proto parallel script
    • Miriam running into a problem where some data doesn't have calibrator and setjy fails
      • Need to differentiate bad selection from parallell generating null selection in exception handler
  • Refine and distribute cluster scheduler interface James
    • In progress, awaiting proto parallel script
    • Potentially use OODT on order of a month
  • Need 400+GB continuum case with complex field Sanjay
    • Sanjay will fill to an MS that others can read from
    • Observed last Friday

Deferred Topics

  • Test version of auto-flagging deferred
    • Few weeks from suitable for testing
  • Parallel gridder testing James deferred
  • MS-MFS performance vs image size James Deferred
    • Appears to be an L1-L2 cache issue on older systems (newer Nehalems have one cache line per core)
    • Consider other parallelization approaches
  • Work toward implementing streaming framework in applycal Jim deferred
    • deferred for subtable storage manager rework

Priorities based on 3.3 release plans

  • Parallel framework
  • BDF size independent filler aka streaming filler
  • Async I/O
  • Multims spectral, continuum imaging
  • auto-flagging

ARDG Targets

  • What is negative side effect of CVel in filler, non-reversible
  • What needs to be done to acceptably implement Auto-boxing with large number of channels
  • Broader parallelization of minor cycle for MS-MFS

-- JamesRobnett - 2011-07-25
Topic revision: r1 - 2011-07-25, JamesRobnett
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback