Casa Parallelization meeting minutes

Monday February 14th Room 317, 3:30pm MDT

Attendees: Bryan, Brian, Jim, James, Jeff, Kumar, Sanjay

Targets/tasks for February 14th - February 20th

• Convert default from memmap to cache. Jeff
• Resolve Async I/O issues Jim
• Integrate async I/O with large continuum case Kumar
• Test end script and generate timings James
• Make listobs multi-ms compatable Sanjay
• Further develop parallel script frame work Jeff/Honglin

Discussion/Agenda

• On the fly mosaicing doesn't need pointing table
• Large data meeting in GB
  • James is likely going and discussing Lustre
  • Jeff going
• Memmap vs cache in casarc
  • Timing tests suggest cache better in lustre and equivalent against local disk
  • Memmap generates warnings under lustre and possibly causes data corruption
  • Change active and stable to default to cache
  • Try to enable memmap vs file I/O on a per task basis
• Status of async I/O Jim
  • Checked into active, integrated with clean
  • Any planned further development ?
    • start working framework to get allow single data selection and implementation of async I/O
    • Generate design description
    • uvconsub, applycal could be relatively simple
    • Need priority list of what to tackle (by use, by simplicity)
  • Send switch to enable/disable
• Integrate async I/O with large continuum test case Kumar
• Parallel imaging for large continuum case Kumar
  • Testable
• Parallel proto end to end script ready for testing Jeff
  • Only time domain for now
  • Spectral not functional
• Make listobs multi-ms compatable Sanjay
  • Question about implementing at MS level or visibility iterator level
  • MS level may be simplest
  • Discussion about indexing in multi-ms wrt spectral windows
    • issue of whether filler Re-index and renumber when creating MS from SDM
• General discussion on what's next
  • Global design for how split, filler, flagging will work
  • Script vs task level
    • Honglin and Jeff to work on parallel script framework
    • Need to consider training local staff
  • separately named split to generate multi-ms
• Generate procedures and first timings for all test cases James
  • Broke test Lustre
• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure to test by friday 2/4/11
    • When is less clear now
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Rework timing scripts
  • Small spectral status (needed further cleaning)
    • does not include continuum subtraction as a separate timed step using uvconsub2.
  • Large continuum
  • Large spectral (Bryan will talk to Claire to assign somebody)
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)
• Cluster scheduler scripting with CV James on hold for scripts
  • Discuss with Jim semantics for cluster/CASA interaction
• Cluster resource definitions James/Jim on hold