Casa Parallelization meeting minutes

Monday May 23rd Room 317, 3;30pm MDT

Attendees: Bryan, Sanjay, Kumar, Jim, James

Targets/tasks for May 23rd - May 29th

Discussion/Agenda

• Examine async thread failure problem Jim
  • Progress defining issue
• Work toward implementing streaming framework in applycal Jim
  • Discuss with Ger
    • Use memory tables rather than aipsIO table mananger
    • Discuss applying above to existing MSes rather than aipsIP at fill time
• Refine and distribute cluster scheduler interface James
  • Some progress with OODT
• Spectral serial improvements (avail mem, don't grid uncleaned channels) Kumar
  • Serial improvements
  • Discussion about interaction with split and spectral windows
• LOFAR visitors
  • Dirk Petry is here
• Parallel gridder status Sanjay
• SDM talk from Francois at 2pm
• Gers talk at 11 on Wednesday
• Discussion on migrating AIPS pipeline to CASA
• Complete parallel user interface frame work Jeff/Honglin
• Cluster scheduler scripting with CV James
  • Clean up and make available James
  • Focusing on inhouse script for now
• Test async performance against parallel continuum case James
  • Need to test vs various hardware configs
  • Need 300+GB continuum case with complex field
    • Bryan will look into observing (spectral line and extended emission) IRC216
• General discussion on what's next

Older Issues

• Filling I/O issues update
  • Michel is optimistic of having a streamed I/O approach and use sub-ms infrastructure
  • Waiting on Francois to change data access methods
• Plot-ms suffers similar problem to split but needs lower level solution Ger
• Timing scripts
  • Large spectral (Bryan talked with Claire, Miriam available)
    • Less interesting in absence of parallel cube imaging
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Cube case regression status
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)