Casa Parallelization meeting minutes

Monday November 29nd Room 317, 3:30pm MDT

Attendees: Kumar, James, Jim, Greg, Jeff, Bryan, Brian

Targets/tasks for November 30 - December 5

Discussion/Agenda

• Verify existing small data scripts Sanjay
  • both good
• Continue code development of async I/O Jim
  • Coded, run through unit tests
  • working toward getting imager to make use of it
  • make aipsrc variable to control async vs sync I/O
• Complete remaining timing scripts Greg
  • Small spectral status (needed further cleaning)
    • does not include continuum subtraction as a separate timed step using uvconsub2.
  • Large continuum in progress should be done this week
  • Large spectral on hold
• Generate procedures and first timings for test cases James
  • Rob to check in split this week
  • Tile size effects in spectral line, agreed to redo spectral case to 1MB tile.
    • Kumar created 1MB tile for spectral line test
  • Need other data sets
  • Sanjay to look at scripts and verify them
• Time domain partitioning with multiple MSes per imaging process Kumar
  • Trying to identify 10% small reads, possibly metadata column (either main or sub table)
• Outline of plan for other parallelization tasks (calibration, flagging, filling etc)
  • Calibration pending waiting on George's design work on separating cal tables, Tak is implementing
    • Honglin and Jeff will implement
  • Flagging has no clear labor with Jonas leaving, may move to Jim
    • Flagging perceived as trickier due to mgmt of where to write the flags, flag command to avoid multiple passes through the data.
  • Filling to be examined by Michel in January when he visits
    • Multiple fillers in parallel perceived as simpler approach,
    • How do you decide how to separate data
    • Cluster vs desktop issue, desktop likely to be I/O bound
  • Who generates parallelization design docs for tasks
    • Perception Parallel-go enforcement will help in standardization

• Input MS/output MS methodology may need to change in long term, expensive in terms of disk space

• Memory requirement for multiple images
  • Recognized as an issue but no immediate concern, some examination ongoing into limiting number of images in wide field imaging.

• Cluster scheduler scripting with CV James on hold for scripts
  • Discuss with Jim semantics for cluster/CASA interaction
• Cluster resource definitions James/Jim on hold