CASA Parallelization Meeting Minutes

Thursday September 11th, DSOC 317, 8:00AM MT

  • Polycom Video: 192.33.117.12##8108
  • Voice: 434-817-6523

Attendees:

  • Socorro: Rob, James, Lindsey, Kumar, Urvashi, Sanjay.
  • Charlottesville: N/A
  • Garching: Justo, Sandra.

  • Apologies: N/A

Discussion

  • Review of Action Items from last meeting (see table below).
    • Pipeline bundling has been taking up Lindsey's time, but she will move up the priority on the MPI design/doc review. It will be useful to have this foundation before the pipeline meeting (Oct 20th).
    • Justo has been pushing on feature freeze tasks. Will perform testing on the cluster before leaving on vacation and will send an e-mail to the group.
      • Justo on vacation 19-29 of September.
    • Sandra committed cvel2 today! Interface is very similar, but has a data column parameter now.
      • There are cases where cvel2 will output a monolithic MS. However, MSTransform could be used instead to go from an MMS > MMS.
      • Sandra is working on the documentation for the MMS cases in MSTransform now. Will try to have the documentation distributed before the pipeline meeting. (Oct 20th, Garching)
    • Sandra clarified MSTransform heuristics with Jeff and has implemented the changes.

  • Other Status Updates
    • Parallel imager tests
      • Very good progress over the last few weeks testing out the refactored imager on the cluster. A few issues identified int he process are discussed below.
      • Too many open files / file descriptor limits were reached.
        • OS-level configuration limit of 1024
        • Can work around by reducing processes in major cycle on the head node. Will need some heuristics to keep within limits.
          • James can evaluate this problem next week and characterize the issue.
        • Testing ways of restarting imager w/o redoing work unnecessarily. Looks promising.
        • Some concerns with load balancing given decreasing load with time. Under utilizing memory resources in current tests.
        • Saving in parallel in the continuum imaging case is a bottleneck at the moment.

  • Anything else you would like to discuss (AOB)
    • Justo and Sandra to visit NM this fall
      • Work with James, Kumar, B&T hire, etc, on 1st release issues.
      • November/December time-frame.
      • May want to invite Invite Darrell and Stewart Williams.
      • Rob to setup a doodle poll to pick the week.
    • Logger
      • Single log file & concerns regarding ordering of entries.
      • Pipeline think of log as a data product. Single file.
      • No synchronization at c++ level amongst loggers, but no indication of log overlap in current implementation. Can happen with intense logging activity.
      • Improvements may not be urgent.
      • Terminal messages from each node would be desirable (for imager testing). Eg, seeing progress meters is useful.
    • OSX support for MPI and OSX testing of MPI framework.
      • Priority is unclear. ASAC has to comment.
      • ESO have 10.7, 10.9 machines available. ESO helpdesk does not support these versions though. 10.8 available in department.
      • Characterize any mpi issues on OSX. Watch for blockers to single code base and multi-platform release.

  • Next Meeting
    • 9/18 virtual model column discussion. (May be at a different time).
    • 9/25 next status meeting.

Deferred Items for a future meeting

08/14/14:
  • Logger
    • Single log file & concerns regarding ordering of entries.
    • Pipeline think of log as a data product. Single file.
    • No synchronization at c++ level amongst loggers, but no indication of log overlap in current implementation. Can happen with intense logging activity.
    • Improvements may not be urgent, revisit use cases/requirements when Sandra is getting ready to start work.

  • Imaging
    • Discuss requirement imposed on VI/VB2 to pass required information for MMS processing.

  • MSTransform
    • Discuss time separation axis for MMS creation.

Action Item List

Item # Date Opened Description Leads Status Status Notes
02 8/07/14 Review available documentation (on wiki), in particular the MPI document. Lindsey Partial 9/11/14: Useful to complete before pipeline meeting. 8/21/14: Initial read, but still has questions.
08 8/21/14 Test mpi4casa integration on cluster. Justo Open  
09 8/21/14 Review openmpi features relative to requirements for preferred library. Justo Open  
10 9/11/14 Documentation of MSTransform MMS functionality for users Sandra Open 9/11/14: Target of 10/20/14 (pipeline meeting).
11 9/11/14 Evaluate and characterize file descriptor limit issue in imager James Open 9/11/14: Task for next week.
12 9/11/14 Basic test of mpi4casa on OSX 10.8 to characterize any multi-platform problems. Justo Open 9/11/14: Don't need mpi to work, but just ensure changes we are considering do not create blockers for OSX builds

Closed Item Record

Item # Date Opened Description Leads Status Status Notes
01 8/07/14 Update Issue Chart to reflect current status. James Closed 8/14/14: Contributions from Kumar incorporated. Justo provided separate notes on tasks outstanding. 8/7/14: Contributions from others on the team also welcome.
03 8/07/14 Update reference doc links to point to MPI doc in SVN Rob Closed 8/7/14: Complete.
04 8/14/14 Talk to James, Kumar, Justo and others and bring some resolution to preferred MPI library/implementation issue. Rob Closed 8/21/14: made MPI Implementation page. will leave open until library selection finalized. 8/14/14: Have feedback from Justo, James, Kumar and Martin Pokorny. Will document and distribute.
05 8/14/14 Add new wiki pages for requirements capture, task list, and other project artifacts. Update based on recent meetings, then circulate for iteration by others. Rob Closed 8/21/14: see main page
06 8/14/14 Evaluate feasibility of completing cvel2 for 4.3 release. Sandra Done Committed to r31041, r31056 and 31057
07 8/21/14 Clarify MSTransform use cases with Jeff. Rob Closed  
-- RobSelina - 2014-09-10
Topic revision: r7 - 2014-09-11, RobSelina
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback