Thursday Morning Meeting - 12 October 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro:
  • CV:
  • Garching:
  • SCO:

News / Meetings / Visitors

Build, Release

  • XCode9 broke a number of MacPorts. Most critical ones were fixed early this week but 8.x.x is still the safer bet.
  • OSX10.11 end-of-life
  • 5.1.1 release schedule and branching? Which pipeline branch to use?
  • We have three releases planned for the next few months and have been in a perpetual release mode since 5.0. Is this the new norm and should we plan on monthly releases?

Verification Testing

2017-10-03 13:45:36   INFO   Summary::getResult     antenna CM02 flagged: 3.68408e+06 total: 8.17992e+06 (45%)
2017-10-03 13:45:36   INFO   Summary::getResult     antenna CM03 flagged: 3.68406e+06 total: 8.17992e+06 (45%)
2017-10-03 13:45:36   INFO   Summary::getResult     antenna CM04 flagged: 3.68404e+06 total: 8.17992e+06 (45%)
      • e.g. serial
2017-10-04 12:24:37   INFO   Summary::getResult::MPIServer-2     antenna CM02 flagged: 1.61248e+06 total: 3.99661e+06 (40.3%)
2017-10-04 12:24:37   INFO   Summary::getResult::MPIServer-2     antenna CM03 flagged: 1.61248e+06 total: 3.99661e+06 (40.3%)
2017-10-04 12:24:37   INFO   Summary::getResult::MPIServer-2     antenna CM04 flagged: 1.61245e+06 total: 3.99661e+06 (40.3%)
      • Any further info on this?
      • Urvashi suggests this might need another feature specific to parallel flagdata's summary mode
        • We need to sync the dictionaries to provide one output one.
        • New JIRA ticket to cover this effort?
      • CAS-10202: Find a way to consolidate aggregate RFlag results in 'calculate' mode when running on MMS
        • related ticket for flagdat(mode='rflag')
        • Likely to be a legitimate difference between serial/parallel, and we should confirm, but maybe we can ignore it, or do an approximate combination of thresholds only for the 'rflag,calculate' mode. ( i.e. not critical for 5.2 )
  • Testing some ATCA Casaguides, which are close to being committed
  • CAS-10730: runUnitTest.py -m was not working
    • Akeem has fixed it, now assigned to Sandra for final testing
  • Work has started on unifying runUnit and runRegression scripts
    • Laying the groundwork for adding functionality

Validation Testing

  • Current validation tally
    • 120 Under Validation, 89 Ready to Validate.
    • 5 tickets went RtV this week so far. 5 tickets went through Validation to Resolved this week so far.
    • Sept 22 - 28: 12 tickets assigned for validation. 12 tickets resolved.
    • Sept 29 - Oct 5: 4 tickets assigned for validation. 6 tickets resolved.
  • Build and packaging of 5.1.1 (needs to be kicked off imminently)?
  • Discussion item: update on running PL runs regularly during the build + test stage with each build to catch problems earlier?
  • JDM has been/will be traveling first three weeks of October, having some impact on getting testing out
  • Current Testing Efforts for 5.2 deliverables
    • update from HPC/Bjorn
  • Current Testing Efforts for 5.3 deliverables
    • imaging issues/autoboxing refinements
    • statwt2 (CAS-10530)
      • Urvashi posted a summary of new requests; is this waiting on Josh? Claire?
    • systematic tests of mstransform
    • plone/CASAdocs
    • further parallelization testing (non-ALMA pipeline specific)
    • polarization calibration
    • testing snapshot -- please see https://safe.nrao.edu/wiki/bin/view/Software/CASAUserTestingfor the up-to-date list of tickets and testing status
      • testing ongoing:
        • plotants tickets (Pam, few more)
        • plotms/plotbandpass capability (Pam, CAS-9053)
        • imhistory tickets (Dave, several)
        • phase center shift in plotms (CAS-8431)
        • frequency interpolation of 'linearflag' for BP tables with good solutions (George, CAS-10772)
        • ms tool selectinit (Pam, Sandra, CAS-10818)
        • immath poli/pola testing (Dave, two tickets)
        • simulations (Remy, several)
        • moving ALMA pipeline's findcontinuum method into C++. (Dave, CAS-10521)
        • imfit result in pixel coordinates (Dave, Dirk, CAS-10696)
      • "next up"/looking for testers:
        • CAS-9538 Perley & Butler 2017 flux density scale, tester identified
        • CAS-8270 Implement remaining plotcal functionality in plotms, tester identified
        • CAS-10329 rename plot ms tool from 'plotms' to 'pmtool'
        • CAS-10616 plotms crashes when 'locate" is used on baseline-averaged data, tester identified
        • CAS-10366 Implement ACCOR equivalent, George: help?

Architecture

HPC

* Report about lustre performance.

Pipeline

  • CASA / pipeline 5.1
    • 1 pending ALMA patch, 2 pending VLA patches, waiting for ALMA end to end tests to complete before creating a new tarball
    • ALMA Cycle 5 data transmission and storage concerns
  • CASA / pipeline 5.2
    • image parameter handling patch ported to 5.2
    • don't worry too much about Cycle 2 project pipeline failures, Cycle 2 suffered from a lot of missing / incorrect meta data problems
    • separation of pipeline task, CASA task, CASA tool inefficiencies and impact on total serial vs HPC run times needs to be more clear to everyone
    • some reporting issues are due to serial vs HPC handling of output dictionaries, e.g flagdata, tclean, etc
  • CASA / pipeline 5.3
    • Cycle 6 refactoring and cleanup in progress, flagging, calibration, imaging, templates
    • Session support / HPC framework enhancements in progress
    • Cycle 6 science goals are modest, science ticket review almost complete,

Development

  • CASA 5.2
    • flagdata summary mode for parallel runs may need an explicit merge (no ticket yet)
    • performance of imageanalysis tasks on reference concatenated cubes ( ideas still to be discussed )
    • need to analysis/understand observed differences in tclean speedups (legitimate or not)

  • CASA 5.3
    • Statwt2 requirements update : ongoing
    • Imaging team ironing out details/interactions for iteration control re-write, parallel cube changes, autoboxing connection with iteration control (+ plans to address other issues)
    • George : update on VLBI work from recent visit abroad ?
    • A couple of new bugs - phasecenter related error, tclean selection-related seg-fault(!)
    • Anything else ?

AOB

Developer Reports

Thursday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
    • Prepared pipeline presentation for ICT leads meeting next week
    • Calibration / rendering code cleanup
    • Cycle 6 science tickets review and request for clarification
    • Numerous email discussions with PWG lead
  • Bjorn Emonts
  • Pam Ford
    • ms tool refactor/rename broke Sandra's pipeline test (CAS-10818) - fixed ms.selectinit() to issue warning and remove selection for invalid DDID. Back to Sandra for validation.
    • cal library parser (plotms callib param) bug causing syntax error for quoted strings within strings (CAS-10736) - fixed, pull request merged
    • investigating plotms spawning multiple windows (CAS-10822) - suspect dbus failure is causing it
  • Enrique Garcia
  • Bob Garwood
    • CAS-10738 : ephemeris code in the filler. The work on this is done. Just wrapping up the testing of the most recent tarball and will move it to "read to validate" later today.
    • CAS-10732 and CAS-10665 : investigating the speed (slow) of asdm2MS and bdflags2MS. I've just begin work on this. I plan to spend a week getting a sense of where things seems particular slow before tabling this work to move back to more specific filler bugs/features.
  • Kumar Golap
  • David Mehringer
  • George Moellenbrock
    • At CASA/VLBI workshop at ASTRON/JIVE last week
    • Inhabiting new laptop this week, including installing a CASA dev build
    • Prep for next trip: Poln talk at QUESO meeting Oct 25-27.
  • Dirk Petry
  • Martin Pokorny
    • Investigating methods of testing parallel I/O in casacore (for MSv3)
    • [VLA: Mostly working on realfast project]
  • Federico M Pouzols
  • Urvashi Rao
  • Darrell Schiebel
  • Ville Suoranta
  • Takahiro Tsutsumi
    • Continued working on timing tests of automask components, did some modifications to improve the speed in pruneRegions
    • Bug fix: CAS-10773 (resolved)
Friday NAOJ Meeting
  • Kanako Sugimoto
    • discussion of 'too many open file' issue especially how to investigate use MultiFile works for this issue.
    • questions about MMS-aware task
  • Wataru Kawasaki
  • Takeshi Nakazato
    • worked on CAS-10683: sdimaging migration to new imager framework
    • looked at MultiFile and related documents
    • some MSv3 work
  • Renaud Miel
    • NAOJ CASA development host setup: done
    • CAS-8087
      • Specs clarification
-- MorganGriffith - 2017-10-11
Topic revision: r25 - 2017-10-12, KanaSugimoto
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback