Thursday Morning Meeting : 7 Sept 2017

• DIAL-IN NUMBERS & PASSCODES:
• IP: 192.33.117.12##8110
• Phone: (434) 817-6524

Attendance

• Socorro:
• CV:
• Garching:
• SCO:

News / Meetings / Visitors

• CARTA - ASIAA plans to post the pre-release on github, timing TBD
• Pipeline - ICT agreement to meet to align the Pipeline/Archive/AQUA interface, as an ICT-only issue

Build, Release

• Packaged versions copied to CV/Socorro. Preliminary email sent out to sci-staff for last minute checks. VLASS says they need a later version of the pipeline (r40809).
• We need a new Plone page describing the final steps of our release process (scripts, where executables get copied, etc).

Verification Testing

• Only had time to begin the recruiting process to replace Akeem (ATR is done, iterating with Darrell on the job description)

Validation Testing

• Current validation tally
  • 116 Under Validation, 90 Ready to Validate.
  • 2 tickets went RtV this week so far. 1 tickets went through Validation to Resolved this week so far.
  • Aug 25 - Sept 1: 27 tickets assigned for validation. 33 tickets resolved.
• Current Testing Efforts for 5.1 deliverables
  • final packages built (5.1.0-68), and final testing shows a couple of small issues (still cannot start casadocs on lustre, run plotcal in "casa -pipeline" mode). now there is an issue with VLASS restore script?
• Current Testing Efforts for 5.2 deliverables
  • coordination for the next level of science validation testing between PLWG/HPC group/Bjorn/(user testers as needed); time to start discussing once 5.1 is out
  • HPC needs for testing going through the 5.2 release, as far as nodes required and disk space anticipated (current numbers, future projections over the next months)
• Current Testing Efforts for 5.3 deliverables
  • statwt2 (CAS-10530)
    • VLA testing scheduled early in the development cycle
  • systematic tests of mstransform
  • plone/CASAdocs
  • further parallelization testing (non-ALMA pipeline specific)
  • polarization calibration
  • miscellaneous
    • reporting peak residual in a channel that has no mask: CAS-10692
    • please see https://safe.nrao.edu/wiki/bin/view/Software/CASAUserTesting for the up-to-date list of tickets and testing status

Architecture

• Warnings Summary

HPC

• Not many bugs/issues open in parallel mode, and definitely no more general issues (although 5-1.0-68 is completely broken for parallel mode, more below...). From https://open-confluence.nrao.edu/display/CASA/ALMA+Pipeline+Cycle+5+Testing:
  • CAS-10672 "RuntimeError: Error in running Major Cycle : CountedPtr: null dereference error Open". Only with 2 datasets. Might be hard to reproduce/debug.
  • CAS-10662 "pipeline.infrastructure.launcher produces exceptions sending command requests - MPI_ERR_BUFFER: invalid buffer pointer". Only a couple of test datasets and with certain level of parallelization. Under investigation.
  • CAS-10685 "uvcontfit::pipeline - Error in applycal::VisSetUtil::addScrCols, Adding CORRECTED_DATA column(s) - Invalid Table operation: Table::addColumn; column CORRECTED_DATA already exists". Under investigation.
    • The error comes from a change in the pipeline to force applycal to run in serial on an MMS inside the hif_uvcontsub stage. Because the pipeline needs to run applycal with the old VI/VB for that correction, a VI1CAL environment variable is set inside the pipeline. This setting does not propogate to the MPIServers at run time, therefore bypassing the parallel processing allows applycal to run using the old VIVB. There are 2 ways to bypass the cluster, 1) by making the task process each Sub-MS, one after the other. 2) by making the task process the MMS as a monolithic MS. This second approach is causing the error message, which we are investigating at the moment. The first we will need to do in the pipeline once the 5.2 branch is cut is to set the bypass to 1 instead of 2.
  • CAS-10636 - "Serial mode: pipeline.hif.heuristics.imageparams_base::im::close No MeasurementSet has been assigned, please run open or selectvis". Only one test dataset. Harmless error message.
• For the PLWG we'll need to clarify/process/filter some error and warning messages. For now we have:
  • CAS-10508 "plotbandpass::pipeline.hif.tasks.bandpass.renderer:: Could not get PHASE_SCORE_RMS score for...".
  • WVR SPWs warnings in importasdm, discussed in CAS-10453.
  • The MSSelectionNullSelection that is being addressed in CAS-10453 / CAS-10685.
• Slowness of stats operations on concatenated images ( CAS-10428) becomes a top priority performance issue, in the light of recent parallel tests. The penalty (~10x slowdown in pipeline tasks) can compromise parallelization speedup for medium size datasets. Examples, with dataset T020: pipeline.hif.heuristics.imageparams_base takes 35m (serial) => 5h9m (parallel); with T030, 19m (serial) => 4h42m (parallel). Other pipe.hif and pipe.infra tasks also affected.
• For validation: many more links to serial + parallel weblogs available now with fresh re-runs. First 20 datasets done. Last 10 datasets halfway through.
• Can we allocate some 5.2 effort to improve automated test coverage (~0% now) of the many issues fixed in the last few weeks?

Development

• CASA 5.2
  • Ongoing discussion about how much of the pipeline scripts/setup we (CASA) need to replicate for our September tests of serial/parallel
  • New bug : CAS-10685 : uvcontinuum fitting : Error when CORRECTED_DATA is encountered. Pipeline bug or CASA bug ?
  • New bug : CAS-10672 : CountedPtr dereference error in some datasets.
• CASA 5.3
  • Anything yet ?
• Initial CARTA user manual (from ACDC via Morgan) : http://www.asiaa.sinica.edu.tw/~kswang/carta_doc/html/index.html

Pipeline

AOB

• ALMA correlator upgrade project : We need to produce a note about how CASA will be impacted by (or will handle) this (by Monday Sep 11).
  • Project timescale : 2021 - 2022
  • Correlator specifications :
    • Increase spectral resolution by a factor of 8 => nchan = ~3800 will go to 32K. Limits are 65K, 32K or 16K depending on polarization mode.
    • Increase bandwidth by a factor of 2 => 16 GHz x 2 polarizations. Process the entire 4-12 GHz IF bandwidth.
    • Increase time resolution from 16msec to 1msec.
  • CASA's impact
    • Spectral resolution : NChan : Our spectral-domain parallelization schemes need to handle this. As of now, we expect current designs to scale. But we should verify this over the next year or two and make plans if we need something new. All modules : filling, flagging, plotting, calibration, imaging.
    • Double bandwidth : More use of wideband analysis algorithms. VLA has done this so all basic algorithms are in place. As of now, we expect existing algorithms to suffice. A factor of 2 increase is still modest in terms of a sensitivity increase (1.4 times) and fractional bandwidth (8% currently to 16% later (for comparison, EVLA handles 60% fractional bandwidth) ). Some commissioning of algorithms may still be required.
    • Narrow channels and/r high time resolution => Less sensitivity per visibility. Any algorithmic changes needed ? Can/should we rely on averaging in time and frequency when needed ? Are our averagers expected to scale appropriately ?
    • Anything else ?
      • Calibration : renewed interest in bpoly and bspline to get parameterized fits across channels and time, transfering solutions across spws/scans
      • Calibration parallelization
      • General topic of splitting and combining across frequency ranges (list of MS, multi-MS, imager's cube partitioning, channel chunking, the need for calibration, flagging, etc to gather stats and/or use data that cross SPW boundaries or 'split' boundaries).
      • Image storage format for large cubes. Neither of our methods will scale well. CASA Image format or Reference Concatenated images. Need to figure something out here.
      • Displays : plotms may need cacheless operations. For image viewing, we have no solution right now. CARTA is meant to support this, but it remains to be seen if this is going to be feasible.
      • We should ask if any heuristics discussions are happening - i.e. to think about how to break the spectra into pieces, keeping in mind the downstream processing.
      • Dev items that we will already be doing in the next year or two : Streamlining frequency splitting for parallelization, image formats, cacheless plotms, crossing split boundaries during processing. Dev items we don't yet have a plan for : A scaleable image format, and image display options.
  • Detailed documents are attached here (https://safe.nrao.edu/wiki/pub/Software/CASA/Sep717/CRE_archive.zip)

Developer Reports

Thursday Meeting

• Sanjay Bhatnagar
• Sandra Castro
• Lindsey Davis
• Bjorn Emonts
• Pam Ford
  • CAS-7049 (cal table selection) in the home stretch. tCTSelection.cc updated and successful. Need to create google test.
• Enrique Garcia
• Bob Garwood
  • CAS-10621 : Target in image appear at incorrect coordinates. I believe I've found a bug in how the filler handles the ephemeris in the ASDM. But that may not be the cause of the target being at the wrong coordinates wrt the image center. Dirk is also looking in to this. I'm also gaining a better understanding in how ephemeris data is handled by the filler. I've also discovered an unrelated bug in how ephemeris related filler arguments are handled.
  • Some more discussions on solar VLA data. Eventually this will turn into a new ticket.
  • Started looking at how to remove boost dependencies from the filler code.
• Kumar Golap
• Jeff Kern
• David Mehringer
• George Moellenbrock
• Dirk Petry
• Martin Pokorny
  • CAS-10420, CAS-10670 PRs (for async VI2)
• Federico M Pouzols
  • parallel pipeline tests
  • CAS-10651: clarify confusing paragraph in cvel2 doc.
  • CAS-6215: flagging, rflag mode complications.
• Urvashi Rao
• Darrell Schiebel
• Ville Suoranta
• Takahiro Tsutsumi

Friday NAOJ Meeting

• Kanako Sugimoto
  • working on draft article of CASA news ( CAS-10495)
  • created M100 SD guide for CASA 5.1: ready for release once 5.1 is released
  • collecting development requests for CASA 5.3
• Wataru Kawasaki
• Masaya Kuniyoshi
• Takeshi Nakazato
  • working on CAS-10683: sdimaging migration to new imager framework
• Renaud Miel
  • Completed CAS-8082: plotms header
  • Setting up NAOJ CASA development server
  • Next ticket CAS-8087: plotms / plot pointings

Minutes

• Build release
  • we will release a re-packaged version version 69, wit the new pipeline revision.
  • existing package is availabale at all sites, we won't change that yet. We will rebuild in the next couple of days, and then announce later.

• Validation : Ongoing discussion (outside CASA) of how to allocate hardware resources for the next few months, and how the HPC testing of CASA will fit into that. No problems as of now.