Thursday Morning Meeting : 7 Sept 2017

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro:
  • CV:
  • Garching:
  • SCO:

News / Meetings / Visitors

  • CARTA - ASIAA plans to post the pre-release on github, timing TBD
  • Pipeline - ICT agreement to meet to align the Pipeline/Archive/AQUA interface, as an ICT-only issue

Build, Release

  • Packaged versions copied to CV/Socorro. Preliminary email sent out to sci-staff for last minute checks. VLASS says they need a later version of the pipeline (r40809).
  • We need a new Plone page describing the final steps of our release process (scripts, where executables get copied, etc).

Verification Testing

  • Only had time to begin the recruiting process to replace Akeem (ATR is done, iterating with Darrell on the job description)

Validation Testing

  • Current validation tally
    • 116 Under Validation, 90 Ready to Validate.
    • 2 tickets went RtV this week so far. 1 tickets went through Validation to Resolved this week so far.
    • Aug 25 - Sept 1: 27 tickets assigned for validation. 33 tickets resolved.
  • Current Testing Efforts for 5.1 deliverables
    • final packages built (5.1.0-68), and final testing shows a couple of small issues (still cannot start casadocs on lustre, run plotcal in "casa -pipeline" mode). now there is an issue with VLASS restore script?
  • Current Testing Efforts for 5.2 deliverables
    • coordination for the next level of science validation testing between PLWG/HPC group/Bjorn/(user testers as needed); time to start discussing once 5.1 is out
    • HPC needs for testing going through the 5.2 release, as far as nodes required and disk space anticipated (current numbers, future projections over the next months)
  • Current Testing Efforts for 5.3 deliverables
    • statwt2 (CAS-10530)
      • VLA testing scheduled early in the development cycle
    • systematic tests of mstransform
    • plone/CASAdocs
    • further parallelization testing (non-ALMA pipeline specific)
    • polarization calibration
    • miscellaneous

Architecture

HPC

Development

  • CASA 5.2
    • Ongoing discussion about how much of the pipeline scripts/setup we (CASA) need to replicate for our September tests of serial/parallel
    • New bug : CAS-10685 : uvcontinuum fitting : Error when CORRECTED_DATA is encountered. Pipeline bug or CASA bug ?
    • New bug : CAS-10672 : CountedPtr dereference error in some datasets.
  • CASA 5.3
    • Anything yet ?
  • Initial CARTA user manual (from ACDC via Morgan) : http://www.asiaa.sinica.edu.tw/~kswang/carta_doc/html/index.html

Pipeline

AOB

  • ALMA correlator upgrade project : We need to produce a note about how CASA will be impacted by (or will handle) this (by Monday Sep 11).
    • Project timescale : 2021 - 2022
    • Correlator specifications :
      • Increase spectral resolution by a factor of 8 => nchan = ~3800 will go to 32K. Limits are 65K, 32K or 16K depending on polarization mode.
      • Increase bandwidth by a factor of 2 => 16 GHz x 2 polarizations. Process the entire 4-12 GHz IF bandwidth.
      • Increase time resolution from 16msec to 1msec.
    • CASA's impact
      • Spectral resolution : NChan : Our spectral-domain parallelization schemes need to handle this. As of now, we expect current designs to scale. But we should verify this over the next year or two and make plans if we need something new. All modules : filling, flagging, plotting, calibration, imaging.
      • Double bandwidth : More use of wideband analysis algorithms. VLA has done this so all basic algorithms are in place. As of now, we expect existing algorithms to suffice. A factor of 2 increase is still modest in terms of a sensitivity increase (1.4 times) and fractional bandwidth (8% currently to 16% later (for comparison, EVLA handles 60% fractional bandwidth) ). Some commissioning of algorithms may still be required.
      • Narrow channels and/r high time resolution => Less sensitivity per visibility. Any algorithmic changes needed ? Can/should we rely on averaging in time and frequency when needed ? Are our averagers expected to scale appropriately ?
      • Anything else ?
        • Calibration : renewed interest in bpoly and bspline to get parameterized fits across channels and time, transfering solutions across spws/scans
        • Calibration parallelization
        • General topic of splitting and combining across frequency ranges (list of MS, multi-MS, imager's cube partitioning, channel chunking, the need for calibration, flagging, etc to gather stats and/or use data that cross SPW boundaries or 'split' boundaries).
        • Image storage format for large cubes. Neither of our methods will scale well. CASA Image format or Reference Concatenated images. Need to figure something out here.
        • Displays : plotms may need cacheless operations. For image viewing, we have no solution right now. CARTA is meant to support this, but it remains to be seen if this is going to be feasible.
        • We should ask if any heuristics discussions are happening - i.e. to think about how to break the spectra into pieces, keeping in mind the downstream processing.
        • Dev items that we will already be doing in the next year or two : Streamlining frequency splitting for parallelization, image formats, cacheless plotms, crossing split boundaries during processing. Dev items we don't yet have a plan for : A scaleable image format, and image display options.
    • Detailed documents are attached here (https://safe.nrao.edu/wiki/pub/Software/CASA/Sep717/CRE_archive.zip)

Developer Reports

Thursday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
  • Lindsey Davis
  • Bjorn Emonts
  • Pam Ford
    • CAS-7049 (cal table selection) in the home stretch. tCTSelection.cc updated and successful. Need to create google test.
  • Enrique Garcia
  • Bob Garwood
    • CAS-10621 : Target in image appear at incorrect coordinates. I believe I've found a bug in how the filler handles the ephemeris in the ASDM. But that may not be the cause of the target being at the wrong coordinates wrt the image center. Dirk is also looking in to this. I'm also gaining a better understanding in how ephemeris data is handled by the filler. I've also discovered an unrelated bug in how ephemeris related filler arguments are handled.
    • Some more discussions on solar VLA data. Eventually this will turn into a new ticket.
    • Started looking at how to remove boost dependencies from the filler code.
  • Kumar Golap
  • Jeff Kern
  • David Mehringer
  • George Moellenbrock
  • Dirk Petry
  • Martin Pokorny
  • Federico M Pouzols
    • parallel pipeline tests
    • CAS-10651: clarify confusing paragraph in cvel2 doc.
    • CAS-6215: flagging, rflag mode complications.
  • Urvashi Rao
  • Darrell Schiebel
  • Ville Suoranta
  • Takahiro Tsutsumi
Friday NAOJ Meeting
  • Kanako Sugimoto
    • working on draft article of CASA news ( CAS-10495)
    • created M100 SD guide for CASA 5.1: ready for release once 5.1 is released
    • collecting development requests for CASA 5.3
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
    • working on CAS-10683: sdimaging migration to new imager framework
  • Renaud Miel
    • Completed CAS-8082: plotms header
    • Setting up NAOJ CASA development server
    • Next ticket CAS-8087: plotms / plot pointings

Minutes

  • Build release
    • we will release a re-packaged version version 69, wit the new pipeline revision.
    • existing package is availabale at all sites, we won't change that yet. We will rebuild in the next couple of days, and then announce later.

  • Validation : Ongoing discussion (outside CASA) of how to allocate hardware resources for the next few months, and how the HPC testing of CASA will fit into that. No problems as of now.

-- UrvashiRV - 2017-09-06
Topic attachments
I Attachment Action Size Date Who Comment
CRE_archive.zipzip CRE_archive.zip manage 2 MB 2017-09-06 - 15:34 UrvashiRV ALMA correlator upgrade plans
Topic revision: r23 - 2017-09-07, KanaSugimoto
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback