James Robnett's meetings last week : Made a case for a future where HTC is achieved by making smaller blocks of work that can be easily farmed out over a pool of compute resources and each of which fit within 4-8GB of mem. Consequences for CASA are not immediate, but should be evaluated by us.
Uses CASA table tool to make HDF5/Parquet format MS's
Added Dask example
Added Dataframes tutorial
TODO: Add more sophisticated computations by antenna and/or scan, apply flagging, etc
Telemetry (Ville): Devise a method of pushing telemetry reports to public facing server/website on weekly basis
Performance Tests (Sandra/Federico): Incorporate initial performance measurements in to automated test suite by end of 5.6 cycle
Release Naming scheme changes:
CASA 5.6.0 (no pipeline): July-ish
CASA 6.0.0 (no pipeline): July-ish
ALMA-7.0 (5.6.x + pipeline): October 1st
This has to be branched early due to the build system updates
VLA/SRDP-1.x (5.6.y + pipeline): someday?
5.6 Development Completion, entering Test phase
Casa download area directory structure is changing due to Casa release naming convention update (release 5.6 deliverable).
The new structure has a "casa" directory for the common releases, and separate directories for the various instrument specific variants.
This will require updates for the NRAO synchronization scripts (copies to /home/casa/packages) and launcher scripts, and probably similar changes at ESO an NAOJ.
Please keep the following in mind as we sail past the 5.6 feature freeze
Testing and Characterizing code changes :
New numerical feature : Add a verification test ( gcwrap/python/script/tests/test_taskname.py )
Numerical bug fix : Add a new verification test or modify existing ones. At least one test should touch the code that has changed.
Plumbing change : Add a new verification test or identify at least one existing test that already exercises the modified code.
JIRA Docs :
A note about what was changed/fixed (a line or two of text)
Point to a verification test that exercises this code
Provide some information on what should be done for validation
For changes to numerical output : Characterize the effects and provide examples (plots/images when appropriate) to show that the fix/change has had the desired effect. This could simply be a few lines of text that describes the testing that the developer has done during coding. The depth of analysis will obviously vary depending on the topic. Documents attached with CAS-12148 and CAS-9621 are a trial for some of our more contentious topics (only a small fraction of tickets should need this level of care). Even with this, there will be iteration, but if we do as much as we can before stakeholder validation, it may help reduce costs (human, time, etc).
Validation tarballs :
Merge current master into the branch build before validation.
Manage interactions between tickets/features from a stakeholder/user point of view (mechanism TBD).
Dominant stakeholders :
VLASS needs 5.6 for Epoch 1 imaging to start in September.
ALMA needs 5.6 for Cycle 7 operations.
Calibration and Flagging
CAS-6747 (relatilve freq interp) now in test (still some work for CalLibrary version)
CAS-12434 (interim uvcontsub apply without VI1CAL env var) now in test
Imaging - Mosaicing tests (general user + VLASS):
[vlass/general] Functional verification tests : Cubes OK. MTMFS bugs found.
[vlass/general] Changes in psf generation for accurate wideband mosaic imaging : Algorithm bug found during test-writing. Needs changes in both 'mosaic' and 'awproject'.
[vlass/general] AWProject gridder for vlass with pointing corrections : ARD doing optimizations and cfcache-restart bug fix.
[general] pointing corrections with gridder=mosaic : To be tested but OK to leave out too.
[general] rectangular image support in awproject : Postpone to 5.7
[vlass/alma] Stakeholder verification tests : 1 for ALMA Cubes, 1 for VLASS. Required for 5.6 in order to deliver the above for VLASS, and to convince ALMA that changes to PSFs do not matter (hopefully).
CASR-21/CAS-12495 image sideband atmospheric curve - implementation complete, manually testing and comparing results with plotbandpass
Performance and reliability improvements, ellipse and polygon region fixes, region import/export in progress
Import / Export
Update on memory limits and .casarc (Pam/Kumar)
Schedule for merging casacore STL changes and Bob's python mods for casa5/casa6 compatibility to the 5.x master ? Both VLASS and ALMA depend critically on the CASA 5.6 release, and we'll need to handle validation worries. Should we wait until after we branch for 5.6 ? Can we coordinate with the pipeline group for branch-build tests (ALMA and VLASS) prior to PR ? Thoughts ?