Thursday Morning Meeting 2017-04-06

  • DIAL-IN NUMBERS & PASSCODES:
  • IP: 192.33.117.12##8110
  • Phone: (434) 817-6524

Attendance

  • Socorro: Jeff, Pam, George, Lindsey, Martin, Juergen, Tak, Kumar, Urvashi
  • CV: Darrell, Martin, Ville, Andy, Dave, Morgan, Jen
  • Garching: Fedrico, Enrique
  • SCO:

News / Meetings / Visitors

  • VLBI

Build, Release

  • 5.0 Status
    • CAS-9984 and CAS-9985: Issue unflagging time- or channel-averaged data in plotms (flagging works)
    • Autoboxing: issue when nothing to deconvolve
    • 2 Issues from CalSolve (Solution Interval Boundaries, Frequency access for pipeline)
    • Co-ord issue in tClean
    • Akeem to follow up on Test Suite 4
    • Fedrico: Regrid options
    • Claire's Flag Data ticket

  • Once we switch to the new process, we need to monitor agent disk usage carefully. To help smooth out the transition, please do not create all of your 5.1 branches at once in Bitbucket. We've ordered new disks and are working with CIS to increase the Linux server capacity but we won't know the actual needs until after the switch.

  • Do we have a pipeline branch to use for 5.0.0 tarballs?

Verification Testing

  • CASAGuide work is ongoing
    • Iterating closely with the NAASC DAs
  • Reminder: task level unit test JIRA tickets are in the system and assigned
    • Once you've created the test requested in the ticket, assign to Akeem

Validation Testing

  • T/F is underway
  • if you are ready for science testing use the "validation" state
  • plone reviews due by the end of next week
  • status check on VI/VB2 and cvel going ready for validation

Architecture

HPC

  • ALMA pipeline parallel testing
    • Using VIVB2 for both imaging and calibration. Found 2 issues:
      • Pipeline qa/gpcal.py, line 46 in gpcal_calc(). SEVERE getcol::CHAN_FREQ Exception Reported: ArrayColumn::getColumn cannot be done for column CHAN_FREQ; the array shapes vary: Table array conformance error.
      • Error from tclean when imaging the PHASE calibrator in mfs mode. SEVERE tclean::task_tclean::@nmpost047::MPIServer-4 Exception from task_tclean : Error in running Major Cycle : Cannot open existing image : oussid.s19_0.J13163338_ph.spw16.mfs.I.iter1.model : There is a coordinate system mismatch between existing images on disk and current parameters (The SpectralCoordinates have differing reference values). If you are attempting to restart a run, please change imagename and supply the old model or mask as inputs (via the startmodel or mask parameters) so that they can be regridded to the new coordinate system before continuing.
    • One of our Jenkins datasets runs successfully until the end using VIVB2 for imaging and VIVB1 for calibration.
    • VIVB2-related failures depend on the dataset in use.

Development

  • 5.1 Planning

Pipeline

  • Imaging heuristics refactor nearing completion
  • Support for "exporting" final measurement sets added to exportdata tasks
  • CASA 5.0 / pipeline trunk testing
    • frequency solint problems have been resolved
    • new caltable metadata handling issue
    • new tclean coordinate system problem ?

AOB

Developer Reports

Monday Meeting
  • Sanjay Bhatnagar
  • Sandra Castro
    • Fixed CAS-9945. Split and mstransform wrote wrong values for the TOTAL_BANDWIDTH column. Starting in Cycle3, correct values of EFFECTIVE_BW were written to the ASDMs. Until now, split wrote ToTBW =EFFECTIVE_BW*NUM_CHAN, but the correct is to write ToTBW =CHAN_WIDTH*NUM_CHAN. This change is ready to be validated
    • Started to use git and created my first branch and pull request for CAS-9945.
    • Updated more Plone documentation for CAS-99896, CAS-99895, CAS-99894.
  • Lindsey Davis
    • Modified the pipeline exportdata tasks to support an "export the final mses" option instead of the standard export flags, caltables, and calibration instructions option.
    • 5 days vacation
  • Pam Ford
    • CAS-9879 - fixed bug for plotms flag extensions, created tickets for unflagging as described above. - ready to validate otherwise
    • CAS-9682 - plotms scalar-averaging bug fix, disable vector/scalar averaging radio buttons when no other averaging is enabled. - under validation
    • CAS-2401 - square UV plots in plotms - ready to verify
    • Looked into plotms max memory (.casarc setting or system free memory). Will implement using CASA_MAX_MEMORY environment variable for 5.1.
    • 2 pull requests for git workflow branches (CAS-8295 and CAS-9500)
  • Enrique Garcia
  • Bob Garwood
  • Kumar Golap
    • Fixed bugs related to vi2 move
    • Fixed a bug in mosaic due to change in oversampling implementation
    • Worked on some simulation for VLASS
  • Jeff Kern
  • David Mehringer
    • Statwt2: Asked questions, attended George's talks, code dove, etc etc.. Still struggling to get a basic TVI-related application working. A complete and current design document describing what one must implement to get an entire application (ie, ms tool method) to work would be great, along with sample code for a very simple, complete application. Without a basic level of documentation like this, one is forced to code dive to try and get answers, which is very hit and miss, and asking questions of others who do not necessarily understand the context of the questions and whose responses can be slow. Developer level documentation is as important as user level documentation.
    • (CAS-9941) visibility weights written by exportuvfits: Began initial investigation.
    • (CAS-10034) PSF from parallel cube tclean can not be collapsed: Punted to Ger who apparently introduced a bug with his misc info fix.
    • Responded to Darrell's queries re: creating bitmap images using CASA.
    • (CAS-9983) imfit fixed parameter does not work for 'b' (minor axis) when 'a' (major axis) is not also requested: Fixed. Updated docs. Setting 'b' as fixed but not 'a' means that axial ratio, not minor axis, is fixed during the fit.
    • Had problem with retrieving data repos using git. Was being prompted for password to retrieve each file. Darrell pointed me to docs describing how to avoid that.
  • George Moellenbrock
    • CASA Dev talks re VI2, Mar 28 & Apr 4
    • VI1 vs. VI2 comparisons: mostly mechanical differences
      • (ALMA pipeline) Fixed freq solint in freq units parsing problem in VI2 context
      • (EVLA pipeline) Traced solution SNR issues to solution interval sampling differences; fix pending
  • Dirk Petry
  • Martin Pokorny
    • CAS-7086: developing fix for computing statistics across spectral windows of different sizes
  • Federico M Pouzols
    • CAS-9853: new regridding in mstransform/cvel2 similar to tclean: getting it ready for validation for 5.0. Verifying and updating+extending tests.
    • CAS-9473: new flagging mode for antenna integrations. Following-up validation with VLASS data, fixed and waiting for final validation.
    • CAS-9519: tests of the pipeline in parallel. Run times and memory usage for combinations of VI1, VI1CAL v1 or v2.
  • Urvashi Rao
  • Darrell Schiebel
    • finished up last steps for beowulf decommissioning (svn has moved)
    • finished up documentation and changes for casa-data on git (all the rsync issues)
    • documented rsync hosts/processes (in plone)
    • worked up a solution for trapping T/F errors (and prepending a CASAesque warning)
    • worked on version number docs
    • add the setenv of OMP_NUM_THREADS if it is not set
    • explore "casa.qtapp: Fatal IO error: client killed"
    • numpy/matplotlib version conflict wrt histograms
    • fix casa cli assignment protection for things like "a, b = tupleFunc( )"
    • created test for imview(...)
    • explore plotants
  • Ville Suoranta
    • Configured the new Linux build machines
    • Changed build hosts to use Git version of the data repository. Fixed related issues.
    • Fixed breakpad curl library loading on OSX.
    • Fixed CrashReporter file naming bug on OSX.
    • Talked with Jim about better solution for the CrashReporter on/off toggle.
    • Requested larger hard drives for the Mac build servers.
    • Andy provided 2 larger hard drives for the Macs that are running test suite 4.
    • Asked Tracy to increase working drives on three Linux build/test machines.
    • Created MacPorts patch for the matplotlib issue.
    • Installed matplotlib patches on the build servers.
    • Tested Netapp usage for Casa repository with mixed results. Seems to work otherwise fine, but is too slow for the accepted tests. Will attempt to use local disk only for the accepted tests.
    • Need to have two separate checked out copies of the data repository in order to accommodate both shared and local data repository. This may become an issue for hosts with smaller disks.
    • Prep work for 5.0 branching.
    • New docker build machine configuration
  • Takahiro Tsutsumi
    • automask
      • Made a process speed improvement in region labelling in pruneRegions(). A test with a C++ unit test program showed x10 speed up but still needs to be verified by the testers data
      • Added warnings for no mask cases per Amanda’s request (will be checked in after Urvashi’s change is in)
      • Investigated "no mask" issue when tclean was restarted. This was caused by the bad input parameter (cutthreshold). Another case to issue a warning.
    • Reviewed the robust flux bootstrapping ticket (CAS-9186)
    • Briefly looked at the coordsys mismatch issue in parallel continuum tclean (CAS-9977)
Friday NAOJ Meeting
  • Kanako Sugimoto
  • Wataru Kawasaki
  • Masaya Kuniyoshi
  • Takeshi Nakazato
CONFLICT original 10:
  • Renaud Miel
    CONFLICT version 11:
* Renaud Miel * Renaud Miel
CONFLICT end
  • Renaud Miel
Topic revision: r15 - 2017-04-06, JeffKern
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback