Monday Morning Meeting September 29

• DIAL-IN NUMBERS & PASSCODES:
• USA Toll Free Number: 866-901-8266
• USA Toll Number: +1-203-566-3863
• PARTICIPANT PASSCODE: 725760

Attendance

• Socorro: Jeff, Rob, George, Lindsey, Susan, Jim, Urvashi, Tak, Sanjay, Juergen, Pam, Kumar

• CV: Andy, Dave, Darrell, Mark
• Garching: Sandra, Julian
• SCO:

News / Meetings / Visitors

• Congratulations Sandra Castro - Promoted to Pipeline Group Coordinator at ESO (30% effort)
• Welcome Julian Taylor - member of pipeline team, working at the VLT. Will join CASA at the 30% level.
• Congratulations Rob Selina - moving to Systems Group Lead in Electronics Division. Transition TBD.

• CASA Users Committee
  • No formal report yet. (1-2 weeks out)
  • Headline was strong support for current priorities and general direction of CASA.
  • Issues raised:
    • Communication w/ User Community - a bit too stove-piped. Need to be more open with the user community. They all learned a lot from the presentations, suggested info be shared more widely. Semi-annual newsletter suggested as a fix (offset from the releases). Likely written as a series of small sections, contributed by Scientists/Developers with Juergen as editor. Semi-annual publication is consistent with Eric's practice with AIPS.
      • Juergen may also do a user survey. Welcomes input.
    • SD Effort over-emphasis - May just be an interferometry bias, or that we failed to present the scope of the SD team efforts (development & pipeline). Can make it clear they are not overstaffed with the MS transition, and then there is greater integration with the SD/Interferometry tasks so the concern is also addressed.
    • Emphasize Robustness - Concern with CASA crashing as well as speed.
      • Ctrl-C (impatience) is a source of many crash reports. Part of the difficulty is to understand the nature of various "crashes" since there are a wide variety of problems, many outside of our control. Some solved with education.
      • Need to improve feedback on crashes. Perhaps generate a crash reporter?
      • Exception messages that are not understood may also be part of the problem.
      • May want to approach the committee to identify a few power users who are willing to be testers. Could have them report seg faults and similar errors and work the issue with a CASA team member.

Build, Release, Testing

• B&T Requirements capture
  • Mark to circulate to casa-staff for final comment.

• 4.2.2-Pipe
  • Almost ready for release.
  • Pipeline group working on documentation.
  • Manually added pipeline to 4.2.2 tar. Enabled with a switch at start up.
  • No technical issues with pipeline on OSX!

• 4.3 pre-release packages (10.8/10.9, EL5/EL6)
  • Waiting on Weight Spectrum in Solves. George is also working on some regression failures from recent commits.
  • Once current commits pass regressions, roll the pre-release package. (Weight Spectrum in Solves will follow after).

• Test builds (off trunk)
  • Now building EL5 and EL6 nightly.
  • When declared stable, will switch to distributing these. Currently can be installed manually.
  • Jeff and Darrell to talk off-line about deploying these builds on the cluster.
  • Semantic change in the definition of "test" and "stable".
  • Darrell is working on a wiki page to describe the change.
  • Concern with stability, including data repository. May need a gate (time-based) to decide when users/testers/pipeline decide to update.
    • Need to watch how we deploy these packages in the building.
    • Should have a consistent policy in NM, CV, ESO, JAO.

• Problem deploying 4.2.2 RHEL5 RPMs at the AOC. K Scott and Darrell are working the issue.
• EL6 deployment of 4.2.2 is working at the moment.

Architecture

• Technical Forums:
  • Push back Build & Test Requirements capture (TBD)
  • CASACore technical meeting (TBD)
  • Sandra and Rob to work with help desks to allow for video link (Socorro, CV, Garching)

Development

• HPC working meeting in Socorro - Dec 1st - 5th
  • Final agenda will be set a little closer to the date (will depend on current status).
  • Likely to focus on (some subset of) environment management, build system integration, testing of features developed for 4.3 cycle, and parallel pipeline.
  • Jeff invited Julian to attend work session.

• Weight Spectrum commit (4.3)
  • Weight spectrum in Solves pending.
  • regression tests failing.

• 4.4 Planning
  • CSSC input may come as last minute deltas.
  • Have feedback from CASA team and pipelines, as well as some collaboration/projects.

Pipeline

• User testing of Linux and OSX pipeline distributions, plotms write lock problem, speed and memory issues.
  • Susan has a fix in for plotms, but trying to avoid rolling a new patch to the release (4.2.3). Eductate users about resource problems instead.
• Operations bug reports
  • WVR bug identified along with 5-6 others so far. Reason for optimism!
  • Recent batch processing results seem well accepted.
• Claire working on a VLA scripted pipeline for 4.2.2.
  • Comparing to CASA pipeline, but it has not been accepted yet.
  • Will run both in parallel for a while, then eventually move to CASA pipeline.

AOB

• Discuss Sandra's polarization issue next meeting. (Put on agenda under development).
• 4826 regression failure coincides with weight spectrum commit, but George isn't convinced that's the nature of the cause. Calibration impact is thin. Mosaic changes could be related?
  • Andy is also fishing around on this failure.

Developer Reports

Monday Meeting

• Sanjay Bhatnagar
• Sandra Castro
  • Fixed a bug in mstransform when datacolumn='data,model,corrected' was requested.
  • Steve's ticket (CAS-6946) seems to show that there is a bug in the VI/VB2 which doesn't take correlation selection into account for the WEIGHT/SIGMA columns. I have created tests for mstransform which are skipped now until the issue is fixed. The ticket is assigned to Jim for verification.
  • I found problems when creating an MMS from an MS that contains the WVR spw. The polarization table is not correct. The fix to CAS-6733 exposed this problem, so I have re-opened this ticket. There is a big method inherited from SubMS (fillDDTables) that probably needs to be re-written to take all these problems into account. (CAS-6941, CAS-6951).
  • We need to make a decision about the POLARIZATION table and whether we want to re-index the POLARIZATION_ID and re-size the POLARIZATION table. With these latest MSs, the old way of slicing the POLARIZATION table columns is not enough. An example follows of an EVLA MS:

SpwID  #Chans      CtrFreq(MHz)    Corrs 
0         256        4968.9961       RR
0         256        4968.9961       LL
1         128        4968.9922       RR  LL
2         64         4968.9844       RR  RL  LR  LL

DATA_DESCRIPTION table:

DD    POLID    FLAG_ROW    SPW_ID
 0     0        0            0
 1     1        0            0
 2     2        0            1
 3     3        0            2

POLARIZATION table:

POL_ID    CORR_TYPE    CORR_PRODUCT    NUM_CORR
0         [5]          [2,1]           1
1         [8]          [2,1]           1
2         [5,8]        [2,2]           2
3         [5,6,7,8]    [2,4]           4

If I select spw='0', correlation='LL', only DATA_DESCRIPTION 1 should be selected, but the output POLARIZATION sub-table will have odd entries because it keeps all the rows in place.

POL_ID    CORR_TYPE    CORR_PRODUCT    NUM_CORR
0         []           [2,0]           0
1         [8]          [2,1]           1
2         [8]          [2,1]           1
3         [8]          [2,1]           1

I think the correct table should have only the selected POLARIZATION, which in this case would be re-index from 1 to 0 and the output would look like this:

POL_ID    CORR_TYPE    CORR_PRODUCT    NUM_CORR
0         [8]           [2,1]           1

This will affect any other task that creates an output MS and needs to re-create the DD and POL tables. This may also affect scripts that rely on the POLARIZATION index. I think for the user's scripts, it may not affect much because they use the correlation name for the selections. XX, RR, etc. will continue to be called this, but the index that points to them will change. All this is described in CAS-6400.

• Lindsey Davis
  • CASA user's committee meeting, pipeine demo
  • Simple spectral window mapping task results prepared and released for tester evaluation
  • Followup on ALMA interferometry pipeline release issues, data set inhomgeneities, web log bugs, SNR problems, ...
  • Followup on EVLA pipeline 4.2.2 issues
  • ADASS poster
• Pam Ford
  • CAS-2884 - r31153 - avgchannel='1' behaves the same way as avgchannel='0', i.e. no channel averaging
  • CAS-6939 - investigating - plotms goes into infinite loop depending on grid dimensions
• Kumar Golap
  • Put some finishing touches and bug fix on mosaic mode in tclean
    • The mosaic mode will take in the static vpmanager definition now in tclean like dirk wanted it
    • vpmanager is the token for beams for mosaic in tclean for ALMA and unkonwn telescopes (will need to expand it to all telescopes)
  • My computer got upgraded ...so lots of time in copying data from one disk to another etc and setting up all my automatic crontab jobs and scripts to work properly.
• Justo González Villalba
• Jim Jacobs
  • Looked into problem where MMSs with large number of subMSs require a huge number of simultaneously open files. Hoped to find a quick fix: failed. Solution requires modifying low-level CasaCore to not keep all MSs in a ConcatTable open at the same time. Given scope for regressions, this should be attacked early in 4.4 cycle.
  • Continued working on state representation for Carta. Some light at the end of the tunnel?
• Jeff Kern
• Susan Loveland
  • Viewer:
    • CAS-6807: Overplotting not correct for certain images when converting to Kelvin & fix of segfault when displaying channels and converting to Kelvin.
    • CAS-6354: Profiler needs to set quality axis for [Data,Error] images in rewrite of getFreqProfile and Optical 2D fit not refreshing.
  • Plotms:
    • CAS-6820: Segfault for certain calibration tables when using an axis of Swpower.
  • CARTA
• David Mehringer
  • CAS-3116 spxfit, after three months, got another batch of comments from the tester to which I responded within 24 hours. Testing for this ticket has now continued for over a year, with most of that time spent with the ticket languishing awaiting new test packages and tester comments. This ticket either needs to be closed or needs to be assigned to a tester who is going to provide a quicker turn around time IMO.
  • Responded to Jim's questions re: my design for the stats framework. Explained that I really desire feedback from team members who would be using it and that now is the best time to provide that feedback since, if I code it up and people then realize they do not like it and do not want to use it, it's a waste of project resources. It would be good if those with a vested interest would actively participate in the design process. If that is not possible, management should re-evaluate the cost/benefit of this work IMO.
  • CAS-6954 imhead fails to put new value in FITS header, no error message: Added error messages and updated docs, re: write modes used for read-only (FITS) images
  • CAS-6354: Much time spent investigating major differences between current spectral plots and those of the 4.2.1 viewer. Finally concluded there was a bug in how the viewer code was selecting the region to pass to PixelValueManipulator::getProfile(). Susan quickly fixed that, and things look much better now. Handed back to Andy for continued testing
• George Moellenbrock
  • CASA Users' Committee preparations and attendance (Calibration Status talk)
  • WEIGHT_SPECTRUM: checked in applycal context usage. Repaired some regression failures.
• Dirk Petry
• Urvashi Rao
• Darrell Schiebel
• Rob Selina
• Julian Taylor
• Takahiro Tsutsumi
  • for last two weeks
    • Setjy scalebychan log message improvement
    • While looking at the above, found a bug in a helper function(that returns min/max frequencies) in MSUtil
    • Fixed test_clean failure. Updated the reference values due to the MSUtil bug fix
    • Made fluxscale to work when multiple reference fields are specified and some of them has been flagged in the input caltable ( CAS-4758)
    • Minor fixes in mask handling code

Friday NAOJ Meeting

• Kanako Sugimoto
  • Listed up CASA 4.4 targets with Bunyo: Next step is to define schedule and assignments, and create JIRA tickets.
  • Presentation at CASA user committee meeting
  • sdplot: enable select data by raster row and map iteration ( CAS-6704): Ready To Test
  • Discussion meeting with Nobeyama staffs on collabolation of NRO and ALMA-J (not limited to but including CASA)
  • Masaya Kuniyoshi, a new developer, will join CASA-SD team in Oct. 1st.
• Wataru Kawasaki
  • add functionality to select data by raster visits in raster mapping ( CAS-6703): RTT
• Takeshi Nakazato
• Shinnosuke Kawakami
  • first skypecon with B&T team
  • GC-casacore + casacode failed ( CAS-6929 : in progress)