Status of wvrgcal in Charlottesville.

As of casapy 3.4 (CAS-3513), wvrgcal is now wrapped as a CASA task. Thus, the rest of this page is mostly obsolete.

This page contains four things:
  1. Links to documentation
  2. Instructions for running wvrgcal on Linux workstations in Charlottesville
  3. The release notes for each version of wvrgcal
  4. Results of regression tests of new versions of wvrgcal copied to (or compiled in) Charlottesville.

Documentation

Latest status for ES Cycle 0 data reduction

Dan Klopp compiled version 0.23.1 which is linked against casa 3.3. Run it at /users/dklopp/wvrgcal/build/bin/wvrgcal ALERT! No, that area (/users/dklopp) no longer exists.

Other versions

As of late February, the Computing group's version of wvrgcal stopped working due to changes in casa's libraries. Remy has compiled a version linked against the casapy 3.3 release. To use it, do the following before you start casapy:
  • .bashrc: export LD_LIBRARY_PATH=/usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib
  • .tcshrc: setenv LD_LIBRARY_PATH /usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib
  • /home/protostar/rindebet/casa/release3.3/linux_64b/bin/wvrgcal

However, this version segfaults on various early science data files. Fortunately, the current builds of casapy-test 3.4 and casapy-stable 3.4 also include a wvrgcal executable linked against the latest casa 3.4 libraries. This resides in /usr/lib64/casapy/test/bin/wvrgcal and is callable from the new wvrgcal task inside casapy. It can also be called from the command line as long as you have set:
  • .tcshrc: setenv LD_LIBRARY_PATH /usr/lib64/casapy/test/lib
  • .bashrc: export LD_LIBRARY_PATH=/usr/lib64/casapy/test/lib

OLD instructions for running wvrgcal on Linux workstations in Charlottesville

  • Starting with version 0.20, the Computing group (Dan Klopp) was tasked with compiling the code from source after each release against the current versions of casa (stable and test). These binaries were located in the following directory in CV (but not on polaris):
    • /home/rhlocal/rhel5/x86_64/stow (also known as: /opt/local/stow)
    • The latest version was 0.23.1 at /opt/local/stow/wvrgcal-23/bin/wvrgcal
  • However, starting with version 1.0, the Computing group was no longer able to produce an executable from the source code. In cv-helpdesk ticket 15457, they decided that they did not need to fix this problem because the casa team was incorporating wvrgcal into casa for the April 2012 release.
  • Starting around Feb 21, 2012, the casa library libmsvis.so.3.4.0 was subsumed into some other library, thus breaking any executable that was linked to it, including the Computing group's version in CV, and the versions linked to casa 3.4 on the mrao groups webpages. Therefore, the only versions that work in CV are the ones in Todd's home area which are linked against libmsvis.so.3.3.0 which is in the November 2011 release. The most recent versions are 1.0 (but 0.23 is also available):
    • /users/thunter/usr/wvrgcal-1.0/bin/wvrgcal
    • /users/thunter/usr/wvrgcal-0.23/bin/wvrgcal
  • You may need to add /usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib to your .tcshrc definition of LD_LIBRARY_PATH to get it to work from Todd's area
  • The intention is that the latest executable linked against casapy-test can always be found at this softlink:
    • /users/thunter/usr/wvrgcal-latest/bin/wvrgcal
  • Older pre-compiled versions are available at Bojan's wvrgcal page

How to run wvrgcal on the regression dataset

The test dataset = /net/yildun/export/data_1/thunter/alma/data/compare_wvrgcal/uid___A002_X15f98e_X141.ms There are three ways to run wvrgcal, from a system call inside of casa, using the attached python wrapper inside of casa, or directly from the linux command-line. The first two are useful for scripts. If it fails, it may be because the casa version on your machine using differs from what wvrgcal was built against. In this case, try the linux command-line option.

Method 1
CASA <2>: os.system('/users/thunter/usr/wvrgcal-latest/bin/wvrgcal --ms /net/yildun/export/data_1/thunter/alma/data/compare_wvrgcal/uid___A002_X15f98e_X141.ms --toffset -1 --output test.wvrgcal')

Method 2
CASA <2>: from wvrgcal import wvrgcal
CASA <3>: wvrgcal(my_ms, my_output_file, tie="2,3", segsource=True)

Method 3
linux% /users/thunter/usr/wvrgcal-latest/bin/wvrgcal --ms /net/yildun/export/data_1/thunter/alma/data/compare_wvrgcal/uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal

If the Linux command line gives an error about loading shared libraries, then you can try appending the following directories to your LD_LIBRARY_PATH:
/usr/lib64/casapy/release/lib:/usr/lib64/casapy/stable/lib:/usr/lib64/casapy/test/lib:/usr/lib64/casapy/lib:
If you still get an error, then the required libraries may not exist on your computer (check with locate) because your casa version is too old or too new. Running 'strace /users/thunter/usr/wvrgcal-latest/bin/wvrgcal' will also tell which libraries it fails on.

Expected output from wvrgcal for the regression dataset

  • Note that the new column "Disc" for Discrepancy between the four channels. This can be used to infer a faulty WVR (e.g. DV05 in the example below) or to infer the presence of clouds. See Bojan's memo for details.

WVRGCAL  -- this is pre-relase software, use at own risk -- Version 0.23

Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)

     Input WVR data information:   
-----------------------------------
Timestamp       Ant. #  [Chn 0, Chn 1, Chn 2, Chn 3]    Elevation       STATE_ID
4.79607e+09     0       [195.17, 97.90, 50.90, 34.45]   0.734991        0

Calculating the coefficients now...done!
       Retrieved parameters      
----------------------------------------------------------------
Evidence        PWV     PWV Error       dTdL    dTdL    dTdL    dTdL
1.38537e-11     0.782041        0.0121128       11.2814 9.45438 5.1288  3.40535

Times used for the statistics caculation (in seconds from first astro datum) 
----------------------------------------------------------------------------
(0, 29.04)
     Antenna/WVR information:                     
-----------------------------------------------------------------------
#       Name    WVR?    Flag?   RMS (um)        Disc (um)
0       DV02    Yes     No      8.95            26.1
1       DV03    Yes     No      8.06            21.5
2       DV04    Yes     No      7.74              18
3       DV05    Yes     No      0.000165                0.000931
4       DV06    Yes     No      18.2              45
5       DV07    Yes     No      5.91            19.4
6       DV08    Yes     No      10.3            24.5
7       PM02    Yes     No      9.03            11.9
  Expected performance 
------------------------------------------------------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 18.1538
* Rough estimate path error due to coefficient error (micron on a baseline): 0.735048

Version 1.1.0 (released Mar 07, 2012)

Release notes for Version 1.1.0

* Fix for bug in tieing phase of sources option (--tie). Multiple sets of sources can now separately have their phase tied together through the use of multiple --tie flags. For example "--tie 0,1,2 --tie 3,4" will prioritising tieing the phase of sources 0,1 and 2 together, and separately will also prioritise tieing the phases of sources 3 and 4.

* (Version 1.0.2) Add an alternative CMake based build system. This should allow building of wvrgcal from within the CASA build system.

* (Version 1.0.1) Add configure option to disable building tests (--disable-buildtests). If this option is enabled, the boost test library is not required. If boost program options library is not found then the building of cmdline subdirectory is automatically disabled. In combination with above change this means that the library can compile with header-only boost libraries

Version 1.0 (released Nov 10, 2011)

Release notes for Version 1.0

* Antennas with names starting with letters "CM" are now assumed not to have a WVR and they are automatically interpolated

* Program wvrgcal has a new option "--scale" which scales the entire phase correction by the amount specified by user (for example "--scale 0.8" applies correction corresponding to 80% of the estimated path fluctuation). The best way to use this option has not yet been investigated in detail so it is not recommended to be used at the present time.

* Antenna RMSs are now corrected for the gross airmass effect (i.e., paths are multiplied by sine of elevation). They should therefore be a better estimate of total stability

* Program wvrgcal now has a new option "--smooth" that allows smoothing in time of the WVR data. The parameter to this option is the width of the smoothing kernel expressed as number of WVR samples.

* Libair can now automatically run its tests by invoking "make check"

Version 0.23.1 (released Aug 23, 2011)

Release notes for version 0.23.1

* New option "--sourceflag" has been added. This option takes a source name, and subsequently disregards the data taken by the WVRs while the antennas were pointing at this source. This means that the phase correction coefficients are not computed for this source and that the output gain calibration has constant zero values during this time.

Note that flagging a source in this way will break its phase relationship to the other sources in the measurement set which are not flagged. You must never flag a phase reference source. This option should be used for example with datasets which have shadowing during absolute flux calibrator observations (it seems to have happened by accident a few times).

Version 0.23 (released July 14, 2011)

Release notes for version 0.23

* Option "--segfield" to the wvrgcal has been depracated. This is because mosaiced observations are recorded as having many different fields and segmenting the phase correction on these boundaries is not appropriate. See the "--segsource" option.

* Option "--segsource" has been added to the wvrgcal program. This option causes the phase correction coefficients to be re-computed when the observing moves to a new source (as opposed to new field with the old "--segfield" option). This option maximises the accuracy of on-source phase correction. Phase transfer between sources may or may not be affected by this option. See also the "--tie" option.

* Option "--tie" has been added to the wvrgcal program. This option instruct the program to ensure that phase transfer between sources in the tie is done as well as possible. This option takes a parameter which is a comma separated list of sources to tie together. Sources can be supplied either as source IDs or source names -- any arguments which have only digits 0-9 will be interrepted as source IDs and everything else as source names.

Note that issue CSV-910 means that source names may not be reliable for some early ALMA data.

* New option "--statsource" has been added to the wvrgcal program. This option works in a similar fashion to the --statfield option, but the statistics are computed for a particular source, not field. As noted for the --segfield, --segsource options, field-based computations are not suitable for most WVR computations because mosaiced observations are recorded as belonging to very many fields.

Version 0.22 (released July 11, 2011)

Release notes for 0.22

* The main improvement in this version is a new feature to help judge the quality of WVR phase corrections (without having to look at lots of noisy phase plots!). This feature consists of two parts: a new column in the output (i.e., the printout) from wvrgcal (labelled "disc") and new option "--statfield".

* The new column is a per-antenna measure of the expected relative quality of WVR phase correction. How this output is expected to work is described in the following draft memo (any comments very much welcome): memoqachannels.pdf

As described in the memo, we think that this quantity should correlate (inversely) with the quality of WVR phase correction but before we assume that it does so it would be much appreciated if you could keep an eye on it and report any cases where the discrepancy is high but correction works well or conversely when the discrepancy is low but the correction is not working well.

* The "--statfield" option allows the user to select which field the "discrepancy" and other statistic output by wvrgcal. Since science measurement sets usually have various calibrators in them, it is necessary to supply this option with name of the science field, e.g., "--statfield NGC253".

Version 0.21.0.1 (released May 17, 2011)

Release notes for 0.21.0.1

* fieldID column of the T-Jones calibration table output by wrvgcal is filled in with the field that was being observed when the WVR data were taken. This fix allows easier per-field calibration application.

Version 0.21 (released April 21, 2011)

Release notes for Version 0.21

* The sign of correction has been reversed to correspond to the ALMA convention. Therefore, the --reverse option should *no longer be needed*

* When WVRs are flagged with --wvrflag, the three nearest antennas are now interpolated as the mean weighted by distance from the flagged antennas (previously this was just an unweighted average). This should improve somewhat the quality of phase correction when WVRs must be flagged

* WVRGCAL now prints path RMS for each antenna in the "Antenna/WVR information" table. These values allow easy identification of problematic WVRs (RMS either much smaller or much greater than the remaining units). These values also give information about overall stability of the atmosphere and different atmospheric conditions across the array

version 0.20.2

  • /home/rhlocal/rhel5/x86_64/stow/wvrgcal-20.2/bin/wvrgcal
[thunter@yildun compare_wvrgcal]$ /home/rhlocal/rhel5/x86_64/stow/wvrgcal-20.2/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal.v19 --wvrflag 3

WVRGCAL  -- this is pre-relase software, use at own risk -- Version 0.20.2

Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)

     Antenna/WVR information:   
--------------------------------
#       Name    WVR?    Flag?
0       DV02    Yes     No
1       DV03    Yes     No
2       DV04    Yes     No
3       DV05    Yes     Yes
4       DV06    Yes     No
5       DV07    Yes     No
6       DV08    Yes     No
7       PM02    Yes     No
     Input WVR data information:   
-----------------------------------
Timestamp       Ant. #  [Chn 0, Chn 1, Chn 2, Chn 3]    Elevation       STATE_ID
4.79607e+09     0       [195.17, 97.90, 50.90, 34.45]   0.734991        0

Calculating the coefficients now...done!
       Retrieved parameters      
---------------------------------
Evidence        PWV     PWV Error       dTdL    dTdL    dTdL    dTdL
1.38537e-11     0.782041        0.0121128       11.2814 9.45438 5.1288  3.40535

  Expected performance 
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603

version 0.20

[thunter@yildun compare_wvrgcal]$ /home/rhlocal/rhel5/x86_64/stow/wvrgcal-20/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal.v19 --wvrflag 3

WVRGCAL  -- this is pre-relase software, use at own risk -- Version 0.20

Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)

     Antenna/WVR information:   
--------------------------------
#       Name    WVR?    Flag?
0       DV02    Yes     No
1       DV03    Yes     No
2       DV04    Yes     No
3       DV05    Yes     Yes
4       DV06    Yes     No
5       DV07    Yes     No
6       DV08    Yes     No
7       PM02    Yes     No
     Input WVR data information:   
-----------------------------------
Timestamp       Ant. #  [Chn 0, Chn 1, Chn 2, Chn 3]    Elevation
4.79607e+09     0       [195.17, 97.90, 50.90, 34.45]   0.734991

Calculating the coefficients now...done!
       Retrieved parameters      
---------------------------------
Evidence        PWV     PWV Error       dTdL    dTdL    dTdL    dTdL
1.38537e-11     0.782041        0.0121128       11.2814 9.45438 5.1288  3.40535

  Expected performance 
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603

Version 0.20b2

  • setenv LD_LIBRARY_PATH /usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib
  • /net/protostar/export/data_1/rindebet/casa/wvr.bojan/bin/wvrgcal
WVRGCAL  -- this is pre-relase software, use at own risk -- Version 0.20b2

Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA
 Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac
.uk/~bn204/alma)

     Antenna/WVR information:   
--------------------------------
#       Name    WVR?    Flag?
0       DV02    Yes     No
1       DV03    Yes     No
2       DV04    Yes     No
3       DV05    Yes     Yes
4       DV06    Yes     No
5       DV07    Yes     No
6       DV08    Yes     No
7       PM02    Yes     No
     Input WVR data information:   
-----------------------------------
Timestamp       Ant. #  [Chn 0, Chn 1, Chn 2, Chn 3]    Elevation
4.79607e+09     0       [195.17, 97.90, 50.90, 34.45]   0.734991

Calculating the coefficients now...done!
       Retrieved parameters      
---------------------------------
Evidence        PWV     PWV Error       dTdL    dTdL    dTdL    dTdL
1.38537e-11     0.782041        0.0121128       11.2814 9.45438 5.1288  3.40535

  Expected performance 
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 
5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
  • OSF-RED
WVRGCAL  -- this is pre-relase software, use at own risk -- Version 0.20

Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)

     Antenna/WVR information:   
--------------------------------
#       Name    WVR?    Flag?
0       DV02    Yes     No
1       DV03    Yes     No
2       DV04    Yes     No
3       DV05    Yes     Yes
4       DV06    Yes     No
5       DV07    Yes     No
6       DV08    Yes     No
7       PM02    Yes     No
     Input WVR data information:   
-----------------------------------
Timestamp       Ant. #  [Chn 0, Chn 1, Chn 2, Chn 3]    Elevation
4.79607e+09     0       [195.17, 97.90, 50.90, 34.45]   0.734991

Calculating the coefficients now...done!
       Retrieved parameters      
---------------------------------
Evidence        PWV     PWV Error       dTdL    dTdL    dTdL    dTdL
1.38537e-11     0.782041        0.0121128       11.2814 9.45438 5.1288  3.40535

  Expected performance 
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603

Version 0.19

  • setenv LD_LIBRARY_PATH /usr/lib64/casapy/lib64/
  • CV: phase vs time
    • /net/protostar/export/data_1/rindebet/casa/wvr/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal.v19 --wvrflag 3
Calculating the coefficients...
Input data for solution #0 is row:13
195.167, 97.9027, 50.9008, 34.4517, 
Retrieval at 4796071280.792000 is: [11281.4,9454.38,5128.8,3405.35,]
done!
** Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
** Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
** Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
Calculating the coefficients...
Input data for solution #0 is row:13
195.167, 97.9027, 50.9008, 34.4517, 
Retrieval at 4796071280.792000 is: [11281.4,9454.38,5128.8,3405.35,]
done!
** Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
** Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
** Rough estimate path error due to coefficient error (micron on a baseline): 0.813603

Version 0.17

ALERT! Dan Klopp has left NRAO; any references to files owned by him needs to be considered deprecated.

  • Howto on nash:
    • export LD_LIBRARY_PATH=/opt/boost/lib:$LD_LIBRARY_PATH
    • /export/data_1/dklopp/libair/bin/wvrgcal ALERT! No, that area no longer exists
  • using dataset: uid___A002_X15f98e_X141
    • 8 antennas: DV02,3,4,5,6,7,8,PM02
    • Band 3
    • 10-Nov-2010/02:01:06.8, total time = 30.672 sec
  • /export/data_1/dklopp/libair/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal ALERT! No, that area no longer exists
  • phase: nash   osf-red
  • amplitude: nash   osf-red

Todd's notes for trying to build on a mac

  • install bnmin1 (get source code
    • sudo ./configure LDFLAGS="-L/Library/Frameworks/Python.framework/Versions/2.7/lib"
    • sudo make
      • Error: SWIG version >= 1.3.31 is required. You have 2.0.3. You should look at http://www.swig.org
  • install libair

-- ToddHunter - 2010-11-12
Topic attachments
ISorted ascending Attachment Action Size Date Who Comment
cv_plotcalpng.pngpng cv_plotcalpng.png manage 55 K 2010-12-13 - 16:44 ToddHunter  
nash_amp_plotcalpng.pngpng nash_amp_plotcalpng.png manage 21 K 2010-11-12 - 15:25 ToddHunter  
nash_plotcalpng.pngpng nash_plotcalpng.png manage 59 K 2010-11-12 - 15:25 ToddHunter  
osfred_amp_plotcalpng.pngpng osfred_amp_plotcalpng.png manage 19 K 2010-11-12 - 15:26 ToddHunter  
osfred_plotcal_v19.pngpng osfred_plotcal_v19.png manage 55 K 2010-12-13 - 16:43 ToddHunter  
osfred_plotcalpng.pngpng osfred_plotcalpng.png manage 58 K 2010-11-12 - 15:26 ToddHunter  
extprog.py.txttxt extprog.py.txt manage 1 K 2011-07-22 - 03:59 ToddHunter Bojan's wrapper function for calling wvrgcal inside casapy
wvrgcal.py.txttxt wvrgcal.py.txt manage 1 K 2012-01-11 - 11:25 ToddHunter  
Topic revision: r35 - 2015-04-28, PatrickMurphy
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback