Status of wvrgcal in Charlottesville.
As of casapy 3.4 (CAS-3513), wvrgcal is now wrapped as a CASA task. Thus, the rest of this page is mostly obsolete.
This page contains four things:
- Links to documentation
- Instructions for running wvrgcal on Linux workstations in Charlottesville
- The release notes for each version of wvrgcal
- Results of regression tests of new versions of wvrgcal copied to (or compiled in) Charlottesville.
Documentation
Latest status for ES Cycle 0 data reduction
Dan Klopp compiled version 0.23.1 which is linked against casa 3.3. Run it at /users/dklopp/wvrgcal/build/bin/wvrgcal No, that area (
/users/dklopp
) no longer exists.
Other versions
As of late February, the Computing group's version of wvrgcal stopped working due to changes in casa's libraries. Remy has compiled a version linked against the casapy 3.3 release. To use it, do the following
before you start casapy:
- .bashrc: export LD_LIBRARY_PATH=/usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib
- .tcshrc: setenv LD_LIBRARY_PATH /usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib
- /home/protostar/rindebet/casa/release3.3/linux_64b/bin/wvrgcal
However, this version segfaults on various early science data files. Fortunately, the current builds of casapy-test 3.4 and casapy-stable 3.4 also include a wvrgcal executable linked against the latest casa 3.4 libraries. This resides in /usr/lib64/casapy/test/bin/wvrgcal and is callable from the new wvrgcal task inside casapy. It can also be called from the command line as long as you have set:
- .tcshrc: setenv LD_LIBRARY_PATH /usr/lib64/casapy/test/lib
- .bashrc: export LD_LIBRARY_PATH=/usr/lib64/casapy/test/lib
OLD instructions for running wvrgcal on Linux workstations in Charlottesville
- Starting with version 0.20, the Computing group (Dan Klopp) was tasked with compiling the code from source after each release against the current versions of casa (stable and test). These binaries were located in the following directory in CV (but not on polaris):
- /home/rhlocal/rhel5/x86_64/stow (also known as: /opt/local/stow)
- The latest version was 0.23.1 at /opt/local/stow/wvrgcal-23/bin/wvrgcal
- However, starting with version 1.0, the Computing group was no longer able to produce an executable from the source code. In cv-helpdesk ticket 15457, they decided that they did not need to fix this problem because the casa team was incorporating wvrgcal into casa for the April 2012 release.
- Starting around Feb 21, 2012, the casa library libmsvis.so.3.4.0 was subsumed into some other library, thus breaking any executable that was linked to it, including the Computing group's version in CV, and the versions linked to casa 3.4 on the mrao groups webpages. Therefore, the only versions that work in CV are the ones in Todd's home area which are linked against libmsvis.so.3.3.0 which is in the November 2011 release. The most recent versions are 1.0 (but 0.23 is also available):
- /users/thunter/usr/wvrgcal-1.0/bin/wvrgcal
- /users/thunter/usr/wvrgcal-0.23/bin/wvrgcal
- You may need to add /usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib to your .tcshrc definition of LD_LIBRARY_PATH to get it to work from Todd's area
- The intention is that the latest executable linked against casapy-test can always be found at this softlink:
- /users/thunter/usr/wvrgcal-latest/bin/wvrgcal
- Older pre-compiled versions are available at Bojan's wvrgcal page
How to run wvrgcal on the regression dataset
The test dataset = /net/yildun/export/data_1/thunter/alma/data/compare_wvrgcal/uid___A002_X15f98e_X141.ms There are three ways to run wvrgcal, from a system call inside of casa, using the attached
python wrapper inside of casa, or directly from the linux command-line. The first two are useful for scripts. If it fails, it may be because the casa version on your machine using differs from what wvrgcal was built against. In this case, try the linux command-line option.
Method 1
CASA <2>: os.system('/users/thunter/usr/wvrgcal-latest/bin/wvrgcal --ms /net/yildun/export/data_1/thunter/alma/data/compare_wvrgcal/uid___A002_X15f98e_X141.ms --toffset -1 --output test.wvrgcal')
Method 2
CASA <2>: from wvrgcal import wvrgcal
CASA <3>: wvrgcal(my_ms, my_output_file, tie="2,3", segsource=True)
Method 3
linux% /users/thunter/usr/wvrgcal-latest/bin/wvrgcal --ms /net/yildun/export/data_1/thunter/alma/data/compare_wvrgcal/uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal
If the Linux command line gives an error about loading shared libraries, then you can try appending the following directories to your LD_LIBRARY_PATH:
/usr/lib64/casapy/release/lib:/usr/lib64/casapy/stable/lib:/usr/lib64/casapy/test/lib:/usr/lib64/casapy/lib:
If you still get an error, then the required libraries may not exist on your computer (check with locate) because your casa version is too old or too new. Running 'strace /users/thunter/usr/wvrgcal-latest/bin/wvrgcal' will also tell which libraries it fails on.
Expected output from wvrgcal for the regression dataset
- Note that the new column "Disc" for Discrepancy between the four channels. This can be used to infer a faulty WVR (e.g. DV05 in the example below) or to infer the presence of clouds. See Bojan's memo for details.
WVRGCAL -- this is pre-relase software, use at own risk -- Version 0.23
Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)
Input WVR data information:
-----------------------------------
Timestamp Ant. # [Chn 0, Chn 1, Chn 2, Chn 3] Elevation STATE_ID
4.79607e+09 0 [195.17, 97.90, 50.90, 34.45] 0.734991 0
Calculating the coefficients now...done!
Retrieved parameters
----------------------------------------------------------------
Evidence PWV PWV Error dTdL dTdL dTdL dTdL
1.38537e-11 0.782041 0.0121128 11.2814 9.45438 5.1288 3.40535
Times used for the statistics caculation (in seconds from first astro datum)
----------------------------------------------------------------------------
(0, 29.04)
Antenna/WVR information:
-----------------------------------------------------------------------
# Name WVR? Flag? RMS (um) Disc (um)
0 DV02 Yes No 8.95 26.1
1 DV03 Yes No 8.06 21.5
2 DV04 Yes No 7.74 18
3 DV05 Yes No 0.000165 0.000931
4 DV06 Yes No 18.2 45
5 DV07 Yes No 5.91 19.4
6 DV08 Yes No 10.3 24.5
7 PM02 Yes No 9.03 11.9
Expected performance
------------------------------------------------------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 18.1538
* Rough estimate path error due to coefficient error (micron on a baseline): 0.735048
Version 1.1.0 (released Mar 07, 2012)
Release notes for Version 1.1.0
* Fix for bug in tieing phase of sources option (--tie). Multiple sets
of sources can now separately have their phase tied together through
the use of multiple --tie flags. For example "--tie 0,1,2 --tie 3,4"
will prioritising tieing the phase of sources 0,1 and 2 together,
and separately will also prioritise tieing the phases of sources 3
and 4.
* (Version 1.0.2) Add an alternative CMake based build system. This should allow
building of wvrgcal from within the CASA build system.
* (Version 1.0.1) Add configure option to disable building tests
(--disable-buildtests). If this option is enabled, the boost test
library is not required. If boost program options library is not found then the building of
cmdline subdirectory is automatically disabled. In combination with
above change this means that the library can compile with
header-only boost libraries
Version 1.0 (released Nov 10, 2011)
Release notes for Version 1.0
* Antennas with names starting with letters "CM" are now assumed not
to have a WVR and they are automatically interpolated
* Program wvrgcal has a new option "--scale" which scales the entire
phase correction by the amount specified by user (for example
"--scale 0.8" applies correction corresponding to 80% of the
estimated path fluctuation). The best way to use this option has not
yet been investigated in detail so it is not recommended to be used
at the present time.
* Antenna RMSs are now corrected for the gross airmass effect (i.e.,
paths are multiplied by sine of elevation). They should therefore be
a better estimate of total stability
* Program wvrgcal now has a new option "--smooth" that allows
smoothing in time of the WVR data. The parameter to this option is
the width of the smoothing kernel expressed as number of WVR samples.
* Libair can now automatically run its tests by invoking "make check"
Version 0.23.1 (released Aug 23, 2011)
Release notes for version 0.23.1
* New option "--sourceflag" has been added. This option takes a source name,
and subsequently disregards the data taken by the WVRs while the antennas
were pointing at this source. This means that the phase
correction coefficients are not computed for this source and that
the output gain calibration has constant zero values during this
time.
Note that flagging a source in this way will break its phase
relationship to the other sources in the measurement set which are
not flagged. You must never flag a phase reference source.
This option should be used for example with datasets which have shadowing during
absolute flux calibrator observations (it seems to have happened by
accident a few times).
Version 0.23 (released July 14, 2011)
Release notes for version 0.23
* Option "--segfield" to the wvrgcal has been depracated. This is
because mosaiced observations are recorded as having many different
fields and segmenting the phase correction on these boundaries is
not appropriate. See the "--segsource" option.
* Option "--segsource" has been added to the wvrgcal program. This
option causes the phase correction coefficients to be re-computed
when the observing moves to a new source (as opposed to new field
with the old "--segfield" option). This option maximises the
accuracy of on-source phase correction. Phase transfer between
sources may or may not be affected by this option. See also the
"--tie" option.
* Option "--tie" has been added to the wvrgcal program. This option
instruct the program to ensure that phase transfer between sources
in the tie is done as well as possible. This option takes a
parameter which is a comma separated list of sources to tie
together. Sources can be supplied either as source IDs or source
names -- any arguments which have only digits 0-9 will be
interrepted as source IDs and everything else as source names.
Note that issue
CSV-910 means that source names may not be reliable
for some early ALMA data.
* New option "--statsource" has been added to the wvrgcal
program. This option works in a similar fashion to the --statfield
option, but the statistics are computed for a particular source, not
field. As noted for the --segfield, --segsource options, field-based
computations are not suitable for most WVR computations because
mosaiced observations are recorded as belonging to very many fields.
Version 0.22 (released July 11, 2011)
Release notes for 0.22
* The main improvement in this version is a new feature to help judge the
quality of WVR phase corrections (without having to look at lots of
noisy phase plots!). This feature consists of two parts: a new column in
the output (i.e., the printout) from wvrgcal (labelled "disc") and new
option "--statfield".
* The new column is a per-antenna measure of the expected relative quality
of WVR phase correction. How this output is expected to work is
described in the following
draft memo (any comments very much
welcome):
memoqachannels.pdf
As described in the memo, we think that this quantity should correlate
(inversely) with the quality of WVR phase correction but before we
assume that it does so it would be much appreciated if you could keep an
eye on it and report any cases where the discrepancy is high but
correction works well or conversely when the discrepancy is low but the
correction is not working well.
* The "--statfield" option allows the user to select which field the
"discrepancy" and other statistic output by wvrgcal. Since science
measurement sets usually have various calibrators in them, it is
necessary to supply this option with name of the science field, e.g.,
"--statfield NGC253".
Version 0.21.0.1 (released May 17, 2011)
Release notes for 0.21.0.1
* fieldID column of the T-Jones calibration table output by wrvgcal is
filled in with the field that was being observed when the WVR data
were taken. This fix allows easier per-field calibration application.
Version 0.21 (released April 21, 2011)
Release notes for Version 0.21
* The sign of correction has been reversed to correspond to the ALMA
convention. Therefore, the --reverse option should *no longer be
needed*
* When WVRs are flagged with --wvrflag, the three nearest antennas are
now interpolated as the mean weighted by distance from the flagged
antennas (previously this was just an unweighted average). This
should improve somewhat the quality of phase correction when WVRs
must be flagged
* WVRGCAL now prints path RMS for each antenna in the "Antenna/WVR
information" table. These values allow easy identification of
problematic WVRs (RMS either much smaller or much greater than the
remaining units). These values also give information about overall
stability of the atmosphere and different atmospheric conditions
across the array
version 0.20.2
- /home/rhlocal/rhel5/x86_64/stow/wvrgcal-20.2/bin/wvrgcal
[thunter@yildun compare_wvrgcal]$ /home/rhlocal/rhel5/x86_64/stow/wvrgcal-20.2/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal.v19 --wvrflag 3
WVRGCAL -- this is pre-relase software, use at own risk -- Version 0.20.2
Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)
Antenna/WVR information:
--------------------------------
# Name WVR? Flag?
0 DV02 Yes No
1 DV03 Yes No
2 DV04 Yes No
3 DV05 Yes Yes
4 DV06 Yes No
5 DV07 Yes No
6 DV08 Yes No
7 PM02 Yes No
Input WVR data information:
-----------------------------------
Timestamp Ant. # [Chn 0, Chn 1, Chn 2, Chn 3] Elevation STATE_ID
4.79607e+09 0 [195.17, 97.90, 50.90, 34.45] 0.734991 0
Calculating the coefficients now...done!
Retrieved parameters
---------------------------------
Evidence PWV PWV Error dTdL dTdL dTdL dTdL
1.38537e-11 0.782041 0.0121128 11.2814 9.45438 5.1288 3.40535
Expected performance
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
version 0.20
[thunter@yildun compare_wvrgcal]$ /home/rhlocal/rhel5/x86_64/stow/wvrgcal-20/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal.v19 --wvrflag 3
WVRGCAL -- this is pre-relase software, use at own risk -- Version 0.20
Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)
Antenna/WVR information:
--------------------------------
# Name WVR? Flag?
0 DV02 Yes No
1 DV03 Yes No
2 DV04 Yes No
3 DV05 Yes Yes
4 DV06 Yes No
5 DV07 Yes No
6 DV08 Yes No
7 PM02 Yes No
Input WVR data information:
-----------------------------------
Timestamp Ant. # [Chn 0, Chn 1, Chn 2, Chn 3] Elevation
4.79607e+09 0 [195.17, 97.90, 50.90, 34.45] 0.734991
Calculating the coefficients now...done!
Retrieved parameters
---------------------------------
Evidence PWV PWV Error dTdL dTdL dTdL dTdL
1.38537e-11 0.782041 0.0121128 11.2814 9.45438 5.1288 3.40535
Expected performance
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
Version 0.20b2
- setenv LD_LIBRARY_PATH /usr/lib64/casapy/release/lib:/usr/lib64/casapy/lib
- /net/protostar/export/data_1/rindebet/casa/wvr.bojan/bin/wvrgcal
WVRGCAL -- this is pre-relase software, use at own risk -- Version 0.20b2
Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA
Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac
.uk/~bn204/alma)
Antenna/WVR information:
--------------------------------
# Name WVR? Flag?
0 DV02 Yes No
1 DV03 Yes No
2 DV04 Yes No
3 DV05 Yes Yes
4 DV06 Yes No
5 DV07 Yes No
6 DV08 Yes No
7 PM02 Yes No
Input WVR data information:
-----------------------------------
Timestamp Ant. # [Chn 0, Chn 1, Chn 2, Chn 3] Elevation
4.79607e+09 0 [195.17, 97.90, 50.90, 34.45] 0.734991
Calculating the coefficients now...done!
Retrieved parameters
---------------------------------
Evidence PWV PWV Error dTdL dTdL dTdL dTdL
1.38537e-11 0.782041 0.0121128 11.2814 9.45438 5.1288 3.40535
Expected performance
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna):
5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
WVRGCAL -- this is pre-relase software, use at own risk -- Version 0.20
Developed by Bojan Nikolic at the University of Cambridge as part of EU FP6 ALMA Enhancement
GPLv2 License -- you have a right to the source code (see http://www.mrao.cam.ac.uk/~bn204/alma)
Antenna/WVR information:
--------------------------------
# Name WVR? Flag?
0 DV02 Yes No
1 DV03 Yes No
2 DV04 Yes No
3 DV05 Yes Yes
4 DV06 Yes No
5 DV07 Yes No
6 DV08 Yes No
7 PM02 Yes No
Input WVR data information:
-----------------------------------
Timestamp Ant. # [Chn 0, Chn 1, Chn 2, Chn 3] Elevation
4.79607e+09 0 [195.17, 97.90, 50.90, 34.45] 0.734991
Calculating the coefficients now...done!
Retrieved parameters
---------------------------------
Evidence PWV PWV Error dTdL dTdL dTdL dTdL
1.38537e-11 0.782041 0.0121128 11.2814 9.45438 5.1288 3.40535
Expected performance
-----------------------
* Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
* Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
* Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
Version 0.19
- setenv LD_LIBRARY_PATH /usr/lib64/casapy/lib64/
- CV: phase vs time
- /net/protostar/export/data_1/rindebet/casa/wvr/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal.v19 --wvrflag 3
Calculating the coefficients...
Input data for solution #0 is row:13
195.167, 97.9027, 50.9008, 34.4517,
Retrieval at 4796071280.792000 is: [11281.4,9454.38,5128.8,3405.35,]
done!
** Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
** Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
** Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
Calculating the coefficients...
Input data for solution #0 is row:13
195.167, 97.9027, 50.9008, 34.4517,
Retrieval at 4796071280.792000 is: [11281.4,9454.38,5128.8,3405.35,]
done!
** Estimated WVR thermal contribution to path fluctuations (micron per antenna): 5.56974
** Greatest Estimated path fluctuation is (micron on a baseline): 20.0939
** Rough estimate path error due to coefficient error (micron on a baseline): 0.813603
Version 0.17
Dan Klopp has left NRAO; any references to files owned by him needs to be considered deprecated.
- Howto on nash:
- export LD_LIBRARY_PATH=/opt/boost/lib:$LD_LIBRARY_PATH
-
/export/data_1/dklopp/libair/bin/wvrgcal No, that area no longer exists
- using dataset: uid___A002_X15f98e_X141
- 8 antennas: DV02,3,4,5,6,7,8,PM02
- Band 3
- 10-Nov-2010/02:01:06.8, total time = 30.672 sec
-
/export/data_1/dklopp/libair/bin/wvrgcal --ms uid___A002_X15f98e_X141.ms --toffset -1 --output uid___A002_X15f98e_X141.wvrgcal No, that area no longer exists
- phase: nash osf-red
- amplitude: nash osf-red
Todd's notes for trying to build on a mac
- install bnmin1 (get source code
- sudo ./configure LDFLAGS="-L/Library/Frameworks/Python.framework/Versions/2.7/lib"
- sudo make
- Error: SWIG version >= 1.3.31 is required. You have 2.0.3. You should look at http://www.swig.org
- install libair
--
ToddHunter - 2010-11-12