ALCHEMI Data Reduction Calibrator Flux Analysis

TIP Last Modified: JeffMangum - 06 August 2019

Table of Contents:


AI M02: Extract all the gain-calibrator flux measurements in the pipeline weblog and check how much the measurements in each EB deviate from a power-law. Are there SPW-based amplitude gaps? How do the gain calibrators look?

Check to see if the SPW-based flux densities of the calibrators deviate significantly from a power-law (or, virtually equivalently, a linear function of freq.) in individual EBs. The pipeline calibration does not use the obvious constraint that any quasar should have a power law (or linear) spectrum among the four SPWs. Not only that, the pipeline does not seem to warn even if its flux-calibration results do not satisfy this power-law (or linear) rule. If such poor flux calibration exists and is transferred to our science data, we will have SPW-to-SPW amplitude gaps, which hurts our SG-based flux normalization using overlapping SPWs. There are a small number of SGs that have visible SPW-to-SPW gaps in the galaxy data. Hence this task to see SPW-to-SPW gaps in gain calibrator flux densities.


Do the following for every individual tunings, both for 12m and 7m observations (separately for 12m-compact and 12m-extended configs in Band 3).
  1. Collect the pipeline flux measurements of gain calibrators (in hifa_gfluxscale) from webLog.
  2. For each execution block, fit the measured flux densities with a linear function (upper panel in the figure below) and then divide each measurement with the fitted value.
  3. Plot those normalized measurements (lower panel in the figure below). If there are significant SPW-to-SPW variation of scaling factors, in particular between the two SPWs in the same sideband, that should show up here as significant deviation from unity.

Sample script for this task: (It does 2 and 3 above. Data collection in 1 is manual, and needs to be automated.)

Result example:


Current Status

JeffMangum picked-up this task and used (see below) to extract all (as of May 2019) phase calibrator information from the ALCHEMI (only project 2017.1.00161.L for now) weblogs. Created (based on above) to read output from (file is fluxes-edit.ipac). The IPAC file written by contains the following columns of information:
  • SB_name
  • UID
  • Calibrator name
  • SPW number
  • Date (observed)
  • Frequency (GHz)
  • Flux (Jy)
  • Flux error (Jy)
  • Catalog flux (Jy)

A sampling of results plots is provided in the following. All results plots can be obtained from this folder.

1 1.pngdiff1-1.pngflux.ngc253 a 03 TM1.pngflux.ngc253 a 03 TM2.pngflux.ngc253 a 04 7M.pngflux.ngc253 a 04 TM1.pngpd1-1.png

I had the script spit-out some statistics related to the linear fit and the residuals of this linear fit. As one can see from this listing that:
  • The mean deviation is rarely more than a few tenths of a percent. This suggests that the SPW-to-SPW absolute flux calibration within a given tuning is very good .
  • Note that any results with significant (10s of percent) deviations are EBs which had flagged SPWs that I have not yet dealt with properly.
  • Between tunings there are often large changes in phase calibrator flux (factors of two are not unusual). Hard to say if this is real phase calibrator flux changes or poor absolute amplitude calibration.
  • The fluxes-edit.ipac file also contains the catalog flux (last column), which I have not used yet.

-- JeffMangum - 2019-08-06

YongxiongWang and GaryFuller have also been looking at the calibrator fluxes.

1 1.png diff1 1.png pd1-1.png

-- GaryFuller - 2019-10-20

Usage Information

Adam Ginsburg in our Software team kindly supplied the following information about how to extract weblog information.

Retrieve Weblogs: and its companion and
  • Extracts and posts all weblogs.
  • Notes for installing and
    • You need to first install astroquery: pip install --upgrade --pre astroquery
      • NOTE: You will need at least version 0.3.10.dev0 or later.
    • Edit the script for your specific ALMA project.
    • The script will automatically move and untar your weblogs into the location where you want to host them (though you will need an additional untar step...see below).
    • python
      • When prompted for your username, enter your ALMA username. If you are using python2.7, you will need to enter your username in single-quotes (i.e. 'jmangum')
    • Run the script as follows:
      • $ cd /path/to/code/dir
      • $ ls
      • . ..
      • $ ipython
      • In [1]: import sys; sys.path.append('/path/to/code/dir')
      • In [2]: import parse_weblog
      • In [3]: run
      • [In a separate terminal] cd /path/to/weblog_tarballs/directory
      • ls *tgz | xargs -I {} tar -xzvf {}
      • mkdir humanreadable

Additional tools:

that produce tables like this (I did it for ALMA-IMF):

which should be straightforward to produce for the ALCHEMI data with a slight modification of this:

Mapping from UID-SB Using astroquery:
  • From astroquery.alma import Alma
  • tbl = Alma.query(payload={'project_code': '2017.1.00161.L'}, public=False)['Member ous id','SB name']
  • mapping = {row['Member ous id']: row['SB name'] for row in tbl}
then, given a uid (which can be extracted from the weblogs; I'll do this), you get the sb name from mapping[uid]

(Oct. 31, 2018) Thomas Greve, Yongxiong Wang, and Gary Fuller are going to work on this task.

-- KazushiSakamoto - 2018-10-04
Topic attachments
I Attachment Action Size Date Who Comment
1_1.pngpng 1_1.png manage 117 K 2019-10-20 - 15:45 GaryFuller Example plot showing linear fit to the flux in the 4 SPW for a particular source.
cal-data.tartar cal-data.tar manage 76 K 2019-10-20 - 15:59 GaryFuller Tar file of ASCII files of the calibrator fluxes extracted from the weblogs.
diff-plots.tartar diff-plots.tar manage 2 MB 2019-10-20 - 15:43 GaryFuller tar file of plots of difference between linear fit to the flux in the 4 SPW.
diff1-1.pngpng diff1-1.png manage 88 K 2019-10-20 - 15:41 GaryFuller Example plot showing different between linear fit to the flux in the 4 SPW.
fits-plots.tartar fits-plots.tar manage 2 MB 2019-10-20 - 15:47 GaryFuller Tar file of plots of linear fit to the flux in the 4 SPW.
pd-plots.tartar pd-plots.tar manage 2 MB 2019-10-20 - 15:51 GaryFuller Tar file of plots of fractional difference between fit and the flux in the 4 SPWs for all calibrators..
pd1-1.pngpng pd1-1.png manage 98 K 2019-10-20 - 15:49 GaryFuller Example plot showing fractional different between linear fit to the flux in the 4 SPW for one source.
Topic revision: r12 - 2019-10-20, GaryFuller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback