Band Width Switching Reduction

Science observations that use very narrow-band ("FDM") observations may be collected in an observing mode where some of the calibrators (phase, flux) are instead observed in broader-bandwidth correlator modes (TDM or sometimes FDM low resolution) to give adequate signal to noise in the calibration solutions. This observing mode is called 'bandwidth switching'. Note that this is distinct from the case where there are narrow-band SPWs collected on both the calibrators (including the phase calibrator) and the science targets, such that (sometimes) the SNR on the phase calibrator is insufficient and a different work-around is needed (spw-mapping); that workaround is documented at LowSnrNarrowBandReduction.

More general information about bandwidth switching observations & data reduction is in the slides for a technical talk that Brian M gave to NAASC data reducers in September 2015.

Is my dataset a bandwidth switching project

An example of a bandwidth-switching observation is shown here (from the listobs of the split ms):

  Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds 
  27-Dec-2013/02:24:36.0 - 02:25:38.1     3      0 J0423-0120               42120  [0, 1, 2, 3] 
              02:27:17.4 - 02:32:33.6     6      0 J0423-0120               70200  [4, 5, 6, 7] 
              02:35:36.3 - 02:38:13.6     9      1 J0522-3627              105300  [0, 1, 2, 3] 
              02:41:15.8 - 02:41:46.1    12      2 J0304+3348               21060  [0, 1, 2, 3]  
              02:42:40.8 - 02:45:50.1    13      3 L1448-IRS2E               5616  [4, 5, 6, 7]  
              02:42:40.8 - 02:45:50.1    13      4 L1448-IRS2E               5616  [4, 5, 6, 7] 
    .
    .
    .   
              02:49:02.5 - 02:52:11.8    15     23 Per-Bolo45                5616  [4, 5, 6, 7] 
              02:52:21.2 - 02:52:51.5    16      2 J0304+3348               21060  [0, 1, 2, 3] 
              02:53:16.9 - 02:56:26.2    17     24 Per-Bolo58                5616  [4, 5, 6, 7]  
    ...
You can see that calibrators are observed in different sets of spws (gain: 0,1,2,3; bandpass: 0,1,2,3,4,5,6,7) than the science source (spw: 4,5,6,7) The spectral windows are:
Spectral Windows:  (8 unique spectral windows and 1 unique polarization setups)
  SpwID  Name                           #Chans   Frame   Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) BBC Num  Corrs
  0      ALMA_RB_03#BB_1#SW-01#FULL_RES    128   TOPO   87568.828    -15625.000   2000000.0       1  XX  YY
  1      ALMA_RB_03#BB_2#SW-01#FULL_RES    128   TOPO   89506.328    -15625.000   2000000.0       2  XX  YY
  2      ALMA_RB_03#BB_3#SW-01#FULL_RES    128   TOPO   97584.453     15625.000   2000000.0       3  XX  YY
  3      ALMA_RB_03#BB_4#SW-01#FULL_RES    128   TOPO   99584.453     15625.000   2000000.0       4  XX  YY
  4      ALMA_RB_03#BB_1#SW-01#FULL_RES   3840   TOPO   89209.225       -15.259     58593.8       1  XX  YY
  5      ALMA_RB_03#BB_2#SW-01#FULL_RES   3840   TOPO   86775.178       -15.259     58593.8       2  XX  YY
  6      ALMA_RB_03#BB_3#SW-01#FULL_RES   3840   TOPO   88652.310       -15.259     58593.8       3  XX  YY
  7      ALMA_RB_03#BB_4#SW-01#FULL_RES   3840   TOPO   85947.184       -15.259     58593.8       4  XX  YY

Here the FDM ('science') windows are 4,5,6,7, and the TDM (gain calibration) windows are 0,1,2,3

Modifications to initial reduction script

generateReducScript() will recognize this as a bandwidth switching observation and attempt to produce a script that deals with the data, but a few modifications may be necessary. Note that a new option in generateReducScript (October 2015) , PhaseDiff, will produce an extra step in the script calculating the phase offsets table, The modifications enumerated here pertain to the manual calibration script that is generated by the analysis Utils distributed with CASA 4.4.0 A few general comments on the approach used are in order:
  1. The "bandpass" step (13) has some extra steps. Most of the steps are because the script now "bootstraps" the bandpass calibrators flux density in order to calibrate the narrow windows. After this process the "bandpass2" table will contain both the bandpass and absolute flux information.
  2. Most of the gain solutions are obtained with combine='spw', resulting in a single SPW in the output table. Therefore the SPWMAP used in applycal will usually map to SPW 0. You don't need to associate narrow windows with wide ones by hand.

Modifications that may be needed:
  1. (on the unsplit MS) Determine which are the gain calibration spws (TDM) and the science spws (FDM). Determine which TDM broad spectral window contains each FDM narrow spectral window. You can do this by looking at the listobs output.
  2. (on the unsplit MS) Determine what tsys SPWs (observed during the scans labeled with CALIBRATE_ATMOSPHERE intents) correspond to the science observations (FDM), and which tsys SPws ones correspond to the gain/flux calibrator observations (TDM)
  3. In the tsys application step, print the generated tsysmap parameter, which should map each spw to its corresponding tsys spw. If the generated tsysmap contains errors, make a copy of tsysmap and tweak it so that tsysmap2[science window]=(corresponding CALIBRATE_ATMOSPHERE) maps each spw to its corresponding tsys spw. Use this new tweaked tsysmap in the applycals() when applying the tsys and wvr solutions.
    1. A common error in the tsysmap is that an FDM window is associated with a TSYS SPW that is not "paired" in the observing sequence with the FDM observations themselves. Ideally you want the FDM window associated with a TSYS SPW that is observed in the CALIBRATE_ATMOSPHERE scan immediately preceding the science (FDM) scan(s). For a given science SPW, choose the TSYS SPW that has a center frequency closest to the center frequency of the SPW in question (also in the same baseband if possible).
  4. Since the dataset contains narrow science spws, smoothing may be necessary of those narrow sps to derive a good bandpass solution. To do that, in the bandpass calibration step, restrict the bandpass() call to the broad spws, and add a second bandpass call for the narrow spws. Add "append=T" in both calls. You can then change solint = 'inf' to something like solint = 'inf,50ch' to smooth the bandpass solution on the narrower spws. If several bandpass solutions are calculated in the default script, repeat this sequence for each bandpass call.
    1. if there is a lot of sample-to-sample scatter in the ap_pre_bandpass solutions you may want to split that gaincal call also and use a larger time averaging for the narrow windows.
  5. in the gain calibration step, the first gaincal call should produce the phase offsets table (phase_diff). Check that the bandpass calibrator is the source being used to derive the phase_diff table. If the bandpass calibrator is also the phase calibrator, also restrict by scan (to the bandpass scans). Note that the option PhaseDiff in generateReducScript will create this additional step in the script, but you can also add it manually to the default script (see example below)
  6. In the gain calibration step, with the exception of the gaincal call producing the phase_diff table, all gaincal() calls should be edited so as to be restricted to the TDM spws. Check that those gaincal calls use combine ='spw'. Note that this will result in the gain tables only having one 'combined' spw, labeled 0.
  7. In the gain calibration step, some gaincal calls use calibration tables on the fly (gaintable parameter). If one or several of the tables listed in gaintable only have one combined spw (which is the case for the phase_int, phase_inf tables for example, or any table created with combine = 'spw'), the spwmap parameter must be added to the gaincal call. This allows for the combined solution to be applied to all spws, and not just to spw 0. For example, for gaintable = ['uid.ms.split.bandpass', 'uid.ms.split.phasediff_inf', 'uid.ms.split.phase_int'], spwmap should be spwmap = [[],[],[0,0,0,0]], or spwmap = calspwmap[0].
  8. In the applycal step -- assuming the new flux bootstrapping of the BP cal has been used -- ensure the following is the case:
    1. the final applycal to the BP cal should include phase_inf (not phase_int); ampli_inf; and (optionally) flux_inf (which should not matter. You may need to add the ampli_inf table -- in particular, if there are multiple observations of that source.
    2. applycals with fieldid's of sources that have had SETJY() run do not need flux_inf (but it should not hurt)
    3. all other sources should have both flux_inf and ampli_inf applied

The modified commands for the example dataset under consideration are shown below:
    .
    .
    .
# in step 7 (tsys application)
  tsysmap2=tsysmap
  tsysmap2[33]=11
  tsysmap2[35]=9
  tsysmap2[37]=11
  tsysmap2[39]=9
  # use tsysmap2 in applycal() statements used to apply tsys and wvr solutions to the
  #  (science) observations that were taken with the narrow SPWs.
    .
    .
    .
# in step 13 (bandpass calibration)
  bandpass(vis = 'uid___A002_X775008_X3d9.ms.split',
    caltable = 'uid___A002_X775008_X3d9.ms.split.bandpass',
    field = '0', # J0423-0120
    spw = '0~3',
    solint = 'inf',
    combine = 'scan',
    refant = 'DV18',
    solnorm = T,
    bandtype = 'B',
    gaintable = 'uid___A002_X775008_X3d9.ms.split.ap_pre_bandpass')
  
  bandpass(vis = 'uid___A002_X775008_X3d9.ms.split',
    caltable = 'uid___A002_X775008_X3d9.ms.split.bandpass',
    field = '0', # J0423-0120
    spw = '4~7',
    solint = 'inf,64ch',
    combine = 'scan',
    refant = 'DV18',
    solnorm = T,
    bandtype = 'B',
    append = T,
    gaintable = 'uid___A002_X775008_X3d9.ms.split.ap_pre_bandpass')

    .
    .
    .
# in step 15 (gain calibration)

# This is an example of GAINCAL() call that gets the explicit spw="0~3" (TDM spws) restriction: 
  gaincal(vis = 'uid___A002_X775008_X3d9.ms.split',
    caltable = 'uid___A002_X775008_X3d9.ms.split.phase_int',
    field = '0~2', # J0423-0120,J0522-3627,J0304+3348
    spw = '0~3',
    solint = 'int',
    refant = 'DV18',
    gaintype = 'G',
    calmode = 'p',
    combine = 'spw',
    gaintable = 'uid___A002_X775008_X3d9.ms.split.bandpass')

WARNING: the observing procedure that has been in use for Cycle 2 bandwidth switching observations is to observe the bandpass calibrator twice, once in the wide-bandwidth setup and once in the narrow-bandwidth setup, with the intent that these two observations are used to determine the spectral window to spectral window instrumental phase offsets. Unfortunately, particularly at the shorter wavelengths, these observations can take a sufficiently long time that the atmospheric phase changes significantly (see this report); it is impossible to separate these two effects with this data. It is important to determine whether your data are badly affected by this. Several ways you can get a handle on this:
  • Look at the phase INT solutions for a bunch of antennas on the two BP calibrator scans (one scan in TDM mode, one in FDM mode -- the TDM scan is likely to be the most sensitive). If the variation in phase as a function of time, for a single polarization, is 50 degrees or more over the duration of the BP scan for most antennas, it is likely that atmospheric phase fluctuations will substantially reduce coherence on your science targets when you apply the SPW offsets determined by the procedure on this wiki page (and no better procedure is known given the way BW-switched data are currently being acquired). An example of a BAD dataset is shown in THIS PLOT, which shows the phase INT solution for a B9 BP observation of a bright QSO dominated by atmospheric phase variations.
  • Another test: self-cal a science source and see how much more flux you get. This is an upper limit to the magnitude of the flux lost due to the flawed observing mode (since flux may also be lost, eg, due to actual short-term atmospheric phase fluctuations). In the B9 test dataset linked above, a factor of 2 more flux was obtained.
  • Another test is to re-calibrate, but simply omit the SPW to SPW offsets, and do the same comparison. You will lose some coherence due to not applying the actual true SPW to SPW phase offsets; but you may gain some coherence by not including the atmospheric phase fluctuations introduced by the less-than-optimal observing method. In the B9 test dataset linked above, the flux increased by 50% when you ignore the SPW offsets.
  • Typically longer-wavelength projects will not be as badly affected as this B9 test dataset, but we currently don't know how much better, how often, and under what conditions. The dataset you are reducing will be helpful in improving our understanding of this envelope.

WARNING 2: It has been reported that, in CASA 4.4, using append = 'T' in the gain solutions may not work CAS-7816 Update 16sep2015 - this was a case where the gaincal calls were not restricted to TDM only windows; if you follow the procedure on this page, this particular limitation of gaincal() will not be a problem for you.

Further cautions , issues under investigation, and areas for improvement:
  1. with the new "BP bootstrap" scheme, ampli_inf and flux_inf are both incremental calibration tables (relative to the absolute gains in the bandpass2 table). the plots of them may be a little weird.
  2. the QA2 plots of source visibilities are messed up. The QA2 script also does not show the TDM windows on the plots, only the science windows.
  3. the results in the fluxscale2 file are bogus with the new scheme since the ampli_inf table is only a relative table and fluxscale2 does not accept the bandpass2 table. may or may not be a problem down stream, but definitely be aware of. (may be fixed by specifying reference field ID by hand?)
  4. The default scheme uses combine='spw' for the amplitude solutions. This is often needed for the phase (in particular the phase_int) solutions; but is less needed, and less important, for the amplitude solutions -- it also has the potential to adversely affect the calibration from SPW to SPW. It would be better to not combine='spw' for the amplitude solutions; this would then require a custom, non-trivial SPW map for application of the corrections.
  5. The script generator sometimes puts in a big chunk of unneeded steps in the bandpass stage (beginning "f=open(" through just before the SETJY on the BP cal). These can be commented out.

-- BrianMason - 2014-03-14 - updated 2015/09/15
Topic attachments
I Attachment Action Size Date Who Comment
uid___A002_X881d57_X410mssplitap_pre_bandpassphaseCM11spw03.pngpng uid___A002_X881d57_X410mssplitap_pre_bandpassphaseCM11spw03.png manage 133 K 2014-10-01 - 14:59 BrianMason an example plot of phase INT solutions on a B9 BP calibrator observation showing lots of atmospheric phase variation.
Topic revision: r25 - 2015-11-02, BrianMason
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback