Low SNR, Narrow-bandwidth Data Reduction

Some data have been collected with narrow/low bandwidth (FDM) spectral windows for all of the scans (science source, phase calibrator, etc.), and the SNR on the phase calibrator can be too low for a reliable calibration. This wiki page documents the procedures to be adopted in reducing the data in this situation. Note that this is distinct from bandwidth switching observations, where the problem is solved in the data collection stage by collecting broader-band data on the calibrators; the data reduction tweaks for that case are documented at NarrowBandWidthSwitchingReduction

Here is a google spreadsheet summarizing narrow bandwidth datasets -- ask Remy if you need permission to view it.

Notes:
  • A good sign that you need to do this is if the flux densities in the fluxscale or fluxscale2 files are very different for the narrow windows than they are for the wide ones, beyond what can be accounted for by the difference in center frequencies given expected source spectra. Other signs are: anomalously high noise in a narrow SPW fluxscale measurements; a point source test image that looks unusally poor compared to the PSF (particularly useful when it's the delaycal); phase int and/or phase inf and/or ap-pre-bp phase solutions look considerably more noisy for narrow SPWs. as a guideline, if you have 58 MHz windows, you should go ahead and pre-emptively remap the phases
  • the phase-int solution is the one that tends to be most affected by SNR, which has a cascading effect through the other solutions.
  • I don't think we should be doing the fluxinf mapping, ie, spwmap_fluxinf should probably be the trivial map. The first thing to try is always just SPW-Mapping the phase solutions only (BSM 10june2015)
  • check the APPLYCAL portion ends up such that: a) BP and flux cal get phase INT; b) science target, phasecal, and delaycal (if any) get phase INF, referenced to the phasecal field. This will ensure that loss of coherence from sub-scan-scale phase fluctuations can be assessed as described at SelfCalAndCoherence

NEW: a more recent manual reduction script is attached, with changes needed indicated by my initials (BSM, Nov 2015)

Example Reduction

Scott Schnee has two scripts that work and illustrate the new procedures in /lustre/naasc/sschnee/Functional/LowSN

The scripts are 1069_spwmap.ms.scriptForCalibration.py and 229_spwmap.ms.scriptForCalibration.py. The equivalent pre-these-changes scripts (i.e., what was generated by au.GenerateReducScript()) are the same but with "independent" instead of "spwmap" in the name.

Changes Needed for Reduction

To improve the bandpass solutions on the narrow spws, one can try smoothing the solutions, using the smoothing interval proposed by au.bandpassPreAverag. For example, one can use solint='inf,62.5MHz' in the bandpass calls.

A brief summary of the changes needed is as follows:
  1. Define some spwmaps at the start of step 15, to be selectively used in the data reduction. In these spwmaps, the problematic narrow SPWs shold be replaced with wide-band SPWs which are as close in frequency as possible. Currently it is only recommended to remap the PHASE solutions, not any amplitude solutions.
  2. Also at the beginning of step 15, we will generate a new "phase-up" cal table. This references all SPWs to zero phase using the bandpass calibrator. The main reason for this is that it corrects any constant phase offset that may exist between SPWs. A side benefit is that all the phase solutions should be centered on zero phase, making visual inspection and quality control easier (180-wraps should be uncommon). Note that the option PhaseDiff in generateReducScript will create this additional step in the script, but you can also add it manually to the default script.
    1. All subsequent gaincal commands should make use of the phase_up table via the gaintable argument.
    2. All subsequent gaincal commands should also make use of the spwmaps that have been defined.
  3. The applycal() statements should also make use of the phase_up gain table and the spwmaps.

The statements affected are shown for illustration below, and the full modified script attached.

# Gain calibration
mystep = 15
if(mystep in thesteps):
  casalog.post('Step '+str(mystep)+' '+step_title[mystep],'INFO')
  print 'Step ', mystep, step_title[mystep]

  spwmap_phaseup  = [0,1,2,3]
  spwmap_bandpass = [0,1,2,3]
  spwmap_phaseint = [0,1,3,3]
  spwmap_phaseinf = [0,1,3,3]
  #spwmap_fluxinf  = [0,1,3,3]
  # this is now recommended to be a trivial map BSM 10june2015 -->
  spwmap_fluxinf = [0,1,2,3]

  os.system('rm -rf 229_spwmap.ms.split.phase_up')
  gaincal(vis = '229_spwmap.ms.split',
    caltable = '229_spwmap.ms.split.phase_up',
    field = '0', # J0522-3627
    solint = 'inf',
    refant = 'DV22',
    gaintype = 'G',
    calmode = 'p',
    gaintable = '229_spwmap.ms.split.bandpass')

  os.system('rm -rf 229_spwmap.ms.split.phase_int')
  gaincal(vis = '229_spwmap.ms.split',
    caltable = '229_spwmap.ms.split.phase_int',
    field = '0~2', # J0006-0623,J2232+117,J2323-0317
    solint = 'int',
    refant = 'DV22',
    gaintype = 'G',
    calmode = 'p',
    gaintable = ['229_spwmap.ms.split.bandpass', '229_spwmap.ms.split.phase_up'])

  os.system('rm -rf 229_spwmap.ms.split.ampli_inf')
  gaincal(vis = '229_spwmap.ms.split',
    caltable = '229_spwmap.ms.split.ampli_inf',
    field = '0~2', # J0006-0623,J2232+117,J2323-0317
    solint = 'inf',
    refant = 'DV22',
    gaintype = 'T',
    calmode = 'ap',
    spwmap = [spwmap_bandpass,spwmap_phaseup,spwmap_phaseint],
    gaintable = ['229_spwmap.ms.split.bandpass', '229_spwmap.ms.split.phase_up', '229_spwmap.ms.split.phase_int'])

...


# Application of the bandpass and gain cal tables
mystep = 17
if(mystep in thesteps):
  casalog.post('Step '+str(mystep)+' '+step_title[mystep],'INFO')
  print 'Step ', mystep, step_title[mystep]

  for i in ['0', '1']: # J0006-0623,J2232+117
    applycal(vis = '229_spwmap.ms.split',
      field = i,
      gaintable = ['229_spwmap.ms.split.bandpass', '229_spwmap.ms.split.phase_up', '229_spwmap.ms.split.phase_int', '229_spwmap.ms.split.flux_inf'],
      gainfield = ['', '', i, i],
      spwmap = [spwmap_bandpass, spwmap_phaseup, spwmap_phaseint, spwmap_fluxinf],
      interp = 'linear,linear',
      calwt = F,
      flagbackup = F)

  applycal(vis = '229_spwmap.ms.split',
    field = '2,3', # BP_Psc
    gaintable = ['229_spwmap.ms.split.bandpass', '229_spwmap.ms.split.phase_up', '229_spwmap.ms.split.phase_inf', '229_spwmap.ms.split.flux_inf'],
    gainfield = ['', '', '2', '2'], # J0541-0541
    spwmap = [spwmap_bandpass, spwmap_phaseup, spwmap_phaseinf, spwmap_fluxinf],
    interp = 'linear,linear',
    calwt = F,
    flagbackup = F)

-- BrianMason - 2014-06-25
Topic revision: r8 - 2015-11-09, BrianMason
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback