Cal Library Quick Reference George Moellenbrock 2015 May 18 (v2) Introduction ------------ The "Cal Library" is a new means of expressing calibration application instructions. It has nominally been available in applycal and the calibration solve tasks since v4.1, via the docallib=True parameter, as an alternative to the traditional gaintable/gainfield/interp parameters that most users continue to use. As of v4.4, we have deployed use of the Cal Library for on-the-fly calibration in plotms. In v4.4, our intent is to demonstrate the Cal Library and begin familiarizing users with it. The capabilities remain limited in some ways, and new features, additional flexibility, and broader deployment in more tasks will be offered in v4.5 and later releases. This brief document describes basic use of the Cal Library mechanism. Please note the section on current (v4.4) limitations. Basic Cal Library Usage ----------------------- The Cal Library is a means of specifying calibration instructions in an ascii file, rather than via the traditional gaintable/gainfield/interp/spwmap/calwt parameters that often become clumsy when many caltables are involved, and which have rather limited flexibility. Instead of specifying the traditional parameters, the file name is specified in the callib parameter in applycal or plotms (in applycal one must also specifiy docallib=T). In a Cal Library file, each row expresses the calibration apply instructions for a particular caltable and (optionally) a specific selection of data in the MS to which it is to be applied. For example: caltable='cal.G' tinterp='linear' calwt=True arranges a caltable called cal.G to be applied (with no detailed selection) to all MS data with linear interpolation in time, and with the weights also calibrated. It corresponds to these settings for the traditional parameters: gaintable='cal.G',gainfield='',interp='linear', spwmap=[],calwt=True If a bandpass table, cal.B, is also available for application, one might use the following Cal Library: caltable='cal.G' tinterp='linear' calwt=True caltable='cal.B' finterp='linear' calwt=False This example arranges the same instructions for cal.G, and adds a bandpass table that will be interpolated linearly in frequency (the default for time-dependent interpolation is linear, if the bandpass table contains more than one time sample), without weight calibration. The corresponding form with the traditional parameters is: gaintable=['cal.G','cal.B'], gainfield=['',''], interp=['linear','linear,linear'],spwmap=[],calwt=[True,False] In general, the Cal LIbrary file should be easier to read and manage than the traditional parameters as the number of specified caltables grows. A more complicated example, involving non-trivial spwmap as well as field selection (fldmap) in the caltable: caltable='cal.G' tinterp='linear' fldmap='nearest' spwmap=[0,1,1,3] calwt=True caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False In this case, solutions from cal.G will be selected based on directional proximity ('nearest') for each MS field via the fldmap parameter, and spw 2 will be calibrated by spw 1 solutions. For cal.B, solutions from field id 3 will be used exclusively, with spw 0 calibrating all MS spws (of which there are apparently 4). The corresponding settings for the traditional parameters is as follows: gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['linear','linear,linear'],spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False] Comment lines may be included in the cal library file by starting a line with the '#' character. (Partial line comments are _not_ supported, as yet.) Existing cal library lines can be turned off (for experimentation purposes by making those lines comments with '#'. More Advanced Cal Library Usage ------------------------------- The real power of the Cal Library arises from the ability to specify calibration instructions for a caltable per MS selection. This enables consolidating what would be multiple applycal runs using the traditional parameters into a single run. Extending the example from above, if the MS field 'cal' should be calibrated by cal.G with 'nearest' interpolation in time, and the field 'sci' with 'linear', the following Cal LIbrary will achieve this: caltable='cal.G' field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0,1,1,3] calwt=True caltable='cal.G' field='sci' tinterp='linear' fldmap='nearest' spwmap=[0,1,1,3] calwt=True caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False Note that the algorithm (fldmap='nearest') for selection solutions from the caltable hasn't been changed, but it could be. In fact, any of the calibration parameters can be adjusted, except calwt, which if set to T for any MS selection, will be T for all (to maintain weight consistency within the MS). MS selection by spw, intent, and obs id can also be used (see the glossary below). The pair of applycal executions corresponding to this Cal Library would be: applycal(vis='my.ms',field='cal',gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['nearest','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False]) applycal(vis='my.ms',field='sci',gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['linear','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False]) General Rules (current, as of 2015 May 28) ------------- - Each non-comment line in the Cal Library file must contain a valid (existing) caltable name. - Blank lines (whitespace only) will be ignored. - All parameters (see glossary below) are name/value pairs using an equals sign (=), delimited with spaces (no commas!) - Only those parameters (see glossary) for which non-default values are required need be specified - Each set of coordinated instructions must occur on a single line (there is no line continuation operator, as yet) - If detailed MS selection is used, care must be exercised to ensure it is mutually exclusive over all rows for the same caltable; there is currently no internal checking for redundancy, and only the last calibration instructions for a particular MS selection will be invoked - Full-line comments are supported by inserting the '#' character as the first non-whitespace character in the line. This mechanism can be used to turn off ordinary cal library lines. Limitations ----------- - Application of parallactic angle corrections is not yet supported within the Cal Library file (this only affects use in plotms, where there is no parang parameter) - Parametrized tables (BPOLY, GSPLINE, antpos, opacity, gaincurve) are not yet supported Conversion from Existing applycal Scripts ----------------------------------------- To convert exiting applycal commands, a simple experimental function, applycaltocallib is available. To access it, type: from callibrary import applycaltocallib Then, chose a filename for the cal library file, and supply existing settings for applycal parameters (field, spw, intent, gaintable, gainfield, interp, spwmap, calwt) to the applycaltocallib function: callibfile='mycallib.txt' applycaltocallib(filename=callibfile,append=F, field,spw,intent,gaintable,gainfield, interp,spwmap,calwt) If append=F, callibfile will be overwritten, if it already exists. If append=T, new entries will be added to an existing callibfile. Only parameters with non-trivial applycal settings need be included. In general, if gaintable is a list, it is best if gainfield, interp, spwmap, and calwt (where non-trivially set) are also lists. For example, if your conventional script contains the following applycal executions (duplicated from above): applycal(vis='my.ms',field='cal', gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['nearest','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]], calwt=[True,False]) applycal(vis='my.ms',field='sci', gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['linear','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]], calwt=[True,False]) ...these can be edited to applycaltocallib executions as: callibfile='mycallib.txt' applycaltocallib(filename='mycallib.txt',append=F, field='cal', gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['nearest','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]], calwt=[True,False]) applycaltocallib(filename='mycallib.txt',append=T, field='sci', gaintable=['cal.G','cal.B'], gainfield=['nearest','3'], interp=['linear','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]], calwt=[True,False]) After running them, mycallib.txt will contain: caltable='cal.B' calwt=False field='cal' tinterp='linear' finterp='linear' fldmap='3' spwmap=[0, 0, 0, 0] caltable='cal.G' calwt=True field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0, 1, 1, 3] caltable='cal.B' calwt=False field='sci' tinterp='linear' finterp='linear' fldmap='3' spwmap=[0, 0, 0, 0] caltable='cal.G' calwt=True field='sci' tinterp='linear' fldmap='nearest' spwmap=[0, 1, 1, 3] Note that 'cal.B' table is specified separately for the 'cal' and 'sci' fields with otherwise the same parameters; thus, those two lines could be manually consolidated to a single line with unified field selection (or none at all), yielding: caltable='cal.B' calwt=False field='cal,sci' tinterp='linear' finterp='linear' fldmap='3' spwmap=[0, 0, 0, 0] caltable='cal.G' calwt=True field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0, 1, 1, 3] caltable='cal.G' calwt=True field='sci' tinterp='linear' fldmap='nearest' spwmap=[0, 1, 1, 3] This function should be considered experimental and used with care, and the resulting file examined thoroughly for correctness, since this fucntion will not do any internal duplication checking or other sanity checks. All other current constraints and limitations on cal libraries (as noted above) will apply. Glossary -------- This is a list of recognized Cal Library parameters. For each, the default is indicated. Additional parameters enhancing flexibility will be added in v4.5 and later. caltable - the name of the caltable for which the instructions on the current line apply; no default; required MS selection: Use these parameters to implement calibration instructions specific to particular MS selections (using standard MS Selection syntax, except where noted. field - the MS field selection for which the calibration instructions on the current line apply; default='' (all fields) spw - the MS spw selection for which the calibration instructions on the current line apply; default='' (all spws) Note that channel selection will be ignored. intent - the MS intent selection for which the calibration instructions on the current line apply; default='' (all intents) obs - the MS observation id selection for which the calibration instructions on the current line apply; default='' (all intents) Interpolation/application: tinterp - the time-dependent interpolation mode; default='linear' options: 'linear','nearest' finterp - the chan-dependent interpolation mode (only relevant for channelized caltables); default='linear' options: 'nearest','linear','cubic','spline' calwt - weight calibration; default=True options: True, False Calibration Mapping: The *map parameters enable selection on the caltable. For each *map parameter, the basic specification is an ordered list indicating the caltable selection indices intended for each MS index on that axis. E.g., spwmap=[0,1,1,3] means MS spws 0,1,3 will each be be calibrated by the same spw index from the caltable, and MS spw 2 will be calibrated by cal spw 1. The map parameters support other short-hand options as well, as indicated below. For defaults, "index identity" means that each MS index will be calibrated by the corresponding caltable index, and 'no explicit mapping' means that no filter will be applied to that axis, and all available solutions on the axis will be included. spwmap - spectral window mapping; default=index identity fldmap - field mapping; default=[] (no explicit mapping); additional options: 'nearest' or a string indicating selection on the caltable (same as traditional gainfield options) antmap - antenna id mapping; default=index identity obsmap - obs id mapping; default=[] (no explicit mapping) Document History ---------------- v1 - initial version (2015May11) v2 - Added comment support, other minor changes (2015May18) v3 - Added applycaltocallib section, other minor fixes (2015May28)