Casapy Flag tool - Functionality and User Interface

Urvashi R.V. (2007-01-16)

This is a short description of the function names and parameter lists to be used in the casapy flag tool. Some specific questions that need to be answered are listed at the end of each section. Suggestions about additional functionality and function/parameter names are most welcome.

This document will be updated based on feedback obtained. The current version includes user feedback from J.McMullin L.Davis, D.Whysong, J.Lightfoot, N.Kanekar, W.Brisken.

Please send feedback to jmcmulli@nrao.edu, rurvashi@nrao.edu.

Data selection

The following function will be used to select a subset of the MS, on which to perform subsequent flagging operations.

>> fg.setdata(antennaName=[],antennaIndex=[],correlation=[],
              fieldName=[],fieldIndex=[],spwIndex=[],
              uvRange=[min,max],timeRange=[min,max],
              chanRange=[nchan,start,width], chanList=[], baselineList=[])

This function will return a record, with some information about the subset of the MS that has been selected. For example, "Number of rows selected".

Qn : What additional parameters would one like to perform a data selection on ? [NK] : baselines
Qn : What additional diagnostic information should be returned ?

Manual flagging

The following function will select specific data, for manual flagging.

>> fg.setmanualflags(antennaName=[],antennaIndex=[],correlation=[],
                     fieldName=[],fieldIndex=[],spwIndex=[],
                     baseline=[],uvRange=[min,max],autocorrelation=T/F,
                     chanList=[],freqRange=[min,max],
                     timeRange=[min,max],timeSlot=[],dtime= ,
                     clipExpr='', clipRange=[min,max], 
                     quackScaninterval= ,quackLength= ,
                     unflag=T/F, rowFlag=T/F)

The flags will get applied to the data, after a call to the fg.run() function.
Note that fg.setmanualflags() can be called multiple times before a call to fg.run(). This will apply flags to all selections during a single pass through the data.

Example : To flag chans 4,5 for antenna 2, and chans 16,17 for antenna 5.
          All corresponding baselines, correlations and times will be flagged.
>> fg.setmanualflags(antennaIndex=[2],chanList=[4,5]);
>> fg.setmanualflags(antennaIndex=[5],chanList=[16,17]);
>> fg.run();

fg.printmanualflagselection() prints out the current list of manual flag selections.
fg.clearmanualflagselection() clears any previous manual flag selections. Note that this only clears flag selection settings, and does not undo any flags that have already been applied to the data.
To undo flags, one must use fg.setmanualflags(...,unflag=True),

There can additionally be separate functions for each of the parameters listed in fg.setmanualflags().

fg.flagantennaName()       fg.flagbaseline()        fg.flagtimeslot(dtime= )
fg.flagantennaIndex()      fg.flaguvRange()         fg.flagtimerange()
fg.flagcorrelation()       fg.flagautocorrelation() fg.clip(expr,min,max)
fg.flagfieldName()         fg.flagchanList()        fg.quack(scaninterval,quacklength)
fg.flagfieldIndex()        fg.flagfreqRange()       fg.flagspwIndex()
fg.flagRow(T/F)

To allow the ability to specify multiple combinations of data to be flagged, an additional function fg.newmanualflagselection() will be provided to distinguish between different combinations of the data.

Example : To flag chans 4,5 for antenna 2, and chans 16,17 for antenna 5.
          All corresponding baselines, correlations and times will be flagged.
>> fg.flagantennaIndex([2]);
>> fg.flagchanList([4,5]);
>> fg.newmanualflagselection();
>> fg.flagantennaIndex([5]);
>> fg.flagchanList([16,17]);
>> fg.run();

Note the difference between the fg.clearmanualflagselection() and fg.newmanualflagselection().

Qn : What other parameters would one like use, to set flags ?

[DW,NK] : Read in manual-flagging parameters from a text file.. This could be done (at the task level ?) via python scripts that can read the text file, and call fg.setmanualflags(). The formats can be compatible with the output of fg.printmanualflagselection() and the history file.
[NK] : Optionally flag the entire timestamp if any data is flagged. Done via a rowFlag=T/F parameter.

Algorithms

The algorithms currently implemented are identical to those in the old autoflag tool. They include sliding-window median filters in time and frequency (time-median, freq-median), and the spectral-rejection, and uv-binning algorithms.
Current algorithm documentation : casa autoflag tool casapy autoflag tool

The following function gathers input parameters for different autoflag algorithms. Each algorithm can have a different parameter list, and this is to be supplied as a record of parameters.

>> fg.setautoflag(algorithm='timemedian',parameters=[column='',expr='',threshold='',...])

Multiple algorithms can be set-up before a call to fg.run(trial=T/F). This creates a list of active algorithms, which are applied in succession to each chunk of data as the program iterates through the data.

If an algorithm is run in trial mode, then flags are not written to disk. The output of the autoflag algorithm can be monitored, and flags later committed to disk via an optional fg.writeflagstodisk() function. One could interactively use this function as follows.

>> inputpars = fg.setautoflag(algorithm='timemedian');
>> print inputpars   
        column = 'data'
        expr = 'abs I'
        thr = 0.5
        ...
>> inputpars.column = 'correcteddata'
>> print inputpars
        column = 'correcteddata'
        expr = 'abs I'
        thr = 0.5
        ...
>> fg.setautoflag(algorithm='timemedian', parameters = inputpars);
>> fg.run();

The data expression used to compute the data on which the algorithm is to be applied, is specified by the following parameters.
column : data/correcteddata/residualdata/weights/invweights
expr : abs/arg/real/imag RR/LL/RL/LR/XX/YY/XY/YX/I/Q/U/V

fg.printautoflagparams() prints out the names of currently active autoflag algorithms and their parameters.
fg.clearautoflagparams() clears any autoflag parameter settings. It can take in an algorithm name as a parameter, to remove only specific algorithms from the currently active list.

Qn : What other known autoflag algorithms should be implemented ?
(ALMA pipeline heuristics, AIPS flagging capability.) : details will be added later, but specific suggestions are welcome.

[JL,LD] : Alma pipeline heuristics :

Algorithms that work on 2D views of the data, and flag based on some 2D statistics.
These 2D views are written to the disk as an image that can be loaded in the viewer.
Autoflag algorithms must return a flag record in some fixed format.

Such algorithms can be implemented and added to the above list.

Flag summaries/statistics

Flag Statistics

The following function will display the percentage of data flagged vs AntennaIndex, Baseline, SpwIndex and Channel, Correlations, TimeRange, FieldIndex, UVRange
An option can be given to take in a list of antennas, channels etc as parameters to restrict the output to only those data values. By default it uses the full range. These results can either be listed out on the logger, or when applicable, sent to a plotter for graphical output. They can also be returned in a suitable data record for automated parsing.

>> fg.showflagsummary(what='antenna')
>> fg.showflagsummary(what='channel',list=[2,3,4,5])
...etc...

[WB] : Flags summary ( % of flagged data ) for any data combination set through a function similar to the setmanualflags().

fg.showflagsummary(antennaName=[],antennaIndex=[],correlation=[],
		   fieldName=[],fieldIndex=[],spwIndex=[],
                  baseline=[],uvRange=[min,max],
                   chanList=[],freqRange=[min,max],
                   timeRange=[min,max])

Qn : What additional statistics would be useful ?

Queries

The following function will return lists of AntennaIndex, Baselines, SpwIndex and Channels, Correlations,FieldIndex with more (or less) than a specified percentage of flagged data (percentage threshold (thr) : [0-1]).

>> fg.queryflags(what = 'antenna',thr = 0.5, [morethan=True])
>> fg.queryflags(what = 'channel',thr = 0.8)
...etc...

Note that these functions can be called to monitor flags before calling any flag/autoflag functions.

Qn : What other kinds of queries would be useful ?
[NK] : Add in queries that list time-ranges and uv-ranges with more than some percentage of data flagged.

Extending flags

Extend existing flags along a specified axis. For example, 'for a given timestamp and baseline, if any channel is flagged, the entire channel.'. The allowed axes along which flags can be extended will be Antenna, Baseline, Channel, Correlation,Time.

>> fg.extendflags(along='channel',width=3) -> Flag 3 channels on either side of a
                                              flagged channel
>> fg.extendflags(along='antenna1')        -> Flag all corresponding visibilities 
                                              for antenna 1 of the flagged baseline.
>> fg.extendflags(along='bothantennas')    -> Flag all corresponding visibilities 
                                              for both antennas of the flagged baseline.

The output from query functions can be sent into the manual flagging functions, to extend flags to other dimensions.
For example, to completely flag all channels that have more than 70% of the data already flagged, do the following.

>> fg.clearmanualflagselection();    -> Cleans up any old manual flag selections 
>> chans = fg.queryflags(what='channel', thr= 0.7) 
                                     -> returns a list of channel numbers
                                         corresponding to channels with more
                                         than 70% of the data being flagged.
>> fg.setmanualflags(chanList=chans) -> Select these channels for manual flagging
>> fg.run();                         -> Apply the flags.

Qn : Will this setup allow for all kinds of flag extensions that one usually requires ? If not, please give examples of what should be possible.

[DW,WB,NK] : A width parameter can control how far the flags should be extended on either side (N) of a flagged visibility point.
[NK] Allow flags to be extended for a uv-range around a flagged point, and in hour-angle (translated internally to time). Also use rowflags.

Other functions

A flag command history can be compiled via a combination of fg.printmanualflagselection() and fg.printautoflagparameters(), with a parameter to append to a history file, instead of printing to the screen/logger.
[DW,JL,LD,NK] : Write flags to an external text file in some simple text format. This should be optionally returned as a python record (for a pipeline).
Functions to reset all flags.
It is currently impossible to distinguish between different kinds of flags, for example, those generated by the online system, vs those applied by the user.
[DW] : Calibration-based flagging.
[DW] : Flag statistics. 2D plot of histograms of the data (with flags applied).
X-axis : time, binned in coarse increments.
Y-axis : the histogram bins (steps in amplitude).
The plot should be a gray-scale (stack up 1-D histograms), or a set of lines for (mean, 1-sigma, 2sigma, etc..)
[NK] : Function to flag antennas for timeranges for which the antenna is shadowed by another.
[NK] : Flag time-ranges based on an elevation limit for a specified source.

Qn : Any other functions ?
Qn : Any comments and/or specific suggestions regarding function names ?
[NK] : fg.setmanflags(). Parameter names should be single-case

About this document ...

Casapy Flag tool - Functionality and User Interface

This document was generated using the LaTeX2HTML translator Version 2002-2-1 (1.70)

The command line arguments were:
latex2html flag_casapy.tex

The translation was initiated by R. V. Urvashi on 2007-01-17

R. V. Urvashi 2007-01-17