You are here: NRAO Public Wiki>ALMA Web>AlmaSci>AlmaTiger>DataReducWorkshop>ALMAFlagPlotQuestions (2011-09-09, JaredCrossley) Edit wiki text Edit Attach Print version

Flagging questions

Q1: I would like to learn to use browsetable and flagcmd -- the flagging tables and using them interactively remain somewhat opaque to me.
  • A: Steve Myers gave a tutorial on the flagcmd task. (It was not recorded.) flagdata's selectdata sub-parameters can accept lists; when flagdata runs, it combines the elements of the lists by index and operates on each set of selections. Calling flagdata only once using lists is faster than calling flagdata separately for each set of selections. flagcmd allows you to us a list of strings rather than multiple lists; this is easier to setup. In addition, flagcmd stores flags in a command table. This table can be used as a record of flagging that has been done.

Q2: At the stage of data inspection, it isn't clear how one knows that the bright resolved source scans are from the last day's. Is there a way to bring up a legend of which scan is which colour?
  • A: plotms cannot currently display a color legend. This is a feature which should be added in the future. However, plotms development is on hold until a new developer is hired.

Q3: Some flagging is done before the Tsys correction using applycal to remove a feature in Tsys versus freq for DV04. Compared to AntennaBand7 guide, we do not flag any strange features in the Tsys plots before applycal, and do so only after we see how they manifest themselves in the calibrated data. Is one method preferred over the other?

Q4: Flag absorption features for all sources" This means you have to completely exclude certain frequencies, and for the broad ozone absorption line, this will ALWAYS be there. In other words, there will always be some frequencies for which it is best to flag, especially for Band 7. Does it make sense to make it standard practice to flag such data in a "pre-processing" phase, and to include these flags as part of the data released to the PI's?
  • A: This service may be provided in the future. It is not provided in cycle 0.

Q5: Antennae: In examining the Tsys data (amp versus time), the CASA guide mentions that the data for DV04 will later be flagged due to high system temperatures. However, in a past Tiger Team meeting, it was stated that since the weights are computed from Tsys, such noisy data would be downweighted, so it's not necessary to flag it. Is there a good "rule of thumb" for deciding whether it is necessary to flag such data, or to just let it get downweighted later on in the imaging stage?
  • A: No, there is no good rule of thumb.

Q6: Antennae: Antenna DV12 exhibits periodic spikes in Tsys vs. frequency for one polarization". Two questions here. First, is there a known cause for such behavior? Second, we do not flag the data at this point: in general, is it better to avoid flagging data based on Tsys plots, and to wait until the tables have been applied before decided which data needs to be flagged?
  • A: Yes, there is probably a known cause... Very discrepant points should be flagged.

Q7: It is often recommended, here and in other casa guides, to flag the edge channels due to roll-off in the sensitivity. It is straightforward to see this in some data-sets, but not others. In cases where it is not obvious, is there a good rule-of-thumb for how many edge channels to flag?
  • A: You can see the roll off in TDM mode but not in FDM mode. This is because the online system flags most of the edge channels in FDM mode but not in TDM mode. It is not necessary to flag channels where roll off is evident until the SNR becomes very small. Therefore, only a few edge channels need to be flagged.

Q8: Does flagging for shadowing work only for antennas in the array? For instance, suppose antenna A was next to antenna B, but antenna A had no power and so wasn't included in a data set. Will its location and orientation still make it into the header so that one can determine if antenna B was shadowed? For CARMA, the answer to this would be "no".

EVLA: ALMA:

Q9: When flagging antenna shadowed data, is it necessary to specify the diameter? Here diameter is specified as 12.0, but the flagdata help claims that leaving the parameter as a negative number will automatically use the antenna diameter. Since eventually antennas with multiple diameters (12 and 7 meter) will be in the array, it seems like letting flagdata figure out which diameter is appropriate should be the safest option. Is this true?

Q10: In the NGC 3256 Band 3 calibration, why don't we need to worry about CO (1-0) in the Titan observations corrupting the flux calibration?
  • A: The CO emission from NGC3256 has been redshifted. The sky frequency used to observe CO from Titan is very different from the sky frequency used to observe CO emission from NGC3256.

Q11: When describing what to look for when flagging, this casaguide states: "For example, if the phase calibrator drops out for a period of time we will not be able to calibrate and will need to flag the data."

Q12: Can I edit or modify a Tsys or WVR cal table if I need to?
  • A: You cannot easily do this right now. A request for this feature is in.

Q13: In the TW Hya band 7 imaging guide, it is noted that "the spw do not have the same amplitude. Unfortunately, the red spw (highest frequency) should be higher than the green one (lowest frequency) because optically thin dust goes as nu^4, and optically thick as nu^2. This inconsistency is due to imperfect amplitude/absolute flux calibration." I'm not sure what calibration error we're seeing here.

Q14: [ Initial Inspection and Flagging: Error: not an image at 'http://casaguides.nrao.edu/index.php' ] It appears that the 'birdies' often show up in common spectral channels of multiple spw's. Is this expected and thus is this something we should be able to flag as a single step?!?
  • A: A birdie has narrow width and appears in all sources. There are several known birdies that are being tracked and removed. All known birdies could be flagged a priori, but this is not currently happening.

Q15: How critical is a priori flagging? Are there observing modes where a priori flagging is a must?
  • A: There will always be things to flag before calibration.

Q16: How to reload in plotms from command line
  • A: Repeating a plotms command on the command line does not force a reload. Only if a selection parameter changes is a reload forced. However, if "force reload" is checked in the GUI, when the subsequent command is issued on the command line, a reload will be forced. Aside: It is not possible to run plotms synchronously, nor is it possible to poll plotms to determine if it is finished plotting.

Q17: Could you explain the antenna parameter choices? The & nomenclature makes no sense, are there other selections for antenna that might be helpful, though are opaque, much like &. The help also mentions *& selects cross correlation baselines, too. 

plotms(vis='X5d8.ms',spw='17,19,21,23',xaxis='frequency',yaxis='amp',field='0,1',
      antenna='*&*',avgtime='1e8',avgscan=T,coloraxis='field',iteraxis='spw',
      xselfscale=T,ydatacolumn='data')
Note: the antenna='*&*' is specifying that we only want to see the cross-correlation data (i.e. not the autocorrelations).

Never use antenna ID numbers. Use antenna names instead.

Q18: For plotting Tsys vs freq, each plot had 99840 points, with 26 scans superposed at each frequency. An option for averaging or better plotting one point with error bars, might be more efficient.
  • A: A new hire is required before this will be implemented.

Q19: casa did not like avgscan=T but worked for avgscan= True but this seems to be inconsistent as timesort=T works in concat.
  • A: This may be an anomaly.

Q20: Is there a systematic way one should check out the data for flagging? I really couldn't see any real pattern.

Q21: Very important, how does one decide when the data are okay and when they are not? In the NGC3256 case, for example, at one point it says that the data look fine, and then later they say to flag the same feature that earlier was okay.
  • A: You can see bad data better after calibration.

Q22: In TW Hya, we flagged out only the YY correlation for PM03, spw 2. Why then is there no bandpass calibration at all for PM03, spw 2?
  • A: This is a feature of CASA. If one correlation is flagged, applycal does not use the other correlation.

Q23: Is there an easy way to get a quantitative measure of the phase and amplitude scatter (e.g. RMS)? this will help us decide whether the wvr and tsys calibrations have worked.
  • A: Yes. There is a task called visstat that allows for selections. visstat is slow, so use a restrictive set of selection criteria to get RMS.

Q24: Is the flagging an AND or OR? ie if you select mode=manualflag with autocorr=True, as well as selectdata=True and antenna=DV04, will it flag only autocorr on DV04?
  • A: Flagging selection is AND. flagdata2 is just a reorganizaton of the inputs available in flagdata.

Q25: During flagging, it's not clear what the purpose is of "storing the current flagging state using flagmanager". Why we are we doing this, and what is it actually doing?
  • A: It is a preference. flagbackup=T can be used, or flagmanager can be used. Crystal recommends using flagmanager over the flagbackup parameter.

Q26: Is it possible to subtract the continuum from an other spectral window ( in frequency mode) that the one we study (if the line is too broad, or if their is a strong pseudo continuum)
  • A: Yes. You can do this in the image or uv planes. Use uvcontsub2. Note: uvcontsub and uvcontsub2 are very different; uvcontsub2 will eventually replace uvcontsub.

Q27: When we check a calibration with a loop in plotms, is it possible to switch between DATA and CORRECTED on the same frame and not begin again the loop?
  • A: If you change any selection parameter the iteration loop starts over.

Q28: plotms crashed several time. How can we restart it? I have to restart CASA.
  • A: Sometimes casa must be restarted.

Q29: Fixplanets: shouldn't this be done before flag for shadowing?
  • A: Probably.

Q30: How do you make the symbols in the CASA PlotMS window larger?
  • A: Go to display tab and select "Custom" for unflagged point symbol. Select the style desired. This cannot be scripted.

Note: Under tools tab, check 'hover' to view the coordinates under the mouse pointer. Display allows you to click and save the coordinates.

Note: Do not annotate a plot in plotms. This annotation will persist until casa is exited.

Q31: Why do you not need to back up the state of the flags before applying the calibration in the NGC 3256 example?

Q32: In the TW Hya example, there was discussion of "birdies". Birdies: how do we know for sure these are features? should they appear in all sources? Is there some easier way of looking for these birdies?

Q33: For 10 MSs, 4 plots (amp/phase vs time/freq) data and corrected over all baselines (~12) there are over 200 plots to look at. What will we do when we have 66 antennas, >4000 baselines?

Q34: There is a missed phase calibration information. The solution applied was to flag the last data set. Is it not a way to avoid to throw away a sizable amount of data, only for the fact that there is not phase cal at the end? Maybe extrapolate the gain solution??

Q35: In the applycal steps, flagbackup=F where when flagdata is run, that is set to T. In TWHydra, when flagdata is run, that is set to F explicitly in the guide. Is there a reason why we would want backup flags in one and not the other? Which should we default too when we tell our users?
Edit  | Attach | Print version | History: r11 < r10 < r9 < r8 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r11 - 2011-09-09, JaredCrossley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback