Discussion on Cycle 0 Data Reduction Expectations

Friday, Sept 9, 2:55pm ET starting

Minutes entered by J. Hibbard

CB starts by setting scope: what will have been done with PI data from time data were taken until it is sent to PIs, & criteria it must pass (QA2).

EF & BVV take over podium.

From Ed's notes:

"Data Reduction Expectations in Manual QA2 in cycle 0"
What should we discuss?

1) What 'data' product(s) for the PI is ALMA's responsibility for Cycle 0
  • Only the ASDMs to...
  • 'Good quality' Images, ready for science analysis in many cases
(suspect we will be closer to the second 'product')
2) When we're tired of this, two 'presentations'
  • Ed
  • BVV

.... (get text file from Ed) ....

Discussion Items (WHAT TO DO)

1) User gets a MS where all the non-standard apriori calibrations have been made (half of the effort right now!)
  • Tsys, wvr, antenna positions. Pointing table, but no entries (JK says this will be fixed before ES)
  • flag shadowing? autocorrelations? flagging wrt tsys, wvr, dead antennas/spw/pol
  • anything else (protect the PI from non-standard stuff)?
  • Additional input information about scientific goals from SB's and PI's needed

SM: will we deliver staff columns? Have to JIRA weights problem
BVV: original MS (but before calibration applied), all scripts, logs, QA tables, flagmanager directories. Desire: PI to start from imaging step.
.....discussion from later brought up to here: give multisource MS with calibration applied; then don't have to deal with many of the problems.

Discussion: time averaging?? Yes. to 10-15sec. But no frequency averaging. CB: please don't time average my data
Liz, Kartik: P2G can capture this input from PIs during ph2 prep
Many people urge JAO to be flexibly with accommodating this desire
(maybe P2G JIRA ticket not best place. Work out exact procedure later)
SM: recommend capture all flagging thru flagcmd - easily transferable
CB: absolutely NO interactive flagging. Definite flagging commands

Back to Ed's file:

2) MS properties:
  • all sb's in one file (within 10hr time span?) or many ms's (1 per sb)? [Consensus: Give separate ms's (until we know better)]
    • Some people like the idea of a single final split target-only ms; CB refers to the many cautions given many times in the last two days
    • only science spw's. No ch0, no TDM tsys stuff etc. [AGREE]

split data???? separate ms
separate sb's => parallel system
[some solution said by Steve that CB understands that has to do with partitions, scan access splits, OBSid axis - JH couldn't follow but CB can probably summarize - this is probably best way to go]

3) Basic Nominal calibration:
  • Must do: bpass, temporal gain cal, absolute flux cal, and flags determined while doing this. Remove birdies, calibrator abs/emission.
  • Is that it?
  • Good calibration uv data is the product. Goes to PI with sufficient documentation
  • does NOT include images. Will be made along the way, but not delivered to PI. Is that ok? [We'll see. JH says we need plots]
4) Continue into imaging
  • Determine spectral emission channels. (don't include imaging of all channels of all spw) BVV: Pipeline plan: make all cubes, but only deliver those with strong lines [Everyone agrees except JH]
    • What about cubes for lines mentioned in proposal, plus any "strong" ones that may not have been mentioned (S/n>10 or so) [NO CONSENSUS ON THIS - SOME PEOPLE WANT THIS, OTHERS SAY JUST REPORT IF LINES DETECTED BUT DON'T MAKE THOSE CUBES, STLL ANOTHER SAY ONLY CUBES OF TARGETS]
  • image/clean target continuum and spectral cubes (those identified above). More flagging?
  • subtract continuum from sp line if applicable
  • Product: reasonable quality clean images with normal calibration

CB: be careful - don't do something now that pipeline will never do [CONSENSUS AGREE]. relates to 1st & 3rd bullets under (4)

3A) Checking of enough images over spws to feel confident that data calibration is ok, but don't deliver all. BVV wants consensus. NOTE: ALL DATA WILL BE CALIBRATED. Below, let ObsUnitSet=SB times number of execution. SB can have multiple sources. For line:
  • PROPOSAL 0: One cube of only one spw per ObsUnitSet. Votes: 18
  • PROPOSAL 0b: One source, all spw per ObsUnitSet. Votes: 25
  • PROPOSAL 2: "a representative sample" across L targets, M lines, N spectral windows: Votes: 7
  • PROPOSAL 1: make small cube of one line per spw per target, make sure noise OK, and then its up to them to take it from there. Votes:5
  • PROPOSAL 3: cubes of entire spectral window/targets: Votes: 1
3B) For continuum... (didn't get to this before JH had to leave.....)

4a) Mosaics: special problems? Lumpy sensitivity over mosaic (KS: especially if SBs end after 1hr - may not get through mosaic. CB: tell Stuartt Corder if he can make control script to make sure mosaics are not interrupted by SB end. KS is emailing SC - he cannot assure this can happen)

5) Deeper imaging
  • selfcal for target editing and improved image quality (CB: if you average some projects to 15sec, you will have destroyed ability to do this)
  • (this will be hard for many PIs) [CONSENSUS: if noise not close to theory, and if enough signal then yes do selfcal]
  • more 'coherent' spectral line results? moment maps, profiles, .... science driven? line identification as well as possible. [CONSENSUS: NO, WE DO NOT DO THIS]. But CB: must put in rest frequency of line, or you won't get a cube
  • Reduction team has to do "reality check" - were right frequencies/ra/dec/field imaged? (do we see lines that were looked at; sources that were targeted)
  • Dirk Petry: need to evaluate noise compared to theoretical expectation.
6) Write the ApJ paper automatically [CONSENSUS: NO]

..... JH had to leave..... See BV
Topic revision: r4 - 2011-09-12, JohnHibbard
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback