Attendees and planned schedule

• Ian Heywood, Oxford ianh at astro ox ac uk
• Remy Indebetouw, NRAO/NAASC rindebet at nrao edu
• Eelco van Kampen, ESO/ALMA evkampen at eso org
• Bojan Nikolic, Cambridge b.nikolic at mrao cam ac uk
• Kana Sugimoto, NAOJ/ALMA kana.sugi at nao ac jp
• Tak Tsutsumi, NRAO ttsusum at aoc nrao edu
• NAASC staff
• wkshp2010.sched.pdf: planned schedule (we deviated somewhat but followed this more or less)

casaguides example [RI]

• CO in a nearby galaxy suggested as an example of how to write up a simulation example for the user.
• comment: CASA should incorporate the exposure time calculator. That code is the same JAVA as in the ObsTool
• action [Harvey Liszt]: debug the OT/online calculator, forward the debugged code to this group [Jan 10: current best knowledge is here: ExposureTimeFormula but HL says it the online calculator is wrong.]
• action [RI]: figure out some way to run JAVA in CASA, or more likely, implement the same formula as a CASA simutil function (will look to the user like a CASA tool function e.g. simutil.alma_etime()) https://bugs.aoc.nrao.edu/browse/CAS-1921
• suggestion: put in the residual random phase noise after WVR calibration (30um/antenna, or use the spec)
• action [RI] document what Tsys, Trx are being used https://bugs.aoc.nrao.edu/browse/CAS-1922

Simulation Database [EvK]

• primary purpose is archive testing. secondary purposes: pipeline heuristics/algorithms testing, user support/example use cases
• we need volunteers to
  • create a project in the OT, with "simulation project" ticked. this will produce an APDM
  • produce corresponding simdata inputs
  • document on a wiki the origin of the sky model, including publication references, the goal of the simulated project, any modifications (brightness scaling etc) required of the sky model, and post the simdata.last inputs file and the model sky data.
  • run simdata
  • post the output, at least the graphic from CASA, and also preferably the ms and clean image, on the web page
  • run ms2asdm in CASA, and send ASDM, APDM, and description to evkampen@eso.org
• Eelco will attach a project ID, run a script to attach the ASDM and APDM, and upload the result to the archive
• we need a variety of projects, to span different kinds of science, but also different kinds of observations:
  • small and large mosaics
  • high and low spectral resolution
  • snapshot mode
  • etc
• The list is being kept by Eelco here - please send modifications and additions to evkampen@eso.org
• timeline: OT v7.1 (April) is supposed to produce APDM. it is not clear that the current version does. There is, however, no reason for people not to start working on their online use cases. We will consolidate them all onto one server eventually, the main goal now is to produce simulation projects, and save these to your local disk or local wiki.
• action: [all] create simulated use case examples

ms2asdm [Petry]

• will be implemented for March release as a separate executable, callable from CASA (old ms.toasdm() is deprecated).
• user inputs the Archive ID and range ID
• it is not clear where the WEATHER table and other meta-data is going to come from. Pipeline would like a WEATHER table that is correlated with data corruption, but more details and prioritization will be required.
• action: provide more input ms for testing
• action [RI] ask Steve if EVLA is interested in ms2asdm
• question: what do we do about single dish data, and total power data? The dedicated 12m total power dishes are expected to provide a separate measurement set, with data in a floatDATA column. It is not clear at this time that there would be significant advantages of combining these before imaging. The interferometric arrays will also produce square law detector total power measurements, and potentially on a much faster time cadence as the visibilities. It could be used in some way to evaluate Tsys, and in a simulation context, could be generated along with a simulated atmospheric opacity. Any such work would require coding effort, probably by RI. For now, we will assume that if such data is taken at ALMA it will be delivered in a separate data structure, and that there is no driving need to simulate such data. Someone in the project who knows otherwise should contact rindebet@nrao.edu
• question: what should we do about subscans? At current, the simulator tool does not do anything in particular about subscans. In the absence of other information, we will stick with one subscan per scan.
• action [RI] verify that simulator tool is filling subscan information thusly
• version 2 of ms2asdm is planned to add CalGAIN and ALMARadiometer tables.
  • The CalGain table is TBD in the ASDM definition. cal tables that are projected to be in the SDM arefrom TelCal, not pipeline, as far as this group understands it. Provided that that understanding is correct, we see no reason not to simply produce the simulated CASA cal tables that are used to corrupt the data (already being done), and then compare them with the CASA cal tables that will be produced by Pipeline attempting to calibrate our simulated data.
  • action [pipeline subsystem]: speak now if this plan is not appropriat, and clarify if there are any plans to define or use the SDM CalGain table.
  • there is a possible plan to create a CASA table manager that reads binary DSM data directly without translating into MS. It is not clear that this helps us much in a simulation context (since we're already working with MS) although it may prove useful for real data.
  • CalWVR table: measured sky brightness T in 4 WVR bands
  • WVMCal: the applied correction to the data
  • AlmaRadiometer table: describes the channels, correlator setup, etc.
  • concern: right now WVR data is being placed in the same MS as visibilities, but as extra spectral windows. It is not clearly marked which SpW are WVR data. This seems ill-advised and asking for processing errors.
  • concern: the current plan is to apply WVR in TelCal on a faster timescale than the visibilities are being recorded, which makes it impossible to remove the calibration.
  • if we are to generate simulated WVR data (still TBD whether this is worth the coding effort - it may be that real data catches up to this effort). CASA will need to generate a tropospheric phase delay cal table on a fast time cadence, the associated WVR T_ebb and calculated correction terms, but then average down the corrupted visibilities to simulate the irreversible operation being performed by TelCal. This will require more thought if it is to be implemented.

ALMA OST [Heywood]

• goal is an "enhanced sensitivity calculator", or simple limited interface to simulation
  • 2d model images and components only, limited to no user-settable clean options
  • suggestion: next version will need some kind of load management / prioritization
  • action: it better agree with the OT, etime calculator, and simdata for RMS noise
  • suggestion: provide the CASA script that gets run (not simdata, but something simpler) for interested users
  • suggestion: write and return to the user the equivalent simdata inputs, so that they could then do more complicated exploration with simdata in CASA offline
  • upgrade: add mosaicing, calculation of mosaic pattern as in simdata.
• input and output screens:
  
• Discussion with D. Halstead and NRAO IT about implementation and concerns
  • action [Halstead] discuss with ESO IT about how they will be hosting one of these
  • suggestion: this could be a very interesting prototype for "science gateway" access, or web interfaces to other parts of CASA, in the same vein as current VO access to compiled applications, and access to Teragrid.
  • action: [to whom?] coordinate CASA HPC efforts with web interfaces, Teragrid access, etc (here RI started understanding less of what is going on

Algorithms and more discussion

• action [RI] change "rm -rf" calls in simdata to os.remove [DONE 2010Jan20]
• use of simulated data for CSV: data from other telescopes, perhaps very deep datasets taken in several configurations, could be used as model sky, and compared with real data.
  • we have AIV/CSV configurations in simdata, so there are no obvious roadblocks for CSV use of simdata, just personpower
  • suggestion: make sure to mosaic the existing data in CASA, since e.g. miriad mosaicing algorithms are quite different from CASA, and we don't want to be confused by differences due to algorithms, rather than due to hardware.
• descriptions of how simdata runs "under the hood", and how simulator tool creates synthetic VisCals in the Measurement Equation
  • wkshp2010.pdf: slides
  • simulator tool: what happens if setnoise is run twice?
  • suggestion: the residual gain after calibration would be well modeled by starting with generalized 1/f noise, and then applying a high-pass filter to simulate the calibration process. This might end up looking like a simple gaussian white noise depending on the original exponent and filter, but probably not.
  • suggestion: there is a purpose to simulating bandpass corruption - at least sometimes the ALMA bandpass amp calibration is done with loads, and nonlinearity in the receivers, plus standing waves in the optics, lead respectively to a nonlinear error in the bandpass amplitude, and a sinusoidal ripple in amplitude only over the band (phase bandpass calibration should be okay, so simulating it as perfectly calibratible is probably okay) https://bugs.aoc.nrao.edu/browse/CAS-1924
  • action [RI]: try to talk to Richard about this
  • there is still interest from CSV to have the antenna-dependent feed angle variation effect simulated, although they may have done their own simulations by now - ask Richard this too.
  • what about off-axis polarization leakage? Since the gains from the two feeds vary independently, one gets differential cross-polarization phase, i.e. off-axis sources from different antennas have different phases, requiring an antenna-based solution.
  • suggestion: in the nearer future, just add a switch as to how correlated the two gains are - if simulating a G, allow either both pols to have identical gain fluctuations, or do them completely independently, or somewhere in between. https://bugs.aoc.nrao.edu/browse/CAS-1925
  • there probably is not a good way to simulate the residual T (phase delay), other than doing the full corruption, then simulating the fast switching and WVR correction process. We can implement residual random phase errors according to the ALMA spec, but it won't have realistic correlated structure in time.
  • simulating WVR data: one can just set up the WVR bands in ATM once, and then use the linear approximation - scale the ATM-output brightness temperatures proportionally to the PWV fluctuations in the screen https://bugs.aoc.nrao.edu/browse/CAS-1926
  • refraction: leads to a course delay error
  • baseline error K terms

SDsim [Kana]

• kana_simmeeting20091119.pdf: Kana's slides from Dec 2009
• right now it is planned for a complete merging with simdata. This is probably still the right thing to do. For now, modularization of each as they are developed independently should make merging later easier.
• sdsim makes a simulated MS, then manually changes some parts of the table so that the ASAP package can read it.
• action: implement reading a list of pointings from a file [DONE 2010 Jan]
• sdsim uses sm.oldsetnoise (ACoh objects) because VisCals don't corrupt autocorrelations (and VisCals have issues with real-valued parameters as opposed to complex ones). (probably longer-term)
  • action: [RI, Moellenbrock] convert sdsim to VisCal corruption by making VisCals act on AC rows.
• action: [Kana, RI] make sdsim work with cubes [DONE untested 2010 Jan]
• action: implement basketweave and Emerson gridding algorithms for processing the simulated SD data back into an image (these are at least partially in ASAP already but not fully attached to sdsim)

Pointing correction and corruption [Sanjay Bhatnagar by phone]

• "normal" application of sky angle-dependent effects: implemented in SkyJones. assumes the same effect for all antennas, and time-invariant effect (e.g. voltage pattern). However, pointing is both time and antenna-dependent, so would have to be applied on each baseline - calculation-intensive
• Instead, it is applied in FTMachine. Sanjay intends to move it into a modified SkyJones framework, on a timescale of several months.
• The corruption must be applied during initial calculation of visibilities - it can't be applied later like VisCal corruptions
• pointing errors can be calculated and corrected for during imaging - this is an iterative process: during clean, we have measured visibilities (with pointing errors), a current list of pointing offsets (to be refined), and a model image (also to be refined). Model visibilities are predicted from the model image, applying the current pointing offsets. The difference between model and observed visibilities forms the χ⊃2 used to iteratively refine both pointing offsets and model image.
• FTMachine produces the model image. nPBWProject is the FTMachine that is capable of dealing with pointing offsets
  • need an aperture illumination model for the antenna
  • from that, calculate the convolution function used in gridding. Normally the convolution function is real-valued prolate spheroidal, but here we make a complex one (so we can include the pointing offsets as phase offsets)
• offsets are stored in a modified Cal table - most cal tables have a complex Gain column with two values for the two poln's. For poining we use 4 real values representing the az,el pointing offsets (there is a parameter in the solver for whether the offsets are the same for the 2 poln's or not. We will want a similar parameter in the simulator when we generate synthetic pointing offsets)
• Plan for simulation: https://bugs.aoc.nrao.edu/browse/CAS-1927
  • [SB] verify that versions of EPJones and nPBWProject in active are (close to) the current state of the art from his sandbox.
  • [RI] add a setSimulate() to EPJones (send SB a proposed implementation first)
  • [SB] have nPBWProject take the EPJones as an argument (is it already this way?)
  • [SB] change nPBWProject to use regular PBMath to calculate aperture illumination - it is sensitive to telescope name, so we'll be able to put in the real ALMA aperture illumination. For the time being, a 12db Gaussian taper with 1m subreflector is pretty good for ALMA.
  • [RI] and [KS] check details of simulating in AC rows of Complex data column as opposed to floatData column, and try to anticipate problems with using the same machinery for SD and synthesis pointing errors.

SBsim work after workshop [RI and KS]

• RI modularized the casting of a model image into 4D CASA-preferred axis order (lat,lon,stokes,spec) into simutil.image4d()
• KS implemented reading pointings from a text file
• we planned for further modularization:
  • encapsulate all calculation, reading of mosaic pointings into a simutil function
  1. model sky manipulation plus mosaic/pointing setup > outputs modelimage.coord and pointings.txt
  2. ms calculation from modelimage.coord and pointings.txt
  3. corruption and calibration
  4. imaging
  5. diff/fidelity/analysis

Other suggestions

• generate data for CASA unit tests and regression tests
• option to display clean residual image, possibly instead of fidelity https://bugs.aoc.nrao.edu/browse/CAS-1923
• clean threshold instead of or and in addition to niter https://bugs.aoc.nrao.edu/browse/CAS-1928
• keep a hardcopy of the graphic output with the project name and number of the run appended https://bugs.aoc.nrao.edu/browse/CAS-1929
• calculate a scalar fidelity https://bugs.aoc.nrao.edu/browse/CAS-1929
• copy a simdata.last file with the time or number of the run https://bugs.aoc.nrao.edu/browse/CAS-1929
• write out a proper clean.last so that people know what clean was run https://bugs.aoc.nrao.edu/browse/CAS-1928