Vex2 Community Notepad

Members of the VLBI community are welcome to leave suggestions here. I hope this page becomes self-organizing but may impose some order if necessary. Please leave your name and date with each comment. New topics should be added to the top with a level 2 heading; comments on existing topics should be nested and use level 4 or level 5 headings.

2014Oct03, Ed Himwich

For reference in discussion of handling non-sidereal sources, the following proposal that was removed from an earlier draft of VEX2 is presented. The original proposal was for an "ephemeris" source type. There was also a suggestion that a similar, but "non-retarded" ephemeris source type should be defined to help support RadioAstron.

Parameter Field Description Type Allowed values Units Comments
datum 1 Time epoch     Timestamp for this table entry
  2 Ref. Frame char J2000   For now, only J2000 is supported
  3 X length     Geocentric X position (appropriately retarded for light travel time)
  4 Y length     Geocentric Y position (appropriately retarded for light travel time)
  5 Z length     Geocentric Z position (appropriately retarded for light travel time)
  (6) VX velocity     Geocentric X rate (appropriately retarded for light travel time)
  (7) VY velocity     Geocentric Y rate (appropriately retarded for light travel time)
  (8) VZ verocity     Geocentric Z rate (appropriately retarded for light travel time)

Notes:
  1. For ephemeris driven objects, the position and velocity provided must be the apparent position and velocity at the specified time stamp, that is, suitably retarded to take into consideration light travel times, referred to the center of the earth.

2014Sep16, Ed Himwich

These are the comments from the VEX2 committee on the Community Notepad items so far. We thank the community, particularly James M Anderson, for the feedback and have tried to address all the issues raised.

We don't think there is an urgent need for many of these items and in the interest of completing VEX2 soon, we recommend that many be deferred to VEX2.1. We expect the time until a VEX2.1 update will be much shorter than has been the case for VEX2.0. This is partly because we expect the changes in VEX2.1 to be more incremental in nature and easier to transition to than VEX2.0.

We also feel that some issues may have been rendered moot by changes already included in VEX2.0. However, we plan to have a community feedback period before going ahead with the implementation of VEX2 and will consider feedback during this period about features that people feel are more urgently needed. Please also let us know if any substantive issue has been overlooked or needs more timely attention.

A summary of our response to the issues raised in reverse chronological order (the order of the notepad):

  1. 2013Apr25 (James)
    • source_model_uvrange
      • This would be easy to implement, but we thought it should be deferred. While source model information is used to construct the schedule, this information seems primarily to be aimed at internal set-up of phased arrays, which is probably outside VEX now, but in the future might not be. So we have deferred this for now. Experimentation with this can occur in the proposed $EXTENSION block.
  2. 2012Nov 15 (James)
    • Phased Arrays
      • For arrays, knowledge of the individual antennas actually used, and their weights (see 'Changes to $SCHED section' below) may be useful in correlation. However, we don't see an immediate need for. Unless those who run correlators see a near future use for this, it will be deferred for now.
    • Change limit on # of BBCs
      • Included in VEX2 but more so, it is now unlimited.
    • Changes to $DAS section.
      • These are pretty much made obsolete by the other changes we have made except of course to allow more equipment types, including specifically "Mark6" and "ALMA", but we don't think there is a need for "VDIF1". We have added "none"; it is already in use.
    • Changes to $SCHED section
      • The suggested parameters seem again very specific to the internal operation of arrays. Two have been superseded by other new features (including pointing_offset). The third (station_array_weights) is related to the definition of the stations used in a phased array and could be useful along with the definition of the stations in the array (see 'Phased Arrays' above) for correlation. If the correlators don't see a near future use for this, it will be deferred for now.
      • The three specified intents in the table have should be included. More intents can be defined of course. (There was a subsequent request for a CALIBRATE_ARRAY_PHASE_NEARBY intent, but maybe this should be APPLY_NEARBY_ARRAY_PHASING instead?). It does need to be clear though that VEX writers CANNOT assume any given station or correlator supports a specific feature. A note was added to that effect.
      • (An additional request from James was received for a FRINGE_FINDER intent, which was added.)
    • Changes to $SOURCE.
      • Apparently RadioAstron provides a non-retarded ephemeris (and perhaps calls it ITRF, instead of ECEF). So making this addition may just be allowing an established practice to be represented, but it is deferred for now. Our feeling was that this needs more definition, in fact, after fruther thought it was felt that the "ephemeris" source type was not well enough defined and was removed from the proposal, at least for now.
      • Since the retarded ephemeris was removed, there was no way to make it explicit that the retarded ephemeris is referred to the center of the earth, but this should be included when the ephemeris source type is added.
      • The "AzEl" coordinate information would support observations of terrestrial calibration beacons. It should be a separate source type, "AzEl" rather than changing the ref_coordinate_frame to "AzEl". This may need more development since apparently its intended purpose is to observe a terrestrial calibration target at ALMA, which would probably not have the same orientation for each antenna.
      • Adding polarization information to the source_model and including a source_model_ephemeris is easy to do. General source model information is used by geodesy for scheduling purposes, so it would not be out of place when (if) it would be useful for that. This would be an opportunity to define a standard that would be useful for correlators. However, there doesn't appear to be am urgent need for this at this time, so it was deferred.
  3. 2012Oct 31 (Walter)
    • FPGA personality by scan
      • Different scans can have different personalities by having different modes referencing different personalities/equipment (this flexibility is one of the strengths of VEX).
  4. 2012 Aug 23 (James)
    • Nitpicking
      1. Fixed block name character exclusion list, make it the same as links
      2. require four digit years, two no longer permitted
      3. Clock rate and higher order terms units are defined in VEX2.
      4. Create a "byte count" unit with decimal powers: B, kB, MB, GB, TB, PB
      5. Change "type" to "real" for axis orientation in axis_type
      6. Spacecraft axes are not being provided at this time.
      7. The mechanical axis offset between the first and second antennaaxes does not effect the parallatic angle. We need more information from users about box offsets for pointing before providing support for these.
      8. See (Nitpicking #3)
      9. See (Nitpicking #3)
      10. sub_lo_frequency example now has units
      11. We are not providing datum entries for orbiting stations at this time
      12. $THREADS is gone, issues are correctly handled in $DATASTREAMS
      13. See (Nitpicking #12)
  5. 2011 Jun 24 (Mark)
    • Remove tape $BLOCKS
      • $HEAD_POS, $PASS_ORDER and $ROLL removed.
  6. 2011 Jun 6 (Mark)
    • VEX2 uses explicit LO frequency "0 MHz" (or other units) for direct sampling of RF
    • note 6 in $MODE updated to use $DATASTREAMS rather than $CHANNELS, still leaves the question how to handle Mark 5A+ playback of Mark5B recording; the VEX file should represent the recording, but Mark says this is no longer an issue anyway since the hardware correlator is gone
    • pointing offsets and intents no longer depend on source "order"
    • $TRACK track_frame_format parameter eliminated
  7. 2011 Jun 2 (Walter)
    • suggestion to finalize VEX2.0, leaving some issues to 2.1
  8. 2011 Jan 2 (Walter)
    • switch power is now in $IF, this is probably sufficient for now, but if not we can add more
  9. 2010 Oct 11 (Mark)
    • pointing source is now specified
  10. 2010 Aug 23 (Ed)
    • All these issues have been superseded
  11. 2010 Aug 18 (Walter)
    • second VEX2 telecon, 13:00 UTC August 24
      1. We defer including correlation parameters until we have a specific proposal for a small subset of simple parameters to include. If having just the three parameters: accumulation period, spectral resolution, and whether to correlate cross-hands would provide significant utility, particularly for eVLBI, there is some interest in adding them.
      2. destination correlator list can be expanded
      3. likewise for rack types
      4. include IP address and other network parameters? probably not
  12. 2010 April 23 (Ed)
    • Nutation information added
  13. 2010 Mar 25 (Walter)
    • First VEX2 telecon, 14:00 UTC April 15
      • Cormac says LBA format can just be covered by existing use of $TRACKS, it is going away fairly soon anyway
  14. 2009 Nov 19 (Mark)
    • $BITSTREAMS is included now and does not include VDIF
  15. 2009 Nov 17 (Walter)
    • Changed to one defining document in Wiki format, will add separate parser test example file.
    • other issues already added
    • pulsar ephemeris: not at this time

2013Apr25, James M Anderson

Here is another comment after thinking about how ALMA may perform phasing for VLBI.

Changes to $SOURCE definition blocks to support phased arrays:

It would be useful to be able to feed in information about source structure to the system that is phasing up an array. Obviously, the source_model parameter can provide information about CLEAN components that make up some source structure. But often, this information is incomplete, either because one has not fully samples all possible (u,v) coordinates and therefore does not know how to describe the full source structure, or because the source structure on some baselines is complicated, and one would prefer to not deal with too complicated details. Therefore, I propose adding a new parameter, source_model_uvrange, that can give a rough idea about the range of baseline lengths for which the CLEAN component model described by the source_model components is valid. The idea is to provide a UVMIN and UVMAX similar to the VLA Calibrator Manual (http://www.vla.nrao.edu/astro/calib/manual/) that can be used by the on-line systems to simplify source structure. Source structure on vastly different (u,v) scales can be supported by using different Band_ID linkwords for the same frequencies... The units are standard lengths to prevent having to define a new kind of unit (wavelength).

ParameterSorted ascending Field Description Type Allowed values Units Comments
  2 Minimum (u,v) distance real >=0 length The minimum projected (u,v) distance which is supported by the source model
  3 Maximum (u,v) distance real >=0 length The maximum projected (u,v) distance which is supported by the source model. If the maximum distance is 0, then treat this as a code for infinity. (This is similar to the AIPS usage of uvrange.)
source_model_uvrange 1 'Band_ID' linkword to selected $FREQ 'def' &link      

2012Nov 15, James M Anderson

The comments below were derived after thinking about the needs of the ALMA Phasing project for phasing ALMA, and for observations with phased arrays in general.

Changes to $SITE definition blocks to support phased arrays:

The idea for these changes is to identify the individual antenna elements that make up a phased array. For many cases, it would be useful to have information about the individual antennas in a phased array, in order to figure out correct (u,v,w) coordinates, for example. LOFAR software uses the positions of each element of the phased array to calculate the beam gain as a function of direction on the sky (E Jones) during calibration and imaging.

For a site that is a phased array, site_position, site_position_epoch, site_position_ref, site_velocity, and so on, refer to the physical location of the phase center of the array. Similarly, zen_atmos, ocean_load_vert, and ocean_load_horiz refer to the values appropriate for the physical phase center of the array.

The allowed values for site_type should be expanded to include
  • fixed_array indicates a phased array with a physical phase reference location that is fixed with respect to the Earth.
  • earth_orbit_array indicates a phased array with a physical phase reference location that is of type earth_orbit

New parameter need to be introduced:

Parameter Field Description Type Allowed values Units Comments
site_array_stations 1..n List of station names that form the array &link     This is a list of station def block names that define the stations (site, antenna, das) of each element making up a phased array. This parameter is only allowed when site_type has the values 'fixed_array' or 'earth_orbit_array'.

Changes to $BBC section

The physical BBC number should not be limited to 16 for the BBC_assign parameter. For ALMA, there will be 256 different "BBCs". Even dividing that among the 4 separate recording systems that will be used, there are 64 BBCs per recording system.

Changes to $DAS section

Changes to parameters:

record_transport_type, field 1, should have an allowed value 'Mark6'. Who is defining the Mark6 stuff? In addition, an allowed value 'None' should be allowed for supporting phased array usage, where no actual output is recorded from a particular antenna site within the phased array.

electronics_rack_type, field 1, should have an allowed value of 'VDIF1' for electronics equipment that generate generic VIDF version 1 output (at least for electronics equipment that is not already covered by one of the existing types). In addition, an allowed value 'None' should be allowed for supporting phased array usage, where no actual electronics rack is being used for generating VLBI output from a particular antenna site within the phased array.

Or, electronics_rack_type, field 1, should have a new electronics rack type 'ALMA'.

Changes to $SCHED section

Add ability to have separate pointing directions for individual stations. This has been used in the past, fox example, to get in-beam calibrators and target objects into the same antenna beams for small dishes (large beams), while pointing at just the target or calibrator with large dishes (small beams).

Next, add the ability to control hierarchical pointing directions for phased arrays. For example, a phased array of dishes can have the dishes pointing in direction X, but being phased up for direction Y. For LOFAR, which contains phased arrays of phased array dipoles, the different levels of phasing can be pointed in different directions.

Then, add in the ability to track which stations in a phased array are actually used for observing, and with what weights. This is intended primarily to be filled in after observations, prior to VLBI correlation, to indicate which antennas were actually used in the phasing, and which ones were thrown out because of poor performance or mechanical/electrical failures --- things that the writer of the VEX observing file could not have known about. Of course, the observing script may try to control such things as well, supposing that the phasing system is capable of supporting such commands.

Finally, add the ability to give specific commands to individual sites to control processing of the scan in a site-dependent fashion. This is intended to give information to phased arrays, as an example, to say which method to use for phasing, to give a hint about how long the coherence time for phasing should be, and so on.

New parameters:

Parameter Field Description Type Allowed values Units Comments
station_pointing 1 Station &link     Link to station from STATION block
  2 Primary Pointing Direction &link     Points to entry in the SOURCE block The Primary Pointing Direction of the site. For standard dish antennas this gives the Pointing Direction of the dish. For phased arrays of dishes, this gives the phasing (not the dish) direction.
  3 Secondary Pointing Direction &link     Points to entry in the SOURCE block Valid only for phased arrays. The Secondary Pointing Direction specifies where the underlying phased array is pointing. For phased arrays of dishes, this specifies the pointing direction of the dishes.
  4 Tertiary Pointing Direction &link     Points to entry in the SOURCE block Valid only for phased arrays of phased arrays. The Tertiary Pointing Direction specifies where the underlying phased arrays of the phased array is pointing. For LOFAR, this indicates where the high band antennas are pointing, which are then phased into tiles, which are then phased into a station.
  5 Quaternary Pointing Direction &link     Points to entry in the SOURCE block Valid only for phased arrays of phased arrays of phased arrays. The Quaternary Pointing Direction specifies where the underlying phased arrays of the phased arrays of the phased array is pointing. For LOFAR, this indicates where the high band antennas are pointing, which are then phased into tiles, which are then phased into stations, which are then phased into tied-array beams.
station_array_weights 1 Station &link     Link to station from STATION block
  2 IF_ID &link     Link to the IF ID link word specifying the particular IF for which the weights listed are valid for this station. If blank, the weights apply to all IFs for this station.
  3 Weight Begin Time real   time Time after 'start' of scan at which the weights provided become valid.
  4 Weight End Time real   time Time after 'start' of scan at which the weights provided cease to be valid.
  5..n+4 Weight real     The weight of each phased array component for the station. The ordering of the components is the same as specified in the site_array_stations parameter in the station def block for this phased array station. Weight values of 1.0 indicate standard weights. Weight values of 0.0 indicate that the phased array component contributes nothing at all to the summation.
station_command 1 Station &link     Link to station from STATION block
  2 Time After Start real   time The time after the 'start' of the scan at which the command sequence provided here should be enacted by the station. Note that the time after start may be negative, allowing commands to be issued prior to the official start of the scan. If multiple command sequences are provided with the same epoch values, the station may receive them in any random order. If commands must come in a specific order, the Time After Start values must be adjusted to indicate the proper time sequence.
  3 Command char     Character sequence for special command to be given to the particular station.

As an alternative to station_command, allow literal blocks within scan definitions?

Questions for ALMA and ALMA phasing:

If spectral lines are used for pointing or phasing antennas, how can the spectral line be specified? Should spectral line information be specified in the SOURCE area in something like source_line_model, or should the information be specified in something like an intent parameter in the SCHED section with Identifier SPECTRAL_LINE_FREQUENCY and the barycentric line frequency provided in the value?

For ALMA, there is a desire to have the on-line system choose appropriate nearby strong calibrators to phase up the array, rather than having the astronomer specify the source to use for phasing up the array. (This is for cases when the target source itself is too weak for phasing.) The idea is that something like an intent parameter with Identifier CALIBRATE_ARRAY_PHASE_NEARBY would tell the on-line system to identify the best calibrator source near the provided source direction, and actually observe that source instead of the source listed in the VEX file. Then, a new VEX file will be written, putting in the correct source direction actually used for the observation, for subsequent use by the VLBI correlator.

Intent identifiers

Standard intent Identifiers should be formulated for phasing of arrays for producing data streams. Something like:

Identifier Origin Expected values Comments
CALIBRATE_ARRAY_PHASING   True/False Should array phasing information be computed from measurements made in this scan?
APPLY_PREVIOUS_ARRAY_PHASING   True/False Should array phasing information from the previously determined (last scan when CALIBRATE_ARRAY_PHASING was True) be applied?
APPLY_CURRENT_ARRAY_PHASING   True/False Should array phasing information be determined from measurements made within this scan (determine the array phasing and apply it during the same scan)?

Changes to $SOURCE section

Allow target sources that have fixed coordinates with respect to the Earth. (ALMA is implementing a polarization beacon for calibration that will have a fixed location, which will be seen at different directions for different antennas in the array.) Also allow target directions to be specified as azimuth and elevation angles, for example, for drift-scan experiments at individual experiments.

Then, update the source_model information to provide full polarization information.

For the source_type parameter:

Allow a new value 'ITRF' for field 1, Generic source type.

For 'source_type=star':

Allow a new value 'AzEl' for the parameter ref_coord_frame field 1, Source-position reference frame. When ref_coord_frame takes on the value 'AzEl', then new parameters 'az', 'el', 'az_rate', and 'el_rate' should be used instead of 'ra', 'dec', 'ra_rate', and 'dec_rate'. Note that azimuth and elevation are interpreted locally for each site in a geographic (not geocentric) sense.

Parameter Field Description Type Allowed values Units Comments
az 1 Azimuth Az   Az Example: -53d04'05.678" Azimuth takes a value of 0 in the direction of Earth North and increases toward Earth East. (So East has a value +90 degrees, South is +180 degrees, and West is +270 or -90 degrees.)
el 1 Elevation El   El Example: +53d04'05.678"
az_rate 1 Az proper motion real   ang rate Typically asec/sec
el_rate 1 El proper motion real   ang rate Typically asec/sec

The source_model parameter needs to be extended to provide polarization information.

Parameter Field Description Type Allowed values Units Comments
source_model 1 Component number int     One source_model=statement for each major source component for each 'Band_ID' link to the selected $FREQ 'def'
  2 'Band_ID' linkword to selected $FREQ 'def' &link      
  3 Component flux-density for Stokes I real   flux-density Stokes I
  4 Component major axis real   angle Angle subtended on sky
  5 Component axis ratio real      
  6 Component position angle real   angle  
  7 Component RA offset wrt specified source position real   angle If ref_coord_frame is 'AzEl', this is the offset in azimuth
  8 Component dec offset wrt specified source position real   angle If ref_coord_frame is 'AzEl', this is the offset in elevation
  (9) Component flux-densityf or Stokes Q real   flux-density Stokes Q
  (10) Component flux-density for Stokes U real   flux-density Stokes U
  (11) Component flux-density for Stokes V real   flux-density Stokes V

For 'source_type=ITRF':

Provide source position in a manner similar to 'source_type=ephemeris'.

Change the meaning of datum so that ITRF positions are supported and that the positions are NOT appropriately retarded for light travel time. At the same time, correct the Type and Units specifications.

Add a new parameter source_model_ephemeris to allow specification of the source model when 'source_type=ephemeris' or 'source_type=ITRF'. To simplify calculations, only allow spherical Gaussian components.

Parameter Field Description Type Allowed values Units Comments
datum 1 Time epoch     Timestamp for this table entry
  2 Ref. Frame char J2000 | ITRF   For now, only J2000 or ITRF are supported
  3 X real   length Geocentric X position (appropriately retarded for light travel time for J2000 coordinates, non-retarded for ITRF)
  4 Y real   length Geocentric Y position (appropriately retarded for light travel time for J2000 coordinates, non-retarded for ITRF)
  5 Z real   length Geocentric Z position (appropriately retarded for light travel time for J2000 coordinates, non-retarded for ITRF)
  (6) VX real   length/time Geocentric X rate (appropriately retarded for light travel time for J2000 coordinates, non-retarded for ITRF)
  (7) VY real   length/time Geocentric Y rate (appropriately retarded for light travel time for J2000 coordinates, non-retarded for ITRF)
  (8) VZ real   length/time Geocentric Z rate (appropriately retarded for light travel time for J2000 coordinates, non-retarded for ITRF)
source_model_ephemeris 1 Component number int     One source_model statement for each major source component for each 'Band_ID' link to the selected $FREQ 'def'
  2 'Band_ID' linkword to selected $FREQ 'def' &link      
  3 Component flux-density for Stokes I real   flux-density Stokes I
  4 Component diameter real   length physical size of the component
  5 Component X offset wrt specified source position real   length  
  6 Component Y offset wrt specified source position real   length  
  7 Component Z offset wrt specified source position real   length  
  8 Component VX offset wrt specified source position real   length/time  
  9 Component VY offset wrt specified source position real   length/time  
  10 Component VZ offset wrt specified source position real   length/time  
  (11) Component flux-density for Stokes Q real   flux-density Stokes Q
  (12) Component flux-density for Stokes U real   flux-density Stokes U
  (13) Component flux-density for Stokes V real   flux-density Stokes V

2012Oct 31, Walter Brisken

Concern about FPGA personalities

I'm noting here so I don't forget to bring it up next week.... The personality is currently slated to be set in the DAS block. If I understand it correctly, an observation that changes the personality within a single vex file will face extreme difficulties, if not impossibilities, due to the reference from STATION to DAS. I'm afraid this brings up a fairly big philosophical complication...

2012 Aug 23, James M Anderson

Nit-picking the details of Rev 1.999 May 26, 2011

1.In the legal character set area, bullet 2, is there a ":" missing from the exclusion list?

2.Can MJD values be accepted as valid epochs?The current specification says that only xxyxxxdxxhxxmxx.xxxs (possibly truncated from the right) is allowed.(And can this please be changed to xxxxyxxxdxxhxxmxx.xxxs with a note that 2 digit years are deprecated?)

3.What are the allowed units of time derivatives of time (for clock rates and higher order derivatives)?Are any units of time / time allowed, or just time / sec?

4.Unit specifications should also include data sizes.B, kB, MB,GB, TB, EB, and so on.And what do they mean?Is 1 kB 1024bytes, or is it 1000 bytes?Should VEX 2.0 use SI prefixes (for decimal or 2^{10}), or should we be using IEC 60027 notation (see,for example, http://en.wikipedia.org/wiki/Mebibyte).

$ANTENNA

5.axis_type:should the "axis orientation" value be specified as

  (3) axis orientation real   angle 0 deg - north/south orientation;
            90 deg - east/west orientation

6.What axis_type should be used for spacecraft that can orient themselves in arbitrary ways?

7.For some antennas (Effelsberg immediately comes to my mind), many receivers are offset from the main optical axis of the telescope, and the telescope main axis actually points in offsets in azimuth and elevation away from the target source that is being observed.These offsets are different for each feed.This impacts the calculation of the axis_offset, as the physical location of the telescope would be different for different receivers.More importantly, it affects the effective parallactic angle of the source depending on the axis orientation of the telescope.(Near zenith, the orientation of the telescope/receiver becomes very different from the orientation of anon-axis alt-az system.)This problem also arises for different"feeds" for phased-array feeds, where some of the feeds are up to several degrees away from the main optical axis of the telescope.Do we need something like an axis_angular_offset to indicate how far off from the target direction the main optical axis of the telescope is actually pointing?I see two angles that need to be specified.For an alt-az mount there is the rotation in azimuth and the rotation in elevation.For an equatorial mount, there is the rotation in hour angle and the rotation in declination.Since this is receiver dependent, should it go into $FREQ, or are different $ANTENNA defs needed for each receiver?Should we include an additional rotation angle of the feed (or the entire dish structure, such as is possible for the ASKAP antennas)?

$CLOCK

8."Clock rate" has often been used without units, leading to an ambiguity.For example, if "clock early" has units of "usec", some people/software assume the units of "Clock rate" are "usec/usec"(equivalent to "sec/sec") while others assume "usec/sec".(See the difference between how the DiFX and JIVE correlators interpret this value?)Can this be fixed in VEX 2.0 to require units?

9.For higher order polynomial terms, how are the units to be specified?Should they look like "sec/sec^2" or "sec/sec/sec"?

$IF

10.In the example section, the sub_lo_frequencies are specified without a frequency unit.Is this allowed?(Note that this is the first such frequency specified, not a continuation list.)

$SITE

11.For site_type==earth_orbit, allow datum entries as for source_type==earth_satellite?

$ THREADS

12.Why is there no "Mbps" unit label in the format example?Also, why is there no "Mbps" for "Data rate" in thread?Is the "Mbps"unit specified in the block description intended to be implied (so that it is never written in the parameter line), or should the units be required to be written out in the file?

13.In "channel", should "Chan_ID" have type &link instead of char?

2011 Jun 24, Mark Kettenis

Removal of tape-related parts of VEX

Blocks that are clear candidates for removal are $HEAD_POS,$PASS_ORDER and $ROLL.All these blocks are explicitly and implicitlyreferenced from other blocks that we can't remove.

Obviously all three blocks are explicitly referenced by the $MODEblock.However, we can easily adapt the examples in the $MODE blockdescription to remove those references.

The $ROLL block isn't really ereferenced from other blocks, but barelrolling is mentioned in the description of the $STATION and $TRACKSblocks.So some editing of the description of those blocks will berequired.

$HEAD_POS is referenced by the $PASS_ORDER block, since it defines thenumeric 'headstack-position reference'.Again some editing of the$STATION and $TRACKS blocks is required if we want to get rid of$HEAD_POS completely.

$PASS_ORDER is reference by the $TRACKS block and the $SCHED block.If we remove the $PASS_ORDER block, we'll have to specify that theSub-pass ID field of the fanout_def and fanin_def parameters in the$TRACKS section are now supposed to be empty ("null").The same forthe "Pass" field of the station parameter in the $SCHED block.Someediting of the description of the $SCHED, $TRACKS and $STATION blockswill be required.

Most of the necessary adjustments to the $SCHED, $TRACKS and $STATIONblock descriptions will simply be the removal of complete sentences.

2011 Jun 6, Mark Kettenis

Some notes:

  • For the lo_chain parameter in the IF block, it doesn't make much sense to list the first LO frequency/sideband as optional, isn't it?
  • In the description of the MODE block, note 6 says that only one of TRACKS, BITSTREAMS and CHANNELS can be referenced. For playback of a Mark5B recording on Mark5A+ it makes some sense to have both a BITSTREAMS section and a TRACKS section though. The TRACKS section would describe how the data would be spit out by the Mark5A+.
  • The description of the multiple phase centers still isn't completely clear to me. The current text suggests that intents have to be provided if multiple source parameters are provided. But intents for pointing center and correlation centers sren't listed. I'm also wondering if it makes sense to have a default meaning, where the first source would be where to point the antenna and additional sources would represent additional correlation centers.
  • Now that we have BITSTREAMS, I'm not sure it makes sense to list Mark5B as a possible track_frame_format in the $TRACKS section.

2011 Jun 2, Walter Brisken

I have added the propsed THREADS and BITSTREAMS blocks to the vex2 definition page and cleaned up a bit the intents in the MODE block.While there are still some features to be considered for eventual inclusion in vex2 (e.g., the switched power mentioned below) I suggest we move toward finalization of vex2.0 with the expectation that vex2.1 will be needed to address implementation issues and perhaps to add a few other things that got left out.

2011 Jan 2, Walter Brisken

Some thoughts on switched power specification within VEX...

The VLBA, and possibly other telescopes, have configurable switched power injectors, which I think should be controllable via DiFX now has switched power detection working too.The key parameters that I'd like to support are switching frequency (e.g., 80 Hz for the VLBA) and amplitude/power level (high, low, or off).Other possible (perhaps optional) parameters would be duty cycle (normally 50%), transition time, measured as fraction of a period, to blank (default 0%), and phase offset from rising edge on the 1 second tick.These parameters should be configurable on a per-IF basis and might be naturally added to the end of the if_def statement of the $IF block.Perhaps it would be cleaner to make a new parameter in the $IF block to store this information.

One could go a step further and add to each channel a Tcal indicating the additive temperature of the switched power.This would be a bit messy as the chan_def statement already supports a variable number of fields.

Thoughts on handling switched power this are welcome.

2010 Oct 11, Mark Kettenis

Issue with multiple phase centers

I think there is a (minor) issue with specifying multiple phase centers in a $SCHED block.The current proposal doesn't specify what the "main" phase center is, in other words, where the antennas should be pointing.The most natural thing would be probably to say that this is the first source defined for the scan.But I think this should be spelled out in the standardard.

2010 Aug 23, Ed Himwich

Here are a few considerations, hopefully without too many errors, but probably added too close to the telecon.

Suggestions for modifications for $CHANNELS

I have been with working folks at Haystack, primarily Alan Whitney and Mike Titus, on some suggested changes to $CHANNELS. In Walter's original proposal, the physical backend that a BBC is located in (via the "channel") is implicitly defined by the "backend number" in the "thread" parameter. We think that it should instead be in the $BBC section. This has implications for VLBI2010, for which we expect to have multiple backends (RDBEs) in use simultaneously. We suggest: (since VEX is already using "rack" for the "backend" concept, the reminder of this proposal uses "rack" instead):

(1) Either add a "rack" field to the "BBC_assign" parameter or define a convention for the construction of the "physical BBC" field, maybe "rack_BBC", with "rack_" being optional absent when there is only one rack (that should be backward compatible). There are pros-and-cons to each approach. We have a slight preference for the later, actual convention TBD.

(2) Define a similar convention for the "physical IF" in the $IF section, maybe (the same idea) "rack_input", with "rack_" being optional absent when there is only one rack (that should be backward compatible). Actual convention TBD.

(3) Remove the "backend number" [sic] from the "thread" parameter.

(4) Rename the $CHANNEL block to $VDIF_THREADS. This section deals with VDIF threads primarily and "channel" already has a different meaning in the world of VEX. This section is a functional replacement for $TRACKS.

(5) Add a "number_of_racks" parameter to the $DAS section. If this parameter is missing the value of its field is assumed to be "1".

Comments by W. Brisken 20100823

In general I agree with all of the suggestions above.Notably in my earlier proposal one could not construct a single thread from more than one "backend", which is indeed limiting; the above proposal gets around this shortcoming.

I have to disagree with the "racks" terminology, but only because I believe it will be common for one "rack" (in the conventional sense of a stack of hardware) to contain electronics for multiple "back-ends".I see vex2 as a chance to shed some now antiquated terminology which could include "racks", at least in new instances.Could we instead consider each "rack" a separate "DAS" and instead of having a "number_of_racks" parameter in $DAS have multiple DAS sections?

Possible alternative to $BITSTREAMS

This section details a possible different approach to $BITSTREAMS for handling Mark 5B. The motivation for this approach is to minimize the number of changes need to VEX, particularly for Mark 5B which we expect to be replaced by Mark 5C in due course. At the same time, this proposal tries to retain the functionality that Mark Kettenis is trying to achieve for Mark 5B. Mark and I have discussed this at length, but I don't know that all of his concerns have been addressed. I am including this alternative here so that it can seen and commented on by a larger audience.

The basic idea of this alternate approach is that Mark 5B can be supported through the existing "fanout_def" statements in the $TRACK section. As Mark would say this is "overloading" (whether that is desirable is another question) the "fanout_def"s compared to how they are used for Mark 5A. To do this, the input bit stream would be used in place of the track number (bit stream counting starts at "0"). This is a straightforward extension of the way track numbers are currently used. Only bit-streams that are actively being recorded would be listed. This has two features: (1) it allows the Mark 5B bit mask to be calculated (the "1"s in the bit mask correspond to the specified input bit streams), and (2) the order of the bit streams on disk, can be determined since they occur in order of the active input bit streams, but with non-active input bits streams "squeezed out". I think Mark's proposal also provides these features, although I think specifying both the input and disk order of the bit streams is redundant (although Mark’s proposal would utilize this redundancy to allow commenting out of “stream_def”s for processing a subset of recorded data, see below). The number of active input bit-streams must correspond to an allowed multiplex ratio (1, 2, 4, 8, 16, or 32). The multiplex ratio could be specified explicitly, although this would require an addition to VEX and might be redundant.

In both approaches you can specify subsets of data (by polarization for example) to process by commenting out the ones to not be processed in the $FREQ section.

In this approach, you can delete bit-streams by making the channel link empty. In Mark's proposal you would comment them out. A related issue is recovering if a station records with the wrong bit mask (i.e., too many "1"s).In this alternative approach, you would add "fanout_def"s with empty channel links as "dummies" to represent the recorded, but not useful data. I think in Mark's approach this would be handled by changing the disk bit stream numbers.

Another issue is eVLBI where JIVE is cleverly using subsets of bit streams to test the connection before using the entire intended bandwidth. In this case, the record bit mask is being controlled by the correlator and is changing in realtime. I think in both Mark’s and this approach you could select the subsets of bit streams (without having to modifying the contents of the “stream_def”s or “fanout_def”s contents) simply by commenting out bit streams that aren’t being transmitted and correlated. Possibly in Mark’s approach you would have to change the bitstream on disk order; I am not sure.

This discussion is not complete and I apologize if I have misrepresented Mark’s proposal. Presumably if there is interested we can articulate these issues more carefully before making a decision.

Other DBBC/RDBE Issues

It should be noted that some extensions may be require for the DBBC as well, in particular perhaps Nyquist zone selection in the "if_def" parameter. Is anyone familiar enough with the DBBC to comment on this? Otherwise maybe we need to consult Gino. Does the RBE have any issue like this we need to consider or is U/L for first and second zones good enough?

2010 Aug 18, Walter Brisken

The second vex2 telecon will occur at 13:00 UTC August 24 (Tuesday).

Agenda

  • Mark5B representation (led by Ed and Mark)
  • Further ruminations of the $CHANNEL (now proposed $VDIF_THREADS) block
  • VLBI2010 needs (led by Alan)
  • Correlator parameters
    • nChan, FFT_Size, integration time
    • Destination correlator (Chris suggests more options added; perhaps this should be made completely free-form?)
  • Chris Phillips suggested topics
    • "Generic" DAS type (e.g., for Westerbork/EVLA/ALMA where phased array is generated by single-built equipment)
    • How to specify complex sampling
    • eVLBI support, including IP addresses, other network params
  • Other?

2010 April 23, Ed Himwich

I edited the 2.0 draft to include the nutation ($EOP block) and real time data transfer ($SCHED block) parameters that are supported by the GSFC parser but not documented for VEX 1.5c. I also added the "nut_origin" ($EOP block) parameter.

2010 Mar 25, Walter Brisken (updated 2010 Apr 7)

Its time for a telecon, I fear.The meeting will occur at 14:00 UTC April 15.This meeting is intended for the 5 core vex2 committee members, but I see no problem having others listen in.Anyone interested should feel free to suggest discussion topics and provide input even if not attending the telecon.Non-committee members wishing to join please contact me so I can keep you up to date with meeting details.I will attempt to summarize the meeting with a set of notes to be made available here.

Agenda

  • Current Document
    • Description of the current status. (Brisken)
    • New format and new vex features added. (Brisken)
    • Is the wiki working? (All)
  • Mark5B
    • What is the best way to properly support this mode, and get support fairly soon? (Kettenis?)
    • Can this work as a subset of a VDIF capable format? (All; WFB: I'm starting to think not)
    • Can we accept $BITSTREAMS as it is defined? (All)
  • LBA
    • Can someone propose a standardized way to describe LBA format? (Cormac?)
      • Can one of $TRACKS, $BITSTREAMS, or $CHANNEL be used to properly describe the recordings?
  • Details
    • What is the purview of vex?
    • Which details should be left up to the site (and/or drudg or equivalent)?
      • I suggest any "flexible" parameter be required to have a formalized log counterpart
    • What actually defines the scope of the standard: a document or software?
      • I.e., can we establish a mechanism that allows fairly frequent updates to the standard?
    • Should we allow intrinsic VEX extensibility? (Alan?)
  • Logging formats
    • Should we standardize logging format for the following?
      • Media (including record format)
      • Tsys
      • Pulse cal
      • Flags
      • Weather
      • GPS-formatter clock offset
    • If so
      • Appoint someone to draft a plan
      • Establish time line for completion?
  • Schedule
    • When do we want to start wrapping things up for vex2.0?
  • Next meeting
    • Do we need to continue a semi-regular telecon until this finishes?
    • Is a face-to-face meeting needed to go the final mile?

2009 Nov 19, Mark Kettenis

For native Mark5B playback, I propose a new $BITSTREAMSblock to be used instead of the $TRACKS block for VSI-H based systems. At this stage it is not clear how this will relate to Walter's suggested $CHANNEL

block. It's not inconceivable that the needs of a software-oriented data format like VDIF are different enough to justify different blocks.

2010 Mar 24, Walter Brisken

This is a useful way to properly define bitstream mappings in Mark5B, however, I have some hesitation to promote it in a formal sense since it won't cleanly apply to VDIF format.I'm hoping to find a block definition that supports VDIF will and will be backward compatible with Mark5B.That may not be possible.For example if the order of sign and magnitude bits is allowed to vary then the VDIF definition will not support the $BITSTREAM block.The proposed $CHANNEL block would at a minimum need to allow more than 2-bits per sample and the concept of thread IDs and it would be good to envision a path forward for complex samples as well.

2009 Nov 17, Walter Brisken

Below are some of my thoughts on the changes I'd like to make to vex:

Some organizational changes
  • move much of the information in the Vex Definition document so that one document serves as the complete set of documentation.

Some new VEX features to put in soon or document for the first time:
  • parameter to enable/disable continuous switching noise cal
  • source calibration codes and qualifiers (per scan) (added)
  • new ephemeris source type -- arbitrary geocentric position tabulated in time
  • specification for multiple phase centers
  • documentation for some undocumented blocks: $SCHED (added), $MODE
  • reference to some newer backend types and formats: K5, Mark5B, LBA, VDIF (added)
  • higher order polynomial suport in $CLOCK block (added)
  • origin parameter in $EOP block (added)
  • breakdown of clock model into peculiar offset, gps, ...

Some features that will require some thinking:
  • New $CHANNEL block. This would be used instead of the $TRACKS block for newer systems
  • Pulsar ephemeris

2009 Nov 16, Walter Brisken

I think we need to develop an updated vex standard
2009 Nov 16, Walter Brisken

And it should be done via wiki!

-- WalterBrisken - 2009-11-16
Topic revision: r29 - 2014-10-03, EdHimwich
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback