Date: May 11, 2023
Time: 11:00 am Mountain, 1:00 pm Eastern
 Meeting will be held over Zoom. Details in email.

NRAO: Jim, Justin, Steven, Robert,
USNO: Bruce, Roxanne, Phillip, Dave H, Sara, Andy, Jon, Megan, Phil C, Dave G, Chris

  • Station Update - MK down for scheduled major maintenance visit May 10-20. BR down due to HVAC problems. Eight antennas operational.

  • P-series Intensives Observation
    • P-series runs every other day.
    • MK suffered a failure of both elevation motors February 19, unable to observe.
      • Motors were replaced March 23.
      • One of the replacement motors failed April 1. Again unable to observe.
      • Another elevation motor installed April 21, operational again.
  • Other Intensives Observations
    • Z-Series runs approximately every other day, alternating with the P-series.
      • Some variability in cadence ok.
        • Toggling between the 5 and 12 degree lower limit is proving useful, probably more important than maintaining the cadence.
    • Q-series intensive observations at MK run on weekends.
      • Addition of Ishioka to the Q-series has introduced some unusual clock issues.
        • Prospect of adding a second VLBA station to tag along to help sort out these clock issues raised.
        • Is this still an issue - no. Ishioka has a new VGOS receiver so will be staying out of these unless/until they get "mixed mode" working.
        • They also had some maser issues that they addressed, could be that was involved in the clock issues seen earlier.
    • I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
  • Post-Monthly Report Review for April 2023
    • Due to the elevation motor failure at MK only 8 of the requested 15 P-series observations ran in April.
    • Three Q-series observation did not get observed due to the elevation motor failure.
    • Sixteen Z-series observations were scheduled, only fifteen ran due to one observation failing to be queued up. Request was for fifteen.
    • April request had 8 hours for Q-series observations but had 9 dates.
    • 48 hours of backlog from March, only 24 hours were caught up. Additionally, another 72 hours of observations in the April request were not observed so 96 hours have carried over into May.
  • Pre-Monthly Request for June 2023
    • Request has 7 hours for Q-series observations but has 8 dates.
    • New project UPXXX (starts in May request) asks for "all ten antennas optimal", does it actually require all ten antennas?
      • No files have been provided yet.

  • Open issues
    • Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator operations:
      • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. .
        • Future tests: June 7, July 13, August 2, September 13, October 4, November 8, December 13
        • May 3 test postponed to May 10 - went ok.
      • Further use of the backup correlator in the processing of P-series and Z-series observations has been proposed.
        • Eventual goal would be to see a demonstration of these being processed on the backup correlator without the involvement of any DC-based USNO personnel.
        • Some testing might involve using the VLBA correlator rather than the backup correlator.
        • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem later in 2023.
        • This will hopefully transition into the ability to do bigger projects for USNO at DSOC such as the 24-hour R4 observations.
        • Initial hope is to do one P and Z series on the backup correlator in April.
          • This did not happen in April.
          • Where to download the data to at DSOC before the deployment of the e-transfer system? Would it take up too much bandwidth at DSOC?
      • USNO would like to be able to check automatically if the backup correlator is up or down but are unable to do so, can NRAO set something up for this?
        • Mark Wainright working with CIS on it.
        • Update? Nothing since mid-April. Will ping Mark Wainright for an update.
    • Replacements for the Xcubes (called Data Buffer Server, DBS) have started rolling out. New software will ultimately cause a change in the naming structure of data files.
      • DBS units have been installed at PT and MK.
        • LA and/or HN will be next, no dates set.
      • Preview of the new naming structure will be provided to USNO.
        • This software change won't happen until all stations have had their Xcubes replaced.
      • An additional change the replacement modules will institute is that data will no longer be recorded in PFB (Mark5B) format, the new software will on the fly convert PFB data to VDIF format.
        • P-series and Z-series intensives currently use PFB format. I-series and Q-series intensives do not.
        • Some tweaks may be needed in correlator jobs to handle the format change. Walter will provide USNO with pertinent information.
        • This also won't happen until after all stations have had their Xcubes replaced.
    • Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
      • Neither Washington correlator nor NRAO set up to do it yet.
        • Blocking issue for NRAO is mostly network speed into the correlator.
        • USNO is not able to directly ingest data to their correlator. Could perhaps have data downloaded to the DMZ in realtime for near-realtime correlation. Will need to be discussed internally.
          • USNO will have faster line installed in early 2023 so will start looking into the possibility after that.
            • Install of higher speed line has been delayed - install expected in March.
              • Delayed again, maybe in April?
              • Has been completed but only recently so testing hasn't started yet.
    • Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
      • E-transfer not really practical at this time so expedited shipping of disk packs would be needed.
      • Record of time to correlation:
        • UD011AA 8 days, UD011AB 8 days, UD011AC 9 days, UD011AD 6 days, UD011AE 7 days, UD011AF 16 days, UD011AG 22 days, UD011AH 13 days, UD011AI 5 days.
        • No UD011 scheduled in February.
        • Two in March:
          • UD011AJ 8 days.
          • UD011AK 8 days.
        • Two in April:
          • UD011AL 7 days
          • UD011AM 10 days
    • New IVS session code naming structure change in 2023
      • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro, currently there are none.
        • Plans mentioned above to correlate some intensives in Socorro would not be impacted since data would be e-transferred instead of correlated from disk pack.
      • IVS will continue to use the existing shorter naming structure through all of 2023.
        • USNO notes they are using the longer naming structure in correlation and archiving of data but will probably hold off on using it for observations until after IVS makes the switch.
      • USNO will send some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
        • Test NT2022224AAA ran on Aug 12, 2022.
        • DiFX does have some project code length limitations, a modified trunk version of has been made for the correlation and the change will hopefully go into the next release (DiFX 2.8,which has now been released.)
          • There is an issue with Mark4 data, it does impact some USNO data (though not VLBA data).
        • SCHED also has some project code length limits but SKED and VieSched ++ may not?
        • Some minor issues were found with archiving of data and some automated emails truncated the longer project code.
  • Long term issues

    • USNO reports starting May 2021 have seen a lot of cable cal issues in the Z series.
      • BR notably bad (affecting Z-series). It has been seen there in non-USNO observing too.
      • A maintenance visit to BR is scheduled for later in 2023.
    • NRAO will at some point begin doing data downloads during and after observations and this could conflict with USNO's own data downloads of the intensives.
      • Currently NRAO is doing downloads about once a week in a post-maintenance day test, usually nowhere near the times of intensive observations.
    • Possible changes coming to login accounts on Xcube/DBS modules, moving to a group USNO account rather than using individual accounts. No ETA.
    • USNO-NRAO contracts:
      • V3 DiFX (correlator upgrade) contract awarded in March.
      • USNO has exercised their option on support contract.
    • There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
      • First IVS needs to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations.
        • Still working through correlator backlog, VGOS has ramped back because of that. Will be revised in early 2023.
          • USNO and Bonn have worked through their backlog, others have not though and probably not done by March so will be revisited again after that.
          • There will be a meeting in March to see how the others are doing with their backlog. Backlog might now be with stations not correlators.
            • Backlog's been dealt with, but nothing will change until 3rd quarter. Next discussion likely in July.
    • Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated and may have to wait until things return to normal after COVID.
      • Possibility of adding KK to an RDV observation raised.
      • Might be able to use WZ rather than KK to get the necessary clock offset data, they are already in RDV observations. Would need data analysis results from USNO.
      • NRAO probably will wait until after new GPS receivers get installed on array.
        • First has been installed at FD this week. Might take about a year to finish the installs.
    • USNO and IVS partners have challenges with creating VLBA-valid VEX files with the SKED and VieSched ++ software packages.
      • Why is there a need to add additional information that is not part of the VEX standard?
        • Tuning VLBA at certain frequencies requires the additional commented information, it's not found in standard VEX setups/catalogs.
      • What is the plan, if there is one, for removing the need for the additional information required by the VLBA?
        • VEX2 standard will cover the missing information for VLBA, though the scheduling software used will still need to have the right setups information.
        • Probably late 2023 before VLBA might switch to using VEX2, hopefully before 2025.
        • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
        • Will start testing of VEX2 on VLBA soon?

  • Fiber Status at each site:
    • 200 Mbps minimum at each site
    • 1 Gbps at PT,KP
    • 2.5 Gbps at HN
    • 10 Gbps at LA,MK
    • 600 Mbps at SC (download up to 400 Mbps)
  • The practical link capacity due to overhead and other traffic is at most 85% of the reported link speed.
Next meeting June 15, 2023

-- JimOgle - 2023-05-09
Topic revision: r4 - 2023-05-11, JimOgle
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback