Date: November 10, 2022
Time: 11:00 am Mountain, 1:00 pm Eastern
 Meeting will be held over Zoom. Details in email.

NRAO: Jim, Walter, Steven, Justin
USNO: Bruce, Dave, Phillip H, Chris, Sara, Lucas, Phil C
  • Daily Observation Report (P-series)
    • P-series runs every other day.
    • A late change to the June request asked that the P-series be increased from 1.5 hours to 3 hours duration.
      • Will revert back to 1.5 hours duration in December.
      • Dropping duration to 1.5 hours for runs which interrupt other observations okay'd by USNO.
  • Other Intensive Observations
    • Z-Series runs approximately every three days.
    • Q-series intensive observations at MK run on weekends.
      • Sometimes having issues with the tsunami connection timing out during data transfers.
        • Happening less often now than a few months back.
      • Addition of Ishioka to the Q-series has introduced some unusual clock issues.
    • I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
  • Post-Monthly Report Review for October 2022
    • UC002 backlog from August was observed in October. The request in August had been for 17.5 hours but the files only used 15.54 hours.
    • UH007Y backlog from September was observed in October.
    • UD015P did not observe in October, 24 hours of backlog carried into November.
    • An extra P-series observation was run over the requested number. Most of the P-series observations in October were only 1.5 hours long due to interrupting other observations.
    • UD016/UF005 requested 24 hours but only used 9.32 hours.
  • Pre-Monthly Request for December 2022
    • As noted above, P-series returns to 1.5 hour duration in December.
    • Note: Beginning with November 2022 the monthly allotment of time to USNO has been updated to 203 hours from 200 hours.

  • Open issues
    • Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator operations:
      • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. .
        • Test will no longer be a fixed date each month but flexibly scheduled based on the actual IVS schedule:
        • Future tests: December 7.
        • October 5 test failed, was unable to download data from WZ. It was accessible a few days later and the November test went without any difficulties.
      • A bigger test, hopefully quarterly, of a 24-hour R4 observation is desired. NRAO would like a requirement document to figure out what needs to be done.
        • Waiting on funding and organizational planning of data management
        • Can begin establishing procedures / permissions for data download prior to operational disk array.
    • Possible changes coming to login accounts on Xcubes, moving it to a group USNO account rather than using individual accounts. No ETA.
      • Part of an overall increase in computer security within NRAO.
      • Unrelated to the security changes, replacements for the Xcubes (called Data Buffer Server, DBS) have started rolling out. Access to data should be unchanged through new software will probably cause a change in the naming structure of data files.
        • First replacement was installed at PT in October. MK is next site scheduled for a DBS.
        • Preview of the new naming structure will be provided to USNO.
          • This software change won't happen until all stations have had their Xcubes replaced, tentatively February 2023.
        • An additional change the replacement modules will institute is that data will no longer be recorded in PFB (Mark5B) format, the new software will on the fly convert PFB data to VDIF format.
          • P-series and Z-series intensives currently use PFB format. I-series and Q-series intensives do not.
          • Some tweaks may be needed in correlator jobs to handle the format change. Walter will provide USNO with pertinent information.
          • This also won't happen until after all stations have had their Xcubes replaced.
    • Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
      • Neither Washington correlator nor NRAO set up to do it yet.
        • Blocking issue for NRAO is mostly network speed into the correlator.
        • USNO is not able to directly ingest data to their correlator. Could perhaps have data downloaded to the DMZ in realtime for near-realtime correlation. Will need to be discussed internally.
          • USNO will have faster line installed in early 2023 so will start looking into the possibility after that.
    • Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
      • E-transfer not really practical at this time so expedited shipping of disk packs would be needed.
        • UD011AA observed on June 21, disk packs were shipped back over the next couple days and it was correlated on June 29. (8 days)
        • UD011AB observed July 18 and was correlated on July 26. (8 days)
        • UD011AC observed August 16, correlation was finished on August 25. (9 days)
        • UD011AD observed September 12. Correlation finished September 18. (6 days)
        • UD011AE observed October 11. Correlation finished October 18. (7 days)
    • New IVS session code naming structure change in 2023
      • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro, currently there are none.
      • While the longer naming structure will become possible in 2023 it looks like IVS continue to use current structure so USNO probably will too at least for part of 2023.
      • USNO will send some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
        • Test NT2022224AAA ran on Aug 12.
        • DiFX does have some project code length limitations, a modified trunk version of has been made for the correlation and the change will hopefully go into the next release.
        • SCHED also has some project code length limits but SKED and VieSched++ may not?
        • USNO did correlate five of the stations with downloaded data and had no obvious problems with the project code length.
        • Correlation on VLBA correlator did not encounter any issues.
        • Some minor issues were found with archiving of data and some automated emails truncated the project code.
  • Long term issues

    • USNO reports starting May 2021 have seen a lot of cable cal issues in the Z series.
      • BR notably bad (affecting Z-series). It has been seen there in non-USNO observing too.
    • NRAO is starting to do more data downloads during and after observations and this could conflict with USNO's own data downloads of the intensives.
      • Currently NRAO is doing it mostly with a once a week test but it will eventually ramp up.
        • Not ramping up yet.
      • In an effort to coordinate download times between the two groups a small group to discuss it will be formed, probably two people from each side.
        • Nothing has been started yet.
    • USNO-NRAO contracts:
      • Contract paperwork is now with NAVSUP. 'BeeGFS' award likely to come first and then the correlator upgrade.
      • V3 DiFX (correlator upgrade) contract likely awarded first quarter of calendar year 2023.
    • There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
      • First IVS needs to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations. Still having concerns about correlator storage and transfer capacity at June 2021 meeting.
        • Discussed a little bit in May 2022 IVS meeting but nothing definitive.
        • Still working through correlator backlog, VGOS has ramped back because of that. Will be revised in early 2023.
    • Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated and may have to wait until things return to normal after COVID.
      • Possibility of adding KK to an RDV observation raised.
      • Might be able to use WZ rather than KK to get the necessary clock offset data, they are already in RDV observations. Would need data analysis results from USNO.
    • USNO and IVS partners have challenges with creating VLBA-valid VEX files with the SKED and VieSched++ software packages.
      • Why is there a need to add additional information that is not part of the VEX standard?
        • Tuning VLBA at certain frequencies requires the additional commented information, it's not found in standard VEX setups/catalogs.
      • What is the plan, if there is one, for removing the need for the additional information required by the VLBA?
        • VEX2 standard will cover the missing information for VLBA, though the scheduling software used will still need to have the right setups information.
        • Probably late 2023 before VLBA might switch to using VEX2, hopefully before 2025.
        • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.

  • Fiber Status at each site:
    • 200 Mbps minimum at each site
    • 1 Gbps at PT,KP
    • 2.5 Gbps at HN
    • 10 Gbps at LA,MK
    • 600 Mbps at SC (download up to 400 Mbps)
  • The practical link capacity due to overhead and other traffic is at most 85% of the reported link speed.
Next meeting December 8, 2022

-- JimOgle - 2022-11-07
Topic revision: r3 - 2022-11-10, JimOgle
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback