Date: February 9, 2023
Time: 11:00 am Mountain, 1:00 pm Eastern
 Meeting will be held over Zoom. Details in email.

NRAO: Robert, Justin, Steven, Jim
USNO: Bruce, Chris, Dave, Phil, Phillip, Roxanne, Sara, Megan
  • Daily Observation Report (P-series)
    • P-series runs every other day.
    • HN dichroic panel badly damaged by falling ice sometime after Jan 27. Awaiting installation of a replacement panel.
      • Should be installed early the week of Feb 13.
  • Other Intensive Observations
    • Z-Series runs approximately every three days.
    • Q-series intensive observations at MK run on weekends.
      • Addition of Ishioka to the Q-series has introduced some unusual clock issues.
        • Prospect of adding a second VLBA station to tag along to help sort out these clock issues raised.
    • I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
  • Post-Monthly Report Review for January 2023
    • Due to poor weather and mechanical issues only 13 of the requested 15 P-series observations were run in January.
    • One extra Z-series observation beyond the requested 10 was observed in January.
    • UY001D did not run in December, the PI requested that it be moved to January. A revised request was not submitted, it was decided that would not be necessary.
      • UY001D ran on January 25.
    • UD015R did not run in December, 24 hours of backlog carry through into January.
      • Ran on January 6.
  • Pre-Monthly Request for March 2023
    • The three 24-hour projects (UD011, UD017, UG004) increased to two observations in March, like was done in January.
      • This might be extended, USNO is considering it.

  • Open issues
    • Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator operations:
      • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. .
        • Test will no longer be a fixed date each month but flexibly scheduled based on the actual IVS schedule:
        • Future tests: February 8, March 1, April 5, May 3, June 7.
        • Results of January 11 test - program newsol not working on the backup correlator, being worked on.
        • February results - correlation okay, there was some confusion with the observing files which delayed availability of the data from KK.
        • USNO has no way of checking automatically if things are up or down due to lack of an RSA key. Can something be set up for them to do this?
          • Did anyone get in touch with USNO about this? No.
          • Will prompt CIS again.
        • Possible changes to COOP procedures coming, having non-Washington correlator personnel run the test. Will need to check that those personnel have the right permissions - this has been done.
          • Attempts to reach Merri S. Carter by CIS to get up her NRAO guest account have not been getting any response.
      • A bigger test, hopefully quarterly, of a 24-hour R4 observation is desired. NRAO would like a requirement document to figure out what needs to be done.
        • Waiting on funding and organizational planning of data management
        • Can begin establishing procedures / permissions for data download prior to operational disk array.
    • Replacements for the Xcubes (called Data Buffer Server, DBS) have started rolling out. Access to data should be unchanged (aside from the possible login change) through new software will probably cause a change in the naming structure of data files.
      • First replacement installed at PT in October, MK in December.
        • LA and HN expected to be next, no date set.
      • Preview of the new naming structure will be provided to USNO.
        • This software change won't happen until all stations have had their Xcubes replaced, tentatively March-April 2023.
      • An additional change the replacement modules will institute is that data will no longer be recorded in PFB (Mark5B) format, the new software will on the fly convert PFB data to VDIF format.
        • P-series and Z-series intensives currently use PFB format. I-series and Q-series intensives do not.
        • Some tweaks may be needed in correlator jobs to handle the format change. Walter will provide USNO with pertinent information.
        • This also won't happen until after all stations have had their Xcubes replaced.
    • Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
      • Neither Washington correlator nor NRAO set up to do it yet.
        • Blocking issue for NRAO is mostly network speed into the correlator.
        • USNO is not able to directly ingest data to their correlator. Could perhaps have data downloaded to the DMZ in realtime for near-realtime correlation. Will need to be discussed internally.
          • USNO will have faster line installed in early 2023 so will start looking into the possibility after that.
            • Install of higher speed line has been delayed - maybe in March.
    • Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
      • E-transfer not really practical at this time so expedited shipping of disk packs would be needed.
      • Record of time to correlation:
        • UD011AA 8 days.
        • UD011AB 8 days.
        • UD011AC 9 days.
        • UD011AD 6 days.
        • UD011AE 7 days.
        • UD011AF 16 days.
        • UD011AG 22 days.
        • UD011AH observed January 01. Correlation done January 14. (13 days)
        • UD011AI observed January 16. Correlation done January 21. (5 days)
        • No UD011 scheduled in February.
        • There will be two again in March. Can VLBA do a fast turnaround on both or just one?
    • New IVS session code naming structure change in 2023
      • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro, currently there are none.
      • While the longer naming structure will become possible in 2023 it looks like IVS continue to use current structure so USNO probably will too at least for part of 2023.
        • IVS likely to continue using the shorter structure through all of 2023.
        • USNO notes they are using the longer naming structure in correlation and archiving of data but will probably hold off on using it for observations until after IVS makes the switch.
      • USNO will send some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
        • Test NT2022224AAA ran on Aug 12.
        • DiFX does have some project code length limitations, a modified trunk version of has been made for the correlation and the change will hopefully go into the next release (DiFX 2.8, release coming soon?)
        • SCHED also has some project code length limits but SKED and VieSched++ may not?
        • USNO did correlate five of the stations with downloaded data and had no obvious problems with the project code length.
        • Correlation on VLBA correlator did not encounter any issues.
        • Some minor issues were found with archiving of data and some automated emails truncated the project code.
  • Long term issues

    • USNO reports starting May 2021 have seen a lot of cable cal issues in the Z series.
      • BR notably bad (affecting Z-series). It has been seen there in non-USNO observing too.
    • NRAO is starting to do more data downloads during and after observations and this could conflict with USNO's own data downloads of the intensives.
      • Currently NRAO is doing it mostly with a once a week test and during USNO intensive observations but it will eventually ramp up.
      • In an effort to coordinate download times between the two groups a small group to discuss it will be formed, probably two people from each side.
        • Nothing has been started yet.
    • Possible changes coming to login accounts on Xcube/DBS modules, moving it to a group USNO account rather than using individual accounts. No ETA.
    • USNO-NRAO contracts:
      • Contract paperwork is now with NAVSUP. 'BeeGFS' award likely to come first and then the correlator upgrade.
        • BeeGFS hardware has arrived at Washington. They need to get their power distribution set up to handle the new correlator, once that's done they'll be in touch about the installation of the new hardware.
      • V3 DiFX (correlator upgrade) contract likely awarded first quarter of calendar year 2023.
        • Expected to be awarded by end of March.
      • USNO has exercised their option on support contract.
    • There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
      • First IVS needs to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations. Still having concerns about correlator storage and transfer capacity at June 2021 meeting.
        • Discussed a little bit in May 2022 IVS meeting but nothing definitive.
        • Still working through correlator backlog, VGOS has ramped back because of that. Will be revised in early 2023.
          • USNO and Bonn have worked through their backlog, others have not though and probably not done by March so will be revisited again after that.
    • Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated and may have to wait until things return to normal after COVID.
      • Possibility of adding KK to an RDV observation raised.
      • Might be able to use WZ rather than KK to get the necessary clock offset data, they are already in RDV observations. Would need data analysis results from USNO.
    • USNO and IVS partners have challenges with creating VLBA-valid VEX files with the SKED and VieSched++ software packages.
      • Why is there a need to add additional information that is not part of the VEX standard?
        • Tuning VLBA at certain frequencies requires the additional commented information, it's not found in standard VEX setups/catalogs.
      • What is the plan, if there is one, for removing the need for the additional information required by the VLBA?
        • VEX2 standard will cover the missing information for VLBA, though the scheduling software used will still need to have the right setups information.
        • Probably late 2023 before VLBA might switch to using VEX2, hopefully before 2025.
        • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.

  • Fiber Status at each site:
    • 200 Mbps minimum at each site
    • 1 Gbps at PT,KP
    • 2.5 Gbps at HN
    • 10 Gbps at LA,MK
    • 600 Mbps at SC (download up to 400 Mbps)
  • The practical link capacity due to overhead and other traffic is at most 85% of the reported link speed.
Next meeting March 9, 2023

-- JimOgle - 2023-02-07
Topic revision: r2 - 2023-02-09, JimOgle
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback