• P-series has changed to fixed times/dates while Kokee is down and MK is substituting for it.
  • Continues into December.
• Z-Series runs approximately every three days.
  
  • BR cable/phase cal problem
    
    • Maintenance visit in late August 2023 at first seemed to have fixed the cable cal problem at BR, there was only one observation at the end of August and one in October that still had jumps in the cable cal but every other observation in September and October has looked okay. However, in November the cable cal problem began appearing again.
    • USNO reports that BR cable cal was less of an issue in March and April 2024.
      
      • VLBA has noticed an increase in cable cal jumps since then. They appear to occur around the same times each day, 2000 and 0800 UT. Site tech speculates late afternoon heat and overnight cooling is causing expansion and contraction of cables.
  • SC began having a cable cal problem in June 2023. Still being investigated.
• Q-series intensive observations at MK run on weekends.
• I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.

• MK substituting for KK I-series and new schedule for P-series observations continues.
• Monthly time allocation to USNO's annual recalculation increased to 214 hours.
• COOP test used MK instead of KK.
• Request asked for 18 P-series but it was agreed not to interrupt the GMVA session with one so the actual count was reduced to 17 in the pre-monthly report sent to USNO.
• UF006 requested 24 hours, used 23.97 hours.
• An extra I-series was observed for the monthly COOP test.

• MK substituting for KK I-series and new schedule for P-series observations continues.
• GC040 A substituted for November UD019 observation.
• COOP test for November rescheduled from November 6 to November 5 to match when MK was running an I-series intensive.

• MK substituting for KK I-series and new schedule for P-series observations continues.

• Data ingest of IVS data to NRAO/Socorro in support of USNO V2 correlator ("backup correlator") continuity of operations (COOP):
  • We will do a monthly COOP test of the V2 correlator by correlating a WZ-KK intensive. .
    
    • Future tests: December 4
    • Dates for 2025 needed.
    • November test used MK-WZ again due to KK's fiber problem.
    • December test will also need to use MK-WZ.
      • December test is scheduled for December 4 but with KK still unavailable this should be rescheduled to when MK is running an I-series intensive.
      • The NRAO correlator software group typically uses Mondays and Tuesdays for patch testing and application onto the V2 correlator so there's a potential of conflict if we use a Monday or Tuesday I-series intensive for the COOP unless correlation waits until after end of business day in Socorro (7pm Eastern time).
      • MOA actually was that the VLBA correlator would be used for the COOP not the V2. Once eTransfer is in place that could be done.
  • Further use of the V2 correlator in the processing of P-series and Z-series observations has been proposed.
    • And eventual goal would be to see a demonstration of these being processed on the V2 correlator without the involvement of any DC-based USNO personnel.
    • Another goal would be processing bigger projects on the V2 correlator such as a 24-hour R4 observation.
    • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem later in 2024.
      • etc/etd will be the only transfer software initially supported by the e-transfer system.
      • Kokee will need to be asked to change to this software for the COOP tests. Wetzell has already switched to it.
    • Testing has been done on both P and Z series observations with the V2 correlator.
      • Storage space in the /datacopy areas is insufficient to hold data from a P series and a Z series at the same time.
      • Data transfer rates were exceptionally slow, probably due to using rsync because tsunami and etc/etd were not available to USNO users on gygax.
        • Will need to get USNO access to the new transfer software before further testing can be done.
        • Probably will have to wait for e-transfer to be in place, now looking like January 2025.
• USNO would like to be able to check automatically if the V2 correlator is up or down but are unable to do so, can NRAO set something up for this?
  • Probably have to wait for e-transfer to be operational at NRAO due to internet security concerns.
• V2 correlator status:
  • Tuesday 5-7pm (Eastern Time) Maintenance Window weekly patches continue, with weekly reporting to USNO at the conclusion of each patch.
  • Troubleshooting as requested to maintain functionality
  • Assisting USNO with RMF ATO process.
• V3 correlator status:
  • NRAO is continuing progress toward milestone 3, software testing in Socorro on a two-processing host system using a representative dataset.
  • BeeGFS volumes have been prepared on test hardware
  • Preparing VM for Tenable Nessus security scanning
  • USNO has advised of change in information security requirements for its IT infrastructure and systems, and is preparing flowdown requirements for the V3 Correlator
• Can some kind of acknowledgement be added to USNO data in the NRAO data archive indicating USNO's involvement?
  • Acknowledgements are in AAT 4.3.2 which was released at the end of October.
  • There is a bug where the acknowledgement is being added to the wrong projects so a fix for that is forthcoming.
• Replacements of the Xcubes with DBS units
  
  • New software will ultimately cause a change in the naming structure of data files and PFB (Mark5B) data format will no longer be supported, instead it will be converted on the fly to VDIF format.
    
    • Software changes will need further testing before deployment and first iteration of the updates may not change naming structure of data files or include the PFB->VDIF conversion.
    • Low priority, won't happen until after eTransfer and v3 correlator. Maybe March 2025 at the earliest.
• Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
  • E-transfer not practical at this time so expedited shipping of disk packs is needed.
  • Might be limited to just attempting it for one of the two UD011 observations per month.
  • Record of time to correlation:
    
    • first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
    • second year of observations (UD011 AS - BQ) average time was 12 days counting all observations, just counting the one observation a month we put a rush on the disk pack shipping the average time was 9 days.
    • third year: UD011 BR - 18 days, UD011 BS - 8 days, UD011 BT - 21 days, UD011 BU - 13 days, BV - observed October 21, awaiting correlation; BW - observed November 4.
      
      • note: VLBA correlator has been experiencing an unexplained slowdown in job runtimes since late September which is lengthening correlation times. Possibly resolved on November 6, being monitored.
• New IVS session code naming structure change in 2024
  • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes. This would only affect any intensives which get correlated in Socorro off of disk packs, but currently there are none.
    • Plans are being developed in NRAO to overhaul Packtrack to cover things such as longer project codes and eTransfer data.
      • Packtrack will first need to be rewritten to work under more up-to-date versions of Linux OS before other changes can be implemented.
        • The rewrite has been postponed to at least Q2 of FY25 due to a lack of resources.
  • IVS will continue to use the existing shorter naming structure through 2024.
    
    • USNO notes they are using the longer naming structure in correlation and archiving of data but will probably hold off on using it for observations until after IVS makes the switch.
      • No firm date yet for IVS to make the switch?
        • There's a goal is to be fully on the new naming convention by January 2025.
        • Yes, USNO plans to start using longer codes on January 1 2025. Note that the first character might be changing which would impact the DBS recording software.
  • USNO sent some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
    • Two test observations have been run.
    • DiFX did have some project code length limitations, a change went into DiFX 2.8 .
      
      • USNO not using DiFX 2.8 yet. Also believe that 2.8 has a problem with mixed mode?
    • SCHED also has some project code length limits but SKED and VieSched ++ may not?
      • Latest SCHED release (v12) increase project code length limit to 16 characters.
    • Some minor issues were found with archiving of data and some automated emails truncated the longer project code.
    • Xcube/DBS recording software does look at project codes to determine to record or not, will need to be addressed.
      • Z2024160A was observed June 8 2024, the DBS units did not catch the longer project code name.
      • Thought to have been fixed, needs to be tested again. .
        • Does USNO need to do any further testing on their side, should we do another intensive (P or Z) with a long project code?
        • They don't need to test on their side so we just need to do a proof of concept test.
• Next DiFX release (2.9) - Slow development continues, still looking like December for release.
  • testing is likely to start soon but there's a patch that needs to be put in concerning Mark4 and that hasn't happened yet.
• VEX2
  • Vex2 standard will cover missing tuning information that SKED and VieSked++ aren't able to include.
    • This information will still be required but VEX2 will have a space for it so manual editing won't be needed.
    • USNO does have a concern about whether comments can safely be ignored in VEX2, Walter says yes.
    • USNO has been sent vex2 samples, will have them send NRAO a file to test and make sure it works.
  • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
  • Some testing of VEX2 has begun at NRAO.
    • Some diagnostic observations have switched to vex2 for testing purposes.
    • Seems to be okay for some regular observing, pointing and other modes still need some work.
    • Sched 12 is out, NRAO will switch to using vex2 internally after that. Vex 1.5 will remain supported but once VNDA is installed and running in 2026 NRAO will probably transition fully to vex2.
• Installation of new GPS receivers at VLBA sites.
  
  • Antennas have been installed at all VLBA sites.
  • installation of first receiver slated for the week of November 18 at FD.

Long term issues

• Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
  
  • Neither Washington correlator nor NRAO set up to do it yet.
    
    • Blocking issue for NRAO is mostly network speed into the correlator. This will probably be another thing that has to wait for the e-transfer hardware to be installed.
      • There also appears to be a bottleneck at the switch into the DSOC, that may need to be upgraded too.
        • New switch is in-house at DSOC, installation TBD.
    • USNO not really set up for realtime with correlator design and workflow, and no real plans to do it in the near future. Wait for next gen correlator.
      
• Possible changes coming to login accounts on DBS modules, moving to a group USNO account rather than using individual accounts?
  
  • NRAO is trying to move away from group accounts for cybersecurity reasons so things will probably remain with individual accounts.
  • Will drop this item since it doesn't appear the change is going to happen.
• There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
  
  • .Not ready to include VLBA yet. Still issues within IVS and mixed mode to address first.
    
    • Some mixed mode issues may have been resolved.
    • Might add some VGOS stations to RDV observations to see how that works, VLBA correlator should be able to handle it
    • 2025 RDV sessions will have at least one VGOS station in mixed mode, USNO might need to assist NRAO in correlating those.
• .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs?
  • Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
    
    • Possibility of adding KK to an RDV observation instead raised..
      • KK will be in the April 2024 RDV observation so could get data from that.
        • RV165 observed April 24. Has been correlated.
        • After USNO (or NASA?) does analysis a copy of the clock offsets can be provided to VLBA for comparisons.
          
          • Analysis finished, but didn't include VLBA stations so still need to do that and get the info to NRAO.
        • Kokee does have a different maser compared to last time VLBA correlated with them, might have an effect?
    • Moving to VNDA will probably change VLBA delays slightly, so probably no effort will be made to update those before that happens.
      • The RV165 clock offset data would be useful for a before and after comparison, but it would not be used to adjust the VLBA offset. A second observation like RV165 would be needed so we could make the comparison.

• 600 Mbps minimum at each site
• 1 Gbps at PT, KP, OV, NL. FD
• 5 Gbps at HN
• 10 Gbps at LA,BR, MK
• 600 Mbps at SC (download up to 400 Mbps) - possibility it could be going up to 1 Gbps

• But there might be a second bottleneck so the new switch might not clear that up right away.
• A DSOC-wide network switch change in 2025 should also lead to faster speeds.