• P-series has changed to fixed times/dates in September and October while Kokee is down and MK is substituting for it.
• Z-Series runs approximately every three days.
  
  • BR cable/phase cal problem
    
    • Maintenance visit in late August 2023 at first seemed to have fixed the cable cal problem at BR, there was only one observation at the end of August and one in October that still had jumps in the cable cal but every other observation in September and October has looked okay. However, in November the cable cal problem began appearing again.
    • USNO reports that BR cable cal was less of an issue in March and April 2024.
      
      • VLBA has noticed an increase in cable cal jumps since then. They appear to occur around the same times each day, 2000 and 0800 UT. Site tech speculates late afternoon heat and overnight cooling is causing expansion and contraction of cables.
  • SC began having a cable cal problem in June 2023. Still being investigated.
• Q-series intensive observations at MK run on weekends.
• I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
  
  • MK substituting for KK in late July though October due to their fiber being damaged and for other work at KK.

• UD019 G from July was reobserved in August, the July run was interrupted by a server room failure in Socorro that prevented us from monitoring the array.
• Ten Z-series observations were requested, eleven ran.
• UF006 requested 24 hours, used 23.08 hours.

• MK substituting for KK in I-series observations until fiber is repaired. Time charged at 2x.
• New schedule for P-series observations due to MK's involvement in the I-series observations.
  • Tuesdays and Thursdays either before or after the I-series.
  • Saturday and Sunday after the Q-series.
  • Exceptions in September - from September 16 - 29 MK will not be doing I-series observations so P-series can be dynamically scheduled during this period.

• MK substituting for KK and new schedule for P-series observations continues in October.

• KK will be down at some point in October/November for six to eight weeks for a receiver replacement.
  
  • With the fiber problems at KK, is the receiver work still scheduled for then or has that changed?
    • Receiver work will probably will be postponed. USNO will be discussing the scheduling of it with KK.
• Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator continuity of operations (COOP):
  • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. .
    
    • September 4 test canceled due to KK's fiber being out and not observing.
      • COOP test done with MK and WZ September 12
    • Future tests: October 1, November 6, December 4
      • October test will probably do MK-WZ again as KK is not expected to have working fiber yet.
      • November test also could be canceled due to scheduled receiver work? TBD.
  • Further use of the backup correlator in the processing of P-series and Z-series observations has been proposed.
    • Eventual goal would be to see a demonstration of these being processed on the backup correlator without the involvement of any DC-based USNO personnel.
    • Some testing might involve using the VLBA correlator rather than the backup correlator.
    • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem later in 2024.
      • etc/etd will be the only transfer software initially supported by the e-transfer system.
      • Kokee will need to be asked to change to this software for the COOP tests. Wetzell has already switched to it.
    • This will hopefully transition into the ability to do bigger projects for USNO at DSOC such as the 24-hour R4 observations.
    • Testing has been done on both P and Z series observations with the backup correlator.
      • Storage space in the datacopy areas is insufficient to hold data from a P series and a Z series at the same time.
      • Data transfer rates were exceptionally slow, probably due to using rsync because tsunami and etc/etd were not available to USNO users on gygax.
        • Will need to get USNO access to the new transfer software before further testing can be done.
        • Probably will have to wait for e-transfer to be in place, hopefully by end of September.
  • USNO would like to be able to check automatically if the backup correlator is up or down but are unable to do so, can NRAO set something up for this?
    
    • Might have to wait for e-transfer to be operational at NRAO due to internet security concerns.
• Replacements of the Xcubes with DBS units
  
  • Installs were completed on May 2.
  • New software will ultimately cause a change in the naming structure of data files and PFB (Mark5B) data format will no longer be supported, instead it will be converted on the fly to VDIF format.
    
    • Software changes will need further testing before deployment. First iteration of the updates may not change naming structure of data files, may also not include the PFB->VDIF format.
• Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
  
  • Neither Washington correlator nor NRAO set up to do it yet.
    
    • Blocking issue for NRAO is mostly network speed into the correlator. This will probably be another thing that has to wait for the e-transfer hardware to be installed.
      • There also appears to be a bottleneck at the switch into the DSOC, that may need to be upgraded too.
        • New switch is in-house at DSOC, installation TBD.
    • USNO made some upgrades to their correlator's networking, up to ~2.5 Gbps on an individual transfer, but still not quite fast enough to do realtime transfers. Further upgrades are being worked on - probably will have to wait for the v3 correlator. USNO is getting a new data server for e-transfer, don't know if it will provide enough overhead to allow for realtime.
      
      • Not really set up for realtime with correlator design and workflow, no real plans to do it in the near future. Wait for next gen correlator. Move to Long Term.
• Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
  • E-transfer not really practical at this time so expedited shipping of disk packs is needed.
  • Might be limited to just attempting it for one of the two observations per month.
  • Record of time to correlation:
    
    • first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
    • second year of observations (UD011 AS - BQ) average time was 12 days counting all observations, just counting the one observation a month we put a rush on the disk pack shipping the average time was 9 days.
    • third year: UD011 BR - observed August 26, awaiting correlation.
• New IVS session code naming structure change in 2024
  • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro off of disk packs, currently there are none.
    • Plans are being developed in NRAO to overhaul Packtrack to cover things such as longer project codes and e-transferred data.
    • Packtrack will need to be rewritten to work under more up-to-date versions of Linux OS before other changes can be implemented.
      • The rewrite has been postponed to FY25 due to a lack of resources.
  • IVS will continue to use the existing shorter naming structure through all of 2023 into 2024.
    
    • USNO notes they are using the longer naming structure in correlation and archiving of data but will probably hold off on using it for observations until after IVS makes the switch.
      • No firm date yet for IVS to make the switch, still working on software.
        • Might start using longer codes on some intensive observations soon? But not all of them immediately, depends on software developments. Goal is to be fully on the new naming convention before FY25.
        • USNO doesn't think any have started using the longer names yet.
          • Haven't started yet, not a priority.
  • USNO sent some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
    • Two test observations have been run.
    • DiFX did have some project code length limitations, a change went into DiFX 2.8 .
      
      • USNO not using DiFX 2.8 yet. Also believe that 2.8 has a problem with mixed mode?
    • SCHED also has some project code length limits but SKED and VieSched ++ may not?
      • Latest SCHED release (v12) will increase project code length limit to 12 characters.
    • Some minor issues were found with archiving of data and some automated emails truncated the longer project code.
    • Xcube/DBS recording software does look at project codes to determine to record or not, will need to be addressed.
      • Z2024160A was observed June 8 2024, the DBS units did not catch the longer project code name.
      • Thought to have been fixed, needs to be tested again.
• Can some kind of acknowledgement be added to USNO data in the NRAO data archive indicating USNO's involvement?
  • Learned in mid-April that this has been pushed back to development cycle AAT 4.4.1 which is expected out in November 2024.
    
    • Walter was able to get this pushed up to AAT 4.3.2 which is now expected out in September 2024.
• Possible changes coming to login accounts on DBS modules, moving to a group USNO account rather than using individual accounts?
  • No ETA on if/when group accounts would be instituted.
  • NRAO is trying to move away from group accounts for cybersecurity reasons so things may remain with individual accounts.
  • Move to long term.
  • Any idea of moving users to crypto keys? USNO would be willing to test that. Would solve some issues with their logins.
• USNO-NRAO contracts:
  
  • V3 DiFX (correlator upgrade) contract awarded in March 2023.
    • Biweekly build status meetings between NRAO and USNO are being held Wednesdays.
      • Monthly reports are available, will get some highlights from that for next month.
    • Some issues in installing DiFX 2.6.1 and a reliance on Python2.
    • Still some issues, especially with needing some backwards compatibility with v2.5, but work continues.
    • Various IT security requirements for procurements at USNO are a bit of a hurdle to forward movement.
    • No cost extension to contract through June 2025 has been granted.
    • USNO visited in August.

Long term issues

• Next DiFX release (2.8.2?) - a blocker issue for moving on to the next version for NRAO and probably USNO too has been resolved so moving to testing phase of the next release candidate can happen. Meetings for that began in May, probably take a couple months to sort everything out. Will be first release of it through GIT, procedures haven't been established for that yet. USNO input should be directed to Walter. Not likely it will be out until October.
• There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
  
  • First IVS needs to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations.
    • Not ready to include VLBA yet. Still issues within IVS and mixed mode to address first.
    • Some mixed mode issues may have been resolved.
    • Might add some VGOS stations to RDV observations to see how that works, VLBA correlator should be able to handle it
• .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
  
  • Possibility of adding KK to an RDV observation raised..
    • KK will be in the April 2024 RDV observation so could get data from that.
      • RV165 observed April 24. Has been correlated.
      • After USNO (or NASA?) does analysis a copy of the clock offsets can be provided to VLBA for comparisons.
        • USNO has started analysis. Still some work to do, this is a low priority item.
        • Analysis finished, but didn't include VLBA stations so still need to do that.
      • Kokee does have a different maser compared to last time VLBA correlated with them, might have an effect?
• Installation of new GPS receivers
  • Antennas have been installed at all VLBA sites.
  • No ETA on installation of the GPS receivers.
    • First site in late October possibly.
• USNO and IVS partners have challenges with creating VLBA-valid VEX files with the SKED and VieSched ++ software packages.
  
  • Tuning VLBA at certain frequencies requires the additional commented information, it's not found in standard VEX setups/catalogs.
  • What is the plan, if there is one, for removing the need for the additional information required by the VLBA?
    • VEX2 standard will cover the missing information for VLBA, though the scheduling software used will still need to have the right setups information.
    • Probably 2024 before VLBA might switch to using VEX2, hopefully before 2025.
    • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
    • Some testing of VEX2 has begun at NRAO.
      • Some diagnostic observations have switched to vex2 for testing purposes.
      • Seems to be okay for some regular observing, pointing and other modes still need some work.
      • Sched 12 comes out fairly soon, NRAO will switch to using vex2 internally after that. Vex 1.5 will remain supported but once VNDA is installed and running in 2026 NRAO will probably transition fully to vex2.
      • USNO would like some examples of differences between vex 1.5 and vex2 to see what would need to change. Some efforts on this have started. Move this up to Open Issues.
        • USNO hasn't received anything yet. Justin will work with Jay to get them something.

• 200 Mbps minimum at each site
• 1 Gbps at PT, KP, OV, NL. FD will be upgraded to 1 Gbps in June (done? still not getting over 400 Mbps.)
• 5 Gbps at HN
• 10 Gbps at LA,MK; BR should be upgraded to 10 Gbps but something is bottlenecking it to 2.5 Gbps.
• 600 Mbps at SC (download up to 400 Mbps)