• P-series runs every other day.
  • Were on hold due to failure of an azimuth wheel bearing at HN (September 15 - October 5).

• Z-Series runs approximately every three days.
  
  • BR cable/phase cal problem
    
    • Maintenance visit in late August seems to have fixed the cable cal problem at BR, there was one observation at the end of August that still showed it but then it looked okay through September.
  • SC began having a cable cal problem in June. Still being investigated.
  • SC was unavailable August 31 - September 15 due to a failed azimuth wheel bearing.
  • BR was unavailable September 8 - 21 for installation of a new backup generator.
• Q-series intensive observations at MK run on weekends.
• I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.

• P-series missed six sessions due to HN wheel bearing failure.

• New project UK002.

• new project UP001
• Full 203 hours requested (though the monthly time allotment is due to be recalculated for FY24.)
• UD011 AZ will be moved from November 20 to November 27 to accommodate fitting a maintenance day into the shortened week (Thanksgiving holiday).

• NRAO login accounts for new USNO analysts
  • Request made July 11 for Marc Murison, Robert Zavala, Remington Sexton, Christopher Dilullo
  • Update - as of September meeting Remington and Christopher's accounts have been set up. Marc and Robert were running into issues with securely sending PII to NRAO. Robert might be set up, no confirmation on that.
  • Couple more people will be requesting accounts in the near future.
• Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator operations:
  • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. .
    
    • Future tests: November 8, December 13
  • Further use of the backup correlator in the processing of P-series and Z-series observations has been proposed.
    • Eventual goal would be to see a demonstration of these being processed on the backup correlator without the involvement of any DC-based USNO personnel.
    • Some testing might involve using the VLBA correlator rather than the backup correlator.
    • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem later in 2023.
    • This will hopefully transition into the ability to do bigger projects for USNO at DSOC such as the 24-hour R4 observations.
    • Initial hope is to do one P and Z series on the backup correlator.
      
      • Datacopy locations available for use with this was provided to USNO on July 14.
      • Plan is for USNO to copy already processed data over, correlate it on the backup and compare the results.
      • Any attempts made?
        • None yet as of September meeting.
        • Still not tried.
  • USNO would like to be able to check automatically if the backup correlator is up or down but are unable to do so, can NRAO set something up for this?
    • Mark Wainright working with CIS on it, no update.
    • Might have to wait for e-transfer to be operational at NRAO due to internet security concerns.
      • E-transfer hardware is at DSOC, might be set up by November.
      • etc/etd will be the only transfer software initially supported.
      • Kokee will need to be asked to change to this software for the COOP tests. Wetzell has already switched to it.
• Replacements for the Xcubes (called Data Buffer Server, DBS) started rolling out but has stalled.
  • DBS units have been installed at MK and PT. No ETA on next install which will be either LA or HN. Goal is to have it done by March 2024.
  • New software will ultimately cause a change in the naming structure of data files and PFB data format will no longer be supported, instead it will be converted on the fly to VDIF format.
    
    • Software changes won't happen until all stations have had their Xcubes replaced.
• Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
  
  • Neither Washington correlator nor NRAO set up to do it yet.
    
    • Blocking issue for NRAO is mostly network speed into the correlator.
    • USNO is not able to directly ingest data to their correlator. Could perhaps have data downloaded to the DMZ in realtime for near-realtime correlation. Will need to be discussed internally.
      • USNO will have faster line installed in early 2023 so will start looking into the possibility after that.
        
        • First round of testing completed with Haystack. Didn't get the speed they expected, not sure if it's a Mark6 issue or networking. More testing to be done. USNO can get ~2.5 Gbps on an individual transfer, not quite good enough for realtime. There's also some security restrictions on the old correlator that might get in the way.
        • USNO working on some hardware upgrades to reach the desired transfer speeds.
• Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
  • E-transfer not really practical at this time so expedited shipping of disk packs would be needed.
  • Might be limited to just attempting it for one of the two observations per month.
  • Record of time to correlation:
    
    • first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
    • UD011 AS 9 days, UD011 AT 11 days, UD011 AU 6 days, UD011 AV 12 days, UD011 AW - observed October 9, waiting on disk packs to arrive.
• New IVS session code naming structure change in 2023
  • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro off of disk packs, currently there are none.
    • Plans are being developed in NRAO to overhaul Packtrack to cover things such as longer project codes and e-transferred data.
  • IVS will continue to use the existing shorter naming structure through all of 2023.
    
    • USNO notes they are using the longer naming structure in correlation and archiving of data but will probably hold off on using it for observations until after IVS makes the switch.
      • Might happen autumn 2023?
  • USNO sent some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
    • Test NT2022224AAA ran on Aug 12, 2022.
    • DiFX does have some project code length limitations, a modified trunk version of has been made for the correlation and the change will hopefully go into the next release (DiFX 2.8,which has now been released.)
      • There is an issue with Mark4 data, it does impact some USNO data (though not VLBA data).
    • SCHED also has some project code length limits but SKED and VieSched ++ may not?
    • Some minor issues were found with archiving of data and some automated emails truncated the longer project code.
      • Xcube/DBS recording software does look at project codes to determine to record or not, will need to be addressed.
• Can some kind of acknowledgement be added to USNO data in the NRAO data archive indicating USNO's involvement?
  • This is planned for the next development cycle of the archive (AAT 4.3.1) so should be coming by early 2024.

Long term issues

• USNO reports starting May 2021 have seen a lot of cable cal issues in the Z series.
  • See notes under Z-series intensives above.
• NRAO will at some point begin doing data downloads during and after observations and this could conflict with USNO's own data downloads of the intensives.
  • Currently NRAO is only regularly doing data downloads once a week during a post-maintenance day test. Sometimes this does come shortly after a Z-series observation during the night.
    • USNO has not noticed any problems with their data downloads even when this has been the case.
  • Data downloads are also being done during the Z-series and P-series observations for realtime fringe checks, but this ends when the observation does so would not interfere with the following data downloads by USNO.
• Possible changes coming to login accounts on Xcube/DBS modules, moving to a group USNO account rather than using individual accounts. No ETA.
• USNO-NRAO contracts:
  
  • V3 DiFX (correlator upgrade) contract awarded in March.
    • First milestone isn't until September.
      • Networking hardware is at DSOC and space in the server room was made for the new correlator. Computing and storage hardware is not expected for a couple more months though (probably arrive mid-November.)
      • Milestone date is March 31. USNO has concerns about the networking at the DSOC and wants to be kept up to date on the work.
  • USNO has exercised their option on support contract.
• There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
  
  • First IVS needs to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations.
    
    • Backlog had to be worked through first. That's been accomplished. Next discussion about it was to happen in July.
    • Not ready to include VLBA yet. Still issues within IVS and mixed mode to address first.
• Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated and may have to wait.
  
  • Possibility of adding KK to an RDV observation raised.
  • Might be able to use WZ rather than KK to get the necessary clock offset data, they are already in RDV observations. Would need data analysis results from USNO.
  • NRAO probably will wait until after new GPS receivers get installed on array.
    • Four antennas have been installed so far - FD, PT, OV, NL.
      • Antenna installations now scheduled to take until May 2024.
      • No ETA on GPS installations, originally planned to begin in January 2024 but now probably delayed until after antenna installations are finished?
• USNO and IVS partners have challenges with creating VLBA-valid VEX files with the SKED and VieSched ++ software packages.
  
  • Why is there a need to add additional information that is not part of the VEX standard?
    • Tuning VLBA at certain frequencies requires the additional commented information, it's not found in standard VEX setups/catalogs.
  • What is the plan, if there is one, for removing the need for the additional information required by the VLBA?
    • VEX2 standard will cover the missing information for VLBA, though the scheduling software used will still need to have the right setups information.
    • Probably 2024 before VLBA might switch to using VEX2, hopefully before 2025.
    • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
    • Testing of VEX2 on VLBA will hopefully start in the near future.

• 200 Mbps minimum at each site
• 1 Gbps at PT,KP,OV
• 5 Gbps at HN
• 10 Gbps at LA,MK
• 600 Mbps at SC (download up to 400 Mbps)