• P-series runs every other day.
• Z-Series runs approximately every three days.
  
  • BR cable/phase cal problem
    
    • Maintenance visit in late August 2023 at first seemed to have fixed the cable cal problem at BR, there was only one observation at the end of August and one in October that still had jumps in the cable cal but every other observation in September and October has looked okay. However, in November the cable cal problem began appearing again.
  • SC began having a cable cal problem in June 2023. Still being investigated.
• Q-series intensive observations at MK run on weekends.
• I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.

• UG005 C did not observe in February due to requiring SC. 24 hours of backlog carrried over into March.
• Ten Z-series observations were requested, only nine observed.
• UF006 requested 24 hours, used 21.01 hours.

• UG005 C and D unlikely to run before end of month due to requiring SC (repairs estimated to be done near the end of next week) so some backlog will carry over into April.
• 20 hours of US006 added on March 21 to March request as something to observe while SC is unavailable.
  • Given the late change it's unlikely all 20 hours can be observed during March.
  • Is this in addition to the 12 hours of US006 in the April request or in place of it?
    • It's an ongoing project so this is in addition to the originally requested hours in April.

• US006 request is a bit confusing, is it one twelve hour session with a target depending on when in the month it observes or is it two sessions with different targets and date requests? If two sessions, are they twelve hours combined or twelve hours each?
  • Will be multiple sessions with the different targets, not a single 12 hour observation.

• S-band performance issues
  • RFI has been increasingly affecting S-band observations for VLBA and VLA.
  • NRAO will be adding warnings to its proposal tools about it.
    • May suggest using narrower bandwidths in S-band observations to avoid the RFI but NRAO currently lacks standardized VLBA setups for that.
  • USNO still gets usable data from all antennas, only one S-band channel at SC is usually lost due to RFI.
  • Does anyone at USNO have time to work with NRAO on investigating this issue? Justin Linford will be initial POC at NRAO.
    • USNO will be in touch with Justin.
      • Justin's been testing things with the P-series setups. Justin will contact Chris Dieck.
• NRAO login accounts for new USNO analysts
  • Accounts have been set up, most issues resolved.
    
    • Marc has been issued credentials but not clear if he's managed to successfully log in yet.
      • Update on Marc? - should be resolved but USNO will double-check.
• Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator operations:
  • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. .
    
    • Future tests: March 6, April 3, May 8, June 5
      • March test results? Nothing reported.
  • Further use of the backup correlator in the processing of P-series and Z-series observations has been proposed.
    • Eventual goal would be to see a demonstration of these being processed on the backup correlator without the involvement of any DC-based USNO personnel.
    • Some testing might involve using the VLBA correlator rather than the backup correlator.
    • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem later in 2024.
      • etc/etd will be the only transfer software initially supported by the e-transfer system.
      • Kokee will need to be asked to change to this software for the COOP tests. Wetzell has already switched to it.
    • This will hopefully transition into the ability to do bigger projects for USNO at DSOC such as the 24-hour R4 observations.
    • Testing has been done on both P and Z series observations with the backup correlator.
      • Storage space in the datacopy areas is insufficient to hold data from a P series and a Z series at the same time.
      • Data transfer rates were exceptionally slow, probably due to using rsync because tsunami and etc/etd were not available to USNO users on gygax.
        • No movement on getting them access to the better programs.
          • Walter will follow up with Mark.
          • Has Mark reached out to Sara net?
            • Not yet. Walter can also help.
  • USNO would like to be able to check automatically if the backup correlator is up or down but are unable to do so, can NRAO set something up for this?
    
    • Might have to wait for e-transfer to be operational at NRAO due to internet security concerns.
      • E-transfer hardware is at DSOC, no ETA on when it will be ready.
        • external facing server has not been certified yet but hopefully by next month there will be news.
        • internal testing of data transfers have so far been successful.
• Replacements for the Xcubes (called Data Buffer Server, DBS) started rolling out but stalled.
  • DBS units have been installed at MK and PT. Goal is to have the remaining eight done by March 2024.
    • Three more are due to be installed this week - SC, HN and LA. If these are successful the remaining five might be done next week depending if some ordered parts come in on time.
      • HN was installed this week, SC and LA have been delayed until next week.
  • New software will ultimately cause a change in the naming structure of data files and PFB data format will no longer be supported, instead it will be converted on the fly to VDIF format.
    
    • Software changes won't happen until all stations have had their Xcubes replaced.
    • Won't happen immediately after the Xcubes are replaced, probably a month or two after that's completed before the software will be changed.
• Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
  
  • Neither Washington correlator nor NRAO set up to do it yet.
    
    • Blocking issue for NRAO is mostly network speed into the correlator. This will probably be another thing that has to wait for the e-transfer hardware to be installed.
    • USNO made some upgrades to their correlator's networking, up to ~2.5 Gbps on an individual transfer, but still not quite fast enough to do realtime transfers. Further upgrades are being worked on.
• Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
  • E-transfer not really practical at this time so expedited shipping of disk packs is needed.
  • Might be limited to just attempting it for one of the two observations per month.
  • Record of time to correlation:
    
    • first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
    • UD011 AS 9 days, UD011 AT 11 days, UD011 AU 6 days, UD011 AV 12 days, UD011 AW - 6 days, UD011 AX - 12 days, UD011 AY - 9 days, UD011 AZ - 15 days, UD011 BA - 7 days, UD011 BB - 17 days, UD011 BC - 8 days, UD011 BD - 20 days, UD011 BE - 8 days, UD011 BF - 18 days, UD011 BG - observed March 11, still waiting on disk packs.
• New IVS session code naming structure change in 2023-24
  • On NRAO's side, the Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro off of disk packs, currently there are none.
    • Plans are being developed in NRAO to overhaul Packtrack to cover things such as longer project codes and e-transferred data.
    • Packtrack will need to be rewritten to work under more up-to-date versions of Linux OS before other changes can be implemented.
      • Scheduled for June-July.
  • IVS will continue to use the existing shorter naming structure through all of 2023.
    
    • USNO notes they are using the longer naming structure in correlation and archiving of data but will probably hold off on using it for observations until after IVS makes the switch.
      • Might happen autumn 2023? Very close to doing it in USNO analysis, so probably will be using it by 2024.
        • No firm date yet, still working on software.
        • I and Q sessions will not change, P and Z sessions will. Some other USNO observations will also switch.
          • Actually I and Q names will probably change first. Goal is to be fully on the new naming convention next quarter.
  • USNO sent some representative "new style" IVS observing codes for NRAO to run through its software looking for any unexpected problems.
    • Test NT2022224 AAA ran on Aug 12, 2022.
    • DiFX does have some project code length limitations, a modified trunk version of has been made for the correlation and the change will hopefully go into the next release (DiFX 2.8,which has now been released.)
      • There is an issue with Mark4 data, it does impact some USNO data (though not VLBA data).
    • SCHED also has some project code length limits but SKED and VieSched ++ may not?
    • Some minor issues were found with archiving of data and some automated emails truncated the longer project code.
      • Xcube/DBS recording software does look at project codes to determine to record or not, will need to be addressed.
• Can some kind of acknowledgement be added to USNO data in the NRAO data archive indicating USNO's involvement?
  • This is planned for the next development cycle of the archive (AAT 4.3.1). Updated expected date now for that is now mid-2024.
    • Walter has been applying pressure to see that this stays on schedule.
• Possible changes coming to login accounts on Xcube/DBS modules, moving to a group USNO account rather than using individual accounts?
  • No ETA on if/when group accounts would be instituted.
  • NRAO is trying to move away from group accounts for cybersecurity reasons so things may remain with individual accounts.
• USNO-NRAO contracts:
  
  • V3 DiFX (correlator upgrade) contract awarded in March 2023.
    • Biweekly build status meetings between NRAO and USNO are being held Wednesdays.
    • Assembly of a subset of the v3 correlator has been completed at DSOC.
• USNO reported users' passwords not working on some machines.
  • According to computing department, password changes do not propagate from computers in the DSOC to computers at the VLBA stations, each machine needs to be individually updated.
  • Computing department recommends USNO users switch to using ssh keys instead of passwords to avoid this problem. Instructions on how to do this are supposed to be forthcoming.
    • Have not received any instructions from NRAO yet.
      • No. Can't put ssh key on machines they don't have a password for. Key is on MK DBS so just needs to be copied from there.
      • Ticket will be filed with NRAO helpdesk.
      • No update yet.
• Missing baseline delays seen in some USNO K-band observations. Possible correlator issue? Not sure where it's happening. USNO's using CASA. Walter & Justin think this might have been reported by other CASA users. Seth will submit a CASA helpdesk ticket. Chris DiLullo has also seen some issues with CASA. Will set up a meeting with Justin to see how to proceed with reporting these issues.

Long term issues

• NRAO will at some point begin doing data downloads during and after observations and this could conflict with USNO's own data downloads of the intensives.
  • Currently NRAO is only regularly doing data downloads once a week during a post-maintenance day test. Sometimes this does come shortly after a Z-series observation during the night.
    • USNO has not noticed any problems with their data downloads even when this has been the case.
  • Data downloads are also being done during the Z-series and P-series observations for realtime fringe checks, but this ends when the observation does so would not interfere with the following data downloads by USNO.
• There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
  
  • First IVS needs to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations.
    • Not ready to include VLBA yet. Still issues within IVS and mixed mode to address first.
    • Some mixed mode issues may have been resolved.
    • Might add some VGOS stations to RDV observations to see how that works, VLBA correlator should be able to handle it
• .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
  
  • Possibility of adding KK to an RDV observation raised.
    • Might be able to use WZ rather than KK to get the necessary clock offset data, they are already in RDV observations. Would need data analysis results from USNO.
    • KK will be in the April 2024 RDV observation so could get data from that.
  • NRAO might wait until after new GPS receivers get installed on array.
    • Seven antennas have been installed so far - NL, FD, LA, PT, OV, KP, SC.
      
      • Remaining antenna installation dates: BR - April 2024, HN - May 2024, MK - June 2024.
      • No firm ETA on GPS installations, originally planned to begin in January 2024 but now delayed.
• USNO and IVS partners have challenges with creating VLBA-valid VEX files with the SKED and VieSched ++ software packages.
  
  • Tuning VLBA at certain frequencies requires the additional commented information, it's not found in standard VEX setups/catalogs.
  • What is the plan, if there is one, for removing the need for the additional information required by the VLBA?
    • VEX2 standard will cover the missing information for VLBA, though the scheduling software used will still need to have the right setups information.
    • Probably 2024 before VLBA might switch to using VEX2, hopefully before 2025.
    • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
    • Some testing of VEX2 has begun at NRAO.
      • Some diagnostic observations have switched to vex2 for testing purposes.
      • Seems to be okay for some regular observing, pointing and other modes still need some work.

• 200 Mbps minimum at each site
• 1 Gbps at PT, KP, OV, NL. FD will be upgraded to 1 Gbps soon
• 5 Gbps at HN
• 10 Gbps at LA,MK; BR should be upgraded to 10 Gbps soon
• 600 Mbps at SC (download up to 400 Mbps)