• First of May we began seeing recording problems at MK in intensives, a small number of scans are dropped on both the DBS and Mark6, though the Mark6 drops fewer scans than the DBS.
  • Not consistent, some observations have had all of their scans.
• P-series has changed to fixed times/dates while Kokee is down and MK is substituting for it in some of the I-series intensives.
  
  • Continues into June
• NZ-Series runs approximately every three days.
  
  • BR cable/phase cal problem. Comes and goes, leading theory is thermal expansion and contraction of cables.
  • SC also has a cable cal problem but is slightly different in effect and doesn't seem to be due to thermal issues like at BR.
• Q-series intensive observations at MK run on weekends.
• I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
  • Also substituting for KK in some regular I-series intensives while their fiber is out.
  • Fiber repairs were expected to start March 17 and finish in early May.
  • This will continue through July, fiber repairs apparently delayed.

• MK substituting for KK I-series and new schedule for P-series observations continues.
• UD020C did not observe in April, backlog for May. (Observed May 1.)
• Two I-series observations in the initial request for April were later dropped.
• Two P-series observations were added late and given a 2x cost multiplier.
• One regular P-series observation and both of the late addition P-series observations did not run.
• UF007 requested 24 hours and used 34.16 hours.

• .MK substituting for KK I-series and new schedule for P-series observations continues.
• COOP test was originally scheduled for May 2, moved to May 22.

• MK substituting for KK I-series and new schedule for P-series observations continues.

• V2 correlator status:
  • Tuesday 5-7pm (Eastern Time) Maintenance Window weekly patches continue, with weekly reporting to USNO at the conclusion of each patch.
  • Troubleshooting as requested to maintain functionality.
  • USNO and NRAO completed the RMF Validation step in December. Since then, USNO and NRAO have been addressing questions and findings of reviewers, as well as remediating new issues. The closeout step of the authorization process is in final polishing by USNO.
  • Three storage drives were replaced in early May, 2 spares remain on hand.
  • NRAO is investigating a few issues - power supply faults (test) and a kernal on one MD node (production).
• V3 correlator status:
  • Progress continues to be slow due to effort on V2 RMF process and V3 delivery approach iterations with USNO.
  • NRAO is continuing progress toward milestone 3, software testing in Socorro on a two-processing host system using a representative dataset.
  • USNO has identified supplemental funding and prepared an updated Performance Work Statement for the V3 Correlator to formalize the changes via NAVSUP/NRAO CAP. However, some programmatic security uncertainties remain and are being studied by USNO and NRAO.

• USNO reports detecting a drift in MK baselines beginning mid-2023 and still happening.
  • Approximately 0.1 microsecond per day drift, works out to about 0.25 cm/year.
  • There was a major maintenance visit to MK in May 2023 during which the dichroic panel and antenna panels damaged by an ice storm in 2022 were replaced. Only other major work done in 2022 at MK was the replacement of an elevation motor in April.
  • It's seen in the Z/NZ series but not P-series? So maybe a correlation issue rather than with MK itself. Possibly seen in Q-series as well. Quick look at GPS data doesn't indicate a physical movement of the station.
  • No update.
• VLBA e-transfer system
  
  • RDV observations having non-VLBA stations e-transfered.
  • Also now testing a direct transfer with Effelsberg instead of going through Haystack.
    • That was successful, moving ahead and testing with other RDV stations.
  • Outgoing transfer to Bonn possibly in the near future.
    • First attempt at an outgoing transfer didn't work, appears to be a networking/firewall issue.
  • Anthony Sowinski new PoC for NRAO.
• Data ingest of IVS data to NRAO/Socorro in support of USNO V2 correlator ("backup correlator") continuity of operations (COOP):
  • We will do a monthly COOP test of the V2 correlator by correlating a WZ-KK intensive. .
    • April COOP test results - ok?
    • Next COOP test scheduled for May 22.
    • An actual use of COOP happened on May 7.
  • MOA actually was that the VLBA correlator would be used for the COOP not the V2. Once eTransfer is in place that could be done.
    • e-transfer is in use now, time to revisit this topic?
    • Different DifX versions are in use, some prep work would be necessary. A larger discussion is needed.
  • Further use of the V2 correlator in the processing of P-series and Z-series observations has been proposed.
    
    • An eventual goal would be to see a demonstration of these being processed without the involvement of any DC-based USNO personnel.
    • Another goal would be processing bigger projects such as a 24-hour R4 observation.
    • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem.
      
      • etc/etd will be the only transfer software initially supported by the e-transfer system.
      • Kokee will need to change to this software for the COOP tests. Wetzell has already switched to it..
        • Switch at Kokee delayed by their fiber outage.
    • Test correlations have been done on both P and Z series observations with the V2 correlator.
      • Storage space in the DSOC /datacopy areas is insufficient to hold data from a P series and a Z series at the same time.
        • More space will be available once e-transfer is in place.
          • Currently we are still using the datacopy areas for transfers from VLBA stations, procedures for using e-transfer area instead need to be developed.
      • Data transfer rates were exceptionally slow, probably due to using rsync because tsunami and etc/etd were not available to USNO users on gygax.
        • Will need to get USNO access to the new transfer software before further testing can be done.
        • Moving to eTransfer would also get the procedures off gygax which isn't really suitable for it.
• USNO would like to be able to check automatically if the V2 correlator is up or down but are unable to do so, can NRAO set something up for this?
  • Probably have to wait for e-transfer to be operational at NRAO due to internet security concerns.
  • If we switch to using VLBA correlator instead of v2 per the MOA would eliminate the need for this.
• Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives but neither Washington correlator nor NRAO set up to do it yet.
  
  • Blocking issue for NRAO is mostly network speed into the correlator.
    • Initial attempts at realtime correlation of VLBA data could begin soon. There will be some limitations - VLBA stations only, limited to observations that record at 512 Mbits/sec or less and NRAO may need some help from USNO in determining sessions, creating files and looking at the output. These observations would still record to Mark6 disk packs but if USNO intensives were used they would not record data to the DBS units.
      • Contact person at USNO to assist on this needed.
  • USNO not really set up for realtime with correlator design and workflow, and no real plans to do it in the near future. Wait for v3 correlator.
• Replacements of the Xcubes with DBS units
  
  • New software will ultimately cause a change in the naming structure of data files and PFB (Mark5B) data format will no longer be supported, instead it will be converted on the fly to VDIF format.
    
    • Software changes will need further testing before deployment and first iteration of the updates may not change naming structure of data files or include the PFB->VDIF conversion.
    • Low priority, won't happen until after eTransfer and v3 correlator.
• Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
  • E-transfer not practical at this time so expedited shipping of disk packs is needed.
  • Might be limited to just attempting it for one of the two UD011 observations per month.
  • Record of time to correlation:
    
    • first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
    • second year of observations (UD011 AS - BQ) average time was 12 days counting all observations, just counting the one observation a month we put a rush on the disk pack shipping the average time was 9 days.
    • third year: UD011 BR - 18 days, UD011 BS - 8 days, UD011 BT - 21 days, UD011 BU - 13 days, BV - 27 days; BW - 15 days; BX - 23 days; BY - 13 days; BZ - 20 days, CA - 17 days, CB - 13 days, CC - 17 days, CD - 17 days, CE - 10 days, CF - 13 days, CG - 14 days, CH - observed May 5
• Next DiFX release (2.9) - Slow development continues.
  
  • testing is likely to start soon but there's a patch that needs to be put in concerning Mark4 and that hasn't happened yet.
    • Lot of progress in the last month. Mark4 issue has been resolved. One issue involving VGOS data in HOPS package still being worked on.
    • Release candidate has been tagged, NRAO will start testing in the next couple weeks with a target release date around June.
    • Minor fixed are being integrated into the next release. Will start taking test data soon for comparison purposes. End of June still possible. HOPS issue has a workaround, a new version of HOPS should fix the issue with DiFX 2.9.
• VEX2
  • Vex2 standard will cover missing tuning information that SKED and VieSked++ aren't able to include.
    • This information will still be required but VEX2 will have a space for it so manual editing won't be needed.
    • USNO has been sent vex2 samples, will have them send NRAO a file to test and make sure it works.
      • NRAO has received samples, no test file yet.
  • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
  • Some testing of VEX2 has begun at NRAO.
    • Some diagnostic observations have switched to vex2 for testing purposes.
    • Seems to be okay for some regular observing and pointing tests. Some modes may still need testing.
    • Sched 12 is out, NRAO will switch to using vex2 internally after that. Vex 1.5 will remain supported but once VNDA is installed and running in 2026 NRAO will probably transition fully to vex2.
      • Vex2 transition for regular observing now likely to happen before VNDA is installed, so probably sometime later this year.But support of v1.5 will continue for at least a year for the observers who provide NRAO with vex files. Switch to VNDA will require going to Vex 2 full-time.
      • NRAO now using vex2 as a default, vex1.5 still supported. Should start a test phase on USNO-provided vex files with vex2 now rather than later.
• Installation of new GPS receivers at VLBA sites.
  
  • Antennas have been installed at all VLBA sites.
  • As of late February all stations had receivers installed except for SC and MK, but only PT and OV have been activated.
  • SC and MK have receivers installed now. Possibly all are active? But not receiving the desired data from them yet.
• Mixed mode correlation
  • 2025 RDV sessions will have at least one VGOS station in mixed mode, USNO might need to assist NRAO in correlating them.
  • Some issues did happen with a mixed mode station in RV169 but they were eventually solved.
    • Problem wasn't solved, the VGOS data was not correlated in RV169.
    • RV170 correlation underway.
• USNO asked about getting CASA to read pulse cal data the way AIPS does, it would help with their imaging efforts on weak astrometry targets at higher frequencies.
  • Walter thinks that this is on the CASA developer's target list.
  • Further discussion needed.
• VLBA planned to start upgrading its correlator from RHEL7 to RHEL8 in January 2025.
  
  • USNO still needs to use DiFX 2.7.
  • Upgrade progress - Mark6's need some hardware updates before VLBA playback units can be upgraded to RHEL8.
    • Procurement of the hardware upgrades is in progress.
    • First testing of new hardware on one Mark6 unit was successful, hardware for remaining upgrades has been ordered.

Long term issues

• New IVS session code naming structure change for intensives.
  
  • Problems encountered with the longer project codes:
    
    • Longer project codes not supported by all IVS stations so not all intensives will change to a longer code.
    • DiFX did have some project code length limitations, a change went into DiFX 2.8 .
      
      • USNO not using DiFX 2.8 yet. They can correlate with 2.8 but fringing doesn't work.
    • NRAO's Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes. This would only affect any intensives which get correlated in Socorro off of disk packs, but currently there are none.
• .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs?
  • Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
    
    • Possibility of adding KK to an RDV observation instead raised..
      • KK will be in the April 2024 RDV observation so could get data from that.
        • RV165 observed April 24. Has been correlated.
        • After USNO (or NASA?) does analysis a copy of the clock offsets can be provided to VLBA for comparisons.
          
          • Analysis finished, but didn't include VLBA stations so still need to do that and get the info to NRAO.
        • Kokee does have a different maser compared to last time VLBA correlated with them, might have an effect?
    • VLBA thinks it will have to wait until after all ten stations have received new GPS units (and they're all in use).
      • Prioritization of this and other lingering issues should be considered once we can start looking at this again.
    • Moving to VNDA will also change VLBA delays slightly, so it'll probably have to wait until after that's in place as well.
      
      • The RV165 clock offset data would be useful for a before and after comparison, but it would not be used to adjust the VLBA offset. A second observation like RV165 would be needed so we could make the comparison.

• 600 Mbps minimum at each site
• 1 Gbps at PT, KP, OV, NL. FD
• 5 Gbps at HN
• 10 Gbps at LA,BR, MK
• 600 Mbps at SC (download up to 400 Mbps) - possibility it could be going up to 1 Gbps