Date: October 17, 2025
Time: 10am Mountain, Noon Eastern. Meeting will be held over Zoom. Details in email.
NRAO: Jim Ogle, Heather Marshall, Thomas Coleman, Justin Linford
USNO: Sara Hardin, Phillip Haftings, Chris Dieck
- Station Update - 10 stations operational. MK weather station in the process of being replaced so temporarily without weather data.
- Intensive Observations
- P-series runs Tuesdays and Thursdays around the time of I-series intensives, Saturdays and Sundays after Q-series intensives.
- NZ-Series runs approximately every three days.
- BR cable/phase cal problem. Comes and goes, leading theory is thermal expansion and contraction of cables.
- SC also has a cable cal problem but is slightly different in effect and doesn't seem to be due to thermal issues like at BR.
- Q-series intensive observations at MK run on weekends.
- I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
- There will be an upcoming period in the future where Kokee will be down, some signal change and receiver upgrades, March 2026 currently the target for that work.
- Post-Monthly Report Review for September 2025
- US008 was postponed to October.
- Request asked for 17 P-series observations, this was reduced to 14 due to scheduled maintenance visit at HN and to GMVA observations at the end of the month. Maintenance visit ran longer than scheduled and two additional P-series observations did not run.
- Ten NZ-observations observations were requested, nine ran.
- UF007 requested 24 hours, used 32.68 hours.
- Request for October 2025
- Global observation GC040 will run in place of the monthly UD020.
- GC040 runs for 48 hours at the end of the month, half of the hours will be counted in October and half in November.
- Pre-Monthly Request for November 2025
- MK substituting for Kokee in two I-series intensives over the Thanksgiving holiday.
- New fixed date observation UF008.
- Global observation GC040 will run in place of the monthly UD020.
- Correlator Statuses
- V2 correlator status:
- Tuesday 5-7pm (Eastern Time) Maintenance Window weekly patches continue, with weekly reporting to USNO at the conclusion of each patch.
- Troubleshooting as requested to maintain functionality.
- Authority to Operate is in place for the NGN/V2, the system is now in the continuous monitoring phase.
- NRAO has begun training a representative of USNO/N6 on ongoing V2 Correlator maintenance, scope for which USNO intends to internalize in CY26. Options for winding down/transitioning other contract scope items were presented to USNO on Oct 8th.
- In September, one storage drive faulted and was replaced; server 5 spare storage drives are available at USNO.
- V3 correlator status:
- Contract modification #5 was executed on September 30, 2025, updating cost and schedule in accordance with NRAO's Proposal/SOW.
- NRAO is proceeding accordingly with purchases, design reports, etc. per the contract milestone schedule. Invoice submitted for hardware purchases milestone is blocked due to mismatch of DODAAC entry in contracting system, unable to resolve due to federal government shutdown.
- Open issues
- DBS software changes
- New software on the DBS units will ultimately cause a change in the naming structure of data files and PFB (Mark5B) data format will no longer be supported, instead it will be converted on the fly to VDIF format.
- Deployment of VNDA would make this change unnecessary?
- Minor hardware change to the DBS file structure made in August 2025, storage hard drive was repartitioned so slightly less storage space for USNO data is available.
- xcuberec program which records the data on the DBS will be replaced by sendit, NRAO could start running it for P-series in parallel with xcuberec for testing and comparison purposes if USNO would be able to do a double correlation. Might be ready to go by the end of September. Sendit does change the file naming structure as mentioned above.
- Sendit testing on hold due to government shutdown.
- VLBA e-transfer system
- RDV observations having non-VLBA stations e-transferred.
- Direct transfers have worked, bypassing need for some stations to go through Haystack. Efforts continuing to add more stations.
- Bonn has successfully pulled GMVA data from DSOC.
- Anthony Sowinski PoC for NRAO.
- Data ingest of IVS data to NRAO/Socorro in support of USNO V2 correlator ("backup correlator") continuity of operations (COOP):
- We will do a monthly COOP test of the V2 correlator by correlating a WZ-KK intensive. .
- October 10 test - data from WZ could not be retrieved.
- USNO found that WZ had changed thelocation where their data gets placed, but WZ didn't notify anyone of the change. USNO will pass along the new location to NRAO.
- MOA actually was that the VLBA correlator would be used for the COOP not the V2. Once eTransfer is in place that could be done.
- e-transfer is in use now, time to revisit this topic?
- Different DiFX versions are in use, some prep work would be necessary. A larger discussion is needed.
- Further use of the V2 correlator in the processing of P-series and NZ-series observations, possibly also types of observations too, has been proposed.
- Test correlations have been done on both P and Z series observations with the V2 correlator but ran into problems with data storage and download speeds.
- Would need to use e-transfer.
- existing storage space in the DSOC datacopy areas is insufficient.
- Procedures to use e-transfer will need to be developed first.
- USNO will need access to etc/etd software on NRAO systems.
- etc/etd will be the only transfer software initially supported by e-transfer.
- Note that it will be a lot of work to get Kokee upgraded to make the switch to etc/etd for COOP test. (Wetzell is already using it.)
- Using e-transfer would get the current procedure off of gygax which isn't really suitable for it.
- An eventual goal would be to see a demonstration of these being processed without the involvement of any DC-based USNO personnel.
- USNO would like to be able to check automatically if the V2 correlator is up or down but are unable to do so.
- Will probably need to be handled via the e-transfer system due to internet security concerns.
- If we switch to using VLBA correlator instead of V2 as per the MOA it would eliminate the need for this.
- This is now a pretty low priority due to the nearing end of the V2 support contract.
- Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
- E-transfer not practical at this time so expedited shipping of disk packs is needed.
- Now that e-transfer is working, should be consider trying it with UD011?
- Under consideration, would require some coordination around observing and bandwidth concerns especially with the slower stations like SC.
- Might be limited to just attempting it for one of the two UD011 observations per month.
- Record of time to correlation:
- first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
- second year of observations (UD011 AS - BQ) average time was 12 days counting all observations, just counting the one observation a month we put a rush on the disk pack shipping the average time was 9 days.
- third year of observations (UD011 BR - CO) average time was 15 days counting all observations, just counting the one per month with rush shipping the average was 13 days.
- fourth year: UD011 CP - 12 days, CQ - 15 days, CR - observed October 6
- A VNDA test with UG006 was done in August. There was a holdup in getting the VNDA stations to record RDBE and VNDA data simultaneously. USNO will check on fringing but will not do any correlation.
- USNO saw some differences in phase cal between the two formats.
- Nothing more on this topic for this meeting.
- Next DiFX release (2.9) - released!
- One issue involving VGOS data in HOPS package still being worked on, a workaround is in place. A new version of HOPS is expected to resolve the issue, still in development.
- VEX2
- Vex2 standard will cover missing tuning information that SKED and VieSked++ aren't able to include.
- This information will still be required but VEX2 will have a space for it so manual editing won't be needed.
- Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
- USNO has been sent vex2 samples, will have them send NRAO a file to test and make sure it works.
- NRAO has received samples, no test file yet.
- NRAO is using VEX2 now.
- Sched 12 is out, NRAO has switched to using vex2 internally.
- Vex 1.5 will remain supported but once VNDA is installed and running in 2026 NRAO will probably end supporting v1.5 after a year.
- Should start a test phase on USNO-provided vex files with vex2 now rather than later.
- Problems reported by USNO with fourfit and vex2, there's a workaround.
- vex2 not supported by the mark4/HOPS data path? This and fourfit might be the same issue?
- for the immediate future the vex 1.5 files can be used to avoid the fourfit and mark4 issues.
- Installation of new GPS receivers at VLBA sites.
- Receivers have been installed at all sites but are not in use yet.
- All stations have been seen to transmit data but it's not usable by NRAO yet. LA has some differences being looked into.
- Also waiting on firmware updates to address using them for timing purposes.
- Done, but not all stations are working identically so some issues are still being tracked down.
- Mixed mode correlation
- 2025 RDV sessions will have at least one VGOS station in mixed mode, NRAO needs assistance in correlating them. Maybe contact Haystack or Bonn?
- Have tried mixed-mode correlation starting with RV169 but no success as of yet.
- Data analysts report some progress but still working on it, have contacted USNO for input.
- VLBA planned to start upgrading its correlator from RHEL7 to RHEL8 in January 2025.
- USNO still needs to use DiFX 2.7.
- Upgrade progress - delayed by some hardware incompatibilities and then finding that DiFX 2.9 is needed to support two expansion chassis' per Mark6 playback unit.
- Now that DiFX 2.9 is out work can start again.
- Work has progressed in fits and starts over the past month but we expect to have all of the Mark6 playback units upgraded by next week which will then allow for work to start on the correlator nodes.
- Starlink RFI mitigation and avoiding damage to receivers and equipment at Kokee.
- Does NRAO have a "chockblock", a physical block that goes over a receiver to protect it? No. Kokee is concerned with damage from the strong RFI and is considering it.
- ODS (Operational Data Sharing) on NRAO's side, tells Starlink where we're looking and satellites are not supposed to transmit when passing over antennas.
- Direct-to-cell is a major concern, especially since Starlink's claims concerning it and RFI protection are not realistic.
- Nothing more for this meeting on this topic,
- USNO asked about getting CASA to read pulse cal data the way AIPS does, it would help with their imaging efforts on weak astrometry targets at higher frequencies.
- A CASA developer has been assigned and work has started on it.
- No update.
-
Long term issues
- USNO reports detecting a drift in MK baselines beginning mid-2023 and still happening.
- Approximately 0.1 microsecond per day drift, works out to about 0.25 cm/year.
- It's seen in the NZ series but not P-series, so maybe a correlation or analysis issue rather than with MK itself.
- Realtime correlation of intensives
- Fast network connections at HN and MK make it theoretically possible but neither NRAO nor USNO are set up at the correlator end to do it yet.
- NRAO could attempt this but there will be some limitations - VLBA stations only, limited to observations that record at 512 Mbits/sec or less and NRAO would need help from USNO in determining sessions, creating files and looking at the output.
- An additional consideration is that while these observations would still record to Mark6 disk packs, they would not record data to the DBS units so would not be easily downloadable should the realtime correlation not work.
- USNO would have to wait for the v3 correlator before they could start planning for realtime correlation.
- New IVS session code naming structure change for intensives.
- Problems encountered with the longer project codes:
- Longer project codes not supported by all IVS stations so not all intensives will change to a longer code.
- DiFX did have some project code length limitations, a change went into DiFX 2.8 .
- USNO not using DiFX 2.8 yet. They can correlate with 2.8 but fringing doesn't work.
- NRAO's Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes. This would only affect any intensives which get correlated in Socorro off of disk packs, but currently there are none.
- Think this can be dropped from the agenda, if necessary can be brought back should developments warrant it.
- .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs?
- Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
- Possibility of adding KK to an RDV observation instead raised..
- KK will be in the April 2024 RDV observation so could get data from that.
- RV165 observed April 24. Has been correlated.
- After USNO (or NASA?) does analysis a copy of the clock offsets can be provided to VLBA for comparisons.
- Analysis finished, but didn't include VLBA stations so still need to do that and get the info to NRAO.
- Kokee does have a different maser compared to last time VLBA correlated with them, might have an effect?
- VLBA thinks it will have to wait until after all ten stations have received new GPS units (and they're all in use).
- Prioritization of this and other lingering issues should be considered once we can start looking at this again. Maybe mid to late 2026?
- Moving to VNDA will also change VLBA delays slightly, so it'll probably have to wait until after that's in place as well.
- The RV165 clock offset data would be useful for a before and after comparison, but it would not be used to adjust the VLBA offset. A second observation like RV165 would be needed so we could make the comparison.
- Fiber Status at each site:
- 600 Mbps minimum at each site
- 1 Gbps at PT, KP, OV, NL. FD
- 5 Gbps at HN
- 10 Gbps at LA,BR, MK
- 600 Mbps at SC (download up to 400 Mbps) - possibility it could be going up to 1 Gbps
- The practical link capacity due to overhead and other traffic is at most 85% of the reported link speed
Next meeting November 14.
--
JimOgle - 2025-10-07