Date: December 12, 2025
Time: 10am Mountain, Noon Eastern. Meeting will be held over Zoom. Details in email.
NRAO: Jim Ogle, Heather Marshall, Thomas Coleman, Justin Linford, Ylva Pihlstrom, Seth Bruzewski
USNO: David Gordon, Phillip Haftings, Chris Dieck, Nathan Secrest, Kristina Nyland, Phil Cigan, Chris DiLullo
- Station Update - all ten stations operational, but BR is recently having a lot of difficulty with its subreflector rotation getting on position. MK weather station not communicating.
- Intensive Observations
- P-series runs Tuesdays and Thursdays around the time of I-series intensives, Saturdays and Sundays after Q-series intensives.
- NZ-Series runs approximately every three days.
- BR cable/phase cal problem. Comes and goes, leading theory is thermal expansion and contraction of cables.
- SC also has a cable cal problem but is slightly different in effect and doesn't seem to be due to thermal issues like at BR.
- Q-series intensive observations at MK run on weekends.
- I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
- Kokee will be down for some signal change and receiver upgrades, currently scheduled for March 2026. MK will substitute for it during that time.
- Post-Monthly Report Review for November 2025
- UD020H backlog from October ran on November 1.
- UF008 was taken out of the request, its four hours were given to UF007 taking that project up to 24 hours. UF007 used 21.18 hours.
- The maximum allotment hours for FY2026 was increased to 218 hours.
- Request for December 2025
- MK substituting for KK in I-series on December 24.
- P-series for December 25 will be scheduled early to avoid the NRAO holiday shutdown period.
- COOP test scheduled for December 5 was moved to December 2.
- Pre-Monthly Request for January 2026
- UD020 incrementing to UD021, UG006 incrementing to UG007.
- UG007 has two sessions instead of the usual one.
- No UF007 in January.
- Correlator Statuses
- V2 correlator status:
- Tuesday 5-7pm (Eastern Time) Maintenance Window weekly patches continue, with weekly reporting to USNO at the conclusion of each patch.
- Troubleshooting as requested to maintain functionality.
- USNO and NRAO received Authorization for the NGN/V2 network as of July 22, 2025.
- NRAO has addressed security vulnerabilities and findings as required.
- One drive was identified as faulted on 12/3; after replacement 5 spare storage drives remain available at USNO.
- NRAO has begun training a representative of USNO/N6 on ongoing V2 maintenance, scope for which USNO intends to internalize in CY26, on Monday Sept 15th. USNO/N6 representative was intended to take a lead role for production system patching starting last month, while NRAO will maintain the test system and provide support. This has not occurred, and in the past week USNO has nominated 3 additional staff for training on the V2 system.
- V3 correlator status:
- NRAO and USNO/NAVSUP have executed a contract mod to incorporate updated technical and process requirements
- NRAO delivered BOM and testing documents on 10/31 - milestone 4 complete
- NRAO has finalized reqs and POs for servers in the last few weeks, following completion of Dell's upgraded InfiniBand adapter testing
- Issues in DoD contract database are blocking processing of invoices. Urgency increasing, with remaining server POs placed as of today 12/12 - milestone 3 complete
- Open issues
- DBS software changes
- New software on the DBS units will ultimately cause a change in the naming structure of data files and PFB (Mark5B) data format will no longer be supported, instead it will be converted on the fly to VDIF format.
- Deployment of VNDA would make this change unnecessary?
- xcuberec program which records the data on the DBS will be replaced by sendit, NRAO could start running it for P-series in parallel with xcuberec for testing and comparison purposes if USNO would be able to do a double correlation. Sendit does change the file naming structure as mentioned above.
- Sendit testing was put on hold due to government shutdown.
- Shutdown has ended but Walter doesn't have a lot of time available through the end of the year so will hopefully start testing in January 2026.
- VLBA e-transfer system
- RDV observations having non-VLBA stations e-transferred.
- Anthony Sowinski PoC for NRAO.
- KK needs to get the etc/etd software running to utilize NRAO's e-transfer system (and the v3 correlator at USNO).
- KK does have etc/etd running but transfers to DSOC aren't working very well.
- KK is also having problems transferring to WACO.
- Data ingest of IVS data to NRAO/Socorro in support of USNO V2 correlator ("backup correlator") continuity of operations (COOP):
- We will do a monthly COOP test of the V2 correlator by correlating a WZ-KK intensive. .
- Next test December 5.
- COOP used December 8 due to a scheduled power outage at USNO.
- Problems downloading data from WZ in a timely manner so technically the COOP failed.
- Discussion will need to start on the future of COOP as V2/V3 correlator developments happen.
- MOA actually was that the VLBA correlator would be used for the COOP not the V2. Once eTransfer is in place that could be done.
- e-transfer is in use now, time to revisit this topic?
- Different DiFX versions are in use, some prep work would be necessary. A larger discussion is needed.
- Further use of the V2 correlator in the processing of P-series and NZ-series observations, possibly also types of observations too, has been proposed.
- Test correlations have been done on both P and Z series observations with the V2 correlator but ran into problems with data storage and download speeds.
- A test correlation was recently done with P2025336 (December 2) alongside the monthly COOP test.
- USNO did have to make some user environment changes on the v2 before they were able to correlate the P-series.
- Would need to use e-transfer if done on a regular basis.
- existing storage space in the DSOC datacopy areas is insufficient.
- Procedures to use e-transfer will need to be developed first.
- USNO will need access to etc/etd software on NRAO systems.
- Using e-transfer would get the current procedure off of gygax which isn't really suitable for it.
- An eventual goal would be to see a demonstration of these being processed without the involvement of any DC-based USNO personnel.
- USNO would like to be able to check automatically if the V2 correlator is up or down but are unable to do so.
- Will probably need to be handled via the e-transfer system due to internet security concerns.
- If we switch to using VLBA correlator instead of V2 as per the MOA it would eliminate the need for this.
- This is now a pretty low priority due to the nearing end of the V2 support contract but some discussion has started on it.
- Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
- E-transfer not practical at this time so expedited shipping of disk packs is needed.
- Now that e-transfer is working, should be consider trying it with UD011?
- Under consideration, would require some coordination around observing and bandwidth concerns especially with the slower stations like SC.
- Might be limited to just attempting it for one of the two UD011 observations per month.
- Record of time to correlation:
- first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
- second year of observations (UD011 AS - BQ) average time was 12 days counting all observations, just counting the one observation a month we put a rush on the disk pack shipping the average time was 9 days.
- third year of observations (UD011 BR - CO) average time was 15 days counting all observations, just counting the one per month with rush shipping the average was 13 days.
- fourth year: UD011 CP - 12 days, CQ - 15 days, CR - 13 days, CS - 13 days, CT - 29 days (problems with correlator updates at DSOC slowed things down), CU - 16 days, CV - observed December 8.
- Next DiFX release (2.9) - released!
- One issue involving VGOS data in HOPS package still being worked on, a workaround is in place. A new version of HOPS is expected to resolve the issue, still in development.
- VEX2
- Vex2 standard will cover missing tuning information that SKED and VieSked++ aren't able to include.
- This information will still be required but VEX2 will have a space for it so manual editing won't be needed.
- Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
- USNO has been sent vex2 samples, will have them send NRAO a file to test and make sure it works.
- NRAO has received samples, no test file yet.
- Question from Ed H at Goddard about status of vex2, and any shortcomings that might push an incremental VEX2 release? USNO has one thing they'll bring up.
- Some progression in IVS meetings, but issues remain to be addressed particularly with old files and if conversion will be necessary.
- NRAO is using VEX2 now.
- Sched 12 is out, NRAO has switched to using vex2 internally.
- Vex 1.5 will remain supported but once VNDA is installed and running in 2026 NRAO will probably end supporting v1.5 after a year.
- Should start a test phase on USNO-provided vex files with vex2 now rather than later.
- Problems reported by USNO with fourfit and vex2, there's a workaround.
- vex2 not supported by the mark4/HOPS data path? This and fourfit might be the same issue?
- for the immediate future the vex 1.5 files can be used to avoid the fourfit and mark4 issues.
- Installation of new GNSS/GPS receivers at VLBA sites.
- Receivers have been installed at all sites but are not in use yet.
- All stations have been seen to transmit data but it's not usable by NRAO yet.
- Also waiting on firmware updates to address using them for timing purposes.
- Done, but not all stations are working identically so some issues are still being tracked down.
- Nine of the ten have valid time offset code running. One more update before end of calendar year hope to have all ten running. Would like to have six months of running data from the sites before they declare it fully operational. VLBA would be able to start own analysis during that time.
- Mixed mode correlation
- 2025 RDV sessions will have at least one VGOS station in mixed mode, NRAO needs assistance in correlating them. Maybe contact Haystack or Bonn?
- Have tried mixed-mode correlation starting with RV169 but no success as of yet.
- Data analysts report some progress but still working on it, have contacted USNO for input.
- USNO asked about getting CASA to read pulse cal data the way AIPS does, it would help with their imaging efforts on weak astrometry targets at higher frequencies.
- A CASA developer has been assigned and work has started on it.
- Hoping to have a test version available in early December.
- No test version yet. Maybe in January?
- USNO asked about this week's changes at NRAO moving guest logins to a new server, they found they were still able to get in through the now NRAO-only login server as well as through the new guest server. USNO probably has a exception due to access needs, also anyone with an @nrao.edu account would retain access through the NRAO-only login server.
-
Long term issues
- USNO reports detecting a drift in MK baselines beginning mid-2023 and still happening.
- Approximately 0.1 microsecond per day drift, works out to about 0.25 cm/year.
- It's seen in the NZ series but not P-series, so maybe a correlation or analysis issue rather than with MK itself.
- Realtime correlation of intensives
- Fast network connections at HN and MK make it theoretically possible but neither NRAO nor USNO are set up at the correlator end to do it yet.
- NRAO could attempt this but there will be some limitations - VLBA stations only, limited to observations that record at 512 Mbits/sec or less and NRAO would need help from USNO in determining sessions, creating files and looking at the output.
- An additional consideration is that while these observations would still record to Mark6 disk packs, they would not record data to the DBS units so would not be easily downloadable should the realtime correlation not work.
- USNO would have to wait for the v3 correlator before they could start planning for realtime correlation.
- .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs?
- Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
- Possibility of adding KK to an RDV observation instead raised..
- KK will be in the April 2024 RDV observation so could get data from that.
- RV165 observed April 24. Has been correlated.
- After USNO (or NASA?) does analysis a copy of the clock offsets can be provided to VLBA for comparisons.
- Analysis finished, but didn't include VLBA stations so still need to do that and get the info to NRAO.
- Kokee does have a different maser compared to last time VLBA correlated with them, might have an effect?
- VLBA thinks it will have to wait until after all ten stations have received new GPS units (and they're all in use).
- Prioritization of this and other lingering issues should be considered once we can start looking at this again. Maybe mid to late 2026?
- Moving to VNDA will also change VLBA delays slightly, so it'll probably have to wait until after that's in place as well.
- The RV165 clock offset data would be useful for a before and after comparison, but it would not be used to adjust the VLBA offset. A second observation like RV165 would be needed so we could make the comparison.
- VNDA timescale looks to be lining up with the v3 correlator so we'll just wait for both of those to be completed.
- Fiber Status at each site:
- 600 Mbps minimum at each site
- 1 Gbps at PT, KP, OV, NL. FD
- 5 Gbps at HN
- 10 Gbps at LA,BR, MK
- 600 Mbps at SC (download up to 400 Mbps) - possibility it could be going up to 1 Gbps
- The practical link capacity due to overhead and other traffic is at most 85% of the reported link speed
Next meeting January 16, 2026.
--
JimOgle - 2025-12-12