Date: January 10, 2025
Time: 10am Mountain, noon Eastern
 Meeting will be held over Zoom. Details in email.

NRAO: Jim Ogle, Walter Brisken, Heather Marshall, Justin Linford, Steven Tremblay
USNO: Chris Dilullo, Chris Dieck, David Gordon, Phil Cigan, Phillip Haftings

  • Station Update - nine stations operational. KP experienced a commercial power failure Thursday evening and the site generator failed so many systems crashed. Power has been restored, site should be operational again over the weekend after receivers have cooled down.

  • Intensive Observations
    • P-series has changed to fixed times/dates while Kokee is down and MK is substituting for it in some of the I-series intensives.
      • Continues into February.
      • New longer project code PYYYYDDD in use.
      • P2025004 did not record to DBS, software update for longer codes apparently did not carry through.
      • P2025005 did not run, human error.
    • Z-Series runs approximately every three days.
      • BR cable/phase cal problem
        • Maintenance visit in late August 2023 at first seemed to have fixed the cable cal problem at BR, there was only one observation at the end of August and one in October that still had jumps in the cable cal but every other observation in September and October has looked okay. However, in November the cable cal problem began appearing again.
        • USNO reports that BR cable cal was less of an issue in March and April 2024.
          • VLBA has noticed an increase in cable cal jumps since then. They appear to occur around the same times each day, 2000 and 0800 UT. Site tech speculates late afternoon heat and overnight cooling is causing expansion and contraction of cables.
          • Seemed to lessen a bit towards the end of summer but colder temperatures seem to have fired it back up again.
      • SC began having a cable cal problem in June 2023. Still being investigated.
      • New longer project code NZYYYYDDD in use.
    • Q-series intensive observations at MK run on weekends.
    • I-series "Holiday" intensive observations at MK, substituting for Kokee on federal holidays.
      • Also substituting for KK in some regular I-series intensives while their fiber is out.
      • No ETA on restoration of KK fiber, replacement is in the early site survey phase.
  • Post-Monthly Report Review for December 2024
    • MK substituting for KK I-series and new schedule for P-series observations continues.
    • COOP test used MK instead of KK.
      • December COOP test was rescheduled from December 4 to December 5 to match when MK was running an I-series intensive
    • .Fourteen I-series observations were initially requested but then due to maintenance work at some of the non-VLBA stations two of these were rescheduled and two were canceled.
    • Eighteen P-series observations were requested, nineteen ran.
    • UF006 requested 24 hours and used 22.27 hours.
    • Two Q-series observations did not run due to problems getting them modified to work on VLBA stations.
    • A 24-hour observation from the January 2025 request, UG006A, technically ran on December 31 by starting at 22:39 UT. It was decided to not include it on the December report.
  • Request for January 2025
    • .MK substituting for KK I-series and new schedule for P-series observations continues.
    • UD019 incrementing to UD020.
    • UG005 incrementing to UG006.
    • UF006 incrementing to UF007.
    • UC003 not incrementing.
    • COOP test scheduled for January 23, 2025
    • P-series and Z-series intensives changing to longer project codes, PYYYYDDDD and NZYYYYDDD respectively. Other intensives not changing.
  • Pre-Monthly Request for February 2025
    • MK substituting for KK I-series and new schedule for P-series observations continues.

  • Open issues
    • VLBA e-transfer system
      • Initial testing possibly next week with transfer of one non-VLBA station's RDV169 data from Haystack.
    • New IVS session code naming structure change for intensives.
      • Problems encountered with the longer project codes:
        • Longer project codes not supported by all IVS stations so not all intensives will change to a longer code.
        • DiFX did have some project code length limitations, a change went into DiFX 2.8 .
          • USNO not using DiFX 2.8 yet. They can correlate with 2.8 but fringing doesn't work.
        • Xcube/DBS recording software looks at project codes to determine to record or not, will need to be addressed.
          • DBS soffware updated to catch new NZ intensive code.
          • It was thought that the code was updated to catch the new longer P-series intensive code but it failed to capture P2025004.
          • Should be fixed before P2025007.
        • NRAO's Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes. This would only affect any intensives which get correlated in Socorro off of disk packs, but currently there are none.
        • move to long term
    • Data ingest of IVS data to NRAO/Socorro in support of USNO V2 correlator ("backup correlator") continuity of operations (COOP):
      • We will do a monthly COOP test of the V2 correlator by correlating a WZ-KK intensive. .
        • COOP test is using WZ-MK while KK's fiber connection is out.
      • MOA actually was that the VLBA correlator would be used for the COOP not the V2. Once eTransfer is in place that could be done.
      • Further use of the V2 correlator in the processing of P-series and Z-series observations has been proposed.
        • Or would this be further use of the VLBA correlator per the MOA?
        • An eventual goal would be to see a demonstration of these being processed without the involvement of any DC-based USNO personnel.
        • Another goal would be processing bigger projects such as a 24-hour R4 observation.
        • Will be easier for VLBA to handle after the deployment of the e-transfer filesystem.
          • etc/etd will be the only transfer software initially supported by the e-transfer system.
          • Kokee will need to change to this software for the COOP tests. Wetzell has already switched to it..
        • Test correlations have been done on both P and Z series observations with the V2 correlator.
          • Storage space in the DSOC /datacopy areas is insufficient to hold data from a P series and a Z series at the same time.
            • More space will be available once e-transfer is in place.
          • Data transfer rates were exceptionally slow, probably due to using rsync because tsunami and etc/etd were not available to USNO users on gygax.
            • Will need to get USNO access to the new transfer software before further testing can be done.
            • Will have to wait for e-transfer to be in place.
    • USNO would like to be able to check automatically if the V2 correlator is up or down but are unable to do so, can NRAO set something up for this?
      • Probably have to wait for e-transfer to be operational at NRAO due to internet security concerns.
    • V2 correlator status:
      • Tuesday 5-7pm (Eastern Time) Maintenance Window weekly patches continue, with weekly reporting to USNO at the conclusion of each patch.
      • Troubleshooting as requested to maintain functionality
      • USNO and NRAO completed the RMF Validation step in December, awaiting disposition eta late Jan.
    • V3 correlator status:
      • NRAO is continuing progress toward milestone 3, software testing in Socorro on a two-processing host system using a representative dataset.
      • BeeGFS volumes have been prepared on test hardware
      • VM established for deployment Tenable Nessus security scanning, installation and configuration underway
      • USNO has advised of change in information security requirements for its IT infrastructure and systems, and is preparing flowdown requirements for the V3 Correlator
    • Replacements of the Xcubes with DBS units
      • New software will ultimately cause a change in the naming structure of data files and PFB (Mark5B) data format will no longer be supported, instead it will be converted on the fly to VDIF format.
        • Software changes will need further testing before deployment and first iteration of the updates may not change naming structure of data files or include the PFB->VDIF conversion.
        • Low priority, won't happen until after eTransfer and v3 correlator. Maybe March 2025 at the earliest.
    • Is it possible to reduce the latency on 24-hr geodetic USNO-only sessions, with a goal of a latency between the completion of observation to analyzed result under 7 days?
      • E-transfer not practical at this time so expedited shipping of disk packs is needed.
      • Might be limited to just attempting it for one of the two UD011 observations per month.
      • Record of time to correlation:
        • first year of observations (UD011 AA - AR) average time was 9 days. (Specifics can be found in minutes of previous meetings.)
        • second year of observations (UD011 AS - BQ) average time was 12 days counting all observations, just counting the one observation a month we put a rush on the disk pack shipping the average time was 9 days.
        • third year: UD011 BR - 18 days, UD011 BS - 8 days, UD011 BT - 21 days, UD011 BU - 13 days, BV - 27 days; BW - 15 days; BX - 23 days; BY - 13 days; BZ - 20 days, CA - awaiting correlation.
          • note: VLBA correlator was experiencing a networking related slowdown September through November (BT through BX) which lengthened correlation times.
    • Next DiFX release (2.9) - Slow development continues.
      • testing is likely to start soon but there's a patch that needs to be put in concerning Mark4 and that hasn't happened yet.
      • there are also some issues with Mark6 units as well.
    • VEX2
      • Vex2 standard will cover missing tuning information that SKED and VieSked++ aren't able to include.
        • This information will still be required but VEX2 will have a space for it so manual editing won't be needed.
        • USNO does have a concern about whether comments can safely be ignored in VEX2, Walter says yes.
        • USNO has been sent vex2 samples, will have them send NRAO a file to test and make sure it works.
      • Things are okay right now with standard USNO setups and observations, could become an issue if USNO starts using non-standard setups in new observations.
      • Some testing of VEX2 has begun at NRAO.
        • Some diagnostic observations have switched to vex2 for testing purposes.
        • Seems to be okay for some regular observing, pointing and other modes still need some work.
        • Sched 12 is out, NRAO will switch to using vex2 internally after that. Vex 1.5 will remain supported but once VNDA is installed and running in 2026 NRAO will probably transition fully to vex2.
    • Installation of new GPS receivers at VLBA sites.
      • Antennas have been installed at all VLBA sites.
      • installation of first receiver was done at FD in November.
      • PT and OV have receivers installed but not turned on yet.
      • Two more stations scheduled to have receivers installed in January.
    • Mixed mode correlation (moved out of Long Term)
      • 2025 RDV sessions will have at least one VGOS station in mixed mode, USNO might need to assist NRAO in correlating them.
      • First 2025 RDV session is January 8, 2025.
    • VLBA will begin upgrading its correlator from RHEL7 to RHEL8 in January 2025. Need to know from USNO which version of DiFX is needed.
      • USNO still needs to use DiFX 2.7.

  • Long term issues

    • Higher speed network connections to HN and MK opens the possibility of realtime correlation of intensives.
      • Neither Washington correlator nor NRAO set up to do it yet.
        • Blocking issue for NRAO is mostly network speed into the correlator.
          • Initial attempts at realtime correlation of VLBA data could begin soon. There will be some limitations - VLBA stations only, limited to observations that record at 512 Mbits/sec or less and NRAO may need some help from USNO in determining sessions, creating files and looking at the output. These observations would still record to Mark6 disk packs but if USNO intensives were used they would not record data to the DBS units.
            • Contact person at USNO to assist on this needed.
            • move to open issues
        • USNO not really set up for realtime with correlator design and workflow, and no real plans to do it in the near future. Wait for v3 correlator.
    • .Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs?
      • Walter reached out to some people, says it looks like it would be a bigger effort than anticipated.
        • Possibility of adding KK to an RDV observation instead raised..
          • KK will be in the April 2024 RDV observation so could get data from that.
            • RV165 observed April 24. Has been correlated.
            • After USNO (or NASA?) does analysis a copy of the clock offsets can be provided to VLBA for comparisons.
              • Analysis finished, but didn't include VLBA stations so still need to do that and get the info to NRAO.
            • Kokee does have a different maser compared to last time VLBA correlated with them, might have an effect?
        • VLBA thinks it will have to wait until after all ten stations have received new GPS units (and they're all in use).
        • Moving to VNDA will also change VLBA delays slightly, so it'll probably have to wait until after that's in place as well.
          • The RV165 clock offset data would be useful for a before and after comparison, but it would not be used to adjust the VLBA offset. A second observation like RV165 would be needed so we could make the comparison.

  • Fiber Status at each site:
    • 600 Mbps minimum at each site
    • 1 Gbps at PT, KP, OV, NL. FD
    • 5 Gbps at HN
    • 10 Gbps at LA,BR, MK
    • 600 Mbps at SC (download up to 400 Mbps) - possibility it could be going up to 1 Gbps
  • The practical link capacity due to overhead and other traffic is at most 85% of the reported link speed
Next meeting February 14, 2025

-- JimOgle - 2024-12-06
Topic revision: r2 - 2025-01-10, JimOgle
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback