Date: June 10, 2021
Time: 11:00 am Mountain, 1:00 pm Eastern


Microsoft Teams: Click here to join the meeting

Dial in backup method: Phone: +1 571-388-3904
Conference ID: 168 008 659#


NRAO: Jim, Justin,
USNO: Chris, Bruce, Matthew, David, Lucas, Nicole, Phillip, Andrew, Megan,
  • Daily Observation Report (P-series)
    • New P-series daily intensive began May 1, replacing the old N-series. It runs concurrent with the IVS daily intensives.
    • USNO has been having occasional download speed problems with MK.
      • Seems to have stopped.
    • Missing scans - USNO reported problems with scans missing in the Xcube data, which NRAO found do get recorded to the Mark6.
      • Missing scans were seen to happen at the three stations of the old N-series and so far with HN in the new P-series.
        • At least one case of missing scans at HN happened on both the Xcube and Mark6, this turned out to be due a problem in the RDBE software.
        • A cause for what's causing scans to be written to the Mark6 but not to the Xcubes has not been determined yet.
          • An attempt to fix it was made by increasing gap time between scans in case the problem was that the Xcube could not keep up but it did not stop the problem.
          • A couple 'knobs' were twiddled in vieSched++ to try and reduce some extraneous recording, also did not have an effect on the problem.
          • There were fewer dropouts in the later part of May so perhaps the tweaks are helping after all?
      • Xcube replacement plan is underway, they have been ordered. Target deployment is still for June, July seems more likely.
    • USNO report on May observations has a lot of cable cal issues noted in both the P and Z series.
      • BR notably bad. It has been seen in non-USNO observing too.
      • Had been noticed last fall but then seemed to go away before coming back recently.
  • Other Intensive Observations
    • The O-series and F-series intensives ended April 30 along with the N-series.
    • New five station Z-series began running May 1 at roughly every other day interval.
    • I-series "Holiday" intensive observations at MK
      • MK will substitute for KK on federal holidays.
    • Q-series MK-WZ backup observation series
      • Was restarted in late June 2020 due to the Ishioka station being unavailable, last observation was May 16, 2021.
  • Post-Monthly Report Review for May 2021
    • UD012/US005 project requested 16 hours in May, only used 7.87 hours (as project UD013).
    • UC002 only got two of its three observations done in April, 14 hours were carried over into May as backlog. Both were observed in early May.
    • Three P-series observations did not run due to warm receivers.
    • Sixteen Z-series were requested for May but only 14 managed to be scheduled.
  • Pre-monthly request for July 2021
    • Nothing out of the ordinary seen in the request.
    • Revision will be sent soon, change in the back-to-back request.

  • Open issues
    • USNO will not be using Microsoft Teams after our June meeting so an alternative will need to be found.
      • Web-based/telephone-call Zoom?
      • Test out Zoom beforehand - Jim will arrange.
    • NRAO is starting to do more data downloads during and after observations and this could conflict with USNO's own data downloads of the intensives.
      • Currently NRAO is doing it mostly with a once a week test but it will eventually ramp up.
      • In an effort to coordinate download times between the two groups a small group to discuss it will be formed, probably two people from each side.
        • Any progress on setting up this group? Bruce and Matt on USNO end? Nothing has been started yet.
    • A minor 'DiFX' release went out in October (2.6.2) and will be installed on the VLBA correlator. USNO is currently using 2.6.1 on their correlator(s), waiting on internal approval to move to the new version.
      • Has been installed.
    • USNO emails to ut1 list are not being forwarded along to other USNO people.
      • Mail is being blocked at the Dept. of Defense level. A possible fix is being tested. Move to Long Term.
      • Bruce Thornton reported in April that he has not been receiving any emails from the ut1 list. Fixed.
    • USNO-NRAO contracts:
      • USNO renewal of MK fiber support contract currently at NAVSUP and being combined with other support contracts.
        • Follow-on contract for six months started December 2020.
        • Might be moved into operational costs/general support.
      • Networking support contract being worked on, might involve changing some dates to spread out USNO contract renewals over different times of the year.
        • Six month extension started December 2020.
      • A follow-on to the correlator support contact is in place for six months, started September 2020.
        • Contract ended March 22.Extension was granted and we've transitioned to the new contract through the end of 2021.
      • Interest in merging the support contracts into the interagency agreement.
        • USNO will bring this to NSF.
        • All contracts have been awareded, good through December 2020.
    • Data ingest of IVS data to NRAO/Socorro in support of USNO backup correlator operations:
      • We will do a monthly test of the backup correlator by correlating a WZ-KK intensive. Nominally scheduled for the first Wednesday of the month.
        • May test was supposed to be on May 5 but no one remembered to do it so it was delayed to May 12.
        • June test happened on schedule, June 2.
        • Reminders will start being sent the day before in July.
      • Data copying will need to be coordinated, to be handled by VLBA operations.
        • April 7 test - an initial glitch downloading WZ, operator error, otherwise went okay.
        • May 12 test - data transfer from WZ kept failing.
        • June 2 test - data transfer went okay from both stations.
      • Test correlator is only correlating on one node instead of four, that needs to be figured out.
        • ACTION: still an issue, NRAO to investigate. Update? Will check with Mark Wainright.
          • Mark had not heard of any issue with the correlator not running on all of its nodes.
          • Bruce Thornton was contacted by Mark for details but possible miscommunication of problem, Bruce was asked about "access issues" to which he reported none.
          • Jim Jacobs had been handling it, Mark may not have been informed.
        • Fringer and correlator can't download data directly, has to be run through gygax first.
        • Synching files also has to be side-loaded through gygax which further complicates process.
        • The ftps download of files through gygax (above two bullet points) stopped working this week.
          • ACTION Something changed on the NRAO side, USNO will contact NRAO helpdesk. Email to helpdesk didn't seem to go through, USNO will resend.
      • A bigger test, hopefully quarterly, of a 24-hour R4 observation is desired. NRAO would like a requirement document to figure out what needs to be done.
        • ACTION: USNO to provide the requirements in two groups. One with the disk space details, and a second with station connection details.
          • Disk space requirements have been received by NRAO.
        • ACTION: NRAO acquire additional storage capacity in support of requirement.
          • Waiting on funding.
    • NRAO data analysts report that the archive group is encountering problems dealing with UD009 data in the old archive and suggest that we update the project code.
      • The issue is related to how the archive system deals with making data public. NRAO policy is that data remains locked until one year after the last observation of a project but this starts to break down with multi-year projects. The software stops seeing new epochs as part of the same project and it stops resetting the proprietary period on the project so it unlocks all data more than one year old, forcing the archive group to manually intervene and re-lock ongoing projects.
      • We did this once before for this project, UD009 started as UD001 and the switch was made when we were about to run out of single letter epoch codes, after UD001 X we went to UD009 A. However, instead of doing this for UD009 we went from UD009 Z to UD009 AA. UD009 AH is queued for June, it or July's run could become UD010 A instead.
      • Switch with July.Double check on next available code.
  • Long term issues

    • Possible issue raised with accessing of USNO data in the VLBA legacy archive. Any way to put in a request people who do access the data to properly credit USNO and limit access?
      • Probably not much can be done with legacy archive, new archive system will have more potential. Currently new data is going into both, sometime soon it will switch to only going into the new archive.
        • Requirements have been set, still several months at least until we switch to just the new archive.
    • There's interest in having VLBA observe along with a VGOS session in "mixed-mode".
      • Heard from IVS OPC in November. First they need to move some correlation off to other correlators and then they can start considering adding more stations like VLBA to the VGOS observations. Possibly start looking into it mid 2021.
      • Probably will come up in a mid-June IVS meeting.
    • New IVS session code naming structure
      • There is likely to be a change to format of the IVS "session code", but maybe not until later in 2021 or 2022.
      • New IVS codes will be similar to existing codes but perhaps slightly longer, for example "NYYYYDOYNN". USNO & Goddard had a discussion about it, length will be more than six characters. NRAO has already used some seven character length codes (such as "N20014B") without it causing any problems.
      • Scheduling OMS program was tested, it will support up to at least ten characters for projects.
      • The Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would only affect any intensives which get correlated in Socorro, something that will end when the switch to the five-station intensive observation eliminates the intensives that are correlated by NRAO instead of USNO.
      • Walter Brisken suggested running an end to end test to make sure all of the software can handle the longer names.
        • NRAO can probably do an end to end test with one of the old intensives such as the F-series, it will take more software work on USNO side depending on what IVS comes up with.
      • Was discussed in the March IVS workshop, there is renewed interest in doing this. Possibly some action on it this year.
    • Peculiar delays - VLBA has a delay compared to Kokee that perhaps should be checked out, maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated and may have to wait until things return to normal after COVID.
    • 'DiFX' Workshops are happening. Walter talked about real-time VLBI Feb 18.
      • Probably will be another one later this year.

  • Fiber Status at each site:
    • 200 Mbps minimum at each site with 190 Mbps the practical limit due to required overhead.
    • 1 Gbps at PT,KP,MK
    • 2.5 Gbps at HN
    • 10 Gbps at LA
Next meeting July 8, 2021

-- JimOgle - 2021-06-03
Topic revision: r4 - 2021-06-10, JimOgle
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback