Date: August 13, 2020

Time: 11:00 am Mountain, 1:00 pm Eastern
Phone: 434-817-6468 ext 228111# (Polycom is working again)


NRAO: Walter, Justin
USNO: Chris, Nicole, Megan, Lucas, Phil, Dave, Bruce, Stuart, Andy, Phillip, Kamyar, Matt
  • VLBA Observations
    • Daily Observation Report (N-series)
      • Three reobservations in July, two because PT was flagged as bad and one because LA which was substituting for PT was flagged as bad. The PT ones were probably RFI, LA it was more likely a real issue.
      • LA had to substitute for PT July 19 - July 24 due to HVAC issues on the PT antenna forcing its shutdown.
      • One LA observation did not get copied to SSD. It was later retrieved off of Mark6 disk pack and made available to USNO for download from Socorro.
      • Some observations had cable cal problems at PT, cause unknown.
      • Changed to use the same setups as the F-series observations on August 1st. Duration also dropped from 90 to 60 minutes and window for observing was changed to be more in line with some IVS observations.
      • Since the change to the F-series setups PT appears to be having more problems with RFI, the checkut1 script has flagged almost every run as possibly bad. A newer diagnostic tool has indicated fringes though.
      • The new window for observing may cause issues by interrupting maintenance day work at PT. The window is early enough on MK time that it will probably not get in the way of their work on maint days.
      • USNO soon to have full use of increased bandwidth into correlator, USNO considering adding SC to daily intensive when the incoming bandwidth is fully available.
        • No operational issues moving to this stance, but do need to consider the impact on the interruption of Open Skies observers. No concerns for the Washington Correlator.
        • Potential future multi-station (~5 stations) intensives would probably not preempt other observations, and would therefore require a larger observing window than is currently present on the daily intensives.
    • Other Intensive Observations
      • O-series test on HN-OV baseline
        • Will run simultaneous with the N-series observations but will not interrupt other observing.
        • Changed to use the same setups as the F-series observations on August 1st. Duration also dropped from 90 to 60 minutes and window for observing was changed to be more in line with some IVS observations.
        • Possible oscillation/tracking issue at HN recently detected, suspect an elevation encoder issue.
          • Only seen at low temperatures, hasn't been noticed since January.
        • Possibly expand in the future to include another baseline (BR-SC?).
      • F-series characterization series on six of the VLBA stations
        • Window for observing changed on August 1st.
        • For the past several months we had been including BR in the F-series sessions, that has been ended.
      • P-series intensive observations roughly every other day on the HN-MK baseline
        • Project code for these observations will be PYYDOY.
        • Using the F-series setups but duration is 1.5 hours.
      • "Holiday" intensive observations at MK
        • Will be substituting MK for Kokee on federal holidays in 2020 such as 7/3/20.
          • Next session will be Labor Day, 9/7/2020
        • Project code for these observations will be IYYDOY.
      • MK-WZ backup observation series
        • Project code for this observation will be QYYDOY.
        • Involves a new post-observing procedure for operators to copy the data to correlator in Japan.
        • Was restarted in late June due to the Ishioka station being unavailable, continuing into September.
          • Dates through the end of the year have been submitted through the pre-monthly request.
        • Six of the observations in July were submitted late so were charged double time.
    • Post-Monthly Report Review
      • No backlog from June that needed to be caught up in July.
      • Sowders replacement observation, UD010 series, requested 24 hours rather than its usual 8 hours. it ended up using 24.96 hours. This was the last month for the UD010 series.
      • A new observation of similar type to the Sowders replacement (but 6cm instead of 13cm) had requested 16 hours a month since April but has used none of it. For July the request actually said zero hours, which is what it used.
      • August request asks that the UD009 observation be scheduled back-to-back with the last UG003 session.
    • Pre-monthly request
      • September request asks like the August request that the UD009 K-band observation be scheduled back-to-back with a new 24-hour S/X observation (led by Lucas Hunt, which is effectively a continuation of the UG003 series).
  • Open issues
    • USNO emails to ut1 list are not being forwarded along to other USNO people.
      • Mail is being blocked at the Dept. of Defense level. A possible fix is being tested.
    • HN-OV baseline is more precise than any of the others USNO has data for, not sure why. Plus with all the recent problems showing up at PT maybe when fiber available to HN switch the N-series to them and PT-MK to the O-series?
      • HN fiber work has finally begun, expected to be finished in early September.
        • HN fiber speed expected to be 2.5 Gbps (maybe limited to 1 Gbps by NRAO hardware initially).
    • LA fiber may be at 10 Gbps when it's installed; this might impact consideration towards future use of it in baselines.
      • Work has not yet started to our knowledge, expected to be done by late September though.
    • USNO needs to renew support contract for MK fiber.
      • This contract is being moved into the larger support contract (see below), will need to coordinate on handoff.
      • Will end up doing a short support contract of a couple months and then a more regular one to start after that.
      • Contract now at NAVSUP and they are combining the contracts.
    • Networking Support Contract: Contract will be renewed.
      • Working on renewal, might involving changing some dates on it to avoid bunching all USNO contracts up at the same time of year. VLBA might all be bundled together at a different date. Moving forward.
    • Correlator Refresh Contract
      • Lustre filesystem and second set of nodes were assembled and tested in Socorro in June and then disassembled for shipping to USNO.
      • The new "BeeGFS" file system has been assembled and configured and tested in Socorro.
      • The second set of nodes cannot be installed until USNO is off of their old correlator first, waiting for confirmation from USNO that they have done so.
        • Have started parallel testing of old and new correlator. Seeing some differences, tracking down what might be causing them.
      • Install was on hold due to COVID-19 work and travel restrictions, a remote installation will most likely be done instead.
        • The old Lustre filesystem can't be updated and is viewed as a security concern so there's pressure to get this done.
        • "BeeGFS" file system hardware arrived and has been installed. Installation of second set of nodes not complete yet.
        • Old correlator and Lustre nodes about ready to be shut down, been running on new stuff for about a week, waiting on some networking setups to get finalized for final switchover.
      • Washington Correlator is now operating on the refreshed compute hardware and the BeeGFS system.
      • Extension through 11/30 for the hardware refresh has been approved.
    • A follow-on to the correlator support contract is being worked on. Walter will coordinate with USNO for NRAO's side. Has been resubmitted and is moving forward.
      • NAVSUP is actively looking at it.
    • USNO is moving to new ingest data server and IP address is not working even though it's on NRAO's whitelist. USNO reports attempts to transfer data to the .179 address does not work. Looks like problem might be firewalls at USNO end.
      • Waiting on hardware and software updates, ETA end of July.
      • Still waiting on the firewall. Updated ETA is end of August, early September.
    • Transition to Mark6 recording system at VLBA is done, including 4 Gbps capability.
      • Tests for transitioning to the higher data rate, perhaps with C-band, will be considered. (Though C-band has ionosphere problems.)
        • Mark Claussen will provide some 4 Gbps setups for USNO to look at for the tests. Hasn't happened yet.
      • Might be some USNO funding for more Mark6 disk drives/packs. Currently VLBA has sufficient supply of Mark6 disk packs though as 4 Gbps ramps up having more packs will be helpful.
        • USNO has sent twenty older disk packs of 32 TB capacity to VLBA for use. These packs have been reconditioned and are being sent out to sites. One pack experienced a board failure and cannot be used.
    • New IVS session code naming structure (long-term issue)
      • There is likely to be a change to format of the IVS "session code", but maybe not until 2021. As the USNO works to make the VLBA intensives look like IVS sessions we want to discuss what impacts such a change might have on this effort. This particularly relates to the 'exper_name' in the .vex files.
      • Intensives are all now using the xYYDOY naming format (ie, N20001, O20001, etc)
      • New IVS codes will be similar to existing codes but perhaps slightly longer, for example "NYYYYDOYNN". USNO & Goddard have discussion about it, length will be more than six characters. NRAO has already used some seven character length codes (such as "N20014B") without it causing any problems.
        • Scheduling OMS program was tested, it will support up to at least ten characters for projects.
        • The Mark6 disk pack shipping/tracking program Packtrack has some problems with lengthy project codes but there are workarounds. This would primarily only affect the O, P and F-series observations which get correlated in Socorro.
    • No CONT2020 session this year. A proposed "mini-CONT" fell through and will not happen either.
      • There's interest in having VLBA observe along with a VGOS session in "mixed-mode". But there's apparently some support problems with latest version of 'DifX' correlator software and VGOS, more technical discussions need to be held to determine what could be done. Some discussion has happened, may not be as big of a problem as initially thought.
      • No new information. It may come up in an IVS OPC which is scheduled for soon.
    • Peculiar delays (long term issue) - VLBA has a delay compared to Kokee that perhaps should be checked out. Maybe put some VLBA stations into a KK observation as tagalongs so comparisons can be made. Walter reached out to some people, says it looks like it would be a bigger effort than anticipated and may have to wait until things return to normal.
    • Next meeting September 10?

-- JimOgle - 2020-08-05
Topic revision: r4 - 2020-08-13, ChristopherDieck
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback