ECSV Daily Log

2011

October

21 Oct (Weekend)

Hi folks --

Lots of testing on various fronts (3-bit, RSRO, holography, gain linearity).
Some progress even! but not closure alas.

There was an Executor update a few days ago, which should have fixed the
early-abort problems (i.e., you should now be able to abort scheduling blocks without
worrying that the correlator will far apart).  We believe this is working; please
call me if you find otherwise!

Old Notes (but still relevant)
------------------------------
- If the fringe display is not updating, please try d10 and if both are failing,
 it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; all other antennas can be used at all bands.

Correlator:
--
CM: 2011-09-29 20:31 UT
CBE: wcbe_20110921.1 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.06 (07 Oct)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 20Oct11
 All ok!

quad1 =
quad2 =
quad3 =
quad4 =


== Friday - Tuesday morning ==

OST (most tests have had their priorities updated except):

  Fri (tonight):
     1700-0030 LST: /home/mchost/evla/scripts/test/Gains/TCAL0004_sb5683956
     **please kill this when the antennas hit the elevation limit, probably
       around 11:30pm**
     0030-0630 LST: /home/mchost/mbrentje/holography/evla-scripts/Q/THOL0001_Q_41x41.evla
       (you can start this immediately after the TCAL0004 script)
       (the exact start time is not important, so if TCAL0004 runs short or
        long that's OK)

  Sat (tomorrow night):
    0030-0630 LST:
      /home/mchost/evla/scripts/opt/2011/10/10C-119_sb4788674_1.evla (array time)

Please run through 1000 LST *Tuesday* morning -- i.e., science all day Monday!

Joe and Ken are out this weekend, so please call me if there are any problems (gulp).

Have a good weekend --

             Michael

Callout: Baseline board communication problem (rebooted).
Callout: 10C-119 not fringing (CBE fell behind; restart; put project on hold).

The operator called this morning to say that the CM had been
complaining about b106-b-5 (= bb2033).  CRM log shows problems
with readback from this board; BlB GUI complains about being unable to
parse the XML returned from the CMIB (premature end of file) when
trying to refresh.  Ping says all is well.  To be on the safe side I added this board (Q3-10) to unavailableBlbPairs.txt, which will prevent its use by scripts run through the OST.  I also asked the oeprator to abort 11B-201.sb5652508.eb5685961.55856.54543798611 , mark it as failed, and re-start the script again through the OST (after clearCorrelator).

b106-b-5 remained configured (and sending frames) after the abort, so
there's clearly a problem. To avoid possible packet collisions I slogin'd
to the BlB and rebooted it as the easiest way to turn off the frames without access to the GUI.  The board is now responding to the BlB GUI
without any problem.  I'm still leaving it out of regular
observing, just in case.

         Michael

Sam called early Sunday morning to say that 10C-119 was experiencing
difficulties (that's 10C-119_sb4788674_1.55857.243612453705).  Indeed
there were several missing BDFs and the CBE was falling further and further
behind.  I collected a bunch of information for Martin, then:

1- Asked Sam to kill 10C-119, then run clearCorrelator

2- Ran wcbetool reset, followed by wcbetool up, to completely (I hope :)
 reset the CBE.
 : It looks like there were problems with two CBE pipeline processed
   since the end of the holography script Saturday morning.  The
   data from subsequent projects look fine (no missing BDFs for instance)
   but this seemed an opportune moment to re-start them anyhow, especially
   since the CBE was clearly unhappy in other ways by this time (presumably
   due to 10C-119).

3- Asked Sam to run C_quad12, then abort & clearCorrelator
 ...to check that all pipeline processes were working properly, for
 a fairly demanding (64 BlB) correlator configuration.

4- Told Sam to go back to OST for further science, and to mark 10C-119
 as *failed*.

So, more things to look at on Monday.  Joy.

For now, we should leave 10C-119 SBs on hold.  We could try letting
C-band-only SBs through but this is probably not the hour to make that
decision ;)

Cheers,

           Michael

20 Oct

Hi,

Lots of testing on various fronts (3-bit, RSRO, holography).
Holography is now cleared and will run tonight and some
over the coming days.

EA08 A/C has fibers swapped; please note in the
logs (might not be relevant for most since A/C seems
not to be reliably fringing):
"EA08 A/C has the cross and parallel hands swapped".

Note (this will go away with the next Executor update!:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out;
 - EA25 use for all.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 20Oct11
as above.

quad1 =
quad2 =
quad3 =
quad4 =


== Thursday  ==
OST (most tests have had their priorities updated except):

1710-1810 10C-145
1810-1910 10C-145

1910-0000 OST
0000-0630 Q /home/mchost/evla/scripts/opt/2011/10/THOL0001_Q_41x41.evla
0630-1030 OST

Stop Friday  morning at 1030 LST for testing.

== Friday - Tuesday morning ==
OST (most tests have had their priorities updated except):

  Fri: 1700-0030 *;
/home/mchost/evla/scripts/opt/2011/10/TCAL0004_sb5681974_1700LSTstart.evla
(*LST start fixed at 1700 day 62580)
  Sat: 0030-0640 L;
/home/mchost/evla/scripts/opt/2011/10/10C-119_sb4788674_1.evla (array
time)

Other possible holography runs pending view of tonight's Q band.

Joe

19 Oct

Hi,

More 3-bit churning; more info from Michael here.
EA23 is out (focus issues). EA25 is on W04.
EA08 A/C has fibers swapped; please note in the
logs:
"EA08 A/C has the cross and parallel hands swapped".

Note (this will go away with the next Executor update!:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA23 please leave out.
 - EA25 use for all.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 14Oct11
as above.

quad1 =  1,      4
quad2 =                                12
quad3 =
quad4 =


== Wednesday  ==
OST (all tests have had their priorities updated except):

OST following TCAL0002 until 2230 LST

syspt2hx.evla (run for 2 hours if possible (2230-0030 LST; manually stop).

Then I would like to try another go at the holography script. I'll
call you at this
time. If it goes well, then we'll run for 6 hours and then OST until 0900 LST.
Otherwise, I'll fix things up and turn things back to you for the OST.

Stop Thursday  morning at 0900 LST for testing.

Joe

18 Oct

Hi,

More 3-bit testing today. Solid sleuthing from Michael/Vivek in understanding
the impact of the testing on the 3-bit enabled antennas. More work on this
tomorrow but there are clear paths to return the system to fully operational
now.
More understanding of the consequences of the Ken-patch to the pointing
function. We'll not be returning to this. We can't yet reproduce the problem
but we'll run with the standard version for a bit to see how the BDF issues
repeat. Martin has a proto-fix that we'll try after further characterization.

Some strangeness following the holography testing; had to do a clearCorrelator
to recover the system (and then it promptly dropped a BDF bomb in the
archive area on lustre (labeled bad??). Michiel will follow up with the
relevant parties.

I'm running Rick's gain test now; after that, it's OST until 0700 LST.

Note (this will go away with the next Executor update!:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA25 use for all.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 14Oct11
as above.

quad1 =  1,      4
quad2 =                                12
quad3 =
quad4 =


== Tuesday  ==
OST (all tests have had their priorities updated):
Stop Wednesday morning at 0700 LST for maintenance day.

Joe

17 Oct

Hi,

3-bit testing today; high winds prevented quite a bit. EA25 is back and
so pointing with it for tonight.
One recurring issue is board 2019: b101-t-2 is still down; removed from
consideration.

Note:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA25 use for LSCXKu.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 14Oct11
as above.

quad1 =  1,      4
quad2 =                                12
quad3 =
quad4 =


== Monday  ==
OST (all tests have had their priorities updated except):

if the winds are good (X band), then please run the following for 2 hours
starting anywhere between 0000-0300 MDT.

/home/mchost/evla/scripts/operations/syspt2hx.evla (must be manually stopped).

Stop Tuesday morning at 0700 LST for maintenance day.

Joe

14 Oct (Weekend)

Hi,

Holography issues from last night have been diagnosed (script
problems); likely to run again soon.
EA25 is ready but has a fan problem. *If* it gets fixed please
include in LSCXKu only.
Ken noted a number of BaselineBoard issues resulting from
the recent change. The expectation was that we would incrementally
lost boards throughout the weekend. Ken/Bruce checked a roll-back
on Q3 and found it was no longer producing the tell-tale errors so
after consulting with Bruce, I decided to roll back the lot.
We retested and found some problems:
- SN:2019: b101-t-2 (Bruce looked and thinks there may be a hardware issue;
to be followed up on Monday; removed from consideration).
- SN200D, SN2021: b101-b-0, b104-b-2 - show low frame rates but Bruce
did not see this out of the board. I removed them anyway.

We're planning to run with the edited pointing script (despite having
no errors during the day).

Note:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA25 use for LSCXKu if fixed; please call
if it is ready as we should also do a pointing run.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 14Oct11
as above.

quad1 =  1,      4
quad2 =                                12
quad3 =
quad4 =


== Weekend  ==
OST (all tests have had their priorities updated except):

  1800-1900 C;
/home/mchost/evla/scripts/opt/2011/10/TRSR0043_sb5614240_1.evla (LST
start time)

Stop at 1000 LST Monday morning (possibly interrupt planned science as
Ken is out the remainder of the week;
if there's nothing pressing, we'll continue on with science until
Tuesday morning).

Joe

13 Oct

Hi,

Work supporting the first official holography run tonight along
with RSRO tests, and missing scan investigations.
For the CBE/missing scan issue, we'll switch back to the
modified pointing script used successfully last night; in the
morning Ken will revert to the original to see if during actual
science runs, the issue can be excited while being monitored
by Martin et al.
Michael et al. did see some issues in the Ka band settings from
the OPT; Dave is aware and will make changes once it's determined
how to adjust the LO setup in a better way, matching what's done
in the Executor.
The EA07 IF A bandpass issue was resolved (StB rebooted, unfortunately
requiring timecode B which is how it will run tonight).

Note:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA25 not yet ready.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 13Oct11
as above.

quad1 =
quad2 =
quad3 =
quad4 =


== Thursday  ==
OST (all tests have had their priorities updated except):

  - start anytime between 2000-2400 LST, 6.5 hours duration
     - /home/mchost/mbrentje/holography/evla-scripts/K/THOL0001_K_41x41.evla

 Continue observing throughout the morning on Friday; Ken will switch
 back to the troubled pointing script and with Martin, review the situation
 during science runs (since this seems to uniquely excite the problem).

Joe

12 Oct

Hi,

More work in hunting down issues with missing scans; no
solutions yet. Ken has put in a proto-fix that will eliminate the
initial short scan in reference pointing loops when you are already
on source. Hopefully this will work better; if we see worse missing
scans, we'll need to restrict the bands to exclude K,Ka, Q (for
now please leave them in to see if this works).
At the transition testing, we found one troubled board; in reviewing
that Ken noted that the interframe delays were set semi-randomly;
those were reset. b103-t-6 was getting a lower than expected rate
(the X4-Y0 LTA would light up intermittently; Ken noted that this
is the "line 28" problem. I excluded it for tonight).
We're running all OST tonight.

Note:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA25 not yet ready.
 - EA07 IFA appears to have a strange bandpass; sent to Eric.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - left in the old and added quad2-3
as above.

quad1 =
quad2 =      3,      7
quad3 =                      12
quad4 =  1


== Wednesday  ==
OST (all tests have had their priorities updated except):

  stop at 0815 Thursday for testing (Michael et al).

Joe

11 Oct

Hi,

The day spent on hunting down some outstanding issues with
missing scans and for 3bit testing. For missing scans, the
test was inconclusive. As a result, I'm going to venture on the
short night for all bands and we'll see what happens.
Note:
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA19 out; EA25 not yet ready.
 - EA07 IFA appears to have a strange bandpass; sent to Eric.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - everything looked fine but I kept these in
as I hadn't directly discussed with MR.

quad1 =
quad2 =            7
quad3 =                      12
quad4 =  1


== Tuesday  ==
OST (all tests have had their priorities updated except):

   stop at 0630 Tuesday for testing

Joe

10 Oct

Hi,

The day spent on hunting down some outstanding issues both
from the weekend and last week. In particular for tonight:
- Michael/Ken have a solid theory on the missing scan issues;
to avoid/limit exciting them for tonight, we would like to restrict
the observing to LSCXKu (i.e., no high frequency observing
tonight).
- In addition, if any SBs are interrupted/aborted, please
run clearCorrelator or just wait 20 seconds.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - everything looked fine but I kept these in
as I hadn't directly discussed with MR.

quad1 =
quad2 =            7
quad3 =                      12
quad4 =  1


== Monday  ==
OST (all tests have had their priorities updated except):
       - 1730-1830 10C-196 (Gibb's phenom test; must be stopped
manually a minute before next project).
       - 0730-0830
/home/mchost/bbutler/planettest/new/TPLA0001_sb5523084_1.evla

    stop at 0830 Tuesday for testing

Joe

07 Oct (Weekend)

Today we mostly continued recovering from the lustre installation problems
from Wednesday.  We believe all is now well...

Upon reviewing old Ka band data, we've concluded that Ka is "good to go"
(if the weather permits of course).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
 if an SB killed before completion (for all but the simplest)
 : if this happens, please run clearCorrelator.evla.

- The fringe display is still our first and best notice for
 possible failures. If the fringe display is not updating, please try
 d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.1 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

Baseline Board status:
   b103-b-6: X1Y2 failure
   b106-b-{0,1} out because of RXP failure (probably
     has to go back to DRAO) -- see Bruce for details.
   b107-t-2: ea04 (wafer 31) lots of errors
UPDATED unavailableBlBprs.txt
quad1 =
quad2 =               7
quad3 =                              12
quad4 = 1


== Friday-Monday morning  ==
OST (all tests have had their priorities updated except):
 - there may be some tests to interject during the
   weekend (e.g., Gibb's test, Holo) but we'll contact
   you and update when ready.

stop at 1100am Monday for testing


                       -- Michael

Callout: 11B-032 not fringing (restart CBE - residual active configuration).
Callout: 11B-106 gaps in BDF data (abort and more on).

06 Oct

Hi,

3-bit testing; Ka band testing - attenuators look fine; I would
still like to hold this back (no Ka band) until we can look at some fringes.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated
quad1 =
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

== Thursday  ==
OST (all tests have had their priorities updated except):
     stop at 0700 Friday for testing (Martin: 0800-1000 MDT).

Joe

05 Oct

Hi,

Maintenance day; transition attempted a bit earlier to
facilitate a rapid response project. Some issues remaining
from the maintenance activities - these will be addressed
fully tomorrow so we will run with limited capacity tonight.

- lustre-switch-networking system is not fully vetted so:

- we have to restrict the data rate; I've done this
artificially by restricting to only a single quadrant
of the correlator.
- data will not arrive in the archive; the data will still
be recorded but until it has been shifted over appropriately,
we don't want to notify the users - so *no* operator logs
should be sent out tonight. 10C-145 PIs will be notified
separately.
- DB for the antennas is also not working properly; please
do *not* try to do any delay setting or loading of parameters.

- Ka band issue has not been fully understood; please disallow
this band tonight.


Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated
quad1 =
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

== Wednesday  ==
OST (all tests have had their priorities updated except):
       10C-145 (two runs of this; the first had issues with the
antenna tracking; second should be good).
       0700-0800 TVER TBD (separate note)

      stop at 0800 Thursday for testing/further recovery

Joe

04 Oct

Hi,

Testing of the attenuator issues exposed in the holography tests.
Not quite resolved. We'll continue to hold back on Ka band
observing tonight (more time tomorrow will be dedicated to
exploring if we have an actual problem).
Keep Q3-8 out still; removed Q1-10 from the list of bad BlB prs.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out; EA08 back in.

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated
quad1 =
quad2 =
quad3 =                    8
quad4 =

== Tuesday  ==
OST (all tests have had their priorities updated):

      stop at 0600 Wednesday for maintenance.

Thanks!

Joe

03 Oct

Hi,

3-bit testing; review of attenuator issues (which have forked into
two behaviors: 1) THOL0001, 2) Ka band instabilities. For tonight,
we will disallow Ka band observing through the OST; work will be
done on this tomorrow. Michael found one bad BlB pair in the
checkout process B106-t-0,1 (Q3-8)

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated
quad1 =                          10
quad2 =
quad3 =                    8
quad4 =

== Monday  ==
OST (all commissioning tests have had their priorities updated):

      stop at 0830 Tuesday for testing.

Joe

September

29 Sep

Hi,

Testing of the new CM version (deployed); testing of
holography script issues from last night.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-09-29 20:31 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated
quad1 =                          10
quad2 =
quad3 =
quad4 =

== Thursday  ==
OST except:

       0100-0130 C;
/home/mchost/evla/scripts/opt/2011/09/TRSR0043_sb5501126_1.evla
(mosaic test; array time)
               if this fails, please send an e-mail and move on.

      stop at 1030 Friday for testing.

Joe

28 Sep

Hi,

Maintenance day.
Had a problem with EA06 IFC; Kerry found that it was still
set for 3-bit mode; fixed.
Had a problem with the X1Y2 problem on b102-t-5 (yeah!);
I removed it from consideration and left as is for further
reflection.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out; EA01 still in A config.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.6.0 (23 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated
quad1 =                          10
quad2 =
quad3 =
quad4 =

== Wednesday  ==
OST (restricting to K band and lower!) except:

       2200-0200 *; /home/mchost/evla/scripts/operations/syscoll.evla
       0200-0600 X; /home/mchost/... (TBD from Bryan)

      stop at 0800 Thursday for testing.

Joe
woops - no problem going to higher frequencies tonight - old note was left in.

Note, Vivek saw that EA23 L band fringes are good in the cross-hands not in
the parallel so likely swapped.

Joe

23 Sep (Weekend)

Hi,

EA01 remains in A config, so we'll continue to be in
the move configuration through Monday. Pointing is
established well enough for K band with reference
pointing but Ka, Q should be disabled.
Baselines are not yet obtained; data from the weekend
will be used to set these (getting EA01 later).

Spate of details from Michael's WIDAR update today
but the key highlights are:
- X1Y2 issue may recur; no fix for this until Bruce
returns; we'll fix in the gaps as possible.
- EA03-D is being used as a test bed this weekend;
please ignore issues with it.
- new MCAF is being used.

We started science circa 1300 LST and will go through
to 0800 Monday morning; note that after this each
Monday will be used for science observing in D configuration
unless otherwise specified.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out; EA01 still in A config.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Friday  ==
OST (restricting to K band and lower!) except:

       Friday: C; 1730-2130;
/home/mchost/evla/scripts/opt/2011/09/TSPE0008_sb5300265_1.evla (array
time, hand-edited)
       Saturday: Q; 0230-0630;
/home/mchost/evla/scripts/opt/2011/09/TSPE0008_sb5300354_1.evla (array
time, hand-edited)
       Saturday: Q; 1930-0030;
/home/mchost/evla/scripts/opt/2011/09/S4172_sb5252322_1.evla (array
time)
       Sunday: Q; 0230-0630;
/home/mchost/evla/scripts/opt/2011/09/TSPE0008_sb5300354_1.evla (array
time, hand-edited)
       Sunday: C; 1730-2130;
/home/mchost/evla/scripts/opt/2011/09/TSPE0008_sb5300265_1.evla (array
time, hand-edited)
       Monday: L; 0030-0630;
/home/mchost/evla/scripts/opt/2011/09/TDEM0007_sb5256868_1.evla (array
time)
       Monday: C; 0630-0700;
/home/mchost/evla/scripts/opt/2011/09/TPHA0001_sb5280282_1.evla (array
time)

      stop at 0800 Monday for testing.

Joe
quick correction:
 Monday: L; 0030-0630;
/home/mchost/evla/scripts/opt/2011/09/TDEM0007_sb5256848_1.evla (array
time)

(SB number was wrong; you usually catch these but I thought I would help!).
and....one more....

can you please squeeze in a 2 hour (or as long as you can manage
between 2300-0500 MDT) pointing run at X band sometime
during the weekend?

syspt2hx.evla is the right script since we'll do the 5 hour run on Monday.

Joe

22 Sep

Hi,

Antennas EA04, EA26 moves completed; EA01 remains in A config,
to be moved on Monday.
Rolled back to the production executor (from test).
We'll stay with the updated CBE.


Kerry switched out 108-t-3 today. Fringes all around on
A1/C1 (and on all 3 new antennas (EA07, EA14, EA06), A2/C2).

EA01,EA02, EA03 don't have a working MIB emulator (on the
deformatter; do not run Kerry's crawler); they are being
left as is for diagnostic purposes.

Delays were set.

We'll do some pointing for the freshly moved antennas.

Found a problem with b102-b-3 X1-Y2 CC(Q1:13; serial 2027);
excluded from consideration tonight and left as is (re-did
manually generated script to avoid that board).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out; EA01 still in A config. EA05 A/B are out at C band.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated 9/22/11
quad1 =                                                  13
quad2 =
quad3 =             8
quad4 =

== Thursday  ==
OST except:

       0.5 hour test run (between 2100-2400)
       C; /home/mchost/evla/scripts/opt/2011/09/TRSR0043_sb5139225_1.evla
(array time)

       2 hour pointing run (after midnight before sunrise);
       C; /home/mchost/evla/scripts/operations/syspt2hc.evla (array
time; must be manually killed).

      stop at 0800 Friday for testing.

Joe

21 Sep

Hi,

More moves today; more pointing tonight.
Ken saw some issues with a station board that will need
to be investigated.
CBE version updated.
Temperature and dew point sensors are stuck (Larry noted
they've been stuck since 1330 local time).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out; only EA01, EA04, and EA26 in A config.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110921.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated from Kerry and Michael
quad1 =
quad2 =
quad3 =             8
quad4 =

== Wednesday  ==
OST except:

       2 hour pointing run (after midnight before sunrise);
       X; /home/mchost/evla/scripts/operations/syspt2hx.evla (array
time; must be manually killed).

       0600-0800 C;
/home/mchost/evla/scripts/opt/2011/09/11B-224_sb5243059_1.evla (array
time; hand-edited)

      stop at 0800 Thursday for testing.

Joe

20 Sep

Hi,

More moves today; more pointing tonight. Quick transition;
things looked gorgeous, both Matt and I marveling at the
full correlator working (C_quad1234).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated from Kerry and Michael
quad1 =
quad2 =
quad3 =             8
quad4 =

== Wednesday  ==
OST except:

       1.5+ hour pointing run (after midnight before sunrise);
       C; /home/mchost/evla/scripts/operations/syspt2hc.evla (array
time; must be manually killed).

       0500-0700 C; 11B-224 makeup TBD (I'll send a note later
tonight if this is viable...)

      stop at 0700 Wednesday for maintenance.

Joe

19 Sep

Hi,

3-bit sampler testing today; EA07, EA14 are now working
based on Ken's tests.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated from Kerry and Michael
quad1 =
quad2 =
quad3 =             8
quad4 =

== Monday  ==
OST except:

       2 hour pointing run (more if possible); C;
/home/mchost/evla/scripts/operations/syspt2hc.evla (array time;
               must be manually killed).

      stop at 0800 Tuesday for testing.

Joe

16 Sep (Weekend)

Hi,

More moves. More testing; Michael hunted down an
issue with baseline board stacking; CM was rolled
back pre-emptively (though it likely wasn't the cause
but was an issue requiring BlB reboots; cf Bruce's note).
- recirculation wasn't working from 13 Sep - now; no
observing files were affected.

K.Scott replaced the failing disk on cbe-node-06

Scheduling is going to rely on the OST; we've bumped up some priorities
for tests and ToOs and will review how it goes; we'll intervene only as
necessary.

New CM version tonight.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA25 out

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - updated from Kerry and Michael
quad1 =
quad2 =
quad3 =             8
quad4 =


== Weekend  ==
OST except:

       Monday morning:
       0200-0800 C;
/home/mchost/evla/scripts/opt/2011/09/TSPE0007_sb5130878_1.evla

       stop at 0800 Monday for testing.

Joe

15 Sep

Hi folks --

 today was a combination of antenna move stuff & some fun solar
work.

Oldies but Goodies

------------------
- we have seen some issues with configurations not updating
 if an SB killed before completion (for all but the simplest)
- if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
 modes; the fringe display is still our first and best notice for
 possible failures. If the fringe display is not updating, please try
 d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

------------------
Antennas:
- EA23 should be used (on MP); EA25 out.

------------------

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt - not changed and likely overkill
quad1 =           6
quad2 =    3,                       15
quad3 =                 9
quad4 =                          14,15

------------------
== Tonight's observing ==

OST for everything except  Ken would like two hours of pointing
to check the moved antennas, sometime late tonight (after 11pm-ish).
I presume this is syspt2hx.evla -- KEN, please correct me if not!

Stop by 0730 LST ~ 0900 MDT for testing & software work during the day.

Cheers,

         Michael

p.s. Joe is still gone, so Michael is the first line of defence:

ea23 in one of the moved antennas today and is now on
the North arm.


OST for everything except  Ken would like two hours of pointing
to check the moved antennas, sometime late tonight (after 11pm-ish).
I presume this is syspt2hx.evla -- KEN, please correct me if not!

C band, please.  Two hours, at least, of syspt2hc, sometime
well after sunset and when the winds are calm.

Ken

14 Sep

Hi,
I'm on travel now but I'm expecting tonight to be some combination of
OST + Ken/Michael tests for the moved antennas (e.g., some pointing).

The only request I have is for:
2.5 hour project; can start anytime 2100-0100 LST:
/home/mchost/bbutler/planettest/new/TPLA0001_sb4950359_1.evla

Thanks

Joe

Thanks Joe.  I've finished up an OST-based script (11B-118), and will continue with the OST through the night except for your requested TPLA0001.  If there's unscheduled time we'll try and fit in 2+ hours of collimation per Ken's request (with only the antennas that have Ku receivers).

Is there a request for an earlier observing stop time, to start testing Thursday morning?  Software time based on the monthly schedule starts at 8:00 LST (9:33 MDT), but I know we tend to start it a little earlier around 8:30-9am MDT.

Cheers,
Matt

Ken's email below:
"The moved antennas are working well enough that I would be
happy to leave pointing for tomorrow night after more antennas
have moved.  Should there be time that would be otherwise unused,
I suggest at least two hours of collimation pointing using only
the antennas (including ea23) which have Ku band receivers. "

13 Sep

Hi,

Maintenance day; 4 more antennas moved (8 total); 3-bit
antennas (EA07/EA14) were equipped with two modules each;
these are currently not working.

New CM version tonight.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 should be used (on MP); EA25 out; EA05 out (FRM).

Correlator:
--
CM: 2011-09-13 18:29 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.5.0 (08 Sep)
OST: 1.11.03 (08 Sep)
TelCal 1.5.41 (08 Sep)

UPDATED unavailableBlBprs.txt - not changed and likely overkill
quad1 =           6
quad2 =    3,                       15
quad3 =                 9
quad4 =                          14,15

== Tuesday  ==
OST except/please run tonight:
       2.0 hour pointing;
/home/mchost/evla/scripts/operations/syspt2hc.evla (midnight-ish only)
       2.0 hour pointing;
/home/mchost/evla/scripts/operations/syspt2hs.evla (anytime; S band
antennas only)
       2.5 hour project; can start anytime 2100-0100 LST:
/home/mchost/bbutler/planettest/new/TPLA0001_sb4950359_1.evla

       stop at 0430 Wednesday for maintenance day

Joe

12 Sep

Hi,

First day of move; all antennas in; EA23 should be included
for all (though lacking an L and Ku currently).
Mostly OST tonight with some additional pointing.

Note, we had an issue with b104-t-4 (X4-Y0 LTA); can
kicked it and it is now working (still included for
tonight).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 should be used (on MP); EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt - not changed and likely overkill
quad1 =           6
quad2 =    3,                       15
quad3 =                 9
quad4 =                          14,15

== Monday  ==
OST except:
       1415-1515 C;
/home/mchost/evla/scripts/opt/2011/09/11B-217_sb5118753_1.evla (array
time)
       1530-1630 LKQ (self-cal);
/home/mchost/evla/scripts/opt/2011/09/10B-218_sb5016215_1.evla (1
hour; array time)

       2200-0100 X;
/home/mchost/evla/scripts/operations/syspt2hx.evla (array time; must
be manually terminated)

       stop at 0700 Tuesday for maintenance day

Joe 

Callout: No fringes 10C-196 (office visit restart):

Tom called with a problem with 10C-196 (just before 0500). Our internet was
out at the house (Quest coming in the morning to fix) so I asked him to confirm
with another calibrator cycle or two before I headed into the office.
Still no fringes (though d10 was working).

The CBE was plugged up pretty badly (wcbetool status showed a backlog
of 57); I stopped and restarted the CBE (many zombies present) and retested.
Things looked okay so we proceeded.

There were two incidents of missing scans: 11B-217 and TSPE0007.
A more detailed note will go to Martin.

We're recovered but there isn't adequate time to resume before maintenance
activities begin...

Joe

06 Sep (week)

Hi,

Coarse plan for the week - to be modified as possible.
Note that Michael and I are both away; please contact me
as needed (in VA so enjoying a 2 hour difference which
should help).

I've updated the unavailable boards list with the current
information; I've checked the hand-edited files to confirm
(they are not using that board; after having regenerated
them).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =           6
quad2 =    3,                       15
quad3 =                 9
quad4 =                          14,15

== Tuesday  ==
OST except:
       1400-1900
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (5
hours; array time; hand edited)
       1900-2000 LKQ (self-cal);
/home/mchost/evla/scripts/opt/2011/09/10B-218_sb5016215_1.evla (1
hour; array time)
       2000-2100 C;
/home/mchost/evla/scripts/operations/syspt2hc.evla (array time; must
end manually)
       2100-0430 SD0487 fixed date

       stop at 0430 Wednesday for maintenance day; run syspt2hc.evla
until activities interrupt.


== Wednesday ==
0430-1500 Maintenance day activities; science observing start by 1500
(hopefully)

OST except:
       anytime between 0700-1500 LST, run:
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (5
hours; array time; hand edited)
       2300-0130 XKu;
/home/mchost/bbutler/planettest/new/TPLA0001_sb4950359_1.evla (array
time)

       stop at 0630 Thursday for testing

== Thursday ==
0630-1500 LST testing activities; science observing start by 1500
(hopefully); Note:
LUNCHTIME OBSERVING (Sorry to shout!): Ken, can we run the following
over lunch please:
       1 hour; L;
/home/mchost/evla/scripts/opt/2011/09/TAST0001_4988750_1.evla (array
time)

OST except:
       anytime between 0700-1500 LST, run:
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (5
hours; array time; hand edited)
       1900-2000 LKQ (self-cal);
/home/mchost/evla/scripts/opt/2011/09/10B-218_sb5016215_1.evla (1
hour; array time)

       Continue through Friday with science observing

== Friday ==

OST except:
       0500-1000 L;
/home/mchost/evla/scripts/opt/2011/09/TLOW0001_sb3370119_1.evla (array
time)
       1000-1500 L;
/home/mchost/evla/scripts/opt/2011/09/TLOW0001_sb4539525_1.evla (array
time)
       1500-1600 K;
/home/mchost/evla/scripts/opt/2011/09/493_1_sb4741317_1.evla (array
time; test)
       1600-1930 L;
/home/mchost/evla/scripts/opt/2011/09/11A-154_sb4700528_1.evla (array
time)

== Saturday ==
OST except:
       0800-1300 L;
/home/mchost/evla/scripts/opt/2011/09/TLOW0001_sb4539525_1.evla (array
time)
       1300-2100 CX;
/home/mchost/evla/scripts/opt/2011/09/TDEM0011_sb4882012_1.evla (array
time)

== Sunday ==
OST except:
       0400-0900 L;
/home/mchost/evla/scripts/opt/2011/09/TLOW0001_sb4896483_1.evla (array
time)
       0800-1300 L;
/home/mchost/evla/scripts/opt/2011/09/TLOW0001_sb4539525_1.evla (array
time)
       1300-1800 L;
/home/mchost/evla/scripts/opt/2011/09/TLOW0001_sb4539976_1.evla (array
time)
       1800-1900 LKQ (self-cal);
/home/mchost/evla/scripts/opt/2011/09/10B-218_sb5016215_1.evla (1
hour; array time)


Stop Monday morning at 0500 LST. Thanks.

Joe 

Callout: Expected file doesn't show up (manually create it).

Hi Tom,
Please let me know if this doesn't come up in the OST; I can make a
backup file for you.
For Ken's pointing, we've only got the one hour planned (2000-2100).
Joe

I have deployed a new version of the status report tool that will handle fixed
SBs again.  I added a fixed start column.  If the SB is dynamic, the value here
will be "---".  Similarly, for fixed sbs, the lst start min/max columns and the
start after column will be "---".

The old version is still available as sbStatReport-old.

The new version is v1.10.02.

Sep 02 (Weekend)

Hi,

All day observing through the long weekend; there may be some changes as we
track the weather. I also may need to update 11A-142 pointer based on its
BlB selection (pending a discussion with Michael).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =           6
quad2 =    3
quad3 =                 9
quad4 =                          15

== Friday  ==
OST except:
  Friday:
       0700-0800 C;
/home/mchost/evla/scripts/opt/2011/09/11B-217_sb4991969_1.evla (array
time)
       0930-1330 SC;
/home/mchost/evla/scripts/opt/2011/09/11A-231_sb4455680_2.evla (array
time)
       1330-1830 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited)
       1830-1930 X;
/home/mchost/evla/scripts/opt/2011/09/10C-168_sb4992740_1.evla (array
time)
       ...
       2130-2230 XKQ;
/home/mchost/evla/scripts/opt/2011/09/11A-266_sb4986877_2.evla (array
time)

  Saturday:
       0800-1300 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited)
       1300-2100 CX;
/home/mchost/evla/scripts/opt/2011/09/TDEM0011_sb4882012_1.evla (array
time)
       2100-0300 CK;
/home/mchost/evla/scripts/opt/2011/09/11A-137_sb4155807_1.evla (array
time)

  Sunday:
       0500-0600 C;
/home/mchost/evla/scripts/opt/2011/09/11A-226_sb4981395_1.evla (array
time)
       0600-0800 C;
/home/mchost/evla/scripts/opt/2011/09/11A-226_sb4245180_1.evla (array
time)
       0800-1300 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited)
       1300-2100 CX;
/home/mchost/evla/scripts/opt/2011/09/TDEM0011_sb4887227_1.evla (array
time)
       2100-0300 CK;
/home/mchost/evla/scripts/opt/2011/09/11A-137_sb4155807_1.evla (array
time)

  Monday:
       0730-0930   ; /home/mchost/evla/scripts/opt/2011/09/11A-182_sb
       0930-1330 SC;
/home/mchost/evla/scripts/opt/2011/09/11A-231_sb4456035_5.evla (array
time)
       1330-1830 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited)
       1830-2130 L;
/home/mchost/evla/scripts/opt/2011/09/11A-129_sb4128829_6.evla (array
time)
       2130-0330 CK;
/home/mchost/evla/scripts/opt/2011/09/11A-137_sb4155807_1.evla (array
time)
...


Currently, we're planning to observe throughout Tuesday morning.

Joe
Hi,
First update, for tonight:

       1830-1900 C;
/home/mchost/evla/scripts/opt/2011/09/10B-212_sb4995240_1.evla (array
time)
       1900-2000 X;
/home/mchost/evla/scripts/opt/2011/09/10C-168_sb4992740_1.evla (array
time)  ** Note insert this half hour block before the 11A-266
       ...
       2130-2230 XKQ;
/home/mchost/evla/scripts/opt/2011/09/11A-266_sb4986877_2.evla (array
time)


In addition, the (Saturday, Sunday, Monday) 11A-137 blocks should be
left to the OST (they may still appear and should be run if they do)
but we don't
need to manually schedule these.

Joe

Hi,
Just an update for today and tomorrow; we missed quite a few scans
from Rick's demo
science SB last night; we watched it throughout and may not be
calamitous but as this
is currently the only program that is known to drop scans, it's an
exciting(!) opportunity
to try to hunt this down. Given that we have other chances to get this
next weekend,
I'd like to pull it from the schedule and beat further on 11A-142 (and
allows us to sneak
in the 10B-218 observation earlier).

Here's what I have for the rest of the weekend's manual runs (all else
is OST); please also
note that we withdrew the manual runs of 11A-137 (see below); it
appears this was still forced
on - we're happy to have this run through the OST but don't need to
force this program on at
this time. Also note that we will stop early-ish on Tuesday to allow
some testing (0900 MDT).

Thanks!

Joe

  Sunday:
       0500-0600 C;
/home/mchost/evla/scripts/opt/2011/09/11A-226_sb4981395_1.evla (array
time)  * DONE
       0600-0800 C;
/home/mchost/evla/scripts/opt/2011/09/11A-226_sb4245180_1.evla (array
time)  * DONE
       0800-1300 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited) * RUNNING
       1300-1800 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited); repeat!
       1800-1830 C;
/home/mchost/evla/scripts/opt/2011/09/10B-218_sb5007436_1.evla (array
time)
       1830-2130 L;
/home/mchost/evla/scripts/opt/2011/09/11A-129_sb4128829_6.evla (array
time; hand edited)

  Monday:
       0830-1330 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited)
       1330-1830 L;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4616455_7.evla (array
time; hand edited)
       1830-2130 L;
/home/mchost/evla/scripts/opt/2011/09/11A-129_sb4128829_6.evla (array
time; hand edited)
...

Stop at 0600 LST Tuesday morning (Martin will take over here).

01 Sep

Hi,

Early turnover to accommodate fixed date observing. Some
ongoing WIDAR issues, outlined separately; we'll continue
to monitor.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =           6
quad2 =    3
quad3 =                 9
quad4 =                          15

== Thursday  ==
OST:
       1400-2100 11A-224 fixed date
       2100-0300 SD0487 fixed date
...

Currently, we're planning to observe throughout Friday.
Please do not schedule past 0730 LST on Friday as we likely
need to inject some specific programs.


Joe

August

31 Aug

Hi,

Early turnover to accommodate fixed date observing. Some
ongoing WIDAR issues (outlined in separate note by Michael).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =           6
quad2 =    3
quad3 =                 9
quad4 =                          15

== Wednesday  ==
OST:

       1300-1400 L;
/home/mchost/evla/scripts/opt/2011/08/11A-291_sb4911125_3.evla (array
time)
       1400-2100 11A-224 fixed date (should come up in OST - if this
fails, let me know so I can
               create a backup).
       ...
       2100-2130 X;
/home/mchost/evla/scripts/opt/2011/08/TPLA0001_sb4987812_1.evla (array
time)
       ...
       0300-0330 X;
/home/mchost/evla/scripts/opt/2011/08/TPLA0001_sb4988578_1.evla (array
time)
       ...

Please stop Thursday at 0600 LST for testing.

Joe

30 Aug

Hi,

Sundry tests including 3-bit. Recovery from maintenance
day was comparatively painless.
Note, b103-t-6 had problems and had to be removed from
consideration (a new flavor of BlB problem).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =    3
quad3 =
quad4 =                          15

== Tuesday  ==
OST:

       1436-1936 L;
/home/mchost/evla/scripts/opt/2011/07/11A-142_sb4616455_1.evla (array
time; hand edited)
       ...
       0300-0530 XKKa;
/home/mchost/evla/scripts/opt/2011/08/TAST0001_sb4961653_1.evla (array
time; astrometry)


Please stop Wednesday at 0600 LST for testing. Note, we have fixed
date observing beginning at 1400 LST so
turnover to operations will need to happen a bit earlier tomorrow.

Joe

A couple additional updates today:
* We are now back to running with TIMECODE A, following the replacement
 of a bad fiber.
* We rebooted all the correlator boards today, following an increase in
 the number of NFS threads and the removal of most (ancient) software
 from the old default boot area -- both changes to avoid problems when
 power cycling the correlator, as described in my looong message
 yesterday evening. * I believe Kerry also put in some new boards, including a working Station
 Board to replace the flaky one in rack 8 (which had led to problems with the
 startup sequence for some of the Baseline Boards).

Cheers,
           Michael

28 Aug

Hi,

Sleuthing of some issues from the weekend revealed that
a fair chunk of data from the weekend were lost due
to a subtle issue in the non-configuration of a set
of BlBs.

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
- Apparently 108-b-6 is used for testing; until this is stopped it
should be removed.
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                          15

== Monday  ==
OST:

       1423-1523 C;
/home/mchost/evla/scripts/opt/2011/08/11B-217_sb4982307_1.evla (array
time)
       1523-2023 L;
/home/mchost/evla/scripts/opt/2011/07/11A-142_sb4616455_1.evla (array
time; hand edited)
       2030-2130 C;
/home/mchost/evla/scripts/opt/2011/08/TSPE0008_sb4563376_1.evla (array
time)
       2130-0230 CX;
/home/mchost/evla/scripts/opt/2011/08/11A-178_sb4679608_1.evla (array
time)


Please stop Tuesday at 0330 LST for maintenance day activities.

Joe

26 Aug (Weekend)

Hi,

Abbreviated testing again today; turned over to science
at 1100 LST and will run through the weekend.

We are currently continuing to run on timecode b.
If *any* issues arise tonight and until this is sorted,
please contact Michael/Ken as the initial POC

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.


Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =           6
quad4 =

== Friday ==
OST except:
       Friday: 1100-1700 TDEM0006
       Friday: 1700-2000 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_1.evla (array
time; hand edited)
       Friday: 2000-2100 C;
/home/mchost/evla/scripts/opt/2011/08/TSPE0008_sb4563376_1.evla (array
time; hand edited)
       ...
       Saturday: 2200-0300 CX;
/home/mchost/evla/scripts/opt/2011/08/11A-178_sb4679608_1.evla (array
time)
       Saturday: 0300-0530 XKKa;
/home/mchost/evla/scripts/opt/2011/08/TAST0001_sb4961653_1.evla (array
time)
       ...gap
       Saturday: 0700-0900 C;
/home/mchost/evla/scripts/opt/2011/08/11A-182_sb4829529_2.evla (array
time)
       Saturday: 0900-1500 Ka;
/home/mchost/evla/scripts/opt/2011/08/TDEM0006_sb4525008_1.evla (array
time; self-cal'able)
       Saturday: 1500-2000 L;
/home/mchost/evla/scripts/opt/2011/07/11A-142_sb4616455_1.evla (array
time; hand edited)
       ...
       Sunday: 0530-1030 L;
/home/mchost/evla/scripts/opt/2011/08/TLOW0001_sb3370119_1.evla (array
time)
       Sunday: 1030-1230 C;
/home/mchost/evla/scripts/opt/2011/08/11A-182_sb4829529_2.evla (array
time)
       Sunday: 1300-2100 LS;
/home/mchost/evla/scripts/opt/2011/08/TDEM0011_sb4908992_1.evla (array
time)


Please stop Monday at 0700 LST for testing.


Joe
Ah, one change already (these ruddy SNs!) :)

Please change:

Saturday: 0700-0900 C;
/home/mchost/evla/scripts/opt/2011/08/11A-182_sb4829529_2.evla (array
time)

to

Saturday: 0800-0900 C;
/home/mchost/evla/scripts/opt/2011/08/11B-217_sb4976573_1.evla (array
time)

Joe

25 Aug

Hi,

Some further work and testing this morning on the timecode
a/b paths; we are currently continuing to run on timecode b.
If *any* issues arise tonight and until this is sorted,
please contact Michael/Ken as the initial POCs


Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.
 - EA20 out

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =             7
quad4 =  1

== Thursday  ==
OST:
       Note, we did pushed on a few programs but we're letting the OST
       do it's thing tonight; Larry looked ahead a bit and the
       things that were popping up were all on the 'must-do' list
       for the configuration.


Please stop Friday at 0600 LST for testing.

We expect to go back to science operations after lunch tomorrow and will
be hitting demo science data as much as possible.

Joe

24 Aug

Hi,

Initially rapid recovery today from the maintenance allows us to
do some science; notes on the fire suppression system
shutdown were sent out by Kevin to widar-wg.
Some issues with the timecode generation in the late afternoon
(running on timecode B tonight - delays took some time for
Ken/Larry to get in order).

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.
 - EA20 out

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =             7
quad4 =  1

== Wednesday  ==
OST except:

       1515-1630 L;
/home/mchost/evla/scripts/opt/2011/08/11A-221_sb4972718_1.evla (array
time)
       1630-1930 L;
/home/mchost/evla/scripts/opt/2011/08/11A-154_sb4700528_1.evla (array
time)

Please stop Thursday at 0600 LST for testing.

Joe
A bit of a sloppy note tonight; Larry handled it with aplomb however
(11A-154 was off but Larry completed it; 11A-221->10B-221 but the
SB was matched by Larry and we ran).
Some issues with antennas (EA06 and EA20) due to power glitches.

Finally, we would like to stop 1 hour earlier; we've received a request
for another key ToO (another SN); if possible (and the timecode work
goes well), we'd like the opportunity to observe over lunch time (trading
the hour in the early morning).

Joe

23 Aug

Hi,

Bonus data between double maintenance days!

Old Notes (but still relevant - note fix coming soon here!):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.
- EA05 temporarily out (cryo team at work)
- EA04 - seeing some issues on IF A

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =             7
quad4 =  1

== Tuesday  ==
OST except:

       1500-1600 L;
/home/mchost/evla/scripts/opt/2011/08/11A-275_sb4963332_1.evla (array
time)
       1600-2100 C;
/home/mchost/evla/scripts/opt/2011/08/10B-124_sb4904375_1.evla (array
time)
       2100-0100 K;
/home/mchost/evla/scripts/opt/2011/08/11A-235_sb4901297_1.evla (array
time)

Please stop Wednesday at 0400 LST for double maintenance day.

Joe

22 Aug

Hi,

Executor and 3-bit testing; some recovery from power issues
over the weekend.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.
 - EA27 out for tonight (L305 stability issues); please keep *out*.
 - EA24 A/C are out for C band (please keep in though).

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =             7
quad4 =  1

== Monday  ==
OST except:

       1315-1415 L;
/home/mchost/evla/scripts/opt/2011/08/11A-291_sb4911115_1.evla (array
time)
       1430-1800 L;
/home/mchost/evla/scripts/opt/2011/08/11A-154_sb4676563_1.evla (array
time)
       1800-2100 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_3.evla (array
time; hand edited; note _3!

Please stop Tuesday at 0400 LST for double maintenance day.

Joe

19 Aug (Weekend)

Hi,

Some testing of the issues from last night:
- script error will be corrected; we'll re-run
planetary testing on Sunday night.
- 11A-129 issue was linked to a problem with the
interframe delays; we'll use a better test of this
for system hand-over (C_quad1234)
- 11A-228 issue is not yet understood; this is on-hold
for now and we have an additional test script from
Michael to check this should symtoms recur.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.
 - EA27 out for tonight; subreflector not functioning.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Friday - Monday morning  ==
OST except:

       Saturday: 0300-0530 XKKa;
/home/mchost/evla/scripts/opt/2011/08/TAST0001_sb4961653_1.evla (array
time; astrometry)
       Saturday: 1100-1600 L;
/home/mchost/evla/scripts/opt/2011/07/11A-142_sb4616455_1.evla (array
time; hand edited)
       Saturday: 1600-2100 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4157844_1.evla (array
time; hand edited)
       Saturday: 2100-2130 x;
/home/mchost/evla/scripts/widar/uvtest.evla (array time; astrometry)

       Sunday:   1330-1830 L;
/home/mchost/evla/scripts/opt/2011/07/11A-142_sb4616455_1.evla (array
time; hand edited)
       Sunday:   1830-2130 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_1.evla (array
time; hand edited)
       Sunday:   2130-2400 x; TBD (array time; planetary script)

Please stop Monday at 0530 LST for development/testing.

       I will pick up 11A-154 early next week.

Joe

Callout: 11A-129 not fringing (looks okay; TelCal eventually picked up).
Callout: Script not in expected location (pushed to Monday).
Callout: Power glitches (confirm recovery looks okay).
Callout: 11A-142 not fringing (problem with BlBs not communicating; removed from consideration).
Callout: 11A-129 not fringing (residual issues with BlBs; MR rebooted them; moved on).

Just an update:

- weathered (ha!) the power glitches yesterday; Tom recovered things
quite quickly and the 11A-129 that followed
seemed to be in good shape if a bit slow in TelCal.
  - EA11 is out until fixed
- Tom noted that 11A-142 was not fringing; I looked in the logs and
there were issues with serial numbers to two boards:
  - b105-b-6
  - b107-t-3
I tried to view them with the BlB gui but they give communications
errors so I removed them from the unavailable list
for the rest of tonight. I pushed in a 2.5 hour RSRO program (11A-126)
to fill the gap + OST to get to 11A-129.

Sunday:   1415-1645 L;
/home/mchost/evla/scripts/opt/2011/0/11A-116_sb4936060_1.evla (array
time)
Sunday:   1830-2130 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_1.evla (array
time; hand edited)
Monday:   2130-2400 x; TBD (array time; planetary script)
Monday:   0300-0530   ;
/home/mchost/evla/scripts/opt/2011/08/TAST0001_sb4961653_1.evla (array
time; astrometry)
  - we missed this Saturday as I hadn't moved the file to proper location

Please stop Monday at 0530 LST for development/testing.

Joe

18 Aug

Hi,

Some 3-bit and fast dump testing throughout the day.
Retaining stable software (though updates on the
obs prep side are coming).

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.
 - EA05 A/B at X band are dead.
 - EA24 at X band has weak fringes.
 - EA27 out for tonight; subreflector not functioning.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Thursday  ==
OST except:

       1600-2100 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4157844_1.evla (array
time; hand edited)

       0430-0530 *;
/home/mchost/evla/scripts/opt/2011/08/TVER0001_sb4956127_1.evla (array
time)

Please stop at 0530 LST for development/testing.

Joe

Callout: 11A-129 not fringing; found only a fraction of the expected frames being received; aborted and moved on (ended up being an issue with the interframe delay for a subset of boards; improved testing in this area).

Just a quick update.

Dave noted that 11A-129 wasn't fringing. I looked and found that the
CBE was only
getting a fraction of the expected frames. I didn't ponder this very
long but aborted
and noted it for review tomorrow.

I've added a planetary test for 2300 (separate note).

Joe

17 Aug

Hi,

Observing all day; new software deployed in the morning
(following some quick sleuthing on the problems last night)
New CBE (solves 1.E37 amplitudes); OST (solves pointer
to bad vlapointing script).

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
 - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110817.0 (using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.02 (17 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Wednesday  ==
OST except:
       SN observation is pending; please don't begin until we've
identified where in the schedule this
       will go.

       1600-2100 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4157844_1.evla (array
time; hand edited)
       If K band weather:
       2100-0100 XK;
/home/mchost/evla/scripts/opt/2011/08/11A-235_sb4901297_1.evla (array
time; EB issue)

Please stop at 0530 LST for development/testing.

Joe

16 Aug

Hi,

Maintenance day; recovery went smoothly (Ken&Michael).

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP; EA25 out.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110811.0 (11 Aug; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.11.01 (16 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

Doing a RSRO weekend:

== Tuesday  ==
OST except:
       1345-1500 L;
/home/mchost/evla/scripts/widar/10C-196_sb3898054_3.evla (Gibb's
check); manually interrupt to get the next
                            program going by 1500.
       1500-1800 C;
/home/mchost/evla/scripts/opt/2011/08/SC0346_sb4932489_1.evla (backup
if OST doesn't pick up; array time)


Observing continues throughout the day tomorrow, so continue to 0600
LST at which point we'll
do the 11A-229 program (0600-1400 LST); we'll call to touch base at ~1700 MDT to
review plans for the night.

Joe

15 Aug

Continued data rate tests (focusing on <1 sec dumps); review
of issues from the weekend; new executor version deployed.
Note that we had a problem following an X_osro where antennas
24+ did not slew to source but had to be manually commanded
to 'Use'.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 MP; EA25 out; EA18 Servo glitch - out currently.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110811.0 (11 Aug; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =


== Monday  ==
OST except:

       1300-1400 L;
/home/mchost/evla/scripts/opt/2011/08/10C-196_sb3898054_2b.evla
(Gibb's phenom check)
       1400-1700 C;
/home/mchost/evla/scripts/opt/2011/08/SC0346_sb4932489_1.evla (backup
if OST doesn't pick up)


Please stop Tuesday 0230 LST (~0630) for development/testing.

Note: 15-16 August Chandra coordinated observations; first night tonight!
- Hide quoted text -

Joe

12 Aug (Weekend)

Hi,

Continued data rate tests (focusing on <1 sec dumps); going
okay but no changes to operations software for the weekend.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 MP

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110811.0 (11 Aug; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

Doing a RSRO weekend:

== Friday  ==
       1400-1600 C;
/home/mchost/evla/scripts/opt/2011/08/10C-222_sb4824939_1.evla (array
time); RSRO
       1600-2100 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4157844_1.evla
(special mode; hand edited; array time); RSRO

== Saturday ==
       2100-2300 C;
/home/mchost/evla/scripts/opt/2011/08/11A-126_sb4923288_1.evla (array
time); RSRO
       2300-0200 Ka;
/home/mchost/evla/scripts/opt/2011/08/AC982_sb4930649_1.evla (array
time); RSRO
       0300-0700 Ka;
/home/mchost/evla/scripts/opt/2011/08/11A-220_sb4128964_1.evla (array
time); RSRO
       0700-1100 C;
/home/mchost/evla/scripts/opt/2011/08/11A-216_sb4796528_1.evla (array
time); RSRO
       1100-1330 C;
/home/mchost/evla/scripts/opt/2011/08/11A-216_sb4733247_1.evla (array
time); RSRO
       1330-1730 C;
/home/mchost/evla/scripts/opt/2011/08/11A-216_sb4581605_1.evla (array
time); RSRO
       1730-2030 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_1.evla (array
time); RSRO

== Sunday ==
OST except:
       2030-0230 K;
/home/mchost/evla/scripts/opt/2011/08/11A-137_sb4155807_1.evla (array
time); RSRO
       ... OST
       0900-1400 LC;
/home/mchost/evla/scripts/opt/2011/08/11A-142_sb4914603_1.evla (array
time); RSRO
       1400-1900 C;
/home/mchost/evla/scripts/opt/2011/08/TDEM0010_sb4698567_1.evla (array
time); DEMO
       1900-1930 X;
/home/mchost/evla/scripts/opt/2011/08/11A-138_sb4218172_1.evla (array
time); High P OSRO
       1930-2030 C;
/home/mchost/evla/scripts/opt/2011/08/11A-226_sb4245182_1.evla (array
time); High P OSRO
       2030-2100 X;
/home/mchost/evla/scripts/opt/2011/08/11A-138_sb4223678_1.evla (array
time); High P OSRO

== Monday ==
       2100-2200 K;
/home/mchost/evla/scripts/opt/2011/08/TDEM0013_sb4735826_1.evla (array
time); DEMO
       2200-0400 K;
/home/mchost/evla/scripts/opt/2011/08/11A-137_sb4155807_1.evla (array
time); RSRO
       0400-0700 CL;
/home/mchost/evla/scripts/opt/2011/08/10C-123_sb4043882_1.evla (array
time); High P OSRO


Please stop Monday 0700 LST (~1100) for development/testing.

Note: 15-16 August Chandra coordinated observations

Joe

Callout: 10C-222 crashed on scan 11 (looks like executor 0000 UT error; restart Executor at break).
Callout: No fringes on 11A-129 (confirm all is working well; guess that d10/TelCal are struggling with the large number of channels in this program).
Callout: 11A-226 is running longer than expected (manual error by me; push other programs by 1.5 hours)

Note: 10C-222 crashed on scan 11;

This was the message:
Class edu.nrao.evla.observe.Array in file Array.java at 546
Class edu.nrao.evla.observe.Array in file Array.java at 137
Class java.lang.String in file String.java at 1937
Uncaught Exception in Array Thread String index out of range: -1
Matrix 4x4 Switch create socket failed: mc-t450

Re-running -in array time- yielded an immediate failure; we tried an X_osro
but the CM was stuck on the 10C-222. I deleted subarrays and flushed
the various queues on the CM to get back to square one.

X_osro then worked so we're retrying 10C-222 again and trying to push
the schedule forward by 30 minutes until the gap all looks good.

Matt noted that this smells like the 0000-0959 UT time precision issue
in the executor; I didn't think we'd advanced version but Matt will confirm.

Some additional notes:

It looks fine to push ahead (we've closed 30 minutes of the 1 hour gap
that we had due
to this failure); 11A-220 needs to start at 0300 LST (or very close to
it even though it
is in array time).

I had the _suffix wrong on the Saturday run of 11A-129 (it should be a _2):
1730-2030 L; /home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_2.evla
(array time); RSRO

Joe

Another update.

Matt noted no fringes for 11A-129 and d10 had no graphical display. We
walked through the following: CM (looks to be on correct config), CBE
(status looks right; log file seems to be keeping up with the 1s
integrations), Main.xml file (seems to be pointing/updating the BDF
pointers), and BDF files (are appearing every minute in the bunker
area)...so we think things
are actually working but perhaps due to the many channels per
sub-band, TelCal and d10 are struggling - we're going to let this run
and look at it with TelCal offline and talk with Ken regarding d10.

On-on...

Matt&Joe

p.s. I withdrew the two 11A-137 files based on Joan's recommendation.

One cause may be the correlator configuration.  d10, by default,
displays data for the subband labelled as 0, but I have seen OPT
generated scripts which do not produce a subband 0.  To check this
d10 has a "summary switch (d10 -summary 1 2) wich asks it to describe
all the data it is getting.  If inspection shows that there is no
subband 0 data, then restart d10 asking for a subband which is
present.  (d10 -subband x 1 2)

d10 reads data from the CBE into a buffer of length 100000, so will
not work reliably if there are more than 8192 channels per subband.
This can be changed if needed.

Ken


11 Aug

Hi,

CBE tests of data rates; these have gone so well that
we're running this version tonight (see e-mail to
widar-wg from Martin&Michael for more details).

Antenna EA05 is having issues and it's expected that
at some point it will lose network connectivity and
auto-stow; please keep in until then.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 barn; see note for EA05 above.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_20110811.0 (11 Aug; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug)
TelCal 1.5.39 (10 Aug)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Thursday  ==
OST except:

       1530-1700 K;
/home/mchost/evla/scripts/opt/2011/08/493_1_sb4741317_1.evla (webtest
version)
       1700-2000 L;
/home/mchost/evla/scripts/opt/2011/08/11A-129_sb4128829_1.evla
(challenge to new CBE)
       ...
       0000-0500 *;
/home/mchost/evla/scripts/opt/2011/08/TPOL0003_sb4672057_1.evla
       or, if weather is Ka quality:
       2300-0200 Ka;
/home/mchost/evla/scripts/opt/2011/08/AC982_sb4931282_1.evla (in case
it doesn't come up in the OST)

Please stop Friday 0500 LST (~0900) for development/testing.

Joe

10 Aug

Hi,

Ongoing testing of data rates. Update to TelCal to
resolve the memory issues; this was tested throughout
the day but we'll roll-back as needed.
All hardware is looking good.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 on MP.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug))
TelCal 1.5.39 (10 Aug) ** New

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Wednesday  ==
OST all night!

Please stop Thursday 0500 LST (~0900) for development/testing.

Tomorrow: Planned lunchtime holography run pending script from Bryan.

Joe

09 Aug

Hi,

Ongoing testing of data rates; no changes in operations
software tonight. All hardware is looking good.

Weather is outrageously good so jumping on some rare
high frequency times...

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 barn;

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug))
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Tuesday  ==
OST all night!

Please stop Wednesday 0230 LST (~0630) for maintenance day.

Joe

Callout: Missing BDFs (check; only a single; continue).

08 Aug

Hi,

Ongoing testing of data rates; no changes in operations
software tonight. All hardware is looking good (though
Ken noted a reboot was required on b101-b-3).

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
  - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
 - EA23 barn; EA06 back in.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug))
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Monday  ==
OST all night!

Please stop Tuesday 0500 LST (~0900) for development/testing.

Joe

05 Aug (Weekend)

Hi,

Hardware is in a lovely state; BlB reboots took place and
Kerry had replaced the misbehaving boards so all 64 pairs
are available (again).

A number of issues worked through:
- apparent issues with TelCal; good news here in that the data
look like they are recoverable - no crazy flags. The TelCal
issue was explored by Keith extensively today; some memory
issues were better understood; a reboot of both TelCal/MCAF
took place and we believe we should be okay for the weekend.
- the executor has been rolled back to 2.1.13 (taking with it
the problem with the 0000-0010 observing restriction).
- OST issue with available blocks not being scheduled has
been resolved.
- we have a new CM version that has some enhancements (error
logs in particular), failed configuration handling.
- CBE is maintaining at the same level.

Old Notes (but still relevant):
-------------------------------
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
   - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
  - EA23 barn; EA06 back in.

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug))
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =

== Friday  ==
OST except/note:

== Saturday ==
OST except/note:
        0000-0130 /home/mchost/evla/scripts/opt/2011/08/TSPE0008_sb4774242_1.evla (H2O maser V Stokes test, array time)
        Fixed date: 0600-1400 LST 11A-229 Spangler
                backup file: /home/mchost/evla/scripts/opt/2011/08/11A-229_sb4128819_1.evla (array time)
        2300-0200 CKa; /home/mchost/evla/scripts/opt/2011/08/AC982_sb4894104_1.evla (array time)
                *IF* Ka weather

== Sunday ==
OST except/note:
        0500-1000 L; /home/mchost/evla/scripts/opt/2011/08/TLOW0001_sb3370119_1.evla (array time)
        1000-1100 L; /home/mchost/evla/scripts/opt/2011/08/10C-196_sb4088789_1.evla (array time; Gibbs test)
                - please stop manually after one hour only! I'll get this in early Sunday.
        1100-1800 LC; /home/mchost/evla/scripts/opt/2011/08/11A-142_sb4913939_1.evla (array time)
        Fixed date: 1800-2200 LST TDEM0013 Miller-Jones
                backup file: /home/mchost/evla/scripts/opt/2011/08/TDEM0013_sb4859266_1.evla (array time)

== Monday ==
OST except/note:


Please stop Monday 0600 LST (~1000) for development/testing.

Joe

04 Aug

Hi,

Two misbehaving baseline boards; these have been removed from
consideration; Kerry will swap these out tomorrow for his two
spares; tomorrow will also see a morning reboot of the BlBs;
Kerry will be on site to help the recovery.

Some data rate and CM testing today; in particular, last night
was a serious loss of data - 1 hour into an 8 hour run, the
BDFs stopped getting produced. Unfortunately our BDF monitor
did not catch this for known reasons. We are continuing to
run an updated version of the CM tonight so:
- we have seen some issues with configurations not updating
if an SB killed before completion (for all but the simplest)
   - if this happens, please run clearCorrelator.evla.
- as above, the BDF monitor has known issues with certain failure
modes; the fringe display is still our first and best notice for
possible failures. If the fringe display is not updating, please try
d10 and if both are failing, it's time for a phone call.

Thanks for your continued vigilance.

----
Antennas:
  - EA23 barn

Correlator:
--
CM: 2011-08-04 15:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.4 (03 Aug)
OST: 1.10.03 (03 Aug))
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12,   14
quad4 =                       14
        - b108-b-4,5, b101-t-6,7

== Thursday  ==
OST through Friday 0430:

Stop at 0430 for development/testing.

Plans for the morning:
- review data from previous night
- BlB reboots
- 11A-142_4864339_1.evla exploration; failure over the weekend - need to montior (0730 LST onward)
        - Martin has requested that we try this with the new CBE version so we'll roll forward
        and test; if this works well, we'll retest the other cases.


Joe

Callout: No fringes on TCAL0001; checked CBE/CM/data on mchammer - look okay; continue.

Jim called after noting that there seemed to be a steady deterioration
on the fringe
display for the TCAL0001 observation (fringe was showing less
antennas, increasing
number flagged and then seemed to stop updating all together); d10 was working.

I checked the CM/CBE both seem to be tracking okay (cbe status shows
the current configuration with the expected and received rates, cm is
tracking the current
observation and exchanging with CBE, logs indicate integrations are
being achieved,
MCAF is updating - Main.xml indicates real addresses for the BDF files, new BDF
files are appearing on lustre). Looking at the TelCal output however
it is clear that
there is an issue - I see the no updates in over an hour and can see the antenna
flags as indicated. I'm not sure I understand what's happening. I'm
kicking a note
to Keith to investigate on the TelCal side and we can have Amy look at the data
later in the morning.

We're continuing on with the last observation of the morning.

Joe

03 Aug

Hi,

Running updated CM version (this works with all of the
previously troubled cases). Note, we have seen some
issues with configurations not updating if an SB was
submitted and quickly killed - if this happens, please
run clearCorrelator.evla.

No work on the hw side that
would impact observing tonight. New L band in EA21.
Scheduling bug fixed (was rounding to next hour rather
than second); should resolve 30 minute gap issues when
SBs are available.

----
Antennas:
  - EA23 barn

Correlator:
--
CM: 2011-08-03 20:44 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12,   14
quad4 =                       14
        - b108-b-5 never finishes programming; restarted but not
        rebooted

== Wednesday ==
OST through Thursday 0430 except:

        1300-2100 LS; /home/mchost/evla/scripts/opt/2011/08/TDEM0011_sb4889378_1.evla (LST time)

Stop at 0430 for development/testing.

- 11A-142_4864339_1.evla exploration; failure over the weekend but apparently this had run successfully before
- TRSR0041_sb4736269_1.evla (11A-218 test; 2 sec)


Joe

02 Aug

Hi,

Running updated CM version (as a trial; will roll-back as needed).

----
Antennas:
 - EA23 barn

Correlator:
--
CM: 2011-07-28 19:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12,   14
quad4 =                       14
       - b108-b-5 never finishes programming; restarted but not
       rebooted

== Tuesday ==
OST through Wednesday 0200 except:

1315-1415 L; /home/mchost/evla/scripts/opt/2011/08/10C-196_sb3898054_1.evla
(Gibbs check)
       - Note this should be manually stopped at 1429 (it's naturally
a 4 hour block!).

Stop at 0200 for maintenance day; note for data rate testing in the morning -

- 0430-0500 L; /home/mchost/evla/scripts/opt/2011/08/TRSR0040_sb4881652_1.evla
(11A-127 test; 3 sec)
  ...go to 4 sec as needed.
- 11A-142_4864339_1.evla exploration; failure over the weekend but
apparently this had run successfully before
- TRSR0041_sb4736269_1.evla (11A-218 test; 2 sec)


Joe

Callout: No fringes/d10 on 10C-220 (rolled back CM version; restarted CBE).

Jim called with a non-fringing/no-d10 program (10C-220).
It had the same issue as from the weekend (CBE configuration failure).

I rolled back the CM to 2011-06-07 (note: I had to restart the CBE afterward).

I'll send details to Sonja/Martin for hunting.

Joe

01 Aug

Hi,

Still running old version of CM.

----
Antennas:
 - EA23 barn

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12,   14
quad4 =                       14
       - b108-b-5 never finishes programming; restarted but not
       rebooted

== Monday ==
OST through Tuesday 0430 except:
       focuscheck observations through ~1330;

       1430-1630 X;
/home/mchost/evla/scripts/opt/2011/08/SC0113_sb4853126_1.evla (array
time)

       (if API is <=7 and wind is <=6):
       0100-0430 QKX
/home/mchost/evla/scripts/opt/2011/07/TCAL0001_sb4160177_1.evla
(calibrator models; array time)

Joe

July

29 July (Weekend)

Hi,

EA05 is back; sundry tests but no new versions; last night's CM version
will continue to be used.

----
Antennas:
 - EA23 barn

Correlator:
--
CM: 2011-07-28 19:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12,   14
quad4 =                       14
       - b108-b-5 out

== Friday ==
OST through 0530 Monday except:

       any 30 minute gaps from 1430-2100 (any time during the weekend):

/home/mchost/evla/scripts/opt/2011/07/11A-120_sb4830850_1.evla (array
time)

/home/mchost/evla/scripts/opt/2011/07/11A-120_sb4832084_1.evla (array
time)

/home/mchost/evla/scripts/opt/2011/07/11A-120_sb4832007_1.evla (array
time)

/home/mchost/evla/scripts/opt/2011/07/11A-120_sb4830773_1.evla (array
time)

/home/mchost/evla/scripts/opt/2011/07/11A-120_sb4830546_1.evla (array
time)

       fixed date:
       1800-2200 TDEM0013
               backup file:
/home/mchost/evla/scripts/opt/2011/07/TDEM0013_sb4856596_1.evla (array
time)

       2300-0100 LSCXKuKKaQ;
/home/mchost/evla/scripts/opt/2011/07/TCAL0004_4614060_1.evla (array
time; rf switch test)

       if API is <=7 and wind is <=6:
       0100-0430 QKX
/home/mchost/evla/scripts/opt/2011/07/TCAL0001_sb4160177_1.evla
(calibrator models; array time)

Joe

Callout: No fringes/d10 11A-254; (restart CBE; test and returned to schedule).
Callout: No fringes/d10 TDEM0013; (roll-back CM version, restart CBE; test and return to schedule).
Callout: OST not picking up SBs; (hand-generate files in preparation; problem worked around).
Callout: No fringes on 11A-142; project should have been on hold; abort and move on.

Two callouts:
- Dave caught 11A-254 misbehaving (no d10, fringes); CBE seemed
confused (no expected frames?); CM seemed to be okay but CBE Config
had failed.
I restarted the CBE did a quick C_quad test and Dave resumed.
- Dave then caught our fixed date (TDEM0013) doing the same thing; at
this stage I have to suspect our new version of the CM (it played coy
for a day plus it's the only thing that has changed recently). I've
rolled back to the 2011-06-07 version and restarted TDEM0013 (40
minutes in) and it seems to be working now.

We'll continue to watch...

Joe

Hi Joe,

See my comments, below.
- Show quoted text -
From what I can see in the CBE log files, the CBE did receive a configuration from the CM, but the CBE didn't configure itself. I suspect that this could be due to "missing" serial numbers for one or more baseline boards. There is no sign that the configuration failed due to an invalid configuration document.


This one is similar to the 11A-254 failure.

-- 
Martin

28 July

Hi,

Sundry tests; new version of CM to be tested overnight; we'll roll back
*if* there are issues. Michael found a bad baseline board which has
been removed from consideration (it never finishes programming; it
was restarted but not rebooted; this should be looked at tomorrow).
EA05 has a new X band receiver (but the FRM broke); EA21 has a new
Ku.

----
Antennas:
 - EA23 barn
 - EA05 FRM problem (keep out)

Correlator:
--
CM: 2011-07-28 19:46 UT/CBE: wcbe_perftrials (21 Jul; using mpiexec.mpd)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12,   14
quad4 =                       14
       - b108-b-5 never finishes programming; restarted but not
       rebooted
       - the quad 3 boards are old but I need to clear them with
       Michael/Ken

== Thursday ==
OST through 0430 except:

       Note: I've up'd the priority of the calibration model
       SBs which haven't been getting run; I reviewed with
       Tom and they are still not showing up in the OST; poking
       in the logs, it looks like it's being considered but
       not selected and I'm not sure why. So, do them
       manually tonight:

       0100-0430 XCL
/home/mchost/evla/scripts/opt/2011/07/TCAL0001_sb4158926_1.evla
(calibrator modesl; array time)
       or (if API is <=7 and wind is <=6):
       0100-0430 QKX
/home/mchost/evla/scripts/opt/2011/07/TCAL0001_sb4160177_1.evla
(calibrator models; array time)

Joe

27 July

No major upgrades during the day -- we tried one (CM) but backed away after
CrsroXrsro failed.

---------------------------------------------------
Antennas:
 - ea03 IF A's deformatter has been replaced -- should be more stable!
 - ea13 IF B's deformatter has been replaced (to correct CRC errors) --
   shouldn't affect observing in any obvious way

 - ea23 is in the barn

---------------------------------------------------
Correlator:
 CM: 2011-06-07 19:16 UT
 CBE: wcbe_perftrials (version of 7/18), with mpd
 MCAF: 1.4.2
 OST: 1.10.01 (20 Jul)
 TelCal 1.5.38 (12 Jul)

 unavailableBlBprs.txt:
   quad1 =
   quad2 =
   quad3 =                 12
   quad4 =

---------------------------------------------------

All OST science tonight.

1- We're hoping the OST will bring up 10C-145 (sb 4832460), which
 is C band (OK for rms<=45d, wind <= 100 m/s) and can start anywhere
 between 1330 and 1730 LST and runs for 1 hour.  If this does NOT come

 up in the OST, please run it anyhow -- there's a backup script in

 /home/mchost/evla/scripts/opt/2011/07/10C-145_sb4832460_1.evla



2- There are several half-hour blocks the OST is not finding for some
 reason.

 If:
    you have a 30-60 minute gap, and
    nothing shows up in the OST,
 please insert one of these by hand.


 All scripts are in
   /home/mchost/evla/scripts/opt/2011/07
 All are L band and so can be run in virtually any weather.
 All are 30mins long.

         Script               LST start range
   11A-120_sb4690106_1.evla   1500-2100 LST
   11A-120_sb4832329_1.evla   1500-0000 LST
   11A-120_sb4832254_1.evla   1500-2300 LST
   11A-120_sb4832161_1.evla   1500-2100 LST
   11A-120_sb4832084_1.evla   1430-2100 LST
   11A-120_sb4832007_1.evla   1430-2100 LST
   11A-120_sb4830850_1.evla   1430-2100 LST
   11A-120_sb4830773_1.evla   1430-2100 LST
   11A-120_sb4830546_1.evla   1430-2100 LST
   11A-120_sb4830475_1.evla   1500-2100 LST

Please stop Thursday morning at 0330 LST (~08:30am MDT).
- Hide quoted text -


Thank you much --

 Michael

p.s. Michael is the first line of defence for problems -- cell is probably
 best, 575 517-6797.

26 July

No major upgrades during the day -- more correlator testing; some progress
on data rates.


---------------------------------------------------
Antennas:
 - ea03 IF A's deformatter misbehaves on an irregular basis, leading to
     odd d10 output and strange fringe display results (sometimes this
     wipes out the fringe display for IF A for *all* antennas).
     -- Leave it in the array and hope...

 - ea23 is in the barn

---------------------------------------------------
Correlator:
 CM: 2011-06-07 19:16 UT
 CBE: wcbe_perftrials (version of 7/18), with mpd (see above)
 MCAF: 1.4.2
 OST: 1.10.01 (20 Jul)
 TelCal 1.5.38 (12 Jul)

 unavailableBlBprs.txt:
   quad1 =
   quad2 =
   quad3 =                 12
   quad4 =
       - had to manually re-phase b105-b-4 today but believe it should
         be ok now.
       - b106-b-1 seems okay but keep it out for now.

---------------------------------------------------

All OST science tonight.

1- We're hoping the OST will bring up 11A-178 (sb 4562999), which
 can stand any weather (despite being C,X bands) and can start anywhere
 between 1830 and 0200 LST and runs for 5 hours.  If this does NOT come
 up in the OST, please run it anyhow -- there's a backup script in

 /home/mchost/evla/scripts/opt/2011/07/11A-178_sb4562999_1.evla


2- There are several half-hour blocks the OST is not finding for some

 reason -- please insert these by hand where appropriate.  All scripts
 are in
   /home/mchost/evla/scripts/opt/2011/07
 All are L band and so can be run in virtually any weather.
 All are 30mins long.

         Script               LST start range
   11A-120_sb4690106_1.evla   1500-2100 LST
   11A-120_sb4691509_1.evla   1500-2100 LST
   11A-120_sb4722088_1.evla   1500-2100 LST
   11A-120_sb4529403_1.evla   1600-1930 LST

Please stop Wednesday morning at 0200 LST (~07:00am MDT).

Thank you much --
- Hide quoted text -


 Michael

p.s. Michael is the first line of defence for problems -- cell is probably
 best, 575 517-6797.

25 July

No major upgrades during the day -- just more correlator testing.

Martin believes the weekend problems were related to MPI (Message Passing
Interface) issues, and switched from hydra to mpd today.  We'll see how
we do -- please call Michael if there are problems tonight.

---------------------------------------------------
Antennas:
 - ea03 IF A's deformatter misbehaves on an irregular basis, leading to
     odd d10 output and strange fringe display results (sometimes this
     wipes out the fringe display for IF A for *all* antennas).
     -- Leave it in the array tonight
 - ea23 is in the barn


---------------------------------------------------
Correlator:
 CM: 2011-06-07 19:16 UT
 CBE: wcbe_perftrials (version of 7/18), with mpd (see above)
 MCAF: 1.4.2
 OST: 1.10.01 (20 Jul)

 TelCal 1.5.38 (12 Jul)

 unavailableBlBprs.txt:
   quad1 =
   quad2 =
   quad3 =                 12
   quad4 =
       - had to manually re-phase b105-b-4 today but believe it should
         be ok now.
       - b106-b-1 seems okay but keep it out for now.

---------------------------------------------------

All OST science tonight.

There are several half-hour blocks the OST is not finding for some
reason -- please insert these by hand where appropriate.  All scripts
are in

  /home/mchost/evla/scripts/opt/2011/07
All are L band and so can be run in virtually any weather.
All are 30mins long.

       Script               LST start range
 11A-120_sb4529478_1.evla   1600-1930 LST
 11A-120_sb4540681_1.evla   1530-1900 LST
 11A-120_sb4722223_1.evla   1500-2100 LST
 11A-120_sb4722423_1.evla   1500-2100 LST
 11A-120_sb4725956_1.evla   1500-2100 LST
 11A-120_sb4690106_1.evla   1500-2100 LST
 11A-120_sb4691509_1.evla   1500-2100 LST
 11A-120_sb4722088_1.evla   1500-2100 LST
 11A-120_sb4529403_1.evla   1600-1930 LST

Please stop Tuesday morning at 0400 LST (~09:00am MDT).

Here's hoping it's a calmer night --

  Michael

p.s. Michael is the first line of defence for problems -- cell is probably
 best, 575 517-6797.

22 July (Weekend)

No major upgrades during the day -- just more correlator testing.

---------------------------------------------------
Antennas:
 - ea23 barn

---------------------------------------------------
Correlator:
 CM: 2011-06-07 19:16 UT
 CBE: wcbe_perftrials (version of 7/18)
 MCAF: 1.4.2
 OST: 1.10.01 (20 Jul)
 TelCal 1.5.38 (12 Jul)

 unavailableBlBprs.txt:
   quad1 =
   quad2 =
   quad3 =                 12
   quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
         fine in operation; leave in.
       - b106-b-1; I believe this is okay (after multiple power
         cyclings but kept it out).

---------------------------------------------------

All OST science except:

 Saturday night
   1700-2300 LST: C band  10B-209 sb 4801245 [fixed date; should be OST-able]
     **this is with Palomar**

Also there are several half-hour blocks the OST is not finding for some
reason -- please insert these by hand where appropriate.  All scripts
are in
   /home/mchost/evla/scripts/opt/2011/07
All are L band and so can be run in virtually any weather.
All are 30mins long.

       Script               LST start range
 11A-120_sb4529478_1.evla   1600-1930 LST
 11A-120_sb4540681_1.evla   1530-1900 LST
 11A-120_sb4722223_1.evla   1500-2100 LST
 11A-120_sb4722423_1.evla   1500-2100 LST
 11A-120_sb4725956_1.evla   1500-2100 LST
 11A-120_sb4690106_1.evla   1500-2100 LST
 11A-120_sb4691509_1.evla   1500-2100 LST
 11A-120_sb4722088_1.evla   1500-2100 LST
 11A-120_sb4529403_1.evla   1600-1930 LST

Please stop Monday morning at 0530 LST (~10:30am MDT).

Have a good weekend --

   Michael & Joan

p.s. Michael is the first line of defence for problems -- cell is probably
 best, 575 517-6797.

Callout: Fringe stopped working (CBE failure).
Callout: Fixed date SB vanished from schedule (manually generated).
Callout: Same issue as before (Aborted and failed SBs - investigate CBE issue on Monday).

The CBE failed after the 13th scan of 10B-124 (Jul 22 20:53:57 in the CBE
log) -- Matt called me when the fringe display stopped updating.  Further
investigation showed that the CBE was receiving messages from the executor,
and frames from the BlBs, but nothing was being written out.  I asked Matt
to run a clearCorrelator; based on a misunderstanding on my part I also had
him re-start the Executor, though I believe now that was not necessary. At that point X_osro was happy and we returned to regular observing.

10B-124 is a single-configuration RSRO observation; we have run many similar
SBs in this project before, without any trouble.  I don't know what caused
the CBE to conk out this time.  Matt noticed that the API stopped
updating at about the same time, and called James to check into a possible
network issue just in case.

This particular observation was

 10B-124.sb4776374.eb4822617.55765.086551064815

I'm hoping Martin can explain all on Monday ;)

Onwards, cautiously --

    Michael

Just to clarify, the API was and is fine.  It's specifically the weather data from the CMP (temperature, dewpoint, etc) that stopped loading into both monitor databases at approx 20:30.  The loaders and databases are running normally and not reporting errors, and other monitor data like the API is loading fine.  I rebooted the CMP but there was no change. The real-time weather data is accessible from the Op interface, but the OST cannot access the wind speed, so we're manually overriding that.

I'm running the fixed date observation 10B-209.sb4801245 manually, after scrambling to create it via m2s (I started it on time though).  Unfortunately it vanished from the OST schedule when it came time to queue it.  I don't know why it disappeared, it was present in the OST schedule yesterday and today right up until when I ran Create Schedule about 15 mins before the 1700 LST start time. It still shows up in the OST Query tab as schedulable.

Cheers,
Matt

To summarize the Widar issues (mainly for operators), and update on the CMP issue:

Two Widar failures 18 hrs apart, at approx 20:53 yesterday and approx 14:47 this afternoon.  Michael and Martin have been investigating.  The cause is unknown at this time, and operators can expect it to occur again.  Current thinking is that it may be a Configuration Mapper problem, although it's odd that the CM software hasn't changed in over a month and the two observations that failed are projects we've run before.

For operators: the common symptom between the two is that the fringes stopped updating on calibrator scans.  In yesterday's case, the d10 continued to work and CBE continued to receive frames, in today's case both stopped.  Michael explained that this is because yesterday's script had only one configuration setup that never changed.  The script that warns of missing BDFs won't catch the current type of failure, I gather this is because the check needs a good scan to follow the bad ones, and in this case the observations have failed completely.

The two observations that failed:
1) 10B-124.4776374 (yesterday) - aborted script, marked SB as failed and released by Joan
2) AC982.4418872 (today) - aborted script, marked SB as failed
*--> this could probably be released?

On a separate note, Hichem fixed the CMP about an hour ago.  The dsc00 weather data is archiving again and the OST is picking up the wind speed.

Cheers,
Matt

21 July

No major upgrades during the day -- just more correlator testing.

---------------------------------------------------
Antennas:
 There was a power glitch during the day; some of the receivers are still
 hot.
 - ea01 X band cooling down
 - ea06 S,C bands cooling down
        C band IFs CD not working
 - ea08 Q band cooling down
 - ea23 barn


---------------------------------------------------
Correlator:
 CM: 2011-06-07 19:16 UT
 CBE: wcbe_perftrials (version of 7/18)

 MCAF: 1.4.2
 OST: 1.09.07 (28 Jun)
 TelCal 1.5.38 (12 Jul)

 unavailableBlBprs.txt:
   quad1 =
   quad2 =
   quad3 =                 12
   quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
         fine in operation; leave in.
       - b106-b-1; I believe this is okay (after multiple power
         cyclings but kept it out).

---------------------------------------------------

Tonight is all OST science except:

 1200-1300 LST: C band
   /home/mchost/evla/scripts/opt/2011/07/10C-145_sb4779497_1.evla
 1300-1400 LST: K band
   /home/mchost/evla/scripts/opt/2011/07/10C-145_sb4676033_1.evla

 1630-2300 LST: Q band
   /home/mchost/evla/scripts/opt/2011/07/11A-144_sb4576963_1.evla

Please stop at 0300 LST -- Martin will take the system to do more
correlator tests.

Enjoy --

    Michael & Joan

20 July

Hi,

In the absence of any specific instructions from schedsoc I am starting with OST at 12:15 LST and running until a reasonable quitting time in the morning (probably 3:00-4:00 LST).  OST has reported one fixed-date SB for TDEM0013.sb4703712 from 18:00-22:00 LST.

All of the antennas seem to be performing well.  The X-band receiver on ea01 is warm and will be so overnight.

A concern is that the weather station seems to have stopped reporting temperature and dewpoint.  Windspeed, barometer, and API RMS phase are all reporting/changing as expected, so I feel confident using OST.

I have no specifics on WIDAR but it seems to be behaving.

Cheers,

Tom Briscoe
VLA Operations

19 July

Hi,

Night between two maintenance days; many updates today
that are still being vetted. Some modest science is
believed possible.

----
Antennas:
 - EA23 barn

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (used throughout the day;
updated 7/18)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul); improved pointing efficiency (keep an eye
       on pointing results tonight!

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12
quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
       fine in operation; leave in.
       - b106-b-1; I believe this is okay (after multiple power
cyclings but kept it out).

I want to get in some C_quad test before we run...

== Tuesday ==
OST from 1300-1830 LST;
       1830-2330 X; /home/mchost/evla/scripts/operations/sysptgx.evla
OST from 2330-0100 LST;

       Stop at 0100 LST for start of double maintenance.

       Pending (notes for me in the scheduling):
       - TDEM0013/11A-144 combo - this Thursday
       - review of blockers for other demo science data


Joe

18 July

Hi,

Sundry tests throughout the day; two antenna moves and
an updated CBE.
EA21 is mostly cool after it's move but we'll push
the pointing run to tomorrow's mid-double-maintenance
day night (at Ken's advice).

----
Antennas:
 - EA21 mostly cool; needs pointing; use for all
 - EA10 FRM problem; out for the night
 - EA23 barn
 - EA13 RCP dead at K,Ka,Q

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (used throughout the day;
updated 7/18)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul); improved pointing efficiency (keep an eye
       on pointing results tonight!

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =                 12
quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
       fine in operation; leave in.
       - b106-b-2 is giving problems (see MR's note to widar-wg)

== Monday ==
OST except:
       1200-1600 S;
/home/mchost/evla/scripts/opt/2011/07/11A-231_sb4456035_1.evla (array
time)

       Submit both of these serially at 0015 (subarray test):
       0015-0100 C;
/home/mchost/evla/scripts/opt/2011/07/TVER0001_sb4670327_1.evla (array
time)
               - uses only antennas: 1,2,3,4,8,9,11,15,17,19,21,25,27,28
       0015-0100 Ku;
/home/mchost/evla/scripts/opt/2011/07/TVER0001_sb4669612_1.evla (array
time)
               - uses only antennas: 5,6,7,10,12,13,14,16,18,20,22,24,26

       Stop at 0100 LST for start of double maintenance.

       Pending (notes for me in the scheduling):
       - TDEM0013/11A-144 combo - this Thursday/Friday
       - review of blockers for other demo science data


Joe
p.s. Please note that I'm headed out of town starting Wednesday, 20
July - 27 July (back in the office on the 28th);
Michael and Joan will be the principal point of contact during this period.

15 July (Weekend)

Hi,

Observing all day; continue through the weekend. Short
30 minute break to review new C band in EA21 (Ken).
Add this for the weekend.

----
Antennas:
 - EA21 use for any observations not requiring L or Ku.
 - EA24 no X band; keep out for observations with X band.
 - EA03 IF A known to be unstable; please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (sundry changes)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul); improved pointing efficiency (keep an eye
       on pointing results tonight!

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
       fine in operation; leave in.

== Friday ==
OST except:
       - look for 10C-145 and 11A-277; if we don't see these tonight;
please e-mail
       me and I'll prep back-ups for tomorrow.

== Saturday ==
OST except:
       2100-2400 /home/mchost/evla/scripts/operations/syspt2hx.evla
(must have 21&28 or skip)
       0000-0100 /home/mchost/evla/scripts/opt/2011/07/TSPE0012_sb470858_1.evla
               - array time; galactic HI test
       if Ka-ish weather; Wind<=7; API<=13:
               1000-1600 Ka;
/home/mchost/evla/scripts/opt/2011/07/TDEM0006_sb4525008_1.evla (array
time)

== Sunday ==
OST except:
       1530-1700 K;
/home/mchost/evla/scripts/opt/2011/07/11A-218_sb464681464_1.evla
(array time)
               - experimental; please call Michael Rupen when this
starts and he will monitor
               throughout the program.

== Monday ==
OST:
       stop at 0300 LST for testing (turn over to Martin/James)

If there is a 30 minute gap anywhere please run:
       /home/mchost/evla/scripts/opt/2011/07/TPHA0001_sb4699418_1.evla
(C band; array time)
               - only need this once!

       Pending (notes for me in the scheduling):
       - longer integrations for 11A-129 worked; ready to go
       - TDEM0013/11A-144 combo - next Thursday
       - review of blockers for other demo science data
       - Monday/Tuesday subarray test (noon/morning)


Joe

14 July

Hi,

EA21 partially ready; will be used in limited capacity tonight.

----
Antennas:
 - EA21 use for any observations not requiring C, L or Ku.
 - EA24 no X band; keep out for observations with X band.
 - EA03 IF A known to be unstable; please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (sundry changes)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul); improved pointing efficiency (keep an eye
       on pointing results tonight!

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
       fine in operation; leave in.

== Thursday (62481)  ==
OST except:
       1100-1500 S;
/home/mchost/evla/scripts/opt/2011/07/11A-231_sb4456035_1.evla (array
time)
               - second try without pre-breakage!

== We'll be observing throughout the day on Friday ==

For Friday morning we have:
       0130-1130 L;
/home/mchost/evla/scripts/opt/2011/07/TCAS0001_sb4554400_1.evla (LST
time)
               - set for 62482 @ 0130 LST

       Pending (notes for me in the scheduling):
       - paused on TPHA0001 (pending MR comment)
       - longer integrations for 11A-129 worked; ready to go
       - X-band pointing run (must include 21,28)
       - TDEM0013/11A-144 combo - next Thursday
       - review of blockers for other demo science data


Joe

13 July

Hi,

Maintenance day; recovery began a bit early (yay!). Some
testing of high frame rates (but modest data rates) by
Michael indicated the particular mode could not be
maintained long term; more follow-up planned.

----
Antennas:
 - EA21 out
 - EA24 no X band; keep out for observations with X band.
 - EA03 IF A known to be unstable; please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (sundry changes)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul); improved pointing efficiency (keep an eye
       on pointing results tonight!

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
       fine in operation; leave in.
       - b105-b-2 looks okay today.

== Tuesday (62480)  ==
OST except:
       1130-1530 S;
/home/mchost/evla/scripts/opt/2011/07/11A-231_sb4456035_1.evla (array
time)

       Postpone phased array test for tonight.

       Stop at 0300 LST for testing.

Joe

Callout: 11A-231 not fringing; (stop file; restart); ended up being the 'stress' test residual.

11A-231 has acted up again (though in a different way than before).
No d10/fringe display; CBE was confused (see below); I have a mild
worry that the
'stress test' CrsroXrsro might have left things in a wonky state (I
received messages
about configurations for it during 11A-231:
Jul 13 17:13:13 cbe-control (wcbe) bdf_mdata_proxy: DEBUG:
CrsroXrsro_3C345_000.55755.96322019676.6 entered expired state
Jul 13 17:13:13 cbe-control (wcbe) bdf_mdata_proxy: DEBUG: Completed
CrsroXrsro_3C345_000.55755.96322019676.6

We stopped at 1200 LST and will let the OST go from here. We'll need
to investigate tomorrow.

Joe

Jul 13 17:12:46 cbe-control (wcbe) executor_listener: DEBUG: 1331+305=3C2863.53925777760.53248521090.00.00.00.00.47790869999153074ObserverName=Dr.
Emmanuel MomjianProjectID=4127696SBID=4456035ScanIntent="OBSERVE_TARGET"SBTYPE=OBSERVERObsCode=11A-231CalibratorCode="
"11widar2349.02349.0
Jul 13 17:12:46 cbe-control (wcbe) executor_listener: DEBUG: command:
start_subscan -d 11A-231_sb4456035_2.55755.9671662037 -c
11A-231_sb4456035_2.55755.9671662037.2 -t telcal 55755.96718402778 1 1
(16759)
Jul 13 17:12:46 cbe-control (wcbe) executor_listener: DEBUG: 1331+305=3C2863.53925777760.53248521090.00.00.00.00.4791006118321093ObserverName=Dr.
Emmanuel MomjianProjectID=4127696SBID=4456035ScanIntent="CALIBRATE_AMPLI"SBTYPE=OBSERVERObsCode=11A-231CalibratorCode="E"21widar2349.02349.0
Jul 13 17:12:46 cbe-control (wcbe) executor_listener: DEBUG: command:
start_subscan -d 11A-231_sb4456035_2.55755.9671662037 -c
11A-231_sb4456035_2.55755.9671662037.2 -t telcal 55755.96837268519 2 1
(16760)
Jul 13 17:12:46 cbe-control (wcbe) start_subscan: DEBUG: start_subscan
11A-231_sb4456035_2.55755.9671662037
11A-231_sb4456035_2.55755.9671662037.2 * 1310598764.700000 0/1/1/
bunker archive telcal
Jul 13 17:12:46 cbe-control (wcbe) bdf_mdata_proxy: DEBUG: received
11A-231_sb4456035_2.55755.9671662037
11A-231_sb4456035_2.55755.9671662037.2 * 1310598764.700000 0/1/1/
bunker archive telcal
Jul 13 17:12:46 cbe-control (wcbe) bdf_mdata_proxy: DEBUG:
11A-231_sb4456035_2.55755.9671662037.2 entered pending state
Jul 13 17:12:46 cbe-control (wcbe) start_subscan: DEBUG: start_subscan
11A-231_sb4456035_2.55755.9671662037
11A-231_sb4456035_2.55755.9671662037.2 * 1310598867.400000 0/2/1/
bunker archive telcal
Jul 13 17:12:46 cbe-control (wcbe) bdf_mdata_proxy: DEBUG: received
11A-231_sb4456035_2.55755.9671662037
11A-231_sb4456035_2.55755.9671662037.2 * 1310598867.400000 0/2/1/
bunker archive telcal
Jul 13 17:12:46 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 39
Jul 13 17:12:47 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 40
Jul 13 17:12:49 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 41
Jul 13 17:12:49 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 42
Jul 13 17:12:50 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 43
Jul 13 17:12:51 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 44
Jul 13 17:12:52 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 45
Jul 13 17:12:53 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 46
Jul 13 17:12:54 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 47
Jul 13 17:12:55 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 48
Jul 13 17:12:56 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 49
Jul 13 17:12:57 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 50
Jul 13 17:12:58 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 51
Jul 13 17:12:59 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 52
Jul 13 17:13:00 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 53
Jul 13 17:13:01 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 54
Jul 13 17:13:02 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 55
Jul 13 17:13:04 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 56
Jul 13 17:13:05 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: add_integration 57
Jul 13 17:13:05 cbe-control wcbe_bdf_mdata[15548]: ** DEBUG: mpi_file_close
Jul 13 17:13:05 cbe-control wcbe_bdf_mdata[15548]: ** INFO:  
/lustre/evla/wcbe/data/telcal/uid____evla_bdf_1310598662265
 uid:///evla/bdf/1310598662265
CrsroXrsro_3C345_000.55755.96322019676
6  1
58  692814221

Jul 13 17:13:13 cbe-control (wcbe) bdf_mdata_proxy: DEBUG:
CrsroXrsro_3C345_000.55755.96322019676.6 entered expired state
Jul 13 17:13:13 cbe-control (wcbe) bdf_mdata_proxy: DEBUG: Completed
CrsroXrsro_3C345_000.55755.96322019676.6
Jul 13 17:14:27 cbe-control (wcbe) executor_listener: DEBUG: 1331+305=3C2863.53925777760.53248521090.00.00.00.00.48048924135946436ObserverName=Dr.
Emmanuel MomjianProjectID=4127696SBID=4456035ScanIntent="CALIBRATE_AMPLI"SBTYPE=OBSERVERObsCode=11A-231CalibratorCode="E"31widar2349.02349.0

12 July

Hi,

Short day of testing focused on some of the issues from last
night; several new versions of underlying software:

----
Antennas:
 - EA21 out
 - EA24 no X band; keep out for observations with X band.
 - EA03 IF A known to be unstable; please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_perftrials (sundry changes)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.38 (12 Jul); improved pointing efficiency (keep an eye
       on pointing results tonight!

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =          5
quad4 =
       - Note: b101-t-3 did not sync for the CRM tests but looks
       fine in operation; leave in.
       - b105-b-2 had a low frame rate and seems to have an issue
with LTA on X1-Y6;
       removed from consideration.

== Tuesday (62479)  ==
OST:

       Stop at 0030 LST for maintenance day.

Joe

11 July

Hi,

Power outage today caused a shutdown in the correlator; much
recovery time from Ken/Michael on this.

----
Antennas:
  - EA21 out
  - EA24 no X band; keep out for observations with X band.
  - EA22 A (band switch issue)
  - EA16 C (deformatter)
  - EA03 IF A known to be unstable; please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110624.0 [mpiexec.mpd]
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
        - Note: b101-t-3 did not sync for the CRM tests but looks
        fine in operation; leave in.

If there is a 30 minute gap in the range below on any night; please run (we've noticed
30 minute gaps might have issues in the scheduling so these are backups):
        1600-1930 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4529403_1.evla (array time; 30 min)
        1600-1930 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4529478_1.evla (array time; 30 min)
        1530-1900 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4540681_1.evla (array time; 30 min)
        1600-1900 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4128824_1.evla (array time; 30 min)


== Monday (62478)  ==
OST except:
        1130-1230 C; /home/mchost/evla/scripts/opt/2011/07/10C-145_sb4668422_1.evla (array time)
        1800-2200 CXK; /home/mchost/evla/scripts/opt/2011/07/TDEM0013_sb4652860_1.evla (array time; demo science)
        2230-2330 C; /home/mchost/evla/scripts/opt/2011/07/StokesV_50Hz_8jul11.evla (array time; dopset run on 8 Jul)

        Stop at 0300 LST on Monday morning.

        Reminder for me (run TDEM0013 associated program tomorrow night).

Joe
Callout: No fringes/d10; (restart CBE).
Callout: Missing BDFs; (only 2; talk with Bryan, decide to continue).

Sam called after catching some missing BDF messages.
It looks like a new issue. d10 was working, CM and CBE status were
purportedly happy
but the CBE logs told a bit of a different story; logs of successful
integrations were happening
very sporadically with lots of surrounding message passing (mostly
patting itself on the back).
The BDFs seemed not to be getting written (those that made it to
/lustre/evla/data/bunker were not
written regularly and the volume/content of the BDFs had dropped).
Based on this, I restarted the CBE - it did find 2 zombies; I re-ran
the same file - it's a fixed
date (in case it was salvageable) which skipped to the current location.
We'll need to review this tomorrow; 11A-120 may also be affected...

Thanks to Sam for his vigilance (and Bryan for assuring us of the problem!). :P
Joe

Further update; Bryan saw further missing scans after the restart.
There are 3 missing scans however it isn't incrementing and we
confirmed that we are getting
new BDFs for the most part (only scans 24,25,26 are the missing ones).

As it is still chunking, I didn't take any other action and we'll
continue to monitor...

Joe

08 July (Weekend)

Hi,

Ken re-ran the CRM tests without the tricky, high-stringency one.
All boards look okay except b107-b-2 (whose chip is not used in
standard operation). Enabling all boards for the weekend.

----
Antennas:
  - EA21 out
  - EA24 no X band; keep out for observations with X band.
  - EA03 IF A known to be unstable; please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110624.0 [mpiexec.mpd]
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
        - Note: b101-t-3 did not sync for the CRM tests but looks
        fine in operation; leave in.

If there is a 30 minute gap in the range below on any night; please run (we've noticed
30 minute gaps might have issues in the scheduling so these are backups):
        1600-1930 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4529403_1.evla (array time; 30 min)
        1600-1930 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4529478_1.evla (array time; 30 min)
        1530-1900 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4540681_1.evla (array time; 30 min)
        1600-1900 L; /home/mchost/evla/scripts/opt/2011/07/11A-120_4128824_1.evla (array time; 30 min)


== Friday (62475)  ==
OST:
        Fixed date for 11A-224

== Saturday (62476) ==
OST except:
        0630-0930 Ka+; /home/mchost/evla/scripts/opt/2011/07/11A-263_sb4564979_1.evla (array time)
                only if Ka weather (API<=7, wind<=6); this is high priority but in a tough time/season
                for getting on so I wanted to see if we can get it with 'close' weather conditions.
                Sunday is expecting storms but we'll keep an eye to see if this is remotely possible
                then as well.

== Sunday (62477) ==
OST except:
        0930-1430 LC; /home/mchost/evla/scripts/opt/2011/07/11A-142_sb4616455_1.evla (array time)

        if there is a 30 minute gap in the range below; please run:
        2200-0000 range; C; /home/mchost/evla/scripts/opt/2011/07/TDEM0009_sb4617266_1.evla (array time; 30 minutes)
        0000-0130; C; /home/mchost/evla/scripts/opt/2011/07/TDEM0009_sb4635086_1.evla (array time)
                - Demo; summer student

== Monday (62478) ==
OST:
        2230-2330 C; /home/mchost/evla/scripts/opt/2011/07/StokesV_50Hz_8jul11.evla (array time; 8 Jul)

        Stop at 0500 LST on Monday morning.

        Reminder (for me): TDEM00013_sb4652860_1.evla (for Monday night!)

Joe

07 July

Executor update led to problems
Hi,

Lots of phased array testing; follow-up on that tomorrow.
CRM results still being discussed so no changes on allowed
boards; C_quad tests look good.

----
Antennas:
 - EA21 out
 - EA24 no X band; keep out for observations with X band.
 - EA03 IF A fixed again (but know to be unstable); please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110624.0 [mpiexec.mpd]
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Thursday (62474)  ==
OST:

       Stop at 0300 LST on Friday morning.

If there is a 15 minute gap please run:
/home/mchost/bbutler/holography/THOL0001_sb4538750_1_new_fast.evla


Joe

Callout: Holography needs to run (yes; prepped and scheduled with Bryan)

Hi Bryan,
Yes, we thought we could do one part but I didn't get any info
after that. This has been something of a comedy of errors
in communication.

I think this can run from 2230-0300; Bryan if you think it's ready
to go, we can try it - do we need any information from your 15
minute test?

I just confirmed the slot with Matt so it will run tonight (though they
still didn't set the LST range - 0000-2400 - it's 3C147 so it should be
good as specified; and it's 4.6 hours which I noted was inconvenient
but we'll end with this so it's okay).

Joe

06 July

Hi,

Maintenance day recovery. CRM tests done today but with
non-convergent answers (Ken/Michael are pondering this
so no change in unavailable boards). Holography tests
in both directions today; still showing the saw-toothed
behavior in amplitudes; need to follow up on this; long
holography test delayed until after tomorrow's testing.

----
Antennas:
 - EA21 out
 - EA24 has L band still cooling; X band dodgy; please keep in.
 - EA03 IF A fixed again (but know to be unstable); please keep in.


Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110624.0 [mpiexec.mpd]
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Wednesday (62473)  ==
OST:

       Stop at 0300 LST on Wednesday morning.

Joe

Ah, spoke to soon; last C_quad exposed a troubled board: b107-b-2;
It is showing a bad LTA on X5-Y6; I tried a soft restart but got a
communication error:


Server Communications Error while Programming all devices:
HttpClientLink.query Exception:
IO Failure with URL:
'http://b107-b-2/mah?%3Csupervisor+hwComponentState=%27program%27%3E%3Cstate+fpgaSource=%27%27%2F%3E%3C%2Fsupervisor%3E'
query: '' --
connect timed out
 java.net.PlainSocketImpl.socketConnect(Native Method)
 java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 java.net.Socket.connect(Socket.java:529)
 sun.net.NetworkClient.doConnect(NetworkClient.java:158)
 sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
 sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
 sun.net.www.http.HttpClient.(HttpClient.java:233)
--
Our query XML: 
--

I'm forgoing further intervention in case it's a repeat of the old
problem (so Bruce has a live example).
I've removed it from consideration for tonight.

Joe

05 July

Hi,

Tests on fast data rates today. All boards seem to be
working (Ken fixed up one bad board).

----
Antennas:
 - EA21 out; EA03 IF A is still dodgy (deformatter instability).

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110624.0 [mpiexec.mpd]
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Tuesday (62472)  ==
OST except:

       Stop at 0030 LST on Wednesday morning.

Joe

Callout: No fringes/d10 (put project on hold; repeatable issue but not clear the origin; investigate during the day)

Dave called due to a problem with 11A-231; no d10 display, no fringes.

Indeed, when we looked at it, the CBE did not seem to have any active
configurations
(frame rate realized/expected was 0/0).
We went back and did a quick check on C_quad1 which worked well.
I went back to 11A-231 (after skipping the first few scans) and found
that the CM
was never getting a response from the CBE (see attached image from the VCI
configuration mapper). I'm not sure what's special about this file but
we'll need to
look at this tomorrow - for now, this project is on-hold.

Joe

01 July (Weekend)

Hi,

Sundry tests today; CBE issue from last night required we roll
back for the weekend; both the CBE version and the underlying
MPI process manager have been changed.

----
Antennas:
 - EA21 out; EA03 IF A is still dodgy (deformatter instability).

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110624.0 [mpiexec.mpd]
MCAF: 1.4.2
OST: 1.09.07 (28 Jun) - not sure of the status here; there seemed to be
       some problems last night so this might be an update on yesterday's.
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Friday (62468)  ==
OST except:
       1400-2030 C;
/home/mchost/evla/scripts/opt/2011/06/10B-209_sb4603030_1.evla
(coordinated; backup)

== Saturday (62469) ==
OST except:
       1800-2200 CXK;
/home/mchost/evla/scripts/opt/2011/06/TDEM0013_sb4559640_1.evla (Demo;
stride)

== Sunday (62470) ==
OST except:
       1530-2200 QX;
/home/mchost/evla/scripts/opt/2011/06/11A-144_sb4576627_1.evla (coord)

== Monday (62471) ==
OST except:
       2300-0100 *;
/home/mchost/evla/scripts/opt/2011/06/TCAL0004_sb4614060_1.evla (RF
switch test)

== Tuesday (62472) ==
OST:

       Stop at 0430 LST on Tuesday morning.

Joe

June

30 June

Lots of data rate, fast dump checks. System looks good.
New CBE version but not using tonight.

----
Antennas:
 - EA21 out

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_mpidebug (20110627)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun) - not sure of the status here; there seemed to be
       some problems last night so this might be an update on yesterday's.
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Thursday (62467)  ==
OST except (both should come up in the OST but if not):
       1400-2000 (fixed date 10B-209); C;
/home/mchost/evla/scripts/opt/2011/06/10B-209_sb4602807_1.evla
(coordinated; backup)
       2000-0100 CX;
/home/mchost/evla/scripts/opt/2011/06/11A-178_sb4399956_1.evla
(coordinated; backup)
       Stop at 0230 for testing.

Joe

29 June

Hi,

Maintenance day; lots of work on antennas; EA03 IF A was
fixed but seems to be unstabled (currently working). EA28
focus resolver was replaced; Ken backed out of his adjustments
and we're doing a focuscheck now. EA24 had an FRM issue but
this was just resolved.

Running through the C_quad checks, it seems the CBE was in
a bit of an odd state (single Node/NIC); I reinitialized
the whole CBE and things seem to be performing well.

----
Antennas:
 - EA21 out

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_mpidebug (20110627)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun) - not sure of the status here; there seemed to be
       some problems last night so this might be an update on yesterday's.
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Wednesday  ==
OST:
       Stop at 0230 LST for testing Thursday morning.

Joe

28 June

Hi,

Some 3-bit testing and ongoing investigations from the BlB/CBE issues from
the weekend. New version of OST to fix the problems from last night; Matt
noted that there is still a problem here (Keith is working on it).

Running through the C_quad tests, I'm seeing problems with:
b101-t-2

Rather than restarting/rebooting this; I'm removing it from
consideration to see if this is a possible repeat of Friday's
issues (rather than another effing issue).

----
Antennas:
 - EA21 out; others as above.

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_mpidebug (20110627)
MCAF: 1.4.2
OST: 1.09.07 (28 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =   1,
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Tuesday  ==
OST except:
       /home/mchost/evla/scripts/opt/2011/06/TRSR0038_sb4589810_1.evla
(can start between 0830-1730; 1 hour SB; LC band)
       /home/mchost/evla/scripts/opt/2011/06/11A-239_sb4278317_1.evla
(can start between 1430-1800; 1.5 hour SB; C band)

       Stop at 0000 for maintenance day.

please observe the following if there are any 30 minute gaps (once per day):
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla
(array time; test)

Joe

Thanks Ken.
I just didn't want to erase clues if it was helpful; we've got a good
spate of boards for
tonight so I can re-start it in the morning.

Joe
- Hide quoted text -

On Tue, Jun 28, 2011 at 5:36 PM, Ken Sowinski  wrote:
> On Tue, 28 Jun 2011, Joseph P. McMullin wrote:
>
>> Running through the C_quad tests, I'm seeing problems with:
>> b101-t-2
>
> This is probably a remnant of testing of b101-t-3 that we were
> doing with Brent this morning.  Re-starting it would likely
> clear things up.
>
> Ken

27 June

Hi,

Some investigations from today on issues from the weekend:
- EA28 focus zero point offset; fixed by Ken
- EA03 unstable deformatter; please keep out of the array tonight
- EA08 problems but seems fine now; please keep in.
- EA27 large delay changes caused by faulty L305; status? keep in for now.

----
Antennas:
 - EA21 out; others as above.

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_test (New - 20110623!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =                          12
quad2 =                  9,10,            15
quad3 =
quad4 =                8,  10,   12,13,14,15  #

C_quadx were run by Michael; I don't have any updates on this so
I've retained the very conservative list.

== Monday  ==
OST:

       Stop at 0200 for testing.

please observe the following if there are any 30 minute gaps (once per day):
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla
(array time; test)

Joe

Callout: OST not working (manually prepped two files; Matt did the rest after it became apparent there would be no fix for the night).

There's a problem with the OST that Keith will fix tomorrow (the OST server scheduling algorithm goes into an infinite loop when an SB matches the End LST parameter).  I've come up with a conservative schedule for the night that is good for winds less than 15 m/s and API < 30 deg.  (Forecast is for gusty winds averaging up to 7m/s, and a 10% chance of rain.)

The plan is to run these as much as possible through the OST but if the infinite loop is triggered, then we'll run the scripts manually.  I've created the scripts for these SBs (in array time) in /home/mchost/evla/scripts/opt/2011/06

AL746.sb4403468 was run manually without the OST
11A-266.4519383 14:20-15:50 (ToO observation, running this now via OST)
11A-138.sb4216744,X:    62464 15:50:01 - 16:20:01 (accepted via OST)
SB0514.sb4464855,X:     62464 16:20:01 - 17:20:01
11A-123.sb4261605,X:    62464 17:20:01 - 17:50:01
11A-120.sb4540681,L:    62464 17:50:01 - 18:20:01
11A-199.sb4499892,L, C: 62464 18:30:00 - 22:30:00
11A-225.sb4242452,L:    62464 22:30:00 - 01:30:00

TVER0001_sb4045730_1 can fill in the half-hour gap bringing us to the 02:00 LST stop time for testing.

- Matt

24 June (Weekend)

Hi,

A new version of the CBE is in place; a number of issues
were worked through earlier in the data (see evlatests
for details).
One issue that recurred upon the transition from testing
to observing; several BlBs were getting unexpectedly low
rates. Looking at the boards there were errors over many
of the CCs. I tried to clear errors and rephase to no avail.
I then tried to restart one of the errant boards but it
immediately lost communication with the board; this then
blocked the CBE which couldn't get the serial number.
Eventually, after talking with Ken/Michael, I rebooted
the boards in question and they seemed to work fine. Apparently
this occurred earlier in the day as well. Unfortunately the
problem only happened with boards in quadrants 1 & 2 or
we would have left it.
The boards were:
b101-b-4,5
b102-b-4,5
b103-b-6,7

----
Antennas:
 - all in (except EA21); X band RCP on EA19 is weak;
       EA13 A/C (L302) X band was fixed and seems to be working.
Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_test (New - 20110623!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Wednesday ==
OST except!:
Friday
       1630-2300 Q;
/home/mchost/evla/scripts/opt/2011/06/11A-144_sb4128868_1.evla
Saturday

Sunday
       1930-0530 C;
/home/mchost/evla/scripts/opt/2011/06/SC0489_sb4289467_1.evla (backup
for FIXED DATE)
       0830-0930 CL;
/home/mchost/evla/scripts/opt/2011/06/TRSR0038_sb4422365_1.evla

Monday
       Stop at 0200 for testing (likely a 1 hour maser check).

please observe the following if there are any 30 minute gaps (once per day):
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla
(array time; test)

Joe

Callout: Not fringing (early anxiety; things were working okay)
Callout: Fixed data times looked incorrect (went through schedule and it was okay in the end).
Callout: Not fringing (BlBs not behaving (effing problem); removed from consideration for observing).
Callout: X_osro not fringing (EA27 behaving oddly; removed and things looked okay).
Callout: Missing BDFs (on SN project; thought better to have some than no data; kept running)
Callout: Missing BDFs (on test program - high data rate - ignore; we'll evaluate on Monday).

23 June

Hi,

A number of changes today (2nd of double maintenance).
New PCMC firmware installed. New StB s005-b-6 (replaced from rack 8);
new BlB 108-b-6; thanks to Kerry; my understanding is that we
now have *no* spares of either StBs or BlBs.
EA16 has a new X (works okay); EA03 has a new L (works okay).
Continue with last night's CBE version.

----
Antennas:
  - all in (except EA21)

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_test (New - 20110622!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Wednesday ==
OST !:
        Note there is a fixed date observation at 1800 - I'm a little worried as the
        start LST range does not reflect this?? Please let me know if this doesn't
        show up in the schedule for tonight.

        Stop at 0200 for testing (likely a 1 hour maser check).

please observe the following if there are any 30 minute gaps:
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Joe

22 June

Hi,

Yet another update to the CBE (tonight to the underlying MPI used).
So far, no issues with the only death script; standard scripts also
appear to be running fine so given it is between two maintenance
days, we'll run with it tonight.
Operator band checkouts and C_quads all look good.

----
Antennas:
 - all in (except EA21)

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_test (New - 20110622!)
 - Note: it looks like we're still pointed at yesterday's - I'll
check with Martin.
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Wednesday ==
OST !:
       Stop at 2330 for maintenance day.

please observe the following between 0900-1800 LST (force if necessary):
Ku; /home/mchost/evla/scripts/opt/2011/06/TCAL0003_sb4557938_1.evla (30 minutes)

please observe the following if there are any 30 minute gaps:
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla
(array time; test)

Joe

21 June

Hi,

A new*er* version of the CBE has been installed with improvements
on the handling of the deconfigurations; we'll run with it tonight and
fall back to 20110607.0 if there are problems.

----
Antennas:
 - all in (except EA21); some band-switchy issues with C band.

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_test (New - 20110621!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
       - Note: C_quads look good but I'm retaining the conservative
list of problem
       boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST !:
       Fixed date observing from 1330 until maintenance at 0030.

please observed the following if there are any 30 minute gaps (not
essential tonight):
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla
(array time; test)

Joe

Callout: OST won't schedule fixed date observation (generated it manually).

Keith, et al...

At approximately 12:50 LST (8pm local tonight) I attempted to accept a scheduling block for SC0666 - a fixed-block SB due to run from 13:30-00:30 LST.  It had showed up in OST as expected the last time I had created a schedule. When I attempted to accept it, I was greeted with the common error about attempting to schedule something in the past.  I rechecked the LST day and time and both were correct; I was trying to schedule something for 40 minutes in the future, not the past.

I attempted to recreate the schedule; OST appeared to work but the schedule it gave me did *not* include the fixed-time SC0666 SB.  I terminated and restarted OST and again created a schedule, but again, the schedule it gave me did not include SC0666.

I called Joe McMullin, who subsequently created a file for me to load at 13:30 manually.  As this is an 11-hour file I have no further need of OST tonight, and we go into maintenance day immediately after SC0666.  But we will probably want it tomorrow after maintenance.

Cheers,

Tom Briscoe
VLA Operations

Hey Tom.
Here is what happened:

At 2011-06-21 23:26:12 GMT a schedule was generated with two blocks:
1. AL746.sb4403468.eb4477273 @ 10:30:00 (VLA day #62458) to 13:30:00 (VLA day #62458) (Dynamic)
2. SC0666.sb4113490.eb4553944 @ 13:30:00 (VLA day #62458) to 00:30:00 (VLA day #62459) (Fixed Date)
At 2011-06-21 23:26:41 GMT AL746.sb4403468.eb4477273 was accepted and sent to the Executor.
At 2011-06-22 02:00:00 GMT SC0666.sb4113490.eb4553944 was attempted to be accepted but failed because "You attempted to schedule something in the past."

Here's why:
When AL746.sb4403468.eb4477273 was scheduled an estimated end time was used, but when it was accepted an actual end time was calculated.  They were slightly different (by less than 1 second).  Unfortunately, we didn't have 1 second to give in the schedule and since the next block was Fixed Date it could not be scooted 1 second later in order to fit.  Officially, AL746.sb4403468.eb4477273 ended at 13:30:00.00021723815553995496 (VLA day #62458) and SC0666.sb4113490.eb4553944 had a start time of 13:29:59.99935087260186155496 (VLA day #62458).  As you can see, they overlap so the Fixed Date one reported that it could not be scheduled in the past.

What you could have done to work around this issue:
When the OST says that "You attempted to schedule something in the past." you can reset its "memory" of what it has already run by moving the LST Start time into "the past".  (Click Reset LST Range then move the time back at least 1 second and create a new schedule.  Say "yes" to the dialog that pops up.)  This will allow you to generate a schedule starting "now", irregardless of what has already been sent to the executor.

I'll give some thought to how we can avoid this situation entirely in the future.

Thanks,
Keith

20 June

Hi,

A new version of the CBE with improved logging and a potential
resolution to some of the missing scan issues; initially the
'death' script of CrsroXrsro was running well, however, recently,
it has been dropping BDFs so more work is needed. I have a call
in to Martin to discuss sticking with this version but based
on the improved logging and the believe that we're no worse
than before, I'm moving ahead with the current version.

Update from Friday; Joan updated all 11A programs with the
correct Program Block priority.

----
Antennas:
  - all in (except EA21)

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_test (New - 20110620!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST !:
        Stop at 1500 LST for further manual V Stokes testing; Michael will be
        leading this and will turn things back between 1600 and 1700 LST.
        After this, more dynamic scheduling stopping at 0130 LST (~0900 MDT)

please observed the following if there are any 30 minute gaps:
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Joe

17 June (Weekend)

Hi,

A good day of testing in sundry areas (V Stokes, Phased array,
holography, Cyg A gain compression/expansion, etc).
All antennas look good (except EA19 X band is warm).
All boards look good.

Joan found an issue with the default Program block priorities
that is in the process of being fixed; this has skewed the
OST priorities toward older (i.e., prior cycle) projects.
This will be fixed throughout the next day.

----
Antennas:
  - all in (except EA21); EA19 X band - keep in for all
  programs except single band X projects.

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110610.0 (New!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Friday-Monday morning 0330 LST (~1100 MDT) ==

Following the TSPE0008 Vstokes test (~1030 LST); all OST throughout the weekend;
please observed the following if there are any 30 minute gaps (only once per day):
C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Joe

Callout: Data not showing up in the archive (CBE had gone down; reset, tested and resumed).
Callout: No fringe display on a set of programs (looked like scrambled abort had compromised the system; restarted CBE to clear old active configurations; review problem data sets).

Tom called when 11A-260 didn't end up in the archive.
I looked and found that we hadn't taken data in a bit (looking at the BDFs on lustre).
I found that the CBE was down 1 NIC (4th on node 8) but otherwise looked okay from a status; however,
it was clear from the logs that we'd stopped taking data between 1855 and 1856 (looks like the transition
between the two 11A-260 and AL746 - AL746 never took data).
I restarted the CBE and things look okay now.
I'm turning things back to Tom but will keep an eye on it.

Bryan, please ignore my call - not an archive issue...

Joe

Sam called ~0330 due to the fact that we weren't getting fringes going into the second project
(11A-228 had none and 11A-120 was also not fringing).
We found that things seemed to be working for the CM/CBE; the only oddity was there
looked like some left over active configurations from 11A-231 that were showing up in
the wcbetool (more expected than received due to this); the wcbe_master.log looked like it
was integrating okay.
We did a quick check on an X_osro and it looked like things were working though TelCal
was only getting 3/4 IFs.
We looked back on Fringe and found that the last several projects had some oddities
(missing IFs BC) through to the 11A-231 project.

So, we'll need to have a quick review on the following tomorrow:
11A-123 (Michael since he's on that) and perhaps an 11A-175 or 11A-182 for good measure
(there were issues with winds prior to this which caused some failures due to antenna stows).

I did bring the CBE down and reset it to clear out the old configurations, we re-ran an X_osro 
and all seems well so we're moving on.

Currently planning to stop at 1100 local time unless Martin can put the system to good use
prior to this (I'll chat with him in the morning).

Joe

16 June

Hi,

Sundry tests and analysis but no major changes in the system.
----
Antennas:
  - all in (except EA21); some problems with EA03 IFA (different subbands).

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110610.0 (New!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST ! except:
        - 2130-2230 C; /home/mchost/evla/scripts/opt/2011/06/TSPE0008_sb1850345_2.evla

We would like to stop at 0130 LST, hoping to do phased array testing from 0130-0230-I'll
resend the note on the SBs.

Joe

15 June

Hi,

Lots of work today on the baselines boards (BlB swaps, deformatter swaps,
delay module swaps and the thermal stress test). All boards looking
okay but we'll retain our conservative approach.

----
Antennas:
  - all in (except EA21).

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110610.0 (New!)
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST !:
        I would like to run the following block if it hasn't been pulled up by
        the OST by 1200 LST:
        1 hour; c band; /home/mchost/evla/scripts/opt/2011/06/TRSR0039_sb4519901_1.evla


Joe

Callout: Data not making it to the archive [confirmed data was still on disk; contacted John who fixed the problem].
Callout: OST can't run SN SB [tried to manually generate script which was not possible; contacted Keith who investigated the problem and found issues in the SB; contacted Dave who investigated and found the instrument configuration was not correct - eventually found it was an old one that had been bulk edited into the SB].

Tom just called after noting that the last 3 projects have not made it into the archive.
I checked both:
- /home/mchammer/evla/mcaf/workspace
- /lustre/evla/wcbe/data/archive
and confirmed that data seemed to be flowing to the disks at least. 
My understanding was that we had space for a few days so we should be okay tonight.
I contacted John and he's working on the problem.
Joe

we have space for much more than a few days - we usually retain a few weeks of data out at the site.  i'm cc:ing john to see if he knows what's up.

       -bryan

john solved the problem - left over from the power failure earlier.
We're struggling with a problem with model2script with 11A-277 (2011dh); the OST barfed on it; Keith
found a funny number but we can't seem to hunt down where in the file the problem is - Dave's on it
now...
AC982 is running now though in excellent weather...
Joe

Well, except for the smoke; got so thick I couldn't see past 4500 meters up the north arm even while it was still daylight.


Tom Briscoe

14 June

Hi,

Some holography work today and investigation into last nights failure
(TRSR0037). We'll be ending even earlier than usual to run a special
version of the CBE in support of the V stokes sleuthing.

----
Antennas:
  - otherwise all in (except EA21).
  - Note: Funny problem with EA11 not changing band (stuck at L - Tom
        was able to clear it but there was no apparent message about
        this - seen through the zero amp fringes).

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110607
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST !:

Stop at 2115 LST for V Stokes test (Michael will change over at ~5am to CBE
version needed and complete by 0630 MDT for maintenance):

If there is a 30 minute gap, please run the following:

   - C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Joe

13 June

Hi,

Sundry tests but nothing disruptive; nothing uncovered in the
testing (C band EA13 not working but suspected to be the RF switch).

----
Antennas:
  - otherwise all in (except EA21).

Correlator:
--
CM: 2011-06-07 19:16 UT/CBE: wcbe_20110607
MCAF: 1.4.2
OST: 1.09.06 (07 Jun)
TelCal 1.5.37 (13 Jun)

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST ! except:
        2130-2230 C; /home/mchost/evla/scripts/opt/2011/06/TSPE0008_sb4451915_1.evla (V stokes test)

Stop at 0115 for 4-band tests as follows:

0115-0145 4; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4254409_1.evla
   (Note: expectation is that card reader is off along with other 'normal' mitigating actions)
0145-0215 4; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4254409_1.evla (repeat)
   (Note: must get call from Operator to confirm card reader has been turned back on).

If there is a 30 minute gap, please run the following:

   - C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Joe

Callout: TRSR0037 missing several scans (confirmed that CBE was falling behind; investigate during the day).

10 June (Weekend)

Hi folks --

 lots of 3-bit testing today, but no new software for the weekend.

We are formally in A config.

Effective immediately and until further notice, please exclude L band
when scheduling science with the OST.

Problem antennas:

 SO: use everyone!

Correlator: same as before:
 CBE: wcbe_20110607.0
 CM: 20911-06-07 19:16 UT
 MCAF: 1.4.2
 OST: 1.09.06

unavailableBlBprs.txt same as before:
 quad1 =
 quad2 =                    10,   12
 quad3 =
 quad4 =                8,9,10,11,12,13,14,15  #

== Friday night through Monday morning ==

Please run:
 Tonight = day 62447
 Day 62447 0930-1000 LST:  TVER0001_sb4045730_1.evla

 Day 62447 1000-1530 LST:  OST dynamic!

 Day 62447 1530-1830 LST:  syspt2hx.evla (in the operations area)
   - ONLY if ea18 is available, and the winds are low (otherwise use the
     OST)

 Day 62447 1830 LST through day 62448 0130 LST: *fixed date* 11A-190
   This is expected to come up in the OST.  If not, please
   use the manually created version:
      /home/mchost/evla/scripts/opt/2011/06/11A-190_sb3684580_1.evla

 Day 62448 0130-0230 LST:  OST dynamic!

 Day 62448 0230-0930 LST:  *fixed date* 11A-190
   This is expected to come up in the OST.  If not, please
   use the manually created version:
      /home/mchost/evla/scripts/opt/2011/06/11A-190_sb3847260_1.evla

 Day 62448 0930 LST through day 62449 0800 LST:  OST dynamic!

 Day 62449 0800-2400 LST:  *fixed date* 11A-246
   This is expected to come up in the OST.  If not, please
   use the manually created version:
      /home/mchost/evla/scripts/opt/2011/06/11A-246_sb3942254_1.evla

 Day 62450 0000-0100 LST:  OST dynamic!

 Day 62450 0100 LST ~ 9am: turn over to Martin Pokorny for CBE work


Please give me a call if there are any problems:

 Michael R.  home: 575 838-2436
             cell: 575 517-6797

     -- Michael

07-12 June (MR/KS)

06 June

Hi,

Testing today for CM/CBE but no new versions.

----
Antennas:
  - EA18 in; EA03 and EA27 IF Ds are dead; let's keep these out to avoid confusing things.
  - otherwise all in.

Correlator:
--
CM: 2011-06-02 22:40 UT/CBE: wcbe_condecon  **NEW
MCAF: 1.4.2
OST: 1.09.05

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
        - Note: C_quads look good but I'm retaining the conservative list of problem
        boards based on the CRM; these need to be cleared by closer inspection.

== Monday ==
OST !
        - except 0900-0930 TVER0001_sb4045730_1.evla (monitor)

Stop at 0030 for testing

Joe

03 June (Weekend)

Hi,

Focused testing on CM/CBE issues. Additional hardware work by Kerry;
We've got 28 antennas in the mix (although EA18 should only be used
for LSC(X)).

----
Antennas:
  - all in depending on frequency

Correlator:
--
CM: 2011-06-02 22:40 UT/CBE: wcbe_condecon  **NEW
MCAF: 1.4.2
OST: 1.09.05

Rollback plan is to move to:
CM: 2011-02-15/CBE: wcbe_20110414.0 if there are major issues.

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
-----------

OST except:
   - 0850-0920 X; /home/mchost/evla/scripts/opt/2011/06/10B-221_sb4315629_1.evla (ToO; array time)
   - 0920-0950 C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (Monitor)

   - 1430-1630 CXKQ; /home/mchost/evla/scripts/opt/2011/06/10B-221_sb4311229_1.evla (ToO; array time)

   - 2130-2230 C; /home/mchost/evla/scripts/opt/2011/06/TSPE0008_sb1850345_2.evla (array time; 2nd day test)

Please fill all gaps in the schedule with:
/home/mchost/evla/scripts/operations/sysptc2.evla

Continue on with OST through to Monday at 0200 LST (~1100 MDT); likely many updates over
the weekend as the SN is tracked by different groups.

Joe

Callout: Confirm just night-time for pointing run (confirmed).
Callout: Problem with high amplitudes on EA28 (looked at; stowed antenna; recurred with EA27; cancel pointing; do X_osro -looked okay); sent mail to Vivek, Ken, Michael.

no, I'm puzzled. The script is OSRO 4.5 and 5.5GHz, first time I've
run this particular one. An identical one except for 4.5, 6.5GHz
settings was run a couple weeks ago, it gave low amplitudes at 6.5
so I tried the current settings. I don't think there is RFI at either
subband, and we use wider 1 or 2GHz bands including these subbands
at other times.
- Hide quoted text -


On Fri, June 3, 2011 9:31 pm, Joseph P. McMullin wrote:
| Hi,
|
| Good news in that BL175 seems to be singing along.
| One problem arose however during Vivek's sysptc2 script. The amplitudes on
| IF A/B suddenly spiked
| by a factor of 300 or so (0.1 -> 30 to 400).
| Matt put it into stow and then suddenly, it happened on antenna 27. I think
| he also went as far as 26.
| We stopped it and ran a quick X_osro which was fine. The script had been
| running for a while when
| this happened.
| Any thoughts on this?
|
| Joe

02 June

Hi,

A full day of testing; key issue in the CM configuration
of BlBs under specific conditions (128 channels) leading
to loss of data; we've rolled back and we're going to
review the OST-produced schedule to check for this instrument
configuration.
Tomorrow we're expecting some work by Kerry on EA18 which will
require some recovery of downstream IFs (digitizer replacement).

----
Antennas:
  - EA18 out.
  - EA03, EA26 should not be included; EA02 IF is bad enough to keep it out as well.
  - otherwise all in.

Correlator:
--
CM: 2011-05-18 18:35 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2
OST: 1.09.04

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
-----------
 ==
OST except:
   - 2130-2230 C; /home/mchost/evla/scripts/opt/2011/05/TSPE0008_TBD (array time; V stokes test)

Stop at 0030 for testing

Joe

Here is the 2130-2230 program:

/home/mchost/evla/scripts/opt/2011/06/TSPE0008_sb1850345_2.evla array time)

Thanks!

Two call-outs last night:

1) Callout: 0005 MDT: No fringe, d10 display on BL175; when I checked it, most of the NIC's frames seemed not to be on their way.
See below. I'm afraid that I didn't look at the CM at that time but initiated restarting the CBE and clearing up the
residual zombies. However, after canceling the SB and restarting the CBE, the CM still thought BL175 was
active and the px display was still lighting up on the relevant boards. I then ran the VCI client GUI to delete
all subarrays and flushed all queues (configuration, activation, control, CM/CMIB). A program wasn't scheduled
for a bit so we ran a quick test on a C_quad1 which looked good.

2) Callout: 0630 MDT: SN went off but we were late to be notified. I blessed the relevant blocks but they were already an hour
past the limiting elevation. They're in the queue now but I've discussed with Dale and these may be replaced by new
SBs.

We had very sparse observing last night, in some cases where I would have expected some selections. I manually 
started an observation just before testing time to get a little bit more. I'll go over this with Keith today.

Joe


CBE during issue:
---------------------------------------------------------------------------------------------------------------------------------------------
ID     | Rate     | Active configs                                                                                | Blb IDs   | Other configs
---------------------------------------------------------------------------------------------------------------------------------------------
01-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2079      |
01-2-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 201A      |
01-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2025      |
01-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2028      |
---------------------------------------------------------------------------------------------------------------------------------------------
02-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 204A      |
02-2-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2037      |
02-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2019      |
02-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 204E      |
---------------------------------------------------------------------------------------------------------------------------------------------
03-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 203F      |
03-2-0 | 569/1140 | BL175.sb4238695.eb4311964.55715.246624363426.2,BL175.sb4238695.eb4311964.55715.246624363426.3 | 203A,2006 |
03-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2071      |
03-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 208B      |
---------------------------------------------------------------------------------------------------------------------------------------------
04-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2031      |
04-2-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2013      |
04-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2036      |
04-4-0 | 564/1140 | BL175.sb4238695.eb4311964.55715.246624363426.2,BL175.sb4238695.eb4311964.55715.246624363426.3 | 206C,204A |
---------------------------------------------------------------------------------------------------------------------------------------------
ID     | Rate     | Active configs                                                                                | Blb IDs   | Other configs
---------------------------------------------------------------------------------------------------------------------------------------------
05-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2046      |
05-2-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 202A      |
05-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 201B      |
05-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2027      |
---------------------------------------------------------------------------------------------------------------------------------------------
06-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 20A2      |
06-2-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2006      |
06-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 208A      |
06-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 204D      |
---------------------------------------------------------------------------------------------------------------------------------------------
07-1-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2033      |
07-2-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2029      |
07-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 201C      |
07-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 20A1      |
---------------------------------------------------------------------------------------------------------------------------------------------
08-1-0 | 569/1140 | BL175.sb4238695.eb4311964.55715.246624363426.2,BL175.sb4238695.eb4311964.55715.246624363426.3 | 202B,2071 |
08-2-0 | 569/1140 | BL175.sb4238695.eb4311964.55715.246624363426.2,BL175.sb4238695.eb4311964.55715.246624363426.3 | 200F,2019 |
08-3-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 2053      |
08-4-0 | 0/570    | BL175.sb4238695.eb4311964.55715.246624363426.2                                                | 208D      |


CBE after restart:

---------------------------------------------------------
ID     | Rate  | Active configs | Blb IDs | Other configs
---------------------------------------------------------
01-1-0 | 567/0 |                |         |
01-2-0 | 569/0 |                |         |
01-3-0 | 569/0 |                |         |
01-4-0 | 571/0 |                |         |
---------------------------------------------------------
02-1-0 | 578/0 |                |         |
02-2-0 | 569/0 |                |         |
02-3-0 | 569/0 |                |         |
02-4-0 | 569/0 |                |         |
---------------------------------------------------------
03-1-0 | 569/0 |                |         |
03-2-0 | 569/0 |                |         |
03-3-0 | 569/0 |                |         |
03-4-0 | 569/0 |                |         |
---------------------------------------------------------
04-1-0 | 569/0 |                |         |
04-2-0 | 569/0 |                |         |
04-3-0 | 569/0 |                |         |
04-4-0 | 569/0 |                |         |
---------------------------------------------------------
ID     | Rate  | Active configs | Blb IDs | Other configs
---------------------------------------------------------
05-1-0 | 576/0 |                |         |
05-2-0 | 577/0 |                |         |
05-3-0 | 569/0 |                |         |
05-4-0 | 569/0 |                |         |
---------------------------------------------------------
06-1-0 | 558/0 |                |         |
06-2-0 | 569/0 |                |         |
06-3-0 | 569/0 |                |         |
06-4-0 | 569/0 |                |         |
---------------------------------------------------------
07-1-0 | 569/0 |                |         |
07-2-0 | 569/0 |                |         |
07-3-0 | 574/0 |                |         |
07-4-0 | 569/0 |                |         |
---------------------------------------------------------
08-1-0 | 569/0 |                |         |
08-2-0 | 566/0 |                |         |
08-3-0 | 569/0 |                |         |
08-4-0 | 569/0 |                |         |

The CBE appears not to have received a deconfig (i.e, delete subarray) for BL175_sb4300165_1_000.55715.59800273148.2 (last night).
Same thing for BL175_sb4300165_1.55715.58009.2 (this morning).

-- 
Martin

Joseph P. McMullin wrote:
We're doing it right now! :)
However it looks like a similar problem is occurring (the script is the BL175 mentioned before).

It seems that the CBE *has* received the "deconfig" documents from the CM this time. The problem is now slightly different, and I believe it lies with the CBE. I'm already working on a solution...

-- 
Martin

Joe

01 June

Hi,

Given some uncertainties in the configuration of the BlBs,
we've rolled back to the mid-May version of the CM for
tonight. The new version of the OST is running and appears
to be doing well (enough to try tonight).
Kerry fixed and mixed boards to improve things; only
one board was identified in the standard tests but several
others came up in the CRM; given the lack of pressure
for more than 32 BlBs, I've adjust the board list to 
be conservative.

----
Antennas:
  - EA18 out.
  - EA01/EA03/EA04/EA27 have moved; EA01 has IFs C/D bad; EA03
        is not usable; EA05 has a repaired FRM.
  - otherwise all in.

Correlator:
-- 
CM: 2011-05-18 18:35 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2
OST: 1.09.04

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #
-----------

== starting 0930 LST ==
OST except:
   - 2130-2230 C; /home/mchost/evla/scripts/opt/2011/05/TSPE0008_sb1850345_2.evla (array time; V stokes test)
   - also, if there are schedule gaps (need at least two hours) please run (otherwise we'll catch it tomorrow):
      - C; /home/mchost/evla/scripts/operations/syspt2hc.evla (manually stop)

Stop at 0030 for testing

Joe

May

31 May

Hi,

New version of CM (hopefully solving the first scan issue); 
several issues with BlBs (time will be spent tomorrow to 
rearrange and repair the known issues by Kerry).

----
Antennas:
  - EA18 out.
  - EA01/EA03 moved (not yet operational).
  - otherwise all in.

Correlator:
--
CM: 2011-05-31 20:34 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                    10
quad3 =                                14
quad4 =   1                            14  #
-----------

TVER0001 should end by 0915 LST.

OST except:
   - 1330-1430 Ka; /home/mchost/evla/scripts/operations/Asysstart.evla (for 1 hour)

   - 2130-2230 C; /home/mchost/evla/scripts/opt/2011/05/TSPE0008_sb1850345_2.evla (array time; V stokes test)


Stop at 2230 for testing/maintenance day

Joe

27 May (Weekend)

Hi,

Good day of testing with phased array, T304, ++.
Some issues were uncovered in the OST that are believed responsible
for the gaps/inefficiencies - a fix will be available next week;
lots of manual programs to ensure completion of key projects and
RSRO as possible, as well as key/coordinated tests.

----
Antennas:
  - EA18 out.
  - EA05 out (FRM).
  - otherwise all in.

Correlator:
--
CM: 2011-05-25 18:35 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =
quad4 =
-----------
b104-t-4 (bent pin - rediscovered - doh!)

Given that we know there are issues with the OST, please try to pull up these programs
in the OST, but if they don't come up, please use the files below (only once - I'll
update them as completed each morning).
Emmanuel/Joe are on duty for SB submissions/ToO blessings over the weekend.

== Friday ==
OST except:
  - 0830-0930 CL; /home/mchost/evla/scripts/opt/2011/05/S3184_sb3999409_1.evla (space; array time)
  - 0930-1000 clear out old active configs (Joe)
  - 1000-1200 C; /home/mchost/evla/scripts/opt/2011/05/10C-205_sb4019291_1.evla (array time)
  - 1330-1800 CKa; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4056329_1.evla (array time)
== Saturday ==
  - 1830-2330 C; /home/mchost/evla/scripts/opt/2011/05/11A-178_sb4129223_1.evla (array time)
  - 0030-0600 C; /home/mchost/evla/scripts/opt/2011/05/10C-225_sb2580347_1.evla (array time)
  - 0930-1230 Ka; /home/mchost/evla/scripts/opt/2011/05/11A-263_sb4075841_1.evla (array time)
== Sunday ==
  - 1530-1800 CK; /home/mchost/evla/scripts/opt/2011/05/10C-186_sb4096690_1.evla (array time)
  - 1800-2030 C; /home/mchost/evla/scripts/opt/2011/05/10C-109_sb4027831_4.evla (array time)
  - 2130-2230 C; /home/mchost/evla/scripts/opt/2011/05/TSPE0008_sb1850345_2.evla (array time; V stokes test)
  - 0030-0600 L; /home/mchost/evla/scripts/opt/2011/05/10C-225_sb4013591_1.evla (array time)
  - 0600-1100 *; /home/mchost/evla/scripts/opt/2011/05/TPOL0003_sb3581703_1.evla (array time; monitoring)
        ** KEN: Can you review this to see if it excites the SB-change-only bug? **
== Monday ==
  - 1430-1830 CXKQ; /home/mchost/evla/scripts/opt/2011/05/10C-182_sb4116685_1.evla (array time)
  - 0000-0100 phased array test (Ken)
  - 0100-0630 L; /home/mchost/evla/scripts/opt/2011/05/10C-225_sb4013591_1.evla (array time)
  - 0630-0830 Q; /home/mchost/evla/scripts/test/zeeman/Qmay2011/TSPE0008_sb1994667_2.evla (array time; V stokes test)
  - 0830-1330 *; /home/mchost/evla/scripts/opt/2011/05/TPOL0003_sb3902307_1.evla (array time; monitoring)
== Tuesday ==
  - 1430-1830 CXKQ; /home/mchost/evla/scripts/opt/2011/05/10C-182_sb4086374_1.evla (array time)
  - 1830-2130 CKa; /home/mchost/evla/scripts/opt/2011/05/10C-141_sb3370123_1.evla (array time)
  - 2130-0200 L; /home/mchost/evla/scripts/opt/2011/05/10B-187_sb4126552_1.evla (array time)

  if you have a 30 minute gap, please run (only once per day):
        - C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Stop at 0200 for testing

Joe

Sorry one correction, for the Monday morning Q band V stokes test:
  - 0630-0830 Q; /home/mchost/evla/scripts/test/zeeman/Qmay2011/TSPE0008_sb1994667_2.evla (array time; V stokes test)

I've made the changes to the fshift fundamental and to get a non-integer integration time; I left the old file in place for
comparison and the new one (which should be run on Monday) is at:
/home/mchost/evla/scripts/opt/2011/05/TSPE0008_sb1994667_2.evla

Joe

Just an update:

I still don't have word back from Bill on the changes tested with regard to 4-band (I poked him again today). This means
that we want to keep these fixes in the webtest version of the OPT for a bit longer. The planned 4-band run may still
suffer from the interference issues as the changes (subband and requantizer gains) are not adjusted for this project.

The plan then for Monday-Tuesday morning is:

== Monday ==
  - 1430-1830 CXKQ; /home/mchost/evla/scripts/opt/2011/05/10C-182_sb4116685_1.evla (array time)
  - 2000-2300 C; /home/mchost/evla/scripts/opt/2011/05/11A-269_sb4207841_1.evla (array time)
  - 0000-0100 phased array test (Ken)
  - 0100-0630 L; /home/mchost/evla/scripts/opt/2011/05/10C-225_sb4013591_1.evla (array time)
  - 0700-1900 4; fixed date 11A-201

== Tuesday ==
  - 1900-2200 CKa; /home/mchost/evla/scripts/opt/2011/05/10C-141_sb3370123_1.evla (array time)
 
Begin move; if not too many antennas are out, please squeeze in (but should not disrupt move activities
which take precedence):
  - 2200-0230 L; /home/mchost/evla/scripts/opt/2011/05/10B-187_sb4126552_1.evla (array time)

Joe

26 May

Hi,

3-bit modules in 12, 15, 22 and 28 were removed; note Ken's comments
regarding the fringe display (underdriven by 2 dB so will not look
as good).

----
Antennas:
  - EA18 out.
  - otherwise all in.

Correlator:
--
CM: 2011-05-25 18:35 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =       
quad2 =                     10
quad3 =
quad4 =
-----------
b104-t-4 (bent pin)

OST except:
  - fixed date for 11A-178 CX; 
        - backup file 1830-2330 CX; /home/mchost/evla/scripts/opt/2011/05/11A-178_sb4128997_1.evla (array time)

  - if you have a 30 minute gap, please run:
        - C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (array time; test)

Stop at 0000 for testing

----

0000-0100 phased array (Vivek, Ken, Amy)
...
0300-0400 (lunch); T304 test TBD

Joe

Callout: Scheduling problem (Manually scheduled).

Larry saw some oddities in the OST (6 hour gap); this could be due to wind jumping around a
bit but there did look to be some low frequency projects to run in that time.
As a result, some manual project options to get us to tomorrow where we can review/understand
better the (non)selections.

0930-1230 *; /home/mchost/evla/scripts/opt/2011/05/11A-263_sb4075841_1.evla (high priority but does need Ka wind)
or
1000-1200 C; /home/mchost/evla/scripts/opt/2011/05/10C-205_sb4019291_1.evla (high priority)

1330-1800 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4056329_1.evla (high priority)

25 May

Hi,

Recovery from maintenance day.

----
Antennas:
  - EA18 out.
  - EA20 AC L302 is not reliably locking; otherwise all in.

Correlator:
--
CM: 2011-05-18 18:35 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =       3
quad2 =                     10
quad3 =
quad4 =
-----------
b101-t-6 (won't configure); b104-t-4 (bent pin)

OST except:
        1030-1330 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4068613_2.evla (array time; if phase <= 7 degrees)
        1330-2030 Ka fixed date on AB1353; backup scripts as needed 
           - 1330-1700 Ka; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4175947_1.evla (MakeMake; fixed date)
           - 1700-2030 Ka; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4176408_1.evla (Pluto; fixed date)
        2030-2130 C; /home/mchost/evla/scripts/opt/2011/05/11A-269_sb4158083_1.evla (array time; ToO)

Stop at 0000 for testing

Joe

23 May

Hi,

Science observing through til maintenance; 3-bit testing concluded; Ken
fixed up delays and checked out boards.

----
Antennas:
  - EA18 out.
  - Otherwise, all in

Correlator:
--
CM: 2011-05-18 18:35 UT/CBE: wcbe_20110516.0
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =
quad4 =
-----------
b104-t-4

OST except:
        1030-1330 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4068613_1.evla (array time; if phase <= 7 degrees)
        1330-2030 Ka fixed date on AB1353; backup scripts as needed (note 4146067 seems not to be submitted though
                I received a note that it was)
           - 1330-2130 Ka; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4146067_1.evla (ephem object; fixed date)
           - 17000-2030 Ka; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4152138_1.evla (ephem object; fixed date)

Stop at 2130 (stop for maintenance day)

        2130-2230 C; /home/mchost/evla/scripts/opt/2011/05/TSPE0008_sb1850345_2.evla (array time; V stokes test)

Joe

18-22 May MR/KS

Callout (21 May): ToO not getting scheduled by OST (Generated the script manually and sent in to operator).

17 May

Hi,

An early edition to be modified by Michael/Ken but I wanted to note what
we already knew was needed tonight.
Testing throughout the day (focusing on CM/CBE updates); these will be
kept in ('New' below, or withdrawn 'Old' below, as results dictate).

----
Antennas:
  - EA18 out.
  - Otherwise, all in; partial baseline update yesterday; additional data
  for troubled antennas taken.

Correlator:
--

Pending testing today:
Old
---
CM: 2011-02-15 22:38 UT (reverted back)/CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
New:
---
CM: 2011-05-16 16:16 UT/CBE: wcbe_subarrays (17 May 2011; 09:25)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =
quad4 =       3,                 12,13,14,15 # issues with bottom crate of this rack
-----------
b104-t-4, b107-t-7 problems; bottom crate of rack 8 is showing some power issues.

OST except:
        1430-1500 4; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4126505_1.evla
        1500-1930 4; fixed date on 11A-184

Stop at 2130 (stop for maintenance day)

Reminder that Michael is the principal point of contact for the next several days:
cell: 575-517-xxxx/home: 575-838-xxxx
I'll be on travel the rest of today, returning Saturday night/Sunday morning.

Joe

14 May

Hi,

Tonight is mostly consumed by a long track with the 3-bit samplers
followed by some fixed date observing...

----
Antennas:
  - EA18 out.
  - Otherwise, all in; baselines determined and updated but for 3 (EA28, EAxx, EAxx).

Correlator:
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =
quad4 =       3,                 12,13,14,15 # issues with bottom crate of this rack
-----------
b104-t-4, b107-t-7 problems; bottom crate of rack 8 is showing some power issues.

0630-1430 C; Long_C3p8bit.evla (Michael will revert back following this observation).
        - please kill this gracefully at this time.
        1430-1500 transition back to 8-bit operation (Michael will be about at around 1400 LST)
1500-1930 4; fixed date observing on 11A-184
        -backup file: /home/mchost/evla/scripts/opt/2011/05/11A-184_sb3684605_1.evla 
1930-2130 X; /home/mchost/evla/scripts/operations/sysptc2.evla (please terminate manually)

Stop at 2400 (stop for testing)

If there are any 30 minute gaps, please run: 
   - /home/mchost/evla/scripts/operations/focuscheck.evla

Joe

13 May (Weekend)

Hi,

In BnA; L305 issues with EA22,EA25 were fixed.
Rolled back to older versions of the CM/CBE.

----
Antennas:
  - EA18 out.
  - EA25 has intermittent B/D issues.

Correlator:
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10,   12
quad3 =
quad4 =                 8,9,10,11,12,13,14,15 # issues with bottom crate of this rack
-----------
Note issues with: b104-t-4 (and associated); issue with rack 8 bottom

I've put in options for the highest priority, highest stringency observations
for the hybrid that are available now; I'd like to see the opportunity for
the OST to select these but if they aren't coming up (and the weather is
appropriate) please run them manually. Please note that I have not put
these on hold, so check that they have not already run if it does come
up later in the weekend (I'll work to ensure that I resolve these each
morning).

Also, if possible, I'd like to record the amount of time not filled
(the gaps in the schedule) given the new algorithm in place; please
call me if you have more than a 2ish hour gap and please let me know
how much time we're not filling each day (thanks!).

== Friday ==
OST
        0600-0630 C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045370_1.evla (Test; array time)
        ...
        0900-1030 Ka; /home/mchost/evla/scripts/opt/2011/05/11A-263_sb4110497_1.evla (array time)
        1030-1330 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4068613_1.evla (array time)
        1330-1530 C; /home/mchost/evla/scripts/operations/syspt2hc.evla (array time; manually terminate)
        1530-1830 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4054040_1.evla (array time)

== Saturday ==
OST
        Will review weather for high frequency or 3-bit testing; 3-bit testing will tentatively run from
        1200-2000 3-bit; Michael; after which there will be a 30 minute setup period to move back to 8-bit
        operation. This corresponds to roughly 2200 MDT so we'll call around 1700 UT to discuss the weather
        (if it's good, we'll punt to Sunday night).

        -if API <=12, wind <=7: 0300-0700 *; /home/mchost/evla/scripts/opt/2011/05/10B-133_sb2978993_1.evla (array time)
        -if API <=7, wind <= 6: 1400-1700 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4059349_1.evla (array time)

== Sunday ==
OST
        Will review weather for high frequency or 3-bit testing
        -if API <= 7, wind <=6: 1400-1700 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4058092_1.evla (array time)

== Monday ==
OST
        -if API <= 7, wind <=6: 1400-1700 Ka; /home/mchost/evla/scripts/opt/2011/05/AC982_sb4054040_1.evla (array time)

Stop at 2400 (stop for testing)

Joe

Callout: No fringes, d10 (CM not deconfiguring? did full restart of CM/CBE; working again).
Callout: OST not picking fixed date SB (manually generated and run).
Callout: OST not giving API/wind; over-ride not helping (just notice to me; Keith working on it).
Callout: OST out of sync messages (contact to Dave H.).
Callout: Completed SBs not showing up in the archive (verified that the data was in the staging area).
Callout: OST won't create schedule (network is completely hosed; James called - 3rd floor switch issue; down several hours, several exchanges).
Callout: AC982 not fringing (everyone's twitchy from this weekend; by the time, I've confirmed that everything seems to be working, fringe results are appearing).

First update (already!):

One Jim reviewed the schedule and many of the projects expected aren't coming up; I looked at those that
were and think that at least one aspect of our problem is that we have residual over-ride priorities from the
B configuration; there was something of a priority-escalation race as some projects were bumped and then
others were bumped to provide further differentiation. I'll review this a bit further to see but for now, I'm
zeroing out all testing O/R priorities to get us back on a level; I'll discuss with Joan the other programs
when she returns.

In addition, Jim will begin keeping a log of things that come up different from our priorities as well as
gap times; in addition, he's already noted an issue with scheduling within the hour you're currently
in (e.g., trying to schedule for 0730 at 0705 seems to create an exception).

More to come...

Joe

Not a good start...
Following the TVER0001 run, the CM apparently didn't deconfigure as a result BL175 was lost. Jim noted
no fringes and when I looked it still thought it was running TVER0001. I brought down the CBE but when
I brought it back up (a complete restart), it still was running on the boards (expected frames high, px display
showing activity, etc).
I tried flushing the sundry queues and deleting subarrays but that also didn't seem to work.
I then stopped and restarted the CM and brought down the CBE again. Still the same; I called Martin
who thought it was a CM issue (as we expected).
I then ran specific C_quad tests to activate those boards that wouldn't turn off and then canceled that
run - in this way, the boards were finally quiescent.

I did a quick wcbetool reset to clean things up, x_osro check and we're back in business but very
strange behavior from our 'stable' setup...

Joe

Scheduler is not picking up the 4-band fixed data observing; manually generated:

1100-1530 4; /home/mchost/evla/scripts/opt/2011/05/11A-217_sb3909504_1.evla (fixed)

Joe

After talking with Michael and reviewing the weather, we postponed the 3-bit testing later
(i.e., conditions are good tonight); unfortunately there is a fixed date program tomorrow night
so it will have to be on Monday.

Jim noted that the last program 11A-266 was not showing up in the archive; the data seem to be
in the staging area however (cc'ing John) so we're continuing on.

Joe

We were down for ~3 hours; the initial symptom was the OST could not connect to generate a schedule.
It turned out most of the connections were down; James was already working on it - filehost seemed to
be down (I'm sure he'll write up a note on this).
He and K.Scott have been at since a bit after 0800; enough of the system was in place that I could at
least get something on the telescope; this will run for 5.5 hours after which hopefully everything will
be back.

Again, some programs may not be showing up in the archive initially (the current example is 10C-141);
the data appear to be in the workspace area but we'll hold off logs to PIs until it's in the archive.

Joe 

Sorry to be adding my own chaos on top of things; I had the day wrong for the start of 11A-184 - it starts Tuesday morning
not Monday.
At this point, I'd prefer to go with science observing tonight as the weather looks to be quite good again.
There are some AC982s (just spoke with Matt to clarify which ones) to try for; oddly, these were not coming up
in the OST even when the API/phase were lowered to catch them (it was picking a priority 3 - not sure why but
it would be good to review).


Joe

12 May

Hi,

In BnA; all antennas up and chugging.
Two pairs of BlBs have problems; removed from consideration.
New versions of CM/CBE (extensive testing done throughout the
afternoon).

----
Antennas:
  - EA18 out.

Correlator:
--
CM: 2011-05-12 16:32 UT *new version*
CBE: wcbe_subarrays (12 May 15:26) *new version*
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =                              13
quad4 =                 8,9,10,11,12,13,14,15 # not checked today following swaps
-----------
Note issues with: b104-t-4 (and associated), b108-b-4 (and associated)


== Thursday ==
OST
        0700-0900 CK; /home/mchost/evla/scripts/opt/2011/05/11A-268_sb4102301_1.evla (array time; ToO; relaxed API/wind)
        0900-0930 C; /home/mchost/evla/scripts/opt/2011/05/TVER0001_sb4045370_1.evla (Test; array time)
        ...
        1330-1530 C; /home/mchost/evla/scripts/opt/2011/05/SC0346_sb4102199_1.evla (array time; ToO)

Stop at 2300 (stop for testing)

Joe

Callout: Missed scan messages (confirm data BDFs are coming in).

11 May

Hi,

All antennas have been moved so a long pointing run tonight after things
settle down.
One Blb has a bent pin (to be fixed tomorrow); we're also removing the
one whose serial number dropped out last night.
All antennas in (though EA21 seems to have some issues; Michael noted that
it was synching to the wrong clock edge earlier).
New versions of the CM and CBE are available but there were some issues
with one of the test scripts (never deconfiguring so we've rolled back
to the stable versions for tonight). Michael I did test the CrsroXrsro over
multiple runs and didn't see any issues.

----
Antennas:
  - EA18 out.

Correlator:
--
CM: 2011-02-15 22:38 UT 
CBE: wcbe_20110414.0
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
# may11 version
# b104-t-4 (2007) has bent pins; b104-b-1 (2060) was flakey last night
quad1 =
quad2 =                    10,   12
quad3 =
quad4 =                8,9,10,11,12,13,14,15  #$ didn't check rack 8

== Monday ==
OST
        except: 
        0715-0745 C; TVER0001_sb4045370_1.evla (Test; array time - 27 minutes); run first thing
        5 hours: /home/mchost/evla/scripts/operations/sysptgx.evla (array time; stop manually)
        2230-2300 4; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4101803_1.evla

Stop at 2300 (stop for 3-bit testing)

Testing tomorrow:
  0900-1200 3-bit (transition back to 8-bit for...)
  1200-1300 gain expansion/compression checks (Rick/Bob)
  1300 TBD

Joe

Callout: Data not in archive (confirmed in mcaf/workspace area; sent note to John).

10 May

Hi,

Continuing the BnA move; several antennas still to move tomorrow.
Continuing 3-bit testing today (now using delays for 3-bit) with the
new version of the Executor. New versions of both CM/CBE.
Matt's done a quick 4-band test and is now cycling through the
other bands; pointing and delays should be okay for the moved
antennas.

----
Antennas:
  - EA05 in; needs baseline update
  - EA18 out.
  - EA16 out (fiber work not completed today)
  - EA12 IF B has issues (may go out but we'll run with it)

Correlator:
--
CM: 2011-05-10 18:47 UT *new version*
CBE: wcbe_subarrays (10 May 11:11) *new version*
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =                              13
quad4 =                 8,9,10,11,12,13,14,15 # not checked today following swaps
-----------
Note issues with: b104-t-4 (and associated), b106-b-2 (and associated)


== Monday ==
OST
        except: TVER0001_sb4045370_1.evla (Test; array time - 27 minutes); run first thing

Stop at 2030 (stop for maintenance day)

Joe

Callout: Missing IFs in FRINGE (rolled back CM/CBE versions)

Some problems tonight.
Matt noticed that we were missing IFs on the incoming data; this smelled like a BlB issue
and when I checked the CBE, I was missing frames over many boards. I tried to rephase
thinking that perhaps that would make everyone happy but alas not. I checked some 
individual boards (BlB GUI) and they seemed not to have any errors. 
I called Ken and we reset the BlB interframe delay in case that had become confused; no
affect. Ken went into the office to look more closely. The two key errors were the missing
frames (and more on some) and a lack of any fringes with Quad 2 due to an apparent
serial number problem.
I called Martin who looked at things and believes the CBE is sending things appropriately
to the different pipeline IP addresses and so perhaps the CM is at fault.
At this stage we rolled back to the stable version. I updated the unavailable list to exclude
the quad two problems and all seems well.

I think this is old ground but even minor changes to these critical systems must be fully
vetted (in practice I think this means that we need the afternoon to confirm they are 
working and that the full quadrant tests (1-4) should be run before signing off a version
change). I'll hold that line.

Matt's back observing...

Joe

09 May

Hi,

Winds prohibited some antenna activities today; focus on CM/CBE update.
Testing, including problematic script, worked fine so we'll run with
the coupled, new versions.
Some swaps of BlBs today; updated serial numbers; some residual issues 
with particular BlBs (noted below).

----
Antennas:
  - EA05 in; needs baseline update
  - EA18 out.
  - EA08 in (needs pointing; okay through C band/with ref ptg)
  - EA20 out (fiber work not completed today).

Correlator:
--
CM: 2011-05-09 20:19 UT
CBE: wcbe_subarrays (09 May 16:26)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =                     10
quad3 =                              13
quad4 =                 8,9,10,11,12,13,14,15 # not checked today following swaps
-----------
Note issues with: b104-t-4 (and associated), b106-b-2 (and associated)


== Monday ==
OST - following TVER0001


Stop at 2200 (stop for testing; software updates)

Joe

Callout: Missing scan (not incrementing).

06 May (Weekend)

Hi,

No testing today; all science/test observations.

----
Antennas:
  - EA05 in; needs baseline update
  - EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now, but flagged for now (along with b105-b-2 (power supply)).
-- 
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 = 
quad2 =   1
quad3 =     2     5
quad4 =  
-----------

Following the correlator checkout:

== Friday ==
OST 

== Saturday ==
OST except/noting:
   - 1400-2030 C; 10B-209 fixed date (backup file at: /home/mchost/evla/scripts/opt/2011/05/10B-209_sb4025069_1.evla)
   - 0230-0930 4-band; 11A-190 fixed data (backup file at: /home/mchost/evla/scripts/opt/2011/05/11A-190_sb3847262_1.evla)

== Sunday ==
OST except/noting:
   - 1400-2030 C; 10B-209 fixed date (backup file at: /home/mchost/evla/scripts/opt/2011/05/10B-209_sb4025628_1.evla)
   - 0100-0900 4-band; 11A-118 fixed date (backup file at: /home/mchost/evla/scripts/opt/2011/05/11A-118_sb3946813_1.evla)

== Monday ==
OST except/noting:
   - 1300-2100 4-band; 11A-203 fixed date (backup file at: /home/mchost/evla/scripts/opt/2011/05/11A-203_sb3684591_1.evla)

Stop at 2100 (reconfiguration begins)

Joe

Tonight if weather conditions permit Ka band observing, we'd like to get:

0900-1200 XKa; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4087712_1.evla (ephem; array time)

Thanks.

Joe

Hi Sam,
Thanks - that makes sense; we'll investigate the one that was missing...
Joe
- Hide quoted text -


On Sat, May 7, 2011 at 6:46 AM, VLA Operations  wrote:
Hey Joe -

Started getting these alerts at about 2:40 am local. The missing scans didn't increase that's why I didn't call and wake you up. It was during the fixed date file from the OST 10B-209_sb4025069_1.

Thnaks

Sam

> Bryan Butler wrote:
>> the BDF for scan 142 in 10B-209 has the 'unknown' marker (.../bdf/X1),
>> triggering these emails.  see Main.xml in
>> /home/mchammer/evla/mcaf/workspace/10B-209.sb4025069.eb4025851.55688.24740097222
>>
>> martin - is this one where the BDF actually exists and we can fix it up
>> after the fact?
>
> Yes, the BdfInfo message for this scan was logged at May  7 02:36:38.
> The regular frequency of the logged messages indicates at first glance
> that there was no problem with completing this scan, so I'll reluctantly
> suggest that the packet to MCAF was lost.
>
> The BDF UID is uid:///evla/bdf/1304757276206. I think that this is
> sufficient information for John to fix up the SDM.
>
> --
> Martin
>

Okay, I fixed the archive. The missing bdf is now cataloged and available.

Sorry for the delay, John

05 May

Hi,

More 3-bit/4-band testing today; addition CM/CBE testing, however there are some
remaining issues and so we've rolled back to the stable versions of that pair.

----
Antennas:
  - EA05 in; needs baseline update
  - EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now, but flagged for now (along with b105-b-2 (power supply)).
-- 
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 = 
quad2 =   1
quad3 =     2     5
quad4 =  
-----------

Following the correlator checkout:

OST except:
        - 0830-0900 4-band; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4049175_1.evla (4-band; array time)
        - 0900-1100 CXK; /home/mchost/evla/scripts/opt/2011/05/10C-145_sb4048544_1.evla (ToO; array time)
        - 
        - 1530-1930 XKa; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4046657_1.evla (Ephem; array time)

end at 2130 LST (~0800 MDT) for testing.

Joe

04 May

Hi,

More 3-bit/4-band testing today; addition CM/CBE testing, however there are some
remaining issues and so we've rolled back to the stable versions of that pair.

----
Antennas:
  - EA05 in; needs baseline update
  - EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now, but flagged for now (along with b105-b-2 (power supply)).
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =   1
quad3 =     2     5
quad4 =
-----------

Following the correlator checkout:

OST except (if we can start earlier, the TSUB0001 and 10C-145 programs can shift earlier):
        - 0830-0900 4-band; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4049175_1.evla (4-band; array time)
        - 0900-1100 CXK; /home/mchost/evla/scripts/opt/2011/05/10C-145_sb4048544_1.evla (ToO; array time)
        - 
        - 1530-1930 XKa; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4046657_1.evla (Ephem; array time)

end at 2130 LST (~0800 MDT) for testing.

Joe

03 May

Hi,

3-bit/4-band testing today.
No known issues with timecode; so perhaps we can relax the following (I'll remove it
tomorrow):
 Let's continue to check the following periodically (note that the messages file updates
 on Sundays so there are no records currently).
   - ssh widar-boot-1
      - cd /var/log
      - cat messages |grep TIMECODE |grep fault |tail

----
Antennas:
  - EA05 in; needs baseline update
  - EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now.
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =           5
quad4 =
-----------

Following the correlator checkout:

OST except:
        - 0700-0730 C; /home/mchost/evla/scripts/widar/C8step.evla (switched power test; array time)
        - 0730-0800 C; /home/mhost/evla/scripts/opt/2011/05/TVER0001_sb4045730_1.evla (e2e test; array time)
        - 0800-0900 XK; /home/mchost/evla/scripts/opt/2011/05/10C-145_sb4045733_1.evla (ToO; array time)
        ...
        - 1530-1600 4-band; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4046141_1.evla
        - 1600-2000 Ka; /hom/mchost/evla/scripts/opt/2011/05/AB1353_sb4042803_1.evla (Ephem; array time)

end at 2000 LST for maintenance.

Joe

02 May

Hi,

3-bit testing and additional antenna activities (4-band installs).
Let's continue to check the following periodically (note that the messages file updates
on Sundays so there are no records currently).
   - ssh widar-boot-1
      - cd /var/log
      - cat messages |grep TIMECODE |grep fault |tail

----
Antennas:
  - EA05 in; needs baseline update
  - EA22 ACU problem; out.
  - EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now.
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
-----------

Following the correlator checkout:

OST except:
        - 1500-1530 4-band; /home/mchost/evla/scripts/opt/2011/05/TSUB0001_sb4037368_1.evla (4-band; array time)
        - 1530-1930 Ka; /home/mchost/evla/scripts/opt/2011/05/AB1353_sb4028889_1.evla (planetary; array time)

end at 2230 LST Monday for testing/antenna works.

Joe

April

29 Apr (Weekend)

Hi,

A day of testing and winds.
The glitches with the timecode loop recurred last evening; today Jim et al worked to
repair all paths. For the B path, a bad fiber was found and fixed. It was decided however
to revert to the A (1) path; a 1.5 dB loss was seen in the optical path but not at either
end of the fiber and so this will need to be investigated next maintenance day (and
some potential resplicing done). In the event that we have a recurrence, we'll need to
watch the following for now:
   - ssh widar-boot-1
      - cd /var/log
      - cat messages |grep TIMECODE |grep fault |tail

   * This will give you the most recent timecode glitches; the last that should have
     occurred was at 13:24:53 today - if you see more recent than that, then we need
     to investigate.

Currently, we continue to be stowed due to winds; once they die down enough to operate,
we'll re-establish delays, confirm the correlator is well, and move on with the plan
below (essentially OST with some interventions to ensure the key ToO/testing).

----
Antennas:
  - EA05 in; needs baseline update
  - EA08 out (receiver issue; CW bleed through at all bands; not understood)
  - EA13 out (LO system failure; worked on but not able to be recovered.
  - EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now.
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
-----------

== Friday ==
OST except for:
   - 0900-1200 C; /home/mchost/evla/scripts/opt/2011/04/11A-265_sb4024637_1.evla (ToO; array time)

== Saturday ==
OST except for:
   - 1500-1530 4; /home/mchost/evla/scripts/opt/2011/04/TSUB0001_sb4022547_1.evla (4-band test; array time)

== Sunday ==
OST

== Monday ==
OST

end at 2400 LST Monday

Joe

Just an addition for Saturday (assuming we start at some point!):

S3124.4023261 should come up in the OST (it's at very high priority); however, if it doesn't,
please run between 1800-0700 LST manually tomorrow (it is 1 hour):

C; /home/mchost/evla/scripts/opt/2011/04/S3124_sb4023261_1.evla (Fermi LAT GRB; array time)

Joe

Callout: Verification delayed due to wind stowage (completed).

Just an update; Dave noted that the winds had died down enough to sustain operations.
He fit the delays and they look pretty good (we could tweak them a bit more but we were 
a bit rushed to get to the ToO).
I ran through the C_quads and had some problems with quadrants 2&3; to save time
(given the number of BlBs showing issues), I did a mass rephase, then rechecked 
and things looked good.
We're getting on the 11A-265 SB 1 hour past its nominal start LST range but thought it
was still worth doing.
Joe

Saturday morning update:

Callout: No updates (CM/CBE interaction issue; restarted CBE).

Sam called after noting that 10C-133 was not providing pointing solutions nor d10 display.
The timecode was checked but no issues there. I looked at the CM and found that it was
still waiting to configure the required BlBs and that the CM seemed to be merrily proceeding
(i.e., wcbe_master.log seemed to be integrating away), however, the wcbetool showed 
that frames were being received but were not expected. I don't remember seeing this particular
failure mode recently. I cc Martin.

We restarted the CBE and things are working again.

Joe

Callout: Bad deformatter reported (contacted Kerry who fixed it).

Just an update; I had a report from Laura regarding a bad deformatter on EA05; this had been checked of course
before we started but the problem occurred sometime Friday night. I talked to Kerry who has now fixed this and noted
that the issue occurred at 2030 local time (on apparently all 4 IFs, though Laura thought one was okay).
If we have a 30 minute gap, it's worth running an X_osro/NXsystart to see if we can recover the delays on EA05;
data from Friday night until those delays are re-set will be bad on that antenna.

Joe

28 Apr

Hi,

Mix of testing today; new, coupled versions of CM/CBE were tested; trial of 64 BlBs with
8x recirculation; CRM work on rack 8; 3-bit test with Vivek/Ken.
Dipole count is: 24 (4 missing still); 4-band testing is pending for tomorrow (02, 03, 04, 27)
as possible.

----
Antennas:
  - EA05 back in array; needs some pointing and baselines; EA18 out.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now.
--
CM: 2011-02-15 22:38 UT (reverted back)
CBE: wcbe_20110414.0 (official tagged version of wcbe_connect)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
-----------

== Thursday ==
OST except for:
   - 0630-1230 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (recirc x 4; array time)

end at 2200 LST (~0900 MDT) for testing.

Joe

27 Apr

Hi,

Second of the double maintenance days; Kerry worked at the site on the clock B issues.
Essentially a second redundant LO path has been installed and tested. Ken and Michael
are back; Ken led the recovery today and all seems well after a rephasing of the BlBs;
Bruce did a reboot of all StB and BlB to bring them up to the same version. Delays
have been worked on and so hopefully a full night of observing ahead.
Note, the dipole installation has gone slower than expected (only 16 thus far) so most
of tomorrow will be needed. Hoping to also get some time for Sonja/Martin for tomorrow
as well as time for Dave. Modest 3-bit testing for the afternoon as needed but the
big thrust will be next week.

----
Antennas:
  - EA05 back in array; needs some pointing and baselines; EA18 out.
  - EA13 baselines updated and loaded.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement; not
        impacting science for now.
  - s106-b-3 (EA24 IF B) was swapped with s105-b-5 (EA18 IF D) to bypass the issues for now;
  station rack 8 did not have any identified versions that could have been swapped
  in without some effort so this was seen as the quickest way of bringing ea24 online.

--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.2

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =
-----------

== Wednesday ==
OST except for:
   - 2100-2200 C; /home/mchost/evla/scripts/opt/2011/04/10C-145_sb4012074_1.evla

end at 2200 LST (~0900 MDT) for testing.

Joe

26 Apr

Hi,

Maintenance day; correlator time was split between Sonja (morning)
and Martin (afternoon); however, Rob et al were working on the master
rack (putting in splitter for the 128MHz) and the correlator clock
was interrupted. Kerry fixed things up as much as possible but we're
still missing clock B (power levels are too low) and so many BlBs
are unhappy. In addition, the delays will need to be reset. Currently,
we can't point the antennas due to the sustained winds. I'll check
in intermittently to see what's possible but we may just punt on tonight
given that things will be interrupted again in the morning.
Anyway, situation is pending (Vivek is also on stand-by to support
settling the delays as possible).

----
Antennas:
  - EA05 back in array; needs some pointing and baselines; EA18 out.

Correlator:
  - Needs to be looked at.

--
CM: 2011-02-15 22:38 UT (reverted back by Sonja).
CBE: wcbe_connect (with mpich2-1.3.1)
      - if we need to fallback, the previous version is: wcbe_connect.20110329
MCAF: 1.4.2 (salient feature is added flagging capabilities).

unavailableBlBps.txt
-----------
TBD
-----------

== Tuesday  ==
TBD

Wednesday morning: Kerry et al. will attack the clock B issue.

Thanks.

Joe

25 Apr

Hi,

Mostly science today so status is relatively unchanged from last week (in both the
good and bad sense). New MCAF was deployed.

----
Antennas:
  - EA13 baselines determined and added into the system.
  - EA05 back in array; needs some pointing and baselines; EA18 out. 

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement.

--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
      - if we need to fallback, the previous version is: wcbe_connect.20110329
MCAF: 1.4.2 (salient feature is added flagging capabilities).


-----------

-----------

== Monday ==
OST 1230-2100
2100-0100 L; /home/mchost/evla/scripts/opt/2011/04/10B-154_sb3537023_1.evla (ECSO; array time)
0100-0200 MCAF deployment
0200-1930 OST

Stop for maintenance day testing at  1930 LST.

Tuesday morning through 1300 MDT: Sonja/Dave/Brent (CM/CRM testing/devel)
Tuesday afternoon 1300-1700 MDT: Martin/Vivek (CBE/3-bit/EA05 checkout)

at this point we'll assess whether anything can be done overnight.

Thanks.

Joe

21 Apr (Weekend)

Hi,

End of day for 3-day weekend; plan is below. Brent/Dave/Bruce did a spate of testing
on the system today. Based on this, I'm taking several boards out of operation until
they can be swapped.
We have a persistent issue with EA24-B (s006-b-3); there are no spares to replace this
so we have to run as is; it is responsible for the missing frames indicated in the CBE
(I tested with/without ea24 to confirm - I did find other issues that were masked by this:
b101-t-2 - issues - rephased
b104-t-3 - seeing Gigabit ethernet interface status with errors pop up intermittently for X6Y5
I took this out as well since we don't have a full complement anyway.

----
Antennas:
  - EA24 IF B is broken; please add a note in the logs that indicates:
"EA24 IF B is not functioning and all data associated with it should be flagged."
  - EA26 had a power issue and is currently out of the array.

Correlator:
  - b103-t-3 (bb2034), b105-t-5 (bb2034) have hard faults and need replacement.
  - b105-t-4 (due to b105-t-5)
  - b104-t-3
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
      - if we need to fallback, the previous version is: wcbe_connect.20110329
MCAF: 1.4.1

# Temporary for 4/21/11
quad1 =
quad2 =    1,            9
quad3 =      2
quad4 =                8,9,10,11,12,13,14,15  #$ didn't check rack 8

== Thursday ==

   - 0900-1000 MDT: Martin/Joe w/ 64 BlBpr test
   - 1000-1300 MDT: Dave del Rizzo w/ CRM
   - 1300-1700 MDT: Vivek/Martin

0700-0800 C; /home/mchost/evla/scripts/opt/2011/04/11A-263_sb4008254_1.evla (ToO; array time)
        expired: sb4007951 (duplicate at different LST range)
OST 0800-1130
1130-1230 CKa; /home/mchost/evla/scripts/opt/2011/04/11A-263_sb4006963_1.evla (ToO; array time)
        expired: sb4007450 (duplicate at different LST range)

== Friday ==
OST 1230-1800
1800-2000 XQ; /home/mchost/evla/scripts/opt/2011/04/TSPE0010_sb3923557_1.evla (SL Dy; array time)
OST 2000-2200
2200-0200 L; /home/mchost/evla/scripts/opt/2011/04/10B-154_sb3537023_1.evla (ECSO; array time)
0200-0730 L; /home/mchost/evla/scripts/opt/2011/04/TDEM0007_sb3591375_1.evla (Demo; array time)
OST 0730-1230

== Saturday ==
OST 1230-2100
2100-0400 L 09A-106 (fixed date)
OST 0400-1230

== Sunday ==
OST 1230-2100
2100-0400 L 09A-106 (fixed date)
OST 0400-1230

== Monday ==
OST 1230-2100
2100-0100 L; /home/mchost/evla/scripts/opt/2011/04/10B-154_sb3537023_1.evla (ECSO; array time)

Stop for testing 0100 LST.

Joe

20 Apr

Hi,

Maintenance day and good recover day! 
Kerry swapped out the PS for b108-t-0. Bruce continued working on the
oddities with CMIB/boards discovered earlier in the week. In particular, the b101-b-4
board issue was sleuthed out (see widar-wg note from today) and is currently running the
older May 25, 2010 binary.
However, after having checked things out, the CMIB code was left running (and it looked
fine except for b101-b-6). Vivek did some subsequent 3-bit testing however and when
we rechecked the system, it was behaving as before, that is, there are many missing
frames indicated in the CBE and the X5Y5 RC chip in general is blinky - i.e., intermittently
showing errors generally associated with BB-4 on the RC-Y5 display.
I fell back to just using racks 2 and 4 for tonight and will discuss with Bruce tomorrow...

----
Antennas:
  - EA05 back in array; needs some pointing and baselines; EA18 out. EA13 hasn't had its baselines set yet.

Correlator:
  - Rolled back from testing to wcbe_connect version; new way to generate missing scans was found so
  we'll be watchful for the alerts from Bryan's scripts.
      - if we need to fallback, the previous version is: wcbe_connect.20110329
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 = 0,1,2,3,4,5,6,7
quad2 = 0,1,2,3,4,5,6,7
quad3 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
-----------

0600-0800 OST
0800-0830 L; /home/mchost/evla/scripts/opt/2011/04/TRFI0001_sb3662404_1.evla (SEFD test; array time)
0830-1330 LX; /home/mchost/evla/scripts/opt/2011/04/TDEM0008_sb3374030_4.evla (Demo; array time)
1330-1530 C; /home/mchost/evla/scripts/opt/2011/04/SC0346_sb4004698_1.evla (coord; array time)
1530-2130 OST

Stop at 2130 LST for testing:

== Thursday ==

   - 0900-1000 MDT: Martin/Joe w/ 64 BlBpr test
   - 1000-1300 MDT: Dave del Rizzo w/ CRM
   - 1300-1700 MDT: Vivek/Martin 3bit/CBE

*Bruce*
All these errors appear to be associated with the malfunctioning station board for ant 24-B (s006-b-3) which happens to favor row/column 5 on the baseline boards. It would be interesting to know if ant 24 is common to the missing frames. The station board has a clock error which typically can be corrected by selecting a different sampling edge, but in this case that change just makes things worse. Kerry states we don't have any spare station boards so the fault was noted and left in place.

-Bruce

19 Apr

Hi,

Work today on CBE (Martin for subarrays); not able to close the testing
loop with the CM today; more tomorrow.
Follow-up with Bruce on possible impact of the CMIB change. Ultimately
Bruce backed out of the changes; there were also issues with deformatters
that were cleaned up by Kerry.
Delays seemed to change a bit and with EA05 back in the array, delays
were found and fit.

----
Antennas:
  - EA05 back in array; needs some pointing and baselines; EA18 out. EA13 hasn't had its baselines set yet.

Correlator:
  - Rolled back from testing to wcbe_connect version; new way to generate missing scans was found so
  we'll be watchful for the alerts from Bryan's scripts.
      - if we need to fallback, the previous version is: wcbe_connect.20110329
-- 
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 = 
quad2 = 
quad3 = 
quad4 = 8,9,10,11,12,13,14,15                
-----------

0700-0900 C; BL175 (from OST)
0930-1530 L; /home/mchost/evla/scripts/opt/2011/04/TDEM0006_sb3548572_1.evla (Demo; array time)
OST 1530-1900 LST

Stop there for maintenance day.

Joe

18 Apr

Hi,

Transition from 3-bit testing today was troubled. The system was
in a wonky state; problems with 3 out of the 6 racks tested; essentially
all of racks 1, 3 and 5 had problems (as indicated in missing frames from
the CBE and then viewing the BlB GUI to see the subsequent errors). In
this case, an attempt at re-starting a set of the boards failed once
with a memory test fail - LTA RAM status fail error but ultimately the
restart wouldn't complete and we would be left with RC chip errors that
I don't know how to clear. I did try a power cycle on rack 1 to see
if that would help but it came back in an even stranger state (with
several boards never quite getting out of the programming state).
I tried to contact Kerry for some additional guidance but given the
late hour, missed him. I then generated a script to confirm that racks
2 and 4 were okay; this worked well with no dropped frames so I set
these as the only available boards for tonight.

Note there was also a new version of BlB CMIB code installed today
(any chance it's only in racks 1,3,5 Bryan?).

Projects: 10C-119 and 10C-187 will fail to generate scripts based on
this; if they come up, please put them on hold. I look at this tomorrow
when I have more guidance available.

----
Antennas:
  - EA05, EA18 are out (MP, Barn)

Correlator:
  - Update to wcbe_connect (clean-up); ran well over the weekend.
      - if we need to fallback, the previous version is: wcbe_connect.20110329
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 = 0,1,2,3,4,5,6,7
quad2 = 0,1,2,3,4,5,6,7
quad3 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,2,2,3,4,5,6,7,8,9,10,11,12,13,14,15
-----------

0630-0730 LC; /home/mchost/evla/scripts/opt/2011/04/S3184_sb3994327_1.evla (array time)
0730-0830 LC; /home/mchost/evla/scripts/opt/2011/04/S3184_sb3999409_1.evla (array time)
OST 0830-2330 LST

Stop there to review testing needs.

Joe

15 Apr (Weekend)

Hi,

Science observing throughout the day; continue through the weekend.

----
Antennas:
  - EA05, EA18 are out (MP, Barn)

Correlator:
  - Update to wcbe_connect (clean-up); we'll run with it tonight.
      - if we need to fallback, the previous version is: wcbe_connect.20110329
  - No problems with Baseline Boards today.
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

== Friday ==

2130-0130 L; 09A-106 fixed date
0130-0530 L; /home/mchost/evla/scripts/opt/2011/04/SB0517_sb3084611_1.evla (Space; array time)
0530-0630 L; /home/mchost/evla/scripts/opt/2011/04/S3184_sb3992559_1.evla (Space; array time)
0630-0730 L; /home/mchost/evla/scripts/opt/2011/04/S3184_sb3991186_1.evla (Space; array time)
0730-0800 XK; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP test; array time)
OST 0800-1200

== Saturday ==

OST 1200-1800
1800-2000 XQ; /home/mchost/evla/scripts/opt/2011/04/TSPE0010_sb3923557_1.evla (spec chan test; array time)
2000-0400 09A-106 (backup script is available at: /home/mchost/evla/scripts/opt/2011/04/09A-106_sb3951739_1.evla (fixed date)
0400-0430 XK; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP test; array time)
--
0430-0530 3-bit testing (Vivek)
--
OST 0530-1200

== Sunday ==

OST 1200-2000
2000-0400 09A-106 (backup script is available at: /home/mchost/evla/scripts/opt/2011/04/09A-106_sb3961583_1.evla (fixed date)
0400-0430 XK; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP test; array time)
OST 0430-1200

== Monday ==

OST 1200-1830
1830-2230 Ka; /home/mchost/evla/scripts/opt/2011/04/10C-221_sb3991710_1.evla (Ephem, array time)

2230 Stop for testing

Joe

14 Apr

Hi,

Testing throughout the day on 3-bit (Vivek), Martin (CBE), and
new receiver checkout (also Vivek). Issues with wind are ongoing.

----
Antennas:
  - EA05, EA18 are out (MP, Barn)
  - EA06 FRM issue; do not include
  - EA25 is stowing (more sensitive); please include if possible
  - EA13, EA17 (L), EA26 (X) are ready to go; pointing as possible
  - EA21 started out in a funny state (didn't update from the band
  in the previous script; Jim kicked it (M301) and it was fringing again.

Correlator:
  - Update to wcbe_connect (clean-up); we'll run with it tonight. 
      - if we need to fallback, the previous version is: wcbe_connect.20110329
  - No problems with Baseline Boards today.
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0630-1500 OST
1500-2100 XK; /home/mchost/evla/scripts/opt/2011/04/10B-124_sb3889033_1.evla (RSRO; array time)  
        - if the weather is good enough only.
2130-0130 L; /home/mchost/evla/scripts/opt/2011/04/09A-106_sb2983800_1.evla (RSRO; array time)
   - Note: only use this if necessary; it should come up in the OST.

-- we'll touch base with you at this point to plan out the rest of the afternoon.

Joe & Vivek

Callout: No data (CBE issue; rolled back CBE version; restarted).

Quick update...
The first science program threw the CBE into convulsions; it never relinquished the previous file (CrsroXrsro_3C84)
and was trying to this and the new file (BL175) at the same time; NICS began to go away.
I cleaned things up, rolled back to the wcbe_connect.20110329 and things will hopefully
go smoothly tonight.

Joe

Hi Steve,
Here's the nominal plan for the next bit; if the winds kick up again as the morning goes on, I'd
like to do the following (space mission/trigger observations); we're still pending some possible
testing in there so I wouldn't load more than one block ahead.
Thanks!

2130-0130 L; 09A-106 fixed date (running now)
0130-0530 L; /home/mchost/evla/scripts/opt/2011/04/SB0517_sb3084611_1.evla (Space; array time)
0530-0630 L; /home/mchost/evla/scripts/opt/2011/04/S3184_sb3992559_1.evla (Space; array time)
0630-0730 L; /home/mchost/evla/scripts/opt/2011/04/S3184_sb3991186_1.evla (Space; array time)

Joe

13 Apr

Hi,

Time today for 3-bit testing (Vivek) and CBE development (Martin). 
Maintenance day activities were many but due to high winds at the
site could not be completed in some cases. See below.
 
----
Antennas:
  - EA05, EA18 are out (MP, Barn)
  - EA26 (new X band but not secured so can't be moved); do not include.
  - EA28 issues with FRM; do not include
  - EA17, EA13 have new L-band receivers; EA17 is cross polarized; EA13
    has issues with the L cryo system (blowing fuses); don't include for L
    but okay for all others.

Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
  - Problems with some Baseline Boards at turnover again, this time in Quad 1;
  - b102-b-1 - all red; tried a restart but ended up doing a power cycle (likely effing).
  - b102-b-0 - a section (6x6) of bad CC/LTAs; rather than rephasing and restarting the
        individuals, I just did a restart and it came back.
-- 
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0630-1500 OST
1500-2100 XK; /home/mchost/evla/scripts/opt/2011/04/10B-124_sb3852398_1.evla (RSRO; array time)

Stop at 2100 for testing.

Joe

12 Apr

Hi,

So issues at turnover today:
- BlB b106-b-7 was showing the effing problem (b106-b-6 seemed to be collateral damage).
I tried to restart the board but it failed at the RXP startup; with Kerry's help we
power cycled both boards and brought them up. Kerry had also noticed some issues with
BlB B108-t-0; we tried to recover this but in the process of recovering from the
power cycling, it went down again. Kerry has recommended swapping this out as possible.
We're currently excluding this from the set of available boards.
We had some problems with the delays on EA22-D; they seemed to be varying with time;
Kerry reconfigured it and after some churning (it reset the time code on downstream
boards including the reference EA24), the delay was set and everything was recovered.

----
Antennas:
  - EA05, EA18 are out
  - EA17 has a new L-band receiver (from EA18); it was weakly fringing but needs
adjustment tomorrow. Let's keep it in for all but L-band observing.
  - EA13 has not been pointed up properly but it should be used for SC observing
(it doesn't have an L-band yet).
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
-- 
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0610-1210 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (RSRO; array time)
1210-1400 X; /home/mchost/evla/scripts/operations/syspt2hx.evla (array time)
1400-1830 OST

Stop at 1830 for maintenance day.

Joe & Vivek

11 Apr

Hi,

Given the good weather and a brief pause in the 3-bit testing; 
here's the early edition plan for today/tonight.

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

**One complication is that the OPT/OST are down from 0000-0300 and so the projects must be
accepted and put into the executor queue to cover that period as nothing can be done in
that zone.**

2230-2330 XK; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP test; array time)
2330-2345 Testing (Joe)
0000-1500 OST
1500-2000 K; /home/mchost/evla/scripts/opt/2011/04/10B-124_sb3673738_1.evla (RSRO; array time)
2000-2100 XK; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP test; array time)

Stop at 2100 for testing.

Joe

Still recovering from the OST outage (some odd holes in the schedule so do some manual; seeing winds/phase gusting up):

0200-0300 KQ; /home/mchost/evla/scripts/opt/2011/04/TSPE0008_sb3930485_1.evla (Zeeman test)
0300-0800 C; /home/mchost/evla/scripts/opt/2011/04/10C-199_sb3654032_1.evla (RSRO; array time but must start at 0300 LST)
0800-1000 C; /home/mchost/evla/scripts/opt/2011/04/BL175_sb3929656_1.evla (RSRO; array time)

1000-1500 OST
1500-2000 K; /home/mchost/evla/scripts/opt/2011/04/10B-124_sb3673738_1.evla (RSRO; array time)
2000-2100 XK; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP test; array time)

Stop at 2100 LST

08 Apr (Weekend)

Hi,

Here's the status/plan for tonight; all antennas/BlBs back in shape
for 8-bit. Program priorities have been adjusted for the end of the
configuration; the OST should be able to do its job (but we'll keep
an eye out). Winds look serious through to Sunday.

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1) 
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

== Friday ==
0600-1200 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (recirc x 4; array time)

== Saturday ==
OST with:
   - 2000-0400 fixed date
   - 0630-1230 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (recirc x 4; array time)

== Sunday == 
OST
   

== Monday ==
1200-1600 X; /home/mchost/evla/scripts/operations/sysptgx.evla (pointing - needs EA13)
        - only if wind/weather are good
1600-1830 OST
1830-2230 Ka; /home/mchost/evla/scripts/opt/2011/04/10C-221_sb3910050_1.evla (ephem; RSRO; array time)

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2230 for testing.

Joe

Hi there.  Due the antennas being stowed for high winds until after 0800 LST on Saturday, April 9, as well as multiple power loss events that afternoon, project 10C-211_sb2561334_1 was not run.  The project had an LST start time range of 0600-0800.

I ran projects using the OST starting at 0830 LST.


Dave

We'll still be planning out the week at 2230 LST (and are short some of our testers); I'd like to push
on and do:

2230-2330 KX; /home/mchost/evla/scripts/opt/2011/04/TSPE0005_sb3699117_1.evla (BP check; array time)

We'll get back in touch with you then to coordinate further testing/observing.

Thanks.

Joe

07 Apr

Hi,

Here's the status/plan for tonight; all antennas/BlBs back in shape
for 8-bit. Further observations of EA13 to bless for reintroduction
to the array.

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
  - Note: EA25 IF B has a bad deformatter; it will be replaced tomorrow but for
        tonight, no fringes are expected (IF A is also a bit dodgy).:
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.2.1) - modified to older library usage
        - this is yet another tweak on the system.
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

--checking delays
0500-0600 L; /home/mchost/evla/scripts/opt/2011/04/TRFI0001_sb3924650_1.evla (L band SEFD test; array time)
0600-0630 C; /home/mchost/evla/scripts/operations/syspt2hc.evla (30 minute sanity check)
0630-0700 X; /home/mchost/evla/scripts/operations/syspt2hx.evla (30 minutes sanity check)
0700-1300 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (recirc x 4; array time)
- OST if possible
1400-1500 LSCX; /home/mchost/evla/scripts/opt/2011/04/TCAL0002_sb3928497_1.evla (cal check; array time)
--
if K band worthy weather 1430-1500 LST (API<=10, wind<=7):
        1500-2100 K; /home/mchost/evla/scripts/opt/2011/04/10B-124_sb3652886_1.evla (RSRO; array time)
else
        OST 1500-2100

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2100 for testing.

Joe

Just a quick update; Vivek had fixed the delays on most antennas and so we just had to set EA13
at C-band (which we did).

One other correction is that I've once again forgotten the correct day for 10B-158; this changes
the complexion of the morning plans, so post 10C-211 (which is already queued):

1400-1500 LSCX; /home/mchost/evla/scripts/opt/2011/04/TCAL0002_sb3928497_1.evla (cal check; array time)
OST 1500-1830
1830-1930 *; /home/mchost/evla/scripts/opt/2011/03/10B-158_sb3848753_1.evla (VLBI coord; array time)
OST 1930-2100

Sorry for the confusion, Jim - and thanks to Mark for being ever vigilant...

Joe

06 Apr

Hi,

Here's the status/plan for tonight; all antennas/BlBs back in shape
for 8-bit.

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
  - Note: EA25 IF B has a bad deformatter; it will be replaced tomorrow but for
        tonight, no fringes are expected (IF A is also a bit dodgy).:
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.2.1) - modified to older library usage
        - this is yet another tweak on the system.
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

OST 0530-2000
  -- if weather looks bad (precipation/thunder) please run:
  0800-1600 LSCX; /home/mchost/evla/scripts/opt/2011/04/TCAL0002_sb3915831_1.evla (thunderstorm ready project)
  OST 1600-2000

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2000 for testing.

Joe

05 Apr

Based on the spectacular weather and the success of the latest fix to the CBE, we're pressing ahead with
science observing throughout the day and into tonight (any interrupt will be coordinated through Michael but
the science observing serves as further confirmation of the fix).
Plan to schedule to about 0500-0600 LST at which point, we'll review and plot out the evening.
Thanks!

Joe

Hi,

Here's the status/plan for tonight; all day science went very well; just
continuing to use the OST throughout the night until maintenance day
starts.

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
  - Note: EA25 IF B has a bad deformatter; it will be replaced tomorrow but for
        tonight, no fringes are expected (IF A is also a bit dodgy).:
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
        - if problems develop will roll back to: wcbe_20101221.0 (with mpich2-1.2.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0000-0600 OST Scheduling
0600-1200 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (RSRO; recircx4; array time)
1200-1815 OST Scheduling

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 1815 for maintenance day.

04 Apr

Hi,

Here's the status/plan for tonight.
----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
  - Note: EA25 IF B has a bad deformatter; it will be replaced tomorrow but for
        tonight, no fringes are expected (IF A is also a bit dodgy).
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (with mpich2-1.3.1)
        - if problems develop will roll back to: wcbe_20101221.0 (with mpich2-1.2.1)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0400-0535 L; /home/mchost/evla/scripts/opt/2011/04/10B-153_sb3851128_1.evla (array time)
0545-0615 L; /home/mchost/evla/scripts/opt/2011/04/TRFI0001_sb3662404_1.evla (array time; SEFD test)
0615-0915 Q; /home/mchost/evla/scripts/opt/2011/04/AC982TMB_sb3885500_1.evla (array time; RSRO)
0915-2000 OST SCHEDULING
2000-2400 Ka; /home/mchost/evla/scripts/opt/2011/04/10C-221_sb3669977_1.evla (array time; planetary)

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2400 LST for further testing.

Joe

01 Apr (Weekend)

Hi,

Here's the status/plan for tonight.
----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_20101221.0 (with mpich2-1.2.1p1); old one (from MR)
MCAF: 1.4.1

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

== Friday ==

0500-0600 X; /home/mchost/evla/scripts/widar/TVER0003_X.evla (phase xfer test; array time)
0600-1000 Ka; /home/mchost/evla/scripts/opt/2011/04/AC982TMB_sb3806838_1.evla (RSRO; array time)
OST 1000-1130 LST

== Saturday ==

OST 1130-1900 LST
1900-2000 C; /home/mchost/evla/scripts/opt/2011/04/10B-158_sb3848753_1.evla (coord observation; array time)
2000-0400 OST Fixed date
        - backup file is located at: /home/mchost/evla/scripts/opt/2011/04/09A-106_sb3861088_1.evla (FIXED DATE)
OST 0400-0800
0800-1400 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (recirc x 4; array time)

== Sunday ==
1400-1500 KQ; /home/mchost/evla/scripts/opt/2011/04/10C-145_sb3879568_1.evla (ToO; array time)
        - only if the OST hasn't pulled this up already
OST 1500-1900
1900-2000 C; /home/mchost/evla/scripts/opt/2011/04/10B-158_sb3848753_1.evla (coord observation; array time)
OST 2000-0600
0600-1200 L; /home/mchost/evla/scripts/opt/2011/04/10C-211_sb2561334_1.evla (recirc x 4; array time)

== Monday ==
1200-1400 CXK; /home/mchost/evla/scripts/opt/2011/04/10C-145_sb3883749_1.evla (ToO; array time)
        - only if the OST hasn't pulled this up already
OST 1400-2400 (tentative; pending stability over the weekend)

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

More here...

Stop at 2300 LST Friday (~1200 MDT) for turn over to DRAO.

Joe

March

31 Mar

Hi,

Here's the status/plan for tonight.
----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_20101221.0 (*rolled back to December version!*)
MCAF: 1.4.1; updated to handle sideband setting properly.

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 =
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

OST all night; we're looking for on project (ran but failed last night)
   - 10C-145
The script is squirreled away in case it's needed:
   - /home/mchost/evla/scripts/opt/2011/03/10C-145_sb3870008_1.evla

0445-0545 C; /home/mchost/evla/scripts/widar/TVER0003_sb3843300_1.evla (phase xfer; array time)
OST 0600-1900 LST
1900-2000 C; /home/mchost/evla/scripts/opt/2011/03/10B-218_sb3872442_1.evla (coord obs; array time)
OST 2000-2300 LST

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2300 LST Friday (~1200 MDT) for turn over to DRAO.

Joe

30 Mar

Hi,

Here's the status/plan for tonight.
Larry is checking the delays now and stepping through the bands as per post-maintenance 
checkout.

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect ( updated again test version ).
MCAF: 1.4.1; updated to handle sideband setting properly.

UPDATED unavailableBlBprs.txt
quad1 =
quad2 =
quad3 = 
quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

OST all night; we're looking for some key projects
   - TPLA0001
   - 10C-145
I'm hoping these get picked up by the OST but if not, we may intervene; I'll
have scripts squirreled away.

OST 0500-2000 LST

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2000 LST Thursday (~0900 MDT) for phased array testing.

Joe

quick update, we have another coordinated observation in the morning; this will likely be the last block run before
the turnover.

1830-1930 *; /home/mchost/evla/scripts/opt/2011/03/10B-158_sb3848753_1.evla (VLBI coord; array time)

29 Mar

Hi,

Here's the status/plan for tonight (between maintenance days).

----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect ( updated again test version ).
MCAF: 1.4.1; updated to handle sideband setting properly.

-----------
unavailableBlbprs.txt:
# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)

quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0530-1030 * /home/mchost/evla/scripts/operations/syscoll.evla
1030-1100 X; /home/mchost/evla/scripts/opt/2011/03/10B-221_sb3847499_1.evla (ToO; array time)
1100-1300 CXK; /home/mchost/evla/scripts/opt/2011/03/10C-145_sb3855976_1.evla (ToO; array time)
1300-1430 C; /home/mchost/evla/scripts/widar/C_Freqstep6.evla (~80 minutes)
1500-1700 Ka; /home/mchost/evla/scripts/opt/2011/03/AL746_sb3244469_1.evla (RSRO; array time)


Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 1800 Wednesday (0645 MDT) for maintenance day.

Joe

28 Mar

Here's the status/plan for tonight.
----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect with additional fixes)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect ( updated again test version ).
MCAF: 1.4.1; updated to handle sideband setting properly.

-----------
unavailableBlbprs.txt:
# standard RSRO (default) listing: mrupen  4feb11 
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0400-0700 OST
0700-1300 L; /home/mchost/evla/scripts/opt/2011/03/10C-211_sb2561334_1.evla (RSRO; 32/recirc; array time)
1300-1500 XK; /home/mchost/evla/scripts/opt/2011/03/10C-145_sb3849857_1.evla (RSRO; ToO; array time))
1530-1630 C; /home/mchost/evla/scripts/opt/2011/03/AS1020_sb3850171_1.evla (ToO; array time)
1630-2030 Ka; /home/mchost/evla/scripts/opt/2011/03/AB1353_sb3847602_1.evla (planetary; array time)

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 2030 Tuesday (0930 MDT) for maintenance day (NOTE: Special extension for today only).

Joe

Hi Tom,
This all sounds fine.
I had verbally told Larry about the 2 hr-1 hr change.
The other programs were fine to run when you did so I think we're okay.
Dancing around these programs will be difficult for the OST so we'll think harder
about how to better fill the gaps.
Thanks for the update (and not waking me). :)
Joe
- Hide quoted text -


On Tue, Mar 29, 2011 at 4:10 AM, Tom Briscoe  wrote:
Joe,

I encountered a slight schedule glitch that I felt did not warrant waking
you up for.

When I arrived at the site Larry pointed out to me that the following file
was actually 1 hour long and that OST should be used to fill the other
hour:

> 1300-1500 XK;
> /home/mchost/evla/scripts/opt/2011/03/10C-145_sb3849857_1.evla
> (RSRO; ToO; array time))

I ran OST for 1300-1400 LST and it gave me AS1020_sb3850171_1, which I
dutifully ran, *before* I realized that it was the same SB that you had
listed for 1530-1630 LST:

> 1530-1630 C; /home/mchost/evla/scripts/opt/2011/03/AS1020_sb3850171_1.evla
> (ToO; array time)

I ran 10C-145 from 1400-1500, then used OST for 1500-16:30 (which,
unfortunately, gave me nothing).

So, everything in the schedule ran successfully, just not entirely at the
scheduled times.  I assumed that AS1020 didn't need to run twice.  The
early start fit into the requested start time range listed in the
comments.

I hope this is clear; if not, please let me know.  Sorry for the confusion.

Cheers,

Tom Briscoe
VLA Operations

25 Mar (Weekend)

Hi,

Here's the status/plan for tonight.
----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect ( updated again test version ).
MCAF: 1.4.1; updated to handle sideband setting properly.

-----------
unavailableBlbprs.txt:
# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

NOTE: There are a lot of hand-generated scripts (both for not-fully-commissioned modes
as well as tests and some stock OSRO/RSRO (for coordination reasons). Please try to
ensure that the same blocks are not subsequently selected by the OST; we'll try to
keep up on this end with marking these as complete throughout the weekend.

== Friday ==
0230-0430 L; /home/mchost/evla/scripts/opt/2011/03/TSPE0009_sb3571118_1.evla (RSRO; array time; test)
0445-0945 C; /home/mchost/evla/scripts/opt/2011/03/10C-199_sb3664212_1.evla (RSRO; *fixed* time)
0945-1100 (really through 1830) OST

== Saturday ==
1100-1830 OST
1830-1930 *; /home/mchost/evla/scripts/opt/2011/03/10B-158_sb1521138_1.evla (OSRO; coordinated)
  -- leave 30 min gap breathing room before fixed data
2000-0400 fixed date schedule (OST)
0400-0700 L; /home/mchost/evla/scripts/opt/2011/03/10C-196_sb3708280_1.evla (OSRO; BP test)
0700-1300 L; /home/mchost/evla/scripts/opt/2011/03/10C-211_sb2561334_1.evla (RSRO; 32; array time)

== Sunday ==
1300-1645 OST
1645-2030 LSCX; /home/mchost/evla/scripts/opt/2011/03/TDEM0011_sb2561334_1.evla (RSRO; *fixed time*)
2030-2400 OST
0000-0300 L; /home/mchost/evla/scripts/opt/2011/03/10C-119_sb3039506_1.evla (RSRO; array time)
0300-0600 OST
0600-1000 Ka; /home/mchost/evla/scripts/opt/2011/03/AC982TMB_sb3507388_1.evla (RSRO; array time)
  - pending weather; otherwise OST

== Monday ==
1130-1530 Q; /home/mchost/evla/scripts/opt/2011/03/10C-187_sb3558833_1.evla (RSRO; 56; array time)
1530-1930 Q; /home/mchost/evla/scripts/opt/2011/03/10C-187_sb3558112_1.evla (RSRO; 56; array time)
1930-2130 KKa; /home/mchost/evla/scripts/opt/2011/03/11A-254_sb3578999_1.evla (RSRO; array time)
1930-2400 OST

   - For the OST, reminder that given the mercurial weather (fronts rolling through, etc) and the
   fixes in the OST, we should not need to select projects more than 15 minutes in
   advance; this should help ensure that the weather matches the project requirements.

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 0000 Monday (1300 MDT) for testing.

Joe

Callout: Nagios alert (CBE nodes down; rebooted, restarted CBE).

Matt called after receiving the now-dreaded nagios notes.
4 nodes were down when I looked I rebooted them and began to recover but then the remaining nodes
continued to drop out.
I cleared off the processes on cbe-control and rebooted the remainder and then brought the cbe
back up again. Based on this, on the next error (and now we must be expecting this as the error
occurred during a standard RSRO observation), I'll just reboot the whole suite.

We're into OST scheduling now.

Some updates are coming for Sunday-Monday (particularly if things go badly, I'll pull the
more challenging SBs out of the mix as I don't think we'll learn anything new).

Joe

Hi,
Just a quick update fro Monday:


== Monday ==
1130-1530 Q; /home/mchost/evla/scripts/opt/2011/03/10C-187_sb3558833_1.evla (RSRO; 56; array time)
   - scary - hoping the CBE hangs in
1530-1830 OST
1830-1930 *; /home/mchost/evla/scripts/opt/2011/03/10B-158_sb3848753_1.evla (VLBI coord; array time)

1930-2130 KKa; /home/mchost/evla/scripts/opt/2011/03/11A-254_sb3578999_1.evla (RSRO; array time)
1930-2400 OST

stop for testing.

Joe

24 Mar

Hi,

Here's the status/plan for tonight.
----
Antennas:
  - All in
Correlator:
  - Continuing to run with updated version (wcbe_connect)
--
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect ( updated test version ).
MCAF: 1.4.1; updated to handle sideband setting properly.

-----------
unavailableBlbprs.txt:
# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

0430-0630 L; /home/mchost/evla/scripts/opt/2011/03/TSPE0009_sb3571118_1.evla (32 BlBprs; array time)
0630-0800 C; /home/mchost/evla/scripts/widar/C_Freqstep3.evla (array time; 3 kHz frequency steps)
OST 0800-0900?
0900-1130 Ka; /home/mchost/evla/scripts/opt/2011/03/AL746_3243518_1.evla (array time; RSRO)
1130-1400 Ka; /home/mchost/evla/scripts/opt/2011/03/AL746_3243518_1.evla (array time; RSRO - run same file again)
OST 1400-1530?
1530-1830 Q; /home/mchost/evla/scripts/opt/2011/03/10C-186_sb3840345_1.evla (array time; RSRO)
1830-1930 *; /home/mchost/evla/scripts/opt/2011/03/10B-158_sb1521138_1.evla (array time; coord VLBA)
1930-2300 Ka; /home/mchost/evla/scripts/opt/2011/03/10B-211_sb1521118_1.evla (array time; RSRO)
2300-2400 C; /home/mchost/evla/scripts/test/zeeman/zeeman_500Hz_24mar11.evla (array time; zeeman test)

   - For the OST, reminder that given the mercurial weather (fronts rolling through, etc) and the
   fixes in the OST, we should need to select projects more than 15 minutes in
   advance; this should help ensure that the weather matches the project requirements.

Note: If there are issues tonight with the CBE (no fringes/d10); please call Joe (838-2635)
or Michael (838-2436).

Stop at 0000 (1300 MDT) for testing.

Joe

Callout: Nagios alerts (CBE nodes are down; rebooted nodes; restarted CBE).

Okay, Steve saw that nagios was reporting a downed CBE node during TSPE0009:

- I rebooted cbe-node-02; while I was doing that cbe-node-05 dropped out
so I also rebooted that.
- I brought the cbe back up but found that I was missing cbe-node-07.
Sure enough, nagios then showed it was down. I rebooted that and 
then brought things back up.

Steve did an X_osro and saw fringes.

We had an hour buffer which we bit into with this and so we're moving on
with the frequency stepping test and hoping for the best...

Joe

22 Mar

Hi,

Here's the status/plan for tonight; still anticipating high winds throughout
so pushing on low-frequency, high priority projects; RSRO should be enabled
though none beyond 16 BlBprs so no hand generated scripts tonight:
----
Antennas:
  - All in
Correlator:
  - A spate of different issues have been excited during testing but Ken managed
to rein things in so we're going to try test the system a bit tonight.

-- some changes to on-line system
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect ( updated test version ).
MCAF: 1.4.1; updated to handle sideband setting properly.

-----------
unavailableBlbprs.txt:
# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------

The key tricky test will be at 0830 LST so hopefully early enough to not wake anyone;
we might want to view the memory as James did to evaluate whether we're getting into
trouble as the program progresses.

0430-0830 L; /home/mchost/evla/scripts/opt/2011/03/TDEM0010_sb3651175_1.evla (std RSRO; array time)
0830-1430 L; /home/mchost/evla/scripts/opt/2011/03/10C-211_sb2561334_1.evla (x4 recirc; array time)
OST 1430-1630 LST
   - Reminder that given the mercurial weather (fronts rolling through, etc) and the
   fixes in the OST, we should need to select projects more than 15 minutes in
   advance; this should help ensure that the weather matches the project requirements.

Note: If there are issues tonight with the CBE (no fringes/d10); please call Michael
(838-2436) or Martin (838-2730).

Stop at 1630 LST for maintenance day.

Joe

21 Mar

Hi,

Here's the status/plan for tonight; anticipating high winds throughout
so pushing on low-frequency, high priority projects (also trying to excite
the problem with memory now that we have additional logging to hunt it down):
----
Antennas:
  - All in
Correlator:
-- 3.5 Quads available.
-- some changes to on-line system
CM: 2011-02-15 22:38 UT
CBE: wcbe_connect (test version with additional logging).
MCAF: 1.4.1; updated to handle sideband setting properly.

-----------
unavailableBlbprs.txt:
# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8
-----------


*OST: 0400-0700* - NOTE: Currently, we can't point due to the high wind gusts; we want to check
delays once the winds calm down; I based the schedule on the current wind advisory timeline.
Please call me once we've had stable winds for about 30 minutes (if it happens before 0730),
otherwise, I'll call to discuss. 
0700-1300 L; /home/mchost/evla/scripts/opt/2011/03/10C-211_sb2561334_1.evla (x4 recirc; array time)
1300-1800 C; /home/mchost/evla/scripts/opt/2011/03/10B-209_sb2504041_1.evla (std RSRO; array time)
*OST: 1800-1900*

Stop at 1900 LST for Martin to review.

Joe

Callout: No data (compromised state; restrict to OSRO and restart).

Okay last minute developments; it looks like end of day efforts have left things in a new but
very troubled state. Currently, even basic RSRO programs are causing the system to fall down.
Please use the OST all night but we'll need to restrict the accepted programs to OSRO-only.
The RSRO projects are here: https://safe.nrao.edu/wiki/bin/view/EVLA/2011BRSROStatus?sortcol=table;up=#2011_B_RSRO_ECSO_Status

If any of these come up, please put it on hold and move to the next; in the morning we'll need
to pull these off of hold so please send a note with that list.

Alternatively, give me a ring if you're in doubt.

Sorry for the churning but at least it's a good night for this to happen...
I still want to take a look at the delays when that's possible. Thanks.

Joe

Callout: No data (RSRO project hung system; residual processes; cleaned up; restart).

James noticed some odd behaviors with wcbe_bdf_mdata processes being left behind.
Sure enough the system went down; we're back up now and hopefully can stay up for the night.
We did inadvertently do a RSRO program (which likely exacerbated things but perhaps is a 
useful data point).
We'll do only OSRO for the rest.
Thanks to Steve for catching things quickly. One thing to note was that after we came up we 
were in a funny state with the antennas - all but 3 were going to the correct direction but those
3 were pointed either at zenith or some completely different direction. Steve tried several times
to command them but ultimately did a restart of the Executor and they joined the rest of the
array after that...

Joe

18 Mar (Weekend)

Hi,

Here's the status/plan for tonight; several tests to be placed in:
----
Antennas:
  - All in
Correlator:
  - All in(!)
     - Recirculation tests (x4) worked well last night so we'll try the actual science
     program tonight.
     - Updated CBE version shows the same issues as before (nodes were power-cycled by
     James). This means that we will want to re-start the CBE each day to try to avoid
     a hard crash of the system (Joe).

# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8

-- Retained 'robust' versions of CM/CBE (rolled-back from the development 2011-03-16 20:42).
CM: 2011-02-15 22:38 UT (rolled back from wcbe_connect)
CBE: wcbe_20110201.0

A fair amount of manual handling to take care of the not-fully-commissioning correlator modes:

== Friday ==
*OST: 0400-0600*
0600-1000 Ka; /home/mchost/evla/scripts/opt/2011/03/AC982TMB_sb3380352_1.evla (16 BlBprs; array time)

== Saturday ==
1000-1400 Q; /home/mchost/evla/scripts/opt/2011/03/10C-187_sb3558833_1.evla (56 BlBprs; array time)
*OST 1400-2000*
2000-0400 fixed date schedule (OST)
** 0400-0430 restart CBE ** -- Joe
*OST 0430-0600*
0600-1200 L; /home/mchost/evla/scripts/opt/2011/03/10C-211_sb2561334_1.evla (x4 recirc; array time)

== Sunday ==
*OST: 1200-1530*
1530-1930 Ka; /home/mchost/evla/scripts/opt/2011/03/10C-187_sb3550909_1.evla (56 BlBprs; array time)
*OST: 1930-0000*
0030-0330 L; /home/mchost/evla/scripts/opt/2011/03/10C-119_sb3039506_1.evla (32 BlBprs)
** 0330-0400 restart CBE ** -- Joe
*OST: 0400-0530 or 0600*
if weather is Ka quality at 0530 LST, wait and do:
   0600-1000 Ka; /home/mchost/evla/scripts/opt/2011/03/AC982TMB_sb3507388_1.evla (16 BlBprs; array time)
if not, do:
   0530-1300 L; /home/mchost/evla/scripts/opt/2011/03/10C-119_sb3699677_1.evla (32 BlBprs; array time)

== Monday ==
*OST: 1300-2130*

Stop at 2130 LST for testing/development

Joe

Callout: No fringe display (CBE nodes were down - no Nagios alerts!; rebooted and restarted CBE).

Tom noted that there was no Fringe display for 40 minutes (though d10 was working).
When I checked, I could see that 4 nodes of the CBE were down (no notes from nagios - these
began to trickle in over the next few minutes). Looking at the logs, it stopped at 02:28:21.
Left message for Martin.
Operator was going to contact James to power cycle the nodes...
Unfortunately, this seems to confirm Martin's worst conclusion (that re-starting the CBE doesn't
really help). Looks to be a long weekend...
Joe

Seem to be 3 memory events after 4:40 or so.  Whatever was running
seemed to either exit or fail and then restart.

  Attached are graphs that show memory and cpu usage.  I don't really
recall seeing that before.

James
- Show quoted text -
2 attachments — Download all attachments   View all images  
	memory-spikes.png
15K   View   Download  
	cpu-spikes.png
76K   View   Download  

Okay, we're back up.
Dave's running a quick check before resuming the schedule.
We'll call 10C-211 a fail.
I'd also like to skip the 10C-187 program at 1530 Sunday morning; please just use the OST instead
since these intense correlator resources seem to be exacerbating the problem.

Joe

17 Mar

Hi,

Here's the status/plan for tonight; several tests to be placed in:
----
Antennas:
  - All in
Correlator:
  - All in(!)
     - Further recirculation tests tonight following successful x2 (and Kerry's clean-up).
     - Martin's CBE updates (wcbe_connect) was not adequately tested; restarted services and
       tools before turning over to operations. Will resume testing tomorrow.

# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8

-- Retained 'robust' versions of CM/CBE (rolled-back from the development 2011-03-16 20:42).
CM: 2011-02-15 22:38 UT
CBE: wcbe_20110201.0 (rolled back from wcbe_connect)

*OST: 0415-0730*
0730-0800 L; /home/mchost/evla/scripts/opt/2011/03/TRFI0001_sb3662404_1.evla (array time; RFI test)
0800-0900 CK; /home/mchost/evla/scripts/opt/2011/03/10B-221_sb3673579_1.evla (array time; ToO)
0900-1000 L; /home/mchost/evla/scripts/opt/2011/03/TRSR0035_sb3656369_1.evla (array time; recircx4)
*OST: 1000-1600*
if 15 minutes before 1600 LST, the weather looks Ka quality, then please run the following:
   1600-2000 Ka; /home/mchost/evla/scripts/opt/2011/03/AB1353_sb3716203_1.evla (array time; planetary)
otherwise, OST: 1600-2000

Stop for testing.

Joe

16 Mar

Here's the status/plan for tonight; several tests to be placed in:
----
Antennas:
  - All in
Correlator:
  - All in(!); time permitting we'll try to test the full correlator for the
    first time tomorrow (CW tests/CBE recovery are higher priority).
     - Work by Kerry to clean those up; summary to come.
  - Recurrence of CBE out-of-memory problem; look for no fringes and no d10 (nagios
    e-mail notification should be in place; we did a preventative restart of the
    cbe at 1530).

# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


# hand setup projects
 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8

-- Retained 'robust' versions of CM/CBE (rolled-back from the development 2011-03-16 20:42).
CM: 2011-02-15 22:38 UT
CBE: wcbe_20110201.0

start with: 1 hour; S; /home/mchost/evla/scripts/widar/rfisurvey.evla (use only antennas with S band)
        - please make sure we include 20 and 22.
*OST: 0515-0800*
0800-0900 L; /home/mchost/evla/scripts/opt/2011/03/TRSR0035_sb3532143_recirc2.evla (array time)
0900-1000 C; /home/mchost/evla/scripts/opt/2011/03/AS1015_sb3671215_1.evla (ToO; array time)
1000-1500 L; /home/mchost/evla/scripts/opt/2011/03/10C-119_sb3693390_1.evla (32 BlBprs run)
*OST: 1500-1930*

Stop for testing (circa 0915 MDT).

15 Mar

Hi,

Here's the status/plan for tonight; pending Ken's startup; we'll modify as needed.

----
Antennas:
  - All in
Correlator:
  - All but second half of quadrant 4 are available as needed; default 16 setup.
  - Recurrence of CBE out-of-memory problem; look for no fringes and no d10 (also hopefully nagios e-mail
    notification is in place).

# standard RSRO (default) listing: mrupen  4feb11 (except for 10C-187)
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


 quad1 =
 quad2 =
 quad3 =
 quad4 =                 8,9,10,11,12,13,14,15  $ didn't check rack 8

-- Retained 'robust' versions of CM/CBE (rolled-back from the development tests).
CM: 2011-02-15 22:38 UT
CBE: wcbe_20110201.0 (rolled back from wcbe_trigalign)

The schedule for tonight is below; please note that *if* there is a 1 hour gap in the
OST scheduling anywhere, we would like to run the following:
   - 1 hour; S; /home/mchost/evla/scripts/widar/rfisurvey.evla (use only antennas with S band)

*OST: 0400-0900*
0900-1100 K; /home/mchost/evla/scripts/opt/2011/03/11A-253_sb3641034_1.evla (ToO; array time)
1100-1500 Ka; /home/mchost/evla/scripts/opt/2011/03/10C-187_sb3533652_1.evla (56 BlBprs; array time)
*OST: 1500-1645*

Stop for maintenance day.

Joe

Callout: No fringes (Nagios alerts; rebooted; restarted CBE).

Hi, 
Jim called and noted that he had lost fringes and d10 wasn't working.
I saw that there were messages from nagios that there were problems with the nodes:
- RECOVERY Host Alert: cbe-node-08 is UP
- PROBLEM Host Alert: cbe-node-04 is DOWN first at 1:41, then at 3:45
- PROBLEM Host Alert: cbe-node-07 is DOWN first at 2:01 then at 4:04
- PROBLEM Host Alert: cbe-node-08 is DOWN first at 1:58 then at 2:29

From the CBE, it looks like the last thing it was trying to do was 10C-187 (the 56 BlBpr science).

I tried to reach Martin but left a message on his house number. 
A slightly different behavior was that I was able to communicate with the cbe even before the reboot.
I began to clean up the cbe (with down, services all stop, removed wcbe_bdf_mdata processes, restart
everything). 
I hoped that we might be able to do some OSRO until maintenance but the OST didn't have anything.
James is an early riser and so caught the nagios messages and rebooted the errant nodes. More info
from him but he sees the memory signature issue was occurring throughout the day; more discussion
to follow this morning on how to proceed.

Done for now...currently there are 4 nodes working; the rebooted ones are still coming up and I can bring
those online as needed.  Maintenance starts in 45 minutes...


Joe

Mar 02

Hi,

Here's the status/plan for tonight; still an emphasis on commissioning test efforts over science (so quite a few
manually run scripts).
----
Antennas:
  - All in!
Correlator:
  - Quad 2 behaving for now; still defer experiments with recirculation and >16 baseline board support to tomorrow.
  - CBE memory issue (last night) has not been resolved so we need to be watchful for a recurrence (no fringes;
   CBE not responsive).
  - This morning a recurrence of a baseline board issue recurred (doesn't respond to configuration commands); 
    the easiest catch for this is neither fringes nor d10 are working.
   
Standard observing:

# standard RSRO (default) listing: mrupen  4feb11
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,
13,14,15
#
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

-- Retained 'robust' versions of CM/CBE (rolled-back from the development tests).
CM: 2011-02-15 22:38 UT
CBE: wcbe_20110201.0 (note tagged version!)

To start the night we'd like to run:
0500-0530 /home/mchost/evla/scripts/opt/2011/03/TVER0003_sb3592732_1.evla (array time)
Note: I provide the file but this should be able to be run via the OST (I bumped the priority up); this
is the blank sky test with switched power.

OST until 0900 LST then:
there's a fading GRB to grab (repeat from last night)
0900-1100 LST ; /home/mchost/evla/scripts/opt/2011/02/11A-253_sb3585694_1.evla (array time)
Tonight, we'd also like to get a pointing/baseline run (repeat from last night):
1100-1600 LST *; /home/mchost/evla/scripts/operations/sysptgx.evla

1600-1700 OST (hoping for 10B-200)

1700-1800 C; /home/mchost/evla/scripts/test/zeeman/zeeman_100Hz.evla (old fshift algorithm control)
1800-1900 C; /home/mchost/evla/scripts/test/zeeman/zeeman_500Hz.evla (new fshift algorithm test)

Stop on Thursday at 0900 MST (~1930 LST).

Thanks.

Joe

01 Mar

Hi,

Here's the status/plan for tonight (following focuscheck).
----
Antennas:
  - EA02 is back in the array.
Correlator:
  - Quad2 is returned to operation but not needed for tonight so no update to available Baseline boards.
   
# standard RSRO (default) listing: mrupen  4feb11
quad1 =   1,        6,7
quad2 = 0,1,2,3,4,5,6,7,8,9,10,11,12,
13,14,15
#
quad3 =       3,4,5,6,7,8,9,10,11,12,13,14,15
quad4 = 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

-- Retained 'robust' versions of CM/CBE (rolled-back from the development tests).
CM: 2011-02-15 22:38 UT
CBE: wcbe_20110201.0 (note tagged version!)

There's a fading GRB to grab:
0900-1100 LST ; /home/mchost/evla/scripts/opt/2011/02/11A-253_sb3585694_1.evla (array time)
Tonight, we'd also like to get a pointing/baseline run:
1100-1600 LST *; /home/mchost/evla/scripts/operations/sysptgx.evla

Tomorrow we need to catch up on our POLCAL observing.

Beyond this restriction please use the OST for scheduling through to Wednesday at 0630 MST (~1700 LST).


Thanks.

Joe

Callout: No fringes.

Hi,

Well we haven't had this in a while. Jim called about 0550 and noted that there were no fringe results from the 10B-200
program and that he suspected something had gone awry during the pointing run.
Indeed, when I tried to get the status of the wcbetool, it hung on me and then looking back in the logs, it appears
there was a problem that began during the 11A-253 run as early as 2352 (MST) on scan 63 of that file. Subsequent
to that I don't see integrations/bdf data being written (for the remaining 36 scans).
It never recovered when it went to sysptgx - there is the message: 
start_subscan: DEBUG: No pipeline instances found under /tmp/wcbe/daily
Martin?



Given that it's a maintenance day and we need to stop at 0630, we stopped and are leaving the system as is for
figuring out the origin of the problem- we'll need to wait on Emmanuel's tests.

Joe

February

January

-- JosephMcMullin - 2011-05-04
Topic revision: r42 - 2011-10-24, JosephMcMullin
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback