Az Oscillation Issues
We had issues with the az axis when in cold-wtx/low rate mode. This has been traced down to an error in setting the CCU's AzFilter
.Vel (Az max velocity) to a reduced value. The fix is to just not reduce this value in cold weather mode. Only the ACU's max azimuth rate should be changed.
What does AzFilter.Vel do?
There are some derived quantities that are calculated from this value. One is the gain of the 'square-root' region. By reducing it, we increase the square-root region gain to twice the normal value. In order to maintain the same gain, when reducing the velocity by half, the acceleration needs to be reduced by a factor of four. In real numbers, when going from 0.6 deg/sec (normal az rate) to 0.3 deg/sec, the acceleration would need to be adjusted from 0.1 to 0.025 deg/sec/sec.
But this is not necessary, since the numbers are relative. So again, the real 'fix' is to not update the CCU
values at all. (
- currently deployed)
Az Oscillation Checks
There is code which attempts to detect an axis oscillation, and when detected disable the axis. We were all asking ourselves why this check didn't trigger during all these runs with oscillations present.
The algorithm works by examining position error only. The algorithm is to checks every time position-error changes sign (i.e. crosses through zero). At each position-error crossing it checks what the maximum position error was since the previous zero-crossing. If the value of maximum position error is greater than a threshold, a counter is incremented. If the counter is incremented more than 5 times in a row, the axis will be disabled. If however the maximum position error falls below the threshold, the counter is reset to zero.
Problems With this Check
The algorithm is making some assumptions here.
First, it is characterizing an oscillation as a position-loop only quantity. If there were issues in the rate loop, the check would not be helpful is detecting it.
Second, it is designed only to catch oscillations where position error is centered about zero. While most typical responses are centered near zero, it is quite plausible that the position error could be off-center due to the position 'lagging' behind. (Although this should be caught by the 'I' part of the PID.)
Third, the threshold (a configuration file value) is currently set rather large, so the trigger is almost never hit. Currently it is set at 0.25 degrees. None of the cases we examined had values near this range.
Finally, the algorithm can be 'fooled' by a waveform which has more than one frequency mixed in with it, causing the oscillation counter to be reset when beating between the two waveforms causing a temporary low-amplitude position error peaks. For example we could have a large-peak (counter goes to '1'); then another large peak (counter goes to '2'); then a small peak (counter resets back to zero) and them more large peaks. The bottom line here is that the small peaks between the large ones delays the triggering of the oscillation check.
During the analysis of the servo problem, we ran into some rather odd issues where the antenna would display a 'Command' value far away from what was specified. This had nothing to do with the servo issues above. The problem here was a bug which caused a strange mixing of ephemeris data from the observers script, with commands the operator was issuing.
In more detail, the antenna accepts commands with an associated time. Normally the time is a quantity relative to the scan start time. However, when an observer is using an ephemeris the commands contain absolute rather than relative times. This is understood by the antenna in some special ways. The bug here is that when the operator starting using cleo, the absolute times were not cleared out as they should have been. The result was a confusing "Go from X at rate Y starting a hour ago". Ron M. mentioned he would have a fix ASAP for this.