Tests of metrology system with appropriate (tiltmeter-based) AN0 and AW0 tentatively suggest that metrology correction is functioning better (at least not producing aberrant AN, AW, and NPAE). Confirmation awaits further analysis.
Started offset pointing tests on VertexRSI #1, but progress thwarted due to computing problems.
Full analysis of all good non-metrology data taken since 2008/08/06. Total of 47 runs (30 day, 17 night). Fixing pointing model derived iteratively using these data suggests:
Mostly good results
33 of 47 have RMS(OSFMW) < 2.0 arcsec
10 of 14 runs with RMS(OSFMW) > 2.0 arcsec are daytime runs
Some clear indications that at least tiltmeter metrology is useful, and perhaps needed.
For several of the observing runs with the poorest RMS, there is a clear uncorrected tilt of the azimuth axis. By allowing the tilt terms (AN and AW) to float and re-fitting, a small (0.5-2.0 arcsec) shift in AW (and sometimes a smaller shift in AN) produces much better results.
Many (most?) of these observing runs which get much better when the azimuth axis tilt terms are allowed to float are sunset runs (around 18-22 UT). Temperature gradient effect?
AIV lost most of 2008/09/01-02 due to software issues. Power outage seems to have resulted in multiple software failures.
Appears that pointing script issues with allowing metrology to activate seem to be solved. Still need to check acureport output to verify.
Metrology on/off sequential measurements designed to see if metrology makes things better/worse under very similar conditions taken 2008/08/30 and 2008/08/31. See VertexRSI Antenna 1 OPT Measurements and Resultsfor details. Conclusion from both measurements (2008/08/31 comparison better of the two) is that metrology is not improving pointing performance. Pere has sent some information to VertexRSI on this issue.
Working on script to process acureport output.
Chugging through seeing measurements. Need to weave position stream client output into analysis (to compensate for any tracking errors during seeing measurement).
Continue to straighten-out observing script which properly handles metrology system latency. In the new version of this script the antenna is sent to the (Az,El) position, with almost enough time to settle to the metrology corrected position (an idle time of 1 sec was added, after trial and error tests), and then the exposure is taken. In total, 12 seconds are spent on source, to be compared with the 18 seconds spent when using the script with a dummy exposure. Also, in the new script there is no double data acquisition, something that was worrying to JM.
Analysis of measurements made 2008/08/27-28:
Several all-sky pointing runs, both day and night. Metrology is significantly changing NPAE, AN, and AW, for unknown reasons.
At this point, it appears that metrology is making matters worse.
Several day and night time seeing measurements. Modified analysis to remove a linear fit to the measurements to take out position drifts. Seeing ~0.2 arcsec at night. Will improve analysis further to remove tracking errors (measured with position stream client).
Will continue to post analysis of these data over the coming days.
Regarding tests done so-far with/without metrology, Pere Planesas notes that: After the failure to use the standard pointing script [OpticalPointingScriptNoOrder.py]when metrology is on, and taking into account that the use of the OpticalPointingScriptMetrologyNoOrder.py script produces a waste of time (18 seconds on a star to get a 5 sec exposure), we have produced a new script that corresponds to the standard one [OpticalPointingScriptNoOrder.py] with the addition of a 5 sec sleep time just before the data acquisition, in order to allow for the local metrology corrections to be applied. This script will be tested tomorrow (2008/08/30) at day time, when the metrology corrections are larger.
Pere made quite a few measurements over the past two nights (and posted information and data today):
Several all-sky pointing runs, both day and night. First-cut analysis suggests little difference between RMS derived from metrology ON and metrology OFF runs. Pere will over the next couple of days make sequential metrology ON/metrology OFF pointing runs during both day and night to try to compare metrology ON/OFF under as close to the exact same conditions as possible.
Several day and night time seeing measurements. Analyzed one night-time seeing measurement so far and derived RMS = 0.22 arcsec (after binning to all-sky pointing integration time of 5.0 seconds).
Will post analysis of these data over the coming days.
Silence from AIV today. Waiting for information...
Troubles with VertexRSI antenna hangups/stalling thwarted attempts to check to see if metrology corrections are now being applied correctly.
Talked with Lewis and Pere (Lewis is cycling-off while Pere is cycling-on shift) to make sure that we are all on the same page. Lewis noted that:
They are still using the "hack" of commanding to each star during an optical pointing run twice (I was led to believe otherwise previously). This is a good thing as I have a concern that commanding once to a star will not give the metrology system enough time to settle down and acquire/apply a correction.
They may also extend star integration time to assure metrology acquisition.
They will regularly (at least once a night/day) make optical seeing measurements.
Some progress toward understanding problems encountered with enabling metrology. Lewis Knee reported the following late on 2008/08/26:
Several hours were spent attempting to track down the pointing data collection problems mentioned in AIVMAN-1194. After restarting, the system switched from the "relative pointing" mode of last night to a mode in which the EL pointing data collected was okay, but the AZ offsets were nonsense numbers. After another restart, the system changed back to relative pointing mode for both axes once again. After much investigation of code, the addition of extra logging code, and another restart, the problem disappeared as mysteriously as it arrived. Speculation is that the problem was originally due to problems in the Notification Channel.Two all-sky models with metrology off were observed and posted to this ticket.As part of the effort to commission metrology, we conducted some tests suggested by Vertex in Germany. We installed a reasonable 7-parameter model in the ACU, zeroed the ABM model, and (with metrology turned off) found that we could acquire stars near the centre of the OPT. This proves that there is no sign error in the ACU code at least for the pointing model, and that the scaling was correct.This led us to suspect that the apparent sign switch and scaling error we had earlier suspected may have been due instead to a mistaken use of the pointing terms AN, AW, and AN0, AW0. Therefore, we
Turned off the ACU pointing model
Turned on a non-metrology ABM pointing model
Installed the non-metrology values of AN and AW into the ACU as AN0 and AW0
Turned on metrology
Did a 100-star pointing run (night-time)
Fitted a model to the resulting data
Installed the resulting model into the ABM but without changing the values of AN0 and AW0
After that, we found that stars were found reasonably close to the OPT centre (certainly better than before) with metrology turned on.This seems faintly promising. In the morning, we plan to make day-time pointing observations with metrology on and off, and see if metrology, applied in the way discussed above, does a better job (daytime should be the time when the difference between metrology and non-metrology should be most stark.
All measurements now done using xx_POSN_RSP instead of GET_xx_ENC monitor pointing information for pointing model evaluation.
All all-sky optical pointing measurements made to-date have used an input tolerance of 2 arcsec and an integration time of 5 seconds. This tolerance is ultimately a larger value than should really be used. AIV is aware of this, and will, as appropriate, move to using a smaller (~1 arcsec) value for the positioning tolerance.
Three problems found in the way ALMA or VertexRSI enable or handle metrology:
Lewis has found an error in the Computing/Antenna ICD which has caused AIV/VertexRSI to insert the tiltmeter offsets (AN0 and AW0) into the wrong locations. Specifically, in ALMA-34.00.00.00-70.35.20.00-E3-ICD, SET_PT_MODEL_COEFF_N (page 56). The ICD lists 15 elements 1-15, and states that AN0 and AW0 go into elements 14 and 15 respectively. However, in reality, this set point has 16 elements, numbered from 0-15, and we have found by experimentation that AN0 and AW0 are actually elements 13 and 14. Until now, we entered them into 14 and 15, which screwed up our metrology measurements. Problem fixed 2008/08/24.
It was found that invoking SetPtModel sets the values of AN0 and AW0 to zero. This is bad. When invoking SetPtModel, either AN0 and AW0 should be left unchanged, or, preferably, get set to the values of AN and AW in the pointing model which is being set. Problem fixed 2008/08/24. ACTION: JGM will pursue issue with Antenna to see if VertexRSI can use some other (perhaps new) control point to host their metrology system zero point values (currently just AN0 and AW0).SOLUTION: 2008/08/25: VertexRSI has agreed to our proposed solution. Computing will coordinate CAN ID allocation and ICD update amongst all antenna vendors.
Following the discovery and correction of items (1) and (2) above, Lewis found that the metrology performance did not improve. Furthermore, the metrology corrections did not do a good job of getting the star into the centre of the OPT. In fact, it appeared that the stars were offset from the centre in AZ and EL by an amount equal in magnitude but opposite in sign to the metrology deltas being reported back by the ACU. This leads Lewis to think that there may be a sign error in the ACU metrology code. Lewis will urgently discuss this with the local Vertex personnel when they return on Monday morning (2008/08/25). UPDATE: Lewis reports that he "worked with local Vertex personnel to convince them of the reality of the software bugs in the metrology system that I uncovered. They seem convinced. There appears to be a scaling error and/or sign error in the ACU metrology deltas. This cannot be fixed at the OSF according to Vertex, so we have to wait until tomorrow morning when Germany is back at work.
Apparently night of 2008/08/25-26 observing thwarted by several problems, including:
OPT power supply (fixed).
OPT computer (fixed).
ALMA M&C software having been reverted to an older version which produces data in "relative pointing" mode. See AIVMAN-1194.
AIV and Computing have modified OPT data acquisition to use xx_POSN_RSP instead of GET_xx_ENC monitor pointing information for pointing model evaluation. First test results, where pointing performance both with and without metrology active, are as follows (same terms used in both models):
NOTE: All all-sky optical pointing measurements made to-date have used an input tolerance of 2 arcsec. This is ultimately a larger value than should really be used. AIV is aware of this, and will, as appropriate, move to using a smaller (~1 arcsec) value for the positioning tolerance.
Metrology ON: RMS = 1.76 arcsec. Scatter in individual terms ~50%-100% larger than that derived from metrology-off fits.
Metrology OFF: RMS = 1.01 arcsec. Could even be better if one or two obvious outliers are removed.
Comparing terms from each fit, see significant difference in NPAE. NPAE(Met ON) = NPAE(Met OFF) + 32 arcsec. Note further that the NPAE(Met OFF) value is very close to that derived previously (~ -6 arcsec).
Forcing fit to non-metrology model then adding terms suggests that there is a residual elevation dependent azimuth dependence.
This can be quantified by including HVCE and HVSE terms (Az/El nonperpendicularity variations as a function of elevation).
These were terms found to be necessary on the AEC prototype (and subsequently attributed to an azimuth bearing problem on that antenna).
Fit where I fix NPAE to non-metrology value (-6 arcsec) and fix all other terms except HVCE and HVSE leads to magnitudes of -13 and -25 arcsec for these terms, respectively, with a very similar RMS residual of 1.83 arcsec.
Experience teaches me to be weary of metrology...
There are currently six (now four) possibilities being considered to explain this metrology on/off discrepant behaviour:
We (ALMA) are still not collecting the proper pointing data when metrology is turned on. Possible, but I don't know what else we can compare to (monitor points, that is). Recall that we saw a similar effect when we were using GET_xx_ENC. The RMS from the set of four tests Lewis did to verify that we need to be using xx_POSN_RSP showed that, when metrology was turned on the RMS residual was worse (though only slightly).
Residual movement of the stars on the CCD in metrology mode is corrupting the data. Certainly possible, as as Lewis suggests easy to test. Lewis will do a test this evening where he puts another dummy integration in the OPT script (i.e. two dummy integrations and a third real one) will at least point us in the right direction on this issue. Should also be able to see any residual movement on the monitor while doing OPT measurements. A similar, though simpler, test would be to do two measurements of a single star. Turn metrology on and command to a star while watching the OPT monitor. Note how long it takes the image of the star to settle. Then do the exact same test with metrology turned off. This, in fact, would be about as conclusive test that one could do.
Something is wrong with how Vertex converts metrology data into position corrections. Since, to my knowledge, the metrology system has not yet been verified, this possibility is to me the most probable. One easy mistake would be that VertexRSI has gotten the sign of the correction to be applied wrong. Lewis is checking on this possibly.
UPDATE: Following the discovery and correction of two software problems noted under (4) below, Lewis found that the metrology performance did not improve. Furthermore, the metrology corrections did not do a good job of getting the star into the centre of the OPT. In fact, it appeared that the stars were offset from the centre in AZ and EL by an amount equal in magnitude but opposite in sign to the metrology deltas being reported back by the ACU. This leads Lewis to think that there may be a sign error in the ACU metrology code. Lewis will urgently discuss this with the local Vertex personnel when they return on Monday morning (2008/08/25).
Another possibility is that pointing terms in the ACU that we don't think we are using are actually being used. Lewis will look into this possibility (by checking internal pointing model values and metrology mode setting) this afternoon after lunch.
UPDATE: Lewis found several bugs in the metrology system.
Lewis has found an error in the Computing/Antenna ICD which has caused AIV/VertexRSI to insert the tiltmeter offsets (AN0 and AW0) into the wrong locations. Specifically, in ALMA-34.00.00.00-70.35.20.00-E3-ICD, SET_PT_MODEL_COEFF_N (page 56). The ICD lists 15 elements 1-15, and states that AN0 and AW0 go into elements 14 and 15 respectively. However, in reality, this set point has 16 elements, numbered from 0-15, and we have found by experimentation that AN0 and AW0 are actually elements 13 and 14. Until now, we entered them into 14 and 15, which screwed up our metrology measurements. SOLUTION: Problem fixed 2008/08/24.
It was found that invoking SetPtModel sets the values of AN0 and AW0 to zero. This is bad. When invoking SetPtModel, either AN0 and AW0 should be left unchanged, or, preferably, get set to the values of AN and AW in the pointing model which is being set. SOLUTION: Problem fixed 2008/08/24.
Metrology is being turned on/off when one think that they are turning it off/on. In other words, the part of the script (or wherever it is done) is logic-backwards.UPDATE: No. See metrology corrections being applied when metrology is turned on.
The metrology mode setting bits are being interpreted wrong. Similar to (5), except a problem on the VertexRSI side. VertexRSI is interpreting what ALMA thinks is metrology on/off as metrology off/on.UPDATE: No. See metrology corrections being applied when metrology is turned on.
Held second telecon with AIV/Computing/Antenna to discuss several issues (see notes below):
Software modifications described in JIRA ticket COMP-2412
Computing will make change from using GET_xx_ENC to xx_POSN_RSP.
UPDATE: Changes made afternoon of 2008/08/22 and tested night of 2008/08/22-23.
Got sign on additional pointing model correction term wrong.
HECE/HESE (Hooke's Law sag) terms are too large.
UPDATE2: Another iteration with correct sign. Behaviour appears to be correct now.
Metrology system latency
All agreed that:
Current metrology latency does not affect our ability to properly characterize all-sky pointing performance.
Current metrology latency does affect out ability to properly characterize other positioning performance specifications (such as fast switching, OTF, etc.). Contractor will be asked to resolve this problem.
Antenna believes that VertexRSI is aware of this problem and is addressing it (though this needs to be confirmed).
Current contractor implementation of GET_METR_DELTAS
Glendenning will coordinate change to post only metrology corrections to this monitor point with all antenna contractors via Computing IPT.
Plan for dealing with (unlikely) testing using ACU internal model (SPEM)
Will deal with this after current software fixes implemented and tested. Computing will think about what software changes are necessary to effect this change.
Extensive discussion and analysis of pointing tests done by AIV (Knee) on 2008/08/21-22. See meeting notes below for details.
Obtained metrology system test report (actually, only deals with displacement sensor tests) from Peter Napier. Will review.
Held telecon with AIV/Computing/Antenna/VA to discuss software modifications described in JIRA ticket COMP-2412. See notes below.
Considerable lack of understanding of this issue in light of tests done to-date.
More tests made night of 2008/08/21-22 by Lewis Knee hopefully will resolve this issue.
Follow-up telecon planned for 2008/08/22 at 19:00 UT to follow-up.
Tried to obtain information on actual state of metrology system, without success.
AIV (Lewis Knee and Joe McMullin) claims that measurement latency forces them to observe using a very non-standard technique. Specifically, AIV states that "with metrology turned on, it is not currently possible to obtain a pointing model; the latency in the corrections causes the source to drift during the exposures leading to a failed fit. All tests with metrology turned on will fail. A workaround was implemented to explore the metrology system (doing two sets of integrations per star; on the second integration, the antenna has settled into the metrology correction and fits are obtained)." I have pointed out that using this workaround is not consistent with the intent of evaluating the astronomical pointing performance of the antenna. RESOLVED: VertexRSI installed an ACU software update on 2008/08/23 which was tested by AIV and confirmed to allow single-command positioning (i.e. no need to command to position twice in succession).
Antenna (Art Symmes) claims that "the Metrology AT (is) 85% complete ... with the bulk of the open items involving "headache problems" such as many missing serial numbers and "device" labels...there are no open issues which were deemed to indicate a negative functionality of the MMS or the Hexapod. Regarding the "latency" - this being the situation in which the ABM begins integrating the target source before the metrology system has "set" the antenna...my understanding is that the ACU is sending a bit in a "status command" that is not currently being read by the ABM...so the Antenna occasionally continues to move after integration has begun (when metrology is active)... VA feels that the MMS is functioning as expected...however, this latency item is being reviewed by both AIV and VA."
Checking with Computing to see if they are aware of VertexRSI's proposal to use a metrology system status bit to signal availability of metrology corrections.
Still waiting to receive report from Willmeroth which purports to describe reasons for current state of metrology system.
In my opinion this is not a correctly functioning metrology system and optical pointing tests should not be made with this system in its current state.
Continued to press AIV to provide all "good" optical pointing measurements made to-date. No success yet.
Discussed software modifications associated with JIRA ticket COMP-2412 with Jeff Kern. Fix has been made, but not committed to active version of M&C software. Apparently word had not passed to Computing that this change should be installed. Kern will instruct Computing in Chile to activate changed software.
Set up a telecon for 2008/08/21 at 17:00 UT involving Computing (Marson), Science (Mangum), Antenna (Symmes), AIV (McMullin), and VA (Willmeroth) to discuss software changes necessitated by JIRA ticket COMP-2412. Goal is to make sure that everyone is on the same page with this change.
Requested past "good" all-sky optical pointing measurements from AIV (McMullin and Knee). No response yet.
McMullin reports that AIV collected all-sky pointing measurements last night (2008/08/20-21).
During ALMA/VA telecon on 2008/08/20 Willmeroth offered to send his report describing reasons for current metrology system performance (i.e. latency issues). Nothing received.