- Title: ALMA HiL Simulation Environment
Grade: 6.0
ALMA
HiL Simulation Environment
Overall:
The basic idea is to buy another (underfilled) quadrant for the correlator
and hook it up to a combination of emulators, "real" software, and actual
dishes, to reduce the amount of maintenance time needed on the full array.
Since ALMA observing time is quite expensive this project could
effectively pay for itself in less than a year.
There are however three major concerns.
(1) The proposal itself is confusing and hard to follow. Language seems
to be a barrier here, and the use of unnecessary jargon (model in the
loop, hardware in the loop) doesn't help. Further, the overall
organization of the proposal is not very clear. This makes it difficult
to evaluate the proposal with any confidence.
(2) How would this approach be affected by changes in the rest of the
system? Long-term ALMA development likely requires changes to the LO
system, the correlator, etc.; will the equipment proposed here become
obsolete as soon as the rest of the telescope is upgraded?
(3) If subarrays were fully implemented, that might take care of much
of the problem, and provide direct scientific benefits as well.
Admittedly no subarray is perfectly insulated from the rest of the
system, but equally, no emulator perfectly simulates actual observations
of the sky. [The proposal's suggestion of a full-up sky emulator seems
unnecessarily ambitious.] Some discussion here would have been useful.
1. Alignment with NA ALMA Partnership strategic goals;
Main aim is to minimize maintenance/commissioning time on-the-sky by
building an "extra" quadrant of the correlator and hooking that into a
combination of telescope emulators and (potentially) real telescopes.
This addresses strategic goals 1.3, "infrastructure upgrades that increase
scientific capability," and 2, "Increase operating efficiency".
2. Strength of the scientific case for the proposed ALMA upgrade concept;
Comment on the relevance to the ALMA 2030 development documents.
The proposal does not directly address the recommended "ALMA 2030"
development paths, although one might argue that it increases wide-field
mapping speed, by providing more observing time for science.
It is difficult to tell how much extra observing time would be made
available.
- Section 3.1 says "...between 14 and 16 hours of full system equivalent
sensitivity will be gained per month..."
- The table in 3.5 seems to be saying that a total of 184 hrs/month (7.7
days~ 25% of 30 days) is currently used for testing (though this is not
very clear in the text), and concludes by saying "It is expected that
big portion of these tasks will be don [sic] in the new testing
environment, and generate an additional 10% of time for scientific
observation". But is this (1) we will save 10% of 184 hrs= 18 hrs/month
which would have been used for testing, or (2) we will increase the
total scientific observing time per month by 10% (~3 days= 72
hrs/month)?
So is it the sum of these two which is gained per month, i.e., 14-16 hrs +
17 or 72 hrs? Understanding this is vital for evaluation of the benefits
of the proposed upgrade.
As a rough comparison, if ALMA costs $77M/yr, that's $6.4M/mo, or
~$210k/day. So 15 hrs costs ~$130k, and 72 hrs is $630k. The total
project costs ~$1.5M, so if you really gain 72hrs/month, naively you are
ahead in about 1.5M/0.63M~ 2.4 months. If it's 15 hrs/mo, you are ahead
in about a year. This neglects all the lost time for these people
spending their time implementing this rather than working on the rest of
the system, among other things...
3. Quality of the upgrade conceptual design;
The proposal is quite difficult to read, presumably largely due to
language issues. But some aspects are more basic.
- The actual project is not really introduced until p.9.
- Why do we spend time in section 5.3, discussing options which are
actually impossible to implement (given the available correlator chips)?
- "It was decided to procure additional hardware to provide a single set
of photonic reference." -- Why? Given that this is ~1/3 of the project
cost this should be more fully justified.
- Must the current system be taken down while the civil works are being
done in the correlator room? (sections 5.6, 6)
4. Readiness for production in the context of the ALMA Development Plan (the
aim is to support a range of upgrades including both those which can be
implemented rapidly and those requiring longer-term research and
development);
The claim is that this is not intrusive and is ready-to-go. Given the
people involved I tend to believe this but it's not really justified
in the text.
5. Strength of the consortium organization (if applicable);
Not applicable I think.
6. Qualifications of the key personnel of the Study;
The people involved are those most involved in day-to-day testing and
commissioning, a very strong plus for this proposal.
7. Technical expertise, past experience (also in series production, if
relevant) and technical facilities in the Institutes taking part in the
Study;
The proposers are embedded in day-to-day ALMA operations and are
intimately familiar with the aspects of the telescope they propose to
modify.
8. Assessment of the level of risk inherent in the design;
Hard to assess based on this proposal. Risk register seems a little
blithe in assigning pretty large risks of e.g. needing additional software
development, but claiming quite low costs if this happens.
9. Strength of the Scientific Team supporting the Study;
Not relevant.
10. Level of support guaranteed by the Institutes;
In-kind staff contributions amount to ~1/6 of the cost.
11. Budgeted cost of the Study;
Suspect risk costs are underestimated. Otherwise it seems well costed,
though one may question the need for some individual items.
============================================================================
Grade: 3.5
Title: ALMA
HiL Simulation Environment
1. Alignment with NA ALMA Partnership strategic goals
The proposed project is to build a small correlator to support ALMA operation. The project, which if implemented well, will help to smoothen the current ALMA operations, and help to allow new firmware to be tested and technical issues to be more readily resolved. In that sense, the proposal will enhance the scientific productivity of ALMA across the board. In this sense, the project aligns with NA ALMA strategic goals.
Strength of the scientific case for the proposed ALMA upgrade concept
The science case is indirect. Since any strengthening of ALMAs operation will help the scientific productivity of ALMA, the successful implementation of a simulation environment will improve the productivity of ALMA.
Quality of the upgrade conceptual design
The plan seems to be fine. In addition, the project will use the existing spare correlator chips to build up new processing boards. This is actually a good move because in this way, a new stack of spares will be made available. This will prove to be important because as the ALMA correlator ages, board failures will be inevitable. In all big projects, there is a move to keep an operating spare to be called up for either replacement of simulation of issues. So, this project goes in the right way.
Readiness for production in the context of the ALMA Development Plan The project is well defined and the approach is methodological.
Strength of the consortium organization
The people involved seem to be people involved in the day-to-day operation of ALMA. Another benefit of the project is that this will provide a training ground for new ALMA technical staff. The project will also help the local staff to deepen their understanding of the system.
Qualifications of the key personnel of the Study Well qualified.
Technical expertise, past experience
Appropriate
Assessment of the level of risk inherent in the design
The counter-balance to the project is that ALMA is going to embark on the development of a second-generation correlator. Depending on the time scale of this new correlator, this project may be an overlap of effort.
Strength of the Scientific Team supporting the Study Non-applicable.
Level of support guaranteed by the Institutes Appropriate
Budgeted cost of the Study
The offerers cost of $1.3M is appropriate for this magnitude of project. The project seems an important infrastructure to ALMA as a mature observatory.
Grade: 4.0
ALMA HIL simulation environment (Corder)
1. Alignment with NA ALMA Partnership strategic goals;
Ideally aligned with the goal of enhancing ALMA's efficiency.
2. Strength of the scientific case for the proposed ALMA upgrade concept;
Comment on the relevance to the ALMA 2030 development documents.
The authors included an exhaustive list of reasons why the implementation of this improved simulation environment (including the 5th quadrant of the correlator) would improve efficiency. Doing software and script testing in a simulation environment that doesn't disrupt science operations is absolutely paramount, and the authors made a good case for how their proposed upgrades would more even more testing away from the full system.
Note that not all of the figures are clear, and most don't have captions. This was not the clearest or best written proposal; however, that is balanced out somewhat by the critical upgrade the authors are proposing.
3. Quality of the upgrade conceptual design;
Excellent.
4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);
Excellent: the duration of the project is only 24 months, which is quite expeditious. Much of the development is building on what has been done before (e.g., the 5th quadrant of the correlator).
5. Strength of the consortium organization (if applicable);
Excellent: includes many ALMA veterans who have been working for years in the environment in which they will be implementing their simulator upgrades.
6. Qualifications of the key personnel of the Study;
Excellent: highly qualified, mainly current ALMA staff.
7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;
Excellent: members of the team have extensive experience in all relevant areas, from software development, correlator programming, antenna hardware, management, etc.
8. Assessment of the level of risk inherent in the design;
Low, mainly having to do with hardware price and availability.
9. Strength of the Scientific Team supporting the Study;
N/A
10. Level of support guaranteed by the Institutes;
Excellent: plenty of in-house support at JAO, provided those supporting the project can afford the additional time.
11. Budgeted cost of the Study;
Very reasonable: an impressively low total number of FTEs, perhaps due to ALMA funding already being in place for most staff?
Grade: 7
ALMA HIL SIMULATION ENVIORNMENT
1. Alignment with NA ALMA Partnership strategic goals;
Increases operations efficiency - claims gain of ~10% (but seems gut-feeling estimate).
There is no direct alignment with there ALMA 2030 document.
2. Strength of the scientific case for the proposed ALMA upgrade concept;
Comment on the relevance to the ALMA 2030 development documents.
The science case for this upgrade is explicitly the increase in operations efficiency obtained by moving part of the testing process to the proposed HiL Simulation environment. The strength of the scientific case is weakened by the lack of detail as to the specific increase in operations that will occur.
3. Quality of the upgrade conceptual design;
A lot of work has gone in to thinking this development through and considering the possible utility of the upgrade. The proposal, however, was not written with the reviewer in mind and it is difficult to quantify the final utility of the upgrade - which is necessary to assess the quality with respect to the Development Plan.
4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);
From the proposal it would appear that significant work has already been undertaken for this project and therefore it appears ready to be implemented.
5. Strength of the consortium organization (if applicable);
The project is led almost exclusively by JAO staff.
6. Qualifications of the key personnel of the Study;
The key personnel are all expert leaders within ALMA and have excellent qualifications.
7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;
n/a
8. Assessment of the level of risk inherent in the design;
There appears to be little risk in putting this project in place but there is very little discussion of the risk that the operational benefit might be significantly lower that anticipated. An additional risk not discussed is how this project might be quickly out-dated by other developments to ALMA
how easily upgradable will this testing environment be (I note that it requires identical parts as used, for example, in the correlator).
9. Strength of the Scientific Team supporting the Study;
n/a
10. Level of support guaranteed by the Institutes;
The support being guaranteed is staff effort and logistics.
11. Budgeted cost of the Study;
The budgeted cost of the project is $1,322,450.
It is entirely unclear why this project is being proposed through North America rather than jointly across all regions.
Grade: 7.0
Title: ALMA
HiL Simulation Environment
1. Alignment with NA ALMA Partnership strategic goals;
This is a proposal to create a more realistic test environment for ALMA operational software. The enhanced environment would include a copy of one quadrant of the ALMA correlator which could be connected to as many as four ALMA antennas, various hardware simulators of antenna subsystems, and other software testing tools. The motive is to increase science observing time by offloading test activities from the main array and by improving the quality of operational software updates. The key strategic goal addressed by this proposal is to increase ALMA operating efficiency.
2. Strength of the scientific case for the proposed ALMA upgrade concept;
Comment on the relevance to the ALMA 2030 development documents.
The proposed work addresses operational inefficiencies which affect all ALMA science.
3. Quality of the upgrade conceptual design;
Fundamentally, the concept of having a software test environment that duplicates the architecture and behavior of the real target environment makes a lot of sense. The hard questions are how to do this efficiently, and how to ensure that the simulation environment remains in sync with the target environment. For example, the extra correlator quadrant proposed here will require significant time effort, and funds to produce, but will become obsolete when the main correlator is upgraded. On a smaller scale, keeping the characteristics of the simulation hardware in sync with the many subsystems of the actual array will be a constant task.
This reviewer doesn't claim to know the best answer to these questions, but does wonder if a better path forward might be to look to subarrays as providing the best default test environment. Subarrays will always look more like the real array than a separately maintained test environment. Perhaps future upgrades to the correlator and other ALMA systems should be designed specifically to facilitate running subarrays as a software test environment, with special monitoring and logging modes, for instance.
4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);
There are no impediments to starting the proposed work, apart from possible obsolete component supply issues as noted by the proposers.
5. Strength of the consortium organization (if applicable);
This is a proposal originating within the JAO.
6. Qualifications of the key personnel of the Study;
The key personnel are eminently qualified. However, reflecting the opinion stated above, it would be a proposal from the same team targeting an evolving test strategy centered on use of subarrays.
7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;
This is an internal proposal; the proposers are the cognizant experts.
8. Assessment of the level of risk inherent in the design;
The primary risk, as noted above, is premature obsolescence of the simulation hardware, especially the duplicated correlator quadrant.
9. Strength of the Scientific Team supporting the Study;
N/A
10. Level of support guaranteed by the Institutes;
N/A
11. Budgeted cost of the Study;
The budgeted cost is in line with the scope of the proposed work.
--
AlWootten - 2017-05-09
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.