• Title: ALMA HiL Simulation Environment

  • Referee1
Grade: 6.0

ALMA HiL Simulation Environment

Overall: The basic idea is to buy another (underfilled) quadrant for the correlator and hook it up to a combination of emulators, "real" software, and actual dishes, to reduce the amount of maintenance time needed on the full array. Since ALMA observing time is quite expensive this project could effectively pay for itself in less than a year.

There are however three major concerns.

(1) The proposal itself is confusing and hard to follow. Language seems to be a barrier here, and the use of unnecessary jargon (model in the loop, hardware in the loop) doesn't help. Further, the overall organization of the proposal is not very clear. This makes it difficult to evaluate the proposal with any confidence.

(2) How would this approach be affected by changes in the rest of the system? Long-term ALMA development likely requires changes to the LO system, the correlator, etc.; will the equipment proposed here become obsolete as soon as the rest of the telescope is upgraded?

(3) If subarrays were fully implemented, that might take care of much of the problem, and provide direct scientific benefits as well. Admittedly no subarray is perfectly insulated from the rest of the system, but equally, no emulator perfectly simulates actual observations of the sky. [The proposal's suggestion of a full-up sky emulator seems unnecessarily ambitious.] Some discussion here would have been useful.

1. Alignment with NA ALMA Partnership strategic goals;

Main aim is to minimize maintenance/commissioning time on-the-sky by building an "extra" quadrant of the correlator and hooking that into a combination of telescope emulators and (potentially) real telescopes. This addresses strategic goals 1.3, "infrastructure upgrades that increase scientific capability," and 2, "Increase operating efficiency".

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The proposal does not directly address the recommended "ALMA 2030" development paths, although one might argue that it increases wide-field mapping speed, by providing more observing time for science.

It is difficult to tell how much extra observing time would be made available. - Section 3.1 says "...between 14 and 16 hours of full system equivalent sensitivity will be gained per month..." - The table in 3.5 seems to be saying that a total of 184 hrs/month (7.7 days~ 25% of 30 days) is currently used for testing (though this is not very clear in the text), and concludes by saying "It is expected that big portion of these tasks will be don [sic] in the new testing environment, and generate an additional 10% of time for scientific observation". But is this (1) we will save 10% of 184 hrs= 18 hrs/month which would have been used for testing, or (2) we will increase the total scientific observing time per month by 10% (~3 days= 72 hrs/month)? So is it the sum of these two which is gained per month, i.e., 14-16 hrs + 17 or 72 hrs? Understanding this is vital for evaluation of the benefits of the proposed upgrade.

As a rough comparison, if ALMA costs $77M/yr, that's $6.4M/mo, or ~$210k/day. So 15 hrs costs ~$130k, and 72 hrs is $630k. The total project costs ~$1.5M, so if you really gain 72hrs/month, naively you are ahead in about 1.5M/0.63M~ 2.4 months. If it's 15 hrs/mo, you are ahead in about a year. This neglects all the lost time for these people spending their time implementing this rather than working on the rest of the system, among other things...

3. Quality of the upgrade conceptual design;

The proposal is quite difficult to read, presumably largely due to language issues. But some aspects are more basic.

- The actual project is not really introduced until p.9.

- Why do we spend time in section 5.3, discussing options which are actually impossible to implement (given the available correlator chips)?

- "It was decided to procure additional hardware to provide a single set of photonic reference." -- Why? Given that this is ~1/3 of the project cost this should be more fully justified.

- Must the current system be taken down while the civil works are being done in the correlator room? (sections 5.6, 6)

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

The claim is that this is not intrusive and is ready-to-go. Given the people involved I tend to believe this but it's not really justified in the text.

5. Strength of the consortium organization (if applicable);

Not applicable I think.

6. Qualifications of the key personnel of the Study;

The people involved are those most involved in day-to-day testing and commissioning, a very strong plus for this proposal.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

The proposers are embedded in day-to-day ALMA operations and are intimately familiar with the aspects of the telescope they propose to modify.

8. Assessment of the level of risk inherent in the design;

Hard to assess based on this proposal. Risk register seems a little blithe in assigning pretty large risks of e.g. needing additional software development, but claiming quite low costs if this happens.

9. Strength of the Scientific Team supporting the Study;

Not relevant.

10. Level of support guaranteed by the Institutes;

In-kind staff contributions amount to ~1/6 of the cost.

11. Budgeted cost of the Study;

Suspect risk costs are underestimated. Otherwise it seems well costed, though one may question the need for some individual items.

============================================================================

  • Referee3
Grade: 3.5

Title: ALMA HiL Simulation Environment

1. Alignment with NA ALMA Partnership strategic goals

The proposed project is to build a small correlator to support ALMA operation. The project, which if implemented well, will help to smoothen the current ALMA operations, and help to allow new firmware to be tested and technical issues to be more readily resolved. In that sense, the proposal will enhance the scientific productivity of ALMA across the board. In this sense, the project aligns with NA ALMA strategic goals.

Strength of the scientific case for the proposed ALMA upgrade concept

The science case is indirect. Since any strengthening of ALMA’s operation will help the scientific productivity of ALMA, the successful implementation of a simulation environment will improve the productivity of ALMA.

Quality of the upgrade conceptual design

The plan seems to be fine. In addition, the project will use the existing spare correlator chips to build up new processing boards. This is actually a good move because in this way, a new stack of spares will be made available. This will prove to be important because as the ALMA correlator ages, board failures will be inevitable. In all big projects, there is a move to keep an operating spare to be called up for either replacement of simulation of issues. So, this project goes in the right way.

Readiness for production in the context of the ALMA Development Plan The project is well defined and the approach is methodological. Strength of the consortium organization

The people involved seem to be people involved in the day-to-day operation of ALMA. Another benefit of the project is that this will provide a training ground for new ALMA technical staff. The project will also help the local staff to deepen their understanding of the system.

Qualifications of the key personnel of the Study Well qualified. Technical expertise, past experience Appropriate

Assessment of the level of risk inherent in the design

The counter-balance to the project is that ALMA is going to embark on the development of a second-generation correlator. Depending on the time scale of this new correlator, this project may be an overlap of effort.

Strength of the Scientific Team supporting the Study Non-applicable. Level of support guaranteed by the Institutes Appropriate Budgeted cost of the Study

The offerer’s cost of $1.3M is appropriate for this magnitude of project. The project seems an important infrastructure to ALMA as a mature observatory.

  • Referee4
Grade: 4.0

ALMA HIL simulation environment (Corder)

1. Alignment with NA ALMA Partnership strategic goals;

Ideally aligned with the goal of enhancing ALMA's efficiency.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The authors included an exhaustive list of reasons why the implementation of this improved simulation environment (including the 5th quadrant of the correlator) would improve efficiency. Doing software and script testing in a simulation environment that doesn't disrupt science operations is absolutely paramount, and the authors made a good case for how their proposed upgrades would more even more testing away from the full system.

Note that not all of the figures are clear, and most don't have captions. This was not the clearest or best written proposal; however, that is balanced out somewhat by the critical upgrade the authors are proposing.

3. Quality of the upgrade conceptual design;

Excellent.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

Excellent: the duration of the project is only 24 months, which is quite expeditious. Much of the development is building on what has been done before (e.g., the 5th quadrant of the correlator).

5. Strength of the consortium organization (if applicable);

Excellent: includes many ALMA veterans who have been working for years in the environment in which they will be implementing their simulator upgrades.

6. Qualifications of the key personnel of the Study;

Excellent: highly qualified, mainly current ALMA staff.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

Excellent: members of the team have extensive experience in all relevant areas, from software development, correlator programming, antenna hardware, management, etc.

8. Assessment of the level of risk inherent in the design;

Low, mainly having to do with hardware price and availability.

9. Strength of the Scientific Team supporting the Study;

N/A

10. Level of support guaranteed by the Institutes;

Excellent: plenty of in-house support at JAO, provided those supporting the project can afford the additional time.

11. Budgeted cost of the Study;

Very reasonable: an impressively low total number of FTEs, perhaps due to ALMA funding already being in place for most staff?

  • Referee6
Grade: 7

ALMA HIL SIMULATION ENVIORNMENT

1. Alignment with NA ALMA Partnership strategic goals;

Increases operations efficiency - claims gain of ~10% (but seems ‘gut-feeling’ estimate). There is no direct alignment with there ALMA 2030 document.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The science case for this upgrade is explicitly the increase in operations efficiency obtained by moving part of the testing process to the proposed ‘HiL’ Simulation environment. The strength of the scientific case is weakened by the lack of detail as to the specific increase in operations that will occur.

3. Quality of the upgrade conceptual design;

A lot of work has gone in to thinking this development through and considering the possible utility of the upgrade. The proposal, however, was not written with the reviewer in mind and it is difficult to quantify the final utility of the upgrade - which is necessary to assess the quality with respect to the Development Plan.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

From the proposal it would appear that significant work has already been undertaken for this project and therefore it appears ready to be implemented.

5. Strength of the consortium organization (if applicable);

The project is led almost exclusively by JAO staff.

6. Qualifications of the key personnel of the Study;

The key personnel are all expert leaders within ALMA and have excellent qualifications.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

n/a

8. Assessment of the level of risk inherent in the design;

There appears to be little risk in putting this project in place but there is very little discussion of the risk that the operational benefit might be significantly lower that anticipated. An additional risk not discussed is how this project might be quickly out-dated by other developments to ALMA … how easily upgradable will this testing environment be (I note that it requires identical parts as used, for example, in the correlator).

9. Strength of the Scientific Team supporting the Study;

n/a

10. Level of support guaranteed by the Institutes;

The support being guaranteed is staff effort and logistics.

11. Budgeted cost of the Study;

The budgeted cost of the project is $1,322,450.

It is entirely unclear why this project is being proposed through North America rather than jointly across all regions.

  • Referee7
Grade: 7.0

Title: ALMA HiL Simulation Environment

1. Alignment with NA ALMA Partnership strategic goals;

This is a proposal to create a more realistic test environment for ALMA operational software. The enhanced environment would include a copy of one quadrant of the ALMA correlator which could be connected to as many as four ALMA antennas, various hardware simulators of antenna subsystems, and other software testing tools. The motive is to increase science observing time by offloading test activities from the main array and by improving the quality of operational software updates. The key strategic goal addressed by this proposal is to increase ALMA operating efficiency.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The proposed work addresses operational inefficiencies which affect all ALMA science.

3. Quality of the upgrade conceptual design;

Fundamentally, the concept of having a software test environment that duplicates the architecture and behavior of the real target environment makes a lot of sense. The hard questions are how to do this efficiently, and how to ensure that the simulation environment remains in sync with the target environment. For example, the extra correlator quadrant proposed here will require significant time effort, and funds to produce, but will become obsolete when the main correlator is upgraded. On a smaller scale, keeping the characteristics of the simulation hardware in sync with the many subsystems of the actual array will be a constant task.

This reviewer doesn't claim to know the best answer to these questions, but does wonder if a better path forward might be to look to subarrays as providing the best default test environment. Subarrays will always look more like the real array than a separately maintained test environment. Perhaps future upgrades to the correlator and other ALMA systems should be designed specifically to facilitate running subarrays as a software test environment, with special monitoring and logging modes, for instance.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

There are no impediments to starting the proposed work, apart from possible obsolete component supply issues as noted by the proposers.

5. Strength of the consortium organization (if applicable);

This is a proposal originating within the JAO.

6. Qualifications of the key personnel of the Study;

The key personnel are eminently qualified. However, reflecting the opinion stated above, it would be a proposal from the same team targeting an evolving test strategy centered on use of subarrays.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

This is an internal proposal; the proposers are the cognizant experts.

8. Assessment of the level of risk inherent in the design;

The primary risk, as noted above, is premature obsolescence of the simulation hardware, especially the duplicated correlator quadrant.

9. Strength of the Scientific Team supporting the Study;

N/A

10. Level of support guaranteed by the Institutes;

N/A

11. Budgeted cost of the Study;

The budgeted cost is in line with the scope of the proposed work.

-- AlWootten - 2017-05-09
Topic revision: r2 - 2017-05-10, AlWootten
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback