• Reviewer 7

Grade: 8

Title A Suite of ALMA Data Analysis Tools.

1. Alignment with NA ALMA Partnership strategic goals; This proposal aligns with the strategic goals ● 1. Improve and extend technical capability by developing and implementing new software and computing technologies that improve and extend capabilities for scientific research (1.2) ● 5. Strengthen the North American Radio Astronomy community by engaging early career individuals who will become the next generation of engineers and scientists ( 5.2)

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the “ALMA 2030” development documents. The scientific case is motivated by noting quite correctly there are analyses that are difficult to do directly in CASA. However, I would like to have seen more examples of the science that the proposed tools enable, whereas the what the authors describe is how the proposed tools would function. The proposed tool with the highest potential impact is probably the HTTP Range request interface, which has application beyond ALMA data to any remote source of large data. Not directly related to ALMA2030, except perhaps via connections with the Archive.

3. Quality of the upgrade conceptual design; The ALMA Operations Plan provides funding for targeted exploratory research and feasibility studies aimed at facilitating or assessing the viability of possible development projects, including assessments of opportunities for collaboration. I don’t get which part of this is a “study” as described above. It is a proposal to implement analysis functionality on a certain platform building upon existing framework and libraries. Some of the proposed functionality overlaps with tools already developed i.e. ADMIT. It’s ok to have similar functionality in different tools, but explain what if anything would be different and why. It is not clear from the proposal that authors did their homework in surveying existing tools. A pro of this is effort is that the implementation is not constrained by the limited number of python packages available in CASA (or their versions), though the just released CASA 5 has much more recent versions of e.g. numpy and matplotlib.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development); Since this is basically an implementation plan, it can be considered close to ready.

5. Strength of the consortium organization (if applicable); N/A

6. Qualifications of the key personnel of the Study; Both Ginsburg and Robitaille are experienced Python developers, and early career astronomers. They have experience in radio astronomy observations, data, and analysis. What are the roles of Rosolowsky, Koch, and Leroy?

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study; The investigators have technical expertise and past experience in developing Python.

8. Assessment of the level of risk inherent in the design; The management plan is weak which constitutes a medium to high risk. There is one week of opening meeting and one closing week and no mention of regular management in between. The personnel includes one (unbudgeted) week for a Project Manager -- what does this PM do that the PI is not doing? The Measure of Success is “If we can do >75% of what we propose, it will be a resounding success.” Ummm….why are you proposing more than you think you can accomplish? What constitutes failure -- 50%? 25%? The Measure of Success “1-2 years” past the study delivery is irrelevant. The funded personnel do not have project management experience (Ginsburg is young and Robitaille is a lone wolf). The study schedule is not a Gantt Chart in that there is no indication of what tasks depend on others or how the FTE of the junior developer will be split for the significant task overlap. I think they have promised much more than a single developer can accomplish in one year.

9. Strength of the Scientific Team supporting the Study; The team all have experience in radio astronomy observations, data, and analysis.

10. Level of support guaranteed by the Institutes; NRAO guarantees 0.25 FTE support of Ginsburg, which is sufficient for his role.

11. Budgeted cost of the Study; This total budget is the highest of all the proposed studies, though I’m not convinced that bang for buck is up there. Table 10 identifies $30,000 in potential risk impact, but this is not in the budget. Is it supposed to be?

  • Reviewer 8
Grade: 3

Title A suite of ALMA data analysis tools

1. Alignment with NA ALMA Partnership strategic goals;

An important goal for ALMA is usability of data by a broad community of stronomers. This This proposal will build on existing efforts to provide ALMA data analysis tools in a standard python framework familiar to astronomers who work with data from a range of wavelengths.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The type of analysis tools proposed here are a "low to moderate" priority in the Pathways document. This is software aimed at end users, and the investment in analysis infrastructure has the potential for long term payoff in science output. An essential element of the proposed work is an open source structure that will enable ongoing contributions and maintenance by the user community. The data analysis tools within CASA are inadequate (by design), and it is difficult to see any path to produce the needed tools be without engaging the resources of the broader community. One could quibble with prioritization of of specific tools the proposers have envisioned for this work, but all have useful applications. The key is to bring the python analysis package to a critical high value for enough users to adopt to become self-sustaining.

3. Quality of the upgrade conceptual design;

These are extensions of quality work that has already been done to develop the previous python ALMA analysis toolkit.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

This off-line software will be usable immediately on delivery.

5. Strength of the consortium organization (if applicable);

6. Qualifications of the key personnel of the Study;

Team members include leaders of previous successful ALMA development studies and projects, as well as leaders in the field of community astronomy software. The PI is a new Jansky Fellow with considerable relevant software experience. The yet to be identified junior programmer must be hired; obtaining a highly qualified junior programmer for the short time frame of this study may prove to be challenging.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

The study will be hosted at NRAO, with close ties to the CASA team.

8. Assessment of the level of risk inherent in the design;

The major risk lies in the junior programmer to be hired, both delays and overall capabilities. The proposers recognize this risk and have a reasonable mitigation plan.

9. Strength of the Scientific Team supporting the Study;

This is a strong team for the proposed work.

10. Level of support guaranteed by the Institutes;

11. Budgeted cost of the Study;

The costs for the junior and senior programmer are essential and reasonable. (But, it is not clear that ALMA development funds should be used to support 0.25 of Jansky Fellow, even for work on a specific ALMA development project.)

  • Reviewer 9
Grade: 8.0

Title: 541 - A Suite of ALMA Data Analysis Tools

1. Alignment with NA ALMA Partnership strategic goals;

Directly aligns with goals 1.2, 3.2, 4.2, 5.2

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The proposal directly addresses one of the four recommended development paths: improvements to the ALMA archive. In this case, the proposal specifically addresses the need to develop archive enrichment toolkits that strengthen the ability of scientists to obtain the maximum science return on data from the archive. One of the particular strengths of this proposal is that it does not focus solely on developing new tools, it also rightly recognizes the importance of quality documentation and usable interfaces to existing tools. Not all users are experts in programming, scripting, or installing specific packages and libraries on their computers, and indeed many users have limited control over what is installed on their own office and department computers. Simplifying access to existing tools while simultaneously developing new tools when needed is critically important to maximize the use of the archive, and this proposal should be commended for recognizing the importance of both. As another plus, the proposal is a follow-up to a previous successful study led by Adam Leroy. With many of the same team members, this a direct follow-up that takes those initial success and builds on them.

3. Quality of the upgrade conceptual design;

While the science case is very strong, and the link to the ALMA development paths is well laid out, the quality of the actual design is low. The proposal reads more like a draft outline than a final, fully fleshed out document. There are no figure captions, some sections are dominated more by lists than full sentences with sufficiently detailed explanation, and critical information is missing throughout the proposal. A few examples are given below (note that this is not intended to be a complete list):

- In the section on using bright spectral lines to ID other lines in the cube, the discussion is overly brief. While this would be a useful tool if fully developed, it’s unclear based on the information given how successful this will be. It appears to assume that all lines follow the same spatial and kinematic morphology, whereas in reality, for many objects, different lines trace very different regions (both in space and in velocity). It’s unclear how useful this tool will be, and thus should be discussed in more detail. - In the section on new tools for signal identification, there are not actually any new tools described. All that is given is a list of existing tools with no discussion of their relative merits and drawbacks, and a sentence about the need to develop an interface to implement all of them directly without having to download and install a variety of scattered packages and libraries. While this is important, it doesn’t actually address anything about “new” tools, or about helping users decide which to use. - The section on “denoising” algorithms gives no information on how the existing algorithms work, what exactly the reader should take away from the figure (which lacks any captions!), and what exactly will be done to improve/expand such algorithms under the work of this proposal. - In the “Link to existing ALMA tools” section, no links are actually given. Instead, overly brief descriptions of two existing programs are given, but how exactly these should and will link to the proposed work is not discussed at all.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

Difficult to evaluate due to the incomplete nature of this proposal.

5. Strength of the consortium organization (if applicable);

N/A.

6. Qualifications of the key personnel of the Study;

Very strong. The team has the expertise and experience necessary to carry out this project. Indeed, these are exactly the right people to be carrying out such a project.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

Very strong. The team has the expertise and experience necessary to carry out this project. Indeed, these are exactly the right people to be carrying out such a project.

8. Assessment of the level of risk inherent in the design;

Difficult to assess because of the lack of critical design details (see comment #3 above).

9. Strength of the Scientific Team supporting the Study;

Very strong. The team has the expertise and experience necessary to carry out this project. Indeed, these are exactly the right people to be carrying out such a project.

10. Level of support guaranteed by the Institutes;

No comments of significance.

11. Budgeted cost of the Study;

The costs appear reasonable and appropriate.

  • Reviewer 10
Grade: 6.0

Title: A Suite of ALMA Data Analysis Tools (PI: Ginsburg)

Summary: This study proposal seeks to deepen and continue a community-based effort to develop third-party astropy-based tools primarily ALMA image cube analysis. This effort was initiated by the PI in 2014 as an ALMA development study. An existing community toolkit exists; current extensions concern enhancements in spectral-line cube coordinate re-projection, spectral-line cube masking, automated cluster/component identification (described as signal identification by the authors), remote HTTP data access for their cube I/O module, spectral-line fitting, and several initiative described in less detail. Integration with existing tools, e.g. ADMIT, where overlapping, is promised.

1. Alignment with NA ALMA Partnership strategic goals

This proposal most closely aligns with NA ALMA Partnership strategic goals in the areas: • (1.2): Improve and extend technical capability by exploring, developing, and implementing new software and computing technologies that improve and extend capabilities for scientific research. • (5.1, 5.2): Strengthen the North American Radio Astronomy Community.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

This proposal aligns with the priority identified in “ALMA 2030” concerning improvements to the ALMA Archive, specifically the difficulty of extracting full scientific information from the “intrinsically rich” ALMA image cubes. Improvements to 3-D coordinate re-projection capabilities to align ALMA, JVLA, and NIR/IFU data cubes is especially valuable scientifically. The other algorithmic capabilities, e.g. component identification or line fitting, exist in other implementations – although the argument that they can be improved is not contested – so it can be argued that effort in these areas would deliver a lower scientific yield.

3. Quality of the upgrade conceptual design;

A great strength of the upgrade conceptual design is that it builds on existing community third-party python development for ALMA image-cube science analysis and has the potential to trigger even broader community engagement by young scientists or algorithm developers. The case for the proposal could be made more clearly if several issues were addressed in more detail: i) the specific algorithmic improvements relative to the current state of practice proposed in the areas of development, including literature citations and greater technical specificity; ii) the relationship between these astropy tools and current capabilities in ADMIT; the linkage here is not described in detail; iii) the argument for using HTTP RANGE requests versus stored procedures or more advanced data access services at the archive is made, but is not as convincing as it might be given the limited nature of RANGE requests.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

As an existing prototype, but usable, toolkit exists, this work could proceed in several areas without extensive delay. There would need to be consideration of the detailed algorithmic plans (see above), and a kickoff meeting is proposed.

5. Strength of the consortium organization (if applicable);

The consortium organization is adequate, but there is a potential mismatch between the ALMA Systems Engineering practices to be adopted and managed by the PI, and their propagation to a more distributed third-party collaboration.

6. Qualifications of the key personnel of the Study;

The key personnel are well qualified to undertake this work.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

The key institutes meet these criteria without reservation. I have no knowledge of Aperio Software, nor is their involvement described in the collaboration.

8. Assessment of the level of risk inherent in the design;

The greatest risks in this study are: i) lack of definition is key areas of proposed algorithm or capability development, leading to less successful outcome that anticipated; and ii) this investment will not necessarily allow this community collaboration to reach a critical mass of self-sustainability, thus impacting access and usefulness of these tools to the broader community.

9. Strength of the Scientific Team supporting the Study;

There is no distinct science team separate from the study team, but they have strong scientific credentials for work in this area.

10. Level of support guaranteed by the Institutes;

This is modest relative to total cost.

11. Budgeted cost of the Study;

The budgeted cost is reasonable. The authors identify the risk in hiring a junior developer. If funded I would recommend consideration of dropping this position and using existing personnel, given the 12 month project duration. There are both hiring delays and the time required to reach familiarity with the existing toolkit.

  • Reviewer 11
Grade: 2.0

A suite of ALMA data analysis tools - 541 - Ginsburg

1. Alignment with NA ALMA Partnership strategic goals;

A uniform, cleanly coded, and quasi-professionally maintained suite of software packages (even by a third party) is, in this reviewer's opinion, one of the absolute best ways to increase accessibility of the observatory. We all benefit from robust, flexible code. While ALMA has made extremely commendable progress in making the facility accessible to the non-interferometry community, the existence of more and simpler interface structures, as part of the broader Astropy ecosystem, would enable even more progress in thsi regard. This is clearly in alignment with ALMA's Strategic Goals and the 2030 Development plan.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ALMA 2030 development documents.

See above.

3. Quality of the upgrade conceptual design;

The study deliverables, outlined in Section 5.2, would be of high value to a broad community.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

As this General program supports the creation and maintenence of a suite of third party tools, interface with ALMA infrastructure is effectively unnecessary. Work on this proposed program can proceed rapidly and in parallel with any other project funded by the Development Program.

5. Strength of the consortium organization (if applicable);

N/A

6. Qualifications of the key personnel of the Study;

The propers are clearly qualified. The propers are ALMA and interferometry expert,s as well as a highly accomplished Python developers. Many of their Python codes are used as foundational componets for the Astropy project. The proposed work is an extension of their long-running efforts, and I have no doubt that they will be able to deliver what is promised.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

See above.

8. Assessment of the level of risk inherent in the design;

Low.

9. Strength of the Scientific Team supporting the Study;

See Section 6.

10. Level of support guaranteed by the Institutes;

N/A

11. Budgeted cost of the Study;

The cost seems low relative to the proposed deliverables, which will be of high and broad value.

-- AlWootten - 2017-08-02
Topic revision: r1 - 2017-08-02, AlWootten
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback