• ALMA Data Mining Toolkit II
  • Referee1
Grade: 3.0

ALMA Data Mining Toolkit II (Mundy)

0. Summary Good incremental project building on past success to provide improvements in a vital area (post-processing software & analysis tools). Not earth-shattering but the proposed analysis tools absolutely should be provided.

Main negative is lack of independent user reviews of the software provided so far. If the users are happy with the current results this project should definitely be done.

1. Alignment with NA ALMA Partnership strategic goals;

Directly addresses stragic goal 1.2 -- new software and computing technologues that improve and extend capabilities for scientific research

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

Overall positives: - Directly addresses ALMA2030 development path #1, Improvements to the ALMA Archive. - Making CASA more user- and science-friendly is essential to making full scientific use of ALMA and other world-class interferometers. - I appreciate the emphasis on user support, and the ability of users to add their own functions and play with the various tasks/flows to match their desires. - The new browser/flow tools seem well thought-out and appealing - Generally good choices for the new tasks -- these are mostly what any scientist would want, pushing the boundaries a bit (e.g., hyperfine line analysis) but of great general interest. : Feature Identification (Clumpfind, dendograms) seems a fairly standard tool these days though personally I do not find the results very informative

Overall negatives: - Quality of current software is vital in deciding whether to continue down this path, but there is no real way to evaluate that based on the information provided. Independent, unbiased "user reviews" of both the quality of ADMIT SDPs, and the ability of users in practice to add new tasks/flows/functions of their own, would have been comprelling. - The scientific case for uv model fitting is not particularly strong, and I was surprised to see this so high up the list. This is much more of a niche application than the other proposed tasks. - Listening to community suggestions sounds good but it's hard to see how that fits with the timeline and all the work that needs doing simply for the tasks already chosen. - The individual analysis tasks are rather hazy -- more details about exact functions and inputs/outputs would have been useful, and easy to provide.

Comments: - Overall the proposal seems more like "please keep funding our software group" than one targeting a specific problem or development. Of course software is incredibly important for instruments producing so much data, and realisitically I think theirs is the correct approach, so I'm reluctant to "ding" the proposers for their honesty!

3. Quality of the upgrade conceptual design;

Overall concept is clear and straightforward, given the work this group has already done for ADMIT.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

The work has effectively already started and could continue fairly seamlessly.

5. Strength of the consortium organization (if applicable);

The same time carried out ADMIT, which was a harder project, so the proposed extensions should be easy for them.

The small size and co-location of the group are definite pluses.

6. Qualifications of the key personnel of the Study;

The individuals involved are all excellent, with expertise on both the astronomical and on the programming sides. I am not sure how experienced they are with direct user support.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

See above.

8. Assessment of the level of risk inherent in the design;

As with all software (and as reflected in their risk register), it is not unlikely that they will not be able to finish all the work they propose in the indicated time. But this group would definitely produce software which the wider community will appreciate and use, and which will add to ALMA's user community and scientific productivity. Basically you are buying several years' effort from some good programmers who are personally excited about the scientific results of their work.

9. Strength of the Scientific Team supporting the Study;

Excellent.

10. Level of support guaranteed by the Institutes;

Should be very good.

11. Budgeted cost of the Study;

$1M sounds about right for a couple years' effort from 4 people.

I am a little confused by Table 2.0, which suggests everyone works half-time for 12 months only, vs. the claimed performance period of 24 months. Perhaps 12 months is supposed to mean, 24 months at half time?

============================================================================

  • Referee4
Grade: 6.0

ALMA Data Mining Toolkit II (Mundy)

1. Alignment with NA ALMA Partnership strategic goals;

This project is fully aligned with the strategic goal to develop "new software and computing technologies that improve and extend capabilities for scientific research." Efficiency and ease-of-use are the goals here.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

Relatively strong: while not improving the archive directly, the results from this project will make their way into the archive, thus making it a more likely clearinghouse for users who want to easily scan results from public datasets.

From the proposal, it is clear that ADMIT I (funded by a Cycle 2 ADP) was a success, and addressed a number of key issues (especially for non-experts) by creating a top-level interface that makes data-cube analysis much easier.

However, the overarching goal of ADMIT II is not particularly clear or cohesive: it seems like something of a grab bag, which tends to be somewhat biased in favor of the science performed by the authors and their research groups. Of course, most of these science cases will have wider reach, and they will reach out to the community for further input into desired future tasks, but again, the plan seems to be simply to make some nice new features on top of an already largely successful and completed software package. It is a safe bet, but isn't groundbreaking.

To put it another way: the authors spent just as long describing ADMIT I (which is fantastic work!!) as they did describing the extensions to the work that they're proposing for ADMIT II. This seemed a bit imbalanced.

3. Quality of the upgrade conceptual design;

A bit scattered: all of the concepts will be helpful to some or all users (e.g., grid view, uv-analysis, dendrograms, etc.), but the choice of which features to add did not seem particularly well motivated, and the delivery of the proposed extension to the project did not seem particularly cohesive.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

Excellent: the team will build upon work they have done with the NAASC and CASA developers.

5. Strength of the consortium organization (if applicable);

N/A

6. Qualifications of the key personnel of the Study;

Excellent: seasoned software developers who already did an excellent job with ADMIT I.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

Excellent: ADMIT I was a success, setting them up for success in the next round.

8. Assessment of the level of risk inherent in the design;

Low, as described in their proposal.

9. Strength of the Scientific Team supporting the Study;

Strong, but biased toward specific star-formation science, which may be biasing the types of tools they're planning to implement in ADMIT II.

10. Level of support guaranteed by the Institutes;

None.

11. Budgeted cost of the Study;

Reasonable: just FTEs paid for their coding work.

  • Referee5
Grade: 6

493 - ALMA Data Mining Toolkit II

1. Alignment with NA ALMA Partnership strategic goals;

This project would enrich the stored data of the ALMA science archive and is thus consistent with the strategic goals.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

Simpler archive access and easier analysis for non-experts are goals that will help ALMA to produce more scientific return. Improving the archive is the first recommendation in ALMA 2030. The “science case” portion of the proposal doesn’t elaborate much on the idea, its “science case” is instead a recounting of ADMIT progress and plans.

3. Quality of the upgrade conceptual design;

The proposed program will continue an existing path of development that is close to providing value added data analysis to the ALMA archive and PIs. The goals of this $0.9M proposal seem a bit underwhelming. At bottom, this is mostly fancy wrapping of existing CASA tasks, and though they propose to do this quite throughly and engage in education efforts to increase the user base, I don’t see why there are not more ambitious goals for 3 FTE-years of senior PhD time and a full time grad student.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

This is at the high-readiness end of the project spectrum

5. Strength of the consortium organization (if applicable);

N/A

6. Qualifications of the key personnel of the Study;

These are experienced programmers, though perhaps this would be better achieved by having fewer person-months of senior astronomers supervising professional programmers.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

The team has adequate expertise and experience, and is building on existing development work.

8. Assessment of the level of risk inherent in the design;

Low

9. Strength of the Scientific Team supporting the Study;

Adequate

10. Level of support guaranteed by the Institutes;

N/A

11. Budgeted cost of the Study;

As noted above, this cost seems high for the return. People are expensive, but return per dollar can be improved by lowering the number of dollars OR increasing the return, and either seem like viable options here.

  • Referee6
Grade: 4

ALMA Data Mining Toolkit II

1. Alignment with NA ALMA Partnership strategic goals;

Aligned with strategic goal to improve and extend technical capability (software). While not explicitly stated, it also supports archive use, an ALMA2030 goal.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The scientific case for this project lies primarily in the ease of use for astronomers. Given that this is an extension of a project, it would have been extremely helpful to have explicit evidence supporting the usefulness to the community of the previous effort. While there is no doubt that these are excellent software additions, it is important to know that they are actually being used and becoming the standard for users (beyond simply being part of the NAASC data distribution effort).

3. Quality of the upgrade conceptual design;

The plans for upgrade are reasonable and will be very helpful for users, assuming that the users are made familiar and adopt their use. The biggest weakness of the proposal is in determining whether these products are likely to be utilized as advertised, not whether they would be useful.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

The team has already produced product for ALMA and these additions appear ready as well (with the exception of the community desired product which is much harder to pin down).

5. Strength of the consortium organization (if applicable);

n/a

6. Qualifications of the key personnel of the Study;

Very capable. Have already produced product for ALMA. Would have been helpful to have an external review of the previous project available for consideration.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

The group are experts that know what they are working on.

8. Assessment of the level of risk inherent in the design;

It is troubling that the risk mitigation is all seen as de-scoping the project (thus no contingency is requested). This reviewer thus wonders if the project already has aspects that could be descaled or an implementation plan that could be slowed.

9. Strength of the Scientific Team supporting the Study;

Excellent but small. The proposal would have been significantly strengthened by showing that the scientific community was enamoured with and successfully using the ADMIT I software.

10. Level of support guaranteed by the Institutes;

None.

11. Budgeted cost of the Study;

The budgeted cost of the project is $898,590 and is primarily the labour cost of three individuals, plus a graduate student, working half time over two years. The cost is significant enough that it would be prudent for the NAASC to assess the community use and support of the ADMIT I software before funding the additional work.

  • Referee7

Grade: 4.0

Title: ALMA Data Mining Toolkit II

1. Alignment with NA ALMA Partnership strategic goals;

This is a proposal to build on prior work (ADMIT) to create an ALMA data browsing and analysis environment that enables users who are not experts in radio interferometry and systhesis imaging to make effective use of ALMA data. As such, it is very well aligned with the NA ALMA strategic goal to develop new software and computing technologies to imporve and extend capabilities for scientific research, as well as with the original concept of ALMA as an observatory accessible to non-specialists.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

This proposal would support the ALMA 2030 strategic goal of better usability and impact of the ALMA archive.

3. Quality of the upgrade conceptual design;

A good model for comparison is the many high-quality tool sets for data mining and visualization developed in the geophysics and climate research communities. These tools have been very effective in making the ever-growing archive of data in these disciplines accessible outside of the domain of expertise where the data were originally generated. The data browsing and tabbed workflow concepts in ADMIT echo approaches that have been used to good effect in geosciences.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

The ADMIT II effort would not be significantly paced by other ALMA development. Indeed, data processing and analysis tool development has lagged behind other ALMA developments and should be fast-tracked. The ADMIT work to date appears to be a high quality foundation for continued development. The proposers also correctly recognize that a large part of their task going forward to reach their target user base through outreach, workshops, and support.

5. Strength of the consortium organization (if applicable);

There are no collaborating institutions outside of UMD. For a software engineering effort such as this, this can be seen as a plus, in the sense that a tightly-connected group helps keep the conceptual integrity of the product under control.

6. Qualifications of the key personnel of the Study;

The project team is very well qualified, both through their experience with CARMA, and the deep software engineering experience of several team members.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

As noted in 6., the team has a successful track record in software systems development across multiple applications in radio astronomy.

8. Assessment of the level of risk inherent in the design;

Base on the work completed in ADMIT, the risk level is low going forward.

9. Strength of the Scientific Team supporting the Study;

This is a small team with solid scientific strength.

10. Level of support guaranteed by the Institutes;

No unusual administrative or infrastructure support is needed from the University.

11. Budgeted cost of the Study;

Roughly $800K for a total of about 2 FTE's appears to be a bit on the high side; it would have been helpful to understand the contribution of UMD overhead to this figure.

-- AlWootten - 2017-05-09
Topic revision: r2 - 2017-05-10, AlWootten
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback