Telescope Time Allocation Updates (Proposal)

Background

This proposal follows from discussions summarized at https://safe.nrao.edu/wiki/bin/view/Software/TTABrainStorm10Feb2017

Currently there are 8 science review panels (SRP), each consisting of 5 SRP members and 1 SRP chair. The SRP members anonymously review (score+text) all proposals in their panel for which they are not conflicted and the SRP chair only reviews proposals with one or more conflicts for which they are not conflicted. The individual scores are between 0.1-9.9, where a lower score is better. After the individual reviews are complete the scores are normalized to have a mean of 5 and a standard deviation of 2. An SRP telecon is held, including all SRP members and the chair, for each panel to form a consensus review (score+text). For each SRP, a normalized, linear rank score is produced between 0-10 (i.e., rank*10/n, where n is the number of proposals). The normalized, linear rank scores from all SRPs are then merged for consideration by the telescope allocation committee (TAC). Here telescope resources and scheduling are considered, along with the scientific and technical reviews, to assign a priority (A, B, C, etc.).

There are several issues with the current panel-based proposal review system.

  1. The number of proposals is not well balanced between science review panels (SRPs). For example, in 17B we have as few as ~20 proposals in NGA and as many as ~70 proposals in AGN. This is a function of VLA configuration (e.g., lots of AGN science in VLA A-conf.), changes in technology (e.g., ALMA was added to GMVA), or trends in science. The large loads are a real problem. We estimate that in a perfect world there would be a maximum of 30 proposals for any given SRP – given the current system. Here are plots of the total number of proposals as a function of semester for each SRP by counts and normalized by the total number of proposals.
  2. The SRPs are not well balanced by telescope. The SRPs were defined with the VLA in mind and do not match the science for the GBT or VLBA. Here are plots of the normalized number of proposals as a function of semester for each SRP by telescope: GBT, VLBA, VLA.
  3. The SRP categories are too broad (e.g., some reviewers feel they are not qualified to review some proposals). Some SRPs are a grab bag with loosely connected topics.

Proposal

Retain the 8 SRP science categories but allow the number of reviewers in an SRP to be flexible and do not require all (unconflicted) members to review every proposal. For example, we typically have N=5 SRP reviewers plus 1 SRP chair. Here will allow the possibility adding supplemental reviewers (r) so we have N+r SRP reviewers plus 1 SRP chair. We expect that for most SRPs r=0, but for those panels that have a trend of larger numbers (e.g., ETP) we will increase the number of reviewers. Provide software to randomly assign primary and secondary reviewers. Here the primary reviewer is responsible for crafting the review and will lead the discussion during the SRP telecon. Each secondary reviewer will produce a review (score + text) and participate in the SRP telecon discussion (unless they are a "reader" in alternative 2 below). If a reviewer is not selected as primary or secondary they are not expected to review the proposal but can comment during the SRP telecon. During the assignment of proposal conflicts reviewers will indicate proposals for which they think they are not qualified to be a primary reviewer. We should have at least four reviewers per proposal with the exception of Large proposals where we should have five. The goal should be to cap the number of proposals to review at 30-35 per reviewer. The SRP chair should be assigned less than 15 proposals to review. The primary/secondary status should be reflected on the My Reviews page and the Reviews Summary page. The SRP chair should have the ability to change the primary/secondary assignments (presumably on the Reviews Summary page).

The number of supplemental reviewers should be r ~ n*4/30 - (N+0.5), where n is the number of proposals and N is the usual number of SRP members (5). In other words we would consider adding supplemental reviewers when n exceeds 50. The current system requires each reviewer to review essentially all proposals, i.e. n reviews per reviewer (ignoring conflicts). Moving to 4 reviews per proposal without adding any supplemental reviewers reduces the number to n*4 / (5+0.5) ignoring conflicts and now counting the reviews by the Chair - assumed to be half the full load. Adding r supplemental reviewers reduces this to n*4 / (5+0.5+r). Here is a table with some examples:

N n n*4/30 - (N+0.5) r Reviews per reviewer
5 45 0.50 0 33
5 50 1.17 1 31
5 55 1.83 1 34
5 60 2.50 2 32
5 65 3.17 3 31
5 70 3.83 3 33
5 75 4.50 4 32
5 80 5.17 4 34

We suggest two alternative paths:

  1. All reviewers will participate in the SRP telecon.
  2. Supplemental reviewers will not participate in the SRP telecons. Therefore, supplemental reviewers cannot be primary reviewers on any proposals. This adds a little more complexity to the system but will help with scheduling the SRP telecons (since the number of participants will remain capped at 6). Also, it may help with recruitment since the load on supplemental reviewers is less than on full members. We could ask regular reviewers to become supplemental reviewers after their term is complete.

The goal is that the added flexibility in the process will partially mitigate the SRP imbalance by naturally allowing for more reviewers per SRP, ease the overall work of reviewers by reducing the number of proposals to review to be 30-35, assist the SRP chair by providing them software to manage the primary/secondary assignments, and potentially help our recruiters since the process will not be so demanding for the reviewers.

-- DanaBalser - 2017-07-28

This topic: Software > TtaUpdateProposalProcess
Topic revision: 2020-12-28, DanaBalser
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback