Telescope Time Allocation Updates (Strawman)


Currently there are 8 science review panels (SRP), each consisting of 5 SRP members and 1 SRP chair. The SRP members anonymously review (score+text) all proposals in their panel for which they are not conflicted and the SRP chair only reviews proposals with one or more conflicts for which they are not conflicted. The individual scores are between 0.1-9.9, where a lower score is better. After the individual reviews are complete the scores are normalized to have a mean of 5 and a standard deviation of 2. An SRP telecon is held, including all SRP members and the chair, for each panel to form a consensus review (score+text). For each SRP, a normalized, linear rank score is produced between 0-10 (i.e., rank*10/N, where N is the number of proposals). The normalized, linear rank scores from all SRPs are then merged for consideration by the telescope allocation committee (TAC). Here telescope resources and scheduling are considered, along with the scientific and technical reviews, to assign a priority (A, B, C, etc.).

There are several issues with the current panel-based proposal review system.

  1. The number of proposals is not well balanced between science review panels (SRPs). For example, in 17B we have as few as ~20 proposals in NGA and as many as ~70 proposals in AGN. This is a function of VLA configuration (e.g., lots of AGN science in VLA A-conf.), changes in technology (e.g., ALMA was added to GMVA), or trends in science. The large loads are a real problem. We estimate that in a perfect world there would be a maximum of 30 proposals. Here are plots of the total number of proposals as a function of semester for each SRP by counts and normalized by the total number of proposals.
  2. The SRPs are not well balanced by telescope. The SRPs were defined with the VLA in mind and do not match the science for the GBT or VLBA. Here are plots of the normalized number of proposals as a function of semester for each SRP by telescope: GBT, VLBA, VLA.
  3. The SRP categories are too broad (e.g., some reviewers feel they are not qualified to review some proposals). Some SRPs are a grab bag with loosely connected topics.

Suggested Updates

Restructure SRPs

Restructure the SRPs to create a better balance. For example: (1) Split ETP into (i) pulsars, XRBs, and other Galactic; and (ii) SNe, GRBs, FRBs, TDEs, and misc. (2) Split AGN into (i) AGN properties and inner jets; and (ii) Lobes, feedback, etc. (3) Combine NGA and EGS. (4) Move SSP proposals to other panels. Stellar non-thermal --> ETP; Novae --> ISM; YSO's --> SFM. Not clear what to do with planets.

These changes may better balance the SRP loads now but how long would that last? The AGN count has varied quite a bit (factor of 2) and depends on VLA configuration. EGS also depends on VLA configuration but not as much as before (HI in B-config). Furthermore we would like to lighten the load in general but this could be part of the solution. One side effect is that changing the SRPs causes problems in consistency when generating metrics.

Add SRP Readers

Add one or more "SRP readers" to each panel. An SRP reader will perform an individual review (score+text) but not attend the SRP telecon. The SRP readers will only be "activated" if necessary. The goal is to cap the number of proposals to review at 35 per reviewer. The number of readers r > n*4/35 - 5.5, where n is the number of proposals. This assumes the SRP chair will do half the number of reviews.

Each proposal will be signed at least 4 reviewers that are designated as either primary or secondary. Both primary and secondary reviewers will perform an individual review (score+text). After the reviews are complete and the scores normalized as above, the primary reviewers will be expected to enter the consensus reviews and to lead the discussion during the telecon. Secondary reviewers will just review the proposal and comment during the telecon (if present). By definition, SRP readers will only be assigned as secondary reviewers since they will not attend the telecon. Each proposal should have three or more reviews from SRP members who will attend the telecon. Unless deemed critical there should be no further contact with the SRP reader.

Reviewers will be assigned to proposals automatically to lighten the load of the chair. During the declaration of conflicts a review will declare their conflicts and select any proposal which they feel unqualified to review. The primary and secondary assignments will then be generated randomly.

With the addition of readers we should be able to cap the number of proposals to review. In some cases we may only want the reader to review one or two proposals. For example, there may be a Large proposal that has a lot of conflicts and thus we need additional reviewers. For this reason it would be prudent to select readers who have been SRP members in the past so we can properly normalize their score.

-- DanaBalser - 2017-04-17
Topic revision: r2 - 2017-07-28, DanaBalser
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback