TTA Brainstorming

TTA Brainstorming: Meeting Minutes (10 February 2017)

Attending: DanaBalser, LewisBall, MarkClaussen, JayLockman

There are several issues with the current panel-based proposal review system.

  1. The number of proposals is not well balanced between science review panels (SRPs). For example, in 17B we have as few as ~20 proposals in NGA and as many as ~70 proposals in AGN. This is a function of VLA configuration (e.g., lots of AGN science in VLA A-conf.), changes in technology (e.g., ALMA was added to GMVA), or trends in science. The large loads are a real problem. We estimate that in a perfect world there would be a maximum of 30 proposals.
  2. The SRPs are not well balanced by telescope. The SRPs were defined with the VLA in mind and do not match the science for the GBT or VLBA.
  3. The SRP categories are too broad (e.g., some reviewers feel they are not qualified to review some proposals). Some SRPs are a grab bag with loosely connected topics. (But there are not too many complaints.)

We had a fairly open discussion jumping from one topic to the next. Here are some of the main discussion points:

  • NRAO 17B Issues. For context we discussed the problems with 17B. Both the AGN and ETP SRPs have 70+ proposals. Just too many to review. Our solution was to add three members to each SRP and only demand that each proposal have 4 reviewers or more. This requires the chair to perform more tasks and the SRP telecons will be harder to schedule and manage.
  • ALMA. The ALMA system was briefly discussed. There is an initial triage where each proposal only has 4 reviews. The bottom (< 50%) are dropped with no further consideration unless the dispersion was high. There are then multiple panels per science category to handle the remaining proposals.
  • SRP Telecons. Could we remove the SRP telecons? Do they add much value? Advantages: reduce some of the work and allow us to more broadly recruit reviewers. Disadvantages: how do we draft the consensus reviews? We could use the raw reviews but then this would require more work since they need to be edited.
  • Reviews by Telescope. We currently rank proposals including all telescopes. That is, the three telescopes compete based on the science, but they are mostly scheduled independently. There are some advantages to having a common TAC since there are synergies between the three telescopes (e.g. HSA, joint proposals), but do we get the best science on each telescope. This may be more important now with limited telescope time for the GBT and VLBA.
  • Outside Reviewers. To help lighten the load we could recruit people from outside the review process. For example, if there are too many proposals in one SRP we could recruit someone from outside to help. We would have to be careful how to incorporate their input since any scores provided would be not normalized. This could be handled by having a pool of people from the outside with a history. For example, there could be a pool of "readers" that are on call.
  • Normalization between SRPs. We normalize scores within each SRP but not between them. For example, what happens if the science is significantly better in one SRP compared to another. The scores are just merged between SRPs. This does come up occasionally in the TAC meeting but it is not formalized. We might ask the TAC to note any proposals with lower scores that are pretty good science.

Action Items:

  • DanaBalser will generate some statistics on the number of proposals per SRP, telescope, etc. over time.
  • JayLockman will determine GBT appropriate SRPs to compare with our current SRPs.


Straw man

Background

Currently there are 8 science review panels (SRP), each consisting of 5 SRP members and 1 SRP chair. The SRP members anonymously review (score+text) all proposals in their panel for which they are not conflicted and the SRP chair only reviews proposals with one or more conflicts for which they are not conflicted. The individual scores are between 0.1-9.9, where a lower score is better. After the individual reviews are complete the scores are normalized to have a mean of 5 and a standard deviation of 2. An SRP telecon is held, including all SRP members and the chair, for each panel to form a consensus review (score+text). For each SRP, a normalized, linear rank score is produced between 0-10 (i.e., rank*10/N, where N is the number of proposals). The normalized, linear rank scores from all SRPs are then merged for consideration by the telescope allocation committee (TAC). Here telescope resources and scheduling are considered, along with the scientific and technical reviews, to assign a priority (A, B, C, etc.).

There are several issues with the current panel-based proposal review system.

  1. The number of proposals is not well balanced between science review panels (SRPs). For example, in 17B we have as few as ~20 proposals in NGA and as many as ~70 proposals in AGN. This is a function of VLA configuration (e.g., lots of AGN science in VLA A-conf.), changes in technology (e.g., ALMA was added to GMVA), or trends in science. The large loads are a real problem. We estimate that in a perfect world there would be a maximum of 30 proposals. Here are plots of the total number of proposals as a function of semester for each SRP by counts and normalized by the total number of proposals.
  2. The SRPs are not well balanced by telescope. The SRPs were defined with the VLA in mind and do not match the science for the GBT or VLBA. Here are plots of the normalized number of proposals as a function of semester for each SRP by telescope: GBT, VLBA, VLA.
  3. The SRP categories are too broad (e.g., some reviewers feel they are not qualified to review some proposals). Some SRPs are a grab bag with loosely connected topics. (But there are not too many complaints.)

Proposed Changes

  • Add one or more "SRP readers" to each panel. An SRP reader will perform an individual review (score+text) but not attend the SRP telecon. The SRP readers will only be "activated" if necessary. The goal is to cap the number of proposals to review at 35 per reviewer. The number of readers r > n*4/35 - 5.5, where n is the number of proposals. This assumes the SRP chair will do half the number of reviews.
  • The SRP chair will be tasked with assigning reviewers to proposals where there are at least 4 reviewers per proposal. (If possible, Large proposals should have more reviewers.) The SRP chair should try to balance the load between all reviewers if possible and assign themselves a smaller number to review. Each assignment will be designated as either primary or secondary. Both primary and secondary reviewers will perform an individual review (score+text). After the reviews are complete and the scores normalized as above, the primary reviewers will be expected to enter the consensus reviews and to lead the discussion during the telecon. Secondary reviewers will just review the proposal and comment during the telecon (if present). By definition, SRP readers will only be assigned as secondary reviewers since they will not attend the telecon. Each proposal should have three or more reviews from SRP members who will attend the telecon. Unless deemed critical there should be no further contact with the SRP reader.
  • For SRPs that have consistently had an overload of proposals create two SRPs. We require at least ~20 proposals for the statistics to be reasonable so this demands that an SPR never fall below ~40 proposals over past semesters. Inspection of the total proposal counts as a function of semester for each SRP reveals that the ETP and AGN SRPs consistently have large loads and could be split into two panels. The AGN SRP varies with VLA configuration and has dipped below 40 proposals for one semester.

PST Impact

  • Currently the SRP can only submit a review for proposals that have one or more conflicts. We need to disable this flag at a minimum.
  • Currently the proposals to review are labeled in the PST. This will now be incorrect. It would be nice to allow the SRP chair to import the reviewers into the PST and have that reflected on the My Reviews page.
  • It would be useful to indicate the primary and secondary reviewers in the PST.
  • There is no distinction between a SRP member and a SRP reader. This is probably okay.
  • Currently an SPR reader's score would be normalized using all of their proposals in the current semester. If the SRP reader only reviewed a couple of proposals this would not be valid. We would want to normalized based on past semesters.
  • We have never modified the the science categories so if we create additional SPRs we need to test this carefully.

Comments

These changes should improve (1) and (3) above. With the addition of readers we should be able to cap the number of proposals to review. In some cases we may only want the reader to review one or two proposals. For example, there may be a Large proposal that has a lot of conflicts and thus we need additional reviewers. For this reason it would be prudent to select readers who have been SRP members in the past so we can properly normalize their score. Why not just add additional SRP members instead of defining a new reviewer class (the readers)? If there are too many SRP members it may become a problem scheduling the SRP telecons which are already difficult to schedule. The SRP telecons become harder to manage with too many people. The discussions become longer and the various conflicts harder to manage. For SRPs that have consistently had 50+ proposals per semester, we can just create two SRPs to reduce the load.

It is more difficult to mitigate (2) under the current scheme. One approach would be to perform the normalized linear rank across telescopes instead of SRPs.

-- DanaBalser - 2017-02-10
Topic attachments
I Attachment Action Size Date Who Comment
gbt.epseps gbt.eps manage 43 K 2017-02-17 - 15:12 DanaBalser GBT Proposal Distribution
tot.epseps tot.eps manage 41 K 2017-02-17 - 15:13 DanaBalser Total Proposal Distribution
tot_number.epseps tot_number.eps manage 40 K 2017-03-30 - 22:45 DanaBalser Total Proposal Distribution (Counts)
vla.epseps vla.eps manage 41 K 2017-02-17 - 15:13 DanaBalser VLA Proposal Distribution
vlba.epseps vlba.eps manage 41 K 2017-02-17 - 15:12 DanaBalser VLBA Proposal Distribution
Topic revision: r5 - 2017-03-30, DanaBalser
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback