Proposal Evaluation and Time Allocation Test Plan

Background Information

Requirements Document

Requirements Document V2.0

Priority

  • Most of the heavy lifting in category B is requirements 3.*. All of these must be implemented by Dec 15 with the exception of 3.13.5-7, which are category C. The 3.* requirements can be grouped in a natural way and prioritized:
    • 3.1-3.3: SRP membership
    • 3.4-3.6: Assign proposals to scientific categories
    • 3.12: Identify and resolve conflicts of interest
    • 3.7-3.10: Science review management
    • 3.13: Proposal scoring and completion
    • 3.11: Related proposals

Release Dates for New Proposal Evaluation Process

Category Test Version Release Date Production Version Release Date Comments
A 15 Nov 2010 07 Jan 2011 Call for proposals (1 Feb 2011 deadline).
B 15 Dec 2010 15 Jan 2011 Technical and scientific review of proposals.
C 15 Feb 2011 25 Mar 2011 SRP review meetings (~08 Apr 2011).
D 15 Mar 2011 25 Apr 2011 TAC meeting (~09 May 2011).
E Beyond 01 May 2011 TBD Anything beyond the above.

Current PST Roles

Role Description
Admin System admin, has access to everything in the system.
Anonymous Used by Registration, and Login page, so anybody who hasn't logged-in has access to these pages.
CASA-Admin CASA admins had permissions to edit CASA help desk tickets that were managed by PST; no longer used.
CASA-User CASA users with access to the CASA tab; no longer used.
CASA-USS CASA users who had access to the CASA help desk tickets but couldn't change them; no longer used.
PST-Group Users who can make changes to PST Groups, mostly Stephan and his team.
PST-Admin Users, Lori, with permissions to additional PST functions like Approving proposals.
PST-Assistance Created for Lori's assistant, who has access to All Proposal List, but can't do much beyond printing All proposals.
PST-Referee-Admin Users who can make changes to Review Categories, and add/delete referees. This will change with the new system.
User PST users.

Test Plan

Summary

Simulate the reviewing process with the test server (mirror.nrao.edu). The test server is completely isolated from the production database and any change to it does not feed to the production database. For testing purposes we can setup a different email list that will send emails to only a select few. Dummy accounts can be created with the proper permission and used by the core testing group to enter information. NRAO staff outside the core testing group and external staff should also test the new system but they can probably use their current accounts. The last cycle (11A) proposals can be used, although this will require some modifications (e.g., translate science categories). We will probably need to add a few dummy proposals to test the "Modified Sessions". Also, it would be useful to have some of the dummy accounts on some of the proposals (e.g., test conflict of interest).

Dummy Accounts

  • Technical Review: the technical reviewers are typically NRAO staff and will be assigned based on the telescope and science category. For testing purposes let's create two accounts per telescope with the following names:
    • tr1EVLA, tr2EVLA, tr1GBT, tr2GBT, tr1VLBA, tr2VLBA
  • Scientific Review (SRP): there will be 5 reviewers plus one chair per scientific category. For testing purposes let's create two accounts per panel plus one chair with the following names:
    • sr1P1, sr2P1, sr1P2, sr2P2, sr1P3, sr2P3, sr1P4, sr2P4, sr1P5, sr2P5, sr1P6, sr2P6, sr1P7, sr2P7, sr1P8, sr2P8
    • srCP1, srCP2, srCP3, srCP4TC, srCP5, srCP6, srCP7, srCP8
  • TAC Review: most of the users for the TAC are already defined since each SRP chair is automatically on the TAC. One of these will be defined as the TAC chair (srCP4TC). Other users will need to be added (e.g., NRAO staff). So let's create two accounts for this purpose.
    • tac1, tac2

Tests for Category A

This is fairly straight forward (e.g., check that the proposal types are correct, make sure the correct science categories are in place, etc.).

Tests for Category B

  1. Check that the various dummy accounts have been created with the correct permissions.
  2. Technical Review
    • Assign technical reviewers for each telescope/science category.
      • Check that only valid users can make these assignments.
      • Check that the reviewer can download the proposal.
    • Input technical reviews.
      • Check that the reviewer cannot input a score.
      • Check that the reviewer can save text, exit, and return.
      • Check that the different text inputs can be viewed by the appropriate users. One that all can see (authors plus TAC); and the other that only the TAC can see.
  3. Scientific Review
    • Assign scientific reviewers for each science category.
      • Check that only valid users can make these assignments.
      • Check that the reviewer can download the proposal (including related proposals).
      • Check that a history of a reviewers tenure is stored in the database.
      • Check that the scientific reviewer cannot see the technical review.
    • Automatically assign proposals to scientific reviewers.
      • Check that the proposals have been assigned correctly.
      • Check that each proposal is assigned to only one panel.
    • Manually reassign several proposals.
      • Check that this indeed has been done correctly.
    • Begin review process with the conflict of interest. Make sure there are SRP members on some proposals in different ways (authors, same institution). Also, for at least one case the SRP panel member and chair should have a conflict.
      • Check that conflicts are handled correctly.
    • Input scientific reviews.
      • Check that the reviewer can save text, exit, and return.
      • Check that a review cannot be submitted unless the reveiwer has entered text, a score, and a percentage.
      • Check that after a review is submitted that it cannot be edited.
      • Check that no user except the reviewer can see the review until all reviews have been submitted.
    • Submit all scientific reviews.
      • Check that each SRP member can view comments, scores, and percentages for their panel, and that TAC members can view all reviews.
  4. Duplicate Sessions
    • Check that the "Modified Sessions" has been created.

Tests for Category C

    • Perform the proposal scoring.
      • Check that this cannot be done unless all reviews have been submitted.
      • Check that the algorithm is correct.

Tests for Category D

-- DanaBalser - 2010-11-10
Topic revision: r4 - 2010-11-12, DanaBalser
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback