PST Feedback (June 2007)

Summary of User's Committee Comments (via Nicole Radziwill)

  • Unclear when to validate and when to save.
  • Should be able to add multiple bands for one source (or multiple sources for one band).
  • When filling in forms that require refreshing the browser screen, the information you already typed in (last source, time, band, rms, etc) should persist so you can edit it instead of starting your new entry from scratch.
  • Sources should be sorted by name.
  • PI list cuts off printing at 8, should be at least 10 and continue to a 2nd page.
  • If you try to exit or go to another tab without having validated or saved (which would cause lost data), warn the user and ask if they are sure.
  • Entering 60-80 source/band combinations noted as an irritation.
  • Others have also mentioned irritation of adding 20-30+ co-proposers, and manual entry of institutions needs to be replaced.

Summary from Joan Wrobel

At the 2007 June 1 deadline, a total of 196 proposals were received
via the PST.  These were split as 114 VLA and 82 GBT.  Feedback was
requested from 152 proposal editors, i.e., the individuals who
initiated the GBT and/or VLA proposals.  A 10-day deadline was
advertised for feedback responses.  20 responses were received.

The table below gives a summary of the types of trouble reported.

 ==================================================================
  Number of            Trouble Bin  Trouble Bin
  Mentions                 Keyword  Description
 ------------------------------------------------------------------
  6      [source/resource/session]  confusion over concepts
  4                     [web comm]  reloads and timeout annoying
  3                     [overload]  overload near deadline
  3                       [userdb]  userdb itself or related to PST
  2          [many pairs/sessions]  entry & checking onerous
  2                [save/validate]  confusion over differences
  2                   [pdf upload]  upload failed
  2                         [docs]  improve docs
  1              [co-editing lock]  lock lingers after edits done
  1                 [many authors]  entry onerous
  1                     [abstract]  save did not save changes
  1                 [many authors]  entry onerous
  1                    [rest freq]  red-shift rest frequency
 ------------------------------------------------------------------

The text of the 20 responses appears below.  Appropriate trouble
keywords have been added to the responses.  Responder 8 sent his
feedback to all proposal editors, and his feedback is cited by some
later responders.  Seven responders, flagged by a +, also suggested
miscellaneous enhancements.

Cheers.  - Joan

---------------------------------------------------------------------

[userdb] +

As requested, I send my feedback about NRAO PST. Generally it is very
well designed tool, and I have only minor comments.

I had quite a big problem with affiliations of my co-investigators. If
the institute/university is not listed in the NROA database the
procedure how to handle it seems obscured to me. I managed to fill in
the affiliation entry myself and it gave the correct output in the
proposal, but when I wanted to correct it, I couldn't find a way to do
it. The "update" button gave "Affiliation not listed" entry in the
proposal. The co-investigator himself could neither change it, because
in the user profile he still has a blank entry as an affiliation and
the note "Please contact me to add my Institution" (there is actually
the same situation in my profile). It would be nice if the affiliation
could be inserted even if it is not registered in the NRAO database.

The second issue is less important. PST applies an automatic and
alphabetical ordering of the proposed sources. In my opinion sometimes
it would be better to use user-defined ordering for example rather
than GRB 050824, GRB 060218, GRB 991208 I would prefer GRB 991208, GRB
050824, GRB 060218.

I hope it could help to improve the PST.

------------------------------------------------------ End Response 1

[rest freq]

Bryan Butler and I had several iterations of email back and forth
during the submission process, but one item I forgot to metion is the
rest frequency field: if the rest freq of a line is outside of the
feed, the tool produces an error, even if the redshift of the
target(s) put the line into the proper band.  I also get an error if I
simply list transitions, even though this is listed as an option in
the documentation.

Overall, however, the PST is certainly continuing to improve and is
excellent at facilitating collaborative proposal preparation.

------------------------------------------------------ End Response 2

[docs] +

Per request, I have some feedback to offer on the PST.
 
My cases are probably a little strange, because I observe solar system
objects for which the positions and velocities change, sometimes
rapidly.  This makes the source entry part a bit more of a challenge,
because I have to enter different "sources" as the object moves across
the sky in time.  This is not a complaint about the system, just
information to guide the fact that I am a slightly non-standard user
of the PST.
 
For the time request, the PST forces entry of a particular number of
hours for each source on the list.  Sometimes I would like to make a
request for adjacent calendar days (time-variable sources) or for a
certain number of tracks in several sets at a spacing of some number
of weeks or months.  There wasn't an easy way to do this -- I can
certainly describe the request in the comments boxes, but how it is
entered affects how the PST calculates the totals that are seen by the
referees.  If there were, for example, a third level in addition to
"repeat" and "separation" then I could enter repeat 2, separated by
one day, and have that entire block repeated N times with a wider
separation.
 
I was confused by the box in which you enter a name for the resource.
Finally I realized this was my name, for bookkeeping purposes, and not
the "official" name of the instrument.  Having a default for the
resource name, or a drop-down menu of receiver+backend combinations
might be helpful.  I was similarly confused by "Group."
 
"Minimum" and "Maximum" LST are a little fuzzy when the source is up
over 0 hours LST... I wasn't sure if I should switch them around, but
ultimately I didn't.
 
As a non-pulsar user, it seemed strange to me to enter my time
resolution in thousands of milliseconds.  Would it be possible to have
a non-pulsar default for those of us intending to integrate 10 seconds
or longer on each source?
 
Under scheduling constraints, it might be helpful, both to proposers
and to referees, to have a check-box if the user is attempting to
coordinate simultaneous operations with the NRAO facility and another
observatory outside NRAO, at the same (Arecibo, Goldstone) or
different frequencies (HST, IR, millimeter...).
 
On the positive side, I really like the way it stores the observer
information and allows copying of past proposals.  This is a major
time-saving feature, and much appreciated.  I like the setup with
source/resource pairs, because it facilitates using different
instruments or resources on the same source.

------------------------------------------------------ End Response 3

[docs] [web comm] +

I have to say it's a pretty clunky product, the interface is poorly
laid out and the screen dialogues are confusing. The 30 minute
time-out is really irritating. Why don't you write data to a pending
area each time a user submits a page? There is no place to enter
(GBT) target rms, which should then calculate integration times.

------------------------------------------------------ End Response 4

The PST works very well. It is great that all previous proposals are
also available in the PST - this helps a lot when new but similar
proposals have to be submitted.

------------------------------------------------------ End Response 5

[web comm] [pdf upload] [many pairs/sessions] +

I have used the web form twice -- for 01FEB07 and now for 01JUN07.

I include below my comments from my 01FEB2007 experience. My recent
01JUN2007 was better because I already had my very extensive source
lists entered and did not have to enter them again.

The main thing I noticed this time (01JUN) was the requirement that
the web form communicate with the NRAO server VERY frequently. Thus
was not only a nuisance when the web server went down for a short
while, but I really pity someone sitting in a hotel room while on
travel and having only a 56KB dial-up modem.  That must be really
painful!

It would seem better if the program would collect changes and only
communicate with the NRAO server when one hit a SUBMIT or SAVE button
or something like that.  That said, the program should also not allow
anyone to take any action which would result in a loss of entered data
without requiring, or at least asking about, saving.

On reading through my points below from 01FEB2007, they still seem
valid, so that the program doesn't seem to have changed much in the
intervening 4 months.

I hope my comments are helpful.

***********************

COMMENTS FROM 01FEB2007

- When filling in forms that require sending the info to NRAO and
getting a refresh, the program should remember what the last source,
time, band, rms was and show it as the initial guess for the new data
line.

- Should be able to add multiple bands for 1 source (or multiple
sources for 1 band)

- It wasn't clear (at least to me) when I needed to "Validate" and/or
"Save" or both.

- If try to exit (or go to the next tab) without having validated
(and/or saved), which would result in a loss of data, be sure to warn
and ask if the author is sure

- Sort sources by name

- At least for me, if I uploaded the PDF of the technical
justification early in the writing process, it was lost or corrupted
before the end of the process.  After the initial filling in of all of
the tabs, however, a new upload seemed to stay around fine.

- PI list cuts off printing at 8 -- too few.  Should be at least 10
and continue to a 2nd page

- Observation list cuts off printing at 30 -- needs to go on to
multiple pages

That was it. In general, except for somewhat of a learning curve (and
the hassle of entering our 60-80 source/band combinations individually
for short observations), it seemed to work fine.

------------------------------------------------------ End Response 6

[many pairs/sessions] [many authors] +

Given my recent experience with the NRAO proposal submission tool I
thought I'd send you some feedback to help you improve the system.
Generally I find that the PST is one of the better submission tools
around, and it seems to me that it would work well for most users who
are looking at single pointings or small samples.  But there are some
minor things that when combined caused major problems for us in
submission because we has such a large source list (99 objects), which
subsequently led to the need to resubmit the proposal after the
deadline, of which you are already aware.  Several of these points
were raised in feedback from my previous proposal submission (late
last year), but they were much more critical during this submission.

Much of the material for the proposal was easily uploaded or entered,
including the science case, abstract, and source list.  The problems
we experienced last time with some of these were avoided (the abstract
was well under the limit, and the source coordinates used min & sec/'
& " rather than decimal) so I'm not sure if these were corrected.  The
addition of large numbers of CoIs still seems to be problematic,
although since most of our CoIs were on the last proposal we were able
to circumvent most of the trouble we had last time.

What caused us the most trouble was setting up the observing sessions
since I needed to do 99 of them.  Last time this wasn't as bad since
the 92 sources we submitted then were in 16 groups, but this time
there was one session per source.  Although the details for each
session was virtually identical (the only difference was that for some
sources we needed CnB observations rather than C array) there was no
easy way to set this up and each source had to be done individually.
With multiple page reloads for each session this took a very long
time.

I think the best way to solve this problem is to do the same thing you
do for uploading large source lists, allow the users to set up a text
file with a specific format that can be uploaded to fill in the
details.  The upload file could be set up in with the following
details in each line:

Session name, various session details (repeat, separation, LSTs,
etc.), number of sources/groups, First source/group, time, RMS,
resource, ...  (repeat for total number of sources/groups in this
session)

Such a file could be set up in five minutes from the sources list for
a large proposal like ours (and small proposals could still use the
old method).  The only things that could cause trouble would be if the
source or resource names did not match any names that had already been
entered.  Implementing it this way would be an enormous help for large
surveys (even if they don't need the number of sessions we did) and
save a lot of time.

Another problem I had was the inability to easily check the sessions
list for mistakes after entering all 99 sessions.  The list of session
details has very little useful information in it, and to see the
details each has to be reloaded one at a time.  With our sample size
this made it virtually impossible to check that all the entries were
correct.  The option of a more detailed view of all sessions at the
same time would be of great help.

One other minor thing you may want to look into is when all the Co-Is
receive emails for submission/withdrawal.  When you withdrew the
proposal after the deadline the system automatically sent an email to
all our collaborators saying the proposal was withdrawn.  I promptly
started to get slightly panicked emails back from them about it until
I had a chance to tell everyone what was happening.  A better way
might be to have an option come up when it confirms you want to submit
saying to uncheck a box if you don't want everyone to get an email.
Or only one email gets sent out to everyone in the team once the
deadline is passed and the proposal is in the "submitted" state, only
the PI getting the confirmation ones.

I hope these comments are helpful in improving the proposal tool and
you are able to implement my suggestions.

------------------------------------------------------ End Response 7

[abstract] [save/validate] [source/resource/session] [web comm]

Once again I was forced to use the dreaded NRAO PST. Honestly, I
understand all the effort that must have gone into it, but from the
many Web-based forms/tools I have used, the PST ranks definitely
bottom.

I don't think the current design reflects in any sense positively on
the NRAO, or VLA for that matter. Also, it scares off potential radio
users rather facilitating access to this wavelength regime. Neither of
this can be in NRAO's interest.

I have sent feedback on previous occasions but haven't seen any
improvement nor change. Perhaps you could inform the users what s
being done with the feedback (am I alone in my opinion? ARe there
changes planned?).

In what follows I give some detailed feedback on my most recent
experience:

A major gripe: The day before submission I revised the Abstract. I
"saved" the Title page (several times as a few changes were made in
succession and to ensure that it really was all saved), looked at the
result with View, found it okay, and concentrated from that moment on
the remaining. Imagine my surprise to find AFTER submission that the
old Abstract showed up...arrghhhhh

Now, should I have "validated", perhaps? What is the subtle difference
between "save" and "validate" in the opinion of the PST? When I click
"save" and want it to do what it says on the tin...! Honest...I
clicked on "save"...I swear!

Then, I'm quite often contacted by colleagues (radio and non-radio
alike) asking me if I understand the "sources/resources/sessions"
section...people are baffled. This is totally non-intuitive and you
need to be experienced (in PST use and VLA use) in order not to make a
mess of it.

Finally, it is EXTREMELY ANNOYING that with every bleeping click (on
the Resources page, for example), the entire page needs a reload. This
slows down the editing enormously.

I fervently hope that you will redesign this interface...it needs it!

------------------------------------------------------ End Response 8
  
For the record I consider myself extremely cranky when it comes to
user interfaces, I think NRAO has an absolutely dismal historical
record there, but I found the pstool quite transparent and easy to
use.  There are a couple of small enhancements but my experience was
overwhelmingly positive and my IRAM collaborator who also used it had
no complaints whatsover.

------------------------------------------------------ End Response 9

[save/validate]

Just read (responder 8's) comments on the PST, and unfortunately, I
have to agree with much of his criticisms. This business of "save"
versus "validate" was confusing to me too. I also just put in a
proposal to use the Australia Telescope Compact Array run by the ATNF
which also uses an on-line submission tool called "Opal". Opal was
simpler to use than PST and I recommend that something like it be
adapted by NRAO. I thank you for soliciting user feedback and am very
glad that NRAO is listening to observers.

----------------------------------------------------- End Response 10

In my experience all worked very smoothly. Very easy to use it.

----------------------------------------------------- End Response 11

[web comm] [source/resource/session]

I agree with much of what was expressed in (responder 8's) email, even
if I don't necessarily feel quite as strongly about it. I liked that
the PST had old proposals stored, so I could start by copying and not
have to enter everything from scratch--but I do think that having to
store the proposal so frequently is a little annoying, and that the
sources/resources/sessions trifecta is hard to understand.

I appreciate you requesting feedback--I don't think NOAO has ever
asked for any, and I think their system is far worse.

---------------------------------------------------- End Response 12

[overload] +

My experience with using the PST has been positive overall (this is
for the GBT).   In addition to providing a good way to communicate the
proposals from the proposers to NRAO to the referees and back, I like
the interface in general.  The ability to add authors from a full
proposer database was nice as was the opportunity to see all proposals
that you're working on for a given deadline.  The ability to enter
objects from NED/SIMBAD or your own lists was well-implemented.

Some areas for improvement:

1) On deadline day, there were many problems in accessing the database
in the morning.  These seem to have resolved later in the day, but it
was ... disconcerting... to be locked out on the day the proposals
were due.

2) The "Sessions" portion of the proposal seem like it needs some
work.  Submitted changes to sessions didn't register and the
"Comments" section would only update occasionally (it usually took 2-3
times of entering comments before they would register).  In addition,
sessions wouldn't immediately appear after being added resulting
having two sets appear once I tried to add one.  These seem like bugs
and they were frustratingly non-reproducible so you might want to take
a look at the logic for this tab.  In general (i.e. in all tabs),
there were a few times when things were input but not saved and
changes were not reflected throughout the proposal.  

3) The sessions area seems like an excellent place to include
estimates of observing overheads and more flexibility at communicating
overheads would make the time estimate in the sessions and the
proposal section correspond better.  For example, if one of my
proposed projects is schedule in 20 separate blocks, the overheads
will be larger than if several blocks are scheduled concurrently.  As
such, the time estimate (which reviewers will key on) will be
different.

I hope this helps,

----------------------------------------------------- End Response 13

[userdb]

I'm quite happy with the PST. The only problem I found was if I made a
mistake when putting in a collaborator - there's no way to edit this
after an email address has been entered. The collaborator's details
have to be put in from scratch, but the same email can't be used since
this is already in the system. I can understand why this is - you
don't want people mucking up your users data base. A solution may be
to allow someone who has just entered a new user to be able, to edit
this during the same session, as I'm sure most people realise straight
away that they've made a mistake. Maybe a colling off period, after
which a users details can't be editted?

----------------------------------------------------- End Response 14

[source/resource/session] [pdf upload]

I have used the PST only once, and that was just for the last deadline
for the VLA (Proposal Code: VLA/07C-118, ID: AW717). I have to say
that generally I am not a friend of these online tools. In my opinion
the standard way of latex templates submitted by email is much more
convenient for the user. However, I see the possibility that online
tools may be easier to handle on your side. I hope that this really is
the case, because otherwise I see no reason for these tools.

My experience with the PST was mixed. Generally I was able to find my
way through it, although it took a while to understand the concept of
resources/sessions etc. (It is explained well, it just takes some
time.)  Most things worked as expected, but still it took much more
time for me than simply filling in the latex form. I had one real
problem with submitting the science case for the proposal as a PDF
file. I was not able to do this with my firefox but had to use an old
version of mozilla to be successful. This is an important point,
because the science case is the core of a proposal and takes most of
the preparation time. If I had noticed the problem only a few hours
before the deadline (and we all know that it is quite common to submit
proposal very late), I would have been in trouble. I found the
workaround (using mozilla) only by chance.

I hope that these comments are helpful. As said, I prefer the latex
template (even though this would need improvement as well), but in the
end it does not make an important difference in which way the
proposals are submitted, as long as everything works.

----------------------------------------------------- End Response 15

[overload]

I am sorry, but I had severe problems with the PST. I had to submit my
proposal rather late (3h before the deadline). At that time the PST
was nearly unusable: In some circumstances it deleted random observing
sessions.  Roughly one hour before I actually submitted I had the
proposal nearly finished and wanted to polish it. However, when I
started to reword one session it deleted a session or the total
session times got mixed up (too high observing hours etc). Thus, I
lost my session setup and spend the next hour repeatedly trying to set
the sessions up again. When I finally succeeded to get it back to a
roughly working version I simply submitted as I did not want to risk
losing the full proposal.

Besides that I think that it is a very good system - it just crashes
under high load.

----------------------------------------------------- End Response 16

[overload] [source/resource/session]

I generally like the PST.  Overall it is fairly simple to use.  I have
two complaints though.  First, there were a lot of problems with the
system going down this year early on the deadline day.  This just
can't happen.  It causes users (and the support scientists) undue
stress and can really muck up the proposal process.  Second, I have
been using some version of the PST for years now and I still don't
find the Sessions/Resources sections sensible at all.  As I see it,
and as every telescope proposal form used to work, I have a source I
want to observe with a given receiver for a given integration time and
I may want to repeat that observation with some frequency.  This makes
sense to me, and is how we observe things.  I confess that I agree
with the sentiment expressed in (responder 8's) email on this point.

I understand that there are a lot of changes going on with how the PST
is being maintained (i.e. that it is being outsourced), but these
things shouldn't be used unless they are ready as a final product.
This was clearly not the case this past deadline.

----------------------------------------------------- End Response 17

[source/resource/session] [userdb]

first of all I think I have to thank the NRAO staff for the effort
that must have gone into the PST.

Two comments:

1) I have found some difficulties in understanding the
"source/resource" and still I have. May be "help" could explain a bit
more about them.

2) I am a Phd student, and it was the first time I submitted a
proposal, so I had to register myself.  Did I have to register me as a
graduate student? what do you mean by "expected graduation year"? Is
it the year that I will finish my Phd? Why can't I register me as a
Phd student?

----------------------------------------------------- End Response 18

[co-editing lock] +

The PST seems much improved to me and I had only two real issues.

1) It still remains a problem to indicate simulataneous dual-frequency
observations, and the work around of setting up sessions for both
bands tends to mess up the hour count.

2) I was trying to help a collaborator out with the technical part
(dual-frequency) of a proposal.  However, even when she logged out
completely from the tool, I was not allowed to edit the proposal.  The
tool claimed it was locked by another user.  After about 15 minutes I
was able to get in and do the edits.

However then she was not able to get back in to finish the proposal
because it was now locked by me even after I logged out.  I actually
shut down my browser completely to try to "unlock" it but it didn't
help.  As there was not a lot of time left before the deadline, she
ended up dictating changes to me over the phone.  Which was a bit
stressful.

While the proposal was "locked" by me, she could view it, and submit
it.  She simply couldn't edit it.

Ideally I think it would be nice that when one user leaves the "edit"
area, it unlocks the file for others.  At the least I would recommend
that logging out would unlock the files of that user immediately.

----------------------------------------------------- End Response 19
[overload] [source/resource/session] [save/validate] +

Sorry this is a bit late but I was travelling. I think there are still
several problematic issues with respect to the PST. Several of these
for a colleague of mine were summarized in an e-mail I sent in
May. The main ones that were apparent for me during the recent
deadline:

- the tool failed to respond during the day of the deadline. This was
apparently an overpressure on the computers but in reality I think the
method of continually saving to an NRAO machine is not efficient. One
should be able to save a template locally on your own machine, set up
the entire proposal then just upload that to an NRAO machine (see the
Chandra RPS for an example of this). When travelling one often has
extra time (say on an airplane) to set-up a proposal but no
connection. If this could be done locally it could then be quickly
submitted to NRAO when a connection is established. There are many
other advantages to this approach.

- the sessions and source/resource pairs are definately not
intuitive. I sort of understand them but have spent a lot of time
answering questions for my non-radio-expert collaborators who cannot
understand what they are supposed to do. They should be able to easily
set up a proposal but instead are getting discouraged from even
trying.

- the whole save and verify parts of the PST are not clear. There are
several of us that routinely lose information because we have not hit
the correct button at the correct time. The tool should be much more
clever in keeping all the information you work on unless you indicate
otherwise.
----------------------------------------------------- End Response 20


-- DanaBalser - 02 Nov 2007
Topic revision: r1 - 2007-11-02, DanaBalser
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback