Software Documentation for the Green Bank Proposal Handling Tool (GBPHT)

1. Intro

The GB PHT is used to prepare GBT proposals from the PST for the TAC meeting and subsequent scheduling in the DSS. Like the Sensitivity Calculator and the Scheduler's tools, the GB PHT consists of two parts: a web server implemented as a django application (in the nell git repository), and a client web application. Here the distinction is that the client code is not implemented in GWT, but in extJS's javascript, and lives right in the static directory of the 'pht' application in nell.

This graph summarizes the GBPHT's role in the life cycle of a proposal.

systems.jpg

2. Background

  • The main wiki for the GBPHT project is here. This was mostly for the development phase of the project.
  • The Project Notes, or memos are a good read.
  • During development, we used Pivotal Tracker.
  • PST - Proposal Submission Tool - users submit proposals to NRAO telescopes with this
  • NRAO PHT - after proposals are submitted, other NRAO telescopes use
  • See Toney Minter's slides (attached as GBPHT.pdf) for a users perspective on the final system (as of Oct 2014)

3. Server

As mentioned above, the GBPHT server was yet another django app in our 'nell' git module (alongside users, scheduler, and calculator). It has a similar collection of models, with similar set of tables as the models/tables defined in the scheduler and user apps, except these tables all begin with 'pht' (ex: 'pht_receivers). Many of the overall patterns are also the same. A big distinction is that the GET/POST patterns implemented are distinct from the other nell apps in order to work correctly with the client javascript we developed (see below).

3.1 Models/Database

Like other DSS Django applications (scheduler, users, calculator), the pht applications data is defined via 'models', which are basically python classes that Django uses to construct your relational DB. The link displays the tables created by the models in our pht Django application.

https://staff.nrao.edu/wiki/pub/GB/PHTool/DataSources/gb_pht_database.pdf

3.2 Serving the client

With the server running on port 9001, the following URL, http://trent.gb.nrao.edu:9001/pht/proposals, gives you a whole bunch of JSON, that looks a bit like this:

{

    "total": 484,
    "proposals": [ … ],
    "success": "ok"

}

where "proposals" is a list of JSON objects that look a bit like this:

{

    "obs_type_codes": [
        "S",
        "C",
        "Y"
    ],
    "srp_to_pi": null,
    "create_date": "01/13/2011",
    "friend_id": null,
...
}

How does that work? How do you get Proposals in a JSON form in your browser when the persist in a database on the server side? Here's the trace:
  • http://trent.gb.nrao.edu:9001/pht/proposals maps to this in pht.views.py: login_required(proposalResource.requestHandler)
  • pht/resources/ProposalResource.py is responsible for using the ProposalHttpAdapter in the appropriate manner to return the appropriate response the GET, POST, or PUT HTTP request.
  • pht/httpadapters/ProposalHttpAdapter is responsible for converting back and forth between our model layer (a Proposal object with several child objects) to a flat JSON structure.

Much of the objects residing in our DB are directly accessible from the client via this RESTful interface (the formatting of the URL, the packaging of the data, etc.)

There's also lots of URLs that provide different services and don't follow this pattern.

4. Client

We implemented the GBPHT's UI as a web application using Ext JS. The library lives in nell/pht/static/js/lib, while our application code lives in nell/pht/static/js/pht.

Here's the great docs!

We learned how to work with stuff from the great [http://docs.sencha.com/ext-js/4-1/[][docs]]. You can make sure you're using the right docs for the version of Ext JS you're running by running the django server and pointing your browser at ?

Examining the directory structure of nell/pht/static/js/pht. gives you an idea of how the Ext JS stuff works. There you'll see these directories:

  • model
  • store
  • controller
  • view

This is a bit like the MVC patter. Models and Stores (M) define data structures and how they are sent and received from the server. In view we define the actual widgets that the user sees (V), and the controller (C) specifies how the interactions work. We'll review each of these layers really quickly:

4.1 Ext JS Models

Each file in nell/pht/static/js/pht/models is a distinct model, and is based off the JSON served up from the server (see above). For example, nell/pht/static/js/pht/models/Proposal.py:

Ext.define('PHT.model.Proposal', {
    extend: 'Ext.data.Model',
    fields: ['id'
           , 'pst_proposal_id'
           , 'proposal_type'
           , 'obs_type_codes'
           , 'srp_to_pi'
           , 'friend_id'
...
    proxy: {
        type: 'rest',
        url: '/pht/proposals',
        timeout: 300000,
        reader: {
            type: 'json',
            root: 'proposals',
            successProperty: 'success',
            totalProperty: 'total',
        }
    }

Note that the fields match one to one in our Proposal JSON. The proxy is where we tell Ext JS how this models get populated from the server. Thus we specified 'rest' for type, and the url points to '/pht/proposals'. This took some experimentation to get right when we first got up and running.

4.2 Ext JS Stores

Stores explain how the models are stored on the client side. Not much to say here, because usually most of what you specify in the store we specified in the model's 'proxy' field. Here's /pht/js/static/pht/store/Proposals.py:

Ext.define('PHT.store.Proposals', {
    extend: 'Ext.data.Store',
    model: 'PHT.model.Proposal',
    autoLoad: true,
});

4.3 Ext JS Views

Here's where things can get a little more complicated. Everything that a user actually sees in the UI is defined in /pht/js/static/pht/store/view. Furthermore, this directory has subdirectories so that related widgets can be grouped together, for instance the proposal dir. has code for the Proposal Explorer, or grid, and other specialized tools for the proposals. Let's take as Allocate.js as example. Here we define a widget that extends PHT.view.proposal.Edit (in Edit.py) that in turn extends PHT.view.FormPanel. This eventually extends Ext.form.Panel. Inside the Allocate class you can see we define our actual form fields (checkboxes etc) and our 'save' button. The loading of this form and the results of hitting 'save', following the MVC pattern are the responsibilities of the Controller.

4.4 Ext JS Controllers

Here's where things get really complicated. Gluing the model and view layers together is the control layer. In nell/pht/static/js/pht/controller/ we have files corresponding roughly to the controllers for various view sub-directories. For example, we have Proposals.js. AT the top, we can see that we define what models, stores, and views we'll be needing. Then, in the class's init function, we map actions to functions in our class. So, our 'save' button from the Allocate form has:

            'proposalallocate button[action=save]': {
                click: this.allocateProposal
            },

Further down, we see that this allocateProposal function gets into more standard looking javascript actions: navigate the DOM to get a handle on the form that has our values. From the values, we get the identifier of the proposal, and use this to get the actual model of this proposal (the data!). This is done by doing a filter on the store, which allows us access to all the proposal models. After this, we have our bussiness logic, which in this case, means setting fields in the retrieved proposal (and session) model (or object if you like). Then, we send these changes down to the server so that they'll persist in the DB by calling the 'sync' routine on our store.

Of course, its way more fucking complicated then this, but this is the high-level view of it all.

4.5 How does the Java Script get loaded?

Here's the path I trace for the pht:

  • web browser requests: trent.gb.nrao.edu:/pht/
  • nell/nell/urls.py: maps this request to nell/pht/urls.py
    • , (r'^pht/', include('pht.urls'))
  • nell/pht/urls.py: maps this to nell/pht/views.py
    • , url(r'^$', root)
    • , (r'^static/(?P.*)$', 'django.views.static.serve', {'document_root': settings.STATIC_PHT})
  • my STATIC_PHT is: STATIC_PHT = NELL_ROOT + '/pht/static'
    • NELL_ROOT = '/home/sandboxes/pmargani/dss/nell'
  • nell/pht/views.py: def root
    • render_to_response("pht/root.html", {'extjs' : settings.EXTJS_URL})
  • root.html in nell/templates/pht, which is in the TEMPLATE_DIRS list
  • settings.EXTJS_URL: EXTJS_URL = '/pht/static/js/lib/extjs'
  • nell/templates/pht/root.html:
    • /pht/static/js/pht/pht.js should be loaded from whatever settings.py has set for NELL_ROOT.

5. Data and Work flow

Let's get back to that cool picture from the beginning:

systems.jpg

This is actually a little misleading (it was made at the beginning of the project). A more accurate graph is taken from the PHT Project Note 3 memo:

ProposalLifeCycle.jpg

5.1 Proposal Life Cycle Summary

Here's just a summary. For gory details see PHT Project Note 3.

  • Proposal Deadline comes around (Feb 1, Aug 1): the Sensitivity Calculator suddenly gets pounded on and there are hundreds of new proposals in the PST
  • The next day GB staff wants these PST proposals imported into the GB PHT
  • GB staff start using GB PHT heavily, finding new and annoying bugs. Note that proposals that were formally in the GB PHT, then the DSS are tracked to provide feedback as to how much available time the GBT has. DSS Lookahead simulations are used to help figure out the 'pressure', which tells us really how much time is available on the GBT by the beginning of the next semester.
  • About 3 months later, the TAC meeting is held: GBT proposals in the NRAO PHT are sync'd with proposals in the GB PHT, as the GB PHT is used to edit these proposals.
  • After the TAC meeting, the next semester is prepared for by 'transferring' accepted proposals in the GB PHT into the GBT DSS.
  • The next proposal deadline approaches and the cycle continues anew.

5.2 Data Interfaces

A whole lot can be said about this and the proposal life cycle, but what's important to the software engineer? The interfaces: what they are and how they work.

  • PST -> GBPHT: The PST DB is a mysql database that may be undergoing refactoring(?). When importing GBT PST proposals to the GB PHT, we simply use mysql queries from python, take the results, and populate our 'models'. We also use a reporting infrastructure to keep track of what was done when and if there were any errors.
  • GB PHT -> NRAO PHT: During the TAC meeting, the committee does not interact with the GB PHT. Instead, GB staff make changes to GBT proposals in the GB NRAO, and these changes are automatically reflected in the NRAO PHT. This achieved by first doing a bulk import of proposals from the GB PHT into the NRAO PHT and then enabling an automatic re-import of each proposal as it is changed in the GB PHT. This is accomplished by 'pinging' a given URL (settings.py) from the GB PHT via the server when changes are made.
  • GB PHT -> DSS: When proposals get approved by the TAC, they then need to get scheduled. This is simple: each proposal (and associated objects/table rows) gets an associated DSS project (and associated objects). Since these live in the same DB (though they are in different Django applications), this is pretty trivial, though there is special code for converting the special needs of the GB PHT to the needs of the DSS. Because each accepted GB PHT proposal has an associated DSS project, we can track the progress of this original proposal as it observes. For example, by the next semester we can tell if a given proposal has completed or not.

6 Testing/Preparing before the TAC

Here's what normally needs to get done before the TAC meeting to make sure the NRAO and GB PHT are working well together:

6.1 Testing with NRAO PHT test system

  • NRAO PHT test: https://webtest.aoc.nrao.edu/pht
  • The NRAO PHT may have an easy way of pointing to a GB PHT sandbox, but for now, they are pointed at the production GB PHT (gollum server, dss DB)
  • NRAO PHT: choose next semester, GBT
  • NRAO PHT: 'Import GBT/16A'
  • Now both GB PHT and NRAO PHT should have same proposals. Do a spot check of the Proposals, some of their sessions. This isn't too hard because the NRAO PHT displays a relatively small amount of information per proposal.
  • Check the 'ping'. That is, make a simple change in a GB PHT Proposal (ex: add '(test)' to beginning of proposal title). Within a few seconds the NRAO PHT Proposal's title should update automagically.
  • Set the 'LST Pressure Tunables', in this order: 0.5, 0.25, 0.25, 0.5. (we originally thought we'd often tune these, and never have).
  • Import the GBT Availability & Carryover: (Need to streamline this)
    • At command line: /home/dss/release/nell>python pht/tools/LstPressures.py > lsts.txt
    • Find line 'Remaining Hours (poor, good, excellent) by LST (0-23)' in lsts.txt
    • Grab the next 24 lines of text and place in a file ('remaining.txt').
    • Do the same as above, but for the lines under 'Carryover Hours (poor, good, excellent) by LST (0-23)'
    • Choose these two files when importing availability and carryover into the NRAO PHT
  • Now you're ready to test the LST pressure plots: they should display the same data in both systems. needs more details

6.2 Preparing for meeting using production NRAO PHT

Basically, you want to spot check the same stuff you did in the NRAO PHT test system, just to be sure. And, of course, you'll have to do the same imports all over again:
  • Import semester proposals
  • Set 'LST Pressure Tunables'
  • upload availability and carryover

6.3 LST Pressure Details.

This section is out-dated

This part sucks. There is a long history of how the pressures in both tools have been tested against one another. Here's a listing of some of the methods used:
  • Just look at the actual plots in both tools. Use identical filters to make sure these match as well. This is fine for finding gross errors, but won't help finding small ones. And it's useless for debugging.
  • Text Reports: this is my preferred method. If both tools can produce text reports with the same exact format, then simple tools like tkdiff can be used to hone in on where the differences lie.
    • complete report: this is a listing of each session, followed by it's 24 LST pressure values. Socorro produces this.
    • complete report by weather type: just like above, yet a separate file for each weather type (good, bad, excellent). Socorro produces these.
    • web page tests: here we swipe the tables listed below NRAO PHT's pressure plot and stick them in a text file. Then we point our pressure code at these files, and run the 'compare' methods to systematically compare our results. This can be done broken down by weather as well.

Lessons Learned:
  • the coefficients for the night time flags are not exactly the same in both tools. This is a known and accepted minor issue. if one wants to be sure this is not a source of error, one can get both tools to use the same coefficients (just send the over the fence).
  • often, a problem in the plots is not a difference in the calculations themselves, but a difference in the inputs (wrong sessions, semester, etc.). The text reports easily show this kind of thing.

Don't forget that NRAO PHT does not compute remaining time, instead we send this to them via a file of 'availability': a CSV file where each line has three weather values for the availability of that LST.

7 DJango / ExtJS Example

A simple Django app that serves a simple ExtJS client has been created to make some of these patterns a little more obvious. It can be fount at: https://safe.nrao.edu/wiki/bin/view/GB/Software/HowToUseExtjsWithDjango

-- PaulMarganian - 2013-01-16
Topic attachments
I Attachment Action SizeSorted ascending Date Who Comment
GBPHT.pdfpdf GBPHT.pdf manage 1 MB 2014-10-23 - 10:11 PaulMarganian  
systems.jpgjpg systems.jpg manage 21 K 2013-01-16 - 13:49 PaulMarganian  
ProposalLifeCycle.jpgjpg ProposalLifeCycle.jpg manage 59 K 2013-02-22 - 12:45 PaulMarganian  
Topic revision: r11 - 2015-11-16, PaulMarganian
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback