Opslog Software Docs

1. Intro

Much of my career at NRAO has been spent replacing Carl Bignell's software. This might be one of the last pieces of his code that we'll be re-writing. Maybe we'll get it done before I retire.

The Operator's Log tool, or Opslog, is what the Operators use while on duty to log what occurs during observations. Their supervisor also uses this tool to manage this information and make sure all the time accounting jives with the DSS and subsequent reports to the NSF.

Could say a lot more here …

2. Background

• OpsLog - a description of Carl's original tool
• The first pass at this tool can be found on the various 'opslog' branches of the nell repo
• See the attached word document for an attempt at distilling the most basic of requirements of this tool
• We are developing this using Pivotal Tracker: https://www.pivotaltracker.com/n/projects/1510872

3. Design

We are using the same server and client technologies as the GB PHT project, so for the basics, see those docs at GbPhtDocs.

Very briefly, here's some big differences between the GB PHT overall design and Opslog:

• At the model level, subclassing is being used due to the overlap between 'basic' entries and more 'complex' ones: that is all types of entries have common shared fields, so this is represented by the parent class.
• For the resource/httpadapter level, this class structure is leveraged again for code sharing
• That same subclassing has a similar impact at the UI level with the various forms for editing these entries.
• Unlike the GB PHT, more then one Viewport is needed on the client side to handle the fact that operator's "log in" to a certain shift before using the tool
• The web app's state (who is logged into what shift) is handled partially through cookies

3.1 DB ERD

3.2 Authentication and Shift Selection

As mentioned above, this Django/ExtJS application is unique in that the client side changes 'viewports', or screens. In addition, the old opslog tool did not make a distinction between user authentication and selection of a shift, it was just all one 'login'. We are making that distinction, and therefore are pointing out our design here.

3.3 User Interface Details

The UI consists of many interacting parts, and the fact that the list of entries is really a collection of different types complicates matters anymore. Here's some stuff from the visio doc that might help:

3.4 Reports

The way reports are created is pretty non-trivial. Here's some stuff to help understand it:

4. Development

4.1 Initial code

We've started a stand-alone Django project, uncoupled from the DSS for the time being. The bare repo can be found at: /home/dss/integration/newTool-bare

Once we're at a good stage to start thinking about coupling functionality with other parts of the DSS, we can simple copy of the 'ops' Django app to the nell Django project.

4.1.1 Sandboxes

Hey, what ports are the development servers going to use? How about:

sandbox	Port Number
release testing	9000
Paul	9001
Thomas	9002

4.2 Testing

We've got unit tests.

We will be making releases using the same mechanisms used with the DSS.

In addition, for test releases, we will attempt to create stand-alone, working versions of each release, leaving each release in place for reference as the project progresses. Some details:

• the releases will all be on the branch 'release'
• each new release version will be tagged with an appropriate version number (starting with '0' until our first official, non-test release).
• a new clone of the repo will be made for each test release, with this working dir on the release branch at the appropriate tag
• each test release will have it's own DB using a simple naming convention (ex: dss_opslog_test_release_x).

4.2.1 Unit Tests

• we have the standard suite of server side unit tests in the Django project. Nothing new there.
• we've introduced client-side unit testing using the recommended jasmine tool:
  • http://docs.sencha.com/extjs/4.1.1/#!/guide/testing
  • our tests can be found in ops/static/js/ops/ops-test
  • to run them, simply open this file in your browser: ops/static/js/ops/run-tests.html

4.3 Test Releases

Here we'll give brief instructions for how to test, and a summary of the release versions and where they live.

4.3.1 All Releases

• All releases live in /home/dss/testing/opslog
• 'release' symbolic link points most recent test release
• all managed using 'dss' account
• settings.py using database NAME 'dss_lopslog_release_0'
• running with development server using port 9000
• therefore, url: http://trent.gb.nrao.edu:9000/ops
• my.nrao.edu accounts needed in order to log in

4.3.2 Release 0

• 'release 0' is a non-tag name: used for /home/dss/testing/opslog/release0 and DB name
• code tagged at 0.1.0
• on 'release' branch
• features:
  • login management:
    • users must have a my.nrao.edu account
    • users must be operators
    • users can login and select a shift
    • users can logout
  • entry management:
    • users can create and edit entries for their selected shift (of all types)
    • users can view all entries from other shifts
  • failure management:
    • users can create a new failure (and edit it)
    • users can restore a failure
    • open failures from all shifts, and failures tracking lost time from current shift can be viewed
  • major features specifically out of scope:
    • all supervisor features
    • all reports
    • 'schedule view' at top of application

4.3.2.1 Release 0 testing instructions

• Get an account at https://my.nrao.edu (if you don't already have one)
• Let Paul or Thomas know what your username is
• Go to the tool: http://trent.gb.nrao.edu:9000/ops
• Before selecting a shift, make sure you are logged in using your personal nrao account, not 'gbtops': do this by making sure your name appears in the top field
• Choose the current shift ( if it's 1/1/2016 1 PM, choose the shift starting at 1/1/2016 7 AM).
• Test all the features that you see are available.
• When you are done, make sure you click the Logout menu option.

4.3.3 Release 1

• 'release 1' is a non-tag name: used for /home/dss/testing/opslog/release1 and DB name
• code tagged at 0.2.0
• on 'release' branch
• features:
  • search and summary web page
  • entry interaction closer to style of original opslog tool
  • entry navigation (go to first, last, previous, next)
  • entry edits tracked (who and when), and partially displayed in Log Details
  • right-click produces pop up menu in style of original opslog tool
  • default time for new entries == time for selected entry
  • distinction made between selecting a shift ('logging into a shift') and logging into/outof the opslog tool

4.3.3.1 Release 1 testing instructions

Nothing much new here. Basically follow the steps in section 4.3.2.1 (#4.3.2.1_Release_0_testing_instructions) However, after playing around with a shift for a while, and creating a number of entries, make sure it's summary page looks correct:

• http://trent.gb.nrao.edu:9000/ops/search

4.3.4 Release 2

• 'release 2' is a non-tag name: used for /home/dss/testing/opslog/release2 and DB name
• code tagged at 0.3.0
• on 'release' branch
• features:
  • Multiple Telescopes introduced:
    • Most entries can be associated with a specific telescope, or 'ALL'
    • Login, Logout, Weather entries are always associated with 'ALL'
    • The entrie opslog tool can be filtered by telescope: defaults to 'ALL'
    • Receiver list changes appropriately according to telescope
  • Shift Summary Report
  • Monthly Summary Report
  • Supervisor is notified via email when a failure from an old shift has been restored
  • Delete entries
  • Supervisor can view modification history of entries
  • Supervisor can view created by/on, last modified by/on at bottom of entry form
  • Entry types can be changed
  • Comments can be made on an entry (as a separate entry)
  • Navigate directly to an entry via it's key in All Entries tab
  • Project Change entry now includes 'observer remote' checkbox
  • Weather information is inserted automatically when Weather entry is created
  • Receiver list uses full description of receiver
  • More then a dozen other minor cosmetic fixes and enhancements

4.3.4.1 Release 2 instructions

Before we get into specifics, it should be noted that this current release has all the features we know of that we were asked to include from Carl's tool, plus a few new ones that we've introduced. So, in a general sense, the instructions for testing could be 'use this like you would Carls tool, or at least try to'. The one big exception is that we've introduced multiple telescopes; a note on that below.

• Get an account at https://my.nrao.edu (if you don't already have one)
• Let Paul or Thomas know what your username is (if you haven't already)
• Go to the tool: http://trent.gb.nrao.edu:9000/ops
• Before selecting a shift, make sure you are logged in using your personal nrao account, not 'gbtops': do this by making sure your name appears in the top field
• Choose the current shift ( if it's 1/1/2016 1 PM, choose the shift starting at 1/1/2016 7 AM) if no one else is testing. If someone else is testing at the same time, coordinate and work on different shifts.
• Test all the features that you see are available.
• When you are done, make sure you click the Logout menu option.

Note on Multiple Telescopes: this is the only major new feature that was not in Carl's tool. For that reason, all the ways to use multiple telescopes haven't quite been worked out yet. This release is our first pass at something we think might work, so let us know what you think. Anyway, here's how they work:

• Most entries can be associated with a specific telescope, or 'ALL'
• Login, Logout, Weather entries are always associated with 'ALL'
• The entrie opslog tool can be filtered by telescope via a combobox at the top: defaults to 'ALL'
• Receiver list changes appropriately according to telescope
• Monthly Summary Report is telescope specific
• I would recommend avoiding the use of 'ALL' with Project, Observer, Receiver Changes, and Failures. But that isn't being enforced right now.

4.3.5 Release 3

• back to using the 'nell' repository
• 'release 3' is a non-tag name: used for /home/dss/testing/opslog/release3 and DB name
• code tagged at 0.4.0
• on 'ops_release' branch of 'nell' repo.
• features:
  • Major:
    • Legacy Data: over 160,000 entries from 2002 to about 2014 have been imported from the test version of Carl's opslog tool
    • Login Screen
      • Default values now make sense (date field is filled in by default and stays synchronized with am/pm buttons and shift ID selector)
      • Shift Info panel up at the top shows information about the next, previous, current and selected shifts
      • There are now warnings for:
        • Selecting a shift that isn't yours
        • Attempting to log into two shifts in a row
        • Selecting a shift that isn't the current shift (or, if it is within 15 minutes of the next shift, the next shift is treated as the current shift)
    • Entry forms have been changed to be easier to use:
      • wasted real estate has been reclaimed to avoid having to scroll to see all fields
      • all read-only fields appear simply as display fields and are not part of the tab-order
      • due to popular demand, and despite programmer protestations, the Edit button has been introduced
    • Monthly Summary Reports: reports should now be numerically identical for months between 2002 and 2014 in both tools
    • View All tab uses paging (shows only 25 entries at a time). This is so it doesn't get swamped by the 160,000 old entries.
    • View All form has been reordered to be easier to use, and so that comment field is full size.
  • Minor:
    • Operators are warned if they are selecting a shift that is not "todays"
    • Daily Shift Summary emails sent out
    • Comments associated with an entry can now be viewed
    • Receivers can be 'retired': thus legacy data contains receivers that will not come up as an option when creating a receiver change
    • 'Maintenance' project does not need a receiver specified.
    • Shift Summary Reports handle multiple telescopes
    • Login to Shift screen includes Next and Previous Shift buttons
    • Login to Shift warns operator if they are trying to log in to two shifts in a row.
    • Comments and Restore entries default to current time, not time of their parent entry
    • 'parent_id' field included in the View All form.
    • When global telescope filter is set to ALL, new entries default to GBT.

4.3.5.1 Release 3 instructions

Here, we pretty much sum up the instructions we've given for previous releases (above):

• Get an account at https://my.nrao.edu (if you don't already have one)
• Let Paul or Thomas know what your username is (if you haven't already)
• Go to the tool: http://trent.gb.nrao.edu:9000/ops
• Before selecting a shift, make sure you are logged in using your personal nrao account, not 'gbtops': do this by making sure your name appears in the top field
• Choose the current shift ( if it's 1/1/2016 1 PM, choose the shift starting at 1/1/2016 7 AM) if no one else is testing. If someone else is testing at the same time, coordinate and work on different shifts.
• Test all the features that you see are available.
• When you are done, make sure you click the Logout menu option.

4.3.6 Release 4

• back to using the 'nell' repository
• 'release 4' is a non-tag name: used for /home/dss/testing/opslog/release4 and DB name
• code tagged at 0.5.0
• on 'ops_release' branch of 'nell' repo.
• features:
  • Major:
    • Schedule Timeline: highest panel (collapsible) contains a timeline of the schedule in the same style as Carl's old tool. Also includes DSS schedule (see below)
    • DSS-OPS coupling:
      • Schedule Timeline's top row is the schedule as defined by the DSS. Row contains project codes, and mouse over produces tool tip with details.
      • Project Change proposal field:
        • Is now a drop down providing all valid DSS project names
        • These options can be ignored and the user can type in and save what ever project name they want
        • Warnings are issued if a) the project name entered is not a valid DSS project name, or b) the Project Change produced is in disagreement with the DSS schedule
      • Project and Observer Change observer field:
        • Is now a drop down providing all valid DSS user names (alphabetical order by last name, displayed as first, last name)
        • These options can be ignored and the user can type in and save what ever user name they want
        • Warnings are issued if a) the user name entered is not a valid DSS user name, or b) the user name entered is in disagreement with the users specified by the DSS for the current project.
      • Project and Receiver Change receiver field: issues warning if the receiver specified is in disagreement with the current DSS schedule.
    • Time field for entry forms is easier to use
  • Minor:
    • Tabbing through form items works just like it does in Carl's tool
    • All entries can be duplicated via the 'Duplicate' button
    • Login/out entries will not be created by Operator Supervisors
    • Weather information more robust: formatting clearer, and warnings about late or missing data included.

4.3.6.1 Release 4 instructions

Here, we pretty much sum up the instructions we've given for previous releases (above):

• Get an account at https://my.nrao.edu (if you don't already have one)
• Let Paul or Thomas know what your username is (if you haven't already)
• Go to the tool: http://trent.gb.nrao.edu:9000/ops
• Before selecting a shift, make sure you are logged in using your personal nrao account, not 'gbtops': do this by making sure your name appears in the top field
• Choose the current shift ( if it's 1/1/2016 1 PM, choose the shift starting at 1/1/2016 7 AM) if no one else is testing. If someone else is testing at the same time, coordinate and work on different shifts.
• Test all the features that you see are available.
• When you are done, make sure you click the Logout menu option.

Special Note about Opslog tool and the DSS

We have tried to integrate parts of the DSS into the Opslog tool. One of the aims of this new feature is to avoid accidentally introducing information that is different from the DSS schedule. You can still do this, but the opslog will warn you about it. Because testing is being done with a separate database from the DSS production system, we needed to provide a schedule for testing that was completely fictitious. This schedule simply repeats itself every 24 hours, with Maintenance every day from 8:00 - 15:00.

4.3.7 Release 5

New features

View All Tab

Three new buttons here:

Go to Start of Current Shift: Navigates to the first entry of the current shift (the one you are logged into) Go to Start of Selected Shift: Navigates to the beginning of the currently selected shift (the one highlighted in the View All tab grid) Go to Shift: Navigates to the first entry of the shift ID entered in the search box

I've also decreased the load times and fixed a bug where entries could not be selected if they were on a different page.

Form Validation

There have been several enhancements to the way forms are validated.

The biggest change is that only the telescope, date, and time are required to be filled in correctly when you save an Entry. All other fields will indicate their validity status with a red box, but will not prevent you from saving. More on this in the next section.

Proposal, Project, and Observer are now only required only when the telescope is the GBT

Receiver is now not required for Maintenance Project Changes

There are now two types of Maintenance observation types: Scheduled and Unscheduled

Changing the observation type to either type of Maintenance will change the Proposal to Maintenance. It is not possible to auto-fill going the other way (Proposal -> obsType) since there are now two Maintenance obsTypes, so just remember to fill out the obsType first.

Logout Validation Checks

Now that Entries are not fully checked for validity at save-time, we needed some mechanism to ensure that they are correctly filled out when the user logs out. So, when you go to log out now, each entry in the current shift will be "validated" -- that is, each field within it will be checked for validity. If any of its fields are invalid, an entry will be marked invalid. This will be designated by red text in the log view shift entry list. If you have any invalid Entries, you will be alerted and given the opportunity to return to your shift and correct them.

After you have corrected an invalid entry, you can press the "Validate Entries" button to re-validate all of the entries.

Shift Summary Printing

You will now be given the option to generate a summary of your shift upon logging out of it. This will create and open a Shift Summary PDF for you, which you can then print out.

Summary Web Page Enhancements

The web-view of the shift summaries has been updated. There are some visual enhancements, and you can now search by ID. It will also now limit the number of days that you can generate summaries for (to avoid terribly long load times).

Observer Combo Box

This now lists Observers by "Last Name, First Name" rather than "First Name Last Name". This allows you to search by last name, rather than trying to remember what an observer's first name is based on their first initial/last name.

As an aside, please use the "Operator, GBT" user when it is an operator running the observation. This will help us keep track of things better.

In the future I hope to greatly enhance the name matching capabilities of this field, but for now we hope that this is a good compromise.

Insert Scheduled Project Changes

It came to my attention early on in the project that you (the operators) will typically create Project Changes for at least some of the scheduled DSS projects. This is done by looking at the DSS schedule and manually typing in the relevant information. I finally found some time to implement a feature I've wanted to give you guys, which is a button that inserts all of the scheduled Project Changes for the current shift (Create Scheduled Project Changes).

The button will only create new Project Changes; if a Project Change already exists that is similar enough (same proposal, date, and shift) to one that exists on the schedule, it won't be created again. The Project Changes that are created are just like any Project Change you would create by hand, so if you need to edit them you can do that. Also remember that if you need to delete an entry, you can do that via the right-click menu.

Here's how this works:

For every Project listed in the shift timeline (at the top), a Project Change is created. It is always populated with the Proposal and the Observation Type. If there is only one Receiver listed for that project, then it is filled in automatically as well. While there is observer and project ID information in the DSS, I know that the actual observer/project isn't known until the observation starts, so these are left blank.

Bug Fixes

Weather input

Previously weather could not be input into a form that had been edited but not saved. Saving the form, then opening it back up to edit would allow you to insert the weather data, but it would replace the current contents.

The new behavior is to simply insert the weather data at the current location of the cursor. It shouldn't matter if the form is dirty or not

Modified Dates

Previously Modified Dates were not being updated upon saving a form, but rather only after refreshing the page. This has been resolved.

"No permissions" Web Page

Previously, when you logged in but did not have valid credentials (because Paul or I messed up that step of the release) you were greeted with a blank white screen. You will now be directed to the proper "no permissions" page, so that you at least know the cause of this. Note that this should never actually happen, since you should all have the proper credentials to view the opslog.

Login Screen Shift ID Combo Box

Not sure if anyone noticed this or not, but when you selected a shift that was sufficiently far in ID from the currently selected one, the box would break, and you would no longer be able to use the next/previous/current buttons. This has been resolved

5. How to Import Old OpsLog Data into the New Opslog

Background

We had to use a tool called dbfExport (from dBase) in order to successfully retrieve all of the old OpsLog data. The export tool developed by SDD was unable to read the datetime columns correctly. It also had issues with the .MDX index files, if I recall correctly.

This is a Windows-only application, and it is currently installed on the SDD laptop HADDOCK. Paul Marganian, Steve Tritapoe, and Thomas Chamberlin have the license and password for activation.

Using dbfExport was rather frustrating, but we were able to settle on a workflow. The .csv files that are output by dbfExport are unable to be read by Python's csv library, so that was a dead-end. However, it is able to output Excel files (.xlsx) without issue. I initially tried to read these with a Python library, but it was terribly slow. So, we settled on the following workflow.

DBF -> .xlsx

• Open dbfExport
• Next
• File system
• Navigate to the .DBF files (C:/opslogData) and open the .DBF file you'd like to import
• Next
• Select either of the Excel 2007
• Make sure it's outputting to the valid directory.
• Column Layout
• Next
• Do Export

You will need to do this for both the logentries.DBF and shifts.DBF files. You will need to ensure that the associated .DBT and .MDX files are copied over along with the .DBF files, and available in the same directory.

.xlsx -> .csv

Excel knows how to output .csv files properly, so we will now use it to create them.

• Open the relevant .xlsx file with Excel. This will probably take a while; they are very large files.
• File -> Save As -> CSV (Comma delimited) (*.csv)
• Save

I'd suggest saving them as I'd suggest saving them as logentries_excel.csv and shiftentries_excel.csv to avoid any confusion, but this probably wouldn't be necessary in the future.

.csv -> OpsLog

Using the scripts ImportLogEntries.py and ImportShifts.py in ops/tools/legacy it is very easy to import the .csv files into the OpsLog.

You will first need to prepare your database for the import. This will involve creating a fresh copy of the DSS DB and running ops/tools/{opsStatics.py,setupOperators.py} on it. However, this will change once the OpsLog is integrated into the DSS release.

Prepare a new DB to import the CSV files into

This is for adding the legacy data from the CSV files into a nell (dss) database - therefore, you can't hold on to any ops data you might have there. You have to delete it.

Start with copy of production DB

• copy the production DB to your test DB
• drop the ops tables that are in there by accident: psql -U dss -d $DB < sql/dropOpsTables.sql
• for some reason we screwed up auth_permission and django_content_type tables: psql -U dss -d $DB < sql/restartDssSeqForOps.sql
• python manage.py syncdb --noinput --no-initial-data
• get rid of time zones: psql -U dss -d $DB < sql/remove_timezones.sql

The above lines are in the clearDB function of nell/resetDB script.

OR: Start with a fresh DB

• create a fresh DB (as opposed to copying the production DB)
• run syncdb on it (w/ --no-initial-data)
• get rid of time zones: psql -U dss -d $DB < sql/remove_timezones.sql

• python ops/tools/opsStatics.py
• python ops/tools/setupOperators.py

Import the CSV files

• python ops/tools/legacy/ImportShifts (.csv file)
• python ops/tools/legacy/ImportLogEntries (.csv file)
• Go into DB and reset the ops_*_id_seq for the shift and entry tables
• Save any feedback/printouts you get for your records

I've verified that all the above directions are accurate (PRM - April 11, 2016)

How to Give a User Access to the OpsLog

There are two systems that a User will need to exist in before they can access the OpsLog.

The user will first need to create an account at http://my.nrao.edu

Once this my.nrao.edu account has been created, that person can then go to any dss url (production or sandbox) where a login is necessary. The get_requestor function looks for the User object associated with the my.nrao.edu account, and if it does not find it, creates it. Again, the PK of the my.nrao.edu account in the PST DB is what is used to link this User object (pst_id field) with the my.nrao.edu account.

Then their User object will need to be given the permissions of scheduler.models.Role.objects.get(role="Staff") and scheduler.models.Role.objects.get(role="Operator").

Operations supervisors should also be given scheduler.models.Role.objects.get(role="OperatorSupervisor").