PlotMS Notes

This page contains accumulated requirements, documentation, and other notes for the new PlotMS. It is intended mainly for use by the developers to keep lists and designs updated in one place as new requirements and information trickles in. While comments from non-developers are of course welcome and encouraged, please do not directly edit this page without consulting one of the developers.

Main Requirements

Flexibility

Very flexible 2D display of MS and Caltable info.

• "Flexible" == lots of axes choices, including support for plots that do not involve the visibility data (e.g., field vs time).
• Some interactive flexibility should be available, e.g., alternating among "standard" axes without re-launching or re-reading the data.

Efficiency

• Minimize bulk I/O, bulk data copies, and re-calculations as much as possible.
• Avoid unnecessary GUI redraws/refreshes, and use layering where possible to separate fast and slow redraws.

Locate and Flagging

• Plotted points must be interactively (within box, near cursor) identifiable with a meta data report to logger ("locate").
• Option to set MS flags for interactively selected data (plot refreshes with flagged data omitted).
• Flagging should be independent of axes choices.
• Interactive toggles should be available to extend selected flags along various meta-data axes.

Averaging

• Flexible time, freq, scan, spw, baseline pre-averaging to improve SNR (and speed) of the plotted data.
• Flagging of unaveraged data must be supported.
• Averaging should be independent of axes choices.

User Interface

UI should be rational and consistent.

• Balance start-up parameters and interactive parameters: more interactive options than current MSPlot, e.g. axes choices; use standardized syntax (e.g., MSSelection) wherever possible.
• Clear, concise and consistent plot labeling, including (especially!) when iterating and multi-plotting (overall data selection is not a label, but should be reviewable).
• Multiple plots with a common axis should share this axis when possible to optimize use of the plotting surface.
• Flexible color and symbol options for distiguishing meta-info in data plots (e.g., different spws in different colors, etc.).
• Good user feedback, especially for time-consuming plots (progress meter, execution time forecasting).
• Efficient refresh and regular user-interrupt opportunities.
• Support for lines (connected dots) in some contexts? (E.g., plots of polynomial bandpasses, etc.).

Other

• Coordinated plots, e.g., a flagging on amp in one panel erases also the corresponding phase in another panel; or, flagging a datum on one baseline also flags the corresponding data on other baselines involving one or both of the antennas in the original baseline (for all other meta data the same). This is probably easily realized by making multiple plots from a common plotting cache.
• Additional on-the-fly data munging, e.g., Hanning smoothing, calibration application. Supported via use of VisIter/VisBuffer, which easily (or already) support such things.
• Eventual suppport for multi-MS, MS+CalTable plotting: each MS, CalTable build their own plotting cache, and each fed to the plotting calls. We probably get this for almost-free, via general support for multi-MS in VisSet/VisIter, etc.

Development Strategies

Learn From Previous Efforts

Recognize short-comings of existing MS plotting (MSPlot, TablePlot).

• Calling (python) matplotlib from C++ is probably a bottleneck.
• There appear to be some redundant copying issues buried in the current MSPlot implementation (not clear where, but possibly in matplotlib).
• New capabilities (multi-color multi-plots) have poor performance, and require actively defeating features of TablePlot to remedy.
• TablePlot very capable for generic applications, but MS (and CalTable) plotting (and the flagging and averaging requirements) make use of TablePlot (TaQLs) fairly complex. Access to MS and CalTable data is probably best realized by using classes tuned specifically to this purpose. Meta-info and data are then much easier to keep coupled, etc.

Use Existing Code

Use (working and efficient) existing code as much as possible

• Some MSPlot/TablePlot code may be recyclable.
• Use VisSet/VisIter/VisBuffer for data traversal, generalized data munging support (e.g., calibration), and averaging.
• Qwt plotting backend on top of Qt GUI backend.

Plotting Cache

Support for automated axes limits, locate, flagging, and averaging pivot on a well-defined and efficient plotting cache.

• Caches the x and y values (and mask) for the plot, as efficiently as possible.
• Caches sufficient meta-data for locate and flagging (including when averaging).
• Supports alternate meta-data choices for axes.
• Should be a virtual table ( not an MS), auto-paged to disk for large plotting cases.
• Filling the disk amounts to a (post-calculation) data copy, but will limit the need to go back to disk for the raw data.
• Is there value in a persistent (post exit) plotting cache?

Flagging Selection

The plotting cache will embody a minimal set of data selection meta-data for communicating flags back to the MS.

• Should establish a well-defined and efficient definition of a flagging selection object.
• Note correspondence to MSSelection.
• Lists of such objects should be sortable and collapsable.
• The flagging selection class should be portable, potentially for using in non-interactive flagging applications in future.

Logical Execution Sequence

1. Plotting application called from the user interface
   1. Global data selection is applied (a la calibration and imaging applications), plotting-specific parameters passed in
   2. Traverse data (I/O) via VisSet/VisIter/VisBuffer mechanism
   3. Plotting cache initialized
   4. Data processed as needed (averaging, cal apply, value formation)
   5. Plotting cache filled iteratively from VisBuffer
2. Initial plot formed and displayed
   1. Interactive session begins
   2. Optionally zoom, pan, page etc.
   3. Optionally change axes, ranges (refresh/refill cache, as needed)
   4. Interactive selection by box, range, point
      1. Interpret selection via cache/MS
      2. Locate (in cache, report meta info)
      3. Flag (Locate plus apply flags to cache/MS and refresh)

Design

High-Level Diagram

Description

PlotMS consists of four main modules: PlotMS, the controller that provides access to the outside world, PlotMSPlotter, the plotting GUI, PlotMSPlot, the plot manager, and PlotMSCache, the data manager. The modules use synchronized parameters that could come from the command-line, the GUI, or scripting via a collection of different types of PlotMSParameters which describe the state of the system.

PlotMSPlotter consists of a Qt main window surrounding a generic Plotter. The main window manages GUI interaction with the user, while the generic Plotter manages the actual plotting using the GenericPlotter class hierarchy. Currently, PlotMSPlotter uses the default (and only) plotter implementation, QwtPlotter, which uses the qwt plotting package (which in turn uses the Qt library).

PlotMSPlotManager is responsible for managing (potentially) many instances of PlotMSPlot, which is a high-level object that manages (potentially) many generic plotting objects across (potentially) many generic plotting canvases. The PlotMSPlot class system is hierarchical to support different plot types; currently the only type is a single plot, which is one generic MaskedScatterPlot associated with one generic PlotCanvas.

Each PlotMSPlot has a PlotMSCache, which is responsible for managing and storing visibility data in a form that can be used by the plot. This caching is necessary mainly because every time the GUI is redrawn (resized, zoomed, cosmetically changed, etc.), each data point must be revisited which would be computationally costly without caching; caching the data also avoids costly I/O for repeated file accesses. PlotMSCache also stores essential meta-information that allows for mapping between plotted points and rows in the MS, which will be used for averaging and flagging. For large datasets that cannot fit in memory, the cache must automatically and transparently page to disk.

GUI Layout

1. Menus.
2. Parameters/tools/customization etc. Needs to present widgets in consistent and sensible ways. Needs to intelligently handle space and layout. Needs to be able to be shown/hidden/resized by the user. Needs to be able to be disabled for non-interactive plots.
3. Toolbars for common tasks. A nice feature would be to allow the user to customize which actions appear on the toolbars. Needs to be able to be shown/hidden by the user.
4. (Blue line.) Main plotting area, uses a single generic Plotter class.
5. (Green line.) One or more plotting surfaces, using one or more generic PlotCanvas classes.

Features

Current

• Standalone application with some command-line options for creating a single plot and setting the log level.
• Parameters and actions are designed to be synchronized across different input sources (command-line, GUI, scripting, etc.)
• Logging of time and memory for major operations (loading cache, drawing plots).
• MS filename and selection parameters for plots.
• Zoom and pan tools (with a stack).
• Select regions, and clear selected regions. Marked regions are drawn in a separate layer to avoid costly redraws of large plots.
• Locate points in selected regions.
• Track mouse position.
• Axis display using time formatting when needed.
• Export plot to JPG, PNG, PDF, and PS.
• Can have multiple single plots, each with its own parameters, each in its own canvas. Each plot can do any supported axis vs. any other supported axis.
• Switch among loaded axes in the cache without additional work beyond drawing cost.
• Efficient indexing of arrays of different shapes in cache.
• Meta-information axes: TIME, TIME_INTERVAL, FIELD, SPW, SCAN, ANTENNA1, ANTENNA2, CHANNEL, FREQUENCY, CORR, and FLAG.
• Other available axes: BASELINE, UVDIST, UVDIST_L, U, V, W, AMP, PHASE, REAL, IMAG, AZIMUTH, ELEVATION, PARANG, and ROW.
• Data axes can use any of the data columns: DATA, CORRECTED, MODEL, or RESIDUAL.

Upcoming

• Cache automatically pages itself to disk when it exceeds memory limits.
• Threading, with progress reports and the ability to stop long operations.
• Flagging: displaying flagged/unflagged data, flagging data in underlying MS, flag extension.
• Scriptable and accessible from Python. Command-line parameters for MS selection.
• Caching for zoom/pan stack to avoid redraws as much as possible.
• Allow user to manage cache (load/release axes, see memory usage, etc.).
• More control over visual/cosmetic features like: custom tick labels, grid, background, fonts, etc.
• More control over GUI layout such as: customizing tool bars, customizing the side panel, etc.

Future

• Tool to let the user add/manage annotations.
• Multiple windows.
• More plot types like: different colors, multi-panel with shared axes and linked selection, iteration, multiple plots on one panel, etc.
• Averaging.
• Additional on-the-fly data processing via MSVis classes.
• Multi-MS, CalTable plotting.
• User help/documentation system.
• Non-interactive and/or non-GUI plots. Infrastructure for automated testing.
• More advanced zoom features, such as a pop-up zoom window, or have an "unaveraging" zoomer.

Under Discussion

• Interaction with matplotlib for annotations, etc.

Results

This section is for comparisons between plotxy and plotms. Comparisons are as fair as possible, but by necessity may be skewed due to different feature sets, parameter handling, etc.

Timing

TODO

Screenshots

TODO

Appendices

Relevant Code Directories

• code/msvis/implement/PlotMS
• code/msvis/apps/casaplotms
• code/casaqt/implement/QwtPlotter
• code/graphics/implement/GenericPlotter

Links

• https://bugs.aoc.nrao.edu/browse/CAS-927
• https://bugs.aoc.nrao.edu/browse/CAS-1095
• http://casa.nrao.edu/docs/programmer/DocTablePlot.html
• http://casa.nrao.edu/Memos/229.html