PlotMS Architecture Review

Background

PlotMS was originally designed to be a Qt GUI application only, then a command-line implementation via the casapy interface was added later to enable scripting. This has resulted in an inconsistent feature set in the two clients. The overall architecture needs to be studied in order to make suggestions for improvement. Some of these recommendations would result in extensive refactoring. Of course, any effort would have to be prioritized against new development.

GUI vs. Command-Line/Scripting

• Inconsistent feature set
  • After investigation, almost all settings available in the GUI are also available as plotms parameters. The few exceptions are:
    • x-axis location (top/bottom) - radio button on GUI only. yaxislocation (left/right) is available as a plotms parameter.
    • "Show X Axis" and "Show Y Axis" - checkboxes on GUI only, no plotms parameters.
  • Recommendation: Add these parameters just for completeness (no tickets exist asking for them).
• Expandable parameters
  • The "expandable parameter" groups have a super-parameter that may/may not need to be enabled to use these sub-parameters. Some default to True, others False/empty string. This is inconsistent and can be confusing to the user.
    • These super-parameters include (with default): selectdata (True), averagedata (True), transform (False), extendflag (False), customsymbol (False), customflaggedsymbol (False), showmajorgrid (False), showminorgrid (False), iteraxis ("" = no iteration), and plotfile ("" = do not export plot).
    • For example, selectdata=True by default, so the user can perform a data selection directly (e.g. spw='0') without explicitly setting selectdata to True. However, transform=False by default, so if the user chooses a transform sub-parameter (e.g. freqframe='TOPO') without setting transform=True, this choice is ignored with no warning to the user except that this choice is not listed in the Plot Data Transformations in the log.
  • Recommendation: Set super-parameters to True where possible, in cases where the sub-parameters are disabled by default so that the default behavior of plotms would be unchanged. These super-parameters include selectdata, averagedata, transform, and extendflag. Unfortunately, they cannot be removed for backward compatibility; plotms currently has 80 parameters! The custom display super-parameters, which include customsymbol, customflaggedsymbol, showmajorgrid, and showminorgrid, must continue to default to False.
  • Recommendation: Warn user if sub-parameter value is ignored because super-parameter is not enabled.
• MVC
  • Plotms was designed with a Model-View-Controller (MVC) pattern, such that the View is updated first with the user's input (checkboxes, text boxes, drop-down lists, etc. changed as needed) and then with the changed Model state (the requested plot). When the Gui is not shown (showgui=False), the View is unneeded.
  • Recommendation: Investigate changing the code such that when the GUI is not used, the plot is created and exported with the Controller and Model only. Depending on the difficulty, the cost may outweigh the benefits.

Plotter interface

• Generic Plotter
  • Most plotter objects in plotms are instances of a GenericPlotter class (code/graphics/GenericPlotter). This design decision was made long ago, apparently for future flexibility in plotter package choice.
  • Some Qt functionality does not exist in the generic base class, which can make adding/changing features in plotms much more difficult if not impossible ("you can't get there from here").
  • However, many Qt classes have been customized in code/casaqt, and they are used in the plotms code. Therefore, to replace Qt would require refactoring plotms anyway. So we have traded off functionality for little gain in flexibility.
  • Recommendation: Evaluate removing GenericPlotter layer.

DBus Interface

Communication between the casapy iPython session and the underlying plotms C++ code is achieved by sending XML messages over a DBus interface.

• XML
  • The decision was made long ago that the messages be sent in XML format, apparently for communications flexibility in future implementations.
  • The content of the message is a casa Record which has been converted to XML, then converted back to a Record after transmission.
  • This adds a layer of code that must be updated for new cases. For example, when OTF calibration was implemented as a plotms parameter, it was necessary to handle array types for the Record value in addition to simple types in the Record<->XML code.
  • Qt has DBus Adaptor classes that do not require this message translation layer. The viewer does not use this XML layer.
  • Recommendation: review need for XML translation and possibly remove this layer.
• DBus timeout
  • When plotms crashes (of course, it should not do this but bugs happen), the user only receives a confusing DBus timeout message that does not indicate the source of the problem:

QDBusError("org.freedesktop.DBus.Error.NoReply", "Message did not receive a reply (timeout by message bus)")
Type= 5
Timeout= 12000000 true (B)
QtTripped= true version= 264197

• • I have been told that DBus is unreliable but I have not found that to be the case; this could be blaming the messenger due to this timeout message.
  • Recommendation: Append/replace timeout message with some kind of "plotms probably crashed" message (first check if any other apps use the DBus interface).

Cache loading

• On-disk vs. real-time
  • Data from the measurement set is retrieved in the MSCache class (subclass of PlotMSCacheBase) using the VisibilityIterator2 / VisBuffer2 / TransformingVi2 / MSTransformIterator framework to read (and possibly transform) the data from disk. The VIVB2 is an integral part of the MSCache class, which is used by the cache thread. However, some metadata is obtained by accessing the measurement set directly with the MeasurementSet class.
  • Alternatively, the CalCache class is used for CalTable access.
  • In the future, it may be desired to plot the data real-time as it is observed/filled. This was being developed in Green Bank when I left using the Google Protocol Buffers framework for monitor & control data first, with a plan to extend it to GBT observed data (could not find any documentation/presentation on this on the GB Software wiki). If data is displayed as it is received, it may not need to be stored as a cache.
  • Plotms is a plotter and interactive flagger for measurement set data, which may also be selected and averaged. With real-time data, some features would have to be disabled, ignored, or generate an error message.
  • Recommendation: Investigate options for real-time data display:
    1. method for data transmission
    2. creating an alternative class for real-time data capture
    3. limiting plotms options for real-time data

Threading

• The background threads CacheThread and ExportThread are subclasses of async::Thread in the msvis package, which includes utilities from the boost package.
• The drawing thread QPDrawThread is a subclass of QThread from the Qt package
• The cache and draw threads communicate with thread controllers to update progress bar widgets.
• As far as I can tell, threads behave as expected and get the job done. There were no threading-related recommendations from the HPC group when they looked at CASA parallelization including plotms.
  • Update: For screen resolution export, PlotMS queries whether the plotter is drawing (thread is running). In testing it happens to check this between the cache and draw threads (thread is null at the time) so it launches the export before drawing is complete.

-- PamFord - 2015-10-15