ArchivistDocumentation

Development plan

Phase 1:

  • Define a few tables in the na_feic database to receive raw data for band 6 noise temp., image rejection, and gain. (Morgan & Michael, done.)
  • Define a configuration file format which maps individual monitor parameter URLs to columns in the tables. Must include method of determining when to write a record. (Morgan, done. Documentation below.)
  • Prototype an Archivist which uses the config file to subscribe to the values, accumulate and write the records. (Morgan, done.)

Decision: In order to be able to archive or otherwise see the values of control parameters which are set by a model from within its own methods:

  1. DataSocket Client Create needs an input "Archive Control Params" which gets saved as an object attribute. It defaults to false.
  2. When the DS client subscribes, in DS Client Subscribe, it passes this value to an input of the same name on DS Listener Create. DS Listener then uses this to decide whether to ignore or accept values having the attribute "Archive" = true.
  3. In Model Base Set Parameter by Name, when control parameters are publishe, set the Archive attribute to true and publish. Go with a raw DS write for now (acknowledgement will be needed soon but not designed yet.)
  4. Archivist creates its DS Client with "Archive Control Params" = true.

Done.

The alternative we considered of publishing everything on the DS Client that every model instance owns (or could own with slight mods.) However this will likely trip up the DataSockets "multiple writers" bug/issue/design travesty.

Phase 2:

  • Modify all Data Description List vis to include an "archive" flag for each parameter. (Tedious, Done)
  • Modify Model Base and Sampler to take the "archive" flag into account in Sampler Publish. (Mike, Done)
  • Devise an ackowledgement scheme whereby Sampler Publish expects an acknowledgement from (one or more) Archivists when a value is published for which "archive" is true. (Mike, Done)
  • Log all "archive" parameters are to a text file on the test system, just in case, to prevent data loss. (Mike, Done)

NEW Phase 3

  • Diagram of improved Ack scheme.:
    Archivist.GIF
  • Improve acknowledgement scheme as illustrated above:
    1. The Archivist is created. It uses the hostname in the Source section of its configuration file to find the Archivists Table.
    2. It reads the Archivists Table, searches for and/or appends its list of entries to the table. Each entry is the target Model instance name, Parameter name, and a string containing the Archivist hostname+instance name.
    3. A Model instance is created. It loads its Parameter entries from the Archivists Table from localhost.
    4. It creates its Parameters. For each Parameter with Archive?=TRUE it searches the Parameter entries from Archivists Table and if a match is found, passes the Archivist host+instance name to the Parameter Create function which stores the info.
    5. When a Monitor Parameter goes to publish, or when a Control Parameter SetValue is called, and Archive? is TRUE, it publishes the data and waits for an acknowledgement from the specified Archivist host+instance name.
  • If no acknowledgement is received, the sampler logs a warning to the error/status notification module.
    • This is proving problematic for now because Sampler Publish is used to call the E/S module and therefore can't be called from within Sampler Publish. What to do?

Phase 4

  • Modify the Band6DAQ to publish the monitor and control parameter values, as appropriate. (Michael)
    • note: The Band 6 DAQ currently logs the values of bias parameters to the spreadsheet. Perhaps these should be logged to a database table as well?
  • Add Archivist to Band 6 beam pattern measurement.
  • Make a new Archivist which deposits data directly into the MeasData table, and not require an intermediate table to stage the data.

possible future work:

  • Add a control parameter to Model Base to allow the "archive" bit to be set or cleared dynamically. The flag in the DDL then serves as the initial value.
  • Add a method to publish a cluster of values and have the Arcivist, using its configuration file, unpack them and write the database record accordingly.
  • Investigate how the data taken as above can be transformed into the FEIC database at cartridge delivery time.

Documentation for MySQL Archivist as of now (27 Apr 2006)

Each instance of MySQL Archivist model archives string, integer, or floating point values (only) from a single Model to a single table in a MySQL database. It does so in a flexible, configurable way. The MySQL Archivist Model takes a path to an INI configuration file as its Resource and it publishes only a couple of monitor parameters indicating its status of connection to the database and the validity of the configuration file. The configuration instructs it to subscribe to monitor and control paramaters of some other Model. The configuration file is interpreted as follows:

[Database]
Connection=<ODBC DSN name>
In the Database section, the Connection gives the name of an ODBC DSN which is configured to connect to an existing MySQL server and database. The DSN contains all the info needed to connect to the server host, log in, and open a database. TODO: Add documentation for setting up the DSN.

[Source]
Host=localhost
Name=<model name>
TermURL=Archive
The Source section tells the Archivist how to build the URLs it uses to subscribe, and how to know when a full record has been accumulated. Host gives the hostname to build the URLs for. Model gives the model name which is publishing the data.

About TermURL:
  • TermURL gives the name to watch for to trigger a database write of the data accumulated for one record. The TermURL shown here would result in a URL like "dstp://localhost/model name/monitor/Archive".
  • When a value (conventionally Boolean TRUE) arrives on this topic, the Current record, if valid, is written to the database. If not valid, an error is logged and the record is dropped.
  • The Archivist can accumulate multiple records simultaneously. Whatever the last valid combination of fields with Key=1 was seen determines the “current” record to receive incoming updates.

[Table]
Name=<table name>
URLString=<name to subscribe>, optional
In the Table section, the Name of the table to write to is given. The URLs are build using the same name unless a URLString is provided in which case that is used instead.

[FieldN]
Name=<table column name>
URLString=<name to subscribe>, optional
Default=<default value>, default 0
Required=0 or 1, default 0
Key=0 or 1, default 0
IsControl=0 or 1, default 0
Subscribe=0 or 1, default 1
Comment=<comment text>, optional, ignored by Archivist.
Next follows one or more Field sections [Field0], [Field1], [Field2], etc. It doesn't realy matter what the sections are called or whether they are numbered consecutively, just that the section names are unique. The Archvist will parse all sections in the file and interpret any besides [Database], [Source], and [Table] to be field sections. The rules for Field sections are as follows:
  • If the section contains Subscribe=0 then the URL is not subscribed and a Default value, if provided, will be used for all records. If no Default is provided, then the database table’s default will be used.
  • The Name is taken to be the column name in the table and is also taken to be part of the URL to subscribe.
  • However, if the section has an URLString entry, that will be used instead of the field name for building the URL.

Example:
[Table] 
URLString=Band6-CartTest
[Field0]
Name=Pol
[Field1] 
Name=keyDataset
URLString=DataSet

Resulting subscriptions:
  • dstp://localhost/Band6-CartTest/monitor/Pol
  • dstp://localhost/Band6-CartTest/monitor/DataSet

Continuing with the [FieldN] contents:
  • If the section has Required=1 then the Archivist will not write out a record until a value for the field has been provided. TODO: this may be not working as of 27 Apr 2006
  • If the section has Key=1 then the Archivist uses it to uniquely identify a record. All Key records are by definition Required, though it doesn’t hurt to include Required=1. The model must publish Key values before non-Key values. The last valid combination of Key values indicates the “current” record. As data arrives it is cached to be saved with the “current” record. There can be multiple records cached in the Archivist’s memory.
  • If the section has a Default value then that will be written with each record unless it has been replaced by an explicit value. Default=0 is assumed for all fields.
  • If the section has IsControl=1 then the URL will be built to subscribe to a control parameter rather than a monitor parameter. IsControl=0 is the default so the item may be left out of the section for monitor parameters.
  • The Comment item is ignored and is only for the benefit of the person editing the INI file.

-- MorganMcLeod - 16 May 2006
Topic revision: r7 - 2006-05-16, MorganMcLeod
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback