About Mark5 modules
A Mark5 module is a metal enclosure the size of a toaster that contains eight 3.5" PATA or SATA hard disks. In the current (as of May 2010) incarnation, these are connected to the Mark5 unit backplane over 4 PATA buses (the SATA modules have a format conversion inside the module). Disks 0 and 1 are master and slave respectively on bus 0, 2 and 3 are M/S on bus 1, and so on. Disk 0 is the one closest to the front handle and disk 7 is the one nearest the back-plane connector. Data is written to the modules in a non-deterministic manner based on the performance of the individual disks.
Each disk is labeled with an 8-digit Volume Serial Number (VSN) which consists of three required parts: a owner-id string consisting only of letters, a type identifier (either
indicating PATA or SATA respectively) and a sequence number consisting only of numeric digits. This VSN is stored both electronically on each disk in the module in a reserved portion of the disks and on a paper label with bar-code that is scanned for inventory purposes at the stations and correlator. It is important to ensure that the paper label and the electronic label match. An extended VSN is sometimes used which adds two fields to the VSN (separated with slashed
) containing the approximate total size in GB and the maximum data record rate that the module can sustain.
The data on a Mark5 module is not stored in a normal filesystem. However, a directory listing with some information about the data contained on the module is supplied. The directory is stored on a special partition of the Mark5 unit that is duplicated on every drive allowing access to the directory even in the presence of disk failure.
About the streamstor card
The Mark5 series of data recorders use the Conduant streamstor card as a disk controller. Due to the non-standard nature of the recording, access to this device does not follow standard POSIX standards. Software wanting access to the streamstor card must go through a C++ API that is exposed through Conduant's Software Development Kit (SDK). Access to the new Mark5C
hardware (a 10 GbE
daughter board that sits on the streamstor card itself) requires SDK version 9.0 or greater. The SDK comes with a linux device driver (
), FPGA firmware for the streamstor card, run-time libraries (
) and include files.
It is important to understand that only one program can talk to the streamstor card at a time. To get around this, the
programs, act as servers. These two programs (the former for Mark5C
devices and the later for Mark5A
devices) take the responsibility for controlling the streamstor device and exposing an interface (using the VLBI Standard Interface format) to potentially several programs simultaneously. These two programs do not allow the full range of capabilities that the streamstor cards provide. It is hoped that
, supplies the full functionality required at a VLBI station for recording data and performing standard monitor and control. Since most problems get resolved at the correlator centers rather than stations, the burden on software is greater there. With the VLBA hardware correlator, the Mark5A
program was used when correlating data, but the
program was stopped when module conditioning was performed. The DiFX
software correlator is compiled against the streamstor libraries directly so does not make use of
functionality. A program called
runs on all of the correlator Mark5 units. This program periodically (every 10 seconds) opens the streamstor card, checks which modules are installed, and then closes the device again. Before opening the device it checks to see if any other process is making use of the streamstor card.
Currently, only 32-bit versions of the SDK exist, preventing installation on 64-bit versions of Linux. As of May 24, 2010, expectations are that this will change shortly.
While effort is made at all levels to maintain compatibility between various software and hardware components, occasionally there is a change that must break compbatibility.
The most prominant kind of compatibility breakage is between major versions of the streamstor SDK (i.e., moving from SDK version X.Y to X+1.0). Compatibility is maintained within minor version changes. The transition from SDK8 to SDK9 will be particularly significant as it changes the on-disk format as well as the software API. Fortunately, it will be possible to read a Mark5 module written with any SDK version with SDK9, but the reverse is not true. An interim period where correlators upgrade their Mark5 units but stations have not will require care when exchanging media. Most software that links directly with the streamstor run-time libraries will need minor changes to accommodate differences in data types.
The module directory is being enhanced with the advent of Mark5C
. Not only will the new directory contain much more detailed information, it will also allow more than the 1024 entries allowed on the current (as of May 2010) format.
Documentation of individual programs
Much of the software described here is either directly related to needs of the DiFX
software correlator when using Mark5 modules directly or are a spin-off (such as
which is based on a software correlator program called
). More detailed information for such programs may thus be found in the DiFX reference manual
Data Recording System (DRS)
This program accesses the streamstor card and extracts various information about a module. It also allows a new Volume Serial Number (VSN) to be set. It should be invoked as follows:
is the bank to look at (either
) and newVSN
is an optional argument that, if supplied, will cause the module VSN to be changed. Running with argument
will cause inline help to be displayed.
vsn B NRAO-234
This program is part of the mk5daemon package.
This program opens a Linux file containing recorded baseband data and attempts to decode it. This program does not directly access the streamstor card. The data file in question must first be read off the Mark5 unit, perhaps using
. Usage is as follows:
m5d filename format n
is the name of the file to read, format
is the data format (Note for RDBE PFB personalities this is usually
) and n
is the number of samples to decode. offset
can be used to skip into the file. A header describing the data and a matrix of numbers will be set to the screen. Run with no command line arguments to get inline help.
This program is found within the mark5access package.
This program is currently homeless! It was a portion of the Mark5A
software package. It, or a new equivalent of it, may get incorporated within the mk5daemon package.
Interacting with the Mark5 units at the VLBA DiFX correlator
Many Linux libraries these days come with a handy pkg-config file (ending in
) that allows Makefiles to easily adapt to changing versions and install locations. I (WFB) have made a
file which gets installed automatically when installing streamstor packages that I have rolled myself. As long as this file is installed in a standard directory or one pointed to by
the compile time flags that are required can be gotten with the command
pkg-config --cflags streamstor
. Likewise, link flags can be gotten with
pkg-congig --libs streamstor
streamstor-X.Y.tar.gz or streamstor-X.Y-1.i386.rpm
This is a repackaged version of Conduant's streamstor SDK. In this I've included the
file for ease in compiling downstream software. On RedHat
Enterprise Linux systems it is preferred to just install a .rpm file that I've made. If compiling by source, it installs using the autotools "configure, make, make install" command series. Whether installing from source or from .rpm, it is necessary to have root permission as the kernel driver needs to be installed in a system directory. By default I install things using