Mark5C information

About Mark5 modules

A Mark5 module is a metal enclosure the size of a toaster that contains eight 3.5" PATA or SATA hard disks. In the current (as of May 2010) incarnation, these are connected to the Mark5 unit backplane over 4 PATA buses (the SATA modules have a format conversion inside the module). Disks 0 and 1 are master and slave respectively on bus 0, 2 and 3 are M/S on bus 1, and so on. Disk 0 is the one closest to the front handle and disk 7 is the one nearest the back-plane connector. Data is written to the modules in a non-deterministic manner based on the performance of the individual disks.

Each disk is labeled with an 8-digit Volume Serial Number (VSN) which consists of three required parts: a owner-id string consisting only of letters, a type identifier (either - or + indicating PATA or SATA respectively) and a sequence number consisting only of numeric digits. This VSN is stored both electronically on each disk in the module in a reserved portion of the disks and on a paper label with bar-code that is scanned for inventory purposes at the stations and correlator. It is important to ensure that the paper label and the electronic label match. An extended VSN is sometimes used which adds two fields to the VSN (separated with slashed / ) containing the approximate total size in GB and the maximum data record rate that the module can sustain.

The data on a Mark5 module is not stored in a normal filesystem. However, a directory listing with some information about the data contained on the module is supplied. The directory is stored on a special partition of the Mark5 unit that is duplicated on every drive allowing access to the directory even in the presence of disk failure.

About the streamstor card

The Mark5 series of data recorders use the Conduant streamstor card as a disk controller. Due to the non-standard nature of the recording, access to this device does not follow standard POSIX standards. Software wanting access to the streamstor card must go through a C++ API that is exposed through Conduant's Software Development Kit (SDK). Access to the new Mark5C hardware (a 10 GbE daughter board that sits on the streamstor card itself) requires SDK version 9.0 or greater. The SDK comes with a linux device driver ( windrvr6.ko ), FPGA firmware for the streamstor card, run-time libraries ( libwdapiXXXX.so and libssapi.a ) and include files.

It is important to understand that only one program can talk to the streamstor card at a time. To get around this, the DRS and Mark5A programs, act as servers. These two programs (the former for Mark5C devices and the later for Mark5A devices) take the responsibility for controlling the streamstor device and exposing an interface (using the VLBI Standard Interface format) to potentially several programs simultaneously. These two programs do not allow the full range of capabilities that the streamstor cards provide. It is hoped that DRS, like Mark5A, supplies the full functionality required at a VLBI station for recording data and performing standard monitor and control. Since most problems get resolved at the correlator centers rather than stations, the burden on software is greater there. With the VLBA hardware correlator, the Mark5A program was used when correlating data, but the Mark5A program was stopped when module conditioning was performed. The DiFX software correlator is compiled against the streamstor libraries directly so does not make use of Mark5A or DRS functionality. A program called mk5daemon runs on all of the correlator Mark5 units. This program periodically (every 10 seconds) opens the streamstor card, checks which modules are installed, and then closes the device again. Before opening the device it checks to see if any other process is making use of the streamstor card.

Currently, only 32-bit versions of the SDK exist, preventing installation on 64-bit versions of Linux. As of May 24, 2010, expectations are that this will change shortly.

Compatibility concerns

While effort is made at all levels to maintain compatibility between various software and hardware components, occasionally there is a change that must break compbatibility.

SDK changes

The most prominant kind of compatibility breakage is between major versions of the streamstor SDK (i.e., moving from SDK version X.Y to X+1.0). Compatibility is maintained within minor version changes. The transition from SDK8 to SDK9 will be particularly significant as it changes the on-disk format as well as the software API. Fortunately, it will be possible to read a Mark5 module written with any SDK version with SDK9, but the reverse is not true. An interim period where correlators upgrade their Mark5 units but stations have not will require care when exchanging media. Most software that links directly with the streamstor run-time libraries will need minor changes to accommodate differences in data types.

Directory changes

The module directory is being enhanced with the advent of Mark5C. Not only will the new directory contain much more detailed information, it will also allow more than the 1024 entries allowed on the current (as of May 2010) format.

Documentation of individual programs

Much of the software described here is either directly related to needs of the DiFX software correlator when using Mark5 modules directly or are a spin-off (such as vex2script which is based on a software correlator program called vex2difx ). More detailed information for such programs may thus be found in the DiFX reference manual.

Data Recording System (DRS)

vsn

This program accesses the streamstor card and extracts various information about a module. It also allows a new Volume Serial Number (VSN) to be set. It should be invoked as follows:

vsn bank [ newVSN ]

where bank is the bank to look at (either A or B ) and newVSN is an optional argument that, if supplied, will cause the module VSN to be changed. Running with argument -h will cause inline help to be displayed.

Example 1: vsn A

Example 2: vsn B NRAO-234

This program is part of the mk5daemon package.

testmod

mk5dir

mk5cp

m5d

This program opens a Linux file containing recorded baseband data and attempts to decode it. This program does not directly access the streamstor card. The data file in question must first be read off the Mark5 unit, perhaps using mk5cp . Usage is as follows:

m5d filename format n [ offset ]

where filename is the name of the file to read, format is the data format (Note for RDBE PFB personalities this is usually Mark5B-2048-16-2 ) and n is the number of samples to decode. offset can be used to skip into the file. A header describing the data and a matrix of numbers will be set to the screen. Run with no command line arguments to get inline help.

Example: m5d testdata.m5b Mark5B-2048-16-2 100

This program is found within the mark5access package.

m5spec

m5fold

vex2script

SSErase

This program is currently homeless! It was a portion of the Mark5A software package. It, or a new equivalent of it, may get incorporated within the mk5daemon package.

mib_mark5c

Interacting with the Mark5 units at the VLBA DiFX correlator

fusemk5

Building software

Configuration

Many Linux libraries these days come with a handy pkg-config file (ending in .pc ) that allows Makefiles to easily adapt to changing versions and install locations. I (WFB) have made a streamstor.pc file which gets installed automatically when installing streamstor packages that I have rolled myself. As long as this file is installed in a standard directory or one pointed to by PKG_CONFIG_PATH the compile time flags that are required can be gotten with the command pkg-config --cflags streamstor. Likewise, link flags can be gotten with pkg-congig --libs streamstor .

streamstor-X.Y.tar.gz or streamstor-X.Y-1.i386.rpm

This is a repackaged version of Conduant's streamstor SDK. In this I've included the .pc file for ease in compiling downstream software. On RedHat Enterprise Linux systems it is preferred to just install a .rpm file that I've made. If compiling by source, it installs using the autotools "configure, make, make install" command series. Whether installing from source or from .rpm, it is necessary to have root permission as the kernel driver needs to be installed in a system directory. By default I install things using prefix=/usr .

-- WalterBrisken - 2009-11-20
Topic revision: r3 - 2010-05-24, WalterBrisken
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback