Metadata repackaging

VLBA and GBT switched power and xcube pulse cal data are stored in compressed text files. A new file is created whenever a project starts, ends, or encounters midnight. These files are transferred to /home/vlba/metadata/switchedpower and /home/vlba/metadata/xcube_pcal within minutes of the file completion making them available for post-observation analysis. To minimize the number of files and to better archive them, a program repackagemetadata is run daily to merge all files from one site from one day into a single file per day. The resultant file is then placed in a subdirectory station/year , such as, br/2020 .

Some documentation on the file formats can be found in VLBA Sensitivity Upgrade Memo 44.

repackagemetadata is part of the rspr package, which is stored in NRAO's SVN Repository.

The crontab entries for this service should look like:

1 3 * * * /usr/bin/python3 /users/vlbamon/vlba/bin/repackagemetadata cron /home/vlba/metadata/switchedpower/ br,fd,gb,hn,kp,la,mk,nl,ov,pt,sc,vlb1 35 5 > /dev/null

1 4 * * * /usr/bin/python3 /users/vlbamon/vlba/bin/repackagemetadata cron /home/vlba/metadata/xcube_pcal/ br,fd,gb,hn,kp,la,mk,nl,ov,pt,sc,vlb1 35 5 > /dev/null

These two entries run at 3:01AM and 4:01AM local time, respectively. Each one looks at data between 35 and 31 days old. This keeps the most recent month's data untouched and allows a few days of reattempts if for some reason the cron job either failed or didn't run. On a typical day each script takes about 5 to 10 minutes to execute. The cron in the crontab entries above invokes a specific operational mode of repackagemetadata .

-- WalterBrisken - 2021-04-18
Topic revision: r3 - 2021-04-23, WalterBrisken
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback