Notes from Eric Johnson 4/13/2009

~3TB Total data

1. calibrate data Raw data pulled from NRAO archives calibrate produce "calibrated data" save to "master store" scp/sftp used to place data in "master store" meta-data associated with calibration - text file from calibration - postscript figures

2. produce "image cube" pull "calibrated data" from "master store" http/ftp? used to pull data from "master store" combining multiple observing segments into a single observation reduce to produce "image cube" push back to "master store" scp/sftp used to place data in "master store" meta-data associated with "image cube" - 2d moment maps - some intermediate steps - figures - text notes

3. basic standard reduction pull "image cube" + meta data from "master store" http/ftp? used to pull data from "master store" produce figures/text/stuff stuff suitable for publications

4. researcher specific reduction pull "image cube" + meta-data http/ftp? used to pull data from "master store" produce figures/text/stuff stuff suitable for publications

Current directory structure, and proposed structure for "master store"

galaxy OPT NEARIR SPITZER VLA VLA data calab Calabrated data figures-XN (X=array N=figure number) notes postcal "image cubes" prod Products (images/text) produced from "image cubes" obs original observe files

Issues:

- Getting data to NRAO site - sending tape/usb disk via mail/fedex seemed to be the preferred method

- Authentication for http/ftp pull operations - per user username/password or shared single username/password?

- Authentication for scp/sftp place operation - ssh key based, not password (probably) May be some user education needed here.

- Mirroring "master store" - FIU available and decent sized network pipeage to UVA/NRAO. - only for "backup purposes, not to be used as secondary pull location - or should it be? - rsync over ssh - how often? - might be pertinent to secondary pull question

- Metadata/data management systems - Software revision control systems not really appropriate (IMHO) - Web based content management systems might be?

-- DavidHalstead - 2009-04-13
Topic revision: r3 - 2009-10-19, CarolynWhite
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback