Data services pipeline page

Pipeline presentations from 1-3 May 2012

Lindsay's talk on infrastructure: PipelineInfrastructure.pdf

Setting up your environment for running the pipeline (May 2012)

Multivac21/22 are currently set up for the pipeline. Multivac22 is set up as the 32-bit machine to deal with the ingest (get data out of the archive and onto Lustre - pipelineMakeRequest command).

Multivac21 has the 64-bit architecture needed to actually run the CASA part of the pipeline

Bash:

From Dan:

If you want to set up your profile to something similar as pipeproc, try this. But I am not 100% certain this will work due to permission issues. If it doesn't work please use pipeproc for the meantime and I'll work on this with you tomorrow. Your .profile.personal should have the following:

export CASA_ROOT=/alma/ACS-current/casapy
export ACSROOT=/alma/ACS-current/ACSSW
export CASAPATH=/alma/ACS-current/casapy
export SCIPIPE_ROOTDIR=/export/lustre/pipeline/root
export SCIPIPE_LOGDIR=/export/lustre/pipeline/logs
export SCIPIPE_SCRIPTDIR=/export/lustre/pipeline/scripts
export SCIPIPE_HEURISTICS=/export/lustre/pipeline/PIPELINE/Heuristics/src
if [ `hostname` = "multivac22" ]; then
source /alma/ACS-current/ACSSW/config/.acs/.bash_profile.acs
fi
if [ `hostname` = "multivac21" ]; then
export PATH=/usr/lib64/casapy/bin:$PATH
export MATPLOTLIBRC=$HOME/.matplotlib
fi

You will also need to add the line:

execute sys.path.insert(0, os.path.expandvars("$SCIPIPE_HEURISTICS"))

To your ~/.casa/ipython/ipythonrc . And add

backend : Agg
interactive : False

to your ~/.matplotlib/matplotlibrc file.

tcsh (CASA part only, ACS requires a bash environment, so change to "bash" on the command line and do above)

(from Amy)

setenv CASA_ROOT /alma/ACS-current/casapy
setenv ACSROOT /alma/ACS-current/ACSSW
setenv CASAPATH /alma/ACS-current/casapy
setenv SCIPIPE_ROOTDIR /export/lustre/pipeline/root
setenv SCIPIPE_LOGDIR /export/lustre/pipeline/logs
setenv SCIPIPE_SCRIPTDIR /export/lustre/pipeline/scripts
setenv SCIPIPE_HEURISTICS /export/lustre/pipeline/PIPELINE/Heuristics/src
if ( `hostname` = "multivac21" ) then
setenv PATH=/usr/lib64/casapy/bin:$PATH
setenv MATPLOTLIBRC=$HOME/.matplotlib
endif

setenv CASA_ROOT /alma/ACS-current/casapy
setenv ACSROOT /alma/ACS-current/ACSSW
setenv CASAPATH /alma/ACS-current/casapy
setenv SCIPIPE_ROOTDIR /export/lustre/pipeline/root
setenv SCIPIPE_LOGDIR /export/lustre/pipeline/logs
setenv SCIPIPE_SCRIPTDIR /export/lustre/pipeline/scripts
setenv SCIPIPE_HEURISTICS /export/lustre/pipeline/PIPELINE/Heuristics/src
if ( `hostname` = "multivac21" ) then
setenv PATH=/usr/lib64/casapy/bin:$PATH
setenv MATPLOTLIBRC=$HOME/.matplotlib
endif

Historical information

Pipeline overview given at the NAASC, Nov 2010: pipe.overview.naasc.oct2010.pdf

Link to wiki for F2F meeting 2010

Heuristics version 1- email from Dirk Muders and Lindsey Davis 2010-09-07

At this time the ALMA Pipeline Group is pleased to announce the first
internal release of the ALMA Pipeline Heuristics scripts to the ALMA staff.
The heuristics scripts will be used by the ALMA Pipeline System to reduce
data taken in standard observing modes and to provide science data
products that can be used for further analysis. The current scripts
support single dish, single field interferometry, and pointed mosaic
observations. The output of the scripts has been tested by external
users during several user tests. See

http://almasw.hq.eso.org/almasw/bin/view/PIPELINE/PipelineHeuristics

for test results.

The single dish scripts are currently able to handle ALMA data in ASDM
format
and data from other observatories in ASAP format. To date the interferometry
scripts have been developed using VLA and IRAM PdB and some SMA data. We
have
begun to reduce early ALMA interferometry data sets, but this capability is
still very experimental and requires special scripts.

The heuristics task scripts, our standard reduction scripts and some of our
test data are available at

http://alma-heuristics.mpifr-bonn.mpg.de/releases/r1

For task documentation please visit

http://almasw.hq.eso.org/almasw/bin/view/PIPELINE/PipelineHeuristicsTaskInterface

This release has been developed and tested for Casa 3.0.2 build 11761. There
are still known limitations in the heuristics scripts and some efficiency
and resource usage issues that will be addressed in future releases.

For questions or problems in the current release or if you wish to reduce
ALMA data, please contact either Lindsey Davis (ldavis@nrao.edu) or
Dirk Muders (dmuders@mpifr-bonn.mpg.de).

-- MarkLacy - 2010-10-25

• Lindsay's talk on infrastructure (May 2nd 2012): PipelineInfrastructure.pdf