OLD STUFF - YMMV

You have a data delivery package and want to apply the included pipeline run to obtain a calibrated MS

  1. untar the package
  2. cd to the lowest level in the resulting tree containing the: calibration, log, product, qa and script directories, and make a new directory called raw
  3. cd into the raw directory and put the raw ASDMs there.
  4. make sure they are named blah.asdm.sdm if they didn't come that way
  5. cd ../script
  6. . /lustre/naasc/pipeline/pipeline_env_r31667_casa422_30986.sh
    (If you're using csh, I think its "source .....csh")
  7. casapy
  8. CASA> execfile("scriptForPI.py")

You have a PPR (perhaps from a data delivery package) and want to rerun the Cycle 2 operations pipeline (casa 4.2.2)

  1. make a directory somewhere you have write access called something like <SBNAME>_PL_rerun
    • make a working directory under that directory:
    • mkdir -p <SBNAME>_PL_rerun/SOUS_uid___/GOUS_uid___/MOUS_uid___/working
  2. copy the PPR and, if you want to use them, flagtemplate file(s) to the rerun working directory
  3. /lustre/naasc/rindebet/pipeline/scripts/calibPipeIF-NA.py --flag=<RERUN DIR>
    • use full path to your <SNAME>_PL_rerun directory for <RERUN DIR> without any quotes and no trailing slash
    • a new pipeline directory will be created in your pipeline/root/ directory in your lustre area corresponding to this rerun regardless of where you run the above command
  4. check casapy....log file for "WARN," "SEVERE" and "ERR" messages

You have a PPR (perhaps from a data delivery package) and want to run the Cycle 3 test pipeline (casa 4.3.0)

  1. you probably can't use your PPR because the pipeline recipe has changed, so follow the instructions below to run the Cycle 3 test pipeline from scratch either with a flagtemplate if you have one or without one if you don't.

You want to run the Cycle 2 production pipeline completely from scratch

  1. you need the member OUS status ID:
    • If you already have the ASDM file on disk, you can extract its OUS using the analysisUtils command au.getOUS, which can be run in any version of casa (where the uid here is the ASDM name):
      • CASA <2>: au.getOUS('uid___A002_X5b06c4_X5c9')
      • This will print: Out[2]: 'uid://A002/X5a9a13/X528'
    • If you don't have the ASDM, you can use the Project Tracker to find the OUS using these instructions:
      • Use project search to find and select the right project.
      • In the lower left window that has column headings "Entity" and "Status", click on the SG of interest, one-level below the "ObsUnitSet" folder. Its icon is a light blue stack of square clocks. (It is one level above where the icon is a single yellow square clock.)
      • In the window in the right, at the top, the "Status entity ID" is the OUS ID.
      • (This is different from the "Sched Block id" and the "Status entity id" that you would see if you clicked on the level below, with the yellow icon.)
  2. /lustre/naasc/rindebet/pipeline/scripts/calibPipeIF-NA.py --mous=MOUS_STATUS_ID
    • (no quotes on MOUS_STATUS_ID)
    • a new pipeline directory will be created in your pipeline/root/ directory in your lustre area corresponding to this rerun regardless of where you run the above command
  3. check casapy....log file for "WARN," "SEVERE" and "ERR" messages

You want to run the Cycle 3 test pipeline completely from scratch

  1. you need the member OUS status ID as above
  2. bash% /lustre/naasc/rindebet/pipeline/scripts/calibPipeIF-NA.py --test --mous=MOUS_STATUS_ID

You want to run the Cycle 2 pipeline from scratch but use an existing flagtemplate file or modify the PPR

  1. you need the member OUS status ID as above
  2. . /lustre/naasc/pipeline/pipeline_env_r31667_casa422_30986.sh
  3. pipelineMakeRequest MOUS_STATUS_ID intents_hifa.xml.r31667_casa422_30986 procedure_hifacal.xml.r31667_casa422_30986 false
  4. find the rawdata directory that it creates eg. /lustre/naasc/USER/pipeline/root/2012.1.00912.S_2013_01_28T18_23_10.196/SOUS_uid___A002_X5a9a13_X526/GOUS_uid___A002_X5a9a13_X527/MOUS_uid___A002_X5a9a13_X528/rawdata/
  5. modify the PPR in that dir if you want
  6. add your flagtemplate files to that dir
  7. /lustre/naasc/rindebet/pipeline/scripts/calibPipeIF-NA.py --flag=<RAWDIR>
    , where <RAWDIR> is that rawdir where you just put flagtemplate files.

You want to run the Cycle 3 test pipeline from scratch but use an existing flagtemplate file or modify the PPR

  1. you need the member OUS status ID as above
  2. . /lustre/naasc/pipeline/pipeline_env_r32164_casa430_31966.sh
  3. pipelineMakeRequest MOUS_STATUS_ID intents_hifa.xml procedure_hifacal.xml false
  4. find the rawdata directory that it creates eg. /lustre/naasc/USER/pipeline/root/2012.1.00912.S_2013_01_28T18_23_10.196/SOUS_uid___A002_X5a9a13_X526/GOUS_uid___A002_X5a9a13_X527/MOUS_uid___A002_X5a9a13_X528/rawdata/
  5. modify the PPR in that dir if you want
  6. add your flagtemplate files to that dir
  7. /lustre/naasc/rindebet/pipeline/scripts/calibPipeIF-NA.py --test  --flag=<RAWDIR>
    , where <RAWDIR> is that rawdir where you just put flagtemplate files.

You want to run the Cycle 2 pipeline but not the "fixes"

  1. you need the member OUS status ID as above
  2. . /lustre/naasc/pipeline/pipeline_env_r31667_casa422_30986.sh
  3. pipelineMakeRequest MOUS_STATUS_ID intents_hifa.xml.r31667_casa422_30986 procedure_hifacal.xml.r31667_casa422_30986 true
  4. find the rawdata directory that it creates eg. /lustre/naasc/USER/pipeline/root/2012.1.00912.S_2013_01_28T18_23_10.196/SOUS_uid___A002_X5a9a13_X526/GOUS_uid___A002_X5a9a13_X527/MOUS_uid___A002_X5a9a13_X528/working/
  5. modify the PPR in that dir if you want
  6. casapy
  7. CASA> eppr.executeppr("PPR.....xml",importonly=False)

You want to run the Cycle 3 test pipeline but not the "fixes" (e.g. fixForCSV2555 which breaks the Cycle 3 pipeline)

  1. you need the member OUS status ID as above
  2. . /lustre/naasc/pipeline/pipeline_env_r32164_casa430_31966.sh
  3. pipelineMakeRequest MOUS_STATUS_ID intents_hifa.xml procedure_hifacal.xml true
  4. find the rawdata directory that it creates eg. /lustre/naasc/USER/pipeline/root/2012.1.00912.S_2013_01_28T18_23_10.196/SOUS_uid___A002_X5a9a13_X526/GOUS_uid___A002_X5a9a13_X527/MOUS_uid___A002_X5a9a13_X528/working/
  5. modify the PPR in that dir if you want
  6. casapy
  7. CASA> eppr.executeppr("PPR.....xml",importonly=False)

Step 2. Modify the PPR (if necessary, e.g. to choose the reference antenna)

  • The pipeline doesn't use the same refant as the manual script generator. To promote comparison, we force the pipeline to use the refant chosen in the manual reduction path (by modifying the PPR).
  • If you want to modify the PPR, you can either edit it by hand, or you can use the automatic editing function that Todd wrote called modifyPPR:
    • bash_or_tcsh% cd to the 'working' directory containing the desired request file (PPR_uid___A002_XXXXXXX_XXXX.xml)
    • bash_or_tcsh% casapy (it can be any version)
    • CASA <2>: import analyzemscal as amc
    • CASA <3>: amc.modifyPPR(put your options here)
    • For example, if you have a multi-EID OUS, and you only want to calibrate one of them, and you want to use DV04 as the refant, then:
      • CASA <3>: amc.modifyPPR('PPR_uid___A002_X391d0b_X281.xml', asdm='X22a', refant='DV04')
      • This will write: PPR_uid___A002_X391d0b_X281_refant_DV04__X22a.xml, which you should reference in the eppr.executeppr() command in Step 3.

Step 4. Run analyzemscal

  • To generate a QA2-like report for the pipeline reduction that appears at the standard webpage you have to run a second casapy in a shell where you have NOT sourced the pipeline setup script. This doesn't have to be a specific version of casapy.
  • You do, however, have to have this line in your init.py, or execute it manually once starting casapy:
    • CASA <03>: sys.path.append('/users/jcrossle/DA_scripts/AIV/science/analysis_scripts/')
  • inside casapy,
    • CASA <04>: import analyzemscal as amc 
      CASA <05>: analyzemscal(asdm='uid___your_asdm', pipecaldir='/lustre/naasc/username/pipeline/root/your_project/....../working',webarea=True, webpagedir='Feb2014_heuristics')
  • If there is a corresponding manual reduction of this dataset, then be sure that you have the original products, including the qa2 subdirectory. If the latter does not exist, then you need to run es.generateQA2Report('your_uid.ms') before running amc.analyzemscal(). You will need to specify the manualcaldir parameter to point to the manual reduction directory (you do not need to have write permission there).
  • You can see a full list of all analyzemscal parameters in the online help: help(amc.analyzemscal)

Step 5. Flagging more data (or setting quasar flux densities) and re-running the pipeline

  • The pipeline still does not flag data (except lines in solar system objects used as flux calibrators), it only flags cal tables, so there will very likely be data that you need to flag. These will show up as outliers in the calibrated data plots crated by analyzemscal
  • If you would like to re-run the pipeline and have it flag data at the beginning, follow the instructions for creating a flagdata text file at the bottom of the page on modifyPPR.
  • Then remove the model, restore the flags, and remove the cal tables (but keep the .ms) by running:
    • delmod('your_uid')
    • flagmanager('your_uid', mode='restore', versionname='Original')
    • rm -rf your_uid.ms.*
  • Now you can re-execute the pipeline with eppr.executeppr() and it will skip the import step, leaving any modifications to the ms in place (like fixplanets).
  • Similarly, if your dataset uses a quasar as the flux calibrator, and you want to change the flux, follow the instructions below for editing the flux.csv file, then remove the model, restore the flags, remove the cal tables and re-execute the pipeline.
    • During conversion of the ASDM to MS fluxes in the source file are loaded into the pipeline context and saved to a file on disk. The fluxes [I, Q, U, V] are defined as a function of MS, field name, and spw. The name of the disk file is 'flux.csv'. It is a text file where each line contains comma delimited fields specifying the MS name, field name, spw, I, Q, U, and V. The first line in this file must not be changed but the flux values themselves may be edited or and new lines may be created. On subsequent runs of the pipeline (which start from the MS not the ASDM) the contents of this file will supersede the contents of the ASDM Source table.

This topic: ALMA > WebHome > NAASC > HowToRunThePipeline > OldPipelineInstructions
Topic revision: 2015-08-21, RemyIndebetouw
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback