CASA Build Notes

Status: DEPRECATED - see How To Build CASA Using Build Tool Wrappers.

Environment Setup - IMPORTANT

Common Environment variables

These are defined in ~srankin/.profile.srankin. This is kept in sync across test systems.

name	value
HOST_CPU_CORES	Unless you know better, set this to "2".

These are build specific

CASA_BRANCH	What ever branch you are building [trunk, branches/test, branches/stable-YYYY-MM, etc].
workDir	Where ever you chose to keep your CASA code.
CASA_BUILD_TYPE	One of Debug, Release, RelWithDebInfo or, MinSizeRel.
CASAPATH	Set to "$workDir $CASA_ARCH".

OS X 10.7

Test build on casabtm1 after a clean install of OS X 10.7, installation of TBD, and OS updates from Apple.

Environment variables:

export PATH="/opt/casa/darwin11/bin:/usr/local/casa/bin:/usr/bin:/bin:/usr/local/bin:/usr/X11/bin:/usr/texbin:/usr/local/bin:/opt/local/bin"
export CASA_ARCH="darwin64"
export DYLD_FALLBACK_LIBRARY_PATH="${workDir}/${CASA_ARCH}/lib:/opt/casa/darwin11/lib:/opt/casa/darwin11/qwt-5.2.3-svn/lib"
export DYLD_FRAMEWORK_PATH="/opt/casa/darwin11/Library/Frameworks"

RHEL 5.6 32 bit

Test build on kusanagi on a recent NRAO RHEL 5.6 install, plus tweaking for ALMA and Redhat updates up to 2011-09-23. Missing CASA development RPMs added as needed.

Environment variables:

export CASA_ARCH=linux_gnu

RHEL 5.7 64 bit

Test build on kirara on an old NRAO RHEL 5.6 install updated to RHEL 5.7, plus tweaking for CASA and Redhat updates up to 2011-09-23. CASA development RPMs added some time ago.

Environment variables:

export CASA_ARCH=linux64

End to End Build

Get code from Subversion

cd <where ever you want to keep your CASA source>
export workDir=`pwd`
svn co https://svn.cv.nrao.edu/svn/casa/$CASA_BRANCH $workDir

Note: If you check out trunk, you will also get some 3rd party libraries (aatm, pypackages, and wcslib). Do not commit changes to these third party libraries. These will be moved out of trunk later.

These third party libraries are not included in any branches. Don't bother deleting these directories from your checked out code. They will just be restored at the next Subversion update.

Add a link to casa data

cd $workDir
ln -s [path/to/a/copy/of/svn.cv.nrao.edu/svn/casa-data/trunk] data

Build casacore

Required.

Configure

1. cd $workDir/casacore; mkdir build; cd build
         

2. Run cmake as required for the platform

OS X 10.7

cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DDATA_DIR=/home/casa/data -DUSE_THREADS=ON -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DLLVMCOMPILER=1&> cmake.log || cat cmake.log

RHEL 5

cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DDATA_DIR=/home/casa/data -DUSE_THREADS=ON -DCFITSIO_INCLUDE_DIR=/usr/include/cfitsio &> cmake.log || cat cmake.log

Notes:

1. -DDATA_DIR=/home/casa/data

   is only used by the Google code version of casacore. It is ignored by the CASA version of casacore. I leave it in the instructions to ensure I build CASA casacore and Google casacore consistantly.

1. cmake messages will be different on different systems. Look for the following at the end of the cmake messages:

-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/casacore/build

1. You will see messages like, "-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS)". This refers to an optional library not used in NRAO CASA builds.

1. The option "-DCMAKE_INSTALL_PREFIX=$orkDir/$CASA_ARCH" is required. casacore defaults to installing in /usr/local/.

Compile, Test, and Install

make -j $HOST_CPU_CORES &> compileLink.log || cat  compileLink.log | strings | grep Error | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking

If you see any errors, stop and investigate.

Note:On Mac, I have been seeing binary noise in logs. Running strings on the noisey log to produce the clean log simplifies checking for errors.

make test &> test.log || cat test.log | strings | grep Failed test.log

If this is a new build and you have not manually set CASAPATH, or if $workDir/data does not exist, expect to see ~53 failures.This is because the CASA CMake configuration (or possibly test code) depends on CASAPATH to find test data, not the casacore -DDATA_DIR option.See CAS-3956.

If CASAPATH was set before before building casacore, expect to see ~7 errors.This is because not all developers who change code in casacore bother to run "make test" and fix broken tests.I will start to report these test failures as Jira tickets in the near future.

make install &> install.log || grep Error install.log  | grep -v Installing | grep -v Up-to-date | grep -v tFITSErrorImage 

If you see errors, stop and investigate.

Build code

Required.

Configure

1. cd $workDir/code; mkdir build; cd build
         

2. Run cmake as required for the platform

OS X 10.7

cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DUSE_ALMAWVR=True -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DLLVMCOMPILER=1 -DINTERACTIVE_ITERATION=0 &> cmake.log || cat cmake.log 

RHEL 5

cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DUSE_ALMAWVR=True -DINTERACTIVE_ITERATION=1 &> cmake.log || cat cmake.log 

Notes:

1. -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran is required on sneffels.aoc.nrao.edu, which has an Intel compiler installed, but has license key issues.
2. cmake messages will be different on different systems. Look for the following at the end of the cmake messages:

-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/code/build

Compile, Test, and Install

make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error compileLink.log | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking

If you see errors, stop and investigate.

make test &> test.log || cat test.log | strings | grep Failed test.log

Expect to see ~112 failures.The CASA code CMake configuration needs work.See CAS-3955.

make install &> install.log || grep Error install.log  | grep -v Installing | grep -v Up-to-date | grep -v tFITSErrorImage 

If you see errors, stop and investigate.

Build gcwrap

Required.

Configure

1. source $workDir/casainit.sh; cd $workDir/gcwrap; mkdir build; cd build
         

2. Run cmake as required for the platform

OS X 10.7

cmake .. -Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DLLVMCOMPILER=1 -DINTERACTIVE_ITERATION=0 &> cmake.log || cat cmake.log

RHEL 5

cmake .. -Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DPYTHON_LIBNAME=2.7 -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DINTERACTIVE_ITERATION=1 &> cmake.log || cat cmake.log

Notes:

1. cmake messages will be different on different systems. Look for the following at the end of the cmake messages:

-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/asap/build

1. The options "-Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH" are required.

Compile, Test, and Install

make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error compileLink.log | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking

If you see errors, stop and investigate.

make test &> test.log || cat test.log | strings | grep Failed test.log

At this time, "make test" does not run any tests.I leave this step in the instructions to pick up these tests as soon as they are available.

make install &> install.log || grep Error install.log  | grep -v Installing | grep -v Up-to-date 

If you see errors, stop and investigate.

Build asap

Optional, but this is so quick you should.

Configure

1. cd $workDir/asap; mkdir build; cd build
         

2. Run cmake as required for the platform

OS X 10.7

cmake .. -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DFORTRAN_LIBRARIES=/usr/lib/gcc/i686-apple-darwin11/4.2.1/x86_64/libgfortran.a -DLLVMCOMPILER=1 &> cmake.log || cat cmake.log

RHEL 5

cmake .. -Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DPYTHON_LIBNAME=2.7 -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE &> cmake.log || cat cmake.log

Notes:

1. cmake messages will be different on different systems. Look for the following at the end of the cmake messages:

-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/asap/build

1. The options "-Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH" are required. asap defaults to installing in /usr/local/.

Compile, Test, and Install

make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error compileLink.log | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking

If you see errors, stop and investigate.

make test &> test.log || cat test.log | strings | grep Failed test.log

According to Kana, ASAP developers are discussing adding automated tests.I leave this step in the instructions to pick up these tests as soon as they are available.

make install &> install.log || grep Error install.log  | grep -v Installing | grep -v Up-to-date | grep -v tFITSErrorImage 

If you see errors, stop and investigate.

Incremental builds after updates from Subversion

Rebuild, test, and install all components

To update your checked out code from Subversion and build any changes:

cd $workDir && svn update

for build_component in casacore code gcwrap asap
do 
    echo $build_component
    cd ${workDir}/${build_component}/build
    make -j $HOST_CPU_CORES 2>&1 | strings > compileLink.log
    make test 2>&1 | strings > test.log
    make install &> install.log
done 
cd $workDir
echo "Checking compileLink.log"
grep Error */build/compileLink.log | grep -v "Building" | grep -v "Built target" 
echo "Checking test.log"
grep Error */build/test.log
echo "Checking install.log"
grep Error */build/install.log | grep -v "Built target" | grep -v "Up-to-date:"

Some errors are expected in the casacore/build/test.log and code/build/test.log files. Check them manually.

Test Builds

Run Functional Tests (runUnitTest.py)

source ${workDir}/casainit.sh
revision=`cd $workDir; svn info | grep "Last Changed Rev" | awk '{print $4}'`
mkdir -p ${workDir}/testRuns/r${revision}/functionalTests
cd ${workDir}/testRuns/r${revision}/functionalTests
functional_test_list=${workDir}/${CASA_ARCH}/python/2.7/tests/unittests_list.txt
functional_test_names=`cat $functional_test_list | grep -v \# | awk '{print $1}'`
for test in $functional_test_names
do
    echo ${test}
    time casa --nogui --log2term -c ${workDir}/${CASA_ARCH}/python/2.7/runUnitTest.py ${test} &> ${test}.log
    if [ $? != 0 ]; then echo ${test} >> nonZeroExit.log; fi
done
echo
if [ -e nonZeroExit.log ]; then cat nonZeroExit.log; fi
echo
grep FAILED test*log | grep -v FIT 
echo
grep "Too many open files" test*log

Run "*_regression.py"

source ${workDir}/casainit.sh
revision=`cd $workDir; svn info | grep "Last Changed Rev" | awk '{print $4}'`
mkdir -p ${workDir}/testRuns/r${revision}/regression
cd ${workDir}/testRuns/r${revision}/regression
regression_test_list=${workDir}/${CASA_ARCH}/python/2.7/regressions/admin/tests_list.txt 
regression_test_names=`cat $regression_test_list | grep -v \# | grep -v unit_tests | awk '{print $1}'`
for test in $regression_test_names
do
    for f in `ls ${workDir}/${CASA_ARCH}/python/2.7/regressions/tests/${test}.py 2>&1 | grep -v "No such file or directory"`
    do
        echo $test
        cp $f .
        time casa --nogui --log2term -c "execfile('$test.py');run(True)" &> ${test}.log
        if [ $? != 0 ]; then echo ${test} >> nonZeroExit.log; fi
    done
    for f in `ls ${workDir}/${CASA_ARCH}/python/2.7/regressions/${test}.py 2>&1 | grep -v "No such file or directory"`
    do
        echo $test
        cp $f .
        time casa --nogui --log2term -c ${test} &> ${test}.log
        if [ $? != 0 ]; then echo ${test} >> nonZeroExit.log; fi
    done
done
echo
if [ -e nonZeroExit.log ]; then cat nonZeroExit.log; fi
echo 
# NOTE: needs work.
grep Regression *_regression.py.log | grep -i failed 
echo
grep "Press enter" *log
echo
grep "Too many open files" test*log

OLD Results

Step	OS X		RHEL 5
	10.6	10.7	32 bit	64 bit
	sneffels	casabtm1	kusanagi	kirara	rishi	warp
configure casacore	S	x	S	S	S	S
build casacore	S	x	S	S	S	S
test casacore	8 of 466 fail	x	8 of 466 fail	8 of 466 fail	8 of 466 fail	8 of 466 fail
configure code	S	x	S	S	S	S
build code	S	x	S	S	S	x
test code	n of 52 fail	x	52 of 52 fail	52 of 52 fail	52 of 52 fail	52 of 52 fail
configure asap	S	x	S	S	S	S
build asap	x	x	S	S	S	S
test asap	x	x	no tests?	no tests?	no tests?	no tests?

key:

• x: not attempted yet
• S: succeeded
• F: failed

F - 1:

-- Looking for ATM header ATMEnumerations.h -- /opt/casa/core2-apple-darwin8/3rd-party/include/ATM/ATMEnumerations.h
-- Looking for ATM header ATMVersion.h
-- Looking for ATM header ATMVersion.h -- NOT FOUND
CMake Error at install/config.cmake:762 (message):
  ATM could not be found.  Please check!
Call Stack (most recent call first):
  CMakeLists.txt:756 (casa_find)

Fixed by building aatm from source (ignoring test build error) and installing into ../darwin.

F - 2:

CMake Error at install/config.cmake:603 (message):
  Could compile and link to, but not run QWT:
Call Stack (most recent call first):
  CMakeLists.txt:862 (casa_find)

Having trouble debugging. cmake deletes test programs after running. The --debug-trycompile option is suppose to save test programs, but I get very different results with this option.

Worked around by adding /opt/casa/core2-apple-darwin8/3rd-party/qwt-5.2.0/lib/ to DYLD_LIBRARY_PATH

-- ScottRankin - 2011-02-10