CASA Build Notes
Status: DEPRECATED - see
How To Build CASA Using Build Tool Wrappers.
Environment Setup - IMPORTANT
Common Environment variables
These are defined in ~srankin/.profile.srankin. This is kept in sync across test systems.
name |
value |
HOST_CPU_CORES |
Unless you know better, set this to "2". |
These are build specific
CASA_BRANCH |
What ever branch you are building [trunk, branches/test, branches/stable-YYYY-MM, etc]. |
workDir |
Where ever you chose to keep your CASA code. |
CASA_BUILD_TYPE |
One of Debug, Release, RelWithDebInfo or, MinSizeRel. |
CASAPATH |
Set to "$workDir $CASA_ARCH". |
OS X 10.7
Test build on casabtm1 after a clean install of OS X 10.7, installation of TBD, and OS updates from Apple.
Environment variables:
export PATH="/opt/casa/darwin11/bin:/usr/local/casa/bin:/usr/bin:/bin:/usr/local/bin:/usr/X11/bin:/usr/texbin:/usr/local/bin:/opt/local/bin"
export CASA_ARCH="darwin64"
export DYLD_FALLBACK_LIBRARY_PATH="${workDir}/${CASA_ARCH}/lib:/opt/casa/darwin11/lib:/opt/casa/darwin11/qwt-5.2.3-svn/lib"
export DYLD_FRAMEWORK_PATH="/opt/casa/darwin11/Library/Frameworks"
RHEL 5.6 32 bit
Test build on kusanagi on a recent NRAO RHEL 5.6 install, plus tweaking for ALMA and Redhat updates up to 2011-09-23. Missing CASA development RPMs added as needed.
Environment variables:
export CASA_ARCH=linux_gnu
RHEL 5.7 64 bit
Test build on kirara on an old NRAO RHEL 5.6 install updated to RHEL 5.7, plus tweaking for CASA and Redhat updates up to 2011-09-23. CASA development RPMs added some time ago.
Environment variables:
export CASA_ARCH=linux64
End to End Build
Get code from Subversion
cd <where ever you want to keep your CASA source>
export workDir=`pwd`
svn co https://svn.cv.nrao.edu/svn/casa/$CASA_BRANCH $workDir
Note: If you check out trunk, you will also get some 3rd party libraries (aatm, pypackages, and wcslib). Do not commit changes to these third party libraries. These will be moved out of trunk later.
These third party libraries are not included in any branches. Don't bother deleting these directories from your checked out code. They will just be restored at the next Subversion update.
Add a link to casa data
cd $workDir
ln -s [path/to/a/copy/of/svn.cv.nrao.edu/svn/casa-data/trunk] data
Build casacore
Required.
-
cd $workDir/casacore; mkdir build; cd build
- Run cmake as required for the platform
- OS X 10.7
-
cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DDATA_DIR=/home/casa/data -DUSE_THREADS=ON -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DLLVMCOMPILER=1&> cmake.log || cat cmake.log
- RHEL 5
-
cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DDATA_DIR=/home/casa/data -DUSE_THREADS=ON -DCFITSIO_INCLUDE_DIR=/usr/include/cfitsio &> cmake.log || cat cmake.log
Notes:
-
-DDATA_DIR=/home/casa/data
is only used by the Google code version of casacore. It is ignored by the CASA version of casacore. I leave it in the instructions to ensure I build CASA casacore and Google casacore consistantly.
- cmake messages will be different on different systems. Look for the following at the end of the cmake messages:
-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/casacore/build
- You will see messages like, "-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS)". This refers to an optional library not used in NRAO CASA builds.
- The option "-DCMAKE_INSTALL_PREFIX=$orkDir/$CASA_ARCH" is required. casacore defaults to installing in /usr/local/.
Compile, Test, and Install
make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking
If you see any errors, stop and investigate.
Note:On Mac, I have been seeing binary noise in logs. Running strings on the noisey log to produce the clean log simplifies checking for errors.
make test &> test.log || cat test.log | strings | grep Failed test.log
If this is a new build and you have not manually set CASAPATH, or if $workDir/data does not exist, expect to see ~53 failures.This is because the CASA CMake configuration (or possibly test code) depends on CASAPATH to find test data, not the casacore -DDATA_DIR option.See
CAS-3956.
If CASAPATH was set before before building casacore, expect to see ~7 errors.This is because not all developers who change code in casacore bother to run "make test" and fix broken tests.I will start to report these test failures as Jira tickets in the near future.
make install &> install.log || grep Error install.log | grep -v Installing | grep -v Up-to-date | grep -v tFITSErrorImage
If you see errors, stop and investigate.
Build code
Required.
-
cd $workDir/code; mkdir build; cd build
- Run cmake as required for the platform
- OS X 10.7
-
cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DUSE_ALMAWVR=True -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DLLVMCOMPILER=1 -DINTERACTIVE_ITERATION=0 &> cmake.log || cat cmake.log
- RHEL 5
-
cmake .. -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DUSE_ALMAWVR=True -DINTERACTIVE_ITERATION=1 &> cmake.log || cat cmake.log
Notes:
- -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran is required on sneffels.aoc.nrao.edu, which has an Intel compiler installed, but has license key issues.
- cmake messages will be different on different systems. Look for the following at the end of the cmake messages:
-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/code/build
Compile, Test, and Install
make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error compileLink.log | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking
If you see errors, stop and investigate.
make test &> test.log || cat test.log | strings | grep Failed test.log
Expect to see ~112 failures.The CASA code CMake configuration needs work.See
CAS-3955.
make install &> install.log || grep Error install.log | grep -v Installing | grep -v Up-to-date | grep -v tFITSErrorImage
If you see errors, stop and investigate.
Build gcwrap
Required.
-
source $workDir/casainit.sh; cd $workDir/gcwrap; mkdir build; cd build
- Run cmake as required for the platform
- OS X 10.7
-
cmake .. -Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DLLVMCOMPILER=1 -DINTERACTIVE_ITERATION=0 &> cmake.log || cat cmake.log
- RHEL 5
-
cmake .. -Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DPYTHON_LIBNAME=2.7 -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DINTERACTIVE_ITERATION=1 &> cmake.log || cat cmake.log
Notes:
- cmake messages will be different on different systems. Look for the following at the end of the cmake messages:
-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/asap/build
- The options "-Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH" are required.
Compile, Test, and Install
make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error compileLink.log | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking
If you see errors, stop and investigate.
make test &> test.log || cat test.log | strings | grep Failed test.log
At this time, "make test" does not run any tests.I leave this step in the instructions to pick up these tests as soon as they are available.
make install &> install.log || grep Error install.log | grep -v Installing | grep -v Up-to-date
If you see errors, stop and investigate.
Build asap
Optional, but this is so quick you should.
-
cd $workDir/asap; mkdir build; cd build
- Run cmake as required for the platform
- OS X 10.7
-
cmake .. -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE -DCMAKE_FIND_ROOT_PATH=/opt/casa/darwin11 -DFORTRAN_LIBRARIES=/usr/lib/gcc/i686-apple-darwin11/4.2.1/x86_64/libgfortran.a -DLLVMCOMPILER=1 &> cmake.log || cat cmake.log
- RHEL 5
-
cmake .. -Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH -DPYTHON_LIBNAME=2.7 -DCMAKE_BUILD_TYPE=$CASA_BUILD_TYPE &> cmake.log || cat cmake.log
Notes:
- cmake messages will be different on different systems. Look for the following at the end of the cmake messages:
-- Configuring done
-- Generating done
-- Build files have been written to: $workDir/asap/build
- The options "-Darch=$CASA_ARCH -DCMAKE_INSTALL_PREFIX=$workDir/$CASA_ARCH" are required. asap defaults to installing in /usr/local/.
Compile, Test, and Install
make -j $HOST_CPU_CORES &> compileLink.log || cat compileLink.log | strings | grep Error compileLink.log | grep -v Scanning | grep -v Building | grep -v Built | grep -v Linking
If you see errors, stop and investigate.
make test &> test.log || cat test.log | strings | grep Failed test.log
According to Kana, ASAP developers are discussing adding automated tests.I leave this step in the instructions to pick up these tests as soon as they are available.
make install &> install.log || grep Error install.log | grep -v Installing | grep -v Up-to-date | grep -v tFITSErrorImage
If you see errors, stop and investigate.
Incremental builds after updates from Subversion
Rebuild, test, and install all components
To update your checked out code from Subversion and build any changes:
cd $workDir && svn update
for build_component in casacore code gcwrap asap
do
echo $build_component
cd ${workDir}/${build_component}/build
make -j $HOST_CPU_CORES 2>&1 | strings > compileLink.log
make test 2>&1 | strings > test.log
make install &> install.log
done
cd $workDir
echo "Checking compileLink.log"
grep Error */build/compileLink.log | grep -v "Building" | grep -v "Built target"
echo "Checking test.log"
grep Error */build/test.log
echo "Checking install.log"
grep Error */build/install.log | grep -v "Built target" | grep -v "Up-to-date:"
Some errors are expected in the casacore/build/test.log and code/build/test.log files. Check them manually.
Test Builds
Run Functional Tests (runUnitTest.py)
source ${workDir}/casainit.sh
revision=`cd $workDir; svn info | grep "Last Changed Rev" | awk '{print $4}'`
mkdir -p ${workDir}/testRuns/r${revision}/functionalTests
cd ${workDir}/testRuns/r${revision}/functionalTests
functional_test_list=${workDir}/${CASA_ARCH}/python/2.7/tests/unittests_list.txt
functional_test_names=`cat $functional_test_list | grep -v \# | awk '{print $1}'`
for test in $functional_test_names
do
echo ${test}
time casa --nogui --log2term -c ${workDir}/${CASA_ARCH}/python/2.7/runUnitTest.py ${test} &> ${test}.log
if [ $? != 0 ]; then echo ${test} >> nonZeroExit.log; fi
done
echo
if [ -e nonZeroExit.log ]; then cat nonZeroExit.log; fi
echo
grep FAILED test*log | grep -v FIT
echo
grep "Too many open files" test*log
Run "*_regression.py"
source ${workDir}/casainit.sh
revision=`cd $workDir; svn info | grep "Last Changed Rev" | awk '{print $4}'`
mkdir -p ${workDir}/testRuns/r${revision}/regression
cd ${workDir}/testRuns/r${revision}/regression
regression_test_list=${workDir}/${CASA_ARCH}/python/2.7/regressions/admin/tests_list.txt
regression_test_names=`cat $regression_test_list | grep -v \# | grep -v unit_tests | awk '{print $1}'`
for test in $regression_test_names
do
for f in `ls ${workDir}/${CASA_ARCH}/python/2.7/regressions/tests/${test}.py 2>&1 | grep -v "No such file or directory"`
do
echo $test
cp $f .
time casa --nogui --log2term -c "execfile('$test.py');run(True)" &> ${test}.log
if [ $? != 0 ]; then echo ${test} >> nonZeroExit.log; fi
done
for f in `ls ${workDir}/${CASA_ARCH}/python/2.7/regressions/${test}.py 2>&1 | grep -v "No such file or directory"`
do
echo $test
cp $f .
time casa --nogui --log2term -c ${test} &> ${test}.log
if [ $? != 0 ]; then echo ${test} >> nonZeroExit.log; fi
done
done
echo
if [ -e nonZeroExit.log ]; then cat nonZeroExit.log; fi
echo
# NOTE: needs work.
grep Regression *_regression.py.log | grep -i failed
echo
grep "Press enter" *log
echo
grep "Too many open files" test*log
OLD Results
Step |
OS X |
RHEL 5 |
|
10.6 |
10.7 |
32 bit |
64 bit |
|
sneffels |
casabtm1 |
kusanagi |
kirara |
rishi |
warp |
configure casacore |
S |
x |
S |
S |
S |
S |
build casacore |
S |
x |
S |
S |
S |
S |
test casacore |
8 of 466 fail |
x |
8 of 466 fail |
8 of 466 fail |
8 of 466 fail |
8 of 466 fail |
configure code |
S |
x |
S |
S |
S |
S |
build code |
S |
x |
S |
S |
S |
x |
test code |
n of 52 fail |
x |
52 of 52 fail |
52 of 52 fail |
52 of 52 fail |
52 of 52 fail |
configure asap |
S |
x |
S |
S |
S |
S |
build asap |
x |
x |
S |
S |
S |
S |
test asap |
x |
x |
no tests? |
no tests? |
no tests? |
no tests? |
key:
- x: not attempted yet
- S: succeeded
- F: failed
F - 1:
-- Looking for ATM header ATMEnumerations.h -- /opt/casa/core2-apple-darwin8/3rd-party/include/ATM/ATMEnumerations.h
-- Looking for ATM header ATMVersion.h
-- Looking for ATM header ATMVersion.h -- NOT FOUND
CMake Error at install/config.cmake:762 (message):
ATM could not be found. Please check!
Call Stack (most recent call first):
CMakeLists.txt:756 (casa_find)
Fixed by building aatm from source (ignoring test build error) and installing into ../darwin.
F - 2:
CMake Error at install/config.cmake:603 (message):
Could compile and link to, but not run QWT:
Call Stack (most recent call first):
CMakeLists.txt:862 (casa_find)
Having trouble debugging. cmake deletes test programs after running. The --debug-trycompile option is suppose to save test programs, but I get very different results with this option.
Worked around by adding /opt/casa/core2-apple-darwin8/3rd-party/qwt-5.2.0/lib/ to DYLD_LIBRARY_PATH
--
ScottRankin - 2011-02-10