OSX CASA Smoke Test [Jenkins] [wip]

==

Setup

OSX

Configurations here and there

CASA Data

The fastest way to get the data is by doing an rsync, e.g.:

rsync -av --progress atejeda@warp.cv.nrao.edu:/home/casa/data/trunk/ /opt/casa/data/

Jenkins

OSX specific configuration

Small summary of steps to be done. And specify OSX and Jenkins version.

After install jenkins from [x], stop any jenkins instance:
launchctl stop org.jenkins-ci

Login as Jenkins, or trough VNC for remote connections:
  • unlock to make changes
  • set a password [ jenkins > Users & Groups Preferences ]
  • [right click on jenkis > Advanced Options...] set the group to staff and check that "Account Name" and "Full Name" are filled with "jenkins" .e.g: [ss]

As administrator/root/superuser/sudo:
mv /Library/LaunchDaemons/org.jenkins-ci.plist /Library/LaunchAgents/org.jenkins-ci.plist # move it from daemons to launcher agents
defaults write /Library/Preferences/org.jenkins-ci heapSize 1024M # increase heapsize to 1024 MB
chgrp -R staff /User/Shared/Jenkins # change group files permission

Edit /Library/LaunchAgents/org.jenkins-ci.plist and change the group name, e.g:
...
    <key>GroupName</key>
    <string>staff</string>
    <key>KeepAlive</key>
...

Reboot and login as jenkins, jenkins instance should be up and running. In order that jenkins can connect to the windows server the jenkins sessions should exists, this is due daemon [users, group, scripts] are not allowed to get a connection to OSX WindowsServer,

To manually load and unload jenkins (start and stop), although these commands should not be use since the login session start jenkins
launchctl unload -w /Library/LaunchAgents/org.jenkins-ci.plist
launchctl load -w /Library/LaunchAgents/org.jenkins-ci.plist

Plugins

Plugin are installed through jenkins configuration, the needed ones are:
  • [Nice to have] Color ANSI Console Output: nice output for ANSI color parser.
  • [Mandatory] Environment Injector Plugin: use to setup environment variables used by the build and test process, also to apply DRY.

Job Configuration

Configuration example [attached ss] and reference to the XML file.

Build CASA

List of hackings done to the current date in order to build the software.

Module used:

Running OSX Smoke Tests

Using a modified version of CasaRegression.py [refer to the Pybot hardcoded paths section in Caveats and Issues].

Caveats and issues

Module casacode lapack linking

Describe what was wrong and how to get the patch [uploaded somewhere].

patch applied

if( ${arch} STREQUAL "darwin" )
    message(STATUS "prefix yes")
    find_package (BLAS REQUIRED)
    find_package (LAPACK REQUIRED)
else()
     message(STATUS "prefix no")
     # BLAS
     casa_find( BLAS
           LIBS blas )

     # LAPACK
     casa_find( LAPACK
           LIBS lapack
           DEPENDS BLAS )
endif()

casacore

-- BLAS library? ......... = /opt/local/lib/libf77blas.a;/opt/local/lib/libatlas.a
-- LAPACK library? ....... = /System/Library/Frameworks/Accelerate.framework;/opt/local/lib/libf77blas.a;/opt/local/lib/libatlas.a


casacode

-- Looking for LAPACK library lapack
-- Looking for LAPACK library lapack -- /opt/local/lib/liblapack.a

problem like

warnings generated.
3 warnings generated.
1 warning generated.
8 warnings generated.
Linking CXX shared library libsynthesis.dylib
Undefined symbols for architecture x86_64:
  "_ATL_dger", referenced from:
      _ATL_dgetf2 in liblapack.a(ATL_dgetf2.o)
  "_ATL_xerbla", referenced from:
      _atl_f77wrap_dgetrf_ in liblapack.a(ATL_f77wrap_dgetrf.o)
      _atl_f77wrap_dgetri_ in liblapack.a(ATL_f77wrap_dgetri.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Jenkins Job Encoding

Even though the encoding is set by the terminal environment, this isn't (in some way) inherited from the terminal for ipython environments, it is needed to force it:
# encoding [jenkins compatibility]
LANG=en_US.UTF-8
PYTHONIOENCODING=UTF-8

This is done when the environment variables are injected into the job. If you are facing a problem with encoding, like the example below, this is your solution e.g.:
Original exception was:
Traceback (most recent call last):
  File "/Users/Shared/Jenkins/Home/workspace/osx-casa-trunk/darwin/python/2.7/casapy.py", line 1424, in <module>
    ipshell.mainloop( )
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/Shell.py", line 76, in mainloop
    self.IP.mainloop(banner)
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/iplib.py", line 1745, in mainloop
    self.exec_init_cmd()
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/iplib.py", line 1780, in exec_init_cmd
    self.push(self.prefilter(self.rc.c, False))
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/iplib.py", line 2461, in multiline_prefilter
    out.append(self._prefilter(l, continue_prompt))
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/iplib.py", line 2443, in _prefilter
    return prefilter.prefilter(line_info, self)
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/prefilter.py", line 153, in prefilter
    return ip.handle_normal(line_info)
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/iplib.py", line 2483, in handle_normal
    self.log(line,line,continue_prompt)
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/Logger.py", line 233, in log
    self.log_write(line_mod)
  File "/opt/casa/01/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/Logger.py", line 245, in log_write
    write('%s\n' % data.encode(sys.stdin.encoding))
TypeError: encode() argument 1 must be string, not None

Find data

CASAPATH is defined as:

/Users/Shared/Jenkins/Home/workspace/osx-casa-trunk/darwin/.. darwin

It can' find the data due the regex while is grepping isn't accepting dots in the path.

STDERR traceback.py:13:   File "/Users/Shared/Jenkins/Home/workspace/osx-casa-trunk/darwin/python/2.7/testbase.py", line 341, in locatedata
STDERR traceback.py:13:     % (datafile, self.dataBaseDirectory)
STDERR traceback.py:13: Exception: Could not find datafile 0420+417.ms in the repository directories ['/Users/Shared/Jenkins/Home/workspace/osx-casa-trunk/darwin/../data']

The problem is related to the the bad regular expression:

find -L /Users/Shared/Jenkins/Home/workspace/osx-casa-trunk/darwin/../data/ -name 0420+417.ms -print 2>/dev/null | grep -vE "^\."  | grep -vE "/\."

The latest can be solved by
  • Replacing both regular expressions by: \.{1}\w
  • Or using fancy find arguments: -not -name
but, in order to not modify more files, these variables are set by jenkins, dynamically, for the test:

export CASAPATH="$WORKSPACE darwin"
export PATH=$WORKSPACE/darwin/bin:/bin:$PATH

Where $WORKSPACE belongs to Jenkins environment variable, which is the root Jenkins job, e.g.: /Users/Shared/Jenkins/Home/workspace/osx-casa-trunk/.

Pybot hardcoded paths

Solved by modifying CasaRegression.py and using environment variables instead [paths, what was modified and where to get it]

Resources

-- AlexisTejeda - 2015-06-10
Topic revision: r8 - 2015-06-19, AlexisTejeda
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback