CASA Requirements: Build and Test Systems
This page lists requirements for build and test systems to be used by CASA Software Engineering to produce CASA packages and ensure CASA meets quality requirements. These systems will be used by automated systems to build, package, and test CASA on all
OSs CASA is required to support.
These systems are distinct from CASA development systems, exploratory debugging and test systems used by CASA Testers, and any systems used for CASA user testering.
STATUS:
- Approved by CASA Management - 2013-06-04 - to be reviewed again after gaps are filled.
- Some details missing, to be filled in as soon as possible.
Build Systems
CASA requires at least one dedicated build system per OS and version
CASA is required to support. Build systems will be used to
- package and test CASA 3rd party development tools and libraries,
- compile and run all "make test" tests on each and every source code change committed to change control, one change at a time, and
- package successful builds for additional testing.
TODO: define "package".
Required Software
Each CASA build system shall include
TODO: define "a full desktop installation" for each OS and version.
Required Hardware
CASA Build systems shall use x86-64 CPUs. These are the only CPUs currently supported by CASA.
Each CASA build system shall be capable of building CASA as described above completely in 60 minutes or less, when the CASA build is the only job. This excludes time required to checkout CASA from change control.
Fast builds are necessary because the CASA source tree can change quickly. Slow builds can render build results irrelevant, due to source changes, before build results are complete. This has lead to developers wasting time trying to resolve problems already resolved by other changes to CASA source. Since developer time is far more expensive than computer hardware, we should optimize to save developer time over saving computer hardware costs.
Build times from existing CASA Build systems
host |
OS |
make -j option |
build times |
casa-dev-07 |
RH 5.7 |
4 |
real |
62m55.237s |
|
|
|
user |
211m12.501s |
|
|
|
sys |
13m52.001s |
casa-dev-07 |
RH 5.7 |
8 |
real |
41m43.854s |
|
|
|
user |
232m42.071s |
|
|
|
sys |
16m14.604s |
casabtm1 |
OS X 10.7 |
4 |
real |
41m27.090s |
|
|
|
user |
142m32.751s |
|
|
|
sys |
9m05.837s |
casabtm1 |
OS X 10.7 |
8 |
real |
25m53.093s |
|
|
|
user |
149m23.389s |
|
|
|
sys |
9m52.931s |
Notes: Builds run in my user account on each system. No other users logged in. Minimal software running during builds. Builds run end-to-end as single jobs using
CASA build tool wrapper scripts building
https://svn.cv.nrao.edu/svn/casa/trunk@24848.
Test Systems
CASA requires at least one dedicated test system per supported OS and version listed in
OSs supported by CASA. Test systems will be used to
- get built packages from a CASA build system running the same OS and version
- run casapy-test acceptance tests,
- run casapy-stable acceptance tests,
- run casapy-prerelease acceptance tests, and
- provide test results back to CASA build systems for possible repackaging.
TODO: define casapy-test acceptance tests, casapy-stable acceptance tests, casapy-prerelease acceptance tests.
Required Software
Each CASA test system shall include
- a full desktop installation, including the OSs default desktop environment, with software and services appropriate for use as a general purpose computer.
CASA test systems shall not include any
CASA 3rd party development tools and libraries for that OS, unless they are installed as part of the OS installation to meet the requirements above.
Required Hardware
The majority of CASA testing will use ALMA and EVLA data. CASA test systems shall use the same hardware
CASA recommends to to users to reduce ALMA and EVLA data.
CASA Testers must be able to test using physical consoles on OS X test systems. Testing CASA on OS X using Screen Sharing, VNC, or other remote desktop software has been an issues because of incomplete support for OS X conventions in remote desktop software (VNC), or protocol issues between versions of OS X Screen Sharing clients and servers.
Each CASA test system shall be capable of running the "reduce-irc10216-data" test in 30 minutes or less, when this test is the only job on that system. This excludes any time required to setup the test environment.
Fast tests are necessary because the CASA source tree can change quickly. Slow tests can render test results irrelevant, due to source changes, before test results are complete. This has lead to developers wasting time trying to resolve problems already resolved by other changes to CASA source. Since developer time is far more expensive than computer hardware, we should optimize to save developer time over saving computer hardware costs.
Test times from existing CASA Build systems
host |
OS |
build times |
casa-dev-07 |
RH 5.x |
real |
28m51.725s |
|
|
user |
30m55.369s |
|
|
sys |
30m55.369s |
casabtm1 |
OS X 10.7 |
real |
15m27.375s |
|
|
user |
13m07.748s |
|
|
sys |
1m47.202s |
Notes: Tests run in my user account on each system. No other users logged in. Minimal software running during tests. Tests run on
https://svn.cv.nrao.edu/svn/casa/trunk@24848.
Separate Build and Test Systems
CASA build and test systems shall be distinct.
Running CASA builds and tests on the same system has masked a variety of CASA build and packaging issues, because the "test" system contained all 3rd party tools and libraries that are suppose to be included in packages. Therefore, testing does not detect that these tools or libraries are missing from packages.
Running CASA builds and tests on the same system has also slowed down builds. CASA tests are mostly large, end to end "regression" style tests that require several hours to run the full suite, and consume a lot of CPU and I/O resources. This competes with compiling and slows down both building and testing.
Naming Convention
These build and test systems shall be named per the following conventions:
cbt-[OS]-[n]
where:
- cbt - identifies this system as a CASA Build/Test system
- [OS] - identifies the Linux distro or Darwin (OS X) version. For example:
- cbt-d11-1 - runs Darwin 11 (OS X 10.7)
- cbt-d12-1 - runs Darwin 12 (OS X 10.8)
- cbt-de7-1 - runs Debian 7.x
- cbt-el5-1 - runs Redhat Enterprise 5.x
- cbt-el6-1 - runs Redhat Enterprise 6.x
- cbt-f17-1 - run Fedora 17
- cbt-u12-1 - runs Ubuntu 12.x
- [n] - is a sequence number
- cbt-el6-1 - the first Redhat 6.x system setup
- cbt-el6-2 - the second Redhat 6.x system setup
Tracking Computers
The site, names, and primary users of CASA build and test systems shall be recorded in the
CASA Build/Test Computer List.
--
ScottRankin - 2013-05-30