CASA Requirements: Build and Test Systems

This page lists requirements for build and test systems to be used by CASA Software Engineering to produce CASA packages and ensure CASA meets quality requirements. These systems will be used by automated systems to build, package, and test CASA on all OSs CASA is required to support.

These systems are distinct from CASA development systems, exploratory debugging and test systems used by CASA Testers, and any systems used for CASA user testering.

STATUS:

• Approved by CASA Management - 2013-06-04 - to be reviewed again after gaps are filled.
• Some details missing, to be filled in as soon as possible.

Build Systems

CASA requires at least one dedicated build system per OS and version CASA is required to support. Build systems will be used to

• package and test CASA 3rd party development tools and libraries,
• compile and run all "make test" tests on each and every source code change committed to change control, one change at a time, and
• package successful builds for additional testing.

TODO: define "package".

Required Software

Each CASA build system shall include

• a full desktop installation, including the OSs default desktop environment, with software and services appropriate for use as a general purpose computer, and
• all CASA 3rd party development tools and libraries for that OS.

TODO: define "a full desktop installation" for each OS and version.

Required Hardware

CASA Build systems shall use x86-64 CPUs. These are the only CPUs currently supported by CASA.

Performance

Each CASA build system shall be capable of building CASA as described above completely in 60 minutes or less, when the CASA build is the only job. This excludes time required to checkout CASA from change control.

Fast builds are necessary because the CASA source tree can change quickly. Slow builds can render build results irrelevant, due to source changes, before build results are complete. This has lead to developers wasting time trying to resolve problems already resolved by other changes to CASA source. Since developer time is far more expensive than computer hardware, we should optimize to save developer time over saving computer hardware costs.

Existing Build Computer Performance

Build times from existing CASA Build systems

host	OS	make -j option	build times
casa-dev-07	RH 5.7	4	real	62m55.237s
			user	211m12.501s
			sys	13m52.001s
casa-dev-07	RH 5.7	8	real	41m43.854s
			user	232m42.071s
			sys	16m14.604s
casabtm1	OS X 10.7	4	real	41m27.090s
			user	142m32.751s
			sys	9m05.837s
casabtm1	OS X 10.7	8	real	25m53.093s
			user	149m23.389s
			sys	9m52.931s

Notes: Builds run in my user account on each system. No other users logged in. Minimal software running during builds. Builds run end-to-end as single jobs using CASA build tool wrapper scripts building https://svn.cv.nrao.edu/svn/casa/trunk@24848.

Test Systems

CASA requires at least one dedicated test system per supported OS and version listed in OSs supported by CASA. Test systems will be used to

• get built packages from a CASA build system running the same OS and version
• run casapy-test acceptance tests,
• run casapy-stable acceptance tests,
• run casapy-prerelease acceptance tests, and
• provide test results back to CASA build systems for possible repackaging.

TODO: define casapy-test acceptance tests, casapy-stable acceptance tests, casapy-prerelease acceptance tests.

Required Software

Each CASA test system shall include

• a full desktop installation, including the OSs default desktop environment, with software and services appropriate for use as a general purpose computer.

CASA test systems shall not include any CASA 3rd party development tools and libraries for that OS, unless they are installed as part of the OS installation to meet the requirements above.

Required Hardware

The majority of CASA testing will use ALMA and EVLA data. CASA test systems shall use the same hardware CASA recommends to to users to reduce ALMA and EVLA data.

CASA Testers must be able to test using physical consoles on OS X test systems. Testing CASA on OS X using Screen Sharing, VNC, or other remote desktop software has been an issues because of incomplete support for OS X conventions in remote desktop software (VNC), or protocol issues between versions of OS X Screen Sharing clients and servers.

Performance

Each CASA test system shall be capable of running the "reduce-irc10216-data" test in 30 minutes or less, when this test is the only job on that system. This excludes any time required to setup the test environment.

Fast tests are necessary because the CASA source tree can change quickly. Slow tests can render test results irrelevant, due to source changes, before test results are complete. This has lead to developers wasting time trying to resolve problems already resolved by other changes to CASA source. Since developer time is far more expensive than computer hardware, we should optimize to save developer time over saving computer hardware costs.

Existing Test Computer Performance

Test times from existing CASA Build systems

host	OS	build times
casa-dev-07	RH 5.x	real	28m51.725s
		user	30m55.369s
		sys	30m55.369s
casabtm1	OS X 10.7	real	15m27.375s
		user	13m07.748s
		sys	1m47.202s

Notes: Tests run in my user account on each system. No other users logged in. Minimal software running during tests. Tests run on https://svn.cv.nrao.edu/svn/casa/trunk@24848.

Separate Build and Test Systems

CASA build and test systems shall be distinct.

Running CASA builds and tests on the same system has masked a variety of CASA build and packaging issues, because the "test" system contained all 3rd party tools and libraries that are suppose to be included in packages. Therefore, testing does not detect that these tools or libraries are missing from packages.

Running CASA builds and tests on the same system has also slowed down builds. CASA tests are mostly large, end to end "regression" style tests that require several hours to run the full suite, and consume a lot of CPU and I/O resources. This competes with compiling and slows down both building and testing.

Naming Convention

These build and test systems shall be named per the following conventions:

cbt-[OS]-[n]

where:

• cbt - identifies this system as a CASA Build/Test system
• [OS] - identifies the Linux distro or Darwin (OS X) version. For example:
  • cbt-d11-1 - runs Darwin 11 (OS X 10.7)
  • cbt-d12-1 - runs Darwin 12 (OS X 10.8)
  • cbt-de7-1 - runs Debian 7.x
  • cbt-el5-1 - runs Redhat Enterprise 5.x
  • cbt-el6-1 - runs Redhat Enterprise 6.x
  • cbt-f17-1 - run Fedora 17
  • cbt-u12-1 - runs Ubuntu 12.x
• [n] - is a sequence number
  • cbt-el6-1 - the first Redhat 6.x system setup
  • cbt-el6-2 - the second Redhat 6.x system setup

Tracking Computers

The site, names, and primary users of CASA build and test systems shall be recorded in the CASA Build/Test Computer List.

-- ScottRankin - 2013-05-30