Build and Test Process Requirements

Overall

  1. Whenever possible, the whole process should conform to industry best practices for Continuous Delivery systems. (see e.g. http://www.drdobbs.com/architecture-and-design/continuous-delivery-the-first-steps/240161356)
  2. Wherever possible, the B&T process should use industry-standard tools, and their associated standard conventions.
  3. The e2e process should be sufficiently well documented that handover of support and maintenance from one B&T engineer to another should be relatively straightforward.
  4. The process should be able to routinely produce functioning, self-contained packages for all supported platforms.
  5. The process should allow for the easy inclusion of new, additionally supported platforms.

Issue Tracking

  1. The CASA team shall use an industry-standard issue tracking system to track progress on bugs, new feature requests, etc.
  2. The issue tracking system should support various standard fields and established (good practice) CASA workflows. These include ticket number, the addition of ticket watchers, a configurable ticket state workflow, relative prioritization, sub-tasks, etc.
  3. The issue tracking system should support a plug-in infrastructure that can be leveraged by the other components of the B&T toolchain

Code Commits & Version Control

  1. All code commits will be managed via an industry-standard Version Control System (VCS).
  2. It should be simple for a developer to roll back any change committed to a previous version.
  3. The commit history should be readily available to the CASA development team, in an automated report form.
  4. It should be possible for developers and user testers to easily determine if any given code commit has been included in any given published package.

Build Process

  1. There should be one established build process used in the production of all packages.
  2. A single unified codebase should be used for routine user (tester)-facing CASA builds. Temporary development branches, etc. are acceptable (and sometimes necessary), but circumstances such as the NRAO/Google Code casacore bifurcation should be avoided.
  3. It should be possible to build CASA in one disk storage location, move it to another and the package should still function.
  4. Code merging between branches should be a straightforward process, and should never be a cause for delays in the deployment of standard CASA packages.
  5. Regular (e.g. nightly) package builds should be run and deployed automatically in both Charlottesville and Socorro. These should ideally be clean end-to-end builds, and also be accessible to developers in Europe, Japan, etc.
  6. As CASA’s internal architecture permits, it should be possible to build the various components/modules of CASA individually.
  7. The build process should automatically generate summary reports on the progress and success/failure of the build. These reports should be available to all members of the development team.
  8. The build process should automatically generate reports summarizing the contents of each package built. These reports should be easily accessible to all CASA developers and user testers.
  9. In the event that a build is broken, the system should be able to give clear indications as to the cause of the build failure, and automatically notify the appropriate individual.
  10. When appropriate, a code commit intended to specifically address a particular tracked issue should trigger a notification to the involved parties (e.g. the developer and user tester) that the committed code is now available for use/testing in a given CASA package. This notification process should end when the issue been resolved.
  11. The build system should natively support the use of a standardized package version numbering scheme.

Testing Process

  1. A standard suite of automated unit tests, integration tests and system regression tests will be automatically run on all routine automated builds (as time permits).
  2. Unit tests will be provided by developers using a common framework as an integral part of the CASA development process.
  3. Andy Hale’s testing group will be responsible for converting CASA Guides examples into automated regression tests, but these shall be run routinely as part of the B&T Group work.
  4. The distinction between “test”, “stable”, “pre-release” and “release”-level packages should be established on the basis of a given package passing a set of standardized “hurdles”, based on increasingly-stringent requirements for passing tests.
  5. All developer machines should have shared access to a standard library of realistic test data for development and automated testing purposes. Whenever possible, these test data should be representative of real science data.
  6. Where possible, automated testing of GUI components should be conducted using industry-standard tools prior to human user testing.
  7. In general, automated unit and integration testing should focus on verification. Regression testing and (human user) acceptance testing should focus on validation.
  8. The testing regime should be designed in such a way that human user testing is naturally the final stage of testing.

Packaging Process

  1. The actual mechanism for producing CASA packages should be a standardized and clearly-documented process.
  2. Clear policies, procedures and a timeline requirement should be established for handling requests for changes to included third party packages.
  3. A standard process for documenting third party package changes should be implemented. (This might be addressed by appropriate VCS usage).
  4. The packaging process should facilitate the early detection of any dependency issues and (ideally) aid in addressing them.
  5. The packaging process should be able to generate packages for one/several/all supported platforms on request.
  6. All packages generated by the new process should be as portable as realistically possible.
  7. All packages produced should be built primarily for end-user environments. Any in-house idiosyncrasies (e.g. NRAO Managed Software deployment systems) should be regarded as special cases.

Publishing/Deployment Process

  1. The process by which CASA packages are published should be automated, ideally in accordance with standard industry practices and tools. Ideally, this should ultimately normally be a “pushbutton” or entirely automated process.
  2. The process by which CASA packages are published should support a simple and immediate process for version “roll-back”.
  3. The process by which CASA packages are publicly published should conform to the IT security requirements of all NRAO and associated sites.

Hardware

  1. The computer systems used for routine CASA package builds should be powerful enough to support a continuous build strategy. At the very least, they should be able to provide automated builds on a frequent, agreed-upon cadence (e.g. nightly).
  2. There should be a sufficient number of computer systems available that are capable of running the necessary standard test suites described above, ideally on a similar cadence to that of the automated routine builds.
  3. There must be sufficient availability of CASA developer computers that the project is not delayed due to a lack of available developer hardware.
  4. Suitable hardware with appropriate OS versions for all currently-supported versions of CASA should be available to developers.
  5. Additional development, build and test hardware should be made available on an as-needed basis, whenever CASA support for additional platforms is advertised.

Other (Additional documentation, Pipeline, ATCA/SA/other considerations, other things I haven’t thought of…)

  1. The system design should not preclude the future introduction of code review tools to the CASA development workflow.
  2. In-line documentation should be regarded as an integral part of the CASA package, and retained in version control along with the associated code.

-- MarkRawlings - 2014-07-23
Topic revision: r1 - 2014-07-23, MarkRawlings
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback