Build and Test Review Meeting - 2014-06-20


Mark, Rob, Jeff, Darrell, Jim (CV 331, SOC 280)

Blue text: Additional notes taken during the meeting.


  • Scope of Build and Test Group Considerations (the below list is available in more detail at: System Scope & Concept of Operations)
    • Version Control System Backups for the vital data for this will be handled by NRAO IT Department
    • Issue Tracking System (Jira)
    • Developer build environment (build system, toolchain)
      • Developer packaging tools (for experimental tar balls)?
    • Developer test environment (unit test development, etc.)
      • Cluster access for debugging at AOC and NAASC (Lustre access needed in many cases. Also necessitates ability to deploy a developer build on the cluster to verify a fix.)?
    • External package integration (Qt, Qwt, Boost, python, ipython, D-Bus, +?)
      • This was generally recognized to be one of the major hurdles that need to be overcome. Darrell would ideally like to see the use of Boost deprecated once C++11 has been adopted (it is able to replace most of the Boost functionality). It may be the case that Boost will no longer build with the current tools (clang). Such an undertaking would eventually be an Action Item for Jim's group. The idea of a review of exactly which 3rd party packages should be retained was generally thought to be a good idea.
      • Although a review of the 3rd party package inclusions would partially fall under the scope of the B&T group, it would be beyond the scope of the review.
    • Continuous Build and Test System (Jenkins/robot framework)
    • Integrated Testing and Validation (Module Tests, Integration Tests, automated GUI testing, e2e regression, user acceptance testing)
      • Overlap/limits with Andy's group?
      • There was some discussion of the delineation of testing work to be handled by the B&T group vs. Andy Hale's group. The developers and B&T will handle the creation and building of unit tests, whereas Andy's group will handle verification (by running regression tests, etc.). Once this is passed, validation is done with a casapy-test/stable package via science user testing, handled by Mark. In terms of module level testing, B&T should be responsible for maintaining the tests, AH's group will execute them.
      • Darrell would be interested in seeing building become more of a developer's responsibility (this probably dovetails with any adoption of a DVCS), and then pass on to AH's group for regression testing, and then Mark for user testing (as per above).
      • It weas also noted that currently, the build and packaging systems are totally separate and different. It would be a good idea to make them more consistent, so that (e.g.) "make tar" and "make app" could be used for Linux and OS X respectively. Conceivably, we might need a version of "make app" for OS X 10.8, and another for 10.9.
      • Possibly a detail for now, but we currently aim to build on RHEL6 and OS X 10.9. We could adopt a strategy where we build for those two, and then have additional test platforms available to AH's group for (say) Ubuntu, and they can then take our known-to-be-good codebase, run the automated tests on those machines, and report back specific issues encountered via Jira tickets, etc. A nifty long-term approach might be that of something like Gradle, in which the build environment is able to pull in at least some binary components that are known to work, in order to produce a workable package for a platform. Then, if one such component is found to fail on a given platform, and a patched binary is made available, then the packaging system could be told to pull in the new one that works. Such an approach would be strongly dependent on our being able to break down the library on functional lines, though, which in turn places additional architectural demands on CASA (such as the definition of and strict adherence to clear modular interfaces).
      • A key goal is for us to have relocatable builds. This would also remove the issue of a developer build not running on lustre.
    • Release Tools / Delivery
    • Other: Software licenses, hardware (and support), staffing, delivery timelines
      • Resource considerations, both one-time and recurring. Setup and training costs for any major changes (e.g., going from SVN to GIT) and subsequent maintenance costs; maybe some of that can come from the IT budget (or maybe we have to "pay" them to provide needed support).
      • Timeline planning should realistically factor in the impact of scheduled software releases.
  • Measures of Effectiveness (Rob will explain)
  • Measures of Performance (Rob will explain)
  • Scope of Review
    • Review Plan - see proposed list of deliverables
    • Expected products?
      • Proposed workflow - system design
      • Proposed tools set (tool role coverage, gap analysis?)
      • Clear migration path
      • "Interfaces": Clear definition of roles, responsibilities and relationships between B+T, NRAO IT, CASA developers, and Andy's group, etc.
      • External "Interfaces"
        • We have partnerships of various forms with Ger et al. for cassacore (Jim is the main contact for this, but it needs to be considered by B&T)
        • ATNF, MEERKAT
        • ESO (to some extent, although that interaction is rather different in nature)
      • Documents
        • Documents describing these: requirements, procedures
        • Living tracking documents: timeline, risk register, ?
        • Jeff emphasized that he wanted the overall approach to be lightweight in terms of documentation i.e. it should be written with an emphasis on friendly, open collaboration,rather than producing reams of legalistic policy/procedures documentation. The focus should be on having a clear "endgame" B&T process in mind, what will be needed, when, etc.
  • Next steps
    • Requirements Definition
    • Mark will start thinking about requirements definitions, and toolchain components that might meet this. These can undergo further technical discussion (including Darrell's thoughts) when Darrell returns from vacation.
  • Future meeting schedule?
  • AOB
    • Immediate Build and Test Concerns
      • Staffing
        • The advertisement for Scott's replacement is out.
        • Darrell will be on vacation from Tuesday 24th, and will be back in the office on July 19th.
        • We should work towards making more use of Shinnosuke. We may also be able to get some effort assigned (Schlake) to help with implementation, as well as a bit of help from Andy Hale's new hire.
        • ACTION ITEM ON MARK: Look into scheduling a weekly (or perhaps in the shorter term, fortnightly) Build and Test group meeting. This can also cover review-related issues. Rob, Jeff and Jim should be included in the invitations to these.
      • Current Build and Test Status and Priorities
        • Darrell noted that for OS X 10.9, the 3rd party packages seemed to build OK, but CASA as a whole will not yet, as it is heavily dependent on the idiosyncrasies of these. Michel makes a lot of use of Boost.
        • The short-term priorities are:
          1. Get the 4.2.2 tarball out to the Pipeline group. Darrell will work on this before he leaves.
          2. Do the release work when Darrell gets back.
      • Other standing issues: googlecode, RHEL6, build on dev machines and run on cluster, Jenkins...

-- MarkRawlings - 2014-06-19
Topic revision: r3 - 2014-06-20, MarkRawlings
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback