CASA Build and Test Group Meeting

Charlottesville: Tuesday, 18th November, 2014, 18:00; Room ER331.

Japan: Wednesday, 19th November, 2014, 08:00.

DIAL-IN NUMBERS & PASSCODES:

USA Toll Free Number: 866-901-8266

USA Toll Number: +1-203-566-3863

(Note: In practice, these telecons are usually conducted via Google+ Hangouts/Voice).

Participants

Mark Rawlings, Darrell Schiebel, Shinnosuke Kawakami, Andy Hale, Akeem Wells, Takeshi Nakazato, Kana Sugimoto

(Apologies from Jim: Technical issues prevented him from calling in).

Post-meeting edits and additions are in blue text.

Agenda

• Note: Due to availability of staff, the libsakura issue (see AOB) was discussed first.
• Status of current CASA packages
  • Release - 4.2.2 out for OS X (10.8) and RHEL 5 and 6. 4.2.2 release has been successfully installed in Japan.
  • 4.3 pre-release tarball packages made available to testers in Chile and Japan. Installed on testing machines in Charlottesville and Socorro. 4.3 pre-release for OS X subsequently made available to testers in Charlottesville.
• Staffing
  • New hire, Akeem Wells, confirmed and starting in the Testing Automation position (Andy Hale's group). And in attendance for this meeting!
  • Shortlist of applications for B&T engineer assembled. Initial phone interviews were conducted by Jeff and Darrell. On-site interviews of subset to follow with full committee. A shortlist of three interview candidates for this position has been established.
• Build and Test Review Status
  • No news. Review timeline still effectively on hold, pending above decisions regarding B&T hiring strategy.
• Casacore unification - Being worked on now. For reference, Jim's proposed plan was as follows:
  • 0.5. Attempt to build against the GC casacorem and identify problems. DONE. Shinnosuke has effectively completed this task.
  • 0.75. Iteratively fix bugs and attempt to build again. This is Shinnosuke's B&T primary assignment for the short-term. Ongoing. Ger has been working with Shinnosuke on this. PROGRESSING Shinnosuke has successfully build code against the Google Code casacore. He is about 30% of the way through doing the same for gcwrap. Discussions with Ger are ongoing.
  • 1. Merge our Casacore into GC casacore. This will require us to flush all casacore mods in and then let Ger to the final merge (finally).
  • 2. Build CASA against GC casacore (platforms?)
  • 3. Test CASA/GC casacore against our various regression tests, etc.
  • 4. Create a Jenkins job that will handle B+T of codebase using GC casacore.
  • 5. Keep a cached, readonly copy of GC casacore locally(?) to prevent a glitch at googlecode from allowing us to build. Also we should maintain periodic a backup copies.
  • 6. Go live and live happily ever after.
• B&T news from Charlottesville.
  • Most of the new package production system is now in place. Already pushed out to CASA developers nodes in Socorro. Holding back on data reduction cluster deployment in CV - probably to be deployed post-4.3 release now.
  • Mark has written up the current and planned structures available so far, including version numbering. Iterated on documentation with Socorro (Rob) and EA (Kana). Still to write up: the process "behind the scenes"...
  • The 4.3 release branch is now pretty mature. Developers should continue to check in subsequent 4.3 bug fixes to both trunk and release.
  • Updates to the launcher wrapper made available, supporting launch of the pre-release packages. User testers now have access to the pre-release package for 4.3 bug fix testing, including an OS X 10.8 package.
  • OS X packaging automation is a work in progress by Darrell. He is currently experimenting with doing this using Gradle and Groovy.
• B&T news from EA.
  • Many/most of the CASA 4.3 Single Dish jobs that had been done have been merged into the release branch now. Lots of testing has been done recently.
  • Shinnosuke has been working on the casacore unification issue (also see above). The work for this is tracked on Jira ticket https://bugs.nrao.edu/browse/CAS-6929. Ger has been communicating with Shinnosuke on this. See above for progress details.
  • build failed for Qt with latest cmake version. Cmake 2.8.12 was affected by this. Shinnosuke had asked Darrell about this. In the past, Darrell had simply switched to an older version of cmake. Action Item on Darrell to reply to Shinnosuke in more detail about this.
• Old Action Items
  • Darrell: Jenkins tests for check-ins for OS X. We may also want to get Alexis in on the discussions of this.
  • Andy Hale: Discuss Jenkins test for check-ins on OS X with Alexis: same (or different?) smoke tests for OS X as for Linux? Related: machines in Socorro for this? (It was noted during the meeting that this AI is not a pressing issue at this point).
• Next meeting: December 2nd, 2014.
• Any Other Business
  • libsakura, libstdc++/libc++ and C++11 ?

    CASA compiled by gcc 4.4             libsakura built by gcc 4.8+ linked with libstdc++.so.6.0.18+
                |                                |
                |                                |
                +---------+----------------------+
                          |
              CASA linked by gcc 4.4 with libsakura and libstdc++.so.6.0.11+(bundled with gcc 4.4)
                          |
              CASA requires libstdc++.so.6.0.18+ at runtime
    

    • libsakura is designed to support Measurement Set (MS)-based data reduction parallelization. The proposal was essentially to supply this as a third-party library for CASA package builds (more details of this are covered in a recent e-mail thread involving Takeshi, Kana, Darrell and Mark). It had not yet been decided if libsakura and the associated libstdc++ deliverables from EA to the Build and Test Group would include RPM packages. One suggestion raised was the inclusion of the relevant gcc version included in the RPMs.
    • A developer branch has been used to build CASA with it.
    • Currently makes use of C++11 features, which means that to build on Linux, newer versions of gcc and libstdc++ are used that are non-native to RHEL5 and 6. It can reportedly be built using the native tools under OS X (Clang and glibc++). It may be the case that the headers would compile with the older gcc (native to RHEL5/6). One possible future strategy might be to retain the official RHEL gcc compilers as supported compilers and begin to test Clang++ on RHEL6. If this proves fruitful, we could use Clang everywhere as the default compiler for CASA packages.
    • Jim (via subsequent e-mail) indicated that his understanding was that possible reluctance to adopt C++11 features by some of the Google Code casacore people was concern that some users might be unwilling/unable to move forward to C++11. He will discuss this with them in more detail in the near future. Jim makes the point that once the new features are adopted, there will be no going back. He advocated resurrecting and updating the CASA programming guidelines. These could still evolve with experience with the C++11 (and 14) features; thus the first cut can be somewhat conservative and should not require a lengthy adoption process.
    • Some were concerns voiced about having two different versions of libstdc++ being required to exist in parallel in order to run a developer build. In particular, this could conceivably cause issues for RPM-based sites, such as Charlottesville (in which the newer library might effectively replace the old one everywhere if not handled correctly; backports compatibility is supported by RPM).
    • In principle, both versions of libstdc++ should be binary compatible, and so should not be a problem. However, it was noted that any subtle, unknown incompatibilities could give rise to difficult-to-resolve issues.
    • There was some detailed discussion about alternative filesystem paths in which the newer libstdc++ could be installed, in order to minimize the risk of version confusion (use of a wrapper for CASA preload, negative implications for developers who run the binary directly, etc.). Possibilities involving the inclusion of builds containing paths with additional version number details were also discussed. There is a utility for OS X that permits the version specifics to be built in (shared library name); such a tool may also exist for Linux.
    • Decision: Kana will send notes to Darrell on how to retrieve and build this. This will be attempted in isolation on a test machine in Charlottesville, to allow Darrell to assess its implications for RPM-based installations, developer machine environments, etc.
    • A dialog will continue on this topic going forwards.
  • Other?
    • None

New Action Items Arising

• Action Item on Kana: Kana will send notes to Darrell on how to retrieve and build this. This will be attempted in isolation on a test machine in Charlottesville, to allow Darrell to assess its implications for RPM-based installations, developer machine environments, etc.
• Action item on Darrell: Reply to Shinnosuke's question in more detail about Cmake failures when attempting to build Qt.
  • Done. Darrell has forwarded the discussion so far with Julian Taylor on this.
• Action item on Mark: Mark will consult Jim for a planned timeline for the casacore unification effort.
  • Done. Jim says that as soon as Shinnosuke gets to 100% of gcwrap to build, and we can pass an acceptable level of regression tests, then we can cut over to the GC casacore for routine use, assuming the rest of the infrastructure (e.g., Jenkins, Subversion, etc.) is in place. It is rumored that both LOFAR and MeerKAT are or would like to be using both casacore and CASA. They are already using GC casacore, and so there is actually some urgency on the part of LOFAR to get the unification accomplished so that they can add CASA imaging code to their existing code.

-- MarkRawlings - 2014-11-13