CASA Build and Test Group Meeting
Tuesday, 13th January, 2015, 18:00; Room ER331.
Tuesday, 13th January, 2015, 16:00; Room ?.
Wednesday, 14th January, 2014, 08:00.
DIAL-IN NUMBERS & PASSCODES:
In practice, these telecons are usually conducted via Google+ Hangouts/Voice (look for user: mgrawlings
USA Toll Free Number:
USA Toll Number:
Mark Rawlings, Darrell Schiebel, Andy Hale, Akeem Wells, Shinnosuke Kawakami, Takeshi Nakazato, Kana Sugimoto, Masaya Kuniyoshi
Post-meeting edits and additions are in blue text.
- Follow-up discussion of libsakura
- Takeshi requested that Darrell update the release process to include libsakura. There had been some email exchanges about details of its integration, but these should now be fixed in principle. Darrell confirmed that he could successfully build and install it himself, using the libstdc++, etc. supplied by NAOJ.
- The build of libsakura is based on CMake, so the process is similar to that used when building CASA suites. it is possible to specify the install root path using the "-D CMAKE_INSTALL_PREFIX" parameter. The available options and instructions of build are described in INSTALL.txt at the root directory of the libsakura source code.
- Darrell agreed that if NAOJ can provide RPMs for libsakura and the associated libc++, routine installation should be straightforward ("Plan A"). he would be willing to help NAOJ with their initial production of the RPM packages.
- Darrell reported the existence of version 2.1 of the Red Hat Developer Toolset (hereafter RHDT). This package is provided and supported by Red Hat for RHEL5 and 6, and includes a version of GCC 4.8.2, which could be adopted broadly by the CASA project in the future as the compiler for libsakura ("Plan B").
- Given the relative nearness of the CASA 4.4 feature freeze deadline (about six weeks from now), Kana was wary about adopting the proposed new RHDT GCC compiler now, but there was general agreement that this could very well be a good long-term path for the developers to adopt. It was proposed that a plan be drafted for this in the meantime, and we could then work towards adopting the new compiler in the post-4.4 era. The overall conclusion of the group was to go for "Plan A" for now, with the aim of switching to "Plan B" after CASA 4.4.
- Regarding the implementation of "Plan A": RPMs will need to be provided for RHEL 5 and 6, but Darrell will do the build himself for OS X (the third-party packages are built separately, so it is probably easiest for him to do the OS X build). The OS X third party packages are created by overlaying the MacPorts Portfiles from: https://svn.cv.nrao.edu/view/macports/users/drs/ports/ on the standard MacPorts port files. The overlay is adjusted to accommodate any quirks of the CASA build.
- Darrell noted that the next major goal regarding new package production is the creation of an OS X 10.10 (Yosemite) package (10.9 support will then be attempted as a secondary goal, as time and available effort permits). He noted that if libsakura is only changes slowly, he could perhaps build it for OS X 10.10 in short order and pass it back to NAOJ. Decision: A deadline of the end of January was agreed upon by everyone for the provision of all of the above to the involved developers.
- As for the build on 10.10, NAOJ are currently using an internal svn server for Sakura development. As a workaround, Takeshi will provide a tarball to Darrell containing the Sakura source code.
- Following this deadline, it will be necessary to merge any associated libsakura branch changes back into the trunk. Kana will perform this merge herself.
- Takeshi's proposed work plan was as follows:
- [ASAP] provide Sakura source code to Darrell for 10.10 build (Takeshi)
- [ASAP] provide a build instruction of Sakura on OS X to Darrell (Takeshi)
- [1/23] confirm the build of Sakura on OS X 10.10 (Darrell)
- [1/23] create an instruction for creating Sakura rpm based on provided set of binary and header (Darrell)
- [1/23] feature freeze of Sakura (EA develpers)
- [1/23] stabilize single dish branch (EA developers)
- [1/28] create Sakura rpm based on Darrell's instruction, provide it to Darrell (Shinnosuke, Takeshi)
- [1/28] build stabilized Sakura on 10.10 (Darrell)
- [1/30] put Sakura rpm to NRAO repository (Darrell)
- [1/30] include Sakura into third party for Mac (Darrell)
- [??] minor update of Sakura (EA developers)
- Status of current CASA packages
- Release - 4.3.0 out for OS X (10.8) and RHEL 5 and 6.
- Some recent issues associated with local 4.3.0 deployment in Charlottesville.
- Two candidates eventually interviewed on-site for the B&T engineer position (the third one forgot about his interview!). An offer will be made to one of the candidates. The second candidate may be suitable as a fall-back option.
- Due to a position opening up at NAOJ related to administration and CASA, someone else will soon be taking up Shinnosuke's B&T work responsibilities. It is not yet clear who will replace him, but it may or may not be Masaya. Nevertheless, in anticipation of this, Masaya will be working with Shinnosuke to help ensure a smooth handover period. We wish to thank Shinnosuke for all his work, and also welcome Masaya and thank him for being willing to help with the handover.
- Build and Test Review Status
- No news. Review timeline still effectively on hold, pending above decisions regarding B&T hiring strategy.
- Casacore unification - Being worked on now. Jim asked the following: Shinnosuke was trying to get the entire CASA 4.4 to build on RHEL6 using Ger's branch of Casacore. Is there an ETA on the completion of this yet?For reference, Jim's proposed plan was as follows:
- 0.5. Attempt to build against the GC casacore and identify problems. DONE. Shinnosuke has effectively completed this task.
- 0.75. Iteratively fix bugs and attempt to build again. This is Shinnosuke's B&T primary assignment for the short-term. Ongoing. Ger has been working with Shinnosuke on this. Shinnosuke has been working on gcwrap. Discussions with Ger are ongoing (see Jira ticket https://bugs.nrao.edu/browse/CAS-6929 for details). Jim reported yesterday that the Google Code people are keen for the merge to proceed. PROGRESSING. Shinnosuke reported that he can now build casacode and gcwrap on RHEL6 using the Google Code version of casacore. Many modifications have been required, particularly to the image analysis source code. Shinnosuke has been in regular contact with Ger, who is still making changes to the Google Code version of casacore (e.g. a change in namespaces, changes in directory structure, etc.). Masaya has been performing verification of casacore on local NAOJ machines with the same regression that Darrell has used, and hopes to have completed this within the next week (see also below).
- 1. Merge our Casacore into GC casacore. This will require us to flush all casacore mods in and then let Ger do the final merge (finally).
- 2. Build CASA against GC casacore (platforms?)
- 3. Test CASA/GC casacore against our various regression tests, etc.
- 4. Create a Jenkins job that will handle B+T of codebase using GC casacore.
- 5. Keep a cached, readonly copy of GC casacore locally(?) to prevent a glitch at googlecode from allowing us to build. Also we should maintain periodic a backup copies.
- 6. Go live and live happily ever after.
- B&T news from EA.
- EA OS X hardware purchase update? Two OS X 10.10 machine has been purchased (a 10.9 machine was not available for purchase). The intention here is to verify libsakura on two different versions of OS X (at most) which are supported by CASA.
- Masaya has been working on verification of the casa 4.3 on our machines (NAOJ) with the same regression data (10349) as Darrell used.
- Shinnosuke has been working on the casacore unification issue.
- build gc-casacore(google code) + code(branch) + gcwrap(branch) + asap(trunk) is completed and one regression test passed(FLS3a)
- gc-casacore with namespace casacore instead of casa have to be built with casa-code successfully
- our modification of code, gcwrap, asap have to be verified by appropriate developers.
- source code of casa
- source code of casa-3rd-party is modified at NRAO and then casa is distributed with binary 3rd-party?
- B&T news from Charlottesville.
- Meeting scheduled tomorrow scheduled with Pat Murphy to clarify NRAO site deployment procedure.
- Darrell has had some success with building the HTML version of the CASA Cookbook with Heava (a tool written in OCaml). Building the tool and task documentation is proving more difficult in practice. Mark would ideally like to assign the cleanup of the process to the new B&T hire.
- Mark has written up the current and planned structures available so far, including version numbering. Iterated on documentation with Socorro (Rob) and EA (Kana). Still to write up: the process "behind the scenes"...
- OS X packaging automation is a work in progress by Darrell. This is using Gradle and Groovy. Early runs showed it took 15 hours to complete (c.f. ~7 hours under Linux). More computing horsepower (and more efficient tests) may be needed.
- Two new Macs are currently being set up for Andy Hale's testing group. Any status updates? The new Macs are now at NRAO-CV, but are awaiting external disk systems.
- In the process of getting the the documentation to build, Darrell discovered new supplemental RedHat software bundles which were previously unknown to us. RedHat Developer Toolset and RedHat Software Collections:
- Old Action Items
- Andy: Discuss Jenkins test for check-ins on OS X with Alexis: same (or different?) smoke tests for OS X as for Linux? Related: machines in Socorro for this? (It was noted during the meeting that this AI is not a pressing issue at this point). Andy had started to have some discussion with Alexis about this. Smoke tests will be run on the new Test Group Macs. They still need to discuss test coverage for OS X and Linux. THIS AI IS ONGOING.
- Action Item on the B&T Group: Review Justo's document and come up with a plan forward.
- Action Item on Mark: Raise the Boost issue with the relevant CASA managers: do we want to work towards getting rid of Boost in CASA, or embrace it? In the past, it had been indicated that when we switch to C++11 the HPC/MPI C++ code could transition to C++11 and that Boost MPI could be phased out. Justo feels that this is not the case, but Darrell thinks it could probably still be avoided.Adopting Boost MPI as the basis for our C++ parallelization would effectively mean adopting Boost permanently (in which case, we might as well embrace it?). Darrell advocates a standalone test program to establish whether or not C++11 alone can do the job. Issue raised with Jeff & Jim
- Action Item on Mark: Raise the following question with the relevant CASA managers: Would it be acceptable to the CASA management to ship CASA for Linux with MPI, but for Mac OS X without it? (Foreseen possible objections: Users might be confused/complain; they still might want to use a Mac as a controller system). Issue raised with Jeff & Jim
- Action Item on Darrell (with Justo): Review and discuss further the Boost dependencies. Ongoing?
- Next meeting: December 27th, 2014.
- Any Other Business
- NAOJ expressed some concerns about the software license conformance (specifically the availability of source code). It was thought that most was LGPL2, but Darrell note that our PGPLOT license situation may checking on. As a point of principle, we do honor the licensing terms of the third-party packages, so that if we do modify code, we do make those changes available. New Action Item on Mark: Raise the issue of software license conformance with Jeff and Joe, just to make sure we're doing the right thing.
- There was some casual discussion regarding automated testing, in terms of providing CASA Guides-based tests for MMS handling.
New Action Items Arising
- New Action Item on Mark: Raise the issue of software license conformance with Jeff and Joe, just to make sure we're doing the right thing.