CASA and CasaCore

Overview

Status

4/9/15: CASA's integration with a single, external CasaCore codebase is nearing completion. At the same time, CasaCore is moving from the soon-to-be-defunct GoogleCode to GitHub.

Introduction

CasaCore is an essential part of CASA, but unlike the main CASA codebase (aka “code”) CasaCore is shared with other organizations writing astronomical software. As of October 2014 there are two main instances of CasaCore: one located at the NRAO and one located on !GoogleCode. The two versions are periodically brought back into sync by merges performed at the NRAO and by Ger Van Diepen (ASTRON). If CASA is to benefit from updates to CasaCore provided by developers outside the NRAO we have to maintain a reasonable level of synchronization between the two instances. Because the merge process is tedious and error prone we are in the process of doing away with the NRAO version of CasaCore and use the one on GitHub. CASA's SVN repository will be modified to use a link (analogous to a symbolic link) to the GitHub repository; except for commits (see below) this should be transparent to normal CASA developers.

Goals

Once we have eliminated the NRAO fork of CasaCore, CASA will treat CasaCore as a third-party package much like we do ASAP. A “link” will be put into CASA's SVN repository allowing us to check out the latest version of CasaCore straight from out repository, just like we currently do for ASAP. We can still modify CasaCore but changes will have to be checked into the CasaCore googlecode repository; once a developer has provided authorization information to SVN commits to CasaCore will become as transparent as checkout.

Accessing CasaCore on GitHub

Since Google is pulling the plug on the googlecode web site, CasaCore is moving to Github (it seems many open source projects are migrating to Github). The migration should be completed soon. For CASA developers using CasaCore, the location of CasaCore should be almost invisible; the structure of how CASA will make CasaCore available to CASA users is described below. Developers needing to modify CasaCore will have to know the protocol for modifying it on GitHub. This section describes that process.

CASA and CasaCore

Because not every person outside the NRAO will (or have the ability to ) be building CASA on all of our approved platforms, it is possible that a change may be committed to the GitHub master branch which breaks CASA. CASA will need to protect itself against this eventuality. We will do this by creating two CasaCore repos located on NRAO servers: the CasaCore Trial repo and the CasaCore Approved repo. Changes made to the GitHub repo will be pulled into the Trial repo. A set of Jenkins jobs will be run against the new revision in the Trial Repo. If the Jenkins jobs all pass (as defined by a script or by someone on the build and test team) the revision will be pushed to the Validated repo. All other CASA builds will draw their source from the Approved repo exclusively. Except for an emergency the Approved repo will only be modified by pulling revisions from the Trial repo; thus it is essentially read-only to CASA developers. Emergency modifications to the Approved repo will require approval by the appropriate lead(s).

In the event that a GitHub revision is rejected by the Trial builds, that information will be passed back to the CasaCore group to address the problem. Probably the most likely approach is for the revision to be quickly backed out of the Master branch until the problem can be remedied. Since the author of the revision is unlikely to be able to reproduce the problem, CASA will need to provide them with sufficient information (e.g., build and test logs) so that they can be diagnose and fix the problem; some interaction with the mod's author may be required in order to allow them to address CASA's concerns. We will also have to be sure that the breakage observed is not because we were misusing CasaCore.

Releases and Branching

When CASA makes a pre-release branch, we will create a local subversion tag in CASA's repository. This creates a de facto branch of CasaCore although it's not visible to the GitHub repository. Normally, if we need to pick up CasaCore changes to be integrated into the release we'll roll the tag forward. If CASA decides that the changes between the tagged CasaCore revision and the one containing the needed bug fix are too risky to include into CASA then CASA will create its own, temporary branch; in this scenario, CASA will be responsible for getting any local changes made on the branch back into CasaCore's trunk.

Modifying CasaCore

This section describes how developers can modify CasaCore. In the past CASA developers simply modified our fork of CasaCore; eventually Ger merged most of our changes into the "global" fork and occasionally someone here merged most of the changes back into our fork. These merges were few and far between and could be fairly unpleasant to do because of the number of changes made between the merge times; this made it frustrating for Ger and the CASA developer who got stuck with the task. The current approach should be better but will probably end up having its own rough edges.

Mechanics for Modifying CasaCore on GitHub

Below are the steps that someone wanting to modify CasaCore will need to take:

1. CasaCore modifier opens a GitHub issue (analogous to JIRA) (notifies module coordinator)
2. Modifier creates a branch in the GitHub repo; branch is named by the GitHub issue from the previous step.
3. After the modifier has completed the change, fully tested it and polished it up, it can be committed back to branch.
4. Modifier generates a pull request to the module coordinator.
5. Module coordinator evaluates the modification and if acceptable, merges it into the master branch. Github has a means for going back and forth on between the module coordinator and the modifier.
6. The branch is destroyed (by ?).

See the CasaCore Github Cookbook for more details.

Modifiers and Module Coordinators

NRAO Developers with Modify Access

NRAO developers who need to modify CasaCore need to be granted access to the GitHub repository. The list of developers with access is given below. Access can be granted in fairly short order if the need arises (probably on the order of days); access requires that the developer obtain a GitHub username or provide an existing one.

1. Darrel Scheibel
2. Ville Suoranta
3. Tak Tsutsumi
4. Dave Mehringer
5. Jim Jacobs
6. Sanjay Bhatnagar
7. Kumar Golap
8. Dirk Petry

Module Coordinators

Modules	Coordinator
build system	Malte Marquarding
casa	Ger Van Diepen / Darrell Scheibel
coordinates	Ger Van Diepen
derivedmscal	Ger Van Diepen
fits	Dirk Petry
images	Ger Van Diepen
lattices/LEL	Ger Van Diepen
lattices/LatticeMath
lattices/Lattices	Ger Van Diepen
meas	Ger Van Diepen
measures	Dirk Petry
ms/MSOper
ms/MSSel	Sanjay Bhatnagar
ms/MeasurementSets	Jim Jacobs
msfits	Dirk Petry
python	Ger Van Diepen
scimath(_f)	TBD (Maxim? Kumar?)
tables	Ger Van Diepen

(as of 4/9/15)

Guidelines for CasaCore Modifications

After we switch to the unified repository there will be some differences in how developers approach modifying code in CasaCore. Because we'll be working on a community codebase, problems introduced into the codebase will more quickly impact CasaCore users at the NRAO and elsewhere.

Determine Appropriate Location

Before making a modification to CasaCore, consider whether it might not be appropriate to locate the functionality in CASA proper. If the needed functionality is unlikely to be useful to other users of CasaCore that might be an indication that it doesn't belong there. If the functionality is of general use to CASA it can be placed in a utilities module (we don't have one yet); otherwise the functionality ought to be contained within the CASA component that needs it. A third option might be to break the functionality into two parts: a general-purpose enabling modification to CasaCore and the CASA-specific portion located in CASA proper.

Maintaining backwards compatibility.

Because CASA is not the only project using CasaCore, we cannot unilaterally make changes to CasaCore which are not backward-compatible. In the past non-backwards-compatible changes could be made to CasaCore if the developer went through the rest of the codebase and fixed any affected code. Unfortunately, this approach ends up breaking builds of non-CASA users of CasaCore; in the past such problems were handled when CASA modifications were somewhat painfully merged into GC CasaCore. The problems caused by breaking backwards compatibility are may not appear immediately after the commit (e.g., another organization is not refreshing their local codebase from GC on a very frequent basis; this will require the committing developer to go back to work on a task they thought put to rest.

Although the primary way of breaking backwards compatibility is to alter the existing interface by modification or removal of something in the interface, altering the semantics of the code can break backwards compatibility even if the API remains unchanged. This certainly does not mean that erroneous behavior cannot be fixed.

An easy way to break another organization's build is to change the calling sequence to a method (e.g., adding/removing a parameter). Depending on the CasaCore unit-test coverage, the breakage may not be apparent until the next time the other organization builds on a fresh copy of CasaCore. A safer approach is to add a new, overloaded method which supports the extended functionality; the existing method implementation would then be rewritten to call the new method implementation providing an appropriate default value for any new parameters. This approach shouldn't break other builds and also avoids creating duplicate implementations which can drift apart over time as well as clutter up the codebase. Below is a rather simple example:

-Before-

double f (double x){
    return computeSomething (x);
}

-After-

double f (double x, int exponent){
    return pow (computeSomething (x), exponent);
}

double f (double x){
    return f (x, 1); // use common implementation
}

NOTE: There must be a process for deprecating and removing poor and obsolete interfaces. The presence of such interfaces in perpetuity unnecessarily increases maintenance costs and leads to code rot.

Precommit Testing

The first step is that the code ought to compile cleanly (i.e., no warnings) and be able to generate doxygen output. The second step is to ensure that the code passes unit tests. CasaCore has a large suite of unit tests. After making a modification, these tests can be run by doing make test after CasaCore has been built and installed. If the developer's changes have increased the number of failing tests then further investigation is warranted before committing the code. In addition, to passing any existing tests, the unit-code test suite should be augmented with tests that exercise the new functionality; otherwise future modifiers may inadvertently break the added features without realizing it. The developer should also feel free to augment the existing unit test coverage in case it has gaps.

Some or all of the unit tests should be run using valgrind's memcheck feature to look for possible memory leaks.

Coding Style

Modifications should adhere as much as possible to the common Casacore coding style. The goal here is to maintain the code's readability. Mixing styles reduces the readability of the code. When making very minor changes, it's best to use the existing style. If the change is more substantial, then the style can be changed in the module being modified while leaving the other modules unaffected. For larger changes, the developer may choose to restyle the entire file—be aware that this can make merging much more difficult and might not be appreciated by the module coordinator.

Some basic guidelines:

• Indent by 2 or 4 spaces (8 is too much)

• if, else, and for/while code block must be enclosed in braces unless everything on a single line

• • If/for bracing should be done like

if (condition) {

} else {

}

• Class member variables should be be named either name_p or itsName (the latter preferred).

Abide by CasaCore development guidelines.

Since CasaCore is being modified globally (literally) it is important to employ good housekeeping practices when modifying it. For large modifications, the CasaCore development guidelines should be used; this will include commenting, unit-test coverage and general code-style issues. For smaller modifications, simply make the changes blend in with the existing code. CASA's CasaCore page will provide access to the guidelines. CasaCore is an old codebase so oftentimes a module has been modified by many different developers with different personal styles; feel free to clean up a module's style if your modifications are extensive or the existing style(s) are abhorrent but when doing so make sure that the style you leave the module with is consistent with the guidelines. In the past, reformatting the code, while improving the readability, often made life difficult for the people tasked with merging in often extensive changes; however, the new direct-commit process should eliminate most of the merges and any that remain should be of much smaller scope.

Breaking Other's Builds

CasaCore is shared with a number of other astronomy projects so CASA needs to be mindful of its obligations to that community. If a CASA developer commits a change to CasaCore it's entirely possible it could break another organization's build or worse, their end product; since other organizations may be using platforms CASA is not currently supporting it's likely that some system-specific breakage will be encountered. Unless the fix is to permanently back the change out of CasaCore, CASA will have to work with the other organization to fix the problem in a timely fashion; we will have similar expectations if the problem is introduced by another organization. The CASA project will have to understand the importance of CASA developers providing timely support to resolve breakage as a high priority and unscheduled task.

C++11

Eventually, CasaCore will move to C++11, but until it's been determined that the using organizations can support C++11, we will need to be careful to not introduce C++11 features into CasaCore. Where they are important (e.g., std::shared_ptr) they can be used but need to be protected by #ifdefs and a suitable alternative provided (which could be a throw for new functionality).

Introducing or Modifying External Dependencies

Adding an external dependency to CasaCore is discouraged and must not be done without coordination without consultation of the module coordinator and other CasaCore principals. Adding external dependencies complicates the deployment process and could even prevent the package's deployment on some hosts. Increasing the minimum version of an existing external dependency requires similar consultation. There is no carved-in-stone rule against adding or modifying external dependencies but it is not something that should be done casually.

Risks

This is a new process for CASA and the other CasaCore users so we are likely to encounter problems. The purely mechanical ones should be easy to correct when discovered.

• Bad code entering the Master Anyone making a change to CasaCore will have to make appropriate changes that are relevant, useful, tested and compatible with the existing CasaCore. The module coordinator will have look at the modifications before pulling them into the Master branch, but they cannot be expected to perform any testing or in-depth analysis of the implementation. This risk has always existed with our use of CasaCore. CASA mitigates this risk by only building production and development code against a validated CasaCore repository located on an NRAO server.

• Module coordinator Not Responsive A module coordinator could become a bottleneck to getting changes accepted into mainstream CasaCore. Since being a module coordinator is unlikely to be their highest priority task, they might not provide timely action on requests. If such a bottleneck arises, CASA will need to work with the CasaCore group to resolve this. As an emergency workaround, vital modifications can be made directly to the NRAO validated CasaCore repo.

• Philosophical differences The module coordinator is supposed to preserve the integrity of one or more modules but is not responsible for actually making changes to the module. This is a common way of handling the maintenance. A module coordinator could simply rubber stamp changes to their module or they could be quite pedantic about the nature of changes that can be accepted into the module, even to the point of rejecting a proposed modification outright. A stricter module coordinator is better for the quality of the codebase but at some point an impasse might be reached where the modifier believes they have a vital change that needs to be made while the module coordinator believes that proposed modification is completely inappropriate. If such an impasse is reached, then the CasaCore group can be asked to resolve it. If the outcome of that appeal is not satisfactory then the appropriate authorities in the CasaCore-using organizations will have to become involved. One mitigation to this risk could be to put the functionality into a layer above CasaCore where only CASA design rules need to be met.

Definitions

Module Coordinator

The person or persons charged with overseeing a CasaCore module. Normally, this is a minor and lower-priority part of their fulltime job.

CasaCore Group

This informal group consists primarily of the CasaCore module coordinators. It's job is to manage everyday issues regarding CasaCore.

CasaCore Using Organization

a project or organization using CasaCore as part of their software. The group of all such using organizations is the constituency of CasaCore; however, at this time there is no formal charter or memo-of-understanding involved. It seems likely that something more formal may need to be created in the future but for now a casual, consensual approach is in force.

-- JimJacobs - 2014-10-30