Version Control System (and Continuous Integration) Planning

This page is intended to act as the central point for the planning and execution of migration of the CASA software development project from its existing Version Control System (VCS), Subversion (svn), to a Distributed Version Control System (DVCS; probably based on Git).

(Proposed) Working Group Members

Pam Ford, Jim Jacobs, Masaya Kuniyoshi and/or Renaud Miel, Mark Rawlings, Darrell Scheibel, Ville Suoranta, Julian Taylor

Principal Goals and Deliverables of the Working Group

• Provide a detailed plan for the migration of the CASA software development team from Subversion to a Distributed Version Control System (Git).
  • Generate a planning document (Mark R. will coordinate this)
  • Provide a detailed timeline for the migration (see below)
  • Provide recommendations regarding the governance/gatekeeping/management of any participating external repositories, particularly those shared with other stakeholders
• Perform internal testing of possible solutions with minimal disruption to routine package development and production
• Ensure that suitable support is provided for the CASA development team in terms of on-site expertise and documentation
• Oversee the eventual migration from Subversion to Git

Proposed Timeline

• July 17th, 2015: First internal draft of migration planning document to be circulated by Mark R. to the other working group members.
• July 31st, 2015: First full draft of migration planning document due.
• Early August, 2015: Jeff to present the plan to CASA User's Committee.
• August 31st, 2015: Final version of the migration planning document due.
• October - November, 2015: Closed implementation testing.
• December - February, 2015: Set up and debug final implementation. test in parallel.
• January 2016: Identification of designated in-house Git experts for each site/development group.
• January - February 2016: In-house git experts training period.
• March 2016: Development team training seminar. Creation of switchover reference documentation.
• April, 2016: Following CASA 4.6 public release, cut over from Subversion to Git for the main CASA development team.

Requirements

(These may eventually need to be transferred to a spreadsheet or similar for tracking purposes)

1. Modify the workflow, such that developers commit to a test branch, which produces test/stable packages as they do now. Eventually they will probably be doing their development on their own branches so this would take the form of a single merge. The resultant capability (feature/bugfix) can then go through the "Ready To Test" / "Under Test" science user testing cycle. When the testing has been completed, the specific functionality becomes a pull request to the deployment branch. Under a CI-type model, eventually, each one of these commits could result in a pre-release being developed which we send out to the world. (Kern, June 2015)
2. Allow external users to submit patches / fixes. This is the pull-request model, where a request comes in, is evaluated and put through code review, before being submitted to the test branch (and falling into the workflow above). This is a long-standing and persistent request from some of our external power users. (Kern, June 2015)
3. Allow developers to have and share feature branches, where they can commit their changes and work together, separate from the main test branch. This is an intrinsic strength of a DVCS such as Git, but a CASA-approved workflow needs to be documented. (Kern, June 2015)
4. The CASA development team's ability to build and packager CASA should never be critically dependent on external third-party system failures (e.g. temporary server failures of an external commercial Github service provider, etc.). (Rawlings, June 2015)
5. All code commits will be managed via an industry-standard Version Control System. (Rawlings, July 2014)
6. It should be simple for a developer to roll back any change committed to a previous version. (Rawlings, July 2014)
7. The commit history should be readily available to the CASA development team, in an automated report form. (Rawlings, July 2014)
8. It should be possible for developers and user testers to easily determine if any given code commit has been included in any given published package. (Rawlings, July 2014)

Miscellaneous Other Considerations/Input

• Given the time-zone distribution of the proposed working group members, can this group operate solely via e-mail, or are telecons necessary?
• Jeff has asked Rafael to provide a recommendation regarding the various possible options for in-house Github-style repository management solutions. Github, Gitlab and Stash are currently being evaluated.
  • Stash is the Atlassian-supported solution, and could potentially provide a tightly-integrated solution incorporating Jira, Stash and Bamboo. Such a system could also support Groovy/Gradle builds. Such a solution would work well when building from a single code repository, if we wish to retain some/all of the existing multiple-repository inputs (e.g. CSIRO/asap, the recently-unified casacore), "side-loaded" contributions from these might complicate the adoption of unified Atlassian version tracking. An example of an Atlassian unified CI and version tracking system is explained in some detail in this (24-minute) video: https://www.youtube.com/watch?v=nW5hWc-s7wg
• Code review: This is something that could be introduced into a CASA development workflow. It has pros and cons:
  • Pros: Generally considered a good practice in industry; probably results in fewer bugs and better-written, more maintainable code.
  • Cons: Would require additional time/effort by the development team, or the hiring of additional staff to perform code review. Could we (e.g.) set up a "buddy system" for developers for this? Would require buy-in from all the development team; would require diplomacy.
• Although they do not need to be centrally involved, it will be important to keep Andy Hale's Testing group up-to-date on the general direction of the planning. This is probably also true to some extent for the maintainers of the external Subversion repositories upon which we currently rely.
• Short-term: We need developers to be able to check in code changes to casacore in the meantime. In other words, we need to be able to get "there" from "here", for which "here" also needs to be defined. For the 4.5 and 4.6 releases, the current plan for our organizational structure is as follows:
  • Move https://svn.cv.nrao.edu/svn/casa/trunk/casacore to https://svn.cv.nrao.edu/svn/casa/trunk/casacore.old (in case we need a quick rollback) and add https://github.com/casacore/casacore/trunk as an SVN external.This will be done shortly after we have confirmed that we can achieve successful end-to-end builds using Casacore from GitHub (this is currently a work in progress).
  • The planned workflow would then be as follows:
    1. Casacore changes are made directly to GitHub.
    2. The build system and (non-casacore) developers checkout code directly from SVN.
    3. No changes should be checked directly to SVN during Trunk development. The release branch is a real branch in SVN and changes need to be reconciled (manually) between the release branch and the GitHub trunk.
  • Other possible short-term implications:
    • Keeping the version numbering of casacore consistent with the rest of CASA whenever interfaces change may be something of an issue.
    • It will change the history in the sense that if a developer checks out (e.g.) version 32000, they would get r32000 of "casacore.old" rather than "casacore".

Other Resources

• CASA/NRAO-Specific Resources
  • Summary of initial "straw man" possible approaches to how a CASA svn-git migration might be performed (see attachments here and here).
  • Link to wiki page documenting the Green Bank migration: https://safe.nrao.edu/wiki/bin/view/Main/CVS2GitMigration
  • Jim's guide to modifying Casacore on GitHub: https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasacoreGithub
• General Git Resources
  • Crash course in Git for SVN users: https://git.wiki.kernel.org/index.php/GitSvnCrashCourse
  • git-svn cheatsheet: https://git.wiki.kernel.org/images-git/7/78/Git-svn-cheatsheet.pdf
  • git-svn tips: https://git.wiki.kernel.org/index.php/GitTips#git_svn
  • git ready: http://gitready.com/
  • Migrating a team: https://git.wiki.kernel.org/index.php/SvnMigration
  • Dr. Dobbs article documenting a major industry migration: http://www.drdobbs.com/architecture-and-design/continuous-delivery-the-first-steps/240161356
• Less-serious resources on Version Control Systems:
  • http://importantshock.wordpress.com/2008/08/07/git-vs-mercurial/
  • https://www.plasticscm.com/version-control-history.html

-- MarkRawlings - 2015-06-23