CASA C++ Testing Framework Evaluation

==

Frameworks selection criteria

There's a lot of frameworks out there, Wikipedia has a representative list http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B in which list all the current active c++ testing frameworks. A short list was made considering the following points nor requirements:
  • Free software nor open source
  • xUnit report
  • Mock out-of-the-box or mock framework integration
  • Fixtures and suite support
  • Learning curve and development complexity
  • Support and community
  • GCC and LLVM compilers compatibility
  • Painless CASA integration
Currently, most of the frameworks provides the same testing implementation paradigm by using minimalist c++ macros, but neither offers mock capability out-of-the-box and a portion of them integrates third-party mocking frameworks almost seamlessly. Catch, Boost.Test and Google.Test were based on the previous points and by the simplicity to develop c++ testing sources compatible with CASA.

Google Test and Google Mock

Google Mock

Google mock is a framework to mock behaviors and define expectations for those behaviors, is super-easy to use due relies on C++ class inheritance and macros to mock class methods. The main constraint is that the mocking classes can mock just virtual and const type methods, static methods aren't covered due his nature of static (no class object instance interaction with class members).

Important aspects:
  • Cardinalities: how many times a method execution should be expected.
  • Actions: even though isn't a fake object framework, when the expectation is define, is possible to return an object instance.
  • Multiple expectations: defining multiple expectation of the same type (same mock method), the framework traverse these expectations in reverse order (bottom-up).
  • Ordered calls: strict expectation order defined by sequence blocks.
  • Sticky expectations: Just the latest expectation (latest) is considered by the framework.
  • Default main implemented.
Useful documentation:

Google Test

A portable multi-platform (gnu/linux, Windows, OSX) c++ testing framework compatible with LLVM and GCC compilers, as any other testing framework structure the tests as: test-program with test cases with assertions.The framework is capable to report to the stdout or xUnit XML cobertura stye reports (compatible with Jenkins). Is widely supported and has a community support as well.

Important aspects:
  • Assertions and comparasons: two value, floating point, string
  • Death tests: test cases expected to terminate the test application if a condition is met, intended to avoid memory corruptions, security, etc.. in later test stages.
  • Fixtures and shared resources: defined by classes inheriting :testing::Test members, and instantiated just before the test case. Supports setup, tear down, setup case and tear down case.
  • Parameterized tests.
  • Test events for thirdparty frameworks.
  • Distributed and parallel test.
  • Default main implemented.
Useful documentation:

Testing the test and mock framework

The source code is located at https://github.com/atejeda/cppt-eval, the repository contains a Makefile to download and build all the needed dependencies for a RHEL6.x environment in a NRAO machine with access to the CASA binaries.

The Makefile provide the targets:
  • google: download, build and install google mock and google test 1.7 in a local directory (relative to the cwd).
  • google.all: download and build google test and google mock.
  • google.install: install google mock and google test into include/google and lib/google.
  • google.clean: remove all the google test and google binaries.
  • all: build the test sources using google mock and google test.
  • clean: remove all the already binaries using google mock and google test.

By just executing "make google all" will build everything. Within the NRAO network, at least at AOC get a symlink of CASA data, otherwise the test will fail, e.g.: "ln -s /home/casa/data/trunk ~/data". If you CASA installation is different the Makefile variable CASA_ROOT should be update it as well, by default is pointing to a local (NFS) CASA installation within the AOC NRAO network.

In order to run the test, point LD_LIBRARY_PATH to CASA libraries and google libraries.
  • LD_LIBRARY_PATH=/home/casa/packages/RHEL6/release/casa-release-4.3.1-el6/lib:lib/google test/<testname> --gtest_output=xml [--help]
The command above will execute a test by mocking a class and execute a test by using google test.

Alternatives reviewed

The review in the table details the important aspects considered in the decision making to not use the framework.

c++ frameworks

All of these frameworks evaluated cover pretty much the basic needs: test fixtures, test suites, assertions and a basic reporting system.

framework pros cons
catch
  • just a header is needed
  • default main test
  • multi-report options
  • macro test based
  • macros definition collides with google-mock
  • not a huge community
  • developed and maintained mainly by one developer
xUnit++
  • written entirely in c++11
  • time limit
  • written entirely in c++11
  • developed and maintained mainly by one developer
  • last commit was 11-2014
  • not full LLVM support
  • vague documentation
CxxTest  
  • code generation from test headers
  • test need to be defined in a class
  • 50 issues opened to the date
Hestia  
  • no or vague documentation
boost.test
  • very well maintained
  • huge community
  • large binaries and compilation time

Mock frameworks

All of these frameworks evaluated cover pretty much the basic needs: expect, order and cardinality.

framework pros cons
mockitocpp  
  • no official llvm support
  • 2 years of no development or bug fixing
mockator  
  • binaries depends on boost
  • strong integration with CUTE and Eclipse CDT
mock++/mockcpp
  • covers pretty much the same as google mock
  • last commit was on 2012
  • not sure if is compatible with other test frameworks
opmock  
  • strong dependency on LLVM

-- AlexisTejeda - 2015-05-18
Topic revision: r5 - 2015-06-02, AlexisTejeda
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback