Common CASA Statistics Framework

Background / Introduction

Recent work evaluating new filters and algorithms for robust statistics in imstat() revealed similar needs for robust statistics other CASA tasks that could benefit from the proposed improvements. Issues also became apparent when debugging visstat() segmentation faults, which were due to how the task accumulates the whole data cube in memory leading to a lack of resources.

While these issues are independent, they suggest a need for a revised common statistics framework in CASA. The possible benefits of a unified statistics framework are believed to be:
  1. The availability of a robust statistics framework that can process large data sets for any CASA task or module.
  2. The availability of new algorithms for robust statistics for any CASA task or module.
  3. Reduced long-term maintenance effort across portions of the code base that use the new framework.

However, to realize these benefits, it is critical that:
  1. The new framework be well designed and flexible in its architecture. It must be based on input from team members who understand the current statistics issues and they must actively participate in the design process.
  2. The new framework must be the only one used by developers going forward with new statistics tasks.
  3. Existing statistics code is rewritten to leverage the new framework (rather than maintain the various existing implementations).

Scope

The scope of this effort would be to create a revised common CASA statistics framework that would be used by all tasks and modules that presently generate statistics on a data set. The framework must be sufficiently flexible to accommodate large data sets and incorporate the statistical functions used by the following tasks:
  • calstat
  • imstat
  • sdstat
  • statwt
  • visstat

The framework must also support the following modules which generate statistics on a data set:
  • Imager
  • Pipeline

The scope of the effort includes:
  • Requirements capture for the common statistics framework.
  • Architectural design of the new framework.
  • Detailed design and implementation of the framework.
  • Testing of the new framework.
  • Updating existing tasks and modules to use the new framework.
  • Testing of the modified tasks to ensure expected operation.

The requirements and architectural design would be recorded and archived (in either the wiki or svn) for future reference by the project team.

Meetings

Technical Materials

Topic revision: r4 - 2014-08-19, RobSelina
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback