NAASC Data Services Group Wiki

Current Tasks

Open Action Items

Group Charter:

The Data Services Group (DSG) coordinates the processing, archiving and distribution of ALMA data at the NAASC. The DSG also coordinates the deployment of the NAASC instance of the ALMA User Portal, and the provision of the ALMA helpdesk to the ALMA project. We coordinate with the NRAO Computing and Information Services division to ensure the provision of the hardware necessary to undertake the data transfer, processing and storage tasks, and to ensure that software environments are set up correctly to allow the pipeline and associated software to function efficiently. We also work closely with the Virtual Observatory (VO), its US incarnation, the Virtual Astronomical Observatory, and the ALMA archive development group to ensure the transparent integration of ALMA data into the VO.The NAASC Software Group (NSG) overlaps with the DSG. The NSG is responsible for the development and maintenance of tools for proposal preparation, helpdesk, data processing and data analysis that are produced in North America, but are not part of the main ALMA project. These tools supplement the ALMA project software development effort. Currently the NSG is responsible for Splatalogue and also directs the effort to develop tools to analyze the large datacubes ALMA will produce.

Archive/Science Portal

Hardware

Current specs for desktop systems:

DesktopSystemSpecs

Using the NAASC Cluster and the Lustre filesystem:

Lustre and Cluster page

Instructions for the cluster scheduler:

NAASC Cluster scheduler (nodes 9-24)

Nodes 1-8 are scheduled using Google Calendars (see Lyndele to be added to these)

James Robnett's NRAO HPC wiki is linked off the menu to the left, and contains documentation on Lustre

NAASC/JAO 8-node cluster specs:

ALMA-Pipeline-Cluster-specification.pdf

Benchmarking tests

Page to help define benchmarking tests by identifying common bottlenecks

Splatalogue

  • www.splatalogue.net

WEEDS - http://www.aanda.org/index.php?option=com_article&access=standard&Itemid=129&url=/articles/aa/abs/2011/02/aa15487-10/aa15487-10.html

and http://iram.fr/IRAMFR/GILDAS/doc/html/weeds-html/weeds.html

Comments: Part of GILDAS (as of Dec 2010), uses SLAP to access JPL and CDMS line databases in an online manner, but can cache portions of these catalogs for offline use. The connection to splatalogue is under development (April 2011). Uses LTE to model faint lines of common species (the "weeds").

XCLASS

Comments: Compiled on top of GILDAS, uses JPL/CDMS databases plus a simple LTE model. Requires a mysql server to be set up on your machine. Based on old version of CLASS, and is essentially superseded by WEEDS, except that WEEDS does not support a private catalog, as XCLASS does. Todd has it installed on his home Linux machine.

CASSIS - http://cassis.cesr.fr/

Java application, builds ~1 GB SQL database of lines on your machine, has various models to predict spectra including LTE and RADEX (including collisional rates for a couple dozen molecules). Todd has this installed on his Linux workstation and Mac laptop.

VADMC http://www.vamdc.eu

European molecular and spectral line database - still seems to be in early stages,

Simdata

Server and Network Information / Projects

Pipeline

Historical information for running the pipeline, including presentations given in Charlottesville, May 2012:

Data services pipeline page

Pipeline setup page for NAASC (updated regularly):

http://wikis.alma.cl/bin/view/DSO/NAASCPipelineSetupInstructions

Tools and scripts (mostly historical - see pipeline page above for recent info on the pipeline itself)

Advanced tools

Image Cube Visualization and Analysis

Requirements

  • Platform independence
  • Ease of install
  • 3D real-time visualization
  • Spectral line visualization
  • Spectral line fitting
  • On-the-fly spatial and spectral smoothing
  • VO interface
  • Maximum cube size limited by hardware
  • Automated object/feature detection and characterization

Requirements matrix for image cube analysis s/w

Links:

GAIA - http://star-www.dur.ac.uk/~pdraper/gaia/gaia.html

Karma - http://www.atnf.csiro.au/computing/software/karma/

SDvision - http://irfu.cea.fr/Projets/COAST/visu.htm

Clumpfind - http://www.ifa.hawaii.edu/~jpw/page16/page4/page4.html

QFITSView - http://www.mpe.mpg.de/~ott/QFitsView/

VisIt: https://wci.llnl.gov/codes/visit/
Package Ease of install

3D visualization

/rendering*
Object finding
Spectral line fitting
On-the-fly smoothing
VO interface
*Max Cube size Support
kvis/xray (karma)
Poor - only in /usr/local, requires root access Y N N Y(xray) N Up to RAM, also has decimation capability for larger cubes Individual/none?
sdvision IDL - easy but proprietary Y N N ? N ? ?
Gridview IDL - easy but proprietary N N Y Y Y ? Brian Kent
GAIA
Fair on linux, some environment variables to set, but no root access required.Mac slightly more difficult(?) Y Y(2D) N Y Y 2Gpixel with 64bit version (may be able to improve) Institutional (JACH), but long term uncertain.
CUPID (starlink)
As GAIA N ? ? ? ? As GAIA
Clumpfind IDL - easy but proprietary Y Y ? ? N? Individual
QFITSView Easy N N N N N ? Individual
DS9 Easy N N N N Y ? Institutional (SAO)
CASA Easy N N Y (3.1) but non-interactive N N ? Institutional (NRAO)
VisIt Fair (need to set environment variable) Y N N Y(?) N Arbitrary- can be run on desktop or server DOE (LLNL)

Jeff Mangum's compilation of CASA viewer requirements

Meeting minutes:
Notes From the Advanced Tools Developer Discussion

Notes From the Advanced Tools Developer Discussion 2
Software development meeting webpage

Group Meeting Minutes

Meeting minutes

Topic attachments
I Attachment Action Size DateSorted ascending Who Comment
pipe.overview.naasc.oct2010.pdfpdf pipe.overview.naasc.oct2010.pdf manage 7 MB 2010-10-27 - 15:35 RemyIndebetouw  
P2P_Initiative_For_NRAO_-_2011-03-03-1200.odtodt P2P_Initiative_For_NRAO_-_2011-03-03-1200.odt manage 82 K 2011-03-04 - 09:46 MikeHatz P2P Overview For NRAO
ALMA-Pipeline-Cluster-specification.pdfpdf ALMA-Pipeline-Cluster-specification.pdf manage 514 K 2011-08-16 - 10:22 MarkLacy 8-node cluster spec
This topic: ALMA > WebHome > NAASC > DataServicesGroup
Topic revision: 2012-08-23, JaredCrossley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback