NAASC Data Services Group Wiki
Current Tasks
Open Action Items
Group Charter:
The Data Services Group (DSG) coordinates the processing, archiving and distribution of ALMA data at the
NAASC. The DSG also coordinates the deployment of the
NAASC instance of the ALMA User Portal, and the provision of the ALMA helpdesk to the ALMA project. We coordinate with the NRAO Computing and Information Services division to ensure the provision of the hardware necessary to undertake the data transfer, processing and storage tasks, and to ensure that software environments are set up correctly to allow the pipeline and associated software to function efficiently. We also work closely with the Virtual Observatory (VO), its US incarnation, the Virtual Astronomical Observatory, and the ALMA archive development group to ensure the transparent integration of ALMA data into the VO.The
NAASC Software Group (NSG) overlaps with the DSG. The NSG is responsible for the development and maintenance of tools for proposal preparation, helpdesk, data processing and data analysis that are produced in North America, but are not part of the main ALMA project. These tools supplement the ALMA project software development effort. Currently the NSG is responsible for Splatalogue and also directs the effort to develop tools to analyze the large datacubes ALMA will produce.
Archive/Science Portal
Hardware
Current specs for desktop systems:
DesktopSystemSpecs
Using the
NAASC Cluster and the Lustre filesystem:
Lustre and Cluster page
Instructions for the cluster scheduler:
NAASC Cluster scheduler (nodes 9-24)
Nodes 1-8 are scheduled using Google Calendars (see Lyndele to be added to these)
James Robnett's NRAO HPC wiki is linked off the menu to the left, and contains documentation on Lustre
NAASC/JAO 8-node cluster specs:
ALMA-Pipeline-Cluster-specification.pdf
Benchmarking tests
Page to help define benchmarking tests by identifying common bottlenecks
Splatalogue
WEEDS -
http://www.aanda.org/index.php?option=com_article&access=standard&Itemid=129&url=/articles/aa/abs/2011/02/aa15487-10/aa15487-10.html
and
http://iram.fr/IRAMFR/GILDAS/doc/html/weeds-html/weeds.html
Comments: Part of GILDAS (as of Dec 2010), uses SLAP to access JPL and CDMS line databases in an online manner, but can cache portions of these catalogs for offline use. The connection to splatalogue is under development (April 2011). Uses LTE to model faint lines of common species (the "weeds").
XCLASS
Comments: Compiled on top of GILDAS, uses JPL/CDMS databases plus a simple LTE model. Requires a mysql server to be set up on your machine. Based on old version of CLASS, and is essentially superseded by WEEDS, except that WEEDS does not support a private catalog, as XCLASS does. Todd has it installed on his home Linux machine.
CASSIS -
http://cassis.cesr.fr/
Java application, builds ~1 GB SQL database of lines on your machine, has various models to predict spectra including LTE and RADEX (including collisional rates for a couple dozen molecules). Todd has this installed on his Linux workstation and Mac laptop.
VADMC http://www.vamdc.eu
European molecular and spectral line database - still seems to be in early stages,
Simdata
Server and Network Information / Projects
Pipeline
Pipeline setup page for NAASC (updated regularly):
http://wikis.alma.cl/bin/view/DSO/NAASCPipelineSetupInstructions
Tools and scripts (mostly historical - see pipeline page above for recent info on the pipeline itself)
Image Cube Visualization and Analysis
Requirements
- Platform independence
- Ease of install
- 3D real-time visualization
- Spectral line visualization
- Spectral line fitting
- On-the-fly spatial and spectral smoothing
- VO interface
- Maximum cube size limited by hardware
- Automated object/feature detection and characterization
Requirements matrix for image cube analysis s/w
Links:
GAIA -
http://star-www.dur.ac.uk/~pdraper/gaia/gaia.html
Karma -
http://www.atnf.csiro.au/computing/software/karma/
SDvision -
http://irfu.cea.fr/Projets/COAST/visu.htm
Clumpfind -
http://www.ifa.hawaii.edu/~jpw/page16/page4/page4.html
QFITSView -
http://www.mpe.mpg.de/~ott/QFitsView/
VisIt:
https://wci.llnl.gov/codes/visit/
Package | Ease of install
|
3D visualization
/rendering* | Object finding
| Spectral line fitting
| On-the-fly smoothing
| VO interface
| *Max Cube size | Support |
kvis/xray (karma)
| Poor - only in /usr/local, requires root access | Y | N | N | Y(xray) | N | Up to RAM, also has decimation capability for larger cubes | Individual/none? |
| | | | | | | | |
sdvision | IDL - easy but proprietary | Y | N | N | ? | N | ? | ? |
| | | | | | | | |
Gridview | IDL - easy but proprietary | N | N | Y | Y | Y | ? | Brian Kent |
| | | | | | | | |
GAIA
| Fair on linux, some environment variables to set, but no root access required.Mac slightly more difficult(?) | Y | Y(2D) | N | Y | Y | 2Gpixel with 64bit version (may be able to improve) | Institutional (JACH), but long term uncertain. |
CUPID (starlink)
| As GAIA | N | ? | ? | ? | ? | | As GAIA |
| | | | | | | | |
Clumpfind | IDL - easy but proprietary | Y | Y | ? | ? | N? | | Individual |
| | | | | | | | |
QFITSView | Easy | N | N | N | N | N | ? | Individual |
| | | | | | | | |
DS9 | Easy | N | N | N | N | Y | ? | Institutional (SAO) |
CASA | Easy | N | N | Y (3.1) but non-interactive | N | N | ? | Institutional (NRAO) |
VisIt | Fair (need to set environment variable) | Y | N | N | Y(?) | N | Arbitrary- can be run on desktop or server | DOE (LLNL) |
| | | | | | | | |
| | | | | | | | |
Meeting minutes:
Notes From the Advanced Tools Developer Discussion
Notes From the Advanced Tools Developer Discussion 2
Group Meeting Minutes
Meeting minutes