Data Vault Status

Background and Introduction

As NRAO revamps its web presence, the Data Vault project aims to deliver a single portal to web pages and services related to NRAO data products, complete with a single ("Google-like") free-text search. Data Vault development refers to research and development efforts on the web application and user interfaces which combine all telescope data products into a single entry point.

Archive infrastructure and storage issues, while of strong interest to the development of the Data Vault, are beyond the scope of this status primer.

Recent News

  • Dec 2008 - Data Vault beta available for testing and feedback (survey linked to page).
  • Feb 2008 - Nicole and Ron visited Google to collaborate on Google Sky, which has strong potential as a visual entry point to NRAO data products.
  • Apr 2008 - New Data Vault beta with integrated cross-telescope searches is available at http://archive.cv.nrao.edu/beta/ (which later was published to http://archive.cv.nrao.edu/).
  • May 2008 - Data Vault is released to http://archive.cv.nrao.edu/ for a public beta.
    • users can search for NRAO data with a single free-text search box.
    • search results are linked to various services, including downloads, proposals, scheduling blocks (GBT), Google Sky, and image searches.
    • familiar tools from the NRAO Data Archive System are linked to the Data Vault front page.
    • feedback can be submitted through the linked survey.
  • 28 Jun 2008 - NRAO presents Data Vault Software at SPIE Astronomical Instrumentation, Advanced Software & Control for Astronomy.
  • Aug 2008 - Summer student Bryan Murphy completes a semantic search prototype for Data Vault.
  • Aug 2008 - Summer student Brian Sacash finishes summer work investigating data archival of telescopes 140ft and 12m.
  • Jan 2009 - Data Vault begins refactoring for how to handle download requests for single dish data.
  • Mar 2009 - Data Vault Lite pilot released at http://archive.cv.nrao.edu/download/, providing direct access to GBT raw data.
  • May 2009 - Data Vault Lite (http://archive.cv.nrao.edu/download/) has an index of archived VLA/EVLA, VLBA, and GBT raw data (pages are available for projects from any of these telescopes). This is a first step in providing one-stop, direct access by project ID to archived data from any NRAO telescope. Right now, GBT data is available for immediate download, and others are redirected to the Data Archive System hosted at the AOC.
  • Aug 2009 - Investigating Solr for building a better search index of data collections.

Current Status

  • Completed in Dec 2008 - Release Data Vault (Archive v2.0) Beta.
  • Completed in Apr 2008 - Complete a new Data Vault (Archive v2.0) Beta (demonstrating integrated Google-like search query, results, and download of VLA, VLBA, and GBT data), and deploy for viewing and early testing.
  • Completed in May 2008 - Complete a release candidate of Data Vault and deploy to http://archive.cv.nrao.edu.
  • Completed in May 2008 - Document database schema and diagrams (ERD) for the archive metadata/registry.
  • Completed in May 2008 - Propose & support deployment of Data Vault to http://archive.nrao.edu.
  • Completed in Mar 2009 - Release a direct downloader (i.e. no search needed, just project ID) for GBT raw data, with support for authentication when accessing proprietary data.
  • On-going - Build a search index of historical proposal cover sheets.
  • Goal for ??? - Include 12m and 140ft data in the production single dish archive node (requires capability to reduce data).
  • Goal for ??? - Install the Spectral Line Search Engine (SLiSE) to query the single dish spectra database.
  • Goal for ??? - Develop pipeline to generate calibrated sdfits data.
  • Goal for ??? - Develop pipeline to generate preliminary calibrated, averaged dataset
  • Goal for ??? - Support semantic & categorical searches of archive contents (e.g. molecular clouds, binary pulsars, etc).
  • Completed in 2010 - Completely redesign platform and interface for searching telescope data. This is NRAO Blacklight. Use it.

Discussion

  • Snapshots in development are available for viewing and early testing at link below.
  • There is on-going development to seamlessly integrate proposal information into the data download.

Insert or link here discussions on the status and direction of this project.

-- RonDuPlain - 03 Mar 2008 - 04 May 2009
Topic attachments
I Attachment Action Size Date Who Comment
SPIE08-DataVault-preprint.pdfpdf SPIE08-DataVault-preprint.pdf manage 459 K 2008-05-26 - 17:19 UnknownUser Data Vault: providing simple web access to NRAO data archives
Topic revision: r15 - 2010-09-29, RonDuPlain
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback