LITTLE THINGS (Local Irregulars That Trace Luminosity Extremes The HI Nearby Galaxy Survey)

The VLA Large Proposal LITTLE THINGS (AH927) has completed its observations, and we are now entering the data calibration and mapping phase. There are 8 or more team members who will be working on the data during this phase, and they are located in the UK, Germany, and across the US. As each person begins working on a dataset, they will download the uv data from the NRAO archives, calibrate the data, and put those data on the team data repository, where someone else may pick the files up to take them through the next stage, and so on. Currently the team data respository is a network RAID disk system on the Lowell Observatory network. Lowell has one T1 line. In anticipation of the coming data crunch the Lowell system's person added several DSL lines. But, a few weeks ago we held a team meeting in Socorro and at one point needed to download one uv dataset from the Lowell ftp site. This one data set took 5 hours to download, and then it was corrupted and had to be downloaded a second time. So, we realized that using Lowell as the main team hub will not be feasible. Thus, we are asking NRAO if they would be willing to serve as the primary team hub where we put and get data as we work on them.

We estimate our networking and storage needs as follows: uv files are typically 1-3 GB. Combined uv files will probably be 4-6 GB. Map cubes are of order 0.5 GB. The total data volume is estimated to be 4 TB. The volume will ramp up over the next year, and then increase slowly over the remaining 3 years.

Movement of data will also ramp up over the next year, probably peak for a year, and then decline. During the peak, the movement of data (getting and putting) will be on average roughly about 8 GB per week per person with about 8 people working on data. In addition I would like to back the data (changes) to Lowell once a week. Co-Is at MPIA-Heidelberg and FIU are looking into acting as mirror sites as well.

One of the requirements of the Large Proposals is that we make the data products available to the public long-term. Currently the public web site is resident on the Lowell web site, and certain ancillary data are now populating that site. The team expects to release the HI data cubes and moment maps to the public at the end of 2009. At that time the Lowell web site will no longer be adequate for serving the public. We would like, therefore, to move this service to NRAO by the end of next year. Unlike the team web site, however, hosting the data products for the public is a long-term function. We would expect there to be international downloads at a lower but continuous level compared to the team requirements.

Link to project home site

-- DavidHalstead - 2009-05-05