-- MarkLacy - 2010-09-07

Meeting minutes:

2012-05-21

Software:

3 development study proposals accepted with software or implications for software: - ALMA data rate increase (BG,ML) - Visualization/analysis on server side (Rosolowsky/Goodman) - ALMA software tools (Mundy, JK and AL also involved)

AL and ML will meet with Susan while we are in Socorro

Redeployment of SV data with CASA 3.4: datasets pretty much ready, Mike and Crystal will coordinate deployment in NA and to JAO and other ARCs

3 interviews for CASA performance hire last Friday, 2 more this week

Operations:

Portal: dev version installed (later found to have missing blobs) - need to check CASA links Kelly to fix table sorting issue in Plone (turned out to be a bad sorting app) Conection to DB - helpdesk will need to be able to connect to JAO user database from Socorro if Cville is down. Mike and Kelly to investigate

Japan meeting highlights (Mike) - unify Oracle/NGAS docs across ARCs (Mike to lead) - formed a group for Oracle DBAs, look for a unified point-of-contact at Oracle for all ALMA issues. Oracle patches to be implemented ~2x/yr

-Oracle licences: need to figure out how many STEs there are using Oracle licences

DB is now 550GB, ML will talk to doug about best thing to do about mirroring to Socorro for VAO testing.

- need access to dev database, Mike to talk to Alasdair

- P2P - NAOJ very interested for reasons of transfer speed and disaster recovery

Lustre: Dan will add extra node to NAASC-Lustre at the end of the week, also working on CV-Lustre after Lustre training this week.

Proposal call - requested Kelly look into pulling all email addresses from database in case of proposal emergency

2012-04-30

Operations meeting, J. Parra visiting from JAO

SP upgrade:

Kelly has version on test server, will work today on installing on production server, with different port

Portal still authenticates to Chile (no CAS servers at the ARCs), but reads roles, etc from local DB instance

Kelly will send out status at COB today, still hoping for a Wednesday deployment as Mike, Kelly out of town next week (in Japan for helpdesk meeting 7-8th).

Helpdesk solution was tested in IT5, should work in production. Registration and profile updates will not work until brought back up at JAO on Wednesday

Registration page does not die prettily, need to have an explanatory page and helpdesk news item added.

Pipeline:

Prep complete for Jeff's visit including 32bit machine. Data retrieval from archive tested.

Data transfer discussion with Jose:

Latency problem with EU (300ms)

Trying to improve software as a first step, hardware options possible too ("Riverbed")

JAO already planning to make next datasets smaller

Meeting to discuss Oracle licenses tomorrow, try to get a whole ALMA deal

Also will discuss alerting system between ARCs and JAO for downtimes, NGAS issues etc.

2012-03-12

Software meeting:

Visualization - Adam has a draft questionairre on vis/analysis needs for comments, will send out to 4-5 interested parties when done. Will include piece on producing publication quality figures. SEII hire proceeding, offer letter should be sent out this week. Devloper could start before the end of the month, so identifying priorities a priority.

CREATIV status - Andrew G. sent round a computing section, ML and DH working on theirs for the 3-5 pager. Robin P. has kindly agreed to review.

Single sign-on: ESO plan is vulnerable to security breaches - need to authenticate at both sites before establishing a link between accounts. Single-sign-off probably best accomplished with a timeout.

2012-02-27

Software meeting:

Kelly reported on trip to Chile, suceeded in kicking off SSO discussion. Working on NRAO portal too. ALMA will not move to SSO level-1 (ARC-based) until after cycle 1. Bhola is taking over as software engineer for portal after Cesc leaves ESO in 2-3 months.

Visualization:

ML combined EVLA and ALMA requirements. Missing from both is a requirement that vis. tool handles cubes larger than memory.

Discussion with Remy, Jeff M., Dave M., Adam:

Unclear if current CASA viewer could be adapted for large datacubes. Propose we use new developer to spend 2-3 months working on "quick fixes" to CASA viewer and learning CASA at the same time. Would then be in a good position to assess how much work CASA viewer would need to meet requirements that we are writing. CASA viewer needs to be more modular, separate analysis and rendering from the canvas (Remy). Also need to react to possible software dev awards, but not spend too much time on them until approved and funded by ALMA project. Also Darrell needs to be involved in discussions.

Interim solution using IDL is probably not useful as IDL too slow to deal with large cubes.

AIs:

*Adam/USG - use cases for large cube visualization

* Me - write requirements for dealing with large datacubes > memory

* Me - ping Alyssa and Eric R. for progress on adapting non-astronomy viewers

* All - look at requirements from EVLA and ALMA and identify high priority missing functionality

Splatalogue:

Dong-Chan working on changesto quick picker page, ~2 weeks out.

working on RFI DB and GBTIDL

All the pieces for datasplat in place, expect developer to come up with work plan soon.

2011-02-13

Operations meeting:

Project 454 untarring wierdly, will check before sending to user

Helpdesk failover to Socorro worked well apart from a couple of minor issues and a need to punch a hole in the firewall. Now believed fixed for next time.

Data delivery procedure: ML to review wiki

2011-02-07

Future of DSG:

Splitting into operations and NAASC software. ML continues to lead both groups, reports to Al for operations, Brian Glendenning for software. Operations and software-focussed meetings will be held on alternate weeks to reduce the number of meetings most people will need to attend.

Splatalogue:

Now has a new front page, courtesy of Dong-Chan

Waiting on Paul's statement of work to convert front end from PHP to Javascript, this will include e.g. LTE spectral calculations.

Computing:

Close to release of cluster scheduling software.

SP:

Kelly travelling to Munich Feb 18-20 to work on single sign-on

Tony going out to Munich at end of March to discuss helpdesk and to Japan May 7-8th for a helpdesk managers meeting

Archive:

NGAS update to v4.15; data delivery via ftp in meantime

2011-12-19

SV data for M100 pre-release is here (on my Lustre disk)

Aaron Boley visit: no dates yet (coordinate with Corder).

Policy for visits to remote sites (GB. AOC, HIA): visitor will get a Lustre account and can VNC in - DAs will bundle up results of visits and put on disk to mail to PI (with return label)

Archive replication now 100% caught up, as is mirroring. Asked Mike to log outages when they occur

Data rates - need to consider entire data handling chain, cost out e.g. increased bandwidth/disk shipping and weigh against science gains. ML to write memo.

CREATIV white paper and supporting letters close to going in, UMd S2I2 proposal in last week.

JAO throttling links to ARCs to 20Mb/s each, no traffic prioritization.

2011-12-05

Splatalogue updates: new front page, web service to simulate spectra, VO services enhancement (datasplat, splatslap). Paul will come back with statement of work. Note Splatalogue management transferred to OSO.

Archive transfer rates from SCO lower than expected during first data delivery

Need to collect data reduction information in OT Phase II

Ed made special "NAASC enhance" products for 191.

2011-10-31

Archive/SP/Helpdesk

Backup plan for Oracle streams is for JAO to branch off a local mirror and then dump that once a week to the ARCs. ML sent email suggesting this to the powers that be.

Datapackerlite ready for retest

Working on putting HT Access stuff for user retrieval on Fatman

Hardware

HPC nodes 9-24 running

Simdata/CASA

CASA 3.3 expected this week, 4/5 simguides done, excluding Scott's new one.

2011-10-24

Archive/SP/Helpdesk

No progress on Oracle streams for the NAASC metadata replication - need a backup plan

Datapackerlite not working, sent back to Alasdair

Hardware

Compute nodes in, still waiting for network though

Dan making progress on node allocation program, need to define a "master node" to run code, netboot etc

Need to be sensitive to users running long jobs when their allocation expires. Will need e.g. all nodes for December workshop, so need to boot everyone off.

Simdata

~ 2 new simguides being written

3/5 simdata casguides updated

Splatalogue

Paul visiting Mon/Tues before Thanksgiving

Getting the spectral line viewer adn GB RFI info in.

Intl working group decided to use ISO linelist for >1THz

OT has problems with greek letters

Adv Tools

Brian was at a recent Mathematica conference. Wolfram alpha interested in astronomy databases e.g. splatalogue

GPUs and parallel programming also discussed.

2011-10-17

Archive/SP/Helpdesk

Mike back from Garching.

ASDM Extraction tool and oracle sync expected this week.

Portal-ready data extraction tool should be available by the end of the calendar year, but for internal use only

P2P for the archive getting a lot of attention (particularly for disaster recovery)

Bhola Panta visiting this week from NAOJ. Discussions will concentrate on early science archive data access (for which EU is happy to let NA and EA find our own solution) and SSO.

Kelly has LDAP lookup enabled from helpdesk such that a non-existant helpdesk user can be looked up in the ALMA LDAP and added to the Kayako databae automatically (useful for Contact Scientists)

Hardware

Looks like most, if not all, of the hardware has been delivered for the cluster expansion. Mike unpacking boxes.

Scheduling of the 24 cluster nodes: MOAB on TORQUE looks like the best solution. Front-end will need work, ML to spec. Tony Baer at NCSA? may be able to help.

Use NGC3256 scripts as a test-set.

Pipeline

No Updates

Splatalogue

Tony and Robin out, no updates

Simdata

No obvious problems with 3.3, Andy and Jared working through CASAGuides (manpower limited right now). More CASAguides may be made for simdata. Jared looking at SV casaguides (ML found possible issue with the uvcontsub2 -> uvcontsub change in N3256).

The mosaic problem in CASA is still open, if resolved expect a release by the end of the week.

Advanced tools

software workshop summary posted to https://safe.nrao.edu/wiki/bin/view/ALMA/ALMAScienceAnalysisSoftwareMeeting

Adam re-emphasized the need for use cases for many of these tools, can come out of cycle0 data. He also talked to Steve Myers and thinks that a path for incorporation of user scripts into CASA can be defined.

Jeff thinks that the stepwise approach to the ISM chemistry problem (viewer, WEEDS-like LTE fitter) followed by more complicated stuff is the way to go.

2011-09-26

Data management review planning:

Settled on dates 16-17th Nov (Wed/Thur)

Charges ~3 weeks prior, Dana will chair. DH and ML will begin high level planning this week.

Archive/SP/Helpdesk

Mike has script from Alasdair to get asdms from archive, not documented though

New set of Oracle patches for archive

Splatalogue

Paul coming out end of October

Pipeline

Still waiting on Jeff, Ed working on combining single dish and interferometry.

May want to feature heuristics for science-ready processing at review

Simdata

Jared working through guides, Scott and Jim Braatz will also try to break it.

Need to figure out what to do about cycle 1 configurations. Will the OT produce an

algorithm simdata can use?

Hardware

Prep for December workshop. Will have 24 nodes by then, suggest limit workshop

to ~48 people and have two share a desktop. Have ~10 desktops with monitors, will need another 14 or so.

Can probably run enough hard wires.

Benchmarking: Boromir wins so far, without parallel CASA.

Contributed scripts:

Remy pointed out that there are now a multitude of places contributed CASA scripts can be found:

Rob's repo
http://www.cv.nrao.edu/~rreid/casa/scripts/atoz.html

Miriam's EVLA casaguide
http://casaguides.nrao.edu/index.php?title=CASA_EVLA_Scripts

analysisutils.py - not publically avail but some should be

Adam's stuff
some on the tutorial
http://www.cv.nrao.edu/~aleroy/pytut/pytut.html

the forum
https://science.nrao.edu/forums/viewtopic.php?f=2&t=19

page on science.nrao.edu with other stuff
https://science.nrao.edu/facilities/evla/data-processing/contributed-scripts

Should try to have somewhere central to at least link to these sites. Could be

an OSO thing. Will discusss and revisit in a couple of weeks.

2011-09-19

Archive/SP/Helpdesk

Looked into extracting ASDMs from NGAS, not straightforward as ASDM directory structure not preserved. Alasdair & co working on it...

Plan for cycle 0 data delivery:tar files for each project to be produced at JAO, ingested into NGAS and replicated to the ARCs. ARCs will have to extract the file from NGAS (should be possible if we have the filename using the NGAS "retrieve" command) and deliver to user. Current default is to write disk and mail to user, may be able to use secure ftp for smallish (<~30GB) datasets. Likely target data for shipping the first disks is probably around the end of October.

(ML notes that 32GB USB drives are around $40 at Best Buy, 64GB though are more like $100, external hard disks start at $50 with 320GB drives and a range of attractive colors...).

Archive syncing now running - database queries performed every 10min or so.

Helpdesk - Kelly working on improving the auto-response for staff-initiated tickets and making the user-forum world readable.

Bhola Panta visiting next month from the EA ARC.

Simdata

Stable & test CASA 3.3 built over the weekend, including new simdata tasks simobserve and simanalyse. Global simdata task to be deprecated in next release. Andy and Jared working on updating guides.

Need to try to break it...

Splatalogue

Paul visiting the end of this week to discuss interface improvements

Pipeline

Developers in NM working on refactoring the pipeline to make it more modular and tweakable. Approx timetable is to have the core in place by the end of the year, start incorporating algorithms early next year.

Hardware

Benchmarking proceeding well, early results coming out, but will need a couple more runs to test reproducability before posting, especially as some results are a little puzzling.

ML will email Jeff to see if there is an "idiot's guide" to running CASA parallel.

Also need to understand better the kernel requirements and if CASA can run under RHEL 6

Waiting on the infiniband cables for the 16 new cluster nodes.

Other:

MH asked if CASA can run on data in a read-only directory (to save everyone copying their own version of the SV data). Apparently it cannot.

2011-09-12

Archive/SP/Helpdesk

(reported at 9:45am tag-up) Missing pictures in SP new item due to mis-configured link, corrected now. Would be better to have .zexp files directly.

outcome of ES data processing at data reduction workshop: recommendation that data be taken through calibration, and one target be imaged in one line per up to four spectral windows (or equivalent for continuum) and given to the user as a reference image (though unclear if all other targets/spws needed to be imaged or not). It is thought that the next step will be that Baltasar will try to arrive at a plan and circulate to the ARCs. ML will try to confirm with JEH later this week.

ML, DH, MH and KS to brainstorm ways of getting data out of NGAS.... (also Al etc need to get hold of CSV data, besides users).

Pipeline/hardware

Benchmark tests from Adam and Jarrod almost ready for primetime. Will test on Boromir, Gauss and a cluster node and list total time and top 5 bottlnecks.

Also will be worth trying full 3840 channel FDM cube to see just how painful it is without parallel CASA

Socorro working on scheduler for CASA parallel, unclear what the status is.

Software/misc:

phase-II prep - Skype tested and works OK....Still need to test webex

Splatalogue news

Robin reports:

"It is now required to select at least one field within the quick picker before clicking search. Additionally, there was a problem in which OI, OIII, and NII were not working in the quick picker and this is now fixed."

Simdata news

New simdata with separated observe and analysis parts will be on casapy-test soon (may be on active now), should try next week or so and see if we can break it.

Adv tools:

ALMA SW workshop progress

- reminder to ML to get draft agenda out!

2011-08-22

Archive/SP/helpdesk

New SV data deployed

Archive design:

- compression of image cubes - FITS files mostly don't compress too well unless heavily blanked. Compression on the fly is indeed probably too hard. Could copy image headers before compression if metadata harvesting a concern.

- Filenames: unique filenames a great boon for e.g. disaster recovery. Could use simlinks to link shorter filenames to a unique one.

Archive sync:

- problem last week was caused by UVa primary link (NLR or I2) being down for 3.5d (tunnel runs on primary link only). Back synching again, 20-25% done.

Helpdesk:

- Big Joe still in Cville, expect ship to Socorro soon.

-possibility of using Big Joe in Socorro to monitor link to UVa

Splatalogue

>1THz line discussions slow due to vacations

Updated SLAP, updated quick picker

Benchmarking

Adam working on scripts for benchmarking with SV data.

2011-08-15

Archive/SP/Helpdesk

Archive Synching: ready to go, waiting for EA

Friday SAR approved

No word on SV data.

Alvarro would like Kelly to help with testing tools for Plone 4 migration. Quanta group (who we work with) have a tool, but only for Python 2.5+, JAO still running Python 2.4

Splatalogue

Paul Frieden visit set for Sept 21-23

Minor updates continuing

Pipeline Software:

Filename format currently been discussed by Pipeline/Archive group. Unique filenames good and work well with NGAS (NGAS file ID can = filenameif filename unique). Can get long though depending on what's prepended (e.g. project code etc). Need to make sure no file name length issues in NGAS.

Benchmarking:

Antennae Band 7 takes a "good long time". (Plotcal is exceptionally painful). Proto-CASAguide available at http://casaguides.nrao.edu/index.php?title=AntennaeBand7

Data currently available from Adam:

/export/lustre/aleroy/Antennae
_Band7_UncalibratedMSandTablesForReduction.tar.gz

in addition you will probably want to grab the contents of

/export/lustre/aleroy/for_chile/

which has the antenna position tables, tsys plots, and bandpass plots for these data. In particular the antenna position tables are important.

DAs have cycles available to help with benchmarking tests.

2011-08-08

Archive/SP/Helpdesk

SP update 31st August, will involve a redeployment, distribution targeted for 26th August. Still not clear if that will include the SV data. ML to email Alison.

Archive synching - still beating on a few problems, should be ready to start soon, but all NGAS nodes need to be up and running.

Hardware

Node from Colfax arriving this week for testing and comparison to Dell nodes. Other vendors working on quotes.

Lustre/cluster management: restrict access to tiger team for now as 8 machines already getting booked up. Mike thinks CIS has enough availability for testing though. ML will get some VNC instructions from Dan to broadcast, and add Al.

Splatalogue

Tony will email his international group to decide scope of Splat beyond current focus on molecular lines at <1THz. Demand for >1THz and for atomic/ionic transitions has been noted, but need to keep database size and complexity reasonable.

Simdata

Tony reports some helpdesk tickets still languishing, really need to figure out a maintenance plan if Remy too busy. These tickets typically require code mods though, so beyond the scope of USG. ML will discuss with Remy on his return.

Pipeline

Still don't really know how job scheduling will be done, but OODT (JPL) still looking like the most promising technology. Probably just wait and see what comes ofut of Socorro. New hire for web interface to pipeline probably just needs python skills.

Other news

Mike's ESO trip is 1st-12th September

2011-08-01

Archive/SP/Helpdesk:

- archive sync: will be ready with updated software for archive replication within ~1d. Still getting a lot of disk failures, OK now but will be problematic as storage grows.Luckily disks under warranty. Database backups also running.

- archive ingest of user data. Users should be encouraged to submit their products, but no incentive unless covered by NSF grant. Staff produced products should definitely be archived.

- DB sysadmin hire - ad going out this week, could be used to build database for user data if necessary.

Hardware

-Order for 5xT7500 mid-range machines going out tomorrow. Boromir T3500 (sans SSD) is now the low-end machine. No sensible high end option exists. T7500s will be distributed between visitors and staff.

-Cluster: quotes out to 3 vendors including Dell and Colfax; Colfax coming in cheaper, buying an evaluation node to test.

- Benchmarking - Antennae dataset is about 65GB, should be a good test of a larger dataset.

Splatalogue:

Mass calculator and dump to CASA now done

Simdata

More minor bug fixes

Pipeline

*ML needs to write ad for pipeline scripting job

Ed or Remy should be on hiring committee

Socorro looking at OODT (JPL scheduler/workflow tool). Unclear it has any killer advantages though, may just make things more complicated. Get Butler to explain...

*Need to sit down with Butler,Glendenning and Kern to discuss how to schedule parallel CASA on the clusters

Ed and Remy seeing if existing pipeline tasks are adequate to reproduce a typical reduction recipe.

2011-07-25

Archive/SP/HD

Archive replication - storage nodes ready, Oracle getting there, new hardware configs working well

SP update (July 30?) - No news yet from JAO on exact timing. Dong-Chan will do once we get the new html from Alvarro. Mike will make more torrents for the new data.

Helpdesk - PHP upload limit increased, will reboot to implement

Splatalogue

Robin and Paul converging on mass calculator and dump to CASA, both near completion. Continuing minor updates to the database.

Hardware

Lustre install - functional 56TB system ready, will need to reboot desktop machines to implement kernel change. ML first volunteer, to be followed by CB, RI and SS. Unclear if Macs have a Lustre driver available.

Desktops: David will find out who already has what. Aim is that everyone gets at least a mid-range machine. Stuartt & Todd under consideration for high-end T7500. Have go ahead for purchase of T7500 as 2nd visitor machine.

Macs - need a high end Mac for benchmarking, Adam may be suitable guinea pig. Kartik's mac rumoured to be performing poorly, but unclear if this is an intrinsic problem or not.

Compute nodes - could be up and running soon, would be good to have with Lustre.

(from Mgmt meeting - rumour that Robnett has a benchmarking page for EVLA)

Software tools

Adam can post his scripts to user forum, but would be best if we can have a software repository of form world-readable, authenticated write to store these sorts of products, which could then be linked from forums, KB articles etc.

2011-07-18

SP/Helpdesk/Archive

Kelly continuing to work on requests arising from the Helpdesk WG

SP update procedure now on wiki here

Archive replication in "pre-production" phase, expect it to be working by the end of the month. Disk failures seem to have eased off following reseating of drives.

Big Joe traveling to Socorro in the near future

Splatalogue

Robin working on mass calculator and more database updates

Tony talked to Paul about updating Splat front end, will try to get him out for a visit in August.

Simdata

Remy fixing little bugs, will sit down with Scott and Jim Braatz in Augustto discuss documentation improvements.

Hardware

Lustre cluster nodes in and racked, expect availability soon.

New NGAS nodes getting 3TB disks

Need to talk to Scott about setting up a roster of available machines for scientist and visitor use once we have more than one.

David will go out for bids on cluster nodes, will take a while anyway, in the meantime Robnett may have more information for us.

Herschel advertizing remote cycles on large memory machine for help with analysing large datasets. ML will attempt to investigate further.

Phase-II support

John wants webex or Skype for Phase-II walk-throughs with the PIs. DSG will test and evaluate Skype vs Webex

Analysis software

CASA dev meetings - Dave M. asked to report on relevant material to DSG (though may not have expertize on HPC side, recommends we discuss directly with developers).

Adam Leroy demonstrated python task for finding and masking signals in data cubes, using scipy and numpy within CASA. Looks very promising. Agreed that we should suggest again to CASA that they consider making a "contributed software" page with these. Eventually python tools could be rewritten in C++ and moved into the full CASA release.

2011-06-06

SP - Kelly to update SP and OT installation instructions [including item to check NAASC->SP weblinks). NAASC-Ops ticket assigned.

Archive: Archive scientists meeting this weekend at SCO. Will address ASA development and planning. Still waiting on replication to restart after last week, but have 70% of SCO archive in Cville. Brian starts as staff on July 21st.

Helpdesk: several significant upgrades including searches on both the KB and SP, ticket ID searches, reporting and branding issues resolved or close to resolution.

Pipeline Hardware: master Lustre node in Cville, cables and rest of equipment on its way.

Splatalogue: The new quickpicker is available from today, starting with CO. More lines will be added.

Advanced tools:

ADASS meeting (Paris, November) - will send some people (good visibility for NRAO's data problems). NRAO contributes 2k/yr to ADASS. Mike's torrent downloads would make a nice poster or short talk.

ML getting together SOC for s/w development meeting. Unlikely to make first deadline for proposals (but unclear when that will be anyway). Suggest we shoot for early Fall.

2011-05-16

HD/SP/Archive:

Pre-release version of new Science Portal installed, still no agreed plan for rollout, but expected later this week.

ML to send around email asking Jim and Aaron to look at it.

CDR 9 inputs: ML and Felix working on archive subsystem and related inputs. Reminder to include single-signon for cross-affiliation.

NAASC Ops department helpdesk tickets to be prioritized.

Also helpdesk "tiger team" being set up to merge NRAO and NAASC/ALMA helpdesks. Current ALMA helpdesk solution seems to be working for the other ARCs (for example, separate helpdesks for each ARC with no visibility between ARCs seems to be OK). Tigr team would address:

1) User accounts, login associations, single sign-on

2) A unified ticket view for both users and staff

3) Ticket transfer

4) Knowledgebase unification

Adv tools:

M. McCarty (GB) Starting to look at VisIt code

Splatalogue:

Progress being made on "quick picker" - will be out for test soon, DSG will be first to test

Hardware:

James close to ordering EVLA, JAO, NAASC and maybe EU ARC hardware for 8 node, 1 lustre-node protocluster

Other business:

Archive replication - Mike still slogging through installation scripts, still on target for first replication tests this week

Data accessibility - NSF want statement from NRAO on data management and accessibility by the beginning of June. David to adapt Archive plan. Should include non-astronomy (e.g. education) too.

2011-05-09

HD/SP/Archive:

Archive - report on today's Archive policy meeting

Science Portal 2 - should appear on CVS on Wednesday. May need pgplot, not clear from description.

Helpdesk - special helpdesk meeting tomorrow

Pipeline/Hardware:

Need to be clear that NAASC can supply cluster cycles for reprocessing, e.g. with webform interface for remote job queuing, but observatory functions (heuristics developement, bulk reprocessing, QA3) take priority.

Hardware problem with drive controller on DB machines. OS and Oracle versions/patches now agreed on through project. Sync tests still planned for 16-20th May.

Advanced Tools:

Big data workshop in GB will recommend not storing all raw data, stream processing as much as possible. VisIt a useful visualization tool for large datacubes as is scriptable and can be run over the web on a large server.

Splatalogue

New guide to splatalogue posted. Some problems with the quick picker, version should be available for test 16th May. Dump2CASA needed ASAP.

2011-05-02

HD/SP/Archive:

Tony distributed list of change requests for helpdesk. A couple affect the setup:
1) Search on both knowledgebase and Science Portal information
2) Request for private departments for proposal handling

Kelly working on (1) already
Any new requirements need to go into a requirements document, and be visible for CDR9 (deadline for input May 20).

Each ARC should have an admin person and a backup to make portal changes

Simdata:

More collaboration needed between OST and simdata with respect to optimizing parameters etc. ML can try to discuss with Eelco

Hardware:

Getting quotes, expect to have proto-cluster/Lustre up by the end of July.

VPN tunnel to Chile up and running. Currently expect 20Mb/s shared, will increase once agreement for dedicated line in place.

2011-04-25

HD/SP/Archive

Helpdesk: Heidi Medlin visiting from Socorro this week. Need to meet up with Tony when he gets back to go through the results of the Garching meeting. For example, staff only dept for interacting with JAO and JIRA. Search function in helpdesk could be Google box pointed at specified ALMA websites could be used to search PDF-only documentation.

Helpdesk URLs - Kelly making progress, but need two more licenses (~$200 ea) and some of Pat's time. Agreed we should try to fix this within the next few weeks.

SP: ML and KS attended SP telecon last week. Raised issue of CVS tagging and SV data distribution (Alvarro working on it).

Archive: Upcoming telecon on caltable/flagtables in archive. Also will address WVR correction issue. Archive/pipeline telecon also planned for 2nd week of May.

For our reprocessings at the NAASC, would like to archive the results locally in the NAASC archive. To do so we would need version of datacube ingest software, with additional/substitute DB fields for QA, why the data was rerun, and if the results are better than the initial run at the SCO. Also may need to deal with non-pipeline user supplied data.

Hardware

David now has "final" specs from James for cluster machines. Will spend the $68k fast-filesystem + cluster node money. Cluster nodes are now dual 6-core machines costing ~$3600 each, charged to 782410. No need to update visitor machine specs yet though.

Software

Dan installed the latest version (0.21) of wvrgcal as

/opt/local/stow/wvrgcal-21/bin/wvrgcal

2011-04-18

HD/UP/Archive

Kelly and Tony back from helpdesk meeting with a long list of enhancement requests. No major bug fixes needed though.Tony, Kelly and David to get together in the next few days to figure out priorities.

Kelly has given Maurizio C. the demo code for SSO-level 2. New ALMA plan is to have SSO level 1 authenticate against JAO server with a backup (Socorro??) rather than replicate to the ARCs.

Meeting with Reuna May 2nd as part of process to set up VPN tunnel from SCO. Mauricio Pelleaux(?) from Santiago business office will be NRAO liason.

Plone 4 going live at NRAO next week. ALMA has CVS Plone 4 repository but still empty.

Advanced Tools

ML to ping Carol re job ad

Greenbank meeting: no firm word yet on NRAO invitees. David will try to find out. Also suggest likely participants may want to drop an email to GB to see if they have a room reservation!

Also wide field meeting at STScI in June - Amy K. going, others encouraged, particularly if not going to GB.

Splatalogue

Redeployment of web services still ongoing

Simdata

Crystal may ask Amy to help with checking simdata casaguides when stable 3.2 release candidate is identified.

Hardware

FY 2011 shopping list compiled. David would like to go ahead with getting quotes for fast storage purchases, will wait another week or two until James R. is out for exact cluster node specs before ordering those.

2011-04-11

HD/SP/Archive

Peer-to-peer now working, ready for testing

Can start to look at 1st SV datasets for size (e.g. Band 3 Antennae)

Outgoing datarates from CV:

400Mb/s (peak hours 6am-8pm)

750Mb/s (off-hours)

Incoming rate unknown, 100Mb/s would be great...

Adv tools

Now have some input on names. ML will collate and circulate.

SciPrg 2 draft ad sent to Carol

Simdata

ALMA ES configs into CASA 3.2

New Tsys values to be added, plus some workaround for extra noise from double-sideband in Band 9 (~ x sqrt(2), but exact number unclear, Remy will ask Crystal)

John requested reformatting of Simdata casaguides page, ML to work on that.

Splatalogue

NCSA now out of the SPLAT loop, NRAO now owns this service

Pipeline

Dan working on using Torque scheduler with CASA, seems to be possible to schedule jobs sensibly

Hardware

David released 10k for James to buy two 8-core, probably 24GB proto- cluster nodes.

Kartik requested more info on recommended desktops - now on the wiki at DesktopSystemSpecs

2011-04-04

HD/UP/Archive

NOI issue fixed - "mailto" address was set to localhost, needed to be reset to valid mail host

ESO having problems with helpdesk redirection, Kelly working on it

Simdata

Remy reports by email that new receiver temps etc will be in simdata for next CASA release

ES configs were sent round at the weekend - unclear if they are "blessed" and we can post. Remy will put them into CASA for next release (per email after meeting).

Splatalogue

SLAP VO service redeployed at NRAO - NCSA hosting was proving problematic. WEEDS have also requested some DB format changes and changes to the VO interface.

Adv Tools

Adam talked to Fiegelson at Penn State, overall positive, would be interested in being on SOC for software dev workshop

Adam will work on draft job ad for Sci Prog 2 this week

2011-03-28

Helpdesk/SP/Archive:

Awaiting OT RC5, also "final" pre-call version of the SP, due out in the next day or two

Plenty of disk space for SV data, John is campaigning for us not to serve the 30-60GB of simulations produced by ESO. In any case will not be needed until next major portal release, targeted for May 15th

Peer-2-peer almost ready to go - on tracker.cv.nrao.edu

Simdata:

Receiver temps for bands 6.7.9 and sideband gain ratio for band 9 recently changed in the sensitivity calculator (ESO one only, will take a few days to propagate to the SP). Also need to be changed in simdata, but no-one has the actual numbers. Receiver temps easy to change in simdata code, could be done by April 15 CASA release if we are told what the new numbers actually are. Sideband separation harder, not coded in simdata but could be approximated by increasing the band 9 noise. Remy will email Stuartt to try to get new numbers, Hibbard has sent email to SciOps IPT asking for clarification of which numbers are being used where, and if a place can be set up where these are stored.

Comparison to OST - looks good up to band 9, then diverges from simdata - simdata more likely to be right. Ian "working on" band 9 and 10 discrepancies.

Pipeline SW

Amy working on box clean algorithm. Ed visiting Edinburgh early May - this will be Ed only, not a mini pipeline summit.

Advanced tools:

After several meetings, it has been decided that the skillset required for the NAASC SciProg2 position should be concentrated on improvements to the CASA viewer to improve the interactive image analysis options in CASA. Adam will work on a draft job ad based on inputs from Jeff K and Carol.

Crystal and Jeff K. (and ML, indirectly) have been requested to organize a workshop on software development, will need to be within a couple of months to apply for NRAO seed money to develop proposals for the ALMA dev call.

Participants should include:

Penn State Astrostats group

Alyssa Goodman's group

UVa astrochemisty (CCU) group

Input for others welcome...

Python: Adam organizing a python for astronomers course, starting April 29th. See link at:

http://www.cv.nrao.edu/~aleroy/pytut/pytut.html

Summary of tag-ups 2011-03-23 - 2011-03-28

Helpdesk ticket numbering changed to sequential

Need for better version control and change documentation for SP (and to some extent OT) discussed - Kelly will raise issue during visit to ESO starting April 10th.

OT RC4 installed

2011-03-22 (Tag-up #1)

SP is now deployed at JAO (password protected) and ARCs. LDAP is "live, will not be wiped again. Documents should be added to the portal at JAO over the next few days.

Comments on the web portal content and style should go to Aaron

Helpdesk - may use a sequential ticket numbering system rather than current random one, but may be security issues. Will test sequential numbering over next few days and see how easy it is to hack.

2011-03-21

HD/UP/Archive:

Portal went live at NRAO, tested OT download successfully

"contact us" link on portal goes to helpdesk if logged in, webform is not. Webform sends email to NAASC triage + Kelly for registration issues. Not an ideal solution, will work on something better for cycle 1. Will file a JIRA ticket if there is a real problem.

Peer-2-peer file transfer. Still working on a few issues. B. Glendenning interested as a solution for data distribution from Chile. Would like a memo. There would thus be two scenarios: data distribution to ARCs from Chile, and distribution of "popular data" e.g. SV data from the ARCs.

Helpdesk URL - currently can look like NA to all ARCs. Will work on solution for Cycle 1

Splatalogue

International advisory committee formed, will meet in April or May

Robin + developer working on "quick picker"

Pipeline SW

Meeting with Ed on Friday

Pipeline HW

Purchasing 2 8-core protocluster nodeswith solid-state drives. James will evaluate in Socorro, testing with new CASA in Lustre/infiniband environment, then ship to us. Once specs finalized in a month or so we will buy the next six.

Adv tools

Meeting with Jeff K Wednesday 10am

~30 people interested in a "science with Python" course later this spring.

2011-03-07

The DSG charter is now posted at the top of the DSG wiki for comments

Simdata - Remy aware of doc requests, will work on any bug fixes after CASA code freeze

Pipeline SW - Remy now has latest version, will share with me and Amy when we return from our trips on 21st. Ed F. is back from Chile/NM and will talk to Lindsay about ALMA pipeline. John, Remy, Ed and I will meet to discuss pipeline issues sometime after this.

Pipeline H/W - ML will reply to Dirk Petry's email requesting further information on NAASC computing with our estimated purchasing schedule.

UP/Archive

Mike will get 60GB GBT dataset for P2P tests, probably by the end of the week, on "fatman"

ML to start daily tag-up meetings with deployment team 22nd March onwards.

"User Preference 3" LDAP field is confirmed to be available for SSO level 2.

Advanced Tools

Brian added link to DSG wiki with some GAIA/Cupid evaluations.

Other business

ML in Europe 9th-20th, but emailable most of the time.

2011-02-28

UP/HD/Archive

OT deployment - still expecting to serve the OT from the ARCs by the CfP. Registration and validation will be against SCO LDAP.

UP deployment - testing deployed UPs March 7-8, M. Lacy & J. Braatz will be testers from the NAASC

Peer-to-peer fileserving. Technical portion finished. Would be good to test with e.g. Tony's GBT data before going live to ensure any "security" problems are not showstoppers.

Hardware

Ordered first blade for fast filesystem. Looking to solicit bids for cluster nodes at the end of March.

Dan will be visiting Socorro next week. Should be able to begin defining cluster node configurations in about a month.

Visitor machines: still evaluating the results of Brian's tests, current bottleneck is in 4k CASA reads, so hard to evaluate true i/o performance once these are

removed. 4-node, 24GB RAM with 128GB SSD is still looking like a good base machine though.

Splatalogue

Tony has contacted other ARCs to request participants in Splatalogue WG

Pipeline SW/simdata - Remy out, so postponed until next week

2011-02-14

Splatalogue: versioning now working, Robin working on incorperating new NH3 data

Archive/UP/HD: data tagging at program level probaby adequate for ALMA, most people prefer a higher level view. Kelly will be working on Plone integration of NRAOwebsite.

Pipeline SW: Telcal - still only installable on 32-bit linux. Probably best to wait for Telcal CASA integration (Roche leading, ADASS poster)

P/l scripts - current version is relatively old, Remy trying to get latest set for tests. Once environment variables are set right should run in CASA OK

Pipeline HW:

Plan for serving SV data developing - will try to use peer-2-peer.

Would be good to have someone test the new visitor machine (Boromir) - Brian has volunteered (post-meeting).

Can turn around new machines fairly fast if strong demand for visitors to analyse SV data.

Simdata: CASA code freeze 15 March for 15th Apr nominal release date. Remy will try to upgrade simdata documentation at that point, has input from Scott S..

2011-02-07

Simdata/CASA:

*ML to chase up CASA system dependences with Jeff, try to get something posted to downloads page

Remy will deliver test script shortly for simdata for use prior to tutorials

Pipeline SW:

ML and Amy to write wiki on running wvrgcal and analysis tools. Analysis tools here but not useable yet, need to set up system

whereby e.g. Dan can check them out of SVN

Amy to get pipeline scripts from Remy. Aim is to demonstrate pipeline running in Cville S/W environment.

Pipeline HW:

Mike working on getting longer running datasets for benchmarking. CASA up and running on visitor machine.

Decision taken to buy first cluster nodes for SCO this summer.

SV data:

Distribution of SV data will be from ARCs. Expect 300-800GB, but mostly of raw data which few will download

Typical cube size more like 1GB.

Mike will write up peer-2-peer filesharing plan to reduce load on servers.

Splatalogue:

Functional leads for Splat at other ARCs being identified.

New webpage now live, with database version choice implemented

Would be nice to have LTE calculator, but needs development effort to speed up before it is practical.

IT3/Helpdesk:

OT distribution and proxying from other ARCs to the helpdesk need to be tested.

2011-1-24

Splatalogue:

Robin presented a poster at the AAS. Talking with CASSIS team and others. V2.0 of web interface nearly done, including "dump to CASA" button. Comments

from testing being followed up e.g. missing >THz lines in OT

Simdata

Segfaults found during tutorials were confined to Macs running 10.6, believed to be related to 64-bit/32 bit issues. A suggested workaround is to reboot holding down "6" and "4" to force 64-bit OS, but that didn't work when Crystal tried it. Remy will investigate further. Once we figure it out we need to at least post a knowledgebase article.

Also need to investigate possible issue with integer input images, and change defaults such that image=True

Casaguides: ML, Carol and Tony need to decide who is responsible for writing and maintaining the casaguides between DSG, USG and OSO. Would be good to have a simdata "idoits guide" for people who don't use CASA already.

Pipeline SW

Amy now working for DSG on pipeline scripts.

Need to install new version of WVRGcal and (ultimately) telcal's applywvr, along with the other telcal tasks outside of the ACS environment. ML sent email to David, Mike and Dan re support we can get from IT. Would be good to have everything on /opt/local once stable and tested.

2011-1-3

Splatalogue

Need to set up a list on the wiki of various websites and applications which also use the big molecular and atomic databases (action item for ML). CCU phase 2 application includes some radiative transfer effort, but very complex problem.

~1-2 weeks away from implementing the "quick picker". Other minor changes/upgrades continuing, including porting of lists to CASA

Simdata

No new material needed for AAS

Combining SD and Interferometry data probably doesn't work in the current release, bug fixes being worked.

Hardware

Boromir now ready for Hobbit action. Just need to know where to put him. Mike will ask Scott. Need to benchmark Solid state drives in particular. Wes Young (and eventually Darrell Schiebel) have regression scripts.

Pat has compiled shopping list for JAO

P/L Software

P/L developers waiting on work of which ES modes will need support. Not clear if JAO will use heuristics scripts in ES, or just use offline CASA. Remy looking at scripts.

Adv Tools

Could GAIA be supported by the EU ARC?

General support for the idea of inviting Peter Draper (GAIA developer) to speak at the Big Data GB meeting

Conference on data-intensive computing in San Jose June 8 - someone (Brian?) should go.

2010-12-20

Simdata

Oxford simulator - close to release. Remy volunteered to look at scripts to check for consistency with simdata, ML to send email to Heywood to request these and to reinforce that getting simdata, the Oxford simulator and the ITC to agree may not be trivial, and the need to ensure the configurations (especially for ES) are consistent. (done 12/21/10)

Scripting the new ITC for comparison to simdata: Kelly volunteered to help Remy with scripting the ITC to get out a grid of noise estimates to test simdata against.

Pipeline HW

Mike's trip to Santiago was very successful, he was able to complete his assignment and make several useful contacts

Benchmarking with CASA regression test - Darrell will know how we can checkout the CASA regression test for becnhmarking. It should contain

some non-trivial datasets for estimating speeds.

UP

We would like to start thinking about a common user database for NRAO and ALMA, and ways we can seamlessly integrate the NRAO and ALMA web experience.

2010-12-13

Archive/UP:

Summarized meetings last week regarding implementation of ALMA Science Portal. ESO wish to roll this out monolithically, with minimal local branding. Would be much better if ARCs could retheme the portal - not possible as written, but a correctly-written plone implementation should beeasily rethemable - this will allow anything from minor rebranding to complete change to match parent organization's website.

ML is writing policy for this, will include words on retheming in next version of document. Still unclear though if whole SP will even be in Plone, some internal debate within ESO on this.

Simdata:

Rick White of STScI is our liason for the JWST simulation effort where it relates to ALMA.

Oxford web interface to simulator making progress - ML to find out status.

Advanced Tools

Brian working on wiki page to describe GAIA installation and uses.

2010-11-29

Alison Peck visiting from JAO

Advanced tools:

ML reported on earlier telecon with Goodman,Pfister(CfA/Harvard), O'Niell,Shelton,Ransom(GB/CV), Lonsdale & Halstead about datacube analysis and visualization. Goodman et al. are considering a Cyber-Development Infrastructure (CDI) proposal to NSF (due Jan), would probably be aimed at a client-server based model for visualization of large (~10TB) datacubes. Also are applying for small Keck grant to fund human-focussed tools for smaller datasets. NAASC expressed interest, but would probably not drive this proposal. Also, due Jun 2011 is a "Sustainable innovation in software" proposal (SI2) for maintainable, open-source analysis software, can discuss in May "Big Data" meeting at GB.

Need to make sure ALMA data cube analysis is on agenda. Also some interest from the CCU.

*ML to check with Amy to see how science agenda is shaping up.

CASA plans and software need to be considered. Line fitting (non-interactive) is in 3.1

*ML will ask Dave M. to demo, or at least give him a quick tutorial. Unclear what long-term plans in CASA might be, would be relatively easy to have interactive line fitting, for example. Also AIPS++/CASA code does exist to do 3D rendering, but not documented.

Archive/UP/HD

Go live for HD and UP for IT3 is Dec 20. Likely open up to users in Feb, sometime after completion of IT3

Need URLs for helpdesk deployments.

UP policy is now part of archive policy.

Carol will investigate official NSF/NRAO policy re. open skies, in particular how it relates to support.

Archive timeline posted at http://almasw.hq.eso.org/almasw/bin/view/Archive/ArchiveDeployment2011

Simdata

New simulation of Vega disk via Rob Ried - may use to test SD

Narayanan simulation converted to FITS and run, awaiting 3D datacube.

Richard sent out new ES config file, Remy evaluating

Likely that antennas will be added to the array at the rate of ~2/mo, will probably cut observation times if >16 antennas available.

Pipeline SW

Remy will talk to Jeff and try to figure out how to sync pipeline and CASA releases so that pipeline uses current CASA rather than previous version.

Data capture etc may change during ES - need to have flexible pipeline scripts. Advertize as "NAASC data reduction scripts" rather than "pipeline" in ES.

Pipeline HW

Routers arrived for CV and SOC to improve network speed.

Visitor machine #1 arrived.

2010-11-22

Archive/VO/UP:

Bohla Panta visiting from NAOJ until Wednesday

Redundant server for helpdesk going to Socorro next week

ML to post notes on ALMA/VO meeting at ADASS

Splatalogue:

Tony and ML discussed offline list for OT with Stewart Williams, visiting for the pipeline meeting. A few minor updates. Lines >1THz for high-z objects now in, but currently need to specify a source redshift to find them in the OT interface.

Simdata:

New simulated datasets from theoretical models by Desika Narayanan (SMG) and Pak Li (ISM turbulence). ML will work on getting these simulated. Still unclear who "owns" the writing up (casaguides etc) between USG and DS. Doug Johnstone has also looked at turbulence simulations. Also need to look at simulating combined single dish and array data.

Remy will ping Oxford for status on their interface to the simulator.

Future enhancements for simdata will include simulation of observations off-transit, and including a user-provided gain table.

Benchmarking - need to look into datasets for benchmarking CASA, David will add to agenda for Dec HPC meeting.

Pipeline:

Remy will update pipeline meeting minutes at bottom of Agenda (linked off the main DS page)

Requirement to store 1Hz data rate for WVR correction implies larger data volume than anticipated, plus online corrected data may be stored alongside uncorrected data leading to a doubling of the data volume. This could erase the NAASC's data storage contingency.

WVR correction software will most likely be needed as part of the pipeline.

2010-11-01

Introduced Nuria Marcelino, new NAASC postdoc.

Splat: updates for the offline OT, pretty much converged on formatting issues

Archive/HD/UP:

archive telecon last week, Alasdair is agreed that a relational database is required for the science archive

Helpdesk - reports decided on, Kelly working on implementation, will discuss with ARC managers once in place

Simdata:

Dong-Chan and Remy working on updates to simdata casaguides

Also may need to update regular casaguides, and determine some metric (noise etc) which can be used to decide when an update is required as new versions of CASA are rolled out. Unclear whose remit this falls under, ML will bring up with management team.

ML, AK and SS to meet tomorrow 10am to work on doing phase screens in simdata

Pipeline:

Email Remy with topics for discussion with pipeline group on 18th Nov, so far:
  • Closure phase error flagging
  • Mosaicing recipes
  • Combining total power and interferometry
  • Implementation architechture for NAASC pipeline
  • Archive metadata and image previews.

Visitor machines:

Need to figure out whether to put more fast memory (flash cards) on desktops, or rely more on central processing with fast links to central fast filesystem. Will discuss more in Thursday's 11am archive h.w. meeting.

Adv tools:

ML has been evaluating GAIA. Looks good for visualization, but lacks line fitting ability and long-term future is in doubt.

2010-10-25

Splatalogue:

-finishing up OT offline version - "Lovas" type list satisifies requirements of spectral line working group

-versioning scheme installed, not tested

-RFI list for GBT and EVLA ready to be adde

Archive/UP/Helpdesk:

- Tony and Kelly have identified a couple of useful reports not produced by Kayako, will produce powerpoint mock-ups to discuss with John

Simdata:

- CASA release still "soon"

Pipeline:

Remy's pipeline walk-through Wednesday 1pm ER230

Advanced tools:

-Added "evaluation matrix" to the Data Services wiki to aid in discussions of available 3rd party software for image cube analysis. Still need to fill out some of the blanks/question marks. Comments/ideas welcome.

-ALMA-VO meeting at ADASS being scheduled.

2010-10-18

1) Splatalogue: Tony back from Chile, Robin working on offline database for OT, changes to offline CASA capability implelemented by Dave, will start work on "quick picker" interface by Dec 15th. RFI database begun (for GBT initially). Testing of versioning taking place in Illinois. Spectral line working group meeting to be scheduled after Crystal & Todd return from Chile on Nov 5th.

2) Simdata. Testing continuing, but no major problems identified. ML will circulate clean bias memo draft, and give Remy the new simdata page to proof read.

3) Helpdesk/UP/archive: John had some ideas for helpdesk mods in light of ORR, David to organize meeting. Need to discuss with Japanese ARC details of Japanese language implementation. Archive: science WG telecon last wee to discuss ORR outcome. Will come up with a plan for what needs to be in place for E2E test, and how to serve SV data. Request handler being released internally at ESO in December, ASA replication tests to begin in January.

4) Adv tools: Brian pointed out two BoF sesssions at ADASS, one on visualization and one on handling of large datasets.

5) Pipeline: Remy to give pipeline walkthrough on Oct 27th, 1pm ER 230. ACS etc believed to be working pending feedback from Al.

2010-10-11

1) Splatalogue - database modifications being made for OT line list. Development version of Splatalogue database discussed, agreed would be a good idea. Most of the issues with the CASA interface successfully worked through.

2) Simdata - still testing, code fixed so center of band used when calculating noise.

3) Pipeline - Remy will give a presentation on the pipeline near the end of the month, 1-2 weeks before the pipeline developer's meeting in CV. Nov 18th is reserved for pipeline developers to talk to NAASC staff.

4) New NRAO algorithm group (ARDG) formed, Gareth Hunt leading. Will probably be largely focussed om imaging algorithms.

5) UP/helpdesk - test of cluster failover worked well on Friday, John giving Demo at ORR.

2010-10-04

1) Splatalogue - a few issues still being worked with the CASA interface. Oct 25th is the deadline for the next OT offline version, Tony implementing a couple of new requirements.

2) VAO workshop. Brian gave a summary of the VAO workshop held in Charlottesville. Alll presentations will be available on the wiki: https://staff.nrao.edu/wiki/bin/view/OSO/VAOTeamMeetingSept2010. Brian and Joe Masters talked with Ray Plante about "science gateways" whereby data storage and manipulation is performed on Teragrid machines, with a "gateway" provided by e.g. NRAO.

3) Simdata - differences in noise between simdata and ALMA ITC discussed. Differences are probably due to different implementions of poorly-documented atmospheric libraries, especially how broad bands are integrated over (CASA uses single central frequency, not sure what ITC does). Need disclaimer on casaguides simdata page explaining this, and that ALMA ITC is what people should use as tech assessments are based on that. Also can do narrow band tests as sanity checks.

CASAguides simdata pages need to be reorganised with change of "simdata2" -> "simdata" and archiving of old material. ML to lead with help from Dongchan

4) Item from Carol - need to find out from Crystal what might be required from data services to support tutorials. ML to act as point of contact.

5) Pipeline knobs list. Due ~ next week, will discuss at next meeting. RI and ML to think.

2010-09-27

1) Advanced tools: Brian K demonstrated his Gridview package and the SD Vision package. Both contain useful functionality that is currently missing from CASA.

Unclear whether such capabilities should be developed as part of CASA (or as a plugin) or we should adopt pre-existing packages to recommend to NAASC users. Suggested to collect together some key functionalities (e.g. interactive spectral line fitting, 3D cube visualization) and discuss with Jeff Kern and Darrell Schieben which of those might be realistically achievable.3D visualization was achieved in AIPS++ and Dave has recently written a command line spectral line fitter for CASA.Alyssa Goodman may have some good connections to software from the medical community.

2) Splat: documents completed for ORR, Tony discussing new version of offline catalog for OT use with Alan Bridger

3) Pipeline/ACS - WVRgcal installed, being tested

Need to discuss more details of benchmarking tests with Jeff for desktop recommendations.

2010-08-30

1) Splatalogue:

working to fix downloading issues

splat memo ~50% done

Continuing discussions to create an international spectral line "steering committee" for ALMA-related lab spectroscopy work. Proceeding carefully in the hope of bringing in JPL and Cologne groups. Would be separate from the internal NRAO spectral line working group, but possibly contain some overlap of membership. Aim to write a formal paper along the lines of the HiTRAN? paper.

Minutes of 6th August Splat meeting posted on Data Services wiki

2) Simdata -

USS has 4-5 ES examples for primer, mostly point sources. Remy and Dong-Chan will attend USS meeting this afternoon to finalize details. Dong Chan will help Remy with the simulations.

CASA code freeze is Sept 17th, need to get bug fixes in for that

3) Pipeline - report from User Test 6 nearly done

4) ACS installation - proceeding, Mike and/or Remy will try to get access to OSF machines to figure out software environment that is actually being run.

5) image cubes

Open discussion on software available for image cubes, and what we might need to do.

Casaviewer currently lacks key capabilities, though most are on the eventual development path (need to talk to Darrell (Schiebel?)) who is CASA viewer developer

Need to test/evaluate some existing software and come up with a plan (AI to ML for now):

Nearly software discussed is IDL, need to assess how easy it would be to translate to e.g. python? If translated, would this be part of mainstream CASA development, an add-on package to CASA or standalone?

6) Plone

ML attended Pat's plone demo, was focussed on scienceweb - clearly need an automated import unless only the top 2-3 layers are done as many thousands of web pages to incorporate.

Unclear if Plone is a good solution for NAASC documentation right now.

Recommended to continue using mediawiki for online documents that need to be written in next couple of months. Translation to plone/regular

html unlikely to be severely problematic, particularly if import solution for NRAO is identified. Twiki is right out based on user experiences of ML and KS.

2010-08-23

1) Completed introductions

2) Helpdesk/UP/archive: Kelly completing registrations of testers for Integrated test, ML completing staff manual. Kelly to look into Kayako options for "self-alerting"

3) Simulations: Crystal and Al will meet with Remy to discuss ES use cases for primer, including simulations. Still unclear how "illustrative" versus scientifically driven the simulations should be.

4) JIRA for Data Services. Would probably be feasible to add NAASC/Data Services to NRAO JIRA implementation as a project. Sub-tasks would include aplatlogue. ML will contact James Robnett in Socorro who may be the person to talk to about this.

After the meeting Remy, David, Mike and I discussed obtaining a machine to run the ALMA common software (ACS). Mike will find out what he can about ACS from Brian Glendenning.

2010-08-16:

Introductions, aims of group discussed

Splatalogue: Robin and Tony working on improved documentation

Simdata: Need to convene offline meeting with Remy and Dong-Chan to discuss further simulations, particularly for ES. Jeff pointed out that ES may start with far fewer than 16 working antennas unless the schedule slips.

Pipeline: Mike confirmed that James Robnett is very happy with Lustre. ESO people may have been running old version of Redhat and were definitely running an older version of the Lustre file system which may have contributed to their problems.

Action items:

*ML - check cosmology assumptions for M51 simulations

*Dong-Chan - check noise level on M51 simulations and compare with ALMA exposure time calculator

2010-08-06:

Splatalogue Working Group Meeting (11AM) - Minutes

This topic: ALMA > WebHome > NAASC > DataServicesGroup > DataServicesMeetingMinutes
Topic revision: 2012-05-23, MarkLacy
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback