OODT Coordination Meeting

• Socorro, NM 2013 September 25,26 2013

Agenda

Day 1 - 9/25

• 02:00pm - Introductions
• 02:30pm - Current progress in Apache OODT - Mattmann
• 03:00pm - Discussion
• 03:30pm - NRAO deployments of Apache OODT for ALMA and EVLA
  • ALMA Kern/Davis
  • EVLA Butler/Harland
• 04:00pm - Q&A for OODT by NRAO - NRAO doers
• 04:30pm - Adjourn
• 05:00pm - Drinks/Dinner/Social

Day 2 - 9/26

• 09:00am - MeerKAT/KAT-7 replicated archive
  • Could we use OODT to build a replicated archive in the US
• 10:00am - Scientific Visualization Toolkit for Astronomy Planning - Kern
• 10:30am - JPL BigData other activities (Climate; DARPA XDATA)
• 11:00am - Specific Planning for JPL & NRAO collaborations
  • Visualization - Kern/Mattmann
  • Archive Access Toolv2 - Butler/Mattmann
  • ALMA specific needs - Kern/Davis
  • EVLA specific needs - Butler/Harland
• 11:30am - Upcoming meetings or workshops/papers/proposals that we can/should plan - All
  • WSSSPE http://wssspe.researchcomputing.org.uk/ - Nov 17/2013
  • WICSA http://www.wicsa2014.org/
  • NSF/NASA/etc.
• 12:00pm - Lunch
• 01:00pm - Adjourn

ALMA OODT Experience (Lindsey Davis, personal impressions)

• focus
  • workflow manager
• work period
  • December 2012 - January 2013, September 2013, 2-3 weeks off and on
• get it working (with the help of Keith Cummings !)
  • sss-pipe-oodt-clean project https://safe.nrao.edu/wiki/pub/Software/CASA/Sept2013OODTMeeting/sss-pipe-oodt-clean.txt
    • maven project management system, stripped EVLA project, skeleton EVLA workflow
  • experience
    • maven: new to me, large black box
    • EVLA project structure: lots of directory layers
    • oodt documentation: how to documentation minimal
• make an ALMA namespace friendly project
  • alma-pipe-oodt project https://safe.nrao.edu/wiki/pub/Software/CASA/Sept2013OODTMeeting/alma-pipe-oodt.txt
    • maven: cloned maven setup, edited some path definitions
    • project: constructed skeleton project with simpler directory structure, cloned skeleton workflow
  • experience
    • maven: for me a distraction, complex, difficult for novice to understand and tune
    • workflow manager: workflow interfaces, policy definitions, easy to understand
    • libraries: large number of jar files, most not needed, large overlap with ALMA
• ALMA Common Software (ACS) integration
  • ALMA pipeline project
    • copied policy, tasks, to ACS SVN repository https://safe.nrao.edu/wiki/pub/Software/CASA/Sept2013OODTMeeting/PIPELINE-Science-src.txt
    • extracted required OODT jar files, only ~6 needed, copied to repository https://safe.nrao.edu/wiki/pub/Software/CASA/Sept2013OODTMeeting/PIPELINE-Science-lib.txt
    • built software, ported wmgr, wmgr-client, property files to ACS
      • https://safe.nrao.edu/wiki/pub/Software/CASA/Sept2013OODTMeeting/wmgr
      • https://safe.nrao.edu/wiki/pub/Software/CASA/Sept2013OODTMeeting/wmgr-client
    • configured run time environment
  • experience
    • workflow manager and client: wmgr, wmgr-client scripts straight forward to port but many details
    • libraries: extracting minmal set of jars and comparing to ACS tedious, configuration especially logging painful
    • user how to documentation needed
• Connection to ALMA pipeline infrastructure code
  • steps
    • generated artificial project (pipeine processing request etc), no external metadata or archive connection required
    • integrated OODT code with existing pipeline infrastructure, up to but not including running the CASA script
    • experimented passing an object between OODT tasks, used Base64 encoding assuming serializable object
  • experience
    • documentation minimal
• Issues
  • OODT distribution mechanism
    • tarballs ?, tarballs with jar and source files ?, building selected jars from source ?
  • configuring OODT in third party development environment
    • environment: environment variables, java properties, loggers, hardcoded paths
    • runtime environment: process, logging, and repository directories
  • passing meta data between tasks
    • passing objects: any experience with this
  • starting, stopping, queuing executions
  • documentation
    • ?

Partial EVLA OODT Experience (David Harland, blog of attempt to create "Hello, World" app)

• http://www.aoc.nrao.edu/~dharland/oodt/fileMgrFirstSteps.html