Minutes of I/O Performance Tuning Project - kick-off meeting
Date: 10 Nov 2015
NRAO: Jeff Kern, James Robnett, Martin Pokorny,
ESO: Julian Taylor, Sandra Castro
The idea of this project is to diagnose CASA's I/O system by running several tests. CASA's I/O system is complicated and usually people dont pay attention to what happens in the low level parts. CASA is not using CPUs efficiently. It reads small bits at a time, which is highly inefficient. This project should outline an implementation plan with a timeline and deliverables. It should run in 2 or 3 phases. The first phase will be used to diagnose the I/O system to test various hypothesis such as: using different tile shapes and different sorting orders. We should search for old documentation that defines the tile system (if any exist). Julian has found some documentation online, but they are incomplete and missing important pieces such as the storage manager documentation.
Jeff's idea for Phase 1 is to use task_visstat2 with different data selections. Julian pointed out that visstat2 only reads, therefore we also need a task that writes to the MS. Task split2 might be a good candidate for this. visstat2 doesn't open the sub-tables either. We need to verify how often the sub-tables are read in the tests.
Sandra will write the project outline with an implementation plan, timeline and deliverables and who does what in the team.
Sandra and Julian will define the test cases for Phase 1, which should include roughly: define data sets, platforms, use-cases for tests; SSD, lustre, AWS.