• Building the Next Generation Digital Correlator and Phased Array for ALMA

  • Referee5
Grade: 4.5

494 - Building the Next Generation Digital Correlator and Phased Array for ALMA

1. Alignment with NA ALMA Partnership strategic goals;

This development would chart a path to a next-generation correlator, and is therefore consistent with the development goals.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The scientific case spans much of the ALMA science space. Increased bandwidth and/or increased throughput at high spectral resolution improve most areas of ALMA science, so the case is somewhat self-evident. There may be other applications enabled by the commensal beamforming that is built into the BX portion of the design, perhaps in real-time transient searches or parallel pulsar searches (particularly in Band 1) or solar observing, but applications of the beamforming are not explored. VLBI is the only case made for beamforming at present (and the case is only vaguely made, as the utility of multiple phase beams is not actually explained, nor any claim of need for improved sensitivity when looking at several Jy sources), and VLBI represents a miniscule fraction of ALMA science proposals.

3. Quality of the upgrade conceptual design;

The concept has clearly received a thorough treatment in the study stage. It represents a paradigm shift for the correlator, as it builds around off-the-shelf technology rather than ALMA-specific engineering. This results in significant improvements over the current correlator, though it is not enormously more capable (same resolution, double total bandwidth, presumes single-pass digitization of the full bandwidth from a future antenna digitizer) than the correlator described in the Lacasse proposal, which has been brought to production readiness through past studies. The off-the-shelf aspect of the development has some downsides as well, as the available hardware will continue to improve but the 3-year study seeks to reach a CDR stage for correlator production. If ALMA will pursue a correlator improvement like that in the Lacasse proposal, the improvements described here are not a significant enough improvement to be worthwhile, but waiting a year or two to achieve something more dramatic with the next generation of hardware might be.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

This project seeks to prototype, not deliver to the array.

5. Strength of the consortium organization (if applicable);

Subcontracting the S engine to HIA makes use of their digital experience.

6. Qualifications of the key personnel of the Study;

The team has demonstrated success at a similar development at SMA.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

The central team has worked together on the new SMA correlator, and presumably the ongoing study.

8. Assessment of the level of risk inherent in the design;

The existing study has probably retired a significant portion of the design risk. There are off-budget risks related to the need for a dramatically different antenna digitization and transmission system and software expenses associated with replacing the correlator with a very different conceptual design.

9. Strength of the Scientific Team supporting the Study;

The science case brought in expert input from several ALMA science areas.

10. Level of support guaranteed by the Institutes;

NRC HIA promise a significant in-kind contribution.

11. Budgeted cost of the Study;

The cost of this proposal is not small ($1M/yr), but seems matched to the task.

  • Referee6
Grade: 2 (science grade alone) 3 (science and cost - see 11 below)

Building the Next Generation Digital Correlator and Phased Array for ALMA

1. Alignment with NA ALMA Partnership strategic goals;

This proposal aligns with the strategic goals in many ways. It improves and extends the technical capabilities. It also increases operating efficiency. The proposal also aligns with ALMA 2030.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The scientific case for the proposed upgrade is excellent - access to wider bandwidth, narrower spectral features, and better sampling, along with flexibility.

3. Quality of the upgrade conceptual design;

Looks excellent but should be reviewed very carefully by an expert in correlators. I note that the conceptual design study is only recently completed but that the output from that study should be very carefully externally reviewed by technologists to determine feasibility and provide confidence in the cost.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

The proposers believe that the next step is a 1/16th production to test the correlator and this appears prudent. Nevertheless, given that the conceptual study has only just completed and many documents were produced for the study, this should be externally verified by technological experts.

5. Strength of the consortium organization (if applicable);

Excellent.

6. Qualifications of the key personnel of the Study;

Excellent.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

Excellent record.

8. Assessment of the level of risk inherent in the design;

By phasing the development in two stages, starting here with a 1/16th production, the risk is significantly mitigated.

9. Strength of the Scientific Team supporting the Study;

There is a very strong scientific team associated with this project.

10. Level of support guaranteed by the Institutes;

$385,000 in kind.

11. Budgeted cost of the Study;

$2,755,055 over three years. While the cost of this project falls within the expected funding envelope, it is hard to judge the expected cost of the full project (is it 16x larger?!?) and thus compare the cost against other correlator projects.

  • Referee8
Grade: 3.0 because this leads to a large improvement in science throughput

Title: Building the next generation digital correlator and phased array for ALMA PI: Weintroub

1. Alignment with NA ALMA Partnership strategic goals;

The proposal is well aligned with the strategic goal of increasing the performance of ALMA, in this case by increasing the correlator bandwidth, which will improve the overall efficiency of ALMA. The proposal strengthens the North American radio astronomy community because it brings new researchers into ALMA, and it involves a technical approach that really is new. The proposal is the most interesting of the correlator efforts in the current round partly because it offers a substantial improvement in performance, and partly because the technology is new.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the ‘ALMA 2030’ development documents.

The science case is along the lines of “more is better,” and while large, unbiased samples are clearly good, not much is said about which science questions will actually be answered. Intensity mapping with the wider bandwidth should yield measurements of the CO ladder for normal galaxies, but it is not obvious that the factor 4 increase in bandwidth is critical for progress in this new area. The event horizon observations steal the show, standing out as the only really fundamental measurement in the science case, but these observations will happen in some form even without the proposed correlator. The proposal directly addresses the ALMA 2030 recommended development path of improving speed by increasing bandwidth, in this case by a factor 4. This is clearly where ALMA wants to be in the next decade. The proposal also addresses the the ALMA 2030 recommendation on longer baselines, though it is through VLBI, which is probably not what the ALMA 2030 planners had in mind.

3. Quality of the upgrade conceptual design;

The proposed correlator upgrade is an ambitious design to provide 32GHz bandwidth per polarization using a new architecture based on previous work for the SMA and LEDA. The design is well thought out. I like the idea of using off the shelf hardware, and I am especially pleased to see strong emphasis on an ALMA simulator as part of the design. I did not see an explanation of why the prototype correlator should have 8 stations. Why not 4?

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development);

The readiness of the design is appropriate for the proposed prototype. Expansion to the full ALMA array is several years away.

5. Strength of the consortium organization (if applicable);

A key strength of this proposal is that it combines the skills of two strong correlator groups at SAO and NRC.

6. Qualifications of the key personnel of the Study;

This is a competent team with a good track record.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study;

NRC has expertise and experience from eVLA; SAO from SMA and LEDA. Both groups have delivered large, complex systems.

8. Assessment of the level of risk inherent in the design;

The use of off the shelf hardware reduces some of the risk in the design, but brings new risks in that modern off the shelf computing hardware has a short life cycle. The latter may become important if ALMA has to spread the upgrade work over many years. Making full use of the 32GHz bandwidth correlator will require many upgrades to ALMA. The proposed correlator work only makes sense if ALMA is fully committed.

9. Strength of the Scientific Team supporting the Study;

The science case for the correlator upgrade is part of the broad science case for ALMA 2030.

10. Level of support guaranteed by the Institutes;

Appropriate.

11. Budgeted cost of the Study;

$3M seems reasonable for development of an 8-station correlator prototype, though I wonder if it really needs to be 8 stations. The proposal only makes sense if the projected cost of the full correlator upgrade fits into ALMA cost planning, but unfortunately the proposal does not address this critical issue. Scaling the hardware costs in the proposal with number of baselines gives a scary number, and that does not include any of the IF and signal transmission upgrades.

* Referee9

Building the Next Generation Digital Correlator and Phased Array for ALMA PRINCIPAL INVESTIGATOR: JONATHAN WEINTROUB

Turn the Development Study “Digital Correlator and Phased Array Architectures for Upgrading ALMA” into a detailed design, build, and validate, hardware and software for 8 antennas x 8 GHz for the next generation ALMA Correlator and Phased Array. The design is a packetized FX correlator with a Tunable Filter Bank Front End and coherent phased sum output, using COTS FPGA, GPU and 100Gbps Ethernet switches. System tests off line: proposal includes the design and production of a real time emulator, with fringe Doppler, delay tracking, and capable of producing visibilities for a complex sky image. Performance: 8GHz dual pol x 8 antennas x 1 kHz resolution. Support for 2+ phased arrays and 1 ms sample. Cost $2.8M. Scaled parts cost for 8 GHz x 72 antennas = $4.5M; 32 GHz x 72 antennas = $18M.

Risks include de-scoping the emulator, and FPGA timing issues - $128K

* Referee 9 Comparative ANALYSIS

Both proposals seek to develop new capability for the ALMA correlator. Lacasse using new custom ASICs, leverages existing expertise. Weintroub using COTS hardware and flexible gateware and software, with a growing community of seasoned and of new developers, is an investment into the long term future of ALMA.

Neither proposal addresses the impact of the higher DATA RATES enabled by new correlators. ALMA science capability is severely limited by the data calibration, validation, and users ability to digest the current data rate, which is a small fraction of what the current correlator is capable.

Correlators using COTS FPGA, GPU and 100Gbps Ethernet switches, offer the possibility adding, or re-purposing hardware attached to the Ethernet switches, for close to real time calibration, essential for VLBI and phased array receivers, and enabling much better monitoring of system and atmospheric performance, and higher data rates with pre-calibrated data. Whilst the custom ASIC design offers enhanced correlator performance in 3-years, the COTS packetized correlator using Ethernet switches provides a development path for the future viability and science capability of ALMA. The current receivers are single pixel, resulting in a small FOV. Array receivers could be used to increase the FOV, provide simultaneous multiple beams to observe multiple targets or target and calibrators, or correct for PB astigmatism. These developments would result in a substantial increase in the data rate from the telescope, and it may not be possible to get the data off the mountain. Real time processing of the data is required. A flexible, programmable correlator could provide the computing power to pre-process the data from the telescopes and significantly reduce the data rate. Mapping a source larger than the antenna PB requires mosaicing multiple pointings. At high frequencies, PB astigmatism may severely compromise the fidelity of mosaic images. Since the PB pattern rotates on the sky, and may be time ( temperature, elevation) dependent, off-line correction for PB astigmatism is difficult. PB astigmatism could be corrected by processing data from array receivers . At high frequencies, and large antenna separations, observations are severely limited by atmospheric coherence. The data may be corrected by using water vapor radiometers, or rapid switching between the target and nearby calibrator(s). These observations increase the data rate. A rapid assessment of the efficacy of atmospheric phase correction would significantly improve the productivity of the telescope, and enable high frequency observations more of the time. Simultaneous observations of a target and close calibrators using multiple beams from phased array receivers on the antennas could be used for atmospheric phase correction. After atmospheric phase correction, the data can be integrated, substantially reducing the data rate. These calibrations could be applied in close to real time using a programmable correlator and data processing at the telescope. The requirements for wide field polarization observations are currently undetermined. There may be time and antenna dependent polarization corrections across the FOV which are difficult to correct off- line. On-line data processing of the measured cross correlations could greatly reduce the burden of off- line data processing and lead to more efficient observing and higher quality data. It is hard to access what will be the most urgent need for the long term future of ALMA. I prefer an incremental development approach, rather than a one-time upgrade to the existing correlator, which already far exceeds our capability to process the data. Since DSP capabilities evolve so rapidly, a correlator architecture which allows incremental improvements and additions to its correlation and data processing engines offers a more flexible approach to address future needs. I am concerned that a significant one-time upgrade within the existing correlator architecture will significantly delay, or preclude a more capable and flexible correlator/data processor which is capable of addressing future needs. A packetized correlator using Ethernet switches which can route the data to multiple compute engines enables incremental development, which could include ASICs for high volume operations such as FFT or correlation engines. In the best of both worlds, it would be nice if the very capable NRAO team could embrace the longer term future of ALMA.

  • * Referee10*

Grade: 3 Executive Summary: This project proposes to build a scale-model version of a next-generation hybrid FX correlator, based on prior design studies, and in preparation for a full-scale build proposal. The technologies used are well justified and well thought through. The included “antenna simulator” is of significant value as an ALMA testbed environment for future developments. I find this proposal timely, well justified, and part of the path toward an eventual next-generation ALMA.

Weintroub, Jonathan - Building the Next Generation Digital Correlator and Phased Array for ALMA

ALMA Development Review

1. Alignment with NA ALMA Partnership strategic goals; The proposal is well aligned with Strategic Goals (1.x), providing a significant correlator upgrade and novel observing modes such as VLBI beamforming. By using new innovations in correlator technologies, it proposes a decrease in energy requirements (3.4). It appears to draw on a coalition of ALMA partners (4.1), and makes explicit reference to training young engineers (5.2). Overall, the proposal does a good job addressing the strategic goals set out in the NA ALMA Development Program.

2. Strength of the scientific case for the proposed ALMA upgrade concept; Comment on the relevance to the †̃ALMA 2030â€TM development documents. By broadening the band of the correlator, providing higher spectral resolution and supporting both longer baselines a beamforming mode, this proposal seeks to develop & test a requisite subset of the technology needed to address point 2 of the ALMA2030 document, “Larger Bandwidths and better receiver sensitivity”. As they point out, significant additional front-end hardware is still required to fully address the point, but is also under development, making this development timely within the broader instrument upgrade.

3. Quality of the upgrade conceptual design;

The design proposed draws on many of the new technologies and techniques developed in the decade since the original ALMA correlator was deployed: an FX architecture and heterogenous compute units tailored to particular tasks. The system appears to be well studied as a result of a previous development program, and the decision are well motivated.

4. Readiness for production in the context of the ALMA Development Plan (the aim is to support a range of upgrades including both those which can be implemented rapidly and those requiring longer-term research and development); I find the partial build request very reasonable to test the system performance in detail and in real-world circumstances. A scaled -up system would presumably not be ready for full deployment for a few years, but this intermediate build plan is well motivated and justified in the broader ALMA context. The antenna simulator in particular is a valuable addition to future ALMA developments, as it will allow competing designs to be compared on a level field.

5. Strength of the consortium organization (if applicable); The SAO and NRC Hertzberg bring significant strengths and strong track records deploying of complex correlator systems. It is unclear how committed SAO are to the project, as very little in the way of institutional in-kind is provided. The significant (40% of their cash cost) contribution from NRC indicates it’s a strong institutional priority. I have no serious concerns about the consortium partners.

6. Qualifications of the key personnel of the Study; The personnel involved are experts in modern correlator technologies, and well placed to develop this new testbed system.

7. Technical expertise, past experience (also in series production, if relevant) and technical facilities in the Institutes taking part in the Study; The personnel and institutions involved have strong track records with large and / or complex correlator systems deployed to major radio facilities in the US. The have all necessary facilities and personnel to meet the production schedule.

8. Assessment of the level of risk inherent in the design; There are substantial risks in implementing the design: as the proposers themselves point out, it is unclear if they will be able to fit the required logic into their FPGAs. The GPU utilization looks modest, but achieving the requisite I/O bandwidths in asynchronous systems is a potential and unaddressed issue. 


9. Strength of the Scientific Team supporting the Study; While this is largely a technical / hardware proposal, there seems to have been some effort to include scientific justifications in the proposal. The science team still appears thin on the ground, and I will note a concern that they don’t appear to be closely involved in the developments proposed. The advisory board is a welcome addition, but I would have preferred to see a formal schedule of reviews from this panel. 


10. Level of support guaranteed by the Institutes; The NRC Herzberg is providing substantial in-kind, as a fraction of their total sub-contracted budget, but SAO makes no explicit statement of support beyond the as-available time of reviewers and the PI. 


11. Budgeted cost of the Study; 
The study cost is modest, and completely reasonable given the scale of the undertaking. Much of the budget remains devoted to NRE / personnel, but the systems resulting appear to be of great interest to ALMA’s future.

-- AlWootten - 2017-05-09
Topic revision: r3 - 2017-06-27, AlWootten
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback