ALMA Correlator data normalization

Steve Scott's memo on normalization

George M.'s comments:

BTW, I noticed that the real and imag parts of the visibilities are quantized at a level rather coarser than float (R*4) precision. I've seen as much as ~1e-4. This arises from the archive data scaling's rounding to int (and probably a tendency to sample high spectral resolution faster in time than originally supposed). According to Steve Scott's "ALMA Correlator Details" memo, the idea is to scale the (normalized) noise level to 30 (as int), but Steve's noise calculation uses 1/sqrt(Bt) rather than a more correct sqrt(3/8)/sqrt(2Bt)/eta (note the additional 2 in the denominator, req. for XCs (cf ACs) and eta<1 is the quantization loss; the numerator comes from Hanning smoothing, and is the most significant part). So, the noise is, in fact, ~sqrt(3/16)/eta~=0.5 less than his simpler calculation, typically. I.e., the noise is scaled to more like ~15, which makes the quantization more obvious in the filled data. I guess this has an essentially trivial effect on overall sensitivity, but it is something worth appreciating generally, as it may cause panic in the innocent ALMA user....

-- ToddHunter - 2014-01-17