# ALMA Correlator data normalization

Steve Scott's memo on normalization
George M.'s comments:

BTW, I noticed that the real and imag parts of the visibilities are
quantized at a level rather coarser than float (R*4) precision. I've
seen as much as ~1e-4. This arises from the archive data scaling's
rounding to int (and probably a tendency to sample high spectral
resolution faster in time than originally supposed). According to
Steve Scott's "ALMA Correlator Details" memo, the idea is to scale the
(normalized) noise level to 30 (as int), but Steve's noise calculation
uses 1/sqrt(Bt) rather than a more correct sqrt(3/8)/sqrt(2Bt)/eta (note the
additional 2 in the denominator, req. for XCs (cf ACs) and eta<1 is the
quantization loss; the numerator comes from Hanning smoothing, and
is the most significant part). So, the noise is, in fact, ~sqrt(3/16)/eta~=0.5
less than his simpler calculation, typically. I.e., the noise is scaled to more
like ~15, which makes the quantization more obvious in the filled data. I
guess this has an essentially trivial effect on overall sensitivity, but it is
something worth appreciating generally, as it may cause panic in the
innocent ALMA user....

--

ToddHunter - 2014-01-17