Exporting multiple tunings
Rob:
exportuvfits says that combineSpw must be True for a successful export to
AIPS. Could whoever put that in (or anyone else with something to say) explain
why? Is FITLD not able to handle multiple FQ entries?
- (Ed?): AIPS's fundamental problem with multiple FREQIDs is that it gets confused if two visibilities observed at the same time have different FREQIDs.
- George: There may be other problems.
If combineSpw is True it will try to export multiple spectral windows as IFs instead of FREQIDS.
SPWs vs. IFs vs. FREQIDs
There are two different ways of mapping CASA's spectral windows (spws) into UVFITS: IFs and FREQIDs.
- FREQIDs: seem to be roughly equivalent to spws, but as noted above there is limited support for them outside of CASA. aips cannot handle FREQIDs that overlap in time, and difmap (incl. smerf) cannot handle multiple FREQIDs at all.
- IFs: are more commonly accepted, but are much less general than spws:
- Unlike FREQIDs, they must all occupy the same times. So if a UVFITS file has n IFs, each IF must be present for every time and baseline.
- Each IF must have the same # of channels.
- Each channel must have the same width.
- The order of IFs must stay the same throughout the whole file, i.e. 0, 1, 2, 0, 1, 2, 0, 1, 2, ..., 0, 1, 2. This is where exportuvfits is currently (CAS-2048) having most of its trouble. Even if the spws were all observed together throughout the whole MS, frequency-dependent flagging followed by time averaging with split can potentially disrupt the cadence. Note that flagging zeroes is often frequency-dependent. As of mid-April 2010 importevla does not yet apply frequency-dependent flags, but it is planned to start doing that soon.
What can AIPS, miriad, difmap, GIPSY, etc. handle?
If the spws were exported separately, could aips usefully combine them?
dbcon
and/or
vbglu
don't seem to do the trick, but maybe I'm doing something wrong.
I believe miriad also has problems with multiple spws from exportuvfits (with
combineSpw = False) and difmap definitely does.
The UVFITS definition
George:
This is mostly a problem with the "definition" (had one existed)
of UVFITS.
Rob: There is
this sadly non-searchable AIPS document describing the various FITS tables you might find in a UVFITS file. The UVFITS description starts on page 7 as 14.4. But obviously that's not the whole story - what AIPS, miriad, difmap, etc. support in practice also matters.
This does not exactly belong here, but
- it seems that EXPOSURE is dropped unless multisource=True, i.e. in single source UVFITS files is INTERVAL effectively assumed to be the difference between one time and the next?
- Note the relationship between weights and flagging on page 14-9.
Back to spws vs. IFs...
George again: The upshot here is that for spws to be packed as AIPS "IFs", they
need to be simultaneous. Or ought to be, to avoid waaay inefficient
storage. Basically, "IFs" in AIPS are simultaneously observed spws,
and there is an IF axis in the underlying data array. Packing
non-simultaneous spws will be very inefficient, if it works at all,
because space must be allocated for non-existent data. The
inefficiency goes as the number of non-simultaneous spws groups.
In AIPS, spws that are not observed simultaneously, e.g., a VLA
observation serially observed in multiple bands (2 IFs in C-band
and 2 IFs in X-band, alternating, say) should be stored as multiple
FREQIDs. Each FREQID has its own data array (with an
appropriate IF axis), but AIPS (i.e., Eric) tends to prefer that
different FREQIDs be stored in separate files.
It is my understanding that exportuvfits doesn't handle either
of the following cases very well:
1. Multiple spws that are generally not simultaneous exported
as multiple IFs in a single FREQID. Unless the spws are precisely
simultaneous, there will necessarily be flagged "data" that wasn't
even observed within the output data array. This is what
is going on in CAS-2048, possibly with some scrambling of the
data. (I've heard anecdotal claims that phantom data are
not zeroed, but Emmanuel seems to see zeroes in
such (if not all) cases.)
2. Non-simultaneous multiple spws stored as separate FREQIDs.
I'm not sure if this ever worked. Since AIPS prefers that separate
FREQIDs occur in separate datasets within AIPS (I'm not sure
about miriad or difmap, but I suspect similar), it is not
clear that we should bother writing multiple FREQIDs into
a single UVFITS file. And I am not sure how to force exportuvfits
to treat truly simultaneous subsets of spws as simultaneous
IFs within each FREQID---currently the exportuvfits task implies
it is all or nothing via combinespw.
I guess I would tend to want to make sure 1 above works
generally, even when the spws are not simultaneous (and
the phantom data get filled as flagged zeros). Hopefully,
the non-simultaneous cases will usually amount to
an integration here and there with fewer than the full
complement of simultaneous spws. Perhaps a warning
should be issued if a very large fraction of data is
non-simultaneous.
As for CAS-2048, Emmanuel's problem is at least
partly a misunderstanding of the above facts. At best,
he is expecting that the different frequency settings
(non-simultaneous) would have come out as separate
FREQIDs. He should run exportuvfits multiple times,
once for each set of simultaneous settings, and he
will probably get what he (and AIPS) wants, in
separate files. (Even so, we should
make sure that what he tried works correctly, even if
it is painfully inefficient. Or it should gracefully
exit and refuse to do such a silly thing.)
One more thing: If you really start tinkering with
spw-handling in exportuvfits, do make sure that
the frequency labeling of the output (incl ALT*
values in the header and FQ table info, such
as they are) aren't botched somehow. Along
with the clunky IF axis issues are fairly fragile
frequency identification conventions.
Rob:
'>
The upshot here is that for spws to be packed as AIPS "IFs", they
'>
need to be simultaneous. Or ought to be, to avoid waaay inefficient
'>
storage. Basically, "IFs" in AIPS are simultaneously observed spws,
My understanding is that they also each have to have the same number of
channels, and each channel must have the same width.
'>
1. Multiple spws that are generally not simultaneous exported
'>
as multiple IFs in a single FREQID. Unless the spws are precisely
'>
simultaneous, there will necessarily be flagged "data" that wasn't
'>
even observed within the output data array. This is what
'>
is going on in CAS-2048,
No, it'd be nice if it did that. What seems to be happening is that the
different DDIDs are in different scans, but the output has 1 "DDID" with
multiple IFs, and each IF gets rows from each of the spws.
On the face of it (= digging thru UVFITS headers & tables, and AIPS docs)
multiple FREQIDs are the UVFITS equivalent of general spws. But it seems
like the implementation never got past half-baked outside casa. miriad
does not handle it well or at all (I'll check with Michele and/or Crystal
tomorrow, they've been struggling in parallel.). difmap explicitly
disallows multiple FREQIDs - it's multiple IFs or nothing.
'>
I guess I would tend to want to make sure 1 above works
'>
generally, even when the spws are not simultaneous (and
'>
the phantom data get filled as flagged zeros).
I would like that (and I'd like it yesterday) because it's definitely the
only option for difmap, but I think it's going to take time for me to
implement it. In the short term I think I should just split out the
spws^H^H^H^Htunings, i.e.
multituning.ms -> outname_0.UVF, outname_1.UVF, outname_2.UVF, ...