Exporting multiple tunings

Rob: exportuvfits says that combineSpw must be True for a successful export to AIPS. Could whoever put that in (or anyone else with something to say) explain why? Is FITLD not able to handle multiple FQ entries?

  • (Ed?): AIPS's fundamental problem with multiple FREQIDs is that it gets confused if two visibilities observed at the same time have different FREQIDs.
  • George: There may be other problems.

If combineSpw is True it will try to export multiple spectral windows as IFs instead of FREQIDS.

SPWs vs. IFs vs. FREQIDs

There are two different ways of mapping CASA's spectral windows (spws) into UVFITS: IFs and FREQIDs.
  • FREQIDs: seem to be roughly equivalent to spws, but as noted above there is limited support for them outside of CASA. aips cannot handle FREQIDs that overlap in time, and difmap (incl. smerf) cannot handle multiple FREQIDs at all.
  • IFs: are more commonly accepted, but are much less general than spws:
    • Unlike FREQIDs, they must all occupy the same times. So if a UVFITS file has n IFs, each IF must be present for every time and baseline.
    • Each IF must have the same # of channels.
    • Each channel must have the same width.
    • The order of IFs must stay the same throughout the whole file, i.e. 0, 1, 2, 0, 1, 2, 0, 1, 2, ..., 0, 1, 2. This is where exportuvfits is currently (CAS-2048) having most of its trouble. Even if the spws were all observed together throughout the whole MS, frequency-dependent flagging followed by time averaging with split can potentially disrupt the cadence. Note that flagging zeroes is often frequency-dependent. As of mid-April 2010 importevla does not yet apply frequency-dependent flags, but it is planned to start doing that soon.

What can AIPS, miriad, difmap, GIPSY, etc. handle?

If the spws were exported separately, could aips usefully combine them? dbcon and/or vbglu don't seem to do the trick, but maybe I'm doing something wrong.

I believe miriad also has problems with multiple spws from exportuvfits (with combineSpw = False) and difmap definitely does.

The UVFITS definition

George: This is mostly a problem with the "definition" (had one existed) of UVFITS.

Rob: There is this sadly non-searchable AIPS document describing the various FITS tables you might find in a UVFITS file. The UVFITS description starts on page 7 as 14.4. But obviously that's not the whole story - what AIPS, miriad, difmap, etc. support in practice also matters.

This does not exactly belong here, but
  • it seems that EXPOSURE is dropped unless multisource=True, i.e. in single source UVFITS files is INTERVAL effectively assumed to be the difference between one time and the next?
  • Note the relationship between weights and flagging on page 14-9.

Back to spws vs. IFs...

George again: The upshot here is that for spws to be packed as AIPS "IFs", they need to be simultaneous. Or ought to be, to avoid waaay inefficient storage. Basically, "IFs" in AIPS are simultaneously observed spws, and there is an IF axis in the underlying data array. Packing non-simultaneous spws will be very inefficient, if it works at all, because space must be allocated for non-existent data. The inefficiency goes as the number of non-simultaneous spws groups.

In AIPS, spws that are not observed simultaneously, e.g., a VLA observation serially observed in multiple bands (2 IFs in C-band and 2 IFs in X-band, alternating, say) should be stored as multiple FREQIDs. Each FREQID has its own data array (with an appropriate IF axis), but AIPS (i.e., Eric) tends to prefer that different FREQIDs be stored in separate files.

It is my understanding that exportuvfits doesn't handle either of the following cases very well:

1. Multiple spws that are generally not simultaneous exported as multiple IFs in a single FREQID. Unless the spws are precisely simultaneous, there will necessarily be flagged "data" that wasn't even observed within the output data array. This is what is going on in CAS-2048, possibly with some scrambling of the data. (I've heard anecdotal claims that phantom data are not zeroed, but Emmanuel seems to see zeroes in such (if not all) cases.)

2. Non-simultaneous multiple spws stored as separate FREQIDs. I'm not sure if this ever worked. Since AIPS prefers that separate FREQIDs occur in separate datasets within AIPS (I'm not sure about miriad or difmap, but I suspect similar), it is not clear that we should bother writing multiple FREQIDs into a single UVFITS file. And I am not sure how to force exportuvfits to treat truly simultaneous subsets of spws as simultaneous IFs within each FREQID---currently the exportuvfits task implies it is all or nothing via combinespw.

I guess I would tend to want to make sure 1 above works generally, even when the spws are not simultaneous (and the phantom data get filled as flagged zeros). Hopefully, the non-simultaneous cases will usually amount to an integration here and there with fewer than the full complement of simultaneous spws. Perhaps a warning should be issued if a very large fraction of data is non-simultaneous.

As for CAS-2048, Emmanuel's problem is at least partly a misunderstanding of the above facts. At best, he is expecting that the different frequency settings (non-simultaneous) would have come out as separate FREQIDs. He should run exportuvfits multiple times, once for each set of simultaneous settings, and he will probably get what he (and AIPS) wants, in separate files. (Even so, we should make sure that what he tried works correctly, even if it is painfully inefficient. Or it should gracefully exit and refuse to do such a silly thing.)

One more thing: If you really start tinkering with spw-handling in exportuvfits, do make sure that the frequency labeling of the output (incl ALT* values in the header and FQ table info, such as they are) aren't botched somehow. Along with the clunky IF axis issues are fairly fragile frequency identification conventions.


Rob: '> The upshot here is that for spws to be packed as AIPS "IFs", they
'> need to be simultaneous. Or ought to be, to avoid waaay inefficient
'> storage. Basically, "IFs" in AIPS are simultaneously observed spws,

My understanding is that they also each have to have the same number of channels, and each channel must have the same width.

'> 1. Multiple spws that are generally not simultaneous exported
'> as multiple IFs in a single FREQID. Unless the spws are precisely
'> simultaneous, there will necessarily be flagged "data" that wasn't
'> even observed within the output data array. This is what
'> is going on in CAS-2048,

No, it'd be nice if it did that. What seems to be happening is that the different DDIDs are in different scans, but the output has 1 "DDID" with multiple IFs, and each IF gets rows from each of the spws.

On the face of it (= digging thru UVFITS headers & tables, and AIPS docs) multiple FREQIDs are the UVFITS equivalent of general spws. But it seems like the implementation never got past half-baked outside casa. miriad does not handle it well or at all (I'll check with Michele and/or Crystal tomorrow, they've been struggling in parallel.). difmap explicitly disallows multiple FREQIDs - it's multiple IFs or nothing.

'> I guess I would tend to want to make sure 1 above works
'> generally, even when the spws are not simultaneous (and
'> the phantom data get filled as flagged zeros).

I would like that (and I'd like it yesterday) because it's definitely the only option for difmap, but I think it's going to take time for me to implement it. In the short term I think I should just split out the spws^H^H^H^Htunings, i.e.

multituning.ms -> outname_0.UVF, outname_1.UVF, outname_2.UVF, ...
Topic revision: r4 - 2010-04-16, RobReid
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback