Add multi-frequency support to pipeline

[taramurphy]

At the moment the pipeline assumes all images are at the same frequency, which is true for the VAST Pilot Survey. However, it would be good to incorporate images at different frequencies (for example the latest RACS epoch is at a higher frequency. This requires some thought about how (and whether) to combine information from different frequencies.

Text from Adam's wishlist:

Multi-frequency support. This is something that TraP has that we don’t. Essentially the measurements and sources have a frequency band attached to them, but it only makes sense to calculate the metrics on a per band basis. Per band here usually means the central frequency +/- the bandwidth / 2. This is tricky but would be a nice science feature. This would mean multiple sets of metrics to be attached to one source so would likely mean database structure changes.

[ajstewart]

I thought about this a bit and I think you somewhat want to follow what the TraP has already done. It is an oversight on our part to not bake this in but I have a feeling it won't be too painful to achieve, just a bit fiddly to make sure everything flows right.

I think this would involve:

• I think I'm right in that we don't need to worry about different association behaviour or separate average positions, so the whole association stage doesn't need to be changed.
• Split off the source metrics to their own table - i.e. remove all the band dependant metrics from the source table to a new metrics table.
• The relationship from sources to this new table is one to many - a source can have many metric entries but crucially the metric table entires for the same source have to be unique bands.
• In the source metrics calculation in the code (finalise.py) you need to group by band - this may actually be the most tricky bit as you'll need to decide what is part of the same band, I forget how the TraP decides this but we could define our own band rules.
• Two epoch metrics will also have to be grouped per band.
• Change the UI to represent the multiple band metrics and light curves.
• Change the parquet outputs to reflect the metric changes.

For all this I'm assuming that we're not trying to do anything special with the spectral index, but this would also open up the possibility of doing something that that as well.

[marxide]

• I think I'm right in that we don't need to worry about different association behaviour or separate average positions, so the whole association stage doesn't need to be changed.

Just to clarify: would the plan be to run the association while ignoring the frequency?

[dlakaplan]

I think that would make sense.

In terms of the metrics (per band) we'd have to pay a little attention to tolerances, so that (say) a slightly different frequency due to different flagging (depending on how those things get into the headers) don't artificially make a new band. But this isn't hard.

I think we should also have an option to compute a single set of metrics including some spectral correction: while that doesn't make a lot of sense for say a flare star, it would remove the majority of the background sources.

[ajstewart]

The structure we have now just needs a tweak I think in terms of slight band changes, and I think the following is how the band detection works in TraP as well, after going back and reminding myself of it.

We have the separated out band object but currently we just create this based on the image. By that I mean because we haven't considered multi-frequency functionality yet, if a different freq image came along that was only slightly shifted then it would create a new band, which, as you say, is not what we want.

When an image is met for the first time we can (hopefully) read the bandwidth and central frequency from the header. So this becomes the first band recorded. E.g. 900 MHz +/- 144 MHz. From then on when images are read in it is checked to see if the central freq falls within this defined band, if it does then for our purposes it is assigned to the same frequency band object, with the actual frequency being stored in the image table.

Of course we can play around with this definition how we like, maybe with ASKAP bandwidths we want to be a little bit more conservative on what is considered the same band. We also have the advantage that we do not attempt to be a one size fits all solution, so we can make this decision based upon the ASKAP observing bands.

So I guess a first step is to decide how to handle the 'grouping' of frequencies for the purposes of statistic calculations.

[ajstewart]

Also to add I'm pretty sure association doesn't need to be changed, all the relations etc should take care of themselves. The TraP docs have a bit on multifrequency association scenarios but to me I don't see any difference from what we do now. I don't see any pitfalls at the moment.

[dlakaplan]

Tagging @marxide and @dlakaplan to get us in the discussion

[ajstewart]

To comment here that selavy also includes the in-band spectral index, not sure if that would be of any use or if it is considered to be any more reliable. It is stored in the pipeline.

[marxide]

In my experience, the Taylor-term derived spectral indices are unreliable for SNR ~< 50.

If we add a feature that can scale fluxes with a spectral index for metrics as David mentioned above, we could implement it such that it can scale with a user-provided spectral index for all measurements, or use the Selavy spectral index of each measurement (and issue a warning about reliability).