MOC overlap with a MRM: calculation of proba in the intersection

[mcoughlin]

Given a MOC representing a telescope field, such as:

6/32777
7/131083 131085-131087 131096 131098 131105-131107 131112-131113 131120 131122
180241 180243-180244 180246 180249
8/524319 524327 524331 524337-524339 524362-524364 524366 524388 524390 524396
524398 524417-524419 524456-524458 524460 524464-524465 524468 524484 524486
524492 524494-524495 720961 720963 720969 720971 720980-720982 720988
721004-721005 721007-721008 721010 721016 721093 721095
...

and a gravitational-wave localization in multi-order coverage of the form:

UNIQ PROBDENSITY DISTMU DISTSIGMA DISTNORM
1 / sr Mpc Mpc 1 / Mpc2

---

1024 0.0018000524306513587 626.1008456818951 176.9447332144471 2.3623295578298845e-06
1025 0.0022572306351681714 599.5277708633371 184.2451449021805 2.5420891568905436e-06
1026 0.001742950980628719 618.5493307656486 179.67464089457215 2.4103058564805825e-06
1027 0.0021450711199728343 592.2768943296091 186.86994636222323 2.5926309212560773e-06
1028 0.0018645197302460666 612.6289458821037 180.04067149941744 2.4526183970504135e-06
...

Is there a fast way with mocpy to compute the integrated probability in that MOC? This is the same problem healpix-alchemy solves but for postgres: https://github.com/skyportal/healpix-alchemy.

An example for how this works would be useful in the docs for interacting with gravitational-wave skymaps, and I suspect greatly speed-up the python-based codes performing calculations of this kind.

[fxpineau]

Hi @mcoughlin, thank you for your message.

So far, there is no way to do this.
But, I think it can be implemented quite easily.

If I understand correctly, you want to perform the intersection between a MOC and a MultiResolution HEALPix Map (MRM)
and sum the values in the resulting (sub-)MRM.
If a MOC cell is included in a larger MRM cell, I assume that the probability associated with the MRM cell must be multiply by the ratio of the area of the MOC cell over the area of the MRM cell, right?

What about a method: MOC.sum_of_multi_order_map_values(path_to_the_mom_fits_file)?

[mcoughlin]

@fxpineau exactly. I think it should probably take in a table assuming you already loaded the fits file into memory as you will likely do this many times for different fields, but that is a minor nitpick. Thanks!

[ManonMarchand]

additional TODO:

document MOC.from_multiordermap_fits_file and replace the call to astropy_healpix in the bayestar.py example

[ManonMarchand]

@mcoughlin what kind of table do you have? Is it an ~astropy.table.Table?

[mcoughlin]

@ManonMarchand Yeah. After you have read in the skymap with read_sky_map from ligo.skymap.io import read_sky_map, as skymap = read_sky_map(filename, moc=True, distances=True), for example.

[fxpineau]

@mcoughlin, @ManonMarchand,

I just added the functionality in MOC Lib Rust (see this commit) and in moc-cli (see this commit) but it is not released yet (only pushed in the main branch).

We will try to integrate the functionality in MOCPy next week (we are in holidays for the rest of the week).

[mcoughlin]

Thanks @fxpineau. We look forward to the integration in mocpy!

[fxpineau]

@ManonMarchand for the Rust part, as a starting point, see the new branch mom_sum.

[ManonMarchand]

Hi,

We added two methods (not yet merged but available in PR #127):

• MOC.sum_in_multiordermap that takes a table and a column name in arguments and returns the sum of the column given in the intersection between the MOM and the MOC
• MOC.probability_in_multiordermap that will convert a PROBDENSITY column into a probability in the intersection directly.

Is it what you needed?

[mcoughlin]

Thanks @ManonMarchand!

I would guess you have confirmed that you get the same answer with your function as when you flatten both the skymap and the moc and do the sum that way?

One idea (for the sake of potential speed up) is for a function that takes many MOCs and a skymap and returns an array of probabilities (one for each MOC). I suspect most folks will need to be looping over many fields so giving them a function where you do that internally is likely to be very useful.

[fxpineau]

Thank you for your feedback @mcoughlin.

I would guess you have confirmed that you get the same answer with your function as when you flatten both the skymap and the moc and do the sum that way?

We basically checked that the value returned by the function applied to a MOC created from a multi-order map + a probability threshold, and taking as parameter the same input multi-order map, fit with the probability threshold.
For additional tests requiring operations on multi-order map (such a flattening), I think they should be integrated to the multi-order map library (we would like not to add a dependency to the multi-order map library in MOCPy).

[...] a function that takes many MOCs and a skymap and returns an array of probabilities (one for each MOC). [...]

From my point-of-view, this should be a method of the multi-order map object, thus in the multi-order map library (I also wonder if the PROBDENSITY to Proba conversion should not be in the multi-order map library instead of the specific probability_in_multiordermap MOCPy method): the conversion from PROBDENSITY to proba could be performed only once and then internally call MOC.sum_in_multiordermap for each MOC (I tend to think that iterating n times over the mutli-order map will be faster than iterating only once and making for each cell a call on each MOC (depending on the MOM + MOCs size with respect to the CPU cache size), and it is quite easy to multithread in case of performance issues).

Let first test and see if performances have to be improved for multiple MOCs.
Don't hesitate to open a new issue if you were to find that the performances don't allow you to do your calculations.

[mcoughlin]

@fxpineau @ManonMarchand Awesome. I would be happy to test this ASAP, especially if you could get a version up on pypi.

[mcoughlin]

Thanks for the pypi release. Is the same planned for conda?

[fxpineau]

Yep, done: https://anaconda.org/conda-forge/mocpy