Merge pull request #548 from VChristiaens/bpix_interp_dvpt

Iterative spectral deconvolution bad pixel correction + other minor improvements

paper.md

@@ -87,12 +87,39 @@ stellar environments.
 
 # Statement of need
 
-``VIP`` stands for Vortex Image Processing. It is a collaborative project
-which started at the University of Liège, aiming to integrate open-source,
-efficient, easy-to-use and well-documented implementations of state-of-the-art
-algorithms used in the context of high-contrast imaging. The package follows a
-modular architecture, such that its routines cover a wide diversity of tasks,
-including:
+``VIP`` (Vortex Image Processing) is a collaborative project which started at 
+the University of Liège, aiming to integrate open-source, efficient, easy-to-use 
+and well-documented implementations of state-of-the-art algorithms used in the 
+context of high-contrast imaging [@Gomez:2017]. Two other open-source toolkits 
+for high-contrast imaging with similar purpose and extent as ``VIP`` have 
+become available in the last few years: ``pyklip`` and ``pynpoint`` 
+[@pyklip:2015; @pynpoint:2015; @pynpoint:2019]. In each of these, a core (and only) 
+post-processing method is available: the Karhuenen-Loeve Image Projection (KLIP) 
+algorithm [@Soummer:2012], and the (similar) Principal Component Analysis (PCA) 
+algorithm [@Amara:2012], respectively. In contrast, ``VIP`` not only implements 
+the PCA algorithm with a variety of flavours, but it also includes a diversity 
+of other post-processing methods, such as ANDROMEDA, KLIP-FMMF, LLSG, NMF or 
+PACO [@Cantalloube:2015; @Gomez:2016; @Gomez:2017; @Ruffio:2017; @Flasseur:2018]. 
+Furthermore, as opposed to ``VIP``, ``pyklip`` does not offer any preprocessing 
+options (e.g. PCA-based sky subtraction, image centering or bad frame trimming). 
+``pynpoint`` was originally developed as a PCA-based PSF-subtraction mini-package 
+[@pynpoint:2015], which was later significantly expanded into an end-to-end 
+processing pipeline including similar options as ``VIP`` regarding preprocessing 
+[@pynpoint:2019]. Nonetheless, the PCA implementation in ``VIP`` offers a much
+wider diversity of options, such as the possibility to carry it out in 
+concentric annuli, and considering a parallactic angle threshold when creating 
+the PCA library. Depending on the high-contrast imaging dataset at hand, 
+different post-processing methods and reduction parameters can lead to better 
+speckle suppression, hence help with the detection of fainter companions 
+[@Dahlqvist:2021]. In that regard, ``VIP`` is thus better equipped than other 
+existing toolkits. It is also worth mentioning that FFT-based methods are 
+implemented in ``VIP`` (default option) for all image operations (rotation, 
+shift and rescaling) as these outperform interpolation-based methods in terms of
+flux conservation [@Larkin:1997]. To our knowledge, these FFT-based methods for 
+image operations are not available in other high-contrast imaging packages.
+
+The ``VIP`` package follows a modular architecture, such that its routines 
+cover a wide diversity of tasks, including:
 
 * image pre-processing, such as sky subtraction, bad pixel correction, bad
 frame removal, or image alignment and star centering (`preproc` module);
@@ -119,25 +146,25 @@ community for the discovery of low-mass companions
 [@Milli:2017;  @Hirsch:2019;  @Ubeira:2020], their characterization 
 [@Wertz:2017;  @Delorme:2017;  @Christiaens:2018;  @Christiaens:2019], the study 
 of planet formation [@Ruane:2017;  @Reggiani:2018;  @Mauco:2020;  @Toci:2020], 
-the study of high-mass star formation [@Rainot:2020;  @Rainot:2022] ,the study 
+the study of high-mass star formation [@Rainot:2020;  @Rainot:2022], the study 
 of debris disks [@Milli:2017b; @Milli:2019], or the development of new 
 high-contrast imaging algorithms 
-[@Gomez:2018;  @Dahlqvist:2020;  @Pairet:2021;  @Dahlqvist:2021]. Given the 
-rapid expansion of ``VIP``, we summarize here all novelties that were brought 
-to the package over the past five years.
-
-The rest of this manuscript summarizes all major changes since v0.7.0
-[@Gomez:2017], that are included in the latest release of ``VIP`` (v1.3.1). At
-a structural level, ``VIP`` underwent a major change since version v1.1.0, which
-aimed to migrate towards a more streamlined and easy-to-use architecture. The
-package now revolves around five major modules (`fm`, `invprob`, `metrics`,
-`preproc` and `psfsub`, as described above) complemented by four additional
-modules containing various utility functions (`config`, `fits`,
-`stats` and `var`). New `Dataset` and `Frame` classes have also been
-implemented, enabling an object-oriented approach for processing high-contrast
-imaging datasets and analyzing final images, respectively. Similarly, a
-`HCIPostProcAlgo` class and different subclasses inheriting from it have been
-defined to facilitate an object-oriented use of ``VIP`` routines.
+[@Gomez:2018;  @Dahlqvist:2020;  @Pairet:2021;  @Dahlqvist:2021]. 
+
+Given the rapid expansion of ``VIP``, we summarize here all novelties that were 
+brought to the package over the past five years. Specifically, the rest of this 
+manuscript summarizes all major changes since v0.7.0 [@Gomez:2017], that are 
+included in the latest release of ``VIP`` (v1.3.1). At a structural level, 
+``VIP`` underwent a major change since version v1.1.0, which aimed to migrate 
+towards a more streamlined and easy-to-use architecture. The package now 
+revolves around five major modules (`fm`, `invprob`, `metrics`, `preproc` and 
+`psfsub`, as described above) complemented by four additional modules containing 
+various utility functions (`config`, `fits`, `stats` and `var`). New `Dataset` 
+and `Frame` classes have also been implemented, enabling an object-oriented 
+approach for processing high-contrast imaging datasets and analyzing final 
+images, respectively. Similarly, a `HCIPostProcAlgo` class and different 
+subclasses inheriting from it have been defined to facilitate an object-oriented 
+use of ``VIP`` routines.
 
 Some of the major changes in each module of ``VIP`` are summarized below:
 
@@ -174,8 +201,8 @@ Some of the major changes in each module of ``VIP`` are summarized below:
     iterative sigma filtering (`cube_fix_badpix_clump`), the circular symmetry
     of the PSF (`cube_fix_badpix_annuli`), or the radial expansion of the PSF
     with increasing wavelength (`cube_fix_badpix_ifs`), and (ii) the correction
-    of bad pixels based on either median replacement (default) or Gaussian
-    kernel interpolation (`cube_fix_badpix_with_kernel`);
+    of bad pixels with iterative spectral deconvolution [@Aach:2001] or 
+    Gaussian kernel interpolation (both through `cube_fix_badpix_interp`);
     - a new algorithm was added for the recentering of coronagraphic image cubes
     based on the cross-correlation of the speckle pattern, after appropriate
     filtering and log-scaling of pixel intensities [@Ruane:2019].

tests/test_fm_negfc_3d.py

@@ -47,7 +47,7 @@ def injected_cube_position(example_dataset_adi):
                  (pca, firstguess, 3, True, None, False, None),
                  (median_sub, firstguess, None, False, 'sum', False, None),
                  (pca_annulus, mcmc_negfc_sampling, 3, False, 'stddev', False, 'gb'),
-                 (pca_annulus, mcmc_negfc_sampling, 5, True, None, True, 'ac'),
+                 (pca_annulus, mcmc_negfc_sampling, 3, True, None, True, 'ac'),
                  (pca_annulus, nested_negfc_sampling, 3, False, 'sum', False, None)
                  ])
 def test_algos(injected_cube_position, pca_algo, negfc_algo, ncomp, mu_sigma,
@@ -145,7 +145,7 @@ def test_algos(injected_cube_position, pca_algo, negfc_algo, ncomp, mu_sigma,
                 j = i
             ci_max = np.amax(np.abs(ci[lab]))
             aarc(val_max[lab], gt[j], atol=3*ci_max)  # diff within 3 sigma
-            aarc(mu[i], gt[j], atol=3*sigma[i])  # diff within 3 sigma
+            aarc(val_max[lab], gt[j], atol=3*sigma[i])  # diff within 3 sigma
     else:
         # run nested sampling
         res = negfc_algo(init, ds.cube, ds.angles, ds.psf, ds.fwhm,

tests/test_preproc_badpix.py

@@ -6,7 +6,7 @@
 import numpy as np
 from vip_hci.preproc import (cube_fix_badpix_isolated, cube_fix_badpix_clump,
                              cube_fix_badpix_annuli, cube_fix_badpix_ifs,
-                             cube_fix_badpix_with_kernel)
+                             cube_fix_badpix_interp)
 from vip_hci.var import (create_synth_psf, dist, get_annulus_segments,
                          frame_center)
 
@@ -179,17 +179,19 @@ def test_badpix_ann():
 
     # test kernel correction
     cy, cx = frame_center(cube)
-    cube_c = cube_fix_badpix_with_kernel(cube, bpm, fwhm=1)
+    cube_c_gau = cube_fix_badpix_interp(cube, bpm, mode='gauss', fwhm=1)
+    cube_c_fft = cube_fix_badpix_interp(cube, bpm, mode='fft', nit=50)
 
     r0 = dist(cy, cx, idx0, idx0)
     ann = get_annulus_segments(cube_c[0], r0-1, 3, mode='val')
     med_val_ann = np.median(ann)
-    assert (cube_c[0, idx0, idx0]-med_val_ann) < 4*s0
+    assert (cube_c_gau[0, idx0, idx0]-med_val_ann) < 4*s0
+    assert (cube_c_fft[0, idx0, idx0]-med_val_ann) < 4*s0
 
     r1 = dist(cy, cx, idx1, idx1)
     ann = get_annulus_segments(cube_c[1], r1-1, 3, mode='val')
     med_val_ann = np.median(ann)
-    assert (cube_c[1, idx1, idx1]-med_val_ann) < 4*s0
+    assert (cube_c_gau[1, idx1, idx1]-med_val_ann) < 4*s0
 
 
 def test_badpix_ifs1():

vip_hci/fm/negfc_mcmc.py

@@ -2,6 +2,30 @@
 
 """
 Module with the MCMC (``emcee``) sampling for NEGFC parameter estimation.
+
+.. [CHR21]
+   | Christiaens et al. 2021
+   | **A faint companion around CrA-9: protoplanet or obscured binary?**
+   | *MNRAS, Volume 502, Issue 4, pp. 6117-6139*
+   | `https://arxiv.org/abs/astro-ph/2102.10288
+     <https://arxiv.org/abs/astro-ph/2102.10288>`_
+     
+.. [FOR13]
+   | Foreman-Mackey et al. 2013
+   | **emcee: The MCMC Hammer**
+   | *PASP, Volume 125, Issue 925, p. 306*
+   | `https://arxiv.org/abs/astro-ph/1202.3665
+     <https://arxiv.org/abs/astro-ph/1202.3665>`_
+     
+.. [WER17]
+   | Wertz et al. 2017
+   | **VLT/SPHERE robust astrometry of the HR8799 planets at milliarcsecond-level 
+     accuracy. Orbital architecture analysis with PyAstrOFit**
+   | *Astronomy & Astrophysics, Volume 598, Issue 1, p. 83*
+   | `https://arxiv.org/abs/astro-ph/1610.04014
+     <https://arxiv.org/abs/astro-ph/1610.04014>`_
+
+
 """
 
 
@@ -404,18 +428,18 @@ def mcmc_negfc_sampling(cube, angs, psfn, initial_state, algo=pca_annulus,
     circular aperture centered on the initial guess.
     4) We calculate a function of merit :math:`\chi^2` (see below).
     The steps 1) to 4) are then looped. At each iteration, the candidate model
-    parameters are defined by the emcee Affine Invariant algorithm.
+    parameters are defined by the ``emcee`` Affine Invariant algorithm [FOR13]_.
 
     There are different possibilities for the figure of merit (step 4):
         
-        - mu_sigma=None; fmerit='sum' (as in Wertz et al. 2017):\
+        - mu_sigma=None; fmerit='sum' (as in [WER17]_):\
         :math:`\chi^2 = \sum(\|I_j\|)`
         
         - mu_sigma=None; fmerit='stddev' (likely more appropriate when speckle\
         noise still significant): \
         :math:`\chi^2 = N \sigma_{I_j}(values,ddof=1)*`values.size
         
-        - mu_sigma=True or a tuple (as in Christiaens et al. 2021, new default):\
+        - mu_sigma=True or a tuple (as in [CHR21]_, new default):\
         :math:`\chi^2 = \sum\frac{(I_j- mu)^2}{\sigma^2}`
 
     where :math:`j \in {1,...,N}` with N the total number of pixels
@@ -478,7 +502,7 @@ def mcmc_negfc_sampling(cube, angs, psfn, initial_state, algo=pca_annulus,
         The FHWM in pixels.
     mu_sigma: tuple of 2 floats or bool, opt
         If set to None: not used, and falls back to original version of the
-        algorithm, using fmerit (Wertz et al. 2017).
+        algorithm, using ``fmerit`` [WER17]_.
         If a tuple of 2 elements: should be the mean and standard deviation of
         pixel intensities in an annulus centered on the location of the
         companion candidate, excluding the area directly adjacent to the CC.
@@ -494,8 +518,8 @@ def mcmc_negfc_sampling(cube, angs, psfn, initial_state, algo=pca_annulus,
     fmerit : {'sum', 'stddev'}, string optional
         If mu_sigma is not provided nor set to True, this parameter determines
         which figure of merit to be used among the 2 possibilities implemented
-        in Wertz et al. (2017). 'stddev' may work well for point like sources
-        surrounded by extended signals.
+        in [WER17]_. 'stddev' may work well for point like sources surrounded by 
+        extended signals.
     cube_ref : 3d or 4d numpy ndarray, or list of 3d ndarray, optional
         Reference library cube for Reference Star Differential Imaging. Should
         be 3d, except if the input cube is 4d, in which case it can either be a

vip_hci/fm/negfc_nested.py

@@ -3,6 +3,14 @@
 """
 Module with functions for posterior sampling of the NEGFC parameters using
 nested sampling (``nestle``).
+
+.. [BAR13]
+   | K. Barbary 2013
+   | **PyMORESANE**
+   | *GitHub repository*
+   | `https://github.com/kbarbary/nestle
+     <https://github.com/kbarbary/nestle>`_
+
 """
 
 
@@ -28,12 +36,13 @@ def nested_negfc_sampling(init, cube, angs, psfn, fwhm, mu_sigma=True,
                           weights=None, w=(5, 5, 200), method='single',
                           npoints=100, dlogz=0.1, decline_factor=None,
                           rstate=None, verbose=True):
-    """ Runs a nested sampling algorithm in order to determine the position and
-    the flux of the planet using the 'Negative Fake Companion' technique. The
-    result of this procedure is a a ``nestle`` object containing the samples
-    from the posterior distributions of each of the 3 parameters. It provides
-    pretty good results (value plus error bars) compared to a more CPU intensive
-    Monte Carlo approach with the affine invariant sampler (``emcee``).
+    """ Runs a nested sampling algorithm with ``nestle`` [BAR13] in order to 
+    determine the position and the flux of the planet using the 'Negative Fake 
+    Companion' technique. The result of this procedure is a ``nestle`` object 
+    containing the samples from the posterior distributions of each of the 3 
+    parameters. It provides good results (value plus error bars) compared to a 
+    more CPU intensive Monte Carlo approach with the affine invariant sampler 
+    (``emcee``).
 
     Parameters
     ----------

vip_hci/fm/negfc_simplex.py

@@ -3,6 +3,15 @@
 """
 Module with simplex (Nelder-Mead) optimization for defining the flux and
 position of a companion using the Negative Fake Companion.
+
+.. [WER17]
+   | Wertz et al. 2017
+   | **VLT/SPHERE robust astrometry of the HR8799 planets at milliarcsecond-level 
+     accuracy. Orbital architecture analysis with PyAstrOFit**
+   | *Astronomy & Astrophysics, Volume 598, Issue 1, p. 83*
+   | `https://arxiv.org/abs/astro-ph/1610.04014
+     <https://arxiv.org/abs/astro-ph/1610.04014>`_
+     
 """
 
 
@@ -393,7 +402,8 @@ def firstguess(cube, angs, psfn, ncomp, planets_xy_coord, fwhm=4,
                mu_sigma=True, wedge=None, weights=None, force_rPA=False,
                algo_options={}, simplex=True, simplex_options=None, plot=False,
                verbose=True, save=False):
-    """ Determines a first guess for the position and the flux of a planet.
+    """ Determines a first guess for the position and the flux of a planet, as
+    explained in [WER17]_.
 
     We process the cube without injecting any negative fake companion.
     This leads to the visual detection of the planet(s). For each of them,

vip_hci/fm/scattered_light_disk.py

@@ -1,6 +1,14 @@
 #! /usr/bin/env python
 """
 Class definition for ScatteredLightDisk, Dust_distribution and Phase_function
+     
+.. [AUG99]
+   | Augereau et al. 1999
+   | **On the HR 4796 A circumstellar disk**
+   | *Astronomy & Astrophysics, Volume 348, pp. 557-569*
+   | `https://arxiv.org/abs/astro-ph/9906429
+     <https://arxiv.org/abs/astro-ph/9906429>`_
+     
 """
 
 __author__ = 'Julien Milli'
@@ -18,8 +26,8 @@
 class ScatteredLightDisk(object):
     """
     Class used to generate a synthetic disc, inspired from a light version of
-    the GRATER tool (GRenoble RAdiative TransfER, Augereau et al. 1999)
-    written originally in IDL and converted to Python by J. Milli.
+    the GRATER tool (GRenoble RAdiative TransfER) written originally in IDL 
+    [AUG99]_, and converted to Python by J. Milli.
     """
 
     def __init__(self, nx=200, ny=200, distance=50., itilt=60., omega=0.,

vip_hci/invprob/andromeda.py

@@ -41,7 +41,9 @@ def andromeda(cube, oversampling_fact, angles, psf, filtering_fraction=.25,
               homogeneous_variance=True, ditimg=1.0, ditpsf=None, tnd=1.0,
               total=False, multiply_gamma=True, nproc=1, verbose=False):
     """
-    Exoplanet detection in ADI sequences by maximum-likelihood approach.
+    Exoplanet detection in ADI sequences by maximum-likelihood approach, as 
+    implemented in [CAN15]_, itself inspired by the framework presented in 
+    [MUG09]_.
 
     Parameters
     ----------