Slides.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "from astropy.io import fits\n",
    "from astropy.convolution import Gaussian2DKernel\n",
    "base = \"/home/max/Development/semester-project/data/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Multi Mode Source Extraction for the James Webb Space Telescope\n",
    "\n",
    "----\n",
    "_Massimo Bortone, Institute of Particle Physics and Astrophysics, ETH Zürich_\n",
    "![JWST](images/jwst-nasa.jpeg \"JWST\")\n",
    "- _Supervisors at Harvard-Smithsonian Center for Astrophysics: Dr. Sandro Tacchella, Prof. Daniel Eisenstein, Dr. Ben Johnson_\n",
    "- _Supervisor at ETH Zürich: Prof. Alexandre Refregier_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Outline\n",
    "----\n",
    "1. Motivations & Objectives\n",
    "2. SExtractor\n",
    "3. Overshredding\n",
    "4. Kernels\n",
    "5. Multi Mode Source Extraction\n",
    "6. Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Motivations\n",
    "----\n",
    "* Overlapping sources in deep images are a problem for photometry tools\n",
    "* Need for realistic uncertainties and degeneracies in photometric measurements\n",
    "* Use information from all bands of the NIRCam instrument on JWST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- Aperture photometry methods like *Kron's first moment* algorithm fail when sources are overlapping\n",
    "- Better estimates of photometric redshifts, colors and color gradients are need to accurately study the evolution of early galaxies\n",
    "- NIRCam will allow detection of galaxy at redshifts possibly beyond $z>15$, important to include multiple bands do detect dropouts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Objectives\n",
    "----\n",
    "* Detect as many possible sources in astronomical images of deep fields\n",
    "* Identify sub-structure components in galaxies (*overshred detection*)\n",
    "* Combine detections from multiple bands\n",
    "* Forward model the image with `FORCEPho` to obtain probabilistic assignments of fluxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The HXDF\n",
    "----\n",
    "<div style=\"width: 100%;\">\n",
    "    <div style=\"float: left; width: 50%\">\n",
    "        <img src=\"images/xdf.png\" alt=\"XDF and cutout region\" />\n",
    "    </div>\n",
    "    <ul style=\"float: left; width: 45%\">\n",
    "        <li>$20 ~\\text{days}$ total exposure</li>\n",
    "        <li>$2.3\\times 2 ~\\text{arcminutes}^2$ in size, which is about $\\frac{1}{32\\times 10^6}$ of the sky</li>\n",
    "        <li>generated using data from Hubble's ACS/WFC and WFC3/IR cameras</li>\n",
    "        <li>deepest image of the sky in the optical/near-IR regime</li>\n",
    "        <li>about 5500 galaxies, with redshift $z<10$</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Multiple bands\n",
    "----\n",
    "<div style=\"width: 100%;\">\n",
    "    <div style=\"float: left; width: 70%\">\n",
    "        <img src=\"images/H-I-bands.png\" alt=\"H and I bands of the XDF\" />\n",
    "    </div>\n",
    "    <ul style=\"float: left; width: 27%\">\n",
    "        <li>detect dropouts</li>\n",
    "        <li>improve detection of sub-structure components</li>\n",
    "        <li>H band: F160W, 1600nm</li>\n",
    "        <li>I band: F814W, 814nm</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Pipeline Overview](images/pipeline-overview.jpeg \"Data-Analysis Pipeline Overview\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# FORCEPho\n",
    "----\n",
    "<div style=\"width: 100%;\">\n",
    "    <div style=\"float: left; width: 50%\">\n",
    "        <img src=\"images/forcepho-example.png\" alt=\"FORCEPho example\"/>\n",
    "    </div>\n",
    "    <ul style=\"float: left; width: 47%\">\n",
    "        <li>Gaussian mixtures for sources and PSF</li>\n",
    "        <li>2nd-order approximation to pixel integral of a Gaussian to fit undrizzled, undersampled exposures</li>\n",
    "        <li>Sersic profiles with 6 parameters to model galaxies, approximated by GMs</li>\n",
    "        <li>Hamiltonian-Monte-Carlo optimization of $\\chi^2$ ln-likelihood</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# SExtractor\n",
    "----\n",
    "Developed in 1996 by E. Bertin and S. Arnouts, is a signal-based method to extract sources from large astronomical images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Fast and based on well motivated heuristic, but requires good tuning of the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How it works\n",
    "----\n",
    "The complete analysis of an image consists of 6 steps:\n",
    "1. estimation of sky background\n",
    "2. **thresholding**\n",
    "3. **deblending**\n",
    "4. cleaning\n",
    "5. photometry\n",
    "6. star/galaxy separation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "1. assume image is already background subtracted or background is sufficiently flat\n",
    "4. not relevant for our goals\n",
    "5. handled by FORCEPho\n",
    "6. not important in deep images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Thresholding\n",
    "----\n",
    "* convolve original image with a filtering kernel \n",
    "* thresholding is local for every pixel with position $(x,~y)$:\n",
    "$$I_{ij} > \\sigma \\cdot \\sqrt{V_{ij}},~\\forall (i,~j) \\in G= \\{(x \\pm 1,~y), (x,~y\\pm 1), (x,~y\\pm 1)\\}$$\n",
    "where $I_{ij}$ and $V_{ij}$ are the intensity and the variance of the pixel at position $(i, j)$ on the image, while $\\sigma$ is the detection threshold\n",
    "* extract 8-connected contigous pixels above threshold from detection image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Deblending\n",
    "----\n",
    "* re-threshold each extracted set of 8-connected pixels at `NTHRESH` exponentially spaced levels between the primary extraction threshold $\\sigma$ and the peak value in the extracted set\n",
    "* model light distribution of the object as a tree structure\n",
    "* `MINCONT` parameter defines the minimal contrast $\\delta_c$ between the integrated pixel density in the branches and the total density of the composite object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**CONDITIONS FOR SPLITTING**:\n",
    "\n",
    "<div style=\"width: 100%;\">\n",
    "    <ol style=\"float: left; width: 50%; margin-top: 40px;\">\n",
    "        <li>$\\frac{\\text{integrated pixel density in branch}}{\\text{total density}} > \\delta_c$</li>\n",
    "        <li>the first condition is true for another branch at the same level</li>\n",
    "    </ol>\n",
    "    <div style=\"float: left; width: 45%\">\n",
    "        <img src=\"images/sextractor-deblending.png\" alt=\"Deblending in SExtractor\" />\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Overshredding\n",
    "----\n",
    "Apart from $\\sigma$, the detection behavior of SExtractor is controlled by\n",
    "- `MINAREA`: minimal number of pixels that define an object\n",
    "- `NTHRESH`: number of exponentially spaced levels\n",
    "- `MINCONT`: minimal contrast needed to split an object in two components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- `MINAREA` is quite straightforward: the smaller, the more \"objects\" are detected\n",
    "- set `MINAREA` to 5 pixels, corresponds to about 1 FWHM of the PSF or about $0.18\\times 0.18~\\text{arcseconds}^2$\n",
    "- not clear which values of the other parameters are good for our goals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exploring the parameter space\n",
    "----\n",
    "Explore the parameter space to find which combination of values of `NTHRESH` and `MINCONT` yield to overshredding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nthresh = range(32, 1025, 32)\n",
    "mincont = np.logspace(0, -10, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![grid](images/mincont-nthresh-grid.png \"Detections as a function of NTHRESH and MINCONT\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**1. CONCLUSION** \n",
    "\n",
    "`MINCONT` has a larger effect on the number of detections than `NTHRESH`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![grid](images/detection-vs-mincont.png \"Detections as a function MINCONT for NTHRESH=64\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**2. CONCLUSION** \n",
    "\n",
    "Number of detections saturates for `MINCONT`$<10^{-6}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "--------\n",
    "From now on we fix the parameters to:\n",
    "- `NTHRESH`=$64$\n",
    "- `MINCONT`=$10^{-7}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kernel\n",
    "----\n",
    "A kernel is a matrix of a given size that is convolved with the image to increase the sensitivity of the detection towards specific objects\n",
    "\n",
    "## Examples\n",
    "\n",
    "* PSF kernel: good for faint unresolved objects\n",
    "* flat kernel: better for LSB objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "psf_filename = os.path.join(base, \"ZC404221_H_PSF.fits\")\n",
    "psf_hdu = fits.open(psf_filename)\n",
    "psf = psf_hdu[0].data\n",
    "psf_clipped = np.array(psf[40:60, 40:60])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "PSF has FWHM of 3 pixels, which corresponds to an area of $0.18\\times 0.18 ~\\text{arcseconds}^2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "psf_default = np.zeros_like(psf_clipped)\n",
    "psf_default[9:12, 9] = [1, 2, 1]\n",
    "psf_default[9:12, 10] = [2, 4, 2]\n",
    "psf_default[9:12, 11] = [1, 2, 1]\n",
    "psf_default /= 4.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "flat_5 = np.zeros_like(psf_clipped)\n",
    "flat_5[8:13, 8:13] = np.ones((5, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "one_hot = np.zeros_like(psf_clipped)\n",
    "one_hot[10, 10] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![Kernels](images/kernels.png \"4 different kernels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Comparing the number of detections as a function of area for the 4 different kernels we notice:\n",
    "\n",
    "![Histograms](images/hist-kernels.png \"Histograms of detections for 4 different kernels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- `ONE-HOT` kernel detects a lot of small scale structure\n",
    "- `PSF` and `FLAT` kernels are sensible to more extended objects\n",
    "- about 1 order of magnitude difference in average area of object detected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Overplotting all detections we find:\n",
    "![All detections](images/all-kernels.png \"Detected sources for 4 different kernels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- SExtractor with the `ONE-HOT` kernel can be use to detect substructures in bright extended objects\n",
    "- broad kernels should be used to detect bright objects, then replacing them with sub-structure components from narrow kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Multi Mode Source Extractor\n",
    "----\n",
    "**IDEA:** \n",
    "run SExtractor in different modes and on different bands, allowing each mode to specify its own set of SExtractor parameters. Then merge detections into one single catalog."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "----\n",
    "**PROBLEM:** \n",
    "merging detections from multiple modes and bands is not straightforward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Defining modes and bands\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'np' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-9d6cfc14e24f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mkernel_cold\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mGaussian2DKernel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mkernel_hot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mkernel_hot\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'np' is not defined"
     ]
    }
   ],
   "source": [
    "kernel_cold = np.asarray(Gaussian2DKernel(3))\n",
    "kernel_hot = np.zeros((3,3))\n",
    "kernel_hot[1,1] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "modes = {\n",
    "    'H': {\n",
    "        # detect large bright sources\n",
    "        'cold': {\n",
    "            'sigma': 1.5,\n",
    "            'mincont': 1e-7,\n",
    "            'nthresh': 64,\n",
    "            'kernel': kernel_cold,\n",
    "            'minarea': 5\n",
    "        },\n",
    "        # detect small isolated sources\n",
    "        'hot': {\n",
    "            'sigma': 1.5,\n",
    "            'mincont': 1e-7,\n",
    "            'nthresh': 64,\n",
    "            'kernel': kernel_hot,\n",
    "            'minarea': 5\n",
    "        }\n",
    "    },\n",
    "    'I': {\n",
    "        # detect substructure in large bright sources\n",
    "        'hot': {\n",
    "            'sigma': 2.0,\n",
    "            'mincont': 1e-7,\n",
    "            'nthresh': 64,\n",
    "            'kernel': kernel_hot,\n",
    "            'minarea': 5\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "bands = {\n",
    "    'H': {\n",
    "        'image': image_h,\n",
    "        'rms': rms_h\n",
    "    },\n",
    "    'I': {\n",
    "        'image': image_i,\n",
    "        'rms': rms_i\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Finding substructures\n",
    "----\n",
    "The idea is to use a small kernel on the I band to detect sub-structures within the brightest objects detected in the H band. To do this we need to:\n",
    "\n",
    "1. Compute fluxes for all objects\n",
    "2. Select brightest objects in the H band\n",
    "3. Look for sub-structures in the I band"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![H and I detections](images/H-I-detections.png \"Detections in the H and I bands\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 1. Compute flux within elliptical aperture for each detection\n",
    "\n",
    "$$\n",
    "\\texttt{CXX}(x-\\bar{x})^2 + \\texttt{CYY}(y-\\bar{y})^2 + \\texttt{CXY}(x-\\bar{x})(y-\\bar{y}) = R^2\n",
    "$$\n",
    "![Integrated flux](images/H-integrated-flux.png \"Integrated flux for detections in the H band\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- CXX, CYY, CXY are computed by SExtractor from the object's light distribution second moments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 2. Select top 20% brightest objects in the H band\n",
    "![Top 20% detections](images/H-flux-selection.png \"Top 20% brightest detections in the H band\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 3. Search for detections in the I band that are within an elliptical aperture of 2x the dimensions determined by SExtractor\n",
    "![I Sub-structures](images/I-band-substructure.png \"Sub-structure detection in the I band\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 4. Repeat same steps to find sub-structures on the H band\n",
    "![H Sub-structures](images/H-band-substructure.png \"Sub-structure detection in the H band\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 5. Compare sub-structures in both bands\n",
    "![Compare sub-structures](images/H-vs-I-substructure.png \"Comparison between substructure detected in the H and I bands\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Merging\n",
    "----\n",
    "In order to merge the detections obtained for the different bands and modes we need to:\n",
    "\n",
    "1. Remove duplicates across bands and modes\n",
    "2. Remove brightest objects in H band\n",
    "3. Replace sub-structures in H band with those found in the I band"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Finding similar detections\n",
    "----\n",
    "Given two objects $V$ and $W$, we define a proximity function as follows:\n",
    "\n",
    "$$\n",
    " P(V, W) = \\sqrt{(\\bar{x}_V - \\bar{x}_W)^2 + (\\bar{y}_V - \\bar{y}_W)^2} + \\vert s_V - s_W \\vert + \\vert \\theta_V - \\theta_W \\vert\n",
    "$$\n",
    "\n",
    "where $(\\bar{x}_i,~\\bar{y}_i)$ are the center coordinates, $s_i = a_i \\cdot b_i$ is the surface of the ellipse estimated from the light distribution and $\\theta_i$ is its orientation with respect to the horizontal axis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Pairs of similar objects are those for which $P(V, W)$ is below a certain user-defined threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 1.1 Remove duplicates across modes in the H band\n",
    "\n",
    "![Duplicates H band](images/H-band-hot-cold-similar.png \"Duplicates across modes in the H band\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 1.2 Remove duplicates across H and I bands in hot mode\n",
    "\n",
    "![Duplicates hot mode](images/H-I-band-hot-hot-similar.png \"Duplicates across H and I bands in hot mode\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Final positions\n",
    "----\n",
    "![Final posititons](images/final-positions.png \"Final positions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- sub-structure detection (*overshredding*) is effective for bright objects\n",
    "- combining multiple bands helps detecting dropouts\n",
    "- spurious detections in bright halos around large objects need to be further confirmed, maybe using photometry information from `FORCEPho`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Comparison to other catalogs\n",
    "----\n",
    "We compare our detections to those found by:\n",
    "- Bouwens+ 2015 (see http://vizier.cfa.harvard.edu/viz-bin/VizieR?-source=J/ApJ/803/34)\n",
    "- Guo+ 2013 (CANDELS GOODS-S catalog, see http://vizier.cfa.harvard.edu/viz-bin/VizieR?-source=J/ApJS/207/24)\n",
    "- Finkelstein+ 2015 (see http://vizier.cfa.harvard.edu/viz-bin/VizieR?-source=J/ApJ/810/71)\n",
    "- Skelton+2014 (3D-HST catalog, see http://adsabs.harvard.edu/abs/2014ApJS..214...24S)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Source extraction methods:\n",
    "\n",
    "- Bouwens and Finkelstein are optimized towards detection of high-redshift galaxies ($z>4$)\n",
    "- Bouwens stacks $Y_{098}Y_{105}J_{125}H_{160}$ WFC3/IR observations\n",
    "- Finkelstein adopts weighted sum of F125W and F160W images\n",
    "- Guo uses a similar strategy to ours with hot and cold source extraction modes\n",
    "- 3D-HST uses coadded F125W + F140W + F160W images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Overplotting all detections we find:\n",
    "    \n",
    "![All catalogs](images/all-mmse.png \"Detections from all catalogs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Results**:\n",
    "\n",
    "![Detection fraction](images/detection-recovery.png \"Detection fraction\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Stacking\n",
    "----\n",
    "Agreement with `Bouwens` and `Finkelstein` catalogs can be improved by stacking YJH bands together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![Stacked](images/Hstacked-mmse.png \"Stacked H band detections\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What's next\n",
    "----\n",
    "\n",
    "- measure distance between peak and barycentre in each detected object to determine the width of a prior on its final position\n",
    "- determine detection recovery as a function of confusion by introducing artificial objects on the image\n",
    "- using simulated images:\n",
    "    - quantify false positive and true negative detections\n",
    "    - find optimal kernels for different kinds of objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- for points that close to each other, need to possibly fix one in order to avoid source-swapping in FORCEPho\n",
    "- eliminate spurious detections by commparing FORCEPho model with initial detections, remove those with zero posteriors"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "livereveal": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "320px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}