Merge pull request #100 from LSSTDESC/auxtel_fit_PT

Auxtel fit pt
LSSTDESC · Dec 5, 2022 · ff1adf7 · ff1adf7
2 parents ac6ee79 + 447a97d
commit ff1adf7
Show file tree

Hide file tree

Showing 78 changed files with 89,387 additions and 113,473 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -32,7 +32,13 @@ jobs:
         # Might be quicker to install rubin-env plus any necessary additions.
         shell: bash -l {0}
         run: |
-          mamba install -y -q numpy>1.15 scipy matplotlib>3.1 pandas llvmlite numba astropy>=3.2 photutils>=0.7 astroquery coloredlogs scikit-image h5py emcee tqdm mpi4py schwimmbad iminuit>=2 coverage>=3.6 configparser coveralls deprecated pyyaml nose getCalspec
+          mamba install -y -q numpy>1.15 scipy matplotlib>3.1 pandas llvmlite numba astropy>=3.2 photutils>=0.7 astroquery coloredlogs scikit-image h5py emcee tqdm mpi4py schwimmbad iminuit>=2 coverage>=3.6 configparser coveralls deprecated pyyaml nose rubin-libradtran getCalspec>=0.1.6
+          python -c "from getCalspec.rebuild import rebuild_tables; rebuild_tables()"
+          mamba install astrometry
+          wget -r -q -nc http://data.astrometry.net/5000/index-5002-24.fits
+          wget -r -q -nc http://data.astrometry.net/5000/index-5000-40.fits
+          mv data.astrometry.net/5000/index-*.fits $CONDA_PREFIX/data/
+          export ASTROMETRYNET_DIR=$CONDA_PREFIX
 
       - name: List installed packages
         shell: bash -l {0}

diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -1,8 +1,31 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
 version: 2
 
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.9"
+    # You can also specify other tool versions:
+    # nodejs: "16"
+    # rust: "1.55"
+    # golang: "1.17"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: docs/conf.py
+
+# If using Sphinx, optionally build your docs in additional formats such as PDF
+# formats:
+#    - pdf
+
+# Optionally declare the Python requirements required to build your docs
 python:
-  version: 3.7
-  install:
-     - requirements: docs/requirements.txt
-     - method: pip
-       path: .
+   install:
+   - requirements: docs/requirements.txt
+   - method: pip
+     path: .
diff --git a/README.md b/README.md
@@ -18,21 +18,29 @@ Some submodules complete the structures with generic functions:
 - `spectractor.logbook`: tools to read logbook `.csv` text files and get some metadata relative to the data images that are not contained in the header;
 - `spectractor.tools`: contains generic functions shared by all  the subpackages (fitting procedures, plotting functions, etc).
 
-
 ## Installation
 
-Spectractor is written in Python 3.9. The dependencies are listed in the `requirements.txt` file. To install Spectractor, just run
+Spectractor is written in Python 3.9. The mandatory dependencies are listed in the `requirements.txt` file. To install Spectractor, just run
 ```
 pip install -r requirements.txt .
 ```
-For the simulation of spectra, Spectractor needs the following external libraries:
-- [libradtran](http://www.libradtran.org/doku.php) to simulate atmospheric transmission: it needs the installation of [netcdf](https://www.unidata.ucar.edu/software/netcdf/) and a python 2 environment (for the compilation only, not the usage); `uvpsec` executable must in the user `$PATH` or the user has to set an environmental variable `$LIBRADTRAN_DIR` pointing to the install directory.
-- [getCalspec](https://github.com/LSSTDESC/getCalspec) to get the CALSPEC star spectra;
-- [astrometry.net](https://astrometrynet.readthedocs.io/en/latest/) (optional): needed to create World Coordinate System files from the images; `solve-field` executable must in the user `$PATH` or the user has to set an environmental variable `$ASTROMETRYNET_DIR` pointing to the install directory. Version below or equal v0.78 should be used.
 
-Be careful, Spectractor can perform fits using the MCMC library [emcee](https://emcee.readthedocs.io/en/stable/) with [mpi4py](https://mpi4py.readthedocs.io/en/stable/) and [h5py](https://www.h5py.org/).  The latter might be better installed using `conda install ...` command to get their own dependencies (openmp and hdf5).
+To simulate atmospheric transmission, Spectractor needs [libradtran](http://www.libradtran.org/doku.php) (optional). There are two ways to install this software:
+- use the [Rubin observatory conda package](https://anaconda.org/conda-forge/rubin-libradtran)
+```
+conda install -c conda-forge rubin-libradtran
+```
+- compile it from [sources](http://www.libradtran.org/doku.php): it needs the installation of [netcdf](https://www.unidata.ucar.edu/software/netcdf/) and a python 2 environment (for the compilation only, not the usage); `uvpsec` executable must in the user `$PATH` or the user has to set an environmental variable `$LIBRADTRAN_DIR` pointing to the install directory.
 
-Detailled command lines for the installation of Spectractor and the external dependencies can be found in the file `.travis.yml`.
+To find accurate star centroids using astrometry, one can install [astrometry.net](https://astrometrynet.readthedocs.io/en/latest/) (optional). There are two ways to install this software:
+- use the anaconda package
+```
+conda install -c conda-forge astrometry
+```
+and then download and move [index files](http://astrometry.net/doc/readme.html#getting-index-files) in the `$CONDA_PREFIX/data` folder.
+- compile it from [sources](https://astrometrynet.readthedocs.io/en/latest/): it needs the installation of [netpbm](https://netpbm.sourceforge.net/) and [wcslib](https://www.atnf.csiro.au/people/mcalabre/WCS/wcslib/); `solve-field` executable must in the user `$PATH` or the user has to set an environmental variable `$ASTROMETRYNET_DIR` pointing to the install directory.
+
+Spectractor is able to perform parameter fits using the MCMC library [emcee](https://emcee.readthedocs.io/en/stable/) (optional) with [mpi4py](https://mpi4py.readthedocs.io/en/stable/) and [h5py](https://www.h5py.org/).  The latter might be better installed using `conda install ...` command to get their own dependencies (openmp and hdf5).
 
 ## Basic extraction
 
@@ -81,19 +89,3 @@ This object is also returned by the `Spectractor` function.
 
 A tutorial Jupyter notebook is available in the `notebooks` folder.
 
-## Detailed description
-
-### Dispersers
-
-### Shot noise
-
-### Rotation
-
-### Background extraction
-
-### Wavelength calibration
-#### First geometrical calibration
-#### Second calibration with line detection
-
-### Second order subtraction
-
diff --git a/config/auxtel.ini b/config/auxtel.ini
@@ -32,11 +32,11 @@ OBS_TRANSMISSION_SYSTEMATICS = 0.005
 # observed object to choose between STAR, HG-AR, MONOCHROMATOR
 OBS_OBJECT_TYPE = STAR
 # telescope transmission file
-OBS_TELESCOPE_TRANSMISSION = calexp_2020031500162-EMPTY_ronchi90lpmm-det000_auxtel_transmission.txt
+OBS_TELESCOPE_TRANSMISSION = multispectra_holo4_003_HD142331_AuxTel_throughput.txt
 # full instrument transmission file
-OBS_FULL_INSTRUMENT_TRANSMISSON = calexp_2020031500162-EMPTY_ronchi90lpmm-det000_auxtel_transmission.txt
+OBS_FULL_INSTRUMENT_TRANSMISSON = multispectra_holo4_003_HD142331_AuxTel_throughput.txt
 # quantum efficiency of the detector file
-OBS_QUANTUM_EFFICIENCY = calexp_2020031500162-EMPTY_ronchi90lpmm-det000_auxtel_transmission.txt
+OBS_QUANTUM_EFFICIENCY = multispectra_holo4_003_HD142331_AuxTel_throughput.txt
 # Camera (x,y) rotation angle with respect to (north-up, east-left) system
 OBS_CAMERA_ROTATION = 0
 # Camera (x,y) flip signs with respect to (north-up, east-left) system
@@ -91,10 +91,12 @@ ROT_ANGLE_MAX = 10
 LAMBDA_MIN = 300
 # maximum wavelength for spectrum extraction (in nm)
 LAMBDA_MAX = 1100
+# spectrum order to extract
+SPEC_ORDER = 1
 
 [background subtraction parameters]
 # half transverse width of the signal rectangular window in pixels
-PIXWIDTH_SIGNAL = 40
+PIXWIDTH_SIGNAL = 20
 # distance from dispersion axis to analyse the background in pixels
 PIXDIST_BACKGROUND = 140
 # transverse width of the background rectangular window in pixels
@@ -104,11 +106,11 @@ PIXWIDTH_BOXSIZE = 20
 
 [PSF]
 # the PSF model: Gauss, Moffat or MoffatGauss
-PSF_TYPE = Moffat
+PSF_TYPE = MoffatGauss
 # the order of the polynomials to model wavelength dependence of the PSF shape parameters
 PSF_POLY_ORDER = 2
 # regularisation parameter for the chisq minimisation to extract the spectrum
-PSF_FIT_REG_PARAM = 1
+PSF_FIT_REG_PARAM = 0.1
 # step size in pixels for the first transverse PSF1D fit
 PSF_PIXEL_STEP_TRANSVERSE_FIT = 50
 # PSF is not evaluated outside a region larger than max(PIXWIDTH_SIGNAL, PSF_FWHM_CLIP*fwhm) pixels

diff --git a/config/ctio.ini b/config/ctio.ini
@@ -92,7 +92,7 @@ LAMBDA_MAX = 1100
 
 [background subtraction parameters]
 # half transverse width of the signal rectangular window in pixels
-PIXWIDTH_SIGNAL = 30
+PIXWIDTH_SIGNAL = 20
 # distance from dispersion axis to analyse the background in pixels
 PIXDIST_BACKGROUND = 100
 # transverse width of the background rectangular window in pixels

diff --git a/docs/conf.py b/docs/conf.py
@@ -55,7 +55,8 @@ def __getattr__(cls, name):
               'sphinx.ext.napoleon',
               'matplotlib.sphinxext.plot_directive',
               'sphinx.ext.autosummary',
-              'm2r']  #
+              'sphinx_mdinclude',
+              ]  #
 
 plot_html_show_source_link = True
 plot_html_show_formats = False

diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -5,5 +5,6 @@ pandas
 astropy>3.1
 coloredlogs
 recommonmark
-m2r
-mistune==2.0.3
+mistune==2.0.3
+docutils<1.0,>=0.16
+sphinx-mdinclude
diff --git a/requirements.txt b/requirements.txt
@@ -21,4 +21,4 @@ coveralls
 deprecated
 pyyaml
 nose
-getCalspec
+getCalspec>=0.1.5
diff --git a/runExtractor.py b/runExtractor.py
@@ -17,8 +17,8 @@
                         help="Enter verbose (print more stuff).", default=False)
     parser.add_argument("-o", "--output_directory", dest="output_directory", default="outputs/",
                         help="Write results in given output directory (default: ./outputs/).")
-    parser.add_argument("-l", "--logbook", dest="logbook", default="ctiofulllogbook_jun2017_v5.csv",
-                        help="CSV logbook file. (default: ctiofulllogbook_jun2017_v5.csv).")
+    parser.add_argument("-l", "--logbook", dest="logbook", default="./tests/data/ctiofulllogbook_jun2017_v5.csv",
+                        help="CSV logbook file. (default: ./tests/data/ctiofulllogbook_jun2017_v5.csv).")
     parser.add_argument("-c", "--config", dest="config", default="config/ctio.ini",
                         help="INI config file. (default: config.ctio.ini).")
     parser.add_argument("-x", "--xy", dest="target_xy", default="0,0",

diff --git a/runImageSim.py b/runImageSim.py
@@ -15,8 +15,8 @@
                         help="Enter verbose (print more stuff).", default=False)
     parser.add_argument("-o", "--output_directory", dest="output_directory", default="outputs/",
                         help="Write results in given output directory (default: ./outputs/).")
-    parser.add_argument("-l", "--logbook", dest="logbook", default="ctiofulllogbook_jun2017_v5.csv",
-                        help="CSV logbook file. (default: ctiofulllogbook_jun2017_v5.csv).")
+    parser.add_argument("-l", "--logbook", dest="logbook", default="./tests/data/ctiofulllogbook_jun2017_v5.csv",
+                        help="CSV logbook file. (default: ./tests/data/ctiofulllogbook_jun2017_v5.csv).")
     parser.add_argument("-c", "--config", dest="config", default="config/ctio.ini",
                         help="INI config file. (default: config.ctio.ini).")
     args = parser.parse_args()

diff --git a/runSimulator.py b/runSimulator.py
@@ -18,8 +18,8 @@
                         help="Enter verbose (print more stuff).", default=False)
     parser.add_argument("-o", "--output_directory", dest="output_directory", default="outputs/",
                         help="Write results in given output directory (default: ./outputs/).")
-    parser.add_argument("-l", "--logbook", dest="logbook", default="ctiofulllogbook_jun2017_v5.csv",
-                        help="CSV logbook file. (default: ctiofulllogbook_jun2017_v5.csv).")
+    parser.add_argument("-l", "--logbook", dest="logbook", default="./tests/data/ctiofulllogbook_jun2017_v5.csv",
+                        help="CSV logbook file. (default: ./tests/data/ctiofulllogbook_jun2017_v5.csv).")
     parser.add_argument("-c", "--config", dest="config", default="config/ctio.ini",
                         help="INI config file. (default: config.ctio.ini).")
     args = parser.parse_args()

diff --git a/setup.cfg b/setup.cfg
@@ -1,2 +1,2 @@
 [nosetests]
-exclude=test_(extractor_ctio_planetary_nebula|fullchain|atmosphere|simulator|astrometry)
+exclude=test_(extractor_ctio_planetary_nebula|atmosphere|astrometry)
diff --git a/spectractor/_version.py b/spectractor/_version.py
@@ -28,5 +28,5 @@
 # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-__version__ = '2.1'
+__version__ = '2.3'
 __version_info__ = tuple(map(int, __version__.split('.')))
diff --git a/spectractor/astrometry.py b/spectractor/astrometry.py
@@ -1,6 +1,7 @@
 import os
 from copy import deepcopy
 import subprocess
+import shutil
 import numpy as np
 import matplotlib.pyplot as plt
 from astropy.stats import sigma_clipped_stats
@@ -196,7 +197,7 @@ def get_gaia_coords_after_proper_motion(gaia_catalog, date_obs):
     return gaia_coords_after_proper_motion
 
 
-def plot_shifts_histograms(dra, ddec):
+def plot_shifts_histograms(dra, ddec):  # pragma: no cover
     dra_median = np.median(dra.to(u.arcsec).value)
     ddec_median = np.median(ddec.to(u.arcsec).value)
     dra_rms = np.std(dra.to(u.arcsec).value)
@@ -222,7 +223,7 @@ def plot_shifts_histograms(dra, ddec):
         parameters.PdfPages.savefig()
 
 
-class Astrometry(Image):
+class Astrometry(Image):  # pragma: no cover
 
     def __init__(self, file_name, target_label="", disperser_label="", wcs_file_name="", output_directory=""):
         """Class to handle astrometric computations.
@@ -254,7 +255,7 @@ def __init__(self, file_name, target_label="", disperser_label="", wcs_file_name
         self.new_file_name = self.file_name.replace('.fits', '_new.fits')
         self.sources_file_name = set_sources_file_name(file_name, output_directory=output_directory)
         self.wcs_file_name = wcs_file_name
-        self.log_file_name = os.path.join(self.output_directory, self.tag) + ".log"
+        self.match_file_name = os.path.join(self.output_directory, self.tag) + ".match"
         self.wcs = None
         if self.wcs_file_name != "":
             if os.path.isfile(self.wcs_file_name):
@@ -285,7 +286,7 @@ def __init__(self, file_name, target_label="", disperser_label="", wcs_file_name
         self.dist_ra = 0 * u.arcsec
         self.dist_dec = 0 * u.arcsec
         self.target_radec_position_after_pm = self.target.get_radec_position_after_pm(date_obs=self.date_obs)
-        if os.path.isfile(self.log_file_name):
+        if os.path.isfile(self.match_file_name):
             self.quad_stars_pixel_positions = self.get_quad_stars_pixel_positions()
 
     def get_target_pixel_position(self):
@@ -379,17 +380,18 @@ def get_quad_stars_pixel_positions(self):
 
         """
         coords = []
-        f = open(self.log_file_name, 'r')
-        for line in f:
-            if 'field_xy' in line:
-                coord = line.split(' ')[5].split(',')
-                coords.append([float(coord[0]), float(coord[1])])
-        f.close()
+        hdu = fits.open(self.match_file_name)
+        table = Table.read(hdu)
+        hdu.close()
+        for k in range(0, len(table["QUADPIX"][0]), 2):
+            coord = [float(table["QUADPIX"][0][k]), float(table["QUADPIX"][0][k+1])]
+            if np.sum(coord) > 0:
+                coords.append(coord)
         if len(coords) < 4:
             self.my_logger.warning(f"\n\tOnly {len(coords)} calibration stars has been extracted from "
-                                   f"{self.log_file_name}, with positions {coords}. "
+                                   f"{self.match_file_name}, with positions {coords}. "
                                    f"A quad of at least 4 stars is expected. "
-                                   f"Please check {self.log_file_name}.")
+                                   f"Please check {self.match_file_name}.")
         self.quad_stars_pixel_positions = np.array(coords)
         return self.quad_stars_pixel_positions
 
@@ -880,7 +882,7 @@ def find_quad_star_index_in_sources(self, quad_star):
         k = -1
         for k in range(len(self.sources)):
             if abs(self.sources['xcentroid'][k] - quad_star[0]) < eps \
-                    and abs(self.sources['ycentroid'][k] - quad_star[1]):
+                    and abs(self.sources['ycentroid'][k] - quad_star[1]) < eps:
                 break
         return k
 
@@ -985,7 +987,7 @@ def run_simple_astrometry(self, extent=None, sources=None):
         >>> from spectractor.astrometry import Astrometry
         >>> from spectractor import parameters
         >>> parameters.VERBOSE = True
-        >>> logbook = LogBook(logbook='./ctiofulllogbook_jun2017_v5.csv')
+        >>> logbook = LogBook(logbook='./tests/data/ctiofulllogbook_jun2017_v5.csv')
         >>> file_names = ['./tests/data/reduc_20170530_134.fits']
         >>> if os.path.isfile('./tests/data/reduc_20170530_134_wcs/reduc_20170530_134.wcs'):
         ...     os.remove('./tests/data/reduc_20170530_134_wcs/reduc_20170530_134.wcs')
@@ -1027,17 +1029,25 @@ def run_simple_astrometry(self, extent=None, sources=None):
         # write results in fits file
         self.write_sources()
         # run astrometry.net
-        command = f"{os.path.join(parameters.ASTROMETRYNET_DIR, 'bin/solve-field')} --scale-unit arcsecperpix " \
+        if shutil.which('solve-field') != "":
+            exec = shutil.which('solve-field')
+        elif parameters.ASTROMETRYNET_DIR != "":
+            exec = os.path.join(parameters.ASTROMETRYNET_DIR, 'bin/solve-field')
+        else:
+            raise OSError(f"solve-field executable not found in $PATH "
+                          f"or {os.path.join(parameters.ASTROMETRYNET_DIR, 'bin/solve-field')}")
+
+        command = f"{exec} --scale-unit arcsecperpix " \
                   f"--scale-low {0.95 * parameters.CCD_PIXEL2ARCSEC} " \
                   f"--scale-high {1.05 * parameters.CCD_PIXEL2ARCSEC} " \
                   f"--ra {self.target.radec_position.ra.value} --dec {self.target.radec_position.dec.value} " \
                   f"--radius {parameters.CCD_IMSIZE * parameters.CCD_PIXEL2ARCSEC / 3600.} " \
                   f"--dir {self.output_directory} --out {self.tag} " \
                   f"--overwrite --x-column X --y-column Y {self.sources_file_name} " \
-                  f"--width {self.data.shape[1]} --height {self.data.shape[0]}"
+                  f"--width {self.data.shape[1]} --height {self.data.shape[0]} --no-plots"
         self.my_logger.info(f'\n\tRun astrometry.net solve_field command:\n\t{command}')
         log = subprocess.check_output(command, shell=True)
-        log_file = open(self.log_file_name, "w+")
+        log_file = open(self.match_file_name.replace(".match", ".log"), "w+")
         log_file.write(command + "\n")
         log_file.write(log.decode("utf-8") + "\n")
         # save new WCS in original fits file
@@ -1079,7 +1089,7 @@ def run_gaia_astrometry(self):
         >>> from spectractor import parameters
         >>> parameters.VERBOSE = True
         >>> parameters.DEBUG = True
-        >>> logbook = LogBook(logbook='./ctiofulllogbook_jun2017_v5.csv')
+        >>> logbook = LogBook(logbook='./tests/data/ctiofulllogbook_jun2017_v5.csv')
         >>> file_names = ['./tests/data/reduc_20170530_134.fits']
         >>> if os.path.isfile('./tests/data/reduc_20170530_134_wcs/reduc_20170530_134.wcs'):
         ...     os.remove('./tests/data/reduc_20170530_134_wcs/reduc_20170530_134.wcs')
@@ -1257,7 +1267,7 @@ def run_full_astrometry(self, extent=None, maxiter=20):
         >>> parameters.DEBUG = True
         >>> radius = 100
         >>> maxiter = 10
-        >>> logbook = LogBook(logbook='./ctiofulllogbook_jun2017_v5.csv')
+        >>> logbook = LogBook(logbook='./tests/data/ctiofulllogbook_jun2017_v5.csv')
         >>> file_names = ['./tests/data/reduc_20170530_134.fits']
         >>> if os.path.isfile('./tests/data/reduc_20170530_134_wcs/reduc_20170530_134.wcs'):
         ...     os.remove('./tests/data/reduc_20170530_134_wcs/reduc_20170530_134.wcs')

diff --git a/spectractor/config.py b/spectractor/config.py
@@ -115,7 +115,6 @@ def load_config(config_filename, rebin=True):
         apply_rebinning_to_parameters()
     else:
         parameters.CCD_REBIN = 1
-        my_logger.warning("No rebinning: parameters.REBIN is forced to 1.")
 
     # check consistency
     if parameters.PIXWIDTH_BOXSIZE > parameters.PIXWIDTH_BACKGROUND: