V0.2.0rc (#64)

* Add dimension reduction functionality (python impl. of gKDR)

* Tweaks to dimension reduction test

* Add documentation for DimensionReduction

* Add/fix some tests for DimensionReduction

* Fix test (DimensionReduction)

* Docstring -> raw string to fix invalid escape sequence warning

* Add sanity checks for parameters passed to DimensionReduction

* Fix assertion in DimensionReduction constructor

* Fix assertion in DimensionReduction constructor

* Test: optimize structural dimension

* First attempt at tuning the structural dimension (kdr)

* Make GaussianProcess objects callable (for prediction)

* Add 'train_model' interface to GaussianProcess

* DimensionReduction tweak to example

* Introduce X_scale/Y_scale parameters in gKDR kernel (DimensionReduction)

* Fix whitespace

* DimensionReduction: use specialized Hermitian eigensolver

* Tune parameters (structural dimension and kernel lengthscales) within gKDR

- rename tune_structural_dimension -> tune_parameters
- documentation
- improved optimization routine

* Improved tests for parameter tuning (gKDR)

* Tweak to test (DimensionReduction)

* Whitespace cleanup

* Correct naming of variable (DimensionReduction)

* Factor out internal loss function from gKDR.tune_parameters

* Use a smaller test example to reduce test runtime

* MCMC (#33)

* added separate functions to calculated squared exponential kernel

* added matern 5/2 covariance function

* put kernel computations into a separate function and removed conjugate gradient based unit test that always gave problems

* moved kernel functions and tests to separate files

* added function to compute gradient of the squared exponential kernel

* changed GP class to use derivative function

* added derivatives for matern 5/2 kernel

* quick and dirty modification to GP in order to use kernel functions

* cleaned up distance calculation to use standardized euclidean distance

* modified fast GP in MICE code to use kernel interface

* made correction to meaning of nugget parameter for MICE candidate GP to be relative to current variance

* fixed minor issues in MICE design to allow for zero samples and ensuring that parameter values are correctly set

* updated MICE benchmark details

* cosmetic tweaks to MICE benchmark

* full hessian implementation in kernel functions

* refactored kernel functions into objects

* implemented Hessian computation into GP class

* Documented base kernel class

* Documented derived kernel classes

* added documentation pages for kernels

* corrected documentation to include newly implemented classes and fixed some old bugs

* renamed run_init_design to be consistent with other methods that use *_initial_design

* made minor change in MICEFastGP documentation

* broke up prediction methods into single and multiple parameter sets, plus some other changes needed to accomodate them

* added routine to compute local covariance matrix from hessian

* implemented approximate normal hyperparameter sampling

* added utility functions for MCMC sampling

* fully implemented basic MCMC sampler

* working MCMC implementation with full set of tests

* fixed a few bugs in GP and MCMC implmentation

* fixed bug in variance prediction where roundoff error can cause negative variance

* added docstrings for MCMC routines

* added documentation for MCMC-related methods and code additions

* created benchmark for MCMC sampling and added documentation pages for it

* added information on MCMC benchmark to readme

* added additional pages to documentation for MCMC sampling

* removed renamed mcmc benchmark file

* fixed MCMC docstring in GP class

* Fix whitespace in Makefile

* Forward kwargs (gKDR._compute_loss); correct number of cross-validation folds

* Add benchmark for gKDR

* Wrap long lines in docstrings

* Versioning (#38)

* added code needed for versioning to devel branch

* forgot to modify setup.py file

* corrected line accidentally deleted from __init__.py

* added prerelease number to devel branch to track commits on devel

* corrected comments in conf.py to reflect full release numbering

* added simple demos for GP and MICE (#46)

* added simple demos for GP and MICE

* incremented prerelease number for merge

* History Matching (#39)

* initial commit of history matching class and benchmark with minor tweaks

* broke benchmark and sanity checks into two files for history matching

* reindented code to use 4 spaces

* added unit tests and some bug fixes for HistoryMatching

* added tests for implausability plus some other checks and bug fixes in HistoryMatching

* fixed misspelling of implausibility

* changed file name for benchmark in makefile

* fixed documentation in HistoryMatching class to be consistent with others

* improved documentation, cleaned up code, added a few unit tests for HistoryMatching

* fixed some docstring formatting and base rst file for HistoryMatching

* full implementation of history matching with unit tests and documentation

* simplified model discrepancy based on discussion with Danny

* fixes to history matching file and tests

* fixed some comparisons with None in SequentialDesign

* broke up long test for Hessian into parts

* incremented prerelease for history match merge

* Feature/mucmtoolkit (#54)

* added toolkit with converted pages and images

* incremented version number

* Feature/mucmtoolkit (#55)

* fixed bug in documentation to display methods

* version number change for corrected PR

* merge input derivative bugfix into devel (#61)

* Fix/cachefactmat (#62)

* corrected GP class to cache factorized matrix rather than inverse plus cleaned up a few unneeded internal variables

* incremented prerelease version number

* missed a line that should have been deleted

* added test to confirm that variance predictions are stable

* fixed solve routines to use cho_solve in scipy

* Fix/toolkitcorr (#63)

* toolkit proofreading and corrections

* continuing updates of toolkit pages

* edits to toolkit pages

* finished corrections up through meta section

* updated toolkit threads section

* updates to proc section of toolkit

* incremented prerelease version number

* modified version for release v0.2.0

* Adjust the paper references in DimensionReduction.py

* Update paper reference in documentation

.gitignore

@@ -1,4 +1,6 @@
 *.pyc
 *.DS_Store
-*.png
+docs/_build/*
+mogp_emulator/tests/*.png
+mogp_emulator/tests/*.pdf
 mogp_emulator/version.py

README.md

@@ -128,13 +128,21 @@ using 1, 2, 4, and 8 processess and notes the time required to perform the fitti
 will depend on the number of cores on the computer -- once you exceed the number of cores, the performance
 will degrade. As with the other benchmarks, Matplotlib can optionally be used to plot the results.
 
+##### MCMC Benchmark
+
+A benchmark applying the software to fitting an emulator with MCMC sampling is included. The code
+draws hyperparameter samples and compares the resulting posterior distributions with the values
+found via maximum likelihood estimation. If Matplotlib is installed, a histogram of the parameter
+samples is shown.
+
 ##### MICE Benchmark
 
 A benchmark comparing the MICE Sequential design method to Latin Hypercube sampling is also available.
 This creates designs of a variety of sizes and computes the error on unseen data for the 2D Branin
 function. It compares the accuracy of the sequential design to the Latin Hypercube for both the
 predictions and uncertainties.
 
+
 ### Documentation
 
 Building the documentation requires Sphinx/autodoc, which can be installed using `pip`. To build the documentatation, first install Sphinx and change to the `docs` directory. There is a Makefile in the

docs/DimensionReduction.rst

@@ -0,0 +1,33 @@
+.. _DimensionReduction:
+
+*********************************
+The ``DimensionReduction`` module
+*********************************
+
+.. automodule:: mogp_emulator.DimensionReduction
+
+
+---------------------------
+Dimension Reduction Classes
+---------------------------
+
+.. autoclass:: mogp_emulator.gKDR
+
+    .. automethod:: __init__
+    .. automethod:: __call__
+    .. automethod:: tune_parameters
+
+---------
+Utilities
+---------
+
+.. automethod:: mogp_emulator.DimensionReduction.gram_matrix
+
+.. automethod:: mogp_emulator.DimensionReduction.gram_matrix_sqexp
+
+.. automethod:: mogp_emulator.DimensionReduction.median_dist
+
+.. rubric:: References
+.. [LG17] Liu, Xiaoyu, and Serge Guillas. "Dimension reduction for Gaussian process emulation: An application to the influence of bathymetry on tsunami heights." SIAM/ASA Journal on Uncertainty Quantification 5.1 (2017): 787-812. https://epubs.siam.org/doi/10.1137/16M1090648
+.. [Fukumizu1] https://www.ism.ac.jp/~fukumizu/software.html
+.. [FL13] Fukumizu, Kenji and Chenlei Leng. "Gradient-based kernel dimension reduction for regression." Journal of the American Statistical Association 109, no. 505 (2014): 359-370

docs/HistoryMatching.rst

@@ -0,0 +1,13 @@
+.. _HistoryMatching:
+
+**********************************
+The ``HistoryMatching`` Class
+**********************************
+
+.. automodule:: mogp_emulator.HistoryMatching
+    :noindex:
+
+.. autoclass:: mogp_emulator.HistoryMatching
+    :members:
+
+    .. automethod:: __init__

docs/MCMC.rst

@@ -0,0 +1,8 @@
+.. _MCMC:
+
+**********************************
+The ``MCMC`` Module
+**********************************
+
+.. automodule:: mogp_emulator.MCMC
+    :members:

docs/benchmarks/benchmarks.rst

@@ -8,3 +8,4 @@ GP Emulator Benchmarks
    rosenbrock
    branin
    tsunami
+   mcmc_benchmark

docs/benchmarks/mcmc_benchmark.rst

@@ -0,0 +1,7 @@
+.. _mcmc_benchmark:
+
+**********************************
+MCMC Benchmark
+**********************************
+
+.. automodule:: mogp_emulator.tests.benchmark_MCMC

docs/conf.py

@@ -26,7 +26,7 @@
 # get version from package
 import mogp_emulator
 import re
-# The full version X.Y.Z
+# The full version X.Y.Z with development version if needed
 release = mogp_emulator.__version__
 # The short verion X.Y
 version = re.sub(r"(\d+\.\d+)", r"\1", mogp_emulator.__version__)

docs/index.rst

@@ -11,12 +11,21 @@ Welcome to Multi-Output GP Emulator's documentation!
    :caption: Contents:
 
    GaussianProcess
+   DimensionReduction
    MultiOutputGP
    Kernel
    ExperimentalDesign
    SequentialDesign
+   HistoryMatching
+   MCMC
    benchmarks/benchmarks
 
+.. toctree::
+   :maxdepth: 1
+   :caption: Uncertainty Quantification Methods
+
+   methods/methods
+
 
 
 Indices and tables

docs/methods/alt/AltBLPriors.rst

@@ -0,0 +1,142 @@
+.. _AltBLPriors:
+
+Alternatives: Prior specification for BL hyperparameters
+========================================================
+
+Overview
+--------
+
+In the fully :ref:`Bayes linear<DefBayesLinear>` approach to
+emulating a complex :ref:`simulator<DefSimulator>`, the
+:ref:`emulator<DefEmulator>` is formulated to represent prior
+knowledge of the simulator in terms of a :ref:`second-order belief
+specification<DefSecondOrderSpec>`. The BL prior specification
+requires the specification of beliefs about some
+:ref:`hyperparameters<DefHyperparameter>`, as discussed in the
+alternatives page on emulator prior mean function
+(:ref:`AltMeanFunction<AltMeanFunction>`), the discussion page on the
+GP covariance function
+(:ref:`DiscCovarianceFunction<DiscCovarianceFunction>`) and the
+alternatives page on emulator prior correlation function
+(:ref:`AltCorrelationFunction<AltCorrelationFunction>`).
+Specifically, in the :ref:`core problem<DiscCore>` that is the
+subject of the core threads (:ref:`ThreadCoreBL<ThreadCoreBL>`,
+:ref:`ThreadCoreGP<ThreadCoreGP>`) a vector :math:`\beta` defines the
+detailed form of the mean function, a scalar :math:`\sigma^2` quantifies
+the uncertainty or variability of the simulator around the prior mean
+function, while :math:`\delta` is a vector of hyperparameters defining
+details of the correlation function. Threads that deal with variations
+on the basic core problem may introduce further hyperparameters.
+
+A Bayes linear analysis requires hyperparameters to be given prior
+expectations, variances and covariances. We consider here ways to
+specify these prior beliefs for the hyperparameters of the core problem.
+Prior specifications for other hyperparameters are addressed in the
+relevant variant thread. Hyperparameters may be handled differently in
+the fully :ref:`Bayesian<DefBayesian>` approach - see
+:ref:`ThreadCoreGP<ThreadCoreGP>`.
+
+Choosing the Alternatives
+-------------------------
+
+The prior beliefs should be chosen to represent whatever prior knowledge
+the analyst has about the hyperparameters. However, the prior
+distributions will be updated with the information from a set of
+training runs, and if there is substantial information in the training
+data about one or more of the hyperparameters then the prior information
+about those hyperparameters may be irrelevant.
+
+In general, a Bayes linear specification requires statements of
+second-order beliefs for all uncertain quantities. In the current
+version of this Toolkit, the Bayes linear emulation approach does not
+consider the situation where :math:`\sigma^2` and :math:`\delta` are
+uncertain, and so we require the following:
+
+-  :math:`\text{E}[\beta_i]`, :math:`\text{Var}[\beta_i]`,
+   :math:`\text{Cov}[\beta_i,\beta_j]` - expectations, variances and
+   covariances for each coefficient :math:`\beta_i`, and covariances
+   between every pair of coefficients :math:`(\beta_i,\beta_j), i\neq j`
+-  :math:`\sigma^2=\text{Var}[w(x)]` - the variance of the residual
+   stochastic process
+-  :math:`\delta` - a value for the hyperparameters of the correlation
+   function
+
+The Nature of the Alternatives
+------------------------------
+
+Priors for :math:`\beta`
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Given a specified form for the basis functions :math:`h(x)` of :math:`m(x)` as
+described in the alternatives page on basis functions for the emulator
+mean (:ref:`AltBasisFunctions<AltBasisFunctions>`), we must specify
+expectation and variance for each coefficient :math:`\beta_i` and a
+covariance between every pair :math:`(\beta_i,\beta_j)`.
+
+As with the basis functions :math:`h(x)`, there are two primary means of
+obtaining a belief specification for :math:`\beta`.
+
+#. **Expert-led specification** - the specification can be made directly
+   by an expert using methods such as
+
+   a. Intuitive understanding of the magnitude and impact of the
+      physical effects represented by :math:`h(x)` leading to a direct
+      quantification of expectations, variances and covariances.
+   b. Assessing the difference between the model under study and another
+      well-understood model such as a fast approximate version or an
+      earlier version of the same simulator. In this approach, we can
+      combine the known information about the mean behaviour of the
+      second simulator with the belief statements about the differences
+      between the two simulator to construct an appropriate belief
+      specification for the hyperparameters -- see :ref:`multilevel
+      emulation<DefMultilevelEmulation>`.
+
+#. **Data-driven specification** - when prior beliefs are weak and we
+   have ample model evaluations, then prior values for :math:`\beta` are
+   typically not required and we can replace adjusted values for
+   :math:`\beta` with empirical estimates, :math:`\hat{\beta}`, obtained by
+   fitting the linear regression :math:`f(x)=h(x)^T\beta`. Our uncertainty
+   statements about :math:`\beta` can then be deduced from the "estimation
+   error" associated with :math:`\hat{\beta}`.
+
+Priors for :math:`\sigma^2`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The current version of the Toolkit requires a point value for the
+variance about the emulator mean, :math:`\sigma^2`. This corresponds
+directly to making a specification about :math:`\text{Var}[w(x)]`. As with
+the model coefficients above, there are two possible approaches to
+making such a quantification. An expert could make the specification by
+directly quantifying the magnitude of :math:`\sigma^2`. Alternatively, an
+expert assessment of the expected prior adequacy of the mean function at
+representing the variation in the simulator outputs can be combined with
+information on the variation of the simulator output, which allows for
+the deduction of a value of :math:`\sigma^2`. In the case of a data-driven
+assessment, the estimate for the residual variance :math:`\hat{\sigma}^2`
+can be used.
+
+In subsequent versions of the toolkit, Bayes linear methods will be
+developed for :ref:`learning<DefBLVarianceLearning>` about
+:math:`\sigma^2` in the emulation process. This will require making prior
+specifications about the squared emulator residuals.
+
+Priors for :math:`\delta`
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Specification of correlation function hyperparameters is a more
+challenging task. Direct elicitation can be difficult as the
+hyperparameter :math:`\delta` is hard to conceptualise - the alternatives
+page on prior distributions for GP hyperparameters
+(:ref:`AltGPPriors<AltGPPriors>`) provides some discussion on this
+topic, with particular application to the Gaussian correlation function.
+Alternatively, when given a large collection of simulator runs then
+:math:`\delta` can be crudely estimated using methods such as
+:ref:`variogram<ProcVariogram>` fitting on the empirical residuals.
+
+Assessing and updating uncertainties about :math:`\delta` raises both
+conceptual and technical problems as methods which would be optimal for
+assessing such parameters given realisations drawn from a corresponding
+stochastic process may prove to be highly non-robust when applied to
+functional computer output which is only represented very approximately
+by such a process. Methods for approaching this problem will appear in a
+subsequent version of the toolkit.

docs/methods/alt/AltBasisFunctions.rst

@@ -0,0 +1,115 @@
+.. _AltBasisFunctions:
+
+Alternatives: Basis functions for the emulator mean
+===================================================
+
+Overview
+--------
+
+The process of building an :ref:`emulator<DefEmulator>` of a
+:ref:`simulator<DefSimulator>` involves first specifying prior
+beliefs about the simulator and then updating this using a :ref:`training
+sample<DefTrainingSample>` of simulator runs. Prior
+specification may be either using the fully
+:ref:`Bayesian<DefBayesian>` approach in the form of a :ref:`Gaussian
+process<DefGP>` or using the :ref:`Bayes
+linear<DefBayesLinear>` approach in the form of first and second
+order moments. The basics of building an emulator using these two
+approaches are set out in the two core threads
+(:ref:`ThreadCoreGP<ThreadCoreGP>`,
+:ref:`ThreadCoreBL<ThreadCoreBL>`).
+
+In either approach it is necessary to specify a mean function and
+covariance function. Choice of the form of the emulator prior mean
+function is addressed in :ref:`AltMeanFunction<AltMeanFunction>`. We
+consider here the additional problem of specifying the forms of the
+:ref:`basis functions<DefBasisFunctions>` :math:`h(x)` when the form of
+the mean function is linear.
+
+The Nature of the Alternatives
+------------------------------
+
+In order to specify an appropriate form of the global trend component,
+we typically first identify a set of potential basis functions
+appropriate to the simulator in question, and then select which elements
+of that set would best describe the simulator's mean behaviour via the
+methods described above. There are a huge number of possible choices for
+the specific forms of :math:`h(x)`. Common choices include:
+
+-  Monomials - to capture simple large-scale effects
+-  Orthogonal polynomials - to exploit computational simplifications in
+   subsequent calculations
+-  Fourier/trigonometric functions - to adequately represent periodic
+   output
+-  A very fast approximate version of the simulator - to capture the
+   gross simulator behaviour using an existing model
+
+Choosing the Alternatives
+-------------------------
+
+When the mean function takes a linear form, an appropriate choice of the
+basis functions :math:`h(x)` is required. This is particularly true for a
+Bayes linear emulator as the emphasis is placed on a detailed structural
+representation of the simulator's mean behaviour.
+
+Typically, there are two primary methods for determining an appropriate
+collection of trend basis functions:
+
+#. prior information about the model can be used directly to specify an
+   appropriate form for the mean function;
+#. if the number of available model evaluations is very large, then we
+   can empirically determine the form of the mean function from this
+   data alone.
+
+Expert information about :math:`h(x)` can be derived from a variety of
+sources including, but not limited to, the following:
+
+-  knowledge and experience with the computer simulator and its outputs;
+-  beliefs about the behaviour of the actual physical system that the
+   computer model simulates;
+-  experience with similar computer models such as previous versions of
+   the same simulator or alternative models for the same system;
+-  series expansions of the generating equations underlying the computer
+   model (or an appropriately simplified model form);
+-  fast approximate versions of the computer model derived from
+   simplifications to the current simulator.
+
+If the prior information is weak relative to the available number of
+model evaluations and the computer model is inexpensive to evaluate,
+then we may choose instead to determine the form of the trend directly
+from the model evaluations. This empirical approach is reasonable since
+any Bayesian posterior would be dominated by the large volume of data.
+Thus in such situations it is reasonable to apply standard statistical
+modelling techniques. Empirical construction of the emulator mean is a
+similar problem to traditional regression model selection. In this case,
+methods such as stepwise model selection could be applied given a set of
+potential trend basis functions. However, using empirical methods to
+identify the form of the emulator mean function requires many more model
+evaluations than are required to fit an emulator with known form and
+hence is only applicable if a substantial number of simulator runs is
+available. Empirical construction of emulators using cheap simulators is
+a key component of :ref:`multilevel
+emulation<DefMultilevelEmulation>` which will be described in a
+subsequent version of the Toolkit.
+
+Often, the majority of the global variation of the output from a
+computer simulator :math:`f(x)` can be attributed to a relatively small
+subset, :math:`x_A`, of the input quantities called the :ref:`active
+inputs<DefActiveInput>`. In such cases, the emulator mean is
+considered to be a function of only the active inputs combined with a
+modified form of the covariance. Using this active input approach can
+make substantial computational savings; see the discussion page on
+active and inactive inputs
+(:ref:`DiscActiveInputs<DiscActiveInputs>`) for further details. If
+the simulator has a high-dimensional input space then the elicitation of
+information about potential active inputs possibly after a suitable
+transformation of the input space, then and the form of at least some of
+the model effects can be very helpful in emulator construction (see
+Craig *et al.* 1998).
+
+References
+----------
+
+Craig, P. S., Goldstein, M., Seheult, A. H., and Smith, J. A. (1998)
+"Constructing partial prior specifications for models of complex
+physical systems," *Applied Statistics*, **47**:1, 37--53