GP model training fails while RF model training completes #26

EricKeenan · 2022-02-17T22:18:57Z

EricKeenan
Feb 17, 2022

I am able to successfully train a random forest model, but when I replicate the process with Gaussian processes, I get a cryptic error. Any insight into what might be happening? And how I can resolve?

failed command:

gp_model.train()

traceback:

2022-02-17 15:10:04.678579: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2022-02-17 15:10:16.155132: W tensorflow/core/kernels/linalg/cholesky_op.cc:56] Cholesky decomposition was not successful. Eigen::LLT failed with error code 1. Filling lower-triangular output with NaNs.

InvalidArgumentError                      Traceback (most recent call last)
<timed exec> in <module>

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/esem/emulator.py in train(self, verbose, **kwargs)
     71             Print verbose training output to screen
     72         """
---> 73         self.model.train(self.training_params, self.training_data.data, verbose=verbose, **kwargs)
     74 
     75     def predict(self, x, *args, **kwargs):

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/esem/model_adaptor.py in train(self, training_params, training_data, verbose, maxiter, **kwargs)
     87         opt.minimize(self.model.training_loss,
     88                      variables=self.model.trainable_variables,
---> 89                      options=dict(disp=verbose, maxiter=maxiter), **kwargs)
     90 
     91     def predict(self, *args, **kwargs):

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py in minimize(self, closure, variables, method, step_callback, compile, allow_unused_variables, **scipy_kwargs)
     94 
     95         return scipy.optimize.minimize(
---> 96             func, initial_params, jac=True, method=method, **scipy_kwargs
     97         )
     98 

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
    622     elif meth == 'l-bfgs-b':
    623         return _minimize_lbfgsb(fun, x0, args, jac, bounds,
--> 624                                 callback=callback, **options)
    625     elif meth == 'tnc':
    626         return _minimize_tnc(fun, x0, args, jac, bounds, callback=callback,

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, finite_diff_rel_step, **unknown_options)
    306     sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
    307                                   bounds=new_bounds,
--> 308                                   finite_diff_rel_step=finite_diff_rel_step)
    309 
    310     func_and_grad = sf.fun_and_grad

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/optimize.py in _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
    260     # calculation reduces overall function evaluations.
    261     sf = ScalarFunction(fun, x0, args, grad, hess,
--> 262                         finite_diff_rel_step, bounds, epsilon=epsilon)
    263 
    264     return sf

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py in __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step, finite_diff_bounds, epsilon)
    138 
    139         self._update_fun_impl = update_fun
--> 140         self._update_fun()
    141 
    142         # Gradient evaluation

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py in _update_fun(self)
    231     def _update_fun(self):
    232         if not self.f_updated:
--> 233             self._update_fun_impl()
    234             self.f_updated = True
    235 

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py in update_fun()
    135 
    136         def update_fun():
--> 137             self.f = fun_wrapped(self.x)
    138 
    139         self._update_fun_impl = update_fun

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py in fun_wrapped(x)
    132             # Overwriting results in undefined behaviour because
    133             # fun(self.x) will change self.x, with the two no longer linked.
--> 134             return fun(np.copy(x), *args)
    135 
    136         def update_fun():

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/optimize.py in __call__(self, x, *args)
     72     def __call__(self, x, *args):
     73         """ returns the the function value """
---> 74         self._compute_if_needed(x, *args)
     75         return self._value
     76 

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/optimize.py in _compute_if_needed(self, x, *args)
     66         if not np.all(x == self.x) or self._value is None or self.jac is None:
     67             self.x = np.asarray(x).copy()
---> 68             fg = self.fun(x, *args)
     69             self.jac = fg[1]
     70             self._value = fg[0]

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py in _eval(x)
    135 
    136         def _eval(x: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
--> 137             loss, grad = _tf_eval(tf.convert_to_tensor(x))
    138             return loss.numpy().astype(np.float64), grad.numpy().astype(np.float64)
    139 

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/tensorflow/python/util/traceback_utils.py in error_handler(*args, **kwargs)
    151     except Exception as e:
    152       filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153       raise e.with_traceback(filtered_tb) from None
    154     finally:
    155       del filtered_tb

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     53     ctx.ensure_initialized()
     54     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 55                                         inputs, attrs, num_outputs)
     56   except core._NotOkStatusException as e:
     57     if name is not None:

InvalidArgumentError: Graph execution error:

Detected at node 'triangular_solve/MatrixTriangularSolve' defined at (most recent call last):
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/traitlets/config/application.py", line 846, in launch_instance
      app.start()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 677, in start
      self.io_loop.start()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 149, in start
      self.asyncio_loop.run_forever()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
      self._run_once()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
      handle._run()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/asyncio/events.py", line 88, in _run
      self._context.run(self._callback, *self._args)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 457, in dispatch_queue
      await self.process_one()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 446, in process_one
      await dispatch(*args)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 353, in dispatch_shell
      await result
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 648, in execute_request
      reply_content = await reply_content
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/ipkernel.py", line 353, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2915, in run_cell
      raw_cell, store_history, silent, shell_futures)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2960, in _run_cell
      return runner(coro)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
      coro.send(None)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3186, in run_cell_async
      interactivity=interactivity, compiler=compiler, result=result)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3377, in run_ast_nodes
      if (await self.run_code(code, result,  async_=asy)):
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_20067/4182110964.py", line 1, in <module>
      get_ipython().run_cell_magic('time', '', 'gp_model = gp_model(X_train, Y_train)\ngp_model.train()\n')
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2419, in run_cell_magic
      result = fn(*args, **kwargs)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/decorator.py", line 232, in fun
      return caller(func, *(extras + args), **kw)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/magic.py", line 187, in <lambda>
      call = lambda f, *a, **k: f(*a, **k)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/IPython/core/magics/execution.py", line 1340, in time
      out = eval(code_2, glob, local_ns)
    File "<timed exec>", line 2, in <module>
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/esem/emulator.py", line 73, in train
      self.model.train(self.training_params, self.training_data.data, verbose=verbose, **kwargs)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/esem/model_adaptor.py", line 89, in train
      options=dict(disp=verbose, maxiter=maxiter), **kwargs)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py", line 96, in minimize
      func, initial_params, jac=True, method=method, **scipy_kwargs
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_minimize.py", line 624, in minimize
      callback=callback, **options)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py", line 308, in _minimize_lbfgsb
      finite_diff_rel_step=finite_diff_rel_step)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 262, in _prepare_scalar_function
      finite_diff_rel_step, bounds, epsilon=epsilon)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 140, in __init__
      self._update_fun()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 233, in _update_fun
      self._update_fun_impl()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in update_fun
      self.f = fun_wrapped(self.x)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 134, in fun_wrapped
      return fun(np.copy(x), *args)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 74, in __call__
      self._compute_if_needed(x, *args)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 68, in _compute_if_needed
      fg = self.fun(x, *args)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py", line 137, in _eval
      loss, grad = _tf_eval(tf.convert_to_tensor(x))
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py", line 119, in _tf_eval
      if first_call:
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py", line 121, in _tf_eval
      loss, grads = _compute_loss_and_gradients(
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/optimizers/scipy.py", line 224, in _compute_loss_and_gradients
      loss = loss_closure()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/models/training_mixins.py", line 63, in training_loss
      return self._training_loss()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/models/model.py", line 56, in _training_loss
      return -(self.maximum_log_likelihood_objective(*args, **kwargs) + self.log_prior_density())
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/models/gpr.py", line 67, in maximum_log_likelihood_objective
      return self.log_marginal_likelihood()
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/models/gpr.py", line 91, in log_marginal_likelihood
      log_prob = multivariate_normal(Y, m, L)
    File "/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/gpflow/logdensities.py", line 97, in multivariate_normal
      alpha = tf.linalg.triangular_solve(L, d, lower=True)
Node: 'triangular_solve/MatrixTriangularSolve'
Input matrix is not invertible.
	 [[{{node triangular_solve/MatrixTriangularSolve}}]] [Op:__inference__tf_eval_894]

Reproduce:

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from esem.utils import plot_results, prettify_plot, add_121_line, leave_one_out
from esem import rf_model, gp_model
from sklearn.metrics import mean_squared_error
from scipy import stats

import warnings
from sklearn.exceptions import DataConversionWarning
warnings.filterwarnings(action='ignore', category=DataConversionWarning)

train = np.loadtxt("train.txt")
Y = train[:, 0]
# X = train[:, 1:4] # Only use traditional predictors 
X = train[:, 1:] # Only use new predictors in addition to traditional ones 
print(Y.shape)
print(X.shape)

# Define training and test inds
n_train = 14000 # Note that I started with 1000
training_inds = np.arange(0, n_train)
testing_inds = np.arange(n_train, len(Y))
n_sample = len(testing_inds)

X_train = X[training_inds, :]
Y_train = Y[training_inds]

gp_model = gp_model(X_train, Y_train)
gp_model.train
[train.txt](https://github.com/duncanwp/ESEm/files/8092598/train.txt)
()

Answered by EricKeenan

Feb 23, 2022

I take it I need to call GPflow directly? White isn't yet implemented as a kernel in ESEm?

gp_model = gp_model(X_train, Y_train, kernel='White')

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<timed exec> in <module>

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/esem/__init__.py in gp_model(training_params, training_data, data_processors, kernel, kernel_op, active_dims, noise_variance, name, gpu)
     80         kernel = _get_gpflow_kernel(kernel, n_params, active_dims=active_dims, operator=kernel_op)
     81     elif not isinstance(kernel, gpflow.kernels.K…

View full answer

EricKeenan · 2022-02-17T22:19:57Z

EricKeenan
Feb 17, 2022
Author

train.txt for reference

train.txt

0 replies

duncanwp · 2022-02-18T10:48:24Z

duncanwp
Feb 18, 2022
Maintainer

Hi Eric,

I've come across this pretty cryptic error when training GPs sometimes too. I'm pretty sure it's GPFlow's way of telling you that it couldn't find a satisfactory solution to the regression using the kernels provided, although newer versions of GPFlow might be more robust against it (see e.g. GPflow/GPflow#692).

You're training with quite a lot of data so it might be worth using a sparse model, or introducing some 'jitter' by including a White kernel.

Hope that helps!

0 replies

EricKeenan · 2022-02-23T00:08:48Z

EricKeenan
Feb 23, 2022
Author

I take it I need to call GPflow directly? White isn't yet implemented as a kernel in ESEm?

gp_model = gp_model(X_train, Y_train, kernel='White')

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<timed exec> in <module>

/projects/erke2265/software/anaconda/envs/esem/lib/python3.7/site-packages/esem/__init__.py in gp_model(training_params, training_data, data_processors, kernel, kernel_op, active_dims, noise_variance, name, gpu)
     80         kernel = _get_gpflow_kernel(kernel, n_params, active_dims=active_dims, operator=kernel_op)
     81     elif not isinstance(kernel, gpflow.kernels.Kernel):
---> 82         raise ValueError("Invalid kernel specified: {}".format(kernel))
     83     # Else, use the user specified kernel
     84 

ValueError: Invalid kernel specified: White

2 replies

duncanwp Feb 23, 2022
Maintainer

Ah, the kernel argument should be a list that's all. So e.g.: kernel=['White'] or kernel=['White', 'RBF']

EricKeenan Mar 2, 2022
Author

Ahhhh! Easy fix. Thanks!

gp_model = gp_model(X_train, Y_train, kernel=['White'])

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GP model training fails while RF model training completes #26

{{title}}

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

GP model training fails while RF model training completes #26

EricKeenan Feb 17, 2022

Replies: 3 comments · 2 replies

EricKeenan Feb 17, 2022 Author

duncanwp Feb 18, 2022 Maintainer

EricKeenan Feb 23, 2022 Author

duncanwp Feb 23, 2022 Maintainer

EricKeenan Mar 2, 2022 Author

EricKeenan
Feb 17, 2022

Replies: 3 comments 2 replies

EricKeenan
Feb 17, 2022
Author

duncanwp
Feb 18, 2022
Maintainer

EricKeenan
Feb 23, 2022
Author

duncanwp Feb 23, 2022
Maintainer

EricKeenan Mar 2, 2022
Author