Noise estimation is completely broken

[dk-teknologisk-mon]

Describe the bug
When any given system gets covered by "enough" data, our model should be able to accurately capture the underlying noise in the system (measurement noise) and be able to separate this from the overall landscape variation. That is, if I sample many times from a normal distribution like N(loc=42, scale=0.45), then the value of opt.models[-1].noise_ should approach 0.45. Likewise, the standard deviation returned by expected_minimum should also converge on 0.45. Neither of these cases occur.

In the figure below, I show plot_objective_1d (and the data) from a model that has been fed 200 data points in a 1D system on (-1, 1), where the samples are drawn from the normal distribution above:

The 95 % credibility interval is completely wrong (too narrow by an order of magnitude), while the estimated standard deviation returned by expected_minimum(res, return_std=True) is 0.024, too small by a factor of 20. If I query opt.models[-1].noise_ on the other hand, its value is too big at 1.004?!

I suspect there is something completely wrong in the way our noise modelling is handled inside our Gaussian Process model, and/or there is some some hyperparameter bounds that are causing this. So to summarize, just compare these:

True noise: 0.45
Std(y): 0.39587640442367505
Result std from expected_minimum: 0.024106513812696212
Model noise: 1.003861781866922

To Reproduce
Code to reproduce the specific example of this failure:

import numpy as np

from ProcessOptimizer import Optimizer
from ProcessOptimizer.plots import plot_objective_1d
from ProcessOptimizer.utils import expected_minimum
from ProcessOptimizer.model_systems.model_system import ModelSystem

#%% Create a flat system for our test
def flat_score(x):
    return 42

flat_space = [(-1.0, 1.0)]
flat_noise = {
    "model_type": "constant",
    "noise_size": 0.45,
    }

# Build ModelSystem object
model = ModelSystem(
    score=flat_score, 
    space=flat_space, 
    noise_model=flat_noise,
)

#%% Run the model
seed = 42
# Build optimizer
opt = Optimizer(
    dimensions=model.space,
    lhs=False,
    acq_func="EI",
    n_initial_points=1,
    random_state=seed,
)

# Select 20 locations dispersed along our x-axis
next_x = np.linspace(-1, 1, 20).tolist()

x = []
y = []
# Build 200 points, 10 at each x-location
for i in range(10):
    for xx in next_x:
        x.append([xx])
        y.append(model.get_score([xx]))
# Feed results to the model
res = opt.tell(x, y)

result_x, [result_y, result_std] = expected_minimum(res, return_std=True)

print("True noise: {}".format(model.noise_model.noise_size))
print("Std(y): {}".format(np.std(y)))
print("Result std from expected_minimum: {}".format(result_std))
print("Model noise: {}".format(res.models[-1].noise_))

#%% Plot the data and plot_objective
ax = plot_objective_1d(res, size=5)
ax.scatter(opt.Xi, opt.yi, alpha=0.25)
ax.set_ylim(39, 45)
ax.set_xlim(-1.05, 1.05)

[dk-teknologisk-mon]

Adding observations here as I come across them: Changing normalize_y to False has a large effect on the model noise. You can see this by adding opt.base_estimator_.normalize_y = False to the code example above just after constructing the Optimizer. This changes the std from expected_minimum to 0.077 (becomes bigger), and opt.models[-1].noise_ to 0.157 (becomes smaller?!).

[sqbl]

Something of relevance here: https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy.html ?

[sqbl]

This does actually not seem like fully unexpected behaviour, when running Gaussian Processes with a Matern Kernel...
A non-zero std on a data-covered area would mean that the covariation matrix itself encode the noise. (And that is somewhat possible - by using the alpha parameter ("the diagonal of the matrix")).
Normally, I would expect our Whitekernel to take care of the noise modelling. But in po.learning.gaussian_process.gpr.py line 209-230, we set the Whitenoise level to 0 before fitting. EDIT: it is set to 0 before before predicting.
This leads to the unexpected behavior observed by @dk-teknologisk-mon.
If I commend out ll 209-230 as a quick-fix, then the plot above turns into:

But some of our tests fail. So this is work in progress :-)