From 1e23d052f5739be25f9a8862c8dec9d091b99641 Mon Sep 17 00:00:00 2001 From: Leonard Papenmeier Date: Wed, 17 Jan 2024 01:54:45 -0800 Subject: [PATCH] Fix for bug that occurs when splitting single-element bins, use default BoTorch kernel for BAxUS. (#2165) Summary: This commit does two things: First, it fixes a bug that occurs when trying to split a bin with a single element. Also, we now use the default BoTorch Matern kernel instead of using MLE and lengthscale constraints. ## Motivation I received a bug report via email for a slightly different benchmark setup that affects the code in the BAxUS tutorial. The bug occurs in cases when, after splitting, a bin contains only a single element, but other bins contain more than one element. In that case, the previous code attempted to split that bin which later caused an error. This commit fixes this bug and, at the same time, removes the custom Matern kernel we used in the previous version. The kernel does not improve performance but adds overhead to the tutorial. ### Have you read the [Contributing Guidelines on pull requests](https://github.com/pytorch/botorch/blob/main/CONTRIBUTING.md#pull-requests)? Yes Pull Request resolved: https://github.com/pytorch/botorch/pull/2165 Test Plan: I tested this version on multiple benchmark setups to ensure this bug is fixed. ## Related PRs Initial PR for BAxUS tutorial: https://github.com/pytorch/botorch/pull/1559 Reviewed By: SebastianAment Differential Revision: D52718499 Pulled By: saitcakmak fbshipit-source-id: 7b2af5ec988406b3e482baa3ddf9f0becc17e45c --- tutorials/baxus.ipynb | 1997 ++++++++++++++++++++--------------------- 1 file changed, 997 insertions(+), 1000 deletions(-) diff --git a/tutorials/baxus.ipynb b/tutorials/baxus.ipynb index c118ec4c14..d9fb0a3d69 100644 --- a/tutorials/baxus.ipynb +++ b/tutorials/baxus.ipynb @@ -1,1013 +1,1010 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## BO with BAxUS and TS/EI\n", - "\n", - "In this tutorial, we show how to implement **B**ayesian optimization with **a**daptively e**x**panding s**u**bspace**s** (BAxUS) [1] in a closed loop in BoTorch.\n", - "The tutorial is purposefully similar to the [TuRBO tutorial](https://botorch.org/tutorials/turbo_1) to highlight the differences in the implementations.\n", - "\n", - "This implementation supports either Expected Improvement (EI) or Thompson sampling (TS). We optimize the Branin2 function [2] with 498 dummy dimensions$ and show that BAxUS outperforms EI as well as Sobol.\n", - "\n", - "Since BoTorch assumes a maximization problem, we will attempt to maximize $-f(x)$ to achieve $\\max_{x\\in \\mathcal{X}} -f(x)=0$.\n", - "\n", - "- [1]: [Papenmeier, Leonard, et al. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. Advances in Neural Information Processing Systems. 2022](https://openreview.net/pdf?id=e4Wf6112DI)\n", - "- [2]: [Branin Test Function](https://www.sfu.ca/~ssurjano/branin.html)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Running on cuda\n" - ] - } - ], - "source": [ - "import math\n", - "import os\n", - "from dataclasses import dataclass\n", - "\n", - "import botorch\n", - "import gpytorch\n", - "import matplotlib.pyplot as plt\n", - "import numpy as np\n", - "import torch\n", - "from gpytorch.constraints import Interval\n", - "from gpytorch.kernels import MaternKernel, ScaleKernel\n", - "from gpytorch.likelihoods import GaussianLikelihood\n", - "from gpytorch.mlls import ExactMarginalLogLikelihood\n", - "from torch.quasirandom import SobolEngine\n", - "\n", - "from botorch.acquisition.analytic import ExpectedImprovement\n", - "from botorch.exceptions import ModelFittingError\n", - "from botorch.fit import fit_gpytorch_mll\n", - "from botorch.generation import MaxPosteriorSampling\n", - "from botorch.models import SingleTaskGP\n", - "from botorch.optim import optimize_acqf\n", - "from botorch.test_functions import Branin\n", - "\n", - "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "print(f\"Running on {device}\")\n", - "dtype = torch.double\n", - "SMOKE_TEST = os.environ.get(\"SMOKE_TEST\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Optimize the augmented Branin function\n", - "\n", - "The goal is to minimize the embedded Branin function\n", - "\n", - "$f(x_1, x_2, \\ldots, x_{20}) = \\left (x_2-\\frac{5.1}{4\\pi^2}x_1^2+\\frac{5}{\\pi}x_1-6\\right )^2+10\\cdot \\left (1-\\frac{1}{8\\pi}\\right )\\cos(x_1)+10$\n", - "\n", - "with bounds [-5, 10] for $x_1$ and [0, 15] for $x_2$ (all other dimensions are ignored). The function has three minima with an optimal value of $0.397887$.\n", - "\n", - "As mentioned above, since botorch assumes a maximization problem, we instead maximize $-f(x)$." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Define a function with dummy variables\n", - "\n", - "We first define a new function where we only pass the first two input dimensions to the actual Branin function." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "branin = Branin(negate=True).to(device=device, dtype=dtype)\n", - "\n", - "\n", - "def branin_emb(x):\n", - " \"\"\"x is assumed to be in [-1, 1]^D\"\"\"\n", - " lb, ub = branin.bounds\n", - " return branin(lb + (ub - lb) * (x[..., :2] + 1) / 2)" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "fun = branin_emb\n", - "dim = 500\n", - "\n", - "n_init = 10\n", - "max_cholesky_size = float(\"inf\") # Always use Cholesky" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Maintain the BAxUS state\n", - "BAxUS needs to maintain a state, which includes the length of the trust region, success and failure counters, success and failure tolerance, etc. \n", - "In contrast to TuRBO, the failure tolerance depends on the target dimensionality.\n", - "\n", - "In this tutorial we store the state in a dataclass and update the state of TuRBO after each batch evaluation. \n", - "\n", - "**Note**: These settings assume that the domain has been scaled to $[-1, 1]^d$" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "@dataclass\n", - "class BaxusState:\n", - " dim: int\n", - " eval_budget: int\n", - " new_bins_on_split: int = 3\n", - " d_init: int = float(\"nan\") # Note: post-initialized\n", - " target_dim: int = float(\"nan\") # Note: post-initialized\n", - " n_splits: int = float(\"nan\") # Note: post-initialized\n", - " length: float = 0.8\n", - " length_init: float = 0.8\n", - " length_min: float = 0.5**7\n", - " length_max: float = 1.6\n", - " failure_counter: int = 0\n", - " success_counter: int = 0\n", - " success_tolerance: int = 3\n", - " best_value: float = -float(\"inf\")\n", - " restart_triggered: bool = False\n", - "\n", - " def __post_init__(self):\n", - " n_splits = round(math.log(self.dim, self.new_bins_on_split + 1))\n", - " self.d_init = 1 + np.argmin(\n", - " np.abs(\n", - " (1 + np.arange(self.new_bins_on_split))\n", - " * (1 + self.new_bins_on_split) ** n_splits\n", - " - self.dim\n", - " )\n", - " )\n", - " self.target_dim = self.d_init\n", - " self.n_splits = n_splits\n", - "\n", - " @property\n", - " def split_budget(self) -> int:\n", - " return round(\n", - " -1\n", - " * (self.new_bins_on_split * self.eval_budget * self.target_dim)\n", - " / (self.d_init * (1 - (self.new_bins_on_split + 1) ** (self.n_splits + 1)))\n", - " )\n", - "\n", - " @property\n", - " def failure_tolerance(self) -> int:\n", - " if self.target_dim == self.dim:\n", - " return self.target_dim\n", - " k = math.floor(math.log(self.length_min / self.length_init, 0.5))\n", - " split_budget = self.split_budget\n", - " return min(self.target_dim, max(1, math.floor(split_budget / k)))\n", - "\n", - "\n", - "def update_state(state, Y_next):\n", - " if max(Y_next) > state.best_value + 1e-3 * math.fabs(state.best_value):\n", - " state.success_counter += 1\n", - " state.failure_counter = 0\n", - " else:\n", - " state.success_counter = 0\n", - " state.failure_counter += 1\n", - "\n", - " if state.success_counter == state.success_tolerance: # Expand trust region\n", - " state.length = min(2.0 * state.length, state.length_max)\n", - " state.success_counter = 0\n", - " elif state.failure_counter == state.failure_tolerance: # Shrink trust region\n", - " state.length /= 2.0\n", - " state.failure_counter = 0\n", - "\n", - " state.best_value = max(state.best_value, max(Y_next).item())\n", - " if state.length < state.length_min:\n", - " state.restart_triggered = True\n", - " return state" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Create a BAxUS embedding\n", - "\n", - "We now show how to create the BAxUS embedding. The essential idea is to assign input dimensions to target dimensions and to assign a sign $\\in \\pm 1$ to each input dimension, similar to the HeSBO embedding. \n", - "We create the embedding matrix that is used to project points from the target to the input space. The matrix is sparse, each column has precisely one non-zero entry that is either 1 or -1." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 1., 0., -1., 0., 0., 1., 0., 0., -1., 0.],\n", - " [ 0., -1., 0., 0., 0., 0., -1., 0., 0., -1.],\n", - " [ 0., 0., 0., -1., 1., 0., 0., 1., 0., 0.]], device='cuda:0',\n", - " dtype=torch.float64)" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "def embedding_matrix(input_dim: int, target_dim: int) -> torch.Tensor:\n", - " if (\n", - " target_dim >= input_dim\n", - " ): # return identity matrix if target size greater than input size\n", - " return torch.eye(input_dim, device=device, dtype=dtype)\n", - "\n", - " input_dims_perm = (\n", - " torch.randperm(input_dim, device=device) + 1\n", - " ) # add 1 to indices for padding column in matrix\n", - "\n", - " bins = torch.tensor_split(\n", - " input_dims_perm, target_dim\n", - " ) # split dims into almost equally-sized bins\n", - " bins = torch.nn.utils.rnn.pad_sequence(\n", - " bins, batch_first=True\n", - " ) # zero pad bins, the index 0 will be cut off later\n", - "\n", - " mtrx = torch.zeros(\n", - " (target_dim, input_dim + 1), dtype=dtype, device=device\n", - " ) # add one extra column for padding\n", - " mtrx = mtrx.scatter_(\n", - " 1,\n", - " bins,\n", - " 2 * torch.randint(2, (target_dim, input_dim), dtype=dtype, device=device) - 1,\n", - " ) # fill mask with random +/- 1 at indices\n", - "\n", - " return mtrx[:, 1:] # cut off index zero as this corresponds to zero padding\n", - "\n", - "\n", - "embedding_matrix(10, 3) # example for an embedding matrix" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Function to increase the embedding\n", - "\n", - "Next, we write a helper function to increase the embedding and to bring observations to the increased target space." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "def increase_embedding_and_observations(\n", - " S: torch.Tensor, X: torch.Tensor, n_new_bins: int\n", - ") -> torch.Tensor:\n", - " assert X.size(1) == S.size(0), \"Observations don't lie in row space of S\"\n", - "\n", - " S_update = S.clone()\n", - " X_update = X.clone()\n", - "\n", - " for row_idx in range(len(S)):\n", - " row = S[row_idx]\n", - " idxs_non_zero = torch.nonzero(row)\n", - " idxs_non_zero = idxs_non_zero[torch.randperm(len(idxs_non_zero))].squeeze()\n", - "\n", - " non_zero_elements = row[idxs_non_zero].squeeze()\n", - "\n", - " n_row_bins = min(\n", - " n_new_bins, len(idxs_non_zero)\n", - " ) # number of new bins is always less or equal than the contributing input dims in the row minus one\n", - "\n", - " new_bins = torch.tensor_split(idxs_non_zero, n_row_bins)[\n", - " 1:\n", - " ] # the dims in the first bin won't be moved\n", - " elements_to_move = torch.tensor_split(non_zero_elements, n_row_bins)[1:]\n", - "\n", - " new_bins_padded = torch.nn.utils.rnn.pad_sequence(\n", - " new_bins, batch_first=True\n", - " ) # pad the tuples of bins with zeros to apply _scatter\n", - " els_to_move_padded = torch.nn.utils.rnn.pad_sequence(\n", - " elements_to_move, batch_first=True\n", - " )\n", - "\n", - " S_stack = torch.zeros(\n", - " (n_row_bins - 1, len(row) + 1), device=device, dtype=dtype\n", - " ) # submatrix to stack on S_update\n", - "\n", - " S_stack = S_stack.scatter_(\n", - " 1, new_bins_padded + 1, els_to_move_padded\n", - " ) # fill with old values (add 1 to indices for padding column)\n", - "\n", - " S_update[\n", - " row_idx, torch.hstack(new_bins)\n", - " ] = 0 # set values that were move to zero in current row\n", - "\n", - " X_update = torch.hstack(\n", - " (X_update, X[:, row_idx].reshape(-1, 1).repeat(1, len(new_bins)))\n", - " ) # repeat observations for row at the end of X (column-wise)\n", - " S_update = torch.vstack(\n", - " (S_update, S_stack[:, 1:])\n", - " ) # stack onto S_update except for padding column\n", - "\n", - " return S_update, X_update" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "S before increase\n", - "tensor([[-1., -1., 0., 0., 0., -1., -1., 0., -1., 0.],\n", - " [ 0., 0., -1., -1., 1., 0., 0., 1., 0., -1.]], device='cuda:0',\n", - " dtype=torch.float64)\n", - "X before increase\n", - "tensor([[79, 84],\n", - " [85, 65],\n", - " [46, 11],\n", - " [95, 34],\n", - " [14, 36],\n", - " [10, 55],\n", - " [48, 47]])\n", - "S after increase\n", - "tensor([[-1., -1., 0., 0., 0., 0., 0., 0., 0., 0.],\n", - " [ 0., 0., 0., 0., 1., 0., 0., 0., 0., -1.],\n", - " [ 0., 0., 0., 0., 0., -1., 0., 0., -1., 0.],\n", - " [ 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.],\n", - " [ 0., 0., -1., 0., 0., 0., 0., 1., 0., 0.],\n", - " [ 0., 0., 0., -1., 0., 0., 0., 0., 0., 0.]], device='cuda:0',\n", - " dtype=torch.float64)\n", - "X after increase\n", - "tensor([[79, 84, 79, 79, 84, 84],\n", - " [85, 65, 85, 85, 65, 65],\n", - " [46, 11, 46, 46, 11, 11],\n", - " [95, 34, 95, 95, 34, 34],\n", - " [14, 36, 14, 14, 36, 36],\n", - " [10, 55, 10, 10, 55, 55],\n", - " [48, 47, 48, 48, 47, 47]])\n" - ] - } - ], - "source": [ - "S = embedding_matrix(10, 2)\n", - "X = torch.randint(100, (7, 2))\n", - "print(f\"S before increase\\n{S}\")\n", - "print(f\"X before increase\\n{X}\")\n", - "\n", - "S, X = increase_embedding_and_observations(S, X, 3)\n", - "print(f\"S after increase\\n{S}\")\n", - "print(f\"X after increase\\n{X}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Take a look at the state" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "BaxusState(dim=500, eval_budget=500, new_bins_on_split=3, d_init=2, target_dim=2, n_splits=4, length=0.8, length_init=0.8, length_min=0.0078125, length_max=1.6, failure_counter=0, success_counter=0, success_tolerance=3, best_value=-inf, restart_triggered=False)\n" - ] - } - ], - "source": [ - "state = BaxusState(dim=dim, eval_budget=500)\n", - "print(state)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Generate initial points\n", - "This generates an initial set of Sobol points that we use to start of the BO loop." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "def get_initial_points(dim, n_pts, seed=0):\n", - " sobol = SobolEngine(dimension=dim, scramble=True, seed=seed)\n", - " X_init = (\n", - " 2 * sobol.draw(n=n_pts).to(dtype=dtype, device=device) - 1\n", - " ) # points have to be in [-1, 1]^d\n", - " return X_init" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Generate new batch\n", - "Given the current `state` and a probabilistic (GP) `model` built from observations `X` and `Y`, we generate a new batch of points. \n", - "\n", - "This method works on the domain $[-1, +1]^d$, so make sure to not pass in observations from the true domain. `unnormalize` is called before the true function is evaluated which will first map the points back to the original domain.\n", - "\n", - "We support either TS and qEI which can be specified via the `acqf` argument." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "def create_candidate(\n", - " state,\n", - " model, # GP model\n", - " X, # Evaluated points on the domain [-1, 1]^d\n", - " Y, # Function values\n", - " n_candidates=None, # Number of candidates for Thompson sampling\n", - " num_restarts=10,\n", - " raw_samples=512,\n", - " acqf=\"ts\", # \"ei\" or \"ts\"\n", - "):\n", - " assert acqf in (\"ts\", \"ei\")\n", - " assert X.min() >= -1.0 and X.max() <= 1.0 and torch.all(torch.isfinite(Y))\n", - " if n_candidates is None:\n", - " n_candidates = min(5000, max(2000, 200 * X.shape[-1]))\n", - "\n", - " # Scale the TR to be proportional to the lengthscales\n", - " x_center = X[Y.argmax(), :].clone()\n", - " weights = model.covar_module.base_kernel.lengthscale.detach().view(-1)\n", - " weights = weights / weights.mean()\n", - " weights = weights / torch.prod(weights.pow(1.0 / len(weights)))\n", - " tr_lb = torch.clamp(x_center - weights * state.length, -1.0, 1.0)\n", - " tr_ub = torch.clamp(x_center + weights * state.length, -1.0, 1.0)\n", - "\n", - " if acqf == \"ts\":\n", - " dim = X.shape[-1]\n", - " sobol = SobolEngine(dim, scramble=True)\n", - " pert = sobol.draw(n_candidates).to(dtype=dtype, device=device)\n", - " pert = tr_lb + (tr_ub - tr_lb) * pert\n", - "\n", - " # Create a perturbation mask\n", - " prob_perturb = min(20.0 / dim, 1.0)\n", - " mask = torch.rand(n_candidates, dim, dtype=dtype, device=device) <= prob_perturb\n", - " ind = torch.where(mask.sum(dim=1) == 0)[0]\n", - " mask[ind, torch.randint(0, dim, size=(len(ind),), device=device)] = 1\n", - "\n", - " # Create candidate points from the perturbations and the mask\n", - " X_cand = x_center.expand(n_candidates, dim).clone()\n", - " X_cand[mask] = pert[mask]\n", - "\n", - " # Sample on the candidate points\n", - " thompson_sampling = MaxPosteriorSampling(model=model, replacement=False)\n", - " with torch.no_grad(): # We don't need gradients when using TS\n", - " X_next = thompson_sampling(X_cand, num_samples=1)\n", - "\n", - " elif acqf == \"ei\":\n", - " ei = ExpectedImprovement(model, train_Y.max())\n", - " X_next, acq_value = optimize_acqf(\n", - " ei,\n", - " bounds=torch.stack([tr_lb, tr_ub]),\n", - " q=1,\n", - " num_restarts=num_restarts,\n", - " raw_samples=raw_samples,\n", - " )\n", - "\n", - " return X_next" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Optimization loop\n", - "This simple loop runs one instance of BAxUS with Thompson sampling until convergence.\n", - "\n", - "BAxUS works on a fixed evaluation budget and shrinks the trust region until the minimal trust region size is reached (`state[\"restart_triggered\"]` is set to `True`).\n", - "Then, BAxUS increases the target space and carries over the observations to the updated space. \n" - ] - }, + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## BO with BAxUS and TS/EI\n", + "\n", + "In this tutorial, we show how to implement **B**ayesian optimization with **a**daptively e**x**panding s**u**bspace**s** (BAxUS) [1] in a closed loop in BoTorch.\n", + "The tutorial is purposefully similar to the [TuRBO tutorial](https://botorch.org/tutorials/turbo_1) to highlight the differences in the implementations.\n", + "\n", + "This implementation supports either Expected Improvement (EI) or Thompson sampling (TS). We optimize the Branin2 function [2] with 498 dummy dimensions and show that BAxUS outperforms EI as well as Sobol.\n", + "\n", + "Since BoTorch assumes a maximization problem, we will attempt to maximize $-f(x)$ to achieve $\\max_{x\\in \\mathcal{X}} -f(x)=0$.\n", + "\n", + "- [1]: [Papenmeier, Leonard, et al. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. Advances in Neural Information Processing Systems. 2022](https://openreview.net/pdf?id=e4Wf6112DI)\n", + "- [2]: [Branin Test Function](https://www.sfu.ca/~ssurjano/branin.html)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "code", - "execution_count": 11, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "iteration 11, d=2) Best value: -2.74, TR length: 0.4\n", - "iteration 12, d=2) Best value: -2.64, TR length: 0.4\n", - "iteration 13, d=2) Best value: -2.55, TR length: 0.4\n", - "iteration 14, d=2) Best value: -2.55, TR length: 0.2\n", - "iteration 15, d=2) Best value: -2.55, TR length: 0.1\n", - "iteration 16, d=2) Best value: -2.55, TR length: 0.05\n", - "iteration 17, d=2) Best value: -2.55, TR length: 0.025\n", - "iteration 18, d=2) Best value: -2.55, TR length: 0.0125\n", - "iteration 19, d=2) Best value: -2.55, TR length: 0.00625\n", - "increasing target space\n", - "new dimensionality: 6\n", - "iteration 20, d=6) Best value: -2.54, TR length: 0.8\n", - "iteration 21, d=6) Best value: -2.54, TR length: 0.4\n", - "iteration 22, d=6) Best value: -1.79, TR length: 0.4\n", - "iteration 23, d=6) Best value: -0.628, TR length: 0.4\n", - "iteration 24, d=6) Best value: -0.456, TR length: 0.8\n", - "iteration 25, d=6) Best value: -0.456, TR length: 0.4\n", - "iteration 26, d=6) Best value: -0.42, TR length: 0.4\n", - "iteration 27, d=6) Best value: -0.42, TR length: 0.2\n", - "iteration 28, d=6) Best value: -0.42, TR length: 0.1\n", - "iteration 29, d=6) Best value: -0.42, TR length: 0.05\n", - "iteration 30, d=6) Best value: -0.42, TR length: 0.025\n", - "iteration 31, d=6) Best value: -0.41, TR length: 0.025\n", - "iteration 32, d=6) Best value: -0.403, TR length: 0.025\n", - "iteration 33, d=6) Best value: -0.4, TR length: 0.05\n", - "iteration 34, d=6) Best value: -0.399, TR length: 0.05\n", - "iteration 35, d=6) Best value: -0.399, TR length: 0.025\n", - "iteration 36, d=6) Best value: -0.398, TR length: 0.025\n", - "iteration 37, d=6) Best value: -0.398, TR length: 0.0125\n", - "iteration 38, d=6) Best value: -0.398, TR length: 0.00625\n", - "increasing target space\n", - "new dimensionality: 18\n", - "iteration 39, d=18) Best value: -0.398, TR length: 0.4\n", - "iteration 40, d=18) Best value: -0.398, TR length: 0.2\n", - "iteration 41, d=18) Best value: -0.398, TR length: 0.1\n", - "iteration 42, d=18) Best value: -0.398, TR length: 0.05\n", - "iteration 43, d=18) Best value: -0.398, TR length: 0.025\n", - "iteration 44, d=18) Best value: -0.398, TR length: 0.0125\n", - "iteration 45, d=18) Best value: -0.398, TR length: 0.00625\n", - "increasing target space\n", - "new dimensionality: 54\n", - "iteration 46, d=54) Best value: -0.398, TR length: 0.4\n", - "iteration 47, d=54) Best value: -0.398, TR length: 0.2\n", - "iteration 48, d=54) Best value: -0.398, TR length: 0.1\n", - "iteration 49, d=54) Best value: -0.398, TR length: 0.05\n", - "iteration 50, d=54) Best value: -0.398, TR length: 0.025\n", - "iteration 51, d=54) Best value: -0.398, TR length: 0.0125\n", - "iteration 52, d=54) Best value: -0.398, TR length: 0.00625\n", - "increasing target space\n", - "new dimensionality: 162\n", - "iteration 53, d=162) Best value: -0.398, TR length: 0.8\n", - "iteration 54, d=162) Best value: -0.398, TR length: 0.8\n", - "iteration 55, d=162) Best value: -0.398, TR length: 0.4\n", - "iteration 56, d=162) Best value: -0.398, TR length: 0.4\n", - "iteration 57, d=162) Best value: -0.398, TR length: 0.4\n", - "iteration 58, d=162) Best value: -0.398, TR length: 0.2\n", - "iteration 59, d=162) Best value: -0.398, TR length: 0.2\n", - "iteration 60, d=162) Best value: -0.398, TR length: 0.2\n", - "iteration 61, d=162) Best value: -0.398, TR length: 0.1\n", - "iteration 62, d=162) Best value: -0.398, TR length: 0.1\n", - "iteration 63, d=162) Best value: -0.398, TR length: 0.1\n", - "iteration 64, d=162) Best value: -0.398, TR length: 0.05\n", - "iteration 65, d=162) Best value: -0.398, TR length: 0.05\n", - "iteration 66, d=162) Best value: -0.398, TR length: 0.05\n", - "iteration 67, d=162) Best value: -0.398, TR length: 0.025\n", - "iteration 68, d=162) Best value: -0.398, TR length: 0.025\n", - "iteration 69, d=162) Best value: -0.398, TR length: 0.025\n", - "iteration 70, d=162) Best value: -0.398, TR length: 0.0125\n", - "iteration 71, d=162) Best value: -0.398, TR length: 0.0125\n", - "iteration 72, d=162) Best value: -0.398, TR length: 0.0125\n", - "iteration 73, d=162) Best value: -0.398, TR length: 0.00625\n", - "increasing target space\n", - "new dimensionality: 486\n", - "iteration 74, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 75, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 76, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 77, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 78, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 79, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 80, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 81, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 82, d=486) Best value: -0.398, TR length: 0.8\n", - "iteration 83, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 84, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 85, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 86, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 87, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 88, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 89, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 90, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 91, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 92, d=486) Best value: -0.398, TR length: 0.4\n", - "iteration 93, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 94, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 95, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 96, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 97, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 98, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 99, d=486) Best value: -0.398, TR length: 0.2\n", - "iteration 100, d=486) Best value: -0.398, TR length: 0.2\n" - ] - } - ], - "source": [ - "evaluation_budget = 100\n", - "\n", - "state = BaxusState(dim=dim, eval_budget=evaluation_budget - n_init)\n", - "S = embedding_matrix(input_dim=state.dim, target_dim=state.d_init)\n", - "\n", - "X_baxus_target = get_initial_points(state.d_init, n_init)\n", - "X_baxus_input = X_baxus_target @ S\n", - "Y_baxus = torch.tensor(\n", - " [branin_emb(x) for x in X_baxus_input], dtype=dtype, device=device\n", - ").unsqueeze(-1)\n", - "\n", - "\n", - "NUM_RESTARTS = 10 if not SMOKE_TEST else 2\n", - "RAW_SAMPLES = 512 if not SMOKE_TEST else 4\n", - "N_CANDIDATES = min(5000, max(2000, 200 * dim)) if not SMOKE_TEST else 4\n", - "\n", - "# Disable input scaling checks as we normalize to [-1, 1]\n", - "with botorch.settings.validate_input_scaling(False):\n", - "\n", - " for _ in range(evaluation_budget - n_init): # Run until evaluation budget depleted\n", - " # Fit a GP model\n", - " train_Y = (Y_baxus - Y_baxus.mean()) / Y_baxus.std()\n", - " likelihood = GaussianLikelihood(noise_constraint=Interval(1e-8, 1e-3))\n", - " covar_module = (\n", - " ScaleKernel( # Use the same lengthscale prior as in the TuRBO paper\n", - " MaternKernel(\n", - " nu=2.5,\n", - " ard_num_dims=state.target_dim,\n", - " lengthscale_constraint=Interval(0.005, 10),\n", - " ),\n", - " outputscale_constraint=Interval(0.05, 10),\n", - " )\n", - " )\n", - " model = SingleTaskGP(\n", - " X_baxus_target, train_Y, covar_module=covar_module, likelihood=likelihood\n", - " )\n", - " mll = ExactMarginalLogLikelihood(model.likelihood, model)\n", - "\n", - " # Do the fitting and acquisition function optimization inside the Cholesky context\n", - " with gpytorch.settings.max_cholesky_size(max_cholesky_size):\n", - " # Fit the model\n", - " try:\n", - " fit_gpytorch_mll(mll)\n", - " except ModelFittingError:\n", - " # Right after increasing the target dimensionality, the covariance matrix becomes indefinite\n", - " # In this case, the Cholesky decomposition might fail due to numerical instabilities\n", - " # In this case, we revert to Adam-based optimization\n", - " optimizer = torch.optim.Adam([{\"params\": model.parameters()}], lr=0.1)\n", - "\n", - " for _ in range(100):\n", - " optimizer.zero_grad()\n", - " output = model(X_baxus_target)\n", - " loss = -mll(output, train_Y.flatten())\n", - " loss.backward()\n", - " optimizer.step()\n", - "\n", - " # Create a batch\n", - " X_next_target = create_candidate(\n", - " state=state,\n", - " model=model,\n", - " X=X_baxus_target,\n", - " Y=train_Y,\n", - " n_candidates=N_CANDIDATES,\n", - " num_restarts=NUM_RESTARTS,\n", - " raw_samples=RAW_SAMPLES,\n", - " acqf=\"ts\",\n", - " )\n", - "\n", - " X_next_input = X_next_target @ S\n", - "\n", - " Y_next = torch.tensor(\n", - " [branin_emb(x) for x in X_next_input], dtype=dtype, device=device\n", - " ).unsqueeze(-1)\n", - "\n", - " # Update state\n", - " state = update_state(state=state, Y_next=Y_next)\n", - "\n", - " # Append data\n", - " X_baxus_input = torch.cat((X_baxus_input, X_next_input), dim=0)\n", - " X_baxus_target = torch.cat((X_baxus_target, X_next_target), dim=0)\n", - " Y_baxus = torch.cat((Y_baxus, Y_next), dim=0)\n", - "\n", - " # Print current status\n", - " print(\n", - " f\"iteration {len(X_baxus_input)}, d={len(X_baxus_target.T)}) Best value: {state.best_value:.3}, TR length: {state.length:.3}\"\n", - " )\n", - "\n", - " if state.restart_triggered:\n", - " state.restart_triggered = False\n", - " print(\"increasing target space\")\n", - " S, X_baxus_target = increase_embedding_and_observations(\n", - " S, X_baxus_target, state.new_bins_on_split\n", - " )\n", - " print(f\"new dimensionality: {len(S)}\")\n", - " state.target_dim = len(S)\n", - " state.length = state.length_init\n", - " state.failure_counter = 0\n", - " state.success_counter = 0" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "[KeOps] Warning : Cuda libraries were not detected on the system ; using cpu only mode\n", + "Running on cpu\n" + ] + } + ], + "source": [ + "import math\n", + "import os\n", + "from dataclasses import dataclass\n", + "\n", + "import botorch\n", + "import gpytorch\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import torch\n", + "from gpytorch.constraints import Interval\n", + "from gpytorch.kernels import MaternKernel, ScaleKernel\n", + "from gpytorch.likelihoods import GaussianLikelihood\n", + "from gpytorch.mlls import ExactMarginalLogLikelihood\n", + "from torch.quasirandom import SobolEngine\n", + "\n", + "from botorch.acquisition.analytic import LogExpectedImprovement\n", + "from botorch.exceptions import ModelFittingError\n", + "from botorch.fit import fit_gpytorch_mll\n", + "from botorch.generation import MaxPosteriorSampling\n", + "from botorch.models import SingleTaskGP\n", + "from botorch.optim import optimize_acqf\n", + "from botorch.test_functions import Branin\n", + "\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", + "print(f\"Running on {device}\")\n", + "dtype = torch.double\n", + "SMOKE_TEST = os.environ.get(\"SMOKE_TEST\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Optimize the augmented Branin function\n", + "\n", + "The goal is to minimize the embedded Branin function\n", + "\n", + "$f(x_1, x_2, \\ldots, x_{20}) = \\left (x_2-\\frac{5.1}{4\\pi^2}x_1^2+\\frac{5}{\\pi}x_1-6\\right )^2+10\\cdot \\left (1-\\frac{1}{8\\pi}\\right )\\cos(x_1)+10$\n", + "\n", + "with bounds [-5, 10] for $x_1$ and [0, 15] for $x_2$ (all other dimensions are ignored). The function has three minima with an optimal value of $0.397887$.\n", + "\n", + "As mentioned above, since botorch assumes a maximization problem, we instead maximize $-f(x)$." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Define a function with dummy variables\n", + "\n", + "We first define a new function where we only pass the first two input dimensions to the actual Branin function." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "branin = Branin(negate=True).to(device=device, dtype=dtype)\n", + "\n", + "\n", + "def branin_emb(x):\n", + " \"\"\"x is assumed to be in [-1, 1]^D\"\"\"\n", + " lb, ub = branin.bounds\n", + " return branin(lb + (ub - lb) * (x[..., :2] + 1) / 2)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "fun = branin_emb\n", + "dim = 500\n", + "\n", + "n_init = 10\n", + "max_cholesky_size = float(\"inf\") # Always use Cholesky" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Maintain the BAxUS state\n", + "BAxUS needs to maintain a state, which includes the length of the trust region, success and failure counters, success and failure tolerance, etc. \n", + "In contrast to TuRBO, the failure tolerance depends on the target dimensionality.\n", + "\n", + "In this tutorial we store the state in a dataclass and update the state of TuRBO after each batch evaluation. \n", + "\n", + "**Note**: These settings assume that the domain has been scaled to $[-1, 1]^d$" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "@dataclass\n", + "class BaxusState:\n", + " dim: int\n", + " eval_budget: int\n", + " new_bins_on_split: int = 3\n", + " d_init: int = float(\"nan\") # Note: post-initialized\n", + " target_dim: int = float(\"nan\") # Note: post-initialized\n", + " n_splits: int = float(\"nan\") # Note: post-initialized\n", + " length: float = 0.8\n", + " length_init: float = 0.8\n", + " length_min: float = 0.5**7\n", + " length_max: float = 1.6\n", + " failure_counter: int = 0\n", + " success_counter: int = 0\n", + " success_tolerance: int = 3\n", + " best_value: float = -float(\"inf\")\n", + " restart_triggered: bool = False\n", + "\n", + " def __post_init__(self):\n", + " n_splits = round(math.log(self.dim, self.new_bins_on_split + 1))\n", + " self.d_init = 1 + np.argmin(\n", + " np.abs(\n", + " (1 + np.arange(self.new_bins_on_split))\n", + " * (1 + self.new_bins_on_split) ** n_splits\n", + " - self.dim\n", + " )\n", + " )\n", + " self.target_dim = self.d_init\n", + " self.n_splits = n_splits\n", + "\n", + " @property\n", + " def split_budget(self) -> int:\n", + " return round(\n", + " -1\n", + " * (self.new_bins_on_split * self.eval_budget * self.target_dim)\n", + " / (self.d_init * (1 - (self.new_bins_on_split + 1) ** (self.n_splits + 1)))\n", + " )\n", + "\n", + " @property\n", + " def failure_tolerance(self) -> int:\n", + " if self.target_dim == self.dim:\n", + " return self.target_dim\n", + " k = math.floor(math.log(self.length_min / self.length_init, 0.5))\n", + " split_budget = self.split_budget\n", + " return min(self.target_dim, max(1, math.floor(split_budget / k)))\n", + "\n", + "\n", + "def update_state(state, Y_next):\n", + " if max(Y_next) > state.best_value + 1e-3 * math.fabs(state.best_value):\n", + " state.success_counter += 1\n", + " state.failure_counter = 0\n", + " else:\n", + " state.success_counter = 0\n", + " state.failure_counter += 1\n", + "\n", + " if state.success_counter == state.success_tolerance: # Expand trust region\n", + " state.length = min(2.0 * state.length, state.length_max)\n", + " state.success_counter = 0\n", + " elif state.failure_counter == state.failure_tolerance: # Shrink trust region\n", + " state.length /= 2.0\n", + " state.failure_counter = 0\n", + "\n", + " state.best_value = max(state.best_value, max(Y_next).item())\n", + " if state.length < state.length_min:\n", + " state.restart_triggered = True\n", + " return state" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create a BAxUS embedding\n", + "\n", + "We now show how to create the BAxUS embedding. The essential idea is to assign input dimensions to target dimensions and to assign a sign $\\in \\pm 1$ to each input dimension, similar to the HeSBO embedding. \n", + "We create the embedding matrix that is used to project points from the target to the input space. The matrix is sparse, each column has precisely one non-zero entry that is either 1 or -1." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## GP-EI\n", - "As a baseline, we compare BAxUS to Expected Improvement (EI)" + "data": { + "text/plain": [ + "tensor([[ 0., 0., 1., 0., 1., 1., 1., 0., 0., 0.],\n", + " [ 0., 0., 0., 0., 0., 0., 0., 1., 1., -1.],\n", + " [-1., 1., 0., -1., 0., 0., 0., 0., 0., 0.]],\n", + " dtype=torch.float64)" ] - }, + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def embedding_matrix(input_dim: int, target_dim: int) -> torch.Tensor:\n", + " if (\n", + " target_dim >= input_dim\n", + " ): # return identity matrix if target size greater than input size\n", + " return torch.eye(input_dim, device=device, dtype=dtype)\n", + "\n", + " input_dims_perm = (\n", + " torch.randperm(input_dim, device=device) + 1\n", + " ) # add 1 to indices for padding column in matrix\n", + "\n", + " bins = torch.tensor_split(\n", + " input_dims_perm, target_dim\n", + " ) # split dims into almost equally-sized bins\n", + " bins = torch.nn.utils.rnn.pad_sequence(\n", + " bins, batch_first=True\n", + " ) # zero pad bins, the index 0 will be cut off later\n", + "\n", + " mtrx = torch.zeros(\n", + " (target_dim, input_dim + 1), dtype=dtype, device=device\n", + " ) # add one extra column for padding\n", + " mtrx = mtrx.scatter_(\n", + " 1,\n", + " bins,\n", + " 2 * torch.randint(2, (target_dim, input_dim), dtype=dtype, device=device) - 1,\n", + " ) # fill mask with random +/- 1 at indices\n", + "\n", + " return mtrx[:, 1:] # cut off index zero as this corresponds to zero padding\n", + "\n", + "\n", + "embedding_matrix(10, 3) # example for an embedding matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Function to increase the embedding\n", + "\n", + "Next, we write a helper function to increase the embedding and to bring observations to the increased target space." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def increase_embedding_and_observations(\n", + " S: torch.Tensor, X: torch.Tensor, n_new_bins: int\n", + ") -> torch.Tensor:\n", + " assert X.size(1) == S.size(0), \"Observations don't lie in row space of S\"\n", + "\n", + " S_update = S.clone()\n", + " X_update = X.clone()\n", + "\n", + " for row_idx in range(len(S)):\n", + " row = S[row_idx]\n", + " idxs_non_zero = torch.nonzero(row)\n", + " idxs_non_zero = idxs_non_zero[torch.randperm(len(idxs_non_zero))].reshape(-1)\n", + "\n", + " if len(idxs_non_zero) <= 1:\n", + " continue\n", + "\n", + " non_zero_elements = row[idxs_non_zero].reshape(-1)\n", + "\n", + " n_row_bins = min(\n", + " n_new_bins, len(idxs_non_zero)\n", + " ) # number of new bins is always less or equal than the contributing input dims in the row minus one\n", + "\n", + " new_bins = torch.tensor_split(idxs_non_zero, n_row_bins)[\n", + " 1:\n", + " ] # the dims in the first bin won't be moved\n", + " elements_to_move = torch.tensor_split(non_zero_elements, n_row_bins)[1:]\n", + "\n", + " new_bins_padded = torch.nn.utils.rnn.pad_sequence(\n", + " new_bins, batch_first=True\n", + " ) # pad the tuples of bins with zeros to apply _scatter\n", + " els_to_move_padded = torch.nn.utils.rnn.pad_sequence(\n", + " elements_to_move, batch_first=True\n", + " )\n", + "\n", + " S_stack = torch.zeros(\n", + " (n_row_bins - 1, len(row) + 1), device=device, dtype=dtype\n", + " ) # submatrix to stack on S_update\n", + "\n", + " S_stack = S_stack.scatter_(\n", + " 1, new_bins_padded + 1, els_to_move_padded\n", + " ) # fill with old values (add 1 to indices for padding column)\n", + "\n", + " S_update[\n", + " row_idx, torch.hstack(new_bins)\n", + " ] = 0 # set values that were move to zero in current row\n", + "\n", + " X_update = torch.hstack(\n", + " (X_update, X[:, row_idx].reshape(-1, 1).repeat(1, len(new_bins)))\n", + " ) # repeat observations for row at the end of X (column-wise)\n", + " S_update = torch.vstack(\n", + " (S_update, S_stack[:, 1:])\n", + " ) # stack onto S_update except for padding column\n", + "\n", + " return S_update, X_update" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "code", - "execution_count": 12, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "11) Best value: -7.04e-01\n", - "12) Best value: -7.04e-01\n", - "13) Best value: -7.04e-01\n", - "14) Best value: -7.04e-01\n", - "15) Best value: -7.04e-01\n", - "16) Best value: -7.04e-01\n", - "17) Best value: -7.04e-01\n", - "18) Best value: -7.04e-01\n", - "19) Best value: -7.04e-01\n", - "20) Best value: -7.04e-01\n", - "21) Best value: -7.04e-01\n", - "22) Best value: -7.04e-01\n", - "23) Best value: -7.04e-01\n", - "24) Best value: -7.04e-01\n", - "25) Best value: -7.04e-01\n", - "26) Best value: -7.04e-01\n", - "27) Best value: -7.04e-01\n", - "28) Best value: -7.04e-01\n", - "29) Best value: -7.04e-01\n", - "30) Best value: -7.04e-01\n", - "31) Best value: -7.04e-01\n", - "32) Best value: -7.04e-01\n", - "33) Best value: -7.04e-01\n", - "34) Best value: -7.04e-01\n", - "35) Best value: -7.04e-01\n", - "36) Best value: -7.04e-01\n", - "37) Best value: -7.04e-01\n", - "38) Best value: -7.04e-01\n", - "39) Best value: -7.04e-01\n", - "40) Best value: -7.04e-01\n", - "41) Best value: -7.04e-01\n", - "42) Best value: -7.04e-01\n", - "43) Best value: -7.04e-01\n", - "44) Best value: -7.04e-01\n", - "45) Best value: -7.04e-01\n", - "46) Best value: -7.04e-01\n", - "47) Best value: -7.04e-01\n", - "48) Best value: -7.04e-01\n", - "49) Best value: -7.04e-01\n", - "50) Best value: -7.04e-01\n", - "51) Best value: -7.04e-01\n", - "52) Best value: -7.04e-01\n", - "53) Best value: -7.04e-01\n", - "54) Best value: -7.04e-01\n", - "55) Best value: -7.04e-01\n", - "56) Best value: -7.04e-01\n", - "57) Best value: -7.04e-01\n", - "58) Best value: -7.04e-01\n", - "59) Best value: -7.04e-01\n", - "60) Best value: -7.04e-01\n", - "61) Best value: -7.04e-01\n", - "62) Best value: -7.04e-01\n", - "63) Best value: -7.04e-01\n", - "64) Best value: -7.04e-01\n", - "65) Best value: -7.04e-01\n", - "66) Best value: -7.04e-01\n", - "67) Best value: -7.04e-01\n", - "68) Best value: -7.04e-01\n", - "69) Best value: -7.04e-01\n", - "70) Best value: -7.04e-01\n", - "71) Best value: -7.04e-01\n", - "72) Best value: -7.04e-01\n", - "73) Best value: -7.04e-01\n", - "74) Best value: -7.04e-01\n", - "75) Best value: -7.04e-01\n", - "76) Best value: -7.04e-01\n", - "77) Best value: -7.04e-01\n", - "78) Best value: -7.04e-01\n", - "79) Best value: -7.04e-01\n", - "80) Best value: -7.04e-01\n", - "81) Best value: -7.04e-01\n", - "82) Best value: -7.04e-01\n", - "83) Best value: -7.04e-01\n", - "84) Best value: -7.04e-01\n", - "85) Best value: -7.04e-01\n", - "86) Best value: -7.04e-01\n", - "87) Best value: -7.04e-01\n", - "88) Best value: -7.04e-01\n", - "89) Best value: -7.04e-01\n", - "90) Best value: -7.04e-01\n", - "91) Best value: -7.04e-01\n", - "92) Best value: -7.04e-01\n", - "93) Best value: -7.04e-01\n", - "94) Best value: -7.04e-01\n", - "95) Best value: -7.04e-01\n", - "96) Best value: -7.04e-01\n", - "97) Best value: -7.04e-01\n", - "98) Best value: -7.04e-01\n", - "99) Best value: -7.04e-01\n", - "100) Best value: -7.04e-01\n" - ] - } - ], - "source": [ - "X_ei = get_initial_points(dim, n_init)\n", - "Y_ei = torch.tensor(\n", - " [branin_emb(x) for x in X_ei], dtype=dtype, device=device\n", - ").unsqueeze(-1)\n", - "\n", - "# Disable input scaling checks as we normalize to [-1, 1]\n", - "with botorch.settings.validate_input_scaling(False):\n", - " while len(Y_ei) < len(Y_baxus):\n", - " train_Y = (Y_ei - Y_ei.mean()) / Y_ei.std()\n", - " likelihood = GaussianLikelihood(noise_constraint=Interval(1e-8, 1e-3))\n", - " model = SingleTaskGP(X_ei, train_Y, likelihood=likelihood)\n", - " mll = ExactMarginalLogLikelihood(model.likelihood, model)\n", - " optimizer = torch.optim.Adam([{\"params\": model.parameters()}], lr=0.1)\n", - " model.train()\n", - " model.likelihood.train()\n", - " for _ in range(50):\n", - " optimizer.zero_grad()\n", - " output = model(X_ei)\n", - " loss = -mll(output, train_Y.squeeze())\n", - " loss.backward()\n", - " optimizer.step()\n", - "\n", - " # Create a batch\n", - " ei = ExpectedImprovement(model, train_Y.max())\n", - " candidate, acq_value = optimize_acqf(\n", - " ei,\n", - " bounds=torch.stack(\n", - " [\n", - " -torch.ones(dim, dtype=dtype, device=device),\n", - " torch.ones(dim, dtype=dtype, device=device),\n", - " ]\n", - " ),\n", - " q=1,\n", - " num_restarts=NUM_RESTARTS,\n", - " raw_samples=RAW_SAMPLES,\n", - " )\n", - " Y_next = torch.tensor(\n", - " [branin_emb(x) for x in candidate], dtype=dtype, device=device\n", - " ).unsqueeze(-1)\n", - "\n", - " # Append data\n", - " X_ei = torch.cat((X_ei, candidate), axis=0)\n", - " Y_ei = torch.cat((Y_ei, Y_next), axis=0)\n", - "\n", - " # Print current status\n", - " print(f\"{len(X_ei)}) Best value: {Y_ei.max().item():.2e}\")" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "S before increase\n", + "tensor([[ 0., 1., 0., 0., -1., 1., 0., -1., 0., 1.],\n", + " [ 1., 0., -1., -1., 0., 0., 1., 0., -1., 0.]],\n", + " dtype=torch.float64)\n", + "X before increase\n", + "tensor([[98, 46],\n", + " [36, 42],\n", + " [55, 24],\n", + " [ 3, 14],\n", + " [87, 17],\n", + " [53, 10],\n", + " [96, 2]])\n", + "S after increase\n", + "tensor([[ 0., 0., 0., 0., -1., 1., 0., 0., 0., 0.],\n", + " [ 1., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n", + " [ 0., 0., 0., 0., 0., 0., 0., -1., 0., 1.],\n", + " [ 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n", + " [ 0., 0., 0., -1., 0., 0., 0., 0., -1., 0.],\n", + " [ 0., 0., -1., 0., 0., 0., 0., 0., 0., 0.]],\n", + " dtype=torch.float64)\n", + "X after increase\n", + "tensor([[98, 46, 98, 98, 46, 46],\n", + " [36, 42, 36, 36, 42, 42],\n", + " [55, 24, 55, 55, 24, 24],\n", + " [ 3, 14, 3, 3, 14, 14],\n", + " [87, 17, 87, 87, 17, 17],\n", + " [53, 10, 53, 53, 10, 10],\n", + " [96, 2, 96, 96, 2, 2]])\n" + ] + } + ], + "source": [ + "S = embedding_matrix(10, 2)\n", + "X = torch.randint(100, (7, 2))\n", + "print(f\"S before increase\\n{S}\")\n", + "print(f\"X before increase\\n{X}\")\n", + "\n", + "S, X = increase_embedding_and_observations(S, X, 3)\n", + "print(f\"S after increase\\n{S}\")\n", + "print(f\"X after increase\\n{X}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Take a look at the state" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Sobol" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "BaxusState(dim=500, eval_budget=500, new_bins_on_split=3, d_init=2, target_dim=2, n_splits=4, length=0.8, length_init=0.8, length_min=0.0078125, length_max=1.6, failure_counter=0, success_counter=0, success_tolerance=3, best_value=-inf, restart_triggered=False)\n" + ] + } + ], + "source": [ + "state = BaxusState(dim=dim, eval_budget=500)\n", + "print(state)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generate initial points\n", + "This generates an initial set of Sobol points that we use to start of the BO loop." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def get_initial_points(dim, n_pts, seed=0):\n", + " sobol = SobolEngine(dimension=dim, scramble=True, seed=seed)\n", + " X_init = (\n", + " 2 * sobol.draw(n=n_pts).to(dtype=dtype, device=device) - 1\n", + " ) # points have to be in [-1, 1]^d\n", + " return X_init" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generate new batch\n", + "Given the current `state` and a probabilistic (GP) `model` built from observations `X` and `Y`, we generate a new batch of points. \n", + "\n", + "This method works on the domain $[-1, +1]^d$, so make sure to not pass in observations from the true domain. `unnormalize` is called before the true function is evaluated which will first map the points back to the original domain.\n", + "\n", + "We support either TS and qEI which can be specified via the `acqf` argument." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def create_candidate(\n", + " state,\n", + " model, # GP model\n", + " X, # Evaluated points on the domain [-1, 1]^d\n", + " Y, # Function values\n", + " n_candidates=None, # Number of candidates for Thompson sampling\n", + " num_restarts=10,\n", + " raw_samples=512,\n", + " acqf=\"ts\", # \"ei\" or \"ts\"\n", + "):\n", + " assert acqf in (\"ts\", \"ei\")\n", + " assert X.min() >= -1.0 and X.max() <= 1.0 and torch.all(torch.isfinite(Y))\n", + " if n_candidates is None:\n", + " n_candidates = min(5000, max(2000, 200 * X.shape[-1]))\n", + "\n", + " # Scale the TR to be proportional to the lengthscales\n", + " x_center = X[Y.argmax(), :].clone()\n", + " weights = model.covar_module.base_kernel.lengthscale.detach().view(-1)\n", + " weights = weights / weights.mean()\n", + " weights = weights / torch.prod(weights.pow(1.0 / len(weights)))\n", + " tr_lb = torch.clamp(x_center - weights * state.length, -1.0, 1.0)\n", + " tr_ub = torch.clamp(x_center + weights * state.length, -1.0, 1.0)\n", + "\n", + " if acqf == \"ts\":\n", + " dim = X.shape[-1]\n", + " sobol = SobolEngine(dim, scramble=True)\n", + " pert = sobol.draw(n_candidates).to(dtype=dtype, device=device)\n", + " pert = tr_lb + (tr_ub - tr_lb) * pert\n", + "\n", + " # Create a perturbation mask\n", + " prob_perturb = min(20.0 / dim, 1.0)\n", + " mask = torch.rand(n_candidates, dim, dtype=dtype, device=device) <= prob_perturb\n", + " ind = torch.where(mask.sum(dim=1) == 0)[0]\n", + " mask[ind, torch.randint(0, dim, size=(len(ind),), device=device)] = 1\n", + "\n", + " # Create candidate points from the perturbations and the mask\n", + " X_cand = x_center.expand(n_candidates, dim).clone()\n", + " X_cand[mask] = pert[mask]\n", + "\n", + " # Sample on the candidate points\n", + " thompson_sampling = MaxPosteriorSampling(model=model, replacement=False)\n", + " with torch.no_grad(): # We don't need gradients when using TS\n", + " X_next = thompson_sampling(X_cand, num_samples=1)\n", + "\n", + " elif acqf == \"ei\":\n", + " ei = LogExpectedImprovement(model, train_Y.max())\n", + " X_next, acq_value = optimize_acqf(\n", + " ei,\n", + " bounds=torch.stack([tr_lb, tr_ub]),\n", + " q=1,\n", + " num_restarts=num_restarts,\n", + " raw_samples=raw_samples,\n", + " )\n", + "\n", + " return X_next" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Optimization loop\n", + "This simple loop runs one instance of BAxUS with Thompson sampling until convergence.\n", + "\n", + "BAxUS works on a fixed evaluation budget and shrinks the trust region until the minimal trust region size is reached (`state[\"restart_triggered\"]` is set to `True`).\n", + "Then, BAxUS increases the target space and carries over the observations to the updated space. \n" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "code", - "execution_count": 13, - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "X_Sobol = (\n", - " SobolEngine(dim, scramble=True, seed=0)\n", - " .draw(len(X_baxus_input))\n", - " .to(dtype=dtype, device=device)\n", - " * 2\n", - " - 1\n", - ")\n", - "Y_Sobol = torch.tensor(\n", - " [branin_emb(x) for x in X_Sobol], dtype=dtype, device=device\n", - ").unsqueeze(-1)" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "iteration 11, d=2) Best value: -6.42, TR length: 0.4\n", + "iteration 12, d=2) Best value: -4.6, TR length: 0.4\n", + "iteration 13, d=2) Best value: -4.6, TR length: 0.2\n", + "iteration 14, d=2) Best value: -4.6, TR length: 0.1\n", + "iteration 15, d=2) Best value: -3.64, TR length: 0.1\n", + "iteration 16, d=2) Best value: -2.36, TR length: 0.1\n", + "iteration 17, d=2) Best value: -1.73, TR length: 0.2\n", + "iteration 18, d=2) Best value: -1.19, TR length: 0.2\n", + "iteration 19, d=2) Best value: -0.661, TR length: 0.2\n", + "iteration 20, d=2) Best value: -0.518, TR length: 0.4\n", + "iteration 21, d=2) Best value: -0.518, TR length: 0.2\n", + "iteration 22, d=2) Best value: -0.518, TR length: 0.1\n", + "iteration 23, d=2) Best value: -0.518, TR length: 0.05\n", + "iteration 24, d=2) Best value: -0.416, TR length: 0.05\n", + "iteration 25, d=2) Best value: -0.409, TR length: 0.05\n", + "iteration 26, d=2) Best value: -0.409, TR length: 0.025\n", + "iteration 27, d=2) Best value: -0.406, TR length: 0.025\n", + "iteration 28, d=2) Best value: -0.406, TR length: 0.0125\n", + "iteration 29, d=2) Best value: -0.398, TR length: 0.0125\n", + "iteration 30, d=2) Best value: -0.398, TR length: 0.00625\n", + "increasing target space\n", + "new dimensionality: 6\n", + "iteration 31, d=6) Best value: -0.398, TR length: 0.4\n", + "iteration 32, d=6) Best value: -0.398, TR length: 0.2\n", + "iteration 33, d=6) Best value: -0.398, TR length: 0.1\n", + "iteration 34, d=6) Best value: -0.398, TR length: 0.05\n", + "iteration 35, d=6) Best value: -0.398, TR length: 0.025\n", + "iteration 36, d=6) Best value: -0.398, TR length: 0.0125\n", + "iteration 37, d=6) Best value: -0.398, TR length: 0.00625\n", + "increasing target space\n", + "new dimensionality: 18\n", + "iteration 38, d=18) Best value: -0.398, TR length: 0.4\n", + "iteration 39, d=18) Best value: -0.398, TR length: 0.2\n", + "iteration 40, d=18) Best value: -0.398, TR length: 0.1\n", + "iteration 41, d=18) Best value: -0.398, TR length: 0.05\n", + "iteration 42, d=18) Best value: -0.398, TR length: 0.025\n", + "iteration 43, d=18) Best value: -0.398, TR length: 0.0125\n", + "iteration 44, d=18) Best value: -0.398, TR length: 0.00625\n", + "increasing target space\n", + "new dimensionality: 54\n", + "iteration 45, d=54) Best value: -0.398, TR length: 0.4\n", + "iteration 46, d=54) Best value: -0.398, TR length: 0.2\n", + "iteration 47, d=54) Best value: -0.398, TR length: 0.1\n", + "iteration 48, d=54) Best value: -0.398, TR length: 0.05\n", + "iteration 49, d=54) Best value: -0.398, TR length: 0.025\n", + "iteration 50, d=54) Best value: -0.398, TR length: 0.0125\n", + "iteration 51, d=54) Best value: -0.398, TR length: 0.00625\n", + "increasing target space\n", + "new dimensionality: 162\n", + "iteration 52, d=162) Best value: -0.398, TR length: 0.8\n", + "iteration 53, d=162) Best value: -0.398, TR length: 0.8\n", + "iteration 54, d=162) Best value: -0.398, TR length: 0.4\n", + "iteration 55, d=162) Best value: -0.398, TR length: 0.4\n", + "iteration 56, d=162) Best value: -0.398, TR length: 0.4\n", + "iteration 57, d=162) Best value: -0.398, TR length: 0.2\n", + "iteration 58, d=162) Best value: -0.398, TR length: 0.2\n", + "iteration 59, d=162) Best value: -0.398, TR length: 0.2\n", + "iteration 60, d=162) Best value: -0.398, TR length: 0.1\n", + "iteration 61, d=162) Best value: -0.398, TR length: 0.1\n", + "iteration 62, d=162) Best value: -0.398, TR length: 0.1\n", + "iteration 63, d=162) Best value: -0.398, TR length: 0.05\n", + "iteration 64, d=162) Best value: -0.398, TR length: 0.05\n", + "iteration 65, d=162) Best value: -0.398, TR length: 0.05\n", + "iteration 66, d=162) Best value: -0.398, TR length: 0.025\n", + "iteration 67, d=162) Best value: -0.398, TR length: 0.025\n", + "iteration 68, d=162) Best value: -0.398, TR length: 0.025\n", + "iteration 69, d=162) Best value: -0.398, TR length: 0.0125\n", + "iteration 70, d=162) Best value: -0.398, TR length: 0.0125\n", + "iteration 71, d=162) Best value: -0.398, TR length: 0.0125\n", + "iteration 72, d=162) Best value: -0.398, TR length: 0.00625\n", + "increasing target space\n", + "new dimensionality: 485\n", + "iteration 73, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 74, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 75, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 76, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 77, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 78, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 79, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 80, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 81, d=485) Best value: -0.398, TR length: 0.8\n", + "iteration 82, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 83, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 84, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 85, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 86, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 87, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 88, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 89, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 90, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 91, d=485) Best value: -0.398, TR length: 0.4\n", + "iteration 92, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 93, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 94, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 95, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 96, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 97, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 98, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 99, d=485) Best value: -0.398, TR length: 0.2\n", + "iteration 100, d=485) Best value: -0.398, TR length: 0.2\n" + ] + } + ], + "source": [ + "evaluation_budget = 100\n", + "\n", + "state = BaxusState(dim=dim, eval_budget=evaluation_budget - n_init)\n", + "S = embedding_matrix(input_dim=state.dim, target_dim=state.d_init)\n", + "\n", + "X_baxus_target = get_initial_points(state.d_init, n_init)\n", + "X_baxus_input = X_baxus_target @ S\n", + "Y_baxus = torch.tensor(\n", + " [branin_emb(x) for x in X_baxus_input], dtype=dtype, device=device\n", + ").unsqueeze(-1)\n", + "\n", + "NUM_RESTARTS = 10 if not SMOKE_TEST else 2\n", + "RAW_SAMPLES = 512 if not SMOKE_TEST else 4\n", + "N_CANDIDATES = min(5000, max(2000, 200 * dim)) if not SMOKE_TEST else 4\n", + "\n", + "# Disable input scaling checks as we normalize to [-1, 1]\n", + "with botorch.settings.validate_input_scaling(False):\n", + " for _ in range(evaluation_budget - n_init): # Run until evaluation budget depleted\n", + " # Fit a GP model\n", + " train_Y = (Y_baxus - Y_baxus.mean()) / Y_baxus.std()\n", + " likelihood = GaussianLikelihood(noise_constraint=Interval(1e-8, 1e-3))\n", + " model = SingleTaskGP(\n", + " X_baxus_target, train_Y, likelihood=likelihood\n", + " )\n", + " mll = ExactMarginalLogLikelihood(model.likelihood, model)\n", + "\n", + " # Do the fitting and acquisition function optimization inside the Cholesky context\n", + " with gpytorch.settings.max_cholesky_size(max_cholesky_size):\n", + " # Fit the model\n", + " try:\n", + " fit_gpytorch_mll(mll)\n", + " except ModelFittingError:\n", + " # Right after increasing the target dimensionality, the covariance matrix becomes indefinite\n", + " # In this case, the Cholesky decomposition might fail due to numerical instabilities\n", + " # In this case, we revert to Adam-based optimization\n", + " optimizer = torch.optim.Adam([{\"params\": model.parameters()}], lr=0.1)\n", + "\n", + " for _ in range(100):\n", + " optimizer.zero_grad()\n", + " output = model(X_baxus_target)\n", + " loss = -mll(output, train_Y.flatten())\n", + " loss.backward()\n", + " optimizer.step()\n", + "\n", + " # Create a batch\n", + " X_next_target = create_candidate(\n", + " state=state,\n", + " model=model,\n", + " X=X_baxus_target,\n", + " Y=train_Y,\n", + " n_candidates=N_CANDIDATES,\n", + " num_restarts=NUM_RESTARTS,\n", + " raw_samples=RAW_SAMPLES,\n", + " acqf=\"ts\",\n", + " )\n", + "\n", + " X_next_input = X_next_target @ S\n", + "\n", + " Y_next = torch.tensor(\n", + " [branin_emb(x) for x in X_next_input], dtype=dtype, device=device\n", + " ).unsqueeze(-1)\n", + "\n", + " # Update state\n", + " state = update_state(state=state, Y_next=Y_next)\n", + "\n", + " # Append data\n", + " X_baxus_input = torch.cat((X_baxus_input, X_next_input), dim=0)\n", + " X_baxus_target = torch.cat((X_baxus_target, X_next_target), dim=0)\n", + " Y_baxus = torch.cat((Y_baxus, Y_next), dim=0)\n", + "\n", + " # Print current status\n", + " print(\n", + " f\"iteration {len(X_baxus_input)}, d={len(X_baxus_target.T)}) Best value: {state.best_value:.3}, TR length: {state.length:.3}\"\n", + " )\n", + "\n", + " if state.restart_triggered:\n", + " state.restart_triggered = False\n", + " print(\"increasing target space\")\n", + " S, X_baxus_target = increase_embedding_and_observations(\n", + " S, X_baxus_target, state.new_bins_on_split\n", + " )\n", + " print(f\"new dimensionality: {len(S)}\")\n", + " state.target_dim = len(S)\n", + " state.length = state.length_init\n", + " state.failure_counter = 0\n", + " state.success_counter = 0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## GP-EI\n", + "As a baseline, we compare BAxUS to Expected Improvement (EI)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Compare the methods\n", - "\n", - "We show the regret of the different methods." - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "11) Best value: -9.02e+00\n", + "12) Best value: -9.02e+00\n", + "13) Best value: -9.02e+00\n", + "14) Best value: -9.02e+00\n", + "15) Best value: -9.02e+00\n", + "16) Best value: -9.02e+00\n", + "17) Best value: -9.02e+00\n", + "18) Best value: -9.02e+00\n", + "19) Best value: -2.11e+00\n", + "20) Best value: -2.11e+00\n", + "21) Best value: -2.11e+00\n", + "22) Best value: -2.11e+00\n", + "23) Best value: -2.11e+00\n", + "24) Best value: -2.11e+00\n", + "25) Best value: -2.11e+00\n", + "26) Best value: -2.11e+00\n", + "27) Best value: -2.11e+00\n", + "28) Best value: -2.11e+00\n", + "29) Best value: -2.11e+00\n", + "30) Best value: -2.11e+00\n", + "31) Best value: -2.11e+00\n", + "32) Best value: -2.11e+00\n", + "33) Best value: -2.11e+00\n", + "34) Best value: -2.11e+00\n", + "35) Best value: -2.11e+00\n", + "36) Best value: -2.11e+00\n", + "37) Best value: -2.11e+00\n", + "38) Best value: -2.11e+00\n", + "39) Best value: -2.11e+00\n", + "40) Best value: -2.11e+00\n", + "41) Best value: -2.11e+00\n", + "42) Best value: -2.11e+00\n", + "43) Best value: -2.11e+00\n", + "44) Best value: -2.11e+00\n", + "45) Best value: -2.11e+00\n", + "46) Best value: -2.11e+00\n", + "47) Best value: -2.11e+00\n", + "48) Best value: -2.11e+00\n", + "49) Best value: -2.11e+00\n", + "50) Best value: -2.11e+00\n", + "51) Best value: -2.11e+00\n", + "52) Best value: -2.11e+00\n", + "53) Best value: -2.11e+00\n", + "54) Best value: -2.11e+00\n", + "55) Best value: -2.11e+00\n", + "56) Best value: -2.11e+00\n", + "57) Best value: -2.11e+00\n", + "58) Best value: -2.11e+00\n", + "59) Best value: -2.11e+00\n", + "60) Best value: -2.11e+00\n", + "61) Best value: -2.11e+00\n", + "62) Best value: -2.11e+00\n", + "63) Best value: -2.11e+00\n", + "64) Best value: -2.11e+00\n", + "65) Best value: -2.11e+00\n", + "66) Best value: -2.11e+00\n", + "67) Best value: -9.90e-01\n", + "68) Best value: -9.90e-01\n", + "69) Best value: -9.90e-01\n", + "70) Best value: -9.90e-01\n", + "71) Best value: -9.90e-01\n", + "72) Best value: -9.90e-01\n", + "73) Best value: -9.90e-01\n", + "74) Best value: -9.90e-01\n", + "75) Best value: -9.90e-01\n", + "76) Best value: -9.90e-01\n", + "77) Best value: -9.90e-01\n", + "78) Best value: -9.90e-01\n", + "79) Best value: -9.90e-01\n", + "80) Best value: -9.90e-01\n", + "81) Best value: -9.90e-01\n", + "82) Best value: -9.90e-01\n", + "83) Best value: -9.90e-01\n", + "84) Best value: -9.90e-01\n", + "85) Best value: -9.90e-01\n", + "86) Best value: -9.90e-01\n", + "87) Best value: -9.90e-01\n", + "88) Best value: -9.90e-01\n", + "89) Best value: -9.90e-01\n", + "90) Best value: -9.90e-01\n", + "91) Best value: -9.90e-01\n", + "92) Best value: -9.90e-01\n", + "93) Best value: -9.90e-01\n", + "94) Best value: -9.90e-01\n", + "95) Best value: -9.90e-01\n", + "96) Best value: -9.90e-01\n", + "97) Best value: -9.90e-01\n", + "98) Best value: -9.90e-01\n", + "99) Best value: -9.90e-01\n", + "100) Best value: -9.90e-01\n" + ] + } + ], + "source": [ + "X_ei = get_initial_points(dim, n_init)\n", + "Y_ei = torch.tensor(\n", + " [branin_emb(x) for x in X_ei], dtype=dtype, device=device\n", + ").unsqueeze(-1)\n", + "\n", + "# Disable input scaling checks as we normalize to [-1, 1]\n", + "with botorch.settings.validate_input_scaling(False):\n", + " while len(Y_ei) < len(Y_baxus):\n", + " train_Y = (Y_ei - Y_ei.mean()) / Y_ei.std()\n", + " likelihood = GaussianLikelihood(noise_constraint=Interval(1e-8, 1e-3))\n", + " model = SingleTaskGP(X_ei, train_Y, likelihood=likelihood)\n", + " mll = ExactMarginalLogLikelihood(model.likelihood, model)\n", + " optimizer = torch.optim.Adam([{\"params\": model.parameters()}], lr=0.1)\n", + " model.train()\n", + " model.likelihood.train()\n", + " for _ in range(50):\n", + " optimizer.zero_grad()\n", + " output = model(X_ei)\n", + " loss = -mll(output, train_Y.squeeze())\n", + " loss.backward()\n", + " optimizer.step()\n", + "\n", + " # Create a batch\n", + " ei = LogExpectedImprovement(model, train_Y.max())\n", + " candidate, acq_value = optimize_acqf(\n", + " ei,\n", + " bounds=torch.stack(\n", + " [\n", + " -torch.ones(dim, dtype=dtype, device=device),\n", + " torch.ones(dim, dtype=dtype, device=device),\n", + " ]\n", + " ),\n", + " q=1,\n", + " num_restarts=NUM_RESTARTS,\n", + " raw_samples=RAW_SAMPLES,\n", + " )\n", + " Y_next = torch.tensor(\n", + " [branin_emb(x) for x in candidate], dtype=dtype, device=device\n", + " ).unsqueeze(-1)\n", + "\n", + " # Append data\n", + " X_ei = torch.cat((X_ei, candidate), axis=0)\n", + " Y_ei = torch.cat((Y_ei, Y_next), axis=0)\n", + "\n", + " # Print current status\n", + " print(f\"{len(X_ei)}) Best value: {Y_ei.max().item():.2e}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Sobol" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "X_Sobol = (\n", + " SobolEngine(dim, scramble=True, seed=0)\n", + " .draw(len(X_baxus_input))\n", + " .to(dtype=dtype, device=device)\n", + " * 2\n", + " - 1\n", + ")\n", + "Y_Sobol = torch.tensor(\n", + " [branin_emb(x) for x in X_Sobol], dtype=dtype, device=device\n", + ").unsqueeze(-1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Compare the methods\n", + "\n", + "We show the regret of the different methods." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "tags": [] + }, + "outputs": [ { - "cell_type": "code", - "execution_count": 14, - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAKBCAYAAADKqj3oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAACYXElEQVR4nOzdd3hUVf7H8c8kmfSeQEIgIKD0XkVEiiCWtRcUV6Ou+lsNNtQVXRXYVVld3cWSVdeGXewFKyKICtKRXqSXECCk18nM/f3BZshM2iQzk5kk79fz5DH3zLnnfm9ygvOde4rJMAxDAAAAAOCGAF8HAAAAAKD5I7EAAAAA4DYSCwAAAABuI7EAAAAA4DYSCwAAAABuI7EAAAAA4DYSCwAAAABuI7EAAAAA4DYSCwAAAABuI7EAgFZk9+7dMplMMplMOumkk3wdjsd4675OOukke7u7d+/2WLue1FJ/p83NokWL7L+HMWPG+DocwCdILAA/MmfOHPv/mFz9uvHGGxt0jQULFujaa69Vt27dFBERofj4ePXr10/33nuvtmzZ0qi4N2/erHvvvVf9+vVTfHy8IiIi1K1bN6WlpWnBggUut1PXfUZGRiolJUU9e/bUxIkT9cADD+ijjz5Sfn5+o2J2xXXXXdfg3wdvLNDcVX2DXNtXcHCw2rRpo+HDh+uOO+7QypUrfR02AD8Q5OsAADSN/Px83XzzzZo7d65DeXFxsXJycrR+/Xo9/fTTmjlzpu6//36X23300Uc1c+ZMWSwWh/Lt27dr+/bteuONN3TVVVfpxRdfVFRUVKPjLyoqUlFRkTIzM7VlyxZ99913kqSIiAhdeeWVmjp1qnr16tXo9gG4zmKx6OjRozp69KiWL1+uZ555RpMmTdJLL73k1t85gOaNxALwUz169NCZZ55Zb73TTjut3joWi0UXX3yxfvjhB3tZnz59NGjQIJWWluqnn35SZmamLBaLHnjgAVksFj388MP1tvvwww/r73//u/24Xbt2GjVqlEJDQ7Vq1Spt3LhRkvTuu+8qOztbX375pYKCXPtn56KLLlL79u3txxUVFcrJyVF2drbWrFmjY8eOSTqecLzyyit666239Nhjj+muu+6SyWRy6RoN4ervo9Ipp5zi8RgAX0hPT69WVlJSov3792vJkiUqLCyUJM2dO1f79+/XokWLXP47B9DCGAD8xmuvvWZIMiQZaWlpHmv3oYcesrcbGhpqvPvuuw6vl5WVGffee6+9jslkMhYtWlRnm99//729viTj3nvvNcrKyhzqvPPOO0ZoaKi9zsyZM+tss2p7CxcurLPuxo0bjdtuu82IiIhwOC89Pb3O8xoiLS3NK78PX9q1a5f9njp16uTrcDzGW/fVqVMne7u7du3yWLue5Ol7X7hwocPfVF0KCwuNKVOmONTPyMhwOwYAzRNzLIAW7vDhw/rXv/5lP549e7auvPJKhzrBwcF64oknNGnSJEmSYRj1Doeq+vqVV16pJ554QsHBwQ51rrrqKv373/+2Hz/55JM6evRoo++lql69eumZZ57R2rVr1bdvX3t5RkaGnn/+eY9cA0DdIiIi9Oyzz2rixIn2srffftuHEQHwJRILoIV7/fXXVVRUJEnq1q2bbr755lrrPvHEEwoIOP7PwtKlS7VmzZoa661YsUIrVqyQJAUEBOiJJ56otc3/+7//sw8LKigo0Jtvvtmo+6jNySefrIULFyo1NdVe9sADD3h1UjcAR5MnT7Z/v2nTJh9GAsCXSCyAFu7TTz+1f1+5ylFtOnbsqHHjxtmPP/nkk3rbHD9+vMObemcmk0lpaWn1tumOhIQEvfLKK/bj3NxcZWRkePw6njBmzBj7yjqLFi2SJGVmZmrmzJkaOHCg4uPjFRoaqh49emjatGn2uSRV7d+/Xw888IAGDhyouLg4RUVFacCAAXrsscdUUlLSqLgWLFigq666Sl27dlVYWJjatGmjUaNG6bnnnlNZWVmD2ioqKtLzzz+v888/X506dVJ4eLiioqJ0yimn6IYbbnCY6+OKzMxM/fWvf1W/fv0UHR2t6Oho9e7dW3fddZe2bt3aoLYqlZWV6dlnn9WoUaPUpk0bhYWFqWvXrpo8ebIWLlzYqDal5nHv3tCuXTv795UfZNSk6kprc+bMkXT87/Xpp5/WGWecofbt2ysoKEgmk0m5ubkO5x4+fFivvfaa0tLS7H8rZrNZsbGx6tGjh66//np9++23LsU7Y8YMexwzZsyQdHwe1xtvvKHx48erffv2CgkJUbt27XTRRRdp3rx59bbpynKztS0NvHLlSt14443q1q2bwsPDFRcXp2HDhumxxx6r8+cJ+B1fj8UCcIKn51iUlJQYAQEB9jaXLFlS7zmPPvqovf7IkSNrrDNixAh7nccee6zeNn/55Rd7/cDAQKO0tLTGepLrcyxq0rdvX/v5/fr1a/D5zrwxx2L06NEO9/jtt98aCQkJDvde9atTp07G7t277ee/8sorRkhISK31e/fubRw+fLjW6zuPxy8vLzduvvnmWtuTZPTs2dPYunWrS/f3/vvvG8nJyXW2J8n4wx/+YOTm5tbb3scff2zExsbW2k5ISIjx0ksvNWiewaZNm4zu3bvXGd+f//xno7y8vEFzLJrDvbuiIXMsKr3xxhv2+qmpqbXWq/o39dprrxk///yzkZqaWuP95eTk2M97+umnjcDAwHp/tpKMcePGGUePHq0z3unTp9vrT58+3di/f79x2mmn1dnu9ddfb1itVpd+bqNHj66xjvPvymazGQ8//LDDv9POX507dzZ27NhR5/0A/oJlGwA/lZubqw8++EAbN25UXl6eoqOjlZKSohEjRqhv374urXy0detW2Ww2ScefHAwcOLDecwYNGmT/fvPmzTXWqVpetX5tql7XarVq27ZtDvMiPOXyyy/X+vXrJUkbNmxQbm6uYmNjPX4dT1m7dq0eeOABlZSUqEOHDho5cqSioqK0bds2/fTTTzIMQ3v27NE555yj9evXa+7cufrTn/4k6fiqU8OGDVNoaKjWr1+v5cuXS5I2btyoa665Rt98841LMdx3333673//K0nq16+fBgwYIMMwtGrVKvuQls2bN2vcuHFaunRpnU+n/v3vf+vuu++WYRiSpOjoaI0YMUIdOnSQ1WrVxo0btXLlShmGoXnz5mnMmDH65ZdfFB4eXmN7X375pa644gpVVFRIOj7sbuTIkerWrZsKCwu1ePFiZWZm6qabbtIzzzzj0v3u2bNHZ555pjIzM+1lvXv31qBBg2QymbR69Wpt2LBBL7zwQq1xNdd796aqy1iPGjXKpXN+//133XnnncrLy1NUVJTOOOMMpaSkKCcnR4sXL3aoe/DgQVmtVklSly5d1LNnT7Vp00ahoaHKzc3V+vXr7avQ/fDDDxo/frx+/fVXhYSE1BtHYWGhzj77bG3YsEHh4eEaNWqUUlNTVVBQoIULF+rw4cOSpNdee03du3fXfffd59L9uWLmzJn629/+JkkaMGCA+vbtK7PZrLVr12r16tWSpF27dumiiy7S6tWrWW0L/s+XWQ0AR1WfWNT1dcoppxgvv/yyYbPZ6mxv7ty59nOSkpJcimHjxo0O13L+9DsrK8vh9c2bN7vUbps2beznvP/++zXWqdpuY55YfPvttw5tfPvttw1uoypvP7EICQkxzGazkZGRUe2T0EWLFjmsePXYY48ZkZGRRnR0tPHhhx9Wa3fu3LkOn+j++OOPNV6/6iemZrPZkGQkJCTU+LP6/PPPjejoaHv9iRMn1npf33//vf1T1+DgYOMf//iHUVRUVK3emjVrjF69etnbvOWWW2ps7+jRo0bbtm3t9fr27Wts2rTJoY7VajUef/xxw2QyGcHBwS59an/mmWfa68XExBhffPFFtTpfffWVERcX5/AzUh1PLJrLvbuqIU8siouLjXvuucdeNygoyFi1alWt9av+TQUFBRnS8ZXcCgoKHOqVl5c7/E288sorxrPPPmvs37+/1rZ/++03Y8iQIfb2//73v9dat+oTi8ongGlpaUZ2drZDvaKiIuOqq66y142MjDQKCwtrbLOhTyyCg4MNk8lkdO3a1Vi2bFm1uu+//75D/3v99ddrvR/AX5BYAH7E1cSi8usPf/hDrf+TMwzD+M9//mOv6+rQoOzsbIdrbNmyxeH1TZs2Obx+7Ngxl9qtOkzphRdeqLGOu4nF7t27Hdp44403GtxGVVXfBPXo0cNIT093+Wvbtm01tlk1sZBkvPzyy7Ve/5FHHnGoazKZjAULFtRa/8Ybb6z3TWvVNzaSjICAAOOXX36ptc358+c71K/p+lar1TjllFPsdT7++ONa2zMMw8jMzDSSkpLsb9z37dtXrc4DDzzgkBRnZWXV2p7zz6m2N9ffffedw8/yhx9+qLXNxYsXGyaTyaHdmhKL5nLvDeGcWNTUv2+88Ubj7LPPdkg8o6OjjW+++abOtqv+TUkybrzxRrfjrSo3N9c+HK1du3ZGRUVFjfWqJhaSjKuuuqrWNktKShyGa7333ns11mtoYlGZ1B84cKDWa1dN2s4+++zabxzwEyQWgB957bXXjI4dOxp333238dVXXxn79u0zSktLjaKiImPr1q3Gf/7zH6NHjx4O/2O64IILah33+8QTT9jrDR8+3KUYiouLHdpfuXKlw+vLly93eL2kpMSldocNG2Y/58knn6yxjruJRU5OjkMbTz/9dIPbqMr5TVBDvmqLv2pi0b9//zqvv2PHDoc2L7roojrrL1iwwF538ODBNdZxfmNzzTXX1PtzuOSSS+z1r7zyymqvf/rppy7HWGnWrFn2c5566imH12w2m8Nchfr2RXCeC1Hbm+srrrjCXufyyy+vN8bJkyfXm1g0l3tvCOfEwpWvq6++ut55DYbh+DcVGhrq8gcTDXHLLbfYr7Fu3boa61RNLIKDg43MzMw62/zLX/5irz916tQa6zQmsXD+/Tur+kFOQkJCnXUBf8CqUIAfueiii7Rr1y49+eSTOuecc9ShQweFhIQoPDxc3bp10y233KLffvtN119/vf2czz//XO+8806N7ZWWltq/d95jojbOY5KdVxmq2mZj223sykX1iYyMdDguKCjwynU85bLLLqvz9S5duigiIsLl+n369LF/v2vXLpdiuPbaa+utU3VVr5pWTPrqq6/s31dddrQuVVcf+/nnnx1e27x5sw4dOiRJCgoKqrdNs9ns0nWrxt7Q+65Nc7l3b3v77bc1cODABq36dtZZZykuLq7B1zp8+LA+//xzPf7445o2bZpuu+02TZkyxf61cuVKe921a9fW297pp5+u5OTkOutUnSe2e/fuBsdcm8svv7zO13v06KGwsDBJUnZ2tt//mwYwCwjwI65MNA4ODtbLL7+s33//XT/99JMk6fHHH9cf//jHanVDQ0Pt35eXl7sUg/PSopX/U6upzcp2ncvqa9e5TU9x/p9udHS0x9pOS0uzL4/pKVUTgdrExsbal5vs3bt3nXXj4+Pt37uyj4fJZNLw4cPrrTdixAj791lZWcrMzHRYXnTp0qX27z/66CP9+OOP9baZl5dn/37fvn0Or1XdP6VHjx4u/V1UjbEmBw4c0JEjR+zHp556ar1tnnrqqTKZTPYJ2TVpDvfurpru32q1Kjs7W6tWrdILL7ygzz//XPv27dMll1yiF198sc79cioNHjy4QXFs2rRJ9913n77++mv7RO76uLIhpysLSSQkJNi/99QeOTExMXUuhiAd/xuNi4uzfxiTn5+vqKgoj1wf8AYSC6AZCggI0PTp0zV+/HhJx1dA2r9/vzp06OBQr+on+K4+JXCu5/wUwPm4pKTEpcSiarvObXhK1TdskuMbbX8UExNTb52qq8DUV79q3crVhOpSuQdGfSpX36l8WnXkyBGHxOLgwYP276uuDuSqnJwch+OqCUDHjh1daqO+elXbDA8PV2JiYr1tRkdHKyYmptp+ClU1h3v3hsDAQLVt21bnnHOOzjnnHM2YMUMzZ86UJN1+++0aM2aMunXrVmcbbdq0cfl63377rS688MIG76niyif8rvwdms1m+/cWi6VBMbhzXW9dG/AWhkIBzdQZZ5zh8D+cmpaGrfopW1ZWlkvtVg7DqOT85rxqm41t11tv+Lds2eJwXN/wBl9zZclgd+rXpyHLqVYdkuX8Zs05oWso5ySosLDQ/r2rMVaNryaNadOVdpvDvTeFBx980J7glJWVubQErqtPLo8cOaJJkybZk4pOnTpp1qxZ+vnnn3Xw4EEVFxfLZrPJOD5vVNOnT7efW7ncdl08/XflKl9dF/AmEgugmTKbzQ6futb0yL979+727w8fPlxtfkRN9u7da/8+Pj6+2qeKbdu2dRiesWfPnnrbLC0tdfgktkePHvWe0xjLli2zfx8YGKihQ4d65TotRXFxsct1q+7+6/yUo+ob29WrV9vf4Ln65TxmveoTLVdjrG934sa06Uq7zeHem0JQUJDOPPNM+/GCBQs81vZLL71kT+D69++vdevWadq0aRo5cqTatWunsLAwhzfpzEMAfIfEAmjGqr6hqOlTy+7duysg4PifuWEYLk1krNyUSZJ69uxZY52q5VXHhLvSZmBgYL1DJBrrww8/tH/fv39/j86xaIlycnIcPiGvzdGjRx2SUudhRElJSfbvnZ94NUbVZLZqolsX57kKdbVZXFys7OzsetssKCio94lEc7j3plJ1eJwrHzi4qmqS8uCDD9b7d+3JawNoGBILoJnauXOnwyTClJSUanVCQ0MdJqkuWrSo3narTj6tunpNVWPHjm10m6eddppLu+E21HfffacNGzbYj6+88kqPX6OlMQzD4SlPbapOUE5KSqrW16pOAP/ll1/cjqvqCjxbtmxxabhR1Rhr0r59e4c37b/++mu9bf766691TtyWmse9N5WqT1gqP9DwhKrzWOqbaG21Wj3yewDQOCQWQDP16quv2r+PiYnRgAEDaqx30UUX2b+vb1Wjffv2OXw6WPXc2tr8/vvvtX///jrbrXrd2tp0R3Z2tm688Ub7cUJCgm655RaPX6clevPNN+ut88Ybb9i/r5pUVvrDH/5g//7VV191achdXXr06GGfH1NRUaF33323zvqu1JEcY2/ofdemudx7U6j6ZLJ9+/Yea7dqklLf8LBPP/3UI0+OADQOiQXgJ1wZklJpyZIleuqpp+zHV155pcOKQFWlpaXZh0lt3bpVL7/8cq3t3nffffZlHEeMGKFBgwbVWG/o0KH2+QtWq1XTpk2rtc3//ve/2rZtm6TjY/Nd2T+gIX7//XeNGzfOYTjIE0884bWVp1qat956q86nFgsXLtRHH31kP66awFW69NJLdfLJJ0uSMjMzdeutt9b7SX+lwsLCanMEAgICdMMNN9iPZ86c6TBHx9mTTz7p0r4dVWN///33tXjx4lrr/vLLL7XuD1NVc7l3b1u5cqV9+WtJDvMt3NWlSxf7959//nmt9Y4cOaK77rrLY9cF0HAkFoCf+PDDDzVs2DC98cYbtQ5/KC0t1TPPPKPx48fbPxmNjY11WAXFWdu2bTV16lT78e23367333/foY7FYtG0adMcPvmcNWtWnfFWff3tt9/WtGnTqi2F+P777+vOO++0H99zzz0uLfPpis2bN+uOO+7QgAEDtG7dOnv51KlTHd6YoXZms1lWq1V/+MMf9P3331d7/csvv9TFF19sf6M8YcKEGt8wBgYG6vnnn1dgYKAk6bXXXtN5551X40plldauXav77rtPqampNb4xvuuuu+x95dChQ5owYUK1Vb9sNpueeuop/fWvf3Vpo8YJEybYn1oYhqGLLrrIYYO7St99950uuOAC2Ww2h5XXatJc7t2bvv/+e1144YX2fhIUFKTbbrvNY+2ff/759u9nzZqlt956q1qd1atXa/To0dq3b59frJIFtFbsYwH4kRUrVigtLU1BQUHq0aOHevToobi4OFmtVh04cEBLly51mFcRFhamzz77zGHSZE0eeugh/fLLL/rhhx9UUlKiSZMm6ZFHHtGgQYNUWlqqxYsXKzMz015/5syZGj16dJ1tnnnmmXrwwQf1yCOPSDq+Sd+bb76pUaNGKTQ0VKtWrXKY8zBhwgQ98MADLv8snn76aYfJ2BUVFcrNzVV2drbWrFlTbfJtWFiYHn/8cU2ZMsXlazTEsmXLGtz2P//5T69tBugJKSkpuvjiizV79mxNmDBB/fv314ABA2QYhlatWqWNGzfa67Zr104vvfRSrW2NHz9ezz//vG655RZZrVZ9/fXX+uabb9SrVy/169dP0dHRKi4uVmZmpn777bc6P4WXjk8Qf+WVV3TJJZfIarXqt99+U+/evXX66aerW7duKiws1OLFi+3j7//5z3/qjjvuqPeeX3nlFY0YMUJZWVnKycnReeedpz59+mjQoEEymUxas2aNPVGdOnWqPvroo3onAzeXe2+smvq91WrVsWPHtGrVKu3YscPhtSeffLLWhR8aIy0tTU899ZS2bdumsrIyXXPNNXrsscfUv39/hYaGasOGDfbdtvv376+JEyfqiSee8Nj1ATSAAcAvvPbaa4Ykl7+GDRtmbNq0yeX2c3NzjSuuuKLONs1ms/Hoo4+63KbNZjP+/ve/G2azuc52r7zySiMvL6/e9hpy/5VfkZGRxk033WRs2bLF5bhdlZaW1qiYKr9ycnKqtTl69Gj76wsXLqw3hk6dOtnr79q1q976Va9fk127dtlf79Spk1FeXm786U9/qvM+unfvbmzevLneaxuGYfzwww/GKaec4vLPqHfv3saBAwdqbe+DDz4wYmJiaj0/JCTEePHFF6vdV102bNhQb4w33XSTUV5e3qCff3O4d1csXLiwUf09Pj7eePPNN+tsu+rf1GuvveZyTFu3bjW6dOlS5/VHjhxp7N+/35g+fbq9bPr06TW250qd2n4mo0ePbnSdxvyuGvpvAOBLPLEA/MRVV12lbt26acmSJfr111+1Y8cOHT16VNnZ2bLZbIqJiVHnzp116qmn6rLLLtPpp5/eoPZjYmI0d+5c3XTTTXr99de1dOlSZWZmymw2KzU1VRMnTtSf/vSnBn3SaDKZ9OCDD+rSSy/Vyy+/rO+++0779u2TxWJRu3btNGLECKWlpdl3CHdHWFiYYmJiFB0drU6dOmnw4MEaOnSoJkyY4NLu0aiZ2WzWyy+/rMsvv1yvvPKKVqxYoczMTEVERKhnz56aNGmSbr75ZpdX8ho7dqw2b96sTz/9VF9++aV+/fVXHTp0SPn5+QoPD1dSUpJ69Oih0047Teecc06tiw5Uuuyyy3Taaafp2Wef1RdffKE9e/bIZDKpQ4cOGj9+vG655Rb17Nmz2n4Qdendu7fWrVun//73v5o7d662bNmi4uJitWvXTkOHDtWNN96oCRMmuNxec7p3TzGZTIqKilLbtm01YMAATZw4UVdeeaXX5jZ169ZNa9asUUZGhj7++GNt3bpV5eXlSk5OVt++fTV58mRdccUV9iFpAHzDZBguzjIDAAAAgFoweRsAAACA20gsAAAAALiNxAIAAACA20gsAAAAALiNxAIAAACA21hu1kNsNpsOHjyoqKgomUwmX4cDAAAAODAMQwUFBUpJSVFAgOefL5BYeMjBgweVmprq6zAAAACAOu3bt08dOnTweLskFh5SuUHXrl27FB8f7+No4I8sFou+++47nXXWWTKbzb4OB36G/oH60EdQH/oI6nPs2DF17tzZaxvLkli4KSMjQxkZGbJarZKOJxjR0dE+jgr+yGKxKDw8XNHR0fyDj2roH6gPfQT1oY+gPhaLRZK8NmyfydtuSk9P16ZNm7RixQpfhwIAAAD4DIkFAAAAALeRWAAAAABwG4kFAAAAALeRWLgpIyNDvXr10tChQ30dCgAAAOAzJBZuYvI2AAAAQGIBAAAAwANILAAAAAC4jcQCAAAAgNtILAAAAAC4jcTCTawKBQAAAJBYuI1VoQAAAAASCwAAAAAeQGIBAAAAwG0kFgAAAADcRmIBAAAAwG0kFh5WUpLj6xAAAACAJkdi4Sbn5WZ/XJXh44gAAACApkdi4Sbn5WZ/OvSLjyMCAAAAmh6JhYcttxaopPiYr8MAAAAAmhSJhYeVBpi0ZO1Lvg4DAAAAaFIkFl7ww+5vfR0CAAAA0KRILLxgUdlhWSzFvg4DAAAAaDIkFl6QH2DS6vVv+ToMAAAAoMmQWHjJgu2f+ToEAAAAoMmQWHjJD0V7ZNhsvg4DAAAAaBIkFm5y3iCvUlagSRu3fOSjqAAAAICmRWLhJucN8qpasHmuDyICAAAAmh6JhRf9kLfN1yEAAAAATYLEwot2BhratXuRr8MAAAAAvI7EwsMSrYbD8YL1c3wTCAAAANCESCw87PSw9g7HPxz9zUeRAAAAAE2HxMLDzjjpLIfj9QEVyspa56NoAAAAgKZBYuFhA3peqSib43CohWte8lE0AAAAQNMgsfCwBdsKdEZwG8eyrGU+igYAAABoGiQWHvbT9myd2WmCQ9lKo1h5eXt9FBEAAADgfSQWHrb2QJ5GDrpZIVWGQ1WYTFq8+r8+jAoAAADwLhILDzuYW6rCikiNCIx2KP9h/yLfBAQAAAA0gSBfB9ASrdyTo3HtT9ei/V/by36pyFVe3l6FhsS43E5QUJgCg4K9ESIAAADgUSQWbsrIyFBGRoasVqu9bOXuHN028v8UsO8r2UwmSVJJgEmnf3peg9qOshm6MqaXbr/kfY/GDAAAAHgaQ6HclJ6erk2bNmnFihX2slV7jikuvqsGK9SttgsCTHqpYLOWMT8DAAAAfo7Ewgs2HsxXSblVE9ud5pH25m//1CPtAAAAAN5CYuEFFTZDv+3P1aXjntAYU5Tb7S0p2ueBqAAAAADvYY6Fl6zak6NTuyTo2WuX6Nix31VUlOXyubszV+nWzSd2694XKO3b94tSU0d6I1QAAADAbSQWXrJy9zH79/HxJys+/mSXz22fMlzxG/+rYwEme9mSTXM1icQCAAAAfoqhUF6yak+ObFU2yWuIgMAgjQhu41D2S9YqT4QFAAAAeAWJhZfkl1bo9yOFjT5/ZIrjxO/l1jxZLMXuhgUAAAB4BYmFF63cndPoc0f0vcbhuCjApN82sp8FAAAA/BOJhRet2tP4xCIxsYd62Bx/PUt2fuVuSAAAAIBXkFh40ao9x+qvVIfToh0nfC/J2+ZWewAAAIC3kFh40e7sYh0pKGv0+SO7nONwvMlUoZxjO9wNCwAAAPA4EgsPCw92/JG6MxxqQK9JCquyspRhMmnp+jca3R4AAADgLSQWHta3fYzDsTvDoYJDojQ00HHn7l8O/Nzo9gAAAABvIbHwsP5OicVKN55YSNJpbQY6HC8tzZJhs7nVJgAAAOBpJBYe1r+jY2Kx4UCeSi3WRrc3suckh+MjgSZt3/lto9sDAAAAvIHEwsP6psQowHTi2GI1tP5AXqPb69RxlNo75SVLtnzU6PYAAAAAbyCxqOLiiy9WXFycLrvsska3ERUapO7J0Q5l7myUZwoI0Gnh7R3Kfsle1+j2AAAAAG8gsajijjvu0BtvuL/q0uBOsQ7Hbu9n0WG0w/Fqo1glxe61CQAAAHgSiUUVY8aMUVRUVP0V6zGkU7zD8ao9OTIMo5ba9Rve71oFVjm/3GTSqg1vN7o9AAAAwNNaTGKxePFinX/++UpJSZHJZNKnn35arU5GRoZOOukkhYaGavjw4Vq+fLlXYhncKc7hOKfYoh1HihrdXlR0e/VTiEPZL3vmN7o9AAAAwNOCfB2ApxQVFal///664YYbdMkll1R7fe7cuZo6dapeeOEFDR8+XLNnz9bEiRO1detWtW3btsHXKysrU1nZiV218/PzJUkWi0VJ8VFKigpRVpVdt5fvPKpOcSHV2nHViJjuWpO/3n68pGC3LBZLo9tD06v8ffF7Q03oH6gPfQT1oY+gPt7uGybDnTE6fspkMumTTz7RRRddZC8bPny4hg4dqueee06SZLPZlJqaqttuu03Tpk2z11u0aJGee+45ffjhh3VeY8aMGZo5c2a18nfeeUfh4eF6bVuA1mafeCA0vI1Nk09u/P4TuQW/6knrPIeyB0NvVGjoSY1uEwAAAK1HcXGxJk+erLy8PEVHR9d/QgO1mCcWdSkvL9eqVat0//3328sCAgI0fvx4LV26tFFt3n///Zo6dar9OD8/X6mpqRo7dqwSEhKUFbtHa7/ean/9iBGpc889vdH3YK0Yr5fe+0J5VdayNcXu1rnjbm10m2haFotF8+fP14QJE2Q2m30dDvwM/QP1oY+gPvQR1Cc7O9ur7beKxOLo0aOyWq1KSkpyKE9KStKWLVvsx+PHj9dvv/2moqIidejQQR988IFGjBhRY5shISEKCak+tMlsNstsNmt4l0RJJxKLnUeLVVBuKD4iuFH3YDabdao5Xt9aTyxd+23mT0pc8e9Gtecx5jAp8RQpsGH31S6inXrE95DJZKq/cgtT2UeAmtA/UB/6COpDH0FtvN0vWkVi4arvv/++wedkZGQoIyNDVqvjLna9UqIVZg5USZVdt++cu1aJDUgsYsLNunRQB/Vpf3w375HJw/XtgW/sry83lWn59jcbHLO/mNxjsu4ffn/9FQEAAOD3WkVikZiYqMDAQGVlZTmUZ2VlKTk52a2209PTlZ6ervz8fMXExNjLzYEB6p8ao193nthvYvG2Iw1u/8OV+/XFbafrpMQIjehztVQlsWju5m6dq1sH3KqYkJj6KwMAAMCvtZjlZusSHByswYMHa8GCBfYym82mBQsW1DrUyROc97NojIKyCr24eIckKTl5gIYr1O02/YXVsGpv/l5fhwEAAAAPaDFPLAoLC/X777/bj3ft2qW1a9cqPj5eHTt21NSpU5WWlqYhQ4Zo2LBhmj17toqKinT99dd7LaarT+2oOUt2q7Cswq12Plp9QHef1V2JkSH624Tn9fTCe7S+zLuTb+plWCWjyipXobFSeP2J1KGiQ7LYTix1tr9wv/q26euFAAEAANCUWkxisXLlSo0dO9Z+XLliU1pamubMmaNJkybpyJEjevjhh3Xo0CENGDBA33zzTbUJ3Q1V2xwLSWoXE6Z5t52urzZkKr/E9eTCkKFXf94li/X4SsDlFTa9uXSP7prQTSkpQ/T41YvcitkjvrxHWvHSieNBZ0kXPFPvaTd/d7OWZp5YietA4QFvRAcAAIAm1mISizFjxqi+LTmmTJmiKVOmePS6tc2xqHRSYoRuHXNyg9s9WlCuj1bvtx+/+ese3TKmq0LNgW7F6zGhTvdamufSae2j2kuZJ473F+yvvTIAAACajVYxx6I5unFUZ4fjY0Xl+ni1H326HxbreFya69JpHSI7OBzvLySxAAAAaAlILPxUz3bRGnVKokPZyz/vlM3mJxulu/PEogqeWAAAALQMJBZuysjIUK9evTR06FCPt33jqC4OxzuPFGnh1sMev06jhMY6HpfkunRaamSqw/GhokOqsLk3uR0AAAC+R2LhpvT0dG3atEkrVqzweNtnnJKo7klRDmUv/bTT49dplMY+sYh0fGJhNaw6VHTIU1EBAADAR0gs/JjJZNKfnOZa/LrzmDYccO1NvFfVlFjUM3lekmJCYhRpjnQoY54FAABA80di4ecuHJCiNlEhDmV+8dTCefK2YZXKC+s9zWQyVXtqcaDAjyalAwAAoFFILPxcSFCg0kZ0ciibty5TB3NLfBTR/zg/sZBcHg7VIYqVoQAAAFoaEgs3eXPydqWrh3dSqPnEr8pqMzRnyW6vXc8lITGSTI5lLk7gdl5ylicWAAAAzR+JhZu8OXm7UlxEsC4f7Lia0rvL9qqg1OK1a9YrIEAKiXYsa+ySszyxAAAAaPZazM7bLd2fTu+st5btsc+PLiir0N/nbVL/1FifxXRhQIQiVSWZaOQmeQcKeWIBAADQ3JFYNBMnJUZoQs8kfbcpy172/sr9en+l7z7tHxBsVu+qz7wa+cTiWOkxFVuKFW4O92B0AAAAaEoMhWpGbjqjS/2VmlCeEeFY4OIcC+dVoSSGQwEAADR3JBbNyJBOcTqta4Kvw7DLl1Ni4eITi5DAELUNa+tQtr+AxAIAAKA5YyiUmzIyMpSRkSGr1er1a5lMJj03eZCe/n6bNmXme/16dSkorVD+UcehSxUlOS53qA5RHXS45LD9mMQCAACgeSOxcFN6errS09OVn5+vmJga9nbwsPiIYM28sI/Xr1OfY0Xl+niW4xOLnKNH1MbF89tHttfqw6vtx0zgBgAAaN4YCoVGiY8IVmhUnENZXs5Rl89nkzwAAICWhcQCjda2bbLDcVlhtsvnOk/gZpM8AACA5o3EAo3WqX2Kw3FgWb7ySlzbtM/5icWBwgMyKjfpAAAAQLNDYoFG69zB8alDtKlIv+507amF8xOLUmupsktdf+IBAAAA/0JigUYLjnScYxGtYv3yu2vzLNqGt5U5wOxQxspQAAAAzReJhZsyMjLUq1cvDR061NehNL3QWIfDSFOplm475NKpAaaAak8tmMANAADQfJFYuCk9PV2bNm3SihUrfB1K0wutvrzu0ewjOpBb4tLp7aOcEgueWAAAADRbJBZovBoSi2iT68OhOkQ6LTlLYgEAANBskVig8cxhUmCwQ1GMihqdWLBJHgAAQPNFYoHGM5mqPbWofGJhs9W/dCyb5AEAALQcJBZwj9ME7hgV6WhhubZmFdR7qvPk7ayiLFmsru2DAQAAAP9CYgH3VHtiUSRJLg2Hcn5iYcjQwaKDnosNAAAATYbEAu4Ji3U4jNHxxOJnFxKLqOAoRQdHO5QdKGCeBQAAQHNEYgH31DDHQpKW7TymsgprvaczzwIAAKBlILFwU6veIE+qcY6FJJVYrFqzN7fe09kkDwAAoGUgsXBTq94gT6p1joXUuHkW7GUBAADQPJFYwD3OiYWK7d+7Ms+CTfIAAABaBhILuMd58naVJxa/7ctVXkndy8eySR4AAEDLQGIB99QyeVuSbIb0687sOk9vH+U4xyK/PF/55fmeiw8AAABNgsQC7nGavJ0QWOxwXN88i5SIFJlkcihjyVkAAIDmh8QC7nF6YhFpFEky7MfzN2Xp898OatfRItlshpyZA81Kjkh2KGNlKAAAgOYnyNcBoJlzmmMRaFQoTGUqUagkKTOvVLe/u0aSFBUapD4pMerbIUZ92seob/sYdYoPV/vI9sosyrS3wRMLAACA5ofEAu5xGgolSalh5dpWElqtvKC0Qkt3ZmtplXkXkSFBiu0Y7NATd+buVWFZhTei9ZigAJNCggJkMpnqrwwAANAKkFjAPSHR1Yr+b1iC7v6x7tWgKhWWVag8J0IhbU6UffDbb3pj3reeitCrQoICFBIUoFBz4P++AhRQS7JhDjSpnQI00WbI3MRxAgAAeBuJBdwTGCQFR0nlBfaiS3tFqXPvHlq05bDWHcjThgN5OlpYXmsTtvJ4h+MA8zGvhetpZRU2lVXYlF/q2hOW9QrQcwt36J6ze3o5MgAAgKZFYgH3hcY4JBYqzdOg7nEa1DFOkmQYhg7ll2rd/uNJxnqnZMNmcUwsTOYcSTa11LUFXl2yRzeM6qr4iGBfhwIAAOAxJBZwX1islF9lJafSXIeXTSaT2sWEqV1MmCb2Pr4ClGEYysov04YDeVq+L1HvHqpSP8AqU1CBjArHFadaiuJyq176aafuO7uHr0MBAADwGBILN2VkZCgjI0NWq9XXofiO05KzKs2r9xSTyaTkmFAlx4TqzJ5t9cnboSq1ltpff/66TuqbOMjTkXqEYRiy2gyVWmwqtViPf1XYVPa//xpG9WV1P11zQAu3HrEfv75kt24a1YWnFgAAoMUgsXBTenq60tPTlZ+fr5iYlvkJe72cV4YqyW3Q6SaTSe0j22tH3g57WZmOqn1smPux+Ym+7WP047YfVbmVB08tAABAS9MyB7GjaTXiiYWz9lHtHY5b2iZ5XdpE6oJ+7RzKXl+yW8eKap/UDgAA0JzwxALuc9okz3mOhSs6RHZwON6Ru0OHiw83PiY/dOWp0fp84xYZOr4cbYlNenrRSqWPPdkr10sITVBgQKBX2gYAAHBGYgH3eeKJRaTjE4v5e+Zr/p757kTllyJOcTz++Kj08QfeuVabsDZ6eMTDGpM6xjsXAAAAqIKhUHCfm3MsJKlDVIf6K6FBjpQc0cylM2WxubZZIQAAgDtILOA+Dzyx6JXQSybVvGM1Gu9oyVFtPbbV12EAAIBWgMQC7vNAYpEckazbB92usKCWsxKUv1iVtcrXIQAAgFaAORZwnwcmb0vSjX1vVFrvNBVbit0OyR9VVFRo/nfzNeGsCZrxxRZ9se6g/bXw4EB9fccZig03u32df674pz7b8Zn9eHXWaqX1TnO7XQAAgLqQWMB9zk8syvIlm1VqxIpE5gCzYkJa5n4glgCLwgLCFB0crbvO7K8vf8s9sa9FqfTesqMe2ddieLvhDonFmsNrZBiGTCaGmgEAAO9hKBTc5zx5W2rUcKjWpEubSF00wHElLE/tazEoyXHH8pyyHO3K2+V2uwAAAHUhsYD7nJ9YSCQWLpgy7mQFVHmIUFxu1edrD7jdbkpEipLCkxzKVh1mngUAAPAuEgu4LzhCCnAaVdfIeRatSZc2kZrYO9mh7PcjhW63azKZqj21WJ212u12AQAA6kJiAfeZTB5ZGao16p4c5XB8IKfEI+0ObjvY4ZjEAgAAeBuJBTzDA5vktUbtYx2X1z2YW+qRdp2fWBwsOqjMwkyPtA0AAFATEgt4Bk8sGsU5sTiQWyLDMNxut2tsV0UHRzuUMc8CAAB4E4kFPKPaXhYkFq5oH+eYWBSWVSi/tMLtdgNMARrUlnkWAACg6ZBYVDFv3jx1795dp5xyil5++WVfh9O8VHtikeuTMJqb5JjQamWemmfBBG4AANCUSCz+p6KiQlOnTtUPP/ygNWvW6J///Keys7N9HVbzwVCoRgkJClTbqBCHsoO53kksduTtUC4JHwAA8BISi/9Zvny5evfurfbt2ysyMlLnnHOOvvvuO1+H1XwwebvRnIdDHfBQYtErvpdCAx2fiKw+zFMLAADgHS0msVi8eLHOP/98paSkyGQy6dNPP61WJyMjQyeddJJCQ0M1fPhwLV++3P7awYMH1b79iZ2Q27dvrwMH3N+srNXgiUWjpVRbGcoziYU50Kx+bfo5lDEcCgAAeEuLSSyKiorUv39/ZWRk1Pj63LlzNXXqVE2fPl2rV69W//79NXHiRB0+fLiJI22hqk3ezvVFFM1SB6fEYr+HEguphnkWPLEAAABeElR/lebhnHPO0TnnnFPr6//6179000036frrr5ckvfDCC/ryyy/16quvatq0aUpJSXF4QnHgwAENGzas1vbKyspUVlZmP87Pz5ckWSwWWSwWd2+n2TGZIx06k1GSq4pW+HOoS2W/cO4fSVHBDscHcoo91of6J/R3ON6cvVn5JfkKCwqr5Qz4Sm39A6hEH0F96COoj7f7RotJLOpSXl6uVatW6f7777eXBQQEaPz48Vq6dKkkadiwYdqwYYMOHDigmJgYff3113rooYdqbXPWrFmaOXNmtfKFCxcqPDzc8zfh59rkb9NpVY7L8g7r26++8lk8/mz+/PkOx5nHTJIC7cc7D+XqKw/97MqMMgUoQDbZJEkVRoVe/vJldTV39Uj78Dzn/gE4o4+gPvQR1Ka4uNir7beKxOLo0aOyWq1KSkpyKE9KStKWLVskSUFBQXrqqac0duxY2Ww2/eUvf1FCQkKtbd5///2aOnWq/Tg/P1+pqakaO3Zsnee1VKYDydKOf9qPQ4xSnXvuuT6MyP9YLBbNnz9fEyZMkNlstpd3zizQS1uX2o/zLSadedbZCgnyzEjFj7/5WBuPbbQfmzubdW4/fjf+prb+AVSij6A+9BHUx9srnraKxMJVF1xwgS644AKX6oaEhCgkJKRaudlsbp1/zFGJDocma5nMskrm6vs0tHbOfaRTm6hqdbKLK9QpIcIj1xucPNghsfjt6G+ts482E6323xC4jD6C+tBHUBtv94sWM3m7LomJiQoMDFRWVpZDeVZWlpKTk91qOyMjQ7169dLQoUPdaqfZc14VSmICt4uiQ4MUGeKY43tqkzxJGpw02OF43dF1stgYfwsAADyrVSQWwcHBGjx4sBYsWGAvs9lsWrBggUaMGOFW2+np6dq0aZNWrFjhbpjNW42JBUvOusJkMql9rHf2spCkQW0dV4YqqSjR5uzNHmsfAABAakGJRWFhodauXau1a9dKknbt2qW1a9dq7969kqSpU6fqpZde0uuvv67NmzfrlltuUVFRkX2VKLgp0CyZnYbusEmey1JiHYeMeTKxiA2NVdcYx8na7GcBAAA8rcXMsVi5cqXGjh1rP66cWJ2WlqY5c+Zo0qRJOnLkiB5++GEdOnRIAwYM0DfffFNtQjfcEBojWYpOHPPEwmXOu297apO8SoOSBmlH3g778arDq3SdrvPoNQAAQOvWYhKLMWPGyDCMOutMmTJFU6ZM8eh1MzIylJGRIavV6tF2m6WwWKng4Ilj5li4zHn3bU8+sZCOJxYfbPvAfrzm8BrZDJsCTC3moSUAAPAx3lW4iTkWVTjPs+CJhcuc51gczC31aPuD2zpO4M4ry9PO3J0evQYAAGjdWswTC/iB0FjHY+ZYuKymyds2m6GAAJNH2m8X2U7tItopsyjTXjZn4xz1TezrkfbhPqvNqg1lG1S0vUiBAYH1n9DKBQYEakCbATo57mRfhwIA+B8SC3hOtScWuT4JozlynmNRXmFTdlG52kRV3yulsQYlDdKXO7+0H3+24zN9tuMzj7UPz/hixRe+DqHZCDQF6rkzn9Pp7U/3dSgAADEUym3sY1FFWKzjMUOhXNY2KlRBTk8nPD7PwmnZWaC5sxpWzd0y19dhAAD+h8TCTcyxqIInFo0WGGBScozjkrOeXhlqXMdxigquvss30Jztzt/t6xAAAP/DUCh4DpO33ZISG6b9VXbc9uTu25KUGJao58Y9p/e2vqfDxYc92jbcZxiGjmUfU3xCvEwmz8ytaYlKKkq0KXuT/fhA4QFZbVbmpQCAHyCxgOcwedstHWLDtLzKsaeHQknH51kMSmJIlD+yWCz66quvdO74c2U2m30djt86WnJUY98/sWeRxWbR4eLDahfZzodRAQAkhkLBk3hi4RbnCdzeSCyA5i4hNEFhQY5/K/sL9/soGgBAVSQWbmLydhXVJm/n+iKKZst5kzxPz7EAWgKTyaQOUR0cyvYV7PNRNACAqkgs3MTk7SqqPbHIl2w238TSDNW0lwWA6lIjUx2OSSwAwD+QWMBznOdYyJDK8n0RSbPk/MQit9iiorIKH0UD+C/nJxb7CxgKBQD+gMQCnuP8xEJinkUDOD+xkBgOBdQkNYonFgDgj0gs4DkhUZLJqUuRWLgsLDhQ8RHBDmUMhwKqq/bEgsnbAOAXSCzgOSYTm+S5iXkWQP2cn1jkleUpv5xhlwDgayQWbmJVKCcsOeuWlFjv7r4NtAQpESkKcHo6yjwLAPA9Egs3sSqUEzbJc0v72HCHY0/vvg20BOZAs5LDkx3KmGcBAL5HYgHP4omFW6o/sSj1USSAf2NlKADwPyQW8Cw2yXNLB3bfBlzCylAA4H9ILOBZPLFwi/NeFofyS1VhZZNBwBkrQwGA/yGxgGcxx8ItzqtCWW2GsgrKfBQN4L8YCgUA/ofEAp7FEwu3xEcEK9Ts+GfJylBAdamRjkOhMosyZbFZfBQNAEAisXAby806qTbHgsSiIUwmU7XhUKwMBVTn/MTCZtiUWZjpo2gAABKJhdtYbtaJ81AoJm83GJvkAfWLCYlRVHCUQxnDoQDAt0gs4FkMhXIbiQXgGlaGAgD/QmIBz2LyttucEwvmWAA16xDJylAA4E9ILOBZzk8sKkqkClY1agjmWACu4YkFAPgXEgt4lvPkbYnhUA3UvoZN8gzD8FE0gP9iyVkA8C8mg3csHpGfn6+YmBgdPXpUCQkJvg7HdyrKpEfaOpbFd5UCg30Tjx8xZKigoEBRUVEyyVRrPYvVpp1HixzKTm4bqUBT7eeg+XO1f+CEZYFW3RhWaj8ON6Rfi8Jb7M+PPoL60EdQzY3fSyGR9sPs7GwlJiYqLy9P0dHRHr9ckMdbROsWFCIFhR0fAlXp2A7fxeNHTJKiJam07npmSd2dnyUe9UpI8COu9g+c0CEoUEptbz8uNkk52dsUb2uZu9XTR1Af+giqMZr230OGQsHzolN8HQGAViC5wqogp4fu+8x8XgYAvkJi4SY2yKvBaVN8HQGAViBQUkpFhUPZ/iASCwDwFeZYeAhzLJwc2SplbZRE96pUYbVqzZo1GjhwoIICA+us+9ovu7VqT479eFyPtrpkUPs6zkBz15D+gRP+vP1N/ZJ/YrhlespY/bndaB9G5D30EdSHPoJqel4gBZrth8yxQPPUpvvxL9gZFosO7g7RgF7nSmZznXWz9m3RvF0n3izZApN1SZ/B3g4RPtSQ/oETOhRulqokFvvDoqU+l/owIu+hj6A+9BH4GkOhAD/UPjbU4fhALjPxgJqwlwUA+A8SC8APVdvLgk3ygBqx+zYA+A8SC8APOe++fbSwTKUWq4+iAfyX8yZ5h4sPq7SCJ3wA4AskFoAfau+UWEhSZh5vlgBnzomFJB0oPOCDSAAAJBaAH4oKNSsq1HFthYO5DIcCnEWYIxQfGu9Qtr+A4VAA4AskFoCf6hAX7nC8O7vIR5EA/o0J3ADgH0gsAD/VJTHC4XjnERILoCbOw6GYwA0AvkFiAfiprm0cE4sdRwp9FAng33hiAQD+gcQC8FNd20Y6HJNYADWrtuQscywAwCdILAA/1bWNY2KxP6eEJWeBGjg/sdhfsF82w+ajaACg9SKxcFNGRoZ69eqloUOH+joUtDCdneZYGAYTuIGaOM+xKLeV60jxER9FAwCtF4mFm9LT07Vp0yatWLHC16GghYkICVJKTKhD2Y7DJBaAszZhbRQSGOJQxjwLAGh6JBaAH+vShnkWQH1MJlP1eRasDAUATY7EAvBjrAwFuIaVoQDA90gsAD/GylCAa6rtZcHKUADQ5EgsAD/mvDLUjsNFstkMH0UD+C8SCwDwPRILwI85JxYlFqsO5Zf6KBrAf1VbcpY5FgDQ5EgsAD+WFB2iiOBAhzKGQwHVOT+xOFZ6TEUWVlEDgKZEYgH4MZPJVH2exWESC8BZ+8j2MsnkUMZwKABoWkG+DgBA3bq2idS6/Xn24x1H+BQWcBYSGKK24W2VVZxlL3t02aNKCE3wYVSeZTNsOlR0SD/89IMCTHwuiOroI3D22KjHFBYU1mTXI7EA/BxLzgKu6RDVwSGxWHN4jQ+j8Z5N+zb5OgT4OfoIKv3d9vcmvR7pLODnnCdw7+SJBVCjk6JP8nUIANCqkVgAfs55jsWh/FIVllX4KBrAf53f9XwFmXgQDwC+wr/AgJ/rlBCuAJNUdfuKnUcK1a9DrM9iAvzR4KTBevu8t7X04FKV28p9HY7H2aw2bdu+Td1O6aaAQD4XRHX0ETgLDgxu0uuRWAB+LiQoUKnx4dqTXWwv20FiAdSoV0Iv9Uro5eswvMJiseirfV/p3L7nymw2+zoc+CH6CHyNdBZoBmragRsAAMCfkFgAzQArQwEAAH/ndmLRpUsXnXrqqS7XHzVqlLp27eruZb3i4osvVlxcnC677DJfhwI4qPbEgsQCAAD4GbcTi927d2vv3r0u19+/f792797t7mW94o477tAbb7zh6zCAapxXhtp9tFgVVpuPogEAAKiuyYdCVVRUKCDAP0dgjRkzRlFRUb4OA6jG+YlFudWm/TklPooGAACguiZ9h19SUqLDhw836s374sWLdf755yslJUUmk0mffvpptToZGRk66aSTFBoaquHDh2v58uUeiBrwvfiIYMWFO67wwXAoAADgTxq83OzevXurDWUqLy/XTz/9JMMwajzHMAzl5ubq7bfflsViUd++fRscaFFRkfr3768bbrhBl1xySbXX586dq6lTp+qFF17Q8OHDNXv2bE2cOFFbt25V27ZtJUkDBgxQRUX1jcW+++47paSkNCiesrIylZWV2Y/z8/MlHV/qzWKxNKgttA6V/aKx/aNLYoRW7c21H287lK8zTo73RGjwA+72D7R89BHUhz6C+ni7b5iM2rKBWsycOVN/+9vf7MeGYchkMrl0bmXdN998U5MnT25YpFWYTCZ98sknuuiii+xlw4cP19ChQ/Xcc89Jkmw2m1JTU3Xbbbdp2rRpLre9aNEiPffcc/rwww/rrDdjxgzNnDmzWvk777yj8PBwl68HuOrdHQH69fCJh4yntrXpqq7MswAAAK4pLi7W5MmTlZeXp+joaI+336gN8qrmIiaTqdYnFVXrREdHq0+fPvrzn//sVlJRk/Lycq1atUr333+/vSwgIEDjx4/X0qVLPXqtSvfff7+mTp1qP87Pz1dqaqrGjh2rhIQEr1wTzZvFYtH8+fM1YcKERm1cdPDn3fr1223244qweJ177jBPhggfcrd/oOWjj6A+9BHUJzs726vtNzixmD59uqZPn24/DggIUHJysg4ePOjRwBri6NGjslqtSkpKcihPSkrSli1bXG5n/Pjx+u2331RUVKQOHTrogw8+0IgRI2qsGxISopCQkGrlZrOZP2bUqbF95JQkx08Wdh4tpq+1QPwbgvrQR1Af+ghq4+1+0agnFlVde+21io2N9UAovvf99983+JyMjAxlZGTIarV6ISLgBOclZ48VletYUbniI4J9FBEAAMAJbicWc+bM8UAY7klMTFRgYKCysrIcyrOyspScnOzVa6enpys9PV35+fmKiYnx6rXQuqXGhckcaJLFemLo4c4jhYqPYAI3AADwPY8uN2uz2bRixQp9+OGHTbrRXHBwsAYPHqwFCxY4xLJgwYJahzIBzU1QYIBOSohwKGPJWQAA4C/cfmJR6dlnn9Ujjzyio0eP2suuvfZa+/c5OTkaNWqUKioq9OOPP1abD1GfwsJC/f777/bjXbt2ae3atYqPj1fHjh01depUpaWlaciQIRo2bJhmz56toqIiXX/99e7fHOAnuraJ1PbDJ5KJHUeKfBgNAADACR55YpGenq4777xTR44cUVRUVI3Lz8bFxWnQoEHavn27PvjggwZfY+XKlRo4cKAGDhwoSZo6daoGDhyohx9+WJI0adIkPfnkk3r44Yc1YMAArV27Vt98802DE5iGysjIUK9evTR06FCvXgeQpK5tnZ5YHOaJBQAA8A9uJxbffPONnn/+eUVGRuqTTz5Rbm6u2rRpU2PdyZMnyzCMRk2SHjNmjAzDqPZVdY7HlClTtGfPHpWVlWnZsmUaPnx4Y2/LZenp6dq0aZNWrFjh9WsBXds4TuBmKBQAAPAXbicWL7zwgkwmk/72t7/pwgsvrLNu5XyH9evXu3tZoFVyTiz2HitWWQUrkgEAAN9zO7FYtmyZJOmGG26ot25MTIyio6N16NAhdy8LtEpd2jgOhbIZ0p7sYh9FAwAAcILbicWxY8cUExOjqKgo1y4YECCbzebuZf0GcyzQlKJCzUqKdtyYkXkWAADAH7idWERHRys/P18Wi6XeuseOHVNeXp4SExPdvazfYI4FmhrzLAAAgD9yO7Ho27evDMOwD4mqy7vvvivDMDRkyBB3Lwu0Ws6JxU6WnAUAAH7A7cTisssuk2EYmjFjRp1DnH777Tc9+OCDMplMuuqqq9y9LNBqOc+z4IkFAADwB25vkHfTTTfpP//5jxYuXKgJEyborrvuktV6fJWa7du3a/fu3friiy/0yiuvqKSkRCNGjNDll1/uduBAa+X8xGL9gTwNf6z6Es59UmJ0z8Tu6tkuuqlCAwAArZjbiYXZbNaXX36ps88+WwsXLtSiRYvsr/Xo0cP+vWEY6tu3rz766KMaN9BrrjIyMpSRkWFPpgBv69rWMbGwGVJWflm1eln5h3Ugt0Rf3zGqRf3NAQAA/+SRnbc7deqkVatWaebMmerYsWO1TexSUlI0Y8YMLVmyRMnJyZ64pN9g8jaaWrvoUMWGm12qu+VQgXYdZQ4GAADwPrefWFQKDw/XQw89pIceekgHDx7UwYMHZbValZycrE6dOnnqMkCrFxBg0l3ju2n65xtdqr9qT466OA2fAgAA8DS3E4tx48bJZDLpv//9r7p27SpJSklJUUpKitvBAahZ2mknaVyPtjU+jXh+0Q4t3ZltP169N0eXD0ltyvAAAEAr5HZi8fPPP8tsNtuTCgBNIzU+XKnx4dXKtx4qcEgsVu3JacqwAABAK+X2HIukpCQFBwd7IhYAHjCoU5zD8basQuWV1L+BJQAAgDvcTizOOOMM5efna/v27Z6Ip9nJyMhQr169NHToUF+HAkiS+rSPVnCg45/2mr08tQAAAN7ldmJxzz33KCgoSHfffbcMw/BETM0Kq0LB34QEBapvhxiHstUMhwIAAF7mdmIxcOBAvfvuu1q0aJFGjhypTz75RFlZWa0yyQD8xWCn4VCr9+b6JhAAANBquD15OzAw0P79smXLdNlll9V7jslkUkVFhbuXBlCLQR0dE4s1e3NktRkKDGCjPAAA4B1uP7Fw3gzP1S8A3jOoU6zDcVG5VVsPFfgmGAAA0Cq4/cRi4cKFnogDgAe1jQpVx/hw7T1WbC9btTdHvVKifRgVAABoydxOLEaPHu2JOAB42OBOcQ6Jxeo9Obrm1E4+jAgAALRkbg+Fau1Ybhb+ynk/CzbKAwAA3kRi4SaWm4W/Guw0gXvvsWIdLij1UTQAAKClc3so1OLFixtUPzQ0VLGxseratavDilIAPKt7cpQiggNVVG61l63ek6uz+yT7MCoAANBSuZ1YjBkzRiZTw5ewDA0N1Zlnnqm//OUvOv30090NA4CTwACTBnSM1S+/Z9vLVu/NIbEAAABe4ZGhUI1ZbrakpETz5s3TmDFj9PTTT3siDABOnIdDMc8CAAB4i9uJhc1m0+eff664uDj16NFDr7zyinbs2KHS0lKVlpZqx44deuWVV9SzZ0/Fx8dr3rx5OnbsmL799luNGzdONptNd999t1avXu2J+wFQhfME7vX781RWYa2lNgAAQOO5nVisWbNGl19+uQYNGqQ1a9bo+uuvV+fOnRUcHKzg4GB17txZ119/vdasWaOBAwfqsssu0759+zRhwgR9//33Ovfcc2Wz2ZSRkeGJ+wFQxUCnJxblVps2Hsz3UTQAAKAlczuxmDVrlsrLy5WRkaGQkJBa6wUHB+u5555TaWmpZs2aZS+fOXOmpIZPAgdQv5gws7olRTqUrWY4FAAA8AK3E4uff/5Z0dHR6tatW711u3fvrpiYGC1atMheNnjwYIWGhurgwYPuhgKgBoPZzwIAADQBtxOLnJwclZWVyTCMeuvabDaVlpYqJ8fxjU1YWFijVpbyB2yQB383yGk41Mo9OS79vQIAADSE24lFSkqKysrK9MUXX9Rbd968eSorK1NKSoq9rDLRaNOmjbuh+AQb5MHfOT+xOFJQpv05JT6KBgAAtFRuJxYXXHCBDMPQTTfdpCVLltRab+nSpbr55ptlMpl0wQUX2Ms3btwoSerSpYu7oQCoQefECMWFmx3KVu9lOBQAAPAstzfIe/DBB/X+++8rMzNTZ5xxhs444wyNHj1aKSkpMplMOnjwoBYtWqTFixfLZrOpXbt2evDBB+3nv/XWW5KkM888091QANTAZDJpcKc4fb/5sL1s1Z4cXTigvQ+jAgAALY3biUVCQoIWLlyoyy67TBs2bNCiRYv0448/OtSpHM/du3dvffjhh0pISLC/duGFF2rMmDE69dRT3Q0FQC0G1ZBYAAAAeJLbiYUkdevWTatXr9bcuXP1wQcfaPXq1Tpy5IgkqU2bNho0aJAuu+wyTZo0SWaz45CMMWPGeCIEAHVwnsC9OTNfRWUVigjxyD8BAAAAnkksJCkoKEhXX321rr76ak81CcBD+neIVWCASVbb8aeHNkP6bV+uTjs50ceRAQCAloKPK4FWICw4UL1TorVuf5697LGvN+ukhAivXzslNkxXDeuozonevxYAAPAdjycWR44c0Z49e1RcXKwzzjjD080DaKRBHeMcEosNB/K14UB+k1z76w2Z+vqOMxTJ0CsAAFost5ebrfT5559r0KBBSk5O1vDhwzVu3DiH13NycnT22Wfr7LPPVl5eXi2tAPAW5/0smtK+YyX6dsMhn10fAAB4n0cSi3/84x+6+OKLtXbtWhmGYf+qKi4uTmFhYZo/f74+/PBDT1wWQAOc1TtJXdv4bjjSvHUHfXZtAADgfW6PS/j111/117/+VUFBQXriiSd0zTXXqHfv3jp8+HC1un/84x/12Wefaf78+frTn/7k7qUBNEBIUKA++PNp+mp9prLyS71+vb3HivXZ2hPJxE/bjyq3uFyx4cFevzYAAGh6bicWTz/9tCTp/vvv1x133FFn3dGjR0uS1qxZ4+5lATRCfESw/nhqpya5VmFZhb7ZcEhlFTZJUoXN0LcbD2nS0I5Ncn0AANC03B4K9csvv0iSpkyZUm/dxMRERURE6ODBljMkIiMjQ7169dLQoUN9HQrgVyJDgjS2e1uHsnnrMn0UDQAA8Da3E4vDhw8rKipKiYmurYcfEhKi8vJydy/rN9LT07Vp0yatWLHC16EAfucP/ds5HC/Zka3swjIfRQMAALzJ7cQiIiJCxcXFslqt9dYtLCxUbm6u4uPj3b0sgGZgXI+2CjMH2o+tNkNfszoUAAAtktuJRffu3WW1WrVu3bp663766aey2WwaMGCAu5cF0AyEBwfpzJ7Ow6FazlBIAABwgtuJxQUXXCDDMDRr1qw66+3fv1/Tpk2TyWTSpZde6u5lATQTf+jnOBxq2a5jOtwEq1IBAICm5XZiMWXKFLVv314fffSRrr32Wm3YsMH+msVi0fbt2/Wvf/1LgwcP1sGDB9WtWzelpaW5e1kAzcSY7m0VEXxiOJRhSF+tZxI3AAAtjduJRWRkpL744gslJibqrbfeUv/+/e17WISGhqpHjx669957deTIEaWkpOjTTz+V2Wx2O3AAzUOoOVATeiU5lLE6FAAALY9Hdt4eMGCAfvvtN11//fUKCQlx2H3bMAyZzWZdd911Wrlypbp37+6JSwJoRv7QL8XheOWeHB3MLfFRNAAAwBvc3iCvUnJysl555RX95z//0apVq3Tw4EFZrVYlJydr6NChCg8Pl3R8eNSLL77o0r4XAFqGUd0SFRUapILSCnvZV+szdeOoLj6MCgAAeJLHEotKISEhOu2006qVW61WvfLKK3r00Ud14MABEgugFQkJCtTE3sn6cNV+e9kX60gsAABoSdwaClVcXKzffvtNq1evVk5OTo11DMPQnDlz1K1bN91yyy3at2+fDMNw57IAmiHn1aF+25erfceKfRQNAADwtEYlFnl5eUpLS1NCQoIGDRqkoUOHqk2bNrrkkkuUmXliUuaiRYvUr18//elPf9KuXbskSRdeeKGWLVvmmegBNBsjT05UXLjjwg1fsjoUAAAtRoMTi4qKCk2YMEFvvfWWysrK7BO0bTabPvvsM02YMEHl5eV66qmnNH78eG3cuFEBAQGaPHmy1q1bp08++URDhgzxxr0A8GPmwACd3SfZoYzN8gAAaDkaPMfi9ddf18qVKyVJ48aN09lnny3DMPTtt9/qhx9+0ObNm/V///d/ev3112UymXTttdfq4YcfVpcujKUGWrs/9EvRu8v32Y83HMjX7qNFOikxwodRAQAAT2hwYvHBBx/IZDLppptu0gsvvGAvv/fee3XzzTfr5Zdf1htvvKG4uDh9/PHHGj16tEcDBtB8De8cr8TIYB0tLLeXzVt3UFPGneLDqAAAgCc0eCjU+vXrJUkPPvhgtdceeugh+/f/+Mc/SCoAOAiqcTgU8ywAAGgJGpxYZGdnKzw8XB06dKj2Wmpqqn2/igsuuMD96AC0OM6b5W05VKADbJYHAECz1+DEory8XFFRUbW+XvlaUlJS46PygX379mnMmDHq1auX+vXrpw8++MDXIQEt0tCT4hUT5rg61K87sn0UDQAA8BS39rFoSYKCgjR79mxt2rRJ3333ne68804VFRX5OiygxQkMMGl453iHsl93klgAANDckVj8T7t27TRgwABJUnJyshITE3Xs2DHfBgW0UCO6JjgcLyWxAACg2WtUYpGVlaXAwMAavw4fPixJtb4eGBiooKAGL0alxYsX6/zzz1dKSopMJpM+/fTTanUyMjJ00kknKTQ0VMOHD9fy5csbc3tatWqVrFarUlNTG3U+gLqd2sUxsdifU8Iu3AAANHMNf4cvyTAMT8dRr6KiIvXv31833HCDLrnkkmqvz507V1OnTtULL7yg4cOHa/bs2Zo4caK2bt2qtm3bSpIGDBigioqKaud+9913Skk5PqH02LFjuvbaa/XSSy/VGU9ZWZnKysrsx/n5+ZIki8Uii8XS6PtEy1XZL+gfUpf4UMWFm5VTfOJn8cv2w7p0UHsfRuVb9A/Uhz6C+tBHUB9v9w2T0cAsYebMmR658PTp0xt9rslk0ieffKKLLrrIXjZ8+HANHTpUzz33nCTJZrMpNTVVt912m6ZNm+ZSu2VlZZowYYJuuukmXXPNNXXWnTFjRo0/i3feece+MhaA2r2yNUDrjp14aDqsjU1Xn2zzYUQAALRsxcXFmjx5svLy8hQdHe3x9hucWPgD58SivLxc4eHh+vDDDx2SjbS0NOXm5uqzzz6rt03DMDR58mR1795dM2bMqLd+TU8sUlNTlZmZqYSEhDrORGtlsVg0f/58TZgwQWazuf4TWrg3f92rv325xX7cPjZUi+4+w4cR+Rb9A/Whj6A+9BHUJzs7W+3atfNaYtGooVD+5ujRo7JardWWuE1KStKWLVtqOcvRL7/8orlz56pfv372+Rtvvvmm+vbtW2P9kJAQhYSEVCs3m838MaNO9JHjRp7SVtKJv88DuaU6VGBRanzrfuJH/0B96COoD30EtfF2v2gRiYUnnH766bLZGIYBNJVuSZGKjwjWsaJye9nSndmtPrEAAKC5ahHLzSYmJiowMFBZWVkO5VlZWUpOTvbqtTMyMtSrVy8NHTrUq9cBWhqTyaRTuzjtZ8FGeQAANFstIrEIDg7W4MGDtWDBAnuZzWbTggULNGLECK9eOz09XZs2bdKKFSu8eh2gJRrhtOzsrzuzfbLqHAAAcF+zGQpVWFio33//3X68a9curV27VvHx8erYsaOmTp2qtLQ0DRkyRMOGDdPs2bNVVFSk66+/3odRA6iL834WB/NKtfdYsTolRPgoIgAA0FjNJrFYuXKlxo4daz+eOnWqpOMrP82ZM0eTJk3SkSNH9PDDD+vQoUMaMGCAvvnmm2oTugH4j5PbRioxMlhHC0/Ms/h1ZzaJBQAAzVCzSSzGjBlT7xCJKVOmaMqUKU0U0XEZGRnKyMiQ1Wpt0usCLYHJZNLwLgn6cl2mvWzpjmxNGtrRh1EBAIDGaBFzLHyJORaAe6rPszjGPAsAAJohEgsAPuU8z+JQfql2Zxf7KBoAANBYJBYAfKprmwi1iXLcbPLXnSw7CwBAc0Ni4Sb2sQDcc3w/i+rLzgIAgOaFxMJNzLEA3Oe8Ud7SHexnAQBAc0NiAcDnnCdwHy4o066jRT6KBgAANAaJBQCf65wYobZO8yyWMhwKAIBmhcQCgM+ZTCaN6Fp92VkAANB8kFi4icnbgGc4T+BmngUAAM0LiYWbmLwNeIbzPIujhWXacYR5FgAANBckFgD8QqeEcCVHhzqUMc8CAIDmg8QCgF+oeZ4FiQUAAM0FiQUAv1HTfhY2G/MsAABoDkgsAPiN07omOhwfKyrXugN5PooGAAA0BImFm1gVCvCc1PhwdW0T4VC2aOthH0UDAAAagsTCTawKBXjWmO5tHY4Xbj3io0gAAEBDkFgA8CtjnRKLdftzlV1Y5qNoAACAq0gsAPiVoZ3jFB4caD82DGnxdp5aAADg70gsAPiVkKDAapO4FzEcCgAAv0diAcDvjOnexuF48bYjsrLsLAAAfo3EAoDfcU4scoot+m1/rm+CAQAALiGxcBPLzQKe1yEuXKe0jXQoYzgUAAD+jcTCTSw3C3iH81ML9rMAAMC/kVgA8EvO+1ms25+nIwUsOwsAgL8isQDgl4acFKeIKsvOSscncQMAAP9EYgHAL4UEBeq0k52WnSWxAADAb5FYAPBbzrtws+wsAAD+i8QCgN9ynsCdV2LR2n05PooGAADUhcQCgN9KiQ1TtySWnQUAoDkgsQDg15yHQy1k2VkAAPwSiYWb2CAP8K7RTsOhNhzI1+GCUh9FAwAAakNi4SY2yAO8a0ineEWGBDmU/chwKAAA/A6JBQC/FhwUoJEnJziUsewsAAD+h8QCgN9z3oX7p21HVGG1+SgaAABQExILAH7PednZ/NIKrdmX65tgAABAjUgsAPi9djFh6pEc5VC2cAurQwEA4E9ILAA0C87Dob7fnOWjSAAAQE1ILAA0C2f2dEwstmUVaseRQh9FAwAAnJFYAGgWBneMU5uoEIeybzYc8lE0AADAGYkFgGYhIMCks3snO5R9tT7TR9EAAABnJBYAmo1z+jgmFhsP5mtvdrGPogEAAFWRWABoNoZ1jldcuNmh7JuNPLUAAMAfkFgAaDaCAgM0sdpwKOZZAADgD0gs3JSRkaFevXpp6NChvg4FaBXOdhoOtXZfrjLzSnwUDQAAqERi4ab09HRt2rRJK1as8HUoQKtwWtdERYcGOZSxOhQAAL5HYgGgWQkOCtD4XkkOZV8zHAoAAJ8jsQDQ7JzTp53D8Yo9x3S4oNRH0QAAAInEAkAzNOqUREUEB9qPDUP6dmOWDyMCAAAkFgCanVBzoMb1dBwO9c0Glp0FAMCXSCwANEvnOq0O9evOYzpWVO6jaAAAAIkFgGZpdPc2CjWf+CfMajM0fxOTuAEA8BUSCwDNUnhwkMZ0a+tQ9jXLzgIA4DMkFgCarXP6Og6H+uX3o8orsfgoGgAAWjcSCwDN1rgebRUceOKfMYvV0ILNrA4FAIAvkFgAaLaiQs0adUqiQxnDoQAA8I0gXwcAAO44p287Ldhy2H7847YjenbBdplMjvUiQoI0oVeSOsSFN3GEAAC0DiQWAJq1CT2TFBRgUoXNkCSVV9j01PxtNdZ9esF2fXPHGUqOCW3KEAEAaBUYCgWgWYsJN+u0kxPrrygpt9ii75mDAQCAV5BYAGj2rh95kst1s/JLvRcIAACtGEOhADR7Y7u31fNXD9LXGw6pxGJ1eG3roQLtPVZsP85md24AALyCxOJ/cnNzNX78eFVUVKiiokJ33HGHbrrpJl+HBcBF5/Rtp3P6tqtW/q/52/TMgu324+zCsqYMCwCAVoPE4n+ioqK0ePFihYeHq6ioSH369NEll1yihIQEX4cGwA0JEcEOx8d4YgEAgFcwx+J/AgMDFR5+fBnKsrIyGYYhwzB8HBUAdyVEOiYW2YUkFgAAeEOzSSwWL16s888/XykpKTKZTPr000+r1cnIyNBJJ52k0NBQDR8+XMuXL2/QNXJzc9W/f3916NBB9957rxITXVtpBoD/ind6YsEcCwAAvKPZDIUqKipS//79dcMNN+iSSy6p9vrcuXM1depUvfDCCxo+fLhmz56tiRMnauvWrWrbtq0kacCAAaqoqKh27nfffaeUlBTFxsbqt99+U1ZWli655BJddtllSkpKqjGesrIylZWdGKudn58vSbJYLLJYLJ64ZbQwlf2C/tG0YkMCHY7zSiwqLi2TOdC/Plehf6A+9BHUhz6C+ni7b5iMZjjex2Qy6ZNPPtFFF11kLxs+fLiGDh2q5557TpJks9mUmpqq2267TdOmTWvwNW699VaNGzdOl112WY2vz5gxQzNnzqxW/s4779iHVAHwvQKL9OBKx89Q/ja4QjHBtZwAAEALVVxcrMmTJysvL0/R0dEeb7/ZPLGoS3l5uVatWqX777/fXhYQEKDx48dr6dKlLrWRlZWl8PBwRUVFKS8vT4sXL9Ytt9xSa/37779fU6dOtR/n5+crNTVVY8eOZcI3amSxWDR//nxNmDBBZrPZ1+G0GlaboYdWzVfVj1AGnjpKPZKjfBdUDegfqA99BPWhj6A+2dnZXm2/RSQWR48eldVqrTZsKSkpSVu2bHGpjT179ujmm2+2T9q+7bbb1Ldv31rrh4SEKCQkpFq52Wzmjxl1oo80LbOkuPBgh9Wg8kptfvs7oH+gPvQR1Ic+gtp4u1+0iMTCE4YNG6a1a9c2+LyMjAxlZGTIarXWXxmAT8RHOCYW2UXsZQEAgKf51+zFRkpMTFRgYKCysrIcyrOyspScnOzVa6enp2vTpk1asWKFV68DoPGc97JgyVkAADyvRSQWwcHBGjx4sBYsWGAvs9lsWrBggUaMGOHDyAD4A+e9LNgkDwAAz2s2Q6EKCwv1+++/24937dqltWvXKj4+Xh07dtTUqVOVlpamIUOGaNiwYZo9e7aKiop0/fXX+zBqAP4gIcJxPhRDoQAA8Lxmk1isXLlSY8eOtR9XrsiUlpamOXPmaNKkSTpy5IgefvhhHTp0SAMGDNA333xT6z4UnsIcC8D/Vdskj6FQAAB4XLNJLMaMGaP6ttyYMmWKpkyZ0kQRHZeenq709HTl5+crJiamSa8NwDWJkey+DQCAt7WIORYAUJd4p6FQzLEAAMDzSCwAtHjOk7ePFjLHAgAATyOxcFNGRoZ69eqloUOH+joUALVwXm62oLRC5RU2H0UDAEDLRGLhJvaxAPyf8+RtScopZjgUAACeRGIBoMWLDQ9WgMmxjOFQAAB4FokFgBYvMMCkuHA2yQMAwJtILAC0Cs4TuNnLAgAAzyKxcBOTt4HmodomeTyxAADAo0gs3MTkbaB5SIh03MsimzkWAAB4FIkFgFbBeclZ5lgAAOBZJBYAWoUEp923jzLHAgAAjyKxANAqxEc6P7FgKBQAAJ5EYgGgVUhk8jYAAF5FYuEmVoUCmgfnVaGOMRQKAACPIrFwE6tCAc2D86pQBWUVKquw+igaAABaHhILAK2C86pQEitDAQDgSSQWAFqFmDCzAgNMDmXsvg0AgOeQWABoFQICTIoLZwI3AADeQmIBoNWovkkeS84CAOApJBYAWo0Ep70sGAoFAIDnkFi4ieVmgebDeclZhkIBAOA5JBZuYrlZoPlIdFpyNruQoVAAAHgKiQWAVqPaJnk8sQAAwGNILAC0Gs5zLI4yxwIAAI8hsQDQalRfFYrEAgAATyGxANBqJDDHAgAAryGxANBqOM+xKCq3qtRi9VE0AAC0LCQWAFqNxIiQamUsOQsAgGeQWABoNaLDghQUYHIoO8YEbgAAPILEwk1skAc0HyaTSXHVNsljngUAAJ5AYuEmNsgDmhfnlaGyeWIBAIBHkFgAaFWc97JgyVkAADyDxAJAq5LgNIH7KEOhAADwCBILAK2K85KzTN4GAMAzSCwAtCqJkc6Tt0ksAADwBBILAK1KvNNQKBILAAA8g8QCQKviPHk7u5A5FgAAeAKJBYBWxXm5WVaFAgDAM0gsALQqCZGOQ6GKy60qKbf6KBoAAFoOEgsArYrzqlASu28DAOAJJBYAWpXo0CCZA00OZQyHAgDAfSQWAFoVk8lU7alFNntZAADgNhILN2VkZKhXr14aOnSor0MB4CKWnAUAwPNILNyUnp6uTZs2acWKFb4OBYCLqm2Sx5KzAAC4jcQCQKvjPBSKORYAALiPxAJAq5PgNBTqKHMsAABwG4kFgFbHefftYyw3CwCA20gsALQ6zrtvM3kbAAD3kVgAaHVYbhYAAM8jsQDQ6iREOi83y1AoAADcRWIBoNVxHgpVarGpuLzCR9EAANAykFgAaHWcJ29LDIcCAMBdJBYAWp3IkCAFBzr+88cEbgAA3ENiAaDVMZlMNWySxzwLAADcQWIBoFVyHg7FJnkAALiHxAJAq1T9iQWJBQAA7iCxANAqJTovOVvIUCgAANxBYgGgVaq2SR5PLAAAcAuJBYBWyXmOBcvNAgDgHhILJ8XFxerUqZPuueceX4cCwIucN8ljjgUAAO4hsXDy6KOP6tRTT/V1GAC8LCGCORYAAHgSiUUV27dv15YtW3TOOef4OhQAXhbvPBSqqFyGYfgoGgAAmr9mk1gsXrxY559/vlJSUmQymfTpp59Wq5ORkaGTTjpJoaGhGj58uJYvX96ga9xzzz2aNWuWhyIG4M8SnZ5YlFXYVFRu9VE0AAA0f0G+DsBVRUVF6t+/v2644QZdcskl1V6fO3eupk6dqhdeeEHDhw/X7NmzNXHiRG3dulVt27aVJA0YMEAVFRXVzv3uu++0YsUKdevWTd26ddOSJUvqjaesrExlZSeGTuTn50uSLBaLLBZLY28TLVhlv6B/+IeoEFO1snFPLlJA9eImU1oaqFkbf/Rom31SonX3hFN0cttIj7aLpse/IagPfQT18XbfMBnN8Nm/yWTSJ598oosuusheNnz4cA0dOlTPPfecJMlmsyk1NVW33Xabpk2bVm+b999/v9566y0FBgaqsLBQFotFd999tx5++OEa68+YMUMzZ86sVv7OO+8oPDy8cTcGoMkYhnTvskBZDB9mEk2kU6Shu/pYZWr5twoAqENxcbEmT56svLw8RUdHe7z9FpFYlJeXKzw8XB9++KFDspGWlqbc3Fx99tlnDWp/zpw52rBhg5588sla69T0xCI1NVWZmZlKSEho0PXQOlgsFs2fP18TJkyQ2Wz2dTiQdO6zv2j74SJfh9EkFk4dpQ5xYb4OA27g3xDUhz6C+mRnZ6tdu3ZeSyyazVCouhw9elRWq1VJSUkO5UlJSdqyZYtXrhkSEqKQkJBq5WazmT9m1Ik+4j+mjDtFd81dK1uz+3il4dbsz1fntp7/nwiaHv+GoD70EdTG2/2iRSQWnnbdddf5OgQATeDCAe01rHO8tmUV+nxFKKvVqhXLV2josKEKDAx0u73nF+3Qsl3H7Mcrdh/TJYM6uN0uAAC1aRGJRWJiogIDA5WVleVQnpWVpeTkZK9eOyMjQxkZGbJaWU0GaI7axYSpXYzvhwhZLBYVbjd0ximJHvlEaXtWoUNisbzK9wAAeEOzWW62LsHBwRo8eLAWLFhgL7PZbFqwYIFGjBjh1Wunp6dr06ZNWrFihVevAwANMbRzvMPxjiNFbAIIAPCqZvPEorCwUL///rv9eNeuXVq7dq3i4+PVsWNHTZ06VWlpaRoyZIiGDRum2bNnq6ioSNdff70PowYA3+idEq0wc6BKLCeepq7YnaOz+3j3KS4AoPVqNonFypUrNXbsWPvx1KlTJR1f+WnOnDmaNGmSjhw5oocffliHDh3SgAED9M0331Sb0A0ArYE5MECDOsXql9+z7WUrdh8jsQAAeE2zSSzGjBlT7+TKKVOmaMqUKU0U0XHMsQDgr4aeFF8tsQAAwFtaxBwLX2KOBQB/Newkx3kWGw/mq6iswkfRAABaOhILAGihBnaMU1DAie22rTZDq/fm+DAiAEBLRmIBAC1UWHCg+rSPcShbwbKzAAAvIbFwU0ZGhnr16qWhQ4f6OhQAqGaY07Kzy5lnAQDwEhILNzHHAoA/G+o0z2LN3lyVV9h8FA0AoCUjsQCAFmxIpziH47IKm9YfyPNRNACAlozEAgBasLiIYHVLinQoY9lZAIA3kFgAQAvnPBxqJYkFAMALSCzcxORtAP7OeQL3it05stnq3nAUAICGIrFwE5O3Afg75ycWeSUWbT9c6KNoAAAtFYkFALRwKbFhah8b5lDGsrMAAE8jsQCAVqDacCg2ygMAeBiJBQC0As7DoVbsPibDYJ4FAMBzSCwAoBUY1tlxP4vMvFLtzynxUTQAgJaIxMJNrAoFoDno2iZS8RHBDmXsZwEA8CQSCzexKhSA5sBkMlXbhZvEAgDgSSQWANBKOE/gXs4EbgCAB5FYAEAr4TyBe8eRImUXlvkoGgBASxPk6wAAAE2jd0q0woMDVVxutZfNW5ep4V3i6zgL/qLCUqGDRdLWQwUKMvO/b1RHH4GzU9pGKTDA1GTXo9cBQCsRFBigQR3j9PPvR+1l0z/f6MOI0HBBenzdUl8HAb9GH8EJ62ecpahQc5Ndj6FQANCKOA+HAgDAU0gs3MRyswCak/G92srUdE/FAQCtCEOh3JSenq709HTl5+crJibG1+EAQJ16p8Tor+f21Ms/7dLhglJfh4MGMgxDJjJD1IE+Al8isQCAVubGUV1046guvg4DDWSxWPTVV1/p3HPPldncdGOm0XzQR+BrDIUCAAAA4DYSCwAAAABuI7EAAAAA4DYSCwAAAABuI7EAAAAA4DYSCwAAAABuI7FwExvkAQAAACQWbktPT9emTZu0YsUKX4cCAAAA+AyJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAAAAAcFuQrwNoKQzDkCQVFBTIbDb7OBr4I4vFouLiYuXn59NHUA39A/Whj6A+9BHUp6CgQNKJ962eRmLhpoyMDGVkZKisrEyS1LlzZx9HBAAAANQuOztbMTExHm/XZHgrZWllcnNzFRcXp71793rlF4XmLz8/X6mpqdq3b5+io6N9HQ78DP0D9aGPoD70EdQnLy9PHTt2VE5OjmJjYz3ePk8sPCQg4Ph0lZiYGP6YUafo6Gj6CGpF/0B96COoD30E9al83+rxdr3SKgAAAIBWhcQCAAAAgNtILDwkJCRE06dPV0hIiK9DgZ+ij6Au9A/Uhz6C+tBHUB9v9xEmbwMAAABwG08sAAAAALiNxAIAAACA20gsAAAAALiNxAIAAACA20gsPCAjI0MnnXSSQkNDNXz4cC1fvtzXIcFHZs2apaFDhyoqKkpt27bVRRddpK1btzrUKS0tVXp6uhISEhQZGalLL71UWVlZPooYvvSPf/xDJpNJd955p72M/oEDBw7oj3/8oxISEhQWFqa+fftq5cqV9tcNw9DDDz+sdu3aKSwsTOPHj9f27dt9GDGaktVq1UMPPaTOnTsrLCxMXbt21d///ndVXYuHPtK6LF68WOeff75SUlJkMpn06aefOrzuSn84duyYrr76akVHRys2NlZ/+tOfVFhY2OBYSCzcNHfuXE2dOlXTp0/X6tWr1b9/f02cOFGHDx/2dWjwgR9//FHp6en69ddfNX/+fFksFp111lkqKiqy17nrrrv0xRdf6IMPPtCPP/6ogwcP6pJLLvFh1PCFFStW6MUXX1S/fv0cyukfrVtOTo5Gjhwps9msr7/+Wps2bdJTTz2luLg4e50nnnhCzzzzjF544QUtW7ZMERERmjhxokpLS30YOZrK448/rueff17PPfecNm/erMcff1xPPPGEnn32WXsd+kjrUlRUpP79+ysjI6PG113pD1dffbU2btyo+fPna968eVq8eLFuvvnmhgdjwC3Dhg0z0tPT7cdWq9VISUkxZs2a5cOo4C8OHz5sSDJ+/PFHwzAMIzc31zCbzcYHH3xgr7N582ZDkrF06VJfhYkmVlBQYJxyyinG/PnzjdGjRxt33HGHYRj0DxjGfffdZ5x++um1vm6z2Yzk5GTjn//8p70sNzfXCAkJMd59992mCBE+dt555xk33HCDQ9kll1xiXH311YZh0EdaO0nGJ598Yj92pT9s2rTJkGSsWLHCXufrr782TCaTceDAgQZdnycWbigvL9eqVas0fvx4e1lAQIDGjx+vpUuX+jAy+Iu8vDxJUnx8vCRp1apVslgsDn2mR48e6tixI32mFUlPT9d5553n0A8k+gekzz//XEOGDNHll1+utm3bauDAgXrppZfsr+/atUuHDh1y6CMxMTEaPnw4faSVOO2007RgwQJt27ZNkvTbb7/p559/1jnnnCOJPgJHrvSHpUuXKjY2VkOGDLHXGT9+vAICArRs2bIGXS/IM2G3TkePHpXValVSUpJDeVJSkrZs2eKjqOAvbDab7rzzTo0cOVJ9+vSRJB06dEjBwcGKjY11qJuUlKRDhw75IEo0tffee0+rV6/WihUrqr1G/8DOnTv1/PPPa+rUqXrggQe0YsUK3X777QoODlZaWpq9H9T0/x36SOswbdo05efnq0ePHgoMDJTVatWjjz6qq6++WpLoI3DgSn84dOiQ2rZt6/B6UFCQ4uPjG9xnSCwAL0lPT9eGDRv0888/+zoU+Il9+/bpjjvu0Pz58xUaGurrcOCHbDabhgwZoscee0ySNHDgQG3YsEEvvPCC0tLSfBwd/MH777+vt99+W++884569+6ttWvX6s4771RKSgp9BD7HUCg3JCYmKjAwsNqKLVlZWUpOTvZRVPAHU6ZM0bx587Rw4UJ16NDBXp6cnKzy8nLl5uY61KfPtA6rVq3S4cOHNWjQIAUFBSkoKEg//vijnnnmGQUFBSkpKYn+0cq1a9dOvXr1cijr2bOn9u7dK0n2fsD/d1qve++9V9OmTdOVV16pvn376pprrtFdd92lWbNmSaKPwJEr/SE5ObnaokMVFRU6duxYg/sMiYUbgoODNXjwYC1YsMBeZrPZtGDBAo0YMcKHkcFXDMPQlClT9Mknn+iHH35Q586dHV4fPHiwzGazQ5/ZunWr9u7dS59pBc4880ytX79ea9eutX8NGTJEV199tf17+kfrNnLkyGpLVG/btk2dOnWSJHXu3FnJyckOfSQ/P1/Lli2jj7QSxcXFCghwfPsWGBgom80miT4CR670hxEjRig3N1erVq2y1/nhhx9ks9k0fPjwhl3QrannMN577z0jJCTEmDNnjrFp0ybj5ptvNmJjY41Dhw75OjT4wC233GLExMQYixYtMjIzM+1fxcXF9jp//vOfjY4dOxo//PCDsXLlSmPEiBHGiBEjfBg1fKnqqlCGQf9o7ZYvX24EBQUZjz76qLF9+3bj7bffNsLDw4233nrLXucf//iHERsba3z22WfGunXrjAsvvNDo3LmzUVJS4sPI0VTS0tKM9u3bG/PmzTN27dplfPzxx0ZiYqLxl7/8xV6HPtK6FBQUGGvWrDHWrFljSDL+9a9/GWvWrDH27NljGIZr/eHss882Bg4caCxbtsz4+eefjVNOOcW46qqrGhwLiYUHPPvss0bHjh2N4OBgY9iwYcavv/7q65DgI5Jq/HrttdfsdUpKSoxbb73ViIuLM8LDw42LL77YyMzM9F3Q8CnnxIL+gS+++MLo06ePERISYvTo0cP473//6/C6zWYzHnroISMpKckICQkxzjzzTGPr1q0+ihZNLT8/37jjjjuMjh07GqGhoUaXLl2Mv/71r0ZZWZm9Dn2kdVm4cGGN7z3S0tIMw3CtP2RnZxtXXXWVERkZaURHRxvXX3+9UVBQ0OBYTIZRZatGAAAAAGgE5lgAAAAAcBuJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAAAAAcBuJBQAAAAC3kVgAgBeMGTNGJpNJM2bM8HUoPlVcXKyHHnpIPXv2VFhYmEwmk0wmk9auXevr0LxmxowZMplMGjNmjK9DaZTrrrtOJpNJ1113na9DAdDMkFgAaDKVb7hMJpPCw8N18ODBWuvu3r3bXnfRokVNFyQ8atKkSXrkkUe0ZcsWmUwmJSUlKSkpSWaz2dehtTqLFi3SjBkzNGfOHF+HAqCFIrEA4BMlJSWaOXOmr8OAF23ZskXz5s2TJM2dO1fFxcU6dOiQDh06pN69e/s4utZn0aJFmjlzZr2JRbt27dS9e3e1a9euaQID0GKQWADwmVdffVXbtm3zdRjwkvXr10uSEhISdMUVV/g4Grhq1qxZ2rJli2bNmuXrUAA0MyQWAJpcamqq+vXrp4qKCj3wwAO+DgdeUlxcLEmKjIz0cSQAgKZAYgGgyQUEBNg/Df3oo4+0fPnyBp1fdf7F7t27a6130kknyWQyVRv64Xz+nj17dNNNN6ljx44KDQ1V165d9eCDD6qoqMh+zoYNG/THP/5RqampCg0N1SmnnKJHHnlEFoul3njLy8v1j3/8Q/369VNERITi4uI0YcIEff311/Weu2HDBt1888065ZRTFB4ersjISPXr109//etfdfTo0RrPcZ48/NFHH+mss85S27ZtFRAQ0OAJ5aWlpZo9e7ZOO+00xcXFKTQ0VJ06ddK1115b4yTsyutXTv7ds2eP/efd2EnBv/zyi/74xz+qU6dOCg0NVUxMjIYNG6bHH39chYWFDnUtFosSExNlMpn0zDPP1Nnuq6++KpPJpOjoaHsiJEmHDh3Ss88+qwsvvFA9e/ZUTEyMwsLCdPLJJ+vGG2/Uxo0bG3wPkmuT+uua/J2Tk6NXXnlFV1xxhfr27av4+Hj772Py5Mn69ddfq51T2d8rhx7++OOPDr8P578RVyZvL1q0SJdffrnat2+vkJAQJSYm6swzz9Rrr70mq9Xq0n0tWLBA5513ntq0aaPQ0FD17NlTM2fOVGlpaa3X/fbbb3XJJZeoQ4cOCg4OVnR0tLp06aKzzjpLTz75pI4dO1bruQCagAEATWT69OmGJKNTp06GYRjG6NGjDUnG2LFjq9XdtWuXIcmQZCxcuLDW13bt2lXr9Tp16mRIMl577bVaz//oo4+M2NhYQ5IRHR1tBAYG2l8bNWqUUV5ebsybN88IDw83JBkxMTGGyWSy15k0aVKN1668t/vvv98YNWqUIckICgqyX6vya/r06bXG//jjjxsBAQH2uuHh4UZwcLD9uF27dsbq1atr/TmPHj3amDp1qiHJMJlMRlxcnBEYGFjnNZ3t37/f6NOnj/2aZrPZiImJsR8HBAQYzzzzjMM5//znP42kpCQjOjraXicpKcn+dfvtt7t8favVatx+++0OP7PIyEiH31P37t2N3bt3O5yXnp5uSDKGDBlSZ/tjxowxJBnXXXedQ3laWpq9/aCgICM+Pt4ICgqyl4WEhBgffvhhjW1W/fk7q+wXdf0O6jq/8jVJRmBgoBEXF2eEhITYy0wmk/H00087nLN3714jKSnJiIiIsP8Oq/4+kpKSjPfee6/avaelpdUY31133eVwvdjYWIffx7hx44z8/Pw67+uJJ54wTCaT/fyqf1Njx441Kioqqp0/c+ZMh34QHh5uREZGOpQ5/1sBoGmRWABoMs6JxdKlS+1vCL7++muHuk2VWMTGxhpnnnmmsXHjRsMwDKO4uNh45pln7G+UHnzwQSMmJsaYNGmS/c1rQUGB8de//tXexvz586tdu/INZExMjBESEmK88MILRklJiWEYx9/oXXbZZfbzP/vss2rnv/zyy/Y30Y8++qiRmZlpGIZhVFRUGCtXrjTGjRtnSDI6dOhgFBQU1PhzrnzTdd999xmHDx82DMMwSktLq70Jr01FRYUxfPhw+3289dZbRllZmWEYhrFjxw7jD3/4g/3N5VdffVXt/Ndee83h990YDz74oCHJaNu2rZGRkWFkZ2cbhmEY5eXlxsKFC42BAwcakoxBgwYZVqvVft6yZcvsP9/NmzfX2PaePXvsb2h/+OEHh9f+/ve/G//85z+N9evXGxaLxTCM40nOhg0bjKuvvtqQZERERBgHDhyo1q43E4sXX3zRmD59urFy5Ur778Jmsxk7d+407rjjDsNkMhmBgYH1Jpx1qSuxePbZZ+0/15tvvtneLwsLC41///vf9uSrpoS78vqxsbFGQECAcf/99xtHjhwxDMMw8vLyjIcfftje9iuvvOJw7u7du+1J9tSpUx1+7rm5ucZPP/1k3HrrrcbKlSvrvDcA3kViAaDJOCcWhmEYF198sSHJGDBggGGz2ezlTZVY9O7d2ygtLa127jXXXGOvM2HCBIfYKlU+ifjTn/5U7bXKN5A1vUkyjONvUs844wx7DFXl5+fbn2x88803Nd6bxWIxBg8ebEgy/v3vfzu8VvVT7alTp9Z4vivee+89ezvffvttjTFUJh59+vSp9rq7icWuXbuMwMBAIywszFi7dm2NdfLz840OHToYkoxPPvnE4bXu3bvbnxrV5LHHHjMkGR07dqzx91uX8847z5Bk/P3vf6/2mjcTi/pUPqmpqU+6m1gUFxcb8fHxhiTjqquuqvHcZ555xt5nnN/kV+2Xtd3/JZdcYkgyxo8f71A+d+5cQ5LRrVu3OmMH4FvMsQDgU4899pgCAwO1du1avfvuu01+/bvuukshISHVyidOnGj/ftq0aTKZTLXWWbduXa3tp6am6vrrr69WHhAQoAcffFCStHHjRvsKStLxORG5ubkaOHCgQxxVBQUF6aqrrpJ0fNx5TQICAnTffffVGlt95s6dK0kaMWKEzjrrrBpjmD59uqTjc0Gq3oMnzJkzR1arVWeffbb69+9fY52oqChddNFFkqr/HK655hpJ0ttvvy3DMKqd++abb0qSrr766hp/v3U577zzJEk///xzg87zNm/GNX/+fPschtrmiNx66632ZWrfeeedGuuEhITonnvuqfG1Cy+8UFL1v6nY2FhJUkFBgcPcJwD+hcQCgE/16NHD/sb7oYcecmkytCcNGzasxvKkpCT790OHDq2zTk5OTq3tV07WrcmoUaMUFBQkSVq5cqW9/JdffpEkbd68WcnJybV+/e1vf5N0fHJ0TU4++WS1bdu21tjqUxnT+PHja60zduxYBQYGVrsHT6j8OXz33Xd1/hxee+01SdV/Dtdcc41MJpP27t2rH3/80eG1VatWafPmzZKka6+9tsbr//bbb7r11lvVr18/RUdHKyAgwD7Z+dZbb5Uk7d+/36P37IqdO3fqnnvu0eDBgxUbG6vAwEB7XOeee67X4qr8/aampqpbt2411gkMDNS4ceMc6jvr3bt3rSuFpaSkSFK1SdjDhg1TYmKiMjMzNXz4cD333HPasmVLjQkjAN8J8nUAADBjxgy9/fbb2rlzp1544QXddtttTXbtqKioGssr3/C7UqeuZKh9+/a1vhYaGqqEhARlZWXp8OHD9vLKHclLS0vrXCGnUtXVjKpyJ6mQZI+pvntITEysdg+eUPlzKCoqculTauefQ8eOHTV69GgtWrRIb775psMqS5VPK4YOHaoePXpUa+u5557THXfcIZvNJkkymUyKiYmxP90qKSlRfn5+k396/sknn+iqq65SWVmZvSw6OlqhoaEymUwqLy9XTk6OV+JypT9IUocOHRzqO6vt70k68TdVUVHhUB4bG6t3331XkydP1saNG+3/RsTExOiMM87QFVdcoUmTJrGjO+BjPLEA4HPt27e3v1F45JFHqi0f2tpULtc5adIkGcfnwtX5VduSu5VPEpqryp/Dfffd59LPYdGiRdXaqHwa8eGHH6qkpETS8TetlcPuKodLVbV582bdeeedstlsuvzyy7V8+XKVlpYqJyfHvnP4v/71L0lq0k/Ms7Ozdd1116msrEzjxo3TokWLVFxcrLy8PGVlZenQoUP64IMPmiyepjZ+/Hjt2rVLb7zxhtLS0nTKKacoLy9PX3zxha655hoNHDhQBw4c8HWYQKtGYgHAL0ybNk1xcXE6fPiwnnrqqTrrVn2aUNcn+nl5eR6Lr7HqeqNTVlam7OxsSY5PF5KTkyXVPsSpqVTGVNewmtLS0hrvwRM88XO47LLLFBYWpvz8fH322WeSjg+tOnz4sMxms32eSlUffvihrFarevbsqffee09Dhw5VcHCwQ51Dhw41Kp7KvtuYfvvVV18pPz9fcXFx+uKLLzR69GiFhYV5JC5XuNIfqr7u6f4gSREREbrmmms0Z84cbdu2Tfv379fjjz+u0NBQhycZAHyDxAKAX4iLi9O0adMkSU899ZSOHDlSZ91K+/btq7HOtm3blJub69EYG+PHH3+s9VPtn376yT7kY8iQIfbykSNHSjo+DyAzM9P7QdaiMqYFCxbUWmfRokX2e6htLkpjVf4cvv/+e5eGhNWk6uTuyuFPlf8955xzlJiYWO2cyj7Vv39/BQTU/L/J77//vlHxVPbd2vqtJC1btqzG8spzunfvrvDw8AbHVXkvjX3KUtkf9u/fr23bttVYx2q1auHChZI83x9q0r59e/3lL3/R3XffLen4BHMAvkNiAcBv3HbbberQoYMKCgr097//vdZ6ERER6tq1q6TjKyjV5NFHH/VKjA21d+9evf7669XKbTabHnvsMUlSr1691LdvX/trl19+uWJjY2WxWDR16tQ63wjabDavJVBXXnmlJGnp0qX67rvvqr1eUVFhn0Dep08f9enTx6PXv+GGGxQUFKSjR4/aV5+qTXl5ea1D6CqHQ3333Xfavn27/clFbZO2Y2JiJEnr16+v8Wf/9ddf1zjsyhWVq1t9++23Nc6D+OGHH7R06dI649q2bVuNidbatWtrXYlJOj4XQ1Kj+8uECROUkJAgqfZVoV588UX73JiangY1VtU5JTWpfHJTWyIIoGnwFwjAb4SFhdnfsHzxxRd11q180/Lqq6/qP//5j338/L59+3TjjTdq7ty5tX6q25RiYmJ0yy236KWXXrK/Gdy3b5+uuuoq+ye7jzzyiMM5sbGxmj17tiTpvffe03nnnadly5bZJxLbbDZt3rxZTz31lHr37q158+Z5JfZLL71Uw4cPlyRdccUVeuedd+wT1Xft2qVLL73U/ib4iSee8Pj1u3btqoceesje/rXXXqsNGzbYX6+oqNDatWv1t7/9TSeffLLWrl1bYzsTJkxQcnKyKioqNHnyZJWUlCguLk5/+MMfaqx/9tlnSzq+DHB6erp9haKioiK9+OKLuuyyy+xvsBvqiiuuUEBAgLKzs3XVVVfZhw2VlJTo9ddf18UXX6z4+Pgazz3rrLMUEBCgY8eO6eqrr7YPsysvL9f777+vs846q86J0ZWJ38aNG7VkyZIGx1717/Pdd9/Vn//8Z2VlZUk6PnH+mWee0Z133inp+PygwYMHN/gatXn88cd1zjnn6M0333QYilVWVqb3339f//znPyWdWG4XgI802Y4ZAFq9mjbIc1ZRUWH06NHDvpGWatggzzCO737dq1cve52AgAD7pnJms9l49913Xdogr7YN9hYuXGivU5u6NoCr3Ajt/vvvN04//XR7XHFxcQ739uCDD9ba/vPPP28EBwfb64aEhBgJCQmG2Wx2aOOtt95yOM+dDdac7d+/3+jdu7f9WsHBwfafc+XP/emnn67xXE/svG2z2YyHHnrIvkO2JCMsLMxISEiw745e+fXzzz/X2s7UqVMd6v7f//1fnde98sorHerHxsbarzd48GD7DtQ13Vt9P/+qO0zrf7uaV+5YfdFFF9l3G6/p/Pvuu6/auZX9oXPnzsbbb79da7+1WCz2TQMlGXFxcUanTp2MTp06GR988IG9Xl07bxuGYdx11132NkwmkxEXF2ePX5IxduxYIz8/v8E/F8Oo/e+u6uZ6lX0gPj7eoV/07NnTvhM4AN/giQUAvxIYGGgfIlSXyMhI/fzzz5o6dao6d+6soKAgmc1m+6folcN4fC04OFgLFizQY489pu7du6usrEwxMTE688wz9eWXX9Y55OvPf/6ztm7dqnvuuUf9+/dXSEiIcnNzFRkZqSFDhui2227T/PnzPTrkxFn79u21cuVK/etf/9Kpp56qsLAwFRcXKzU1Vddcc41WrVql22+/3WvXN5lM+tvf/qZ169bp1ltvVc+ePRUYGKi8vDzFxcXptNNO07333qslS5bY52TUxHnYU23DoCq9/fbbmj17tvr166eQkBBZrVb17dtXs2bN0i+//FLrPgyumDlzpt58802deuqpioiIkNVq1YABA/TCCy/o448/rnM1r3/84x964403NGzYMIWFhclisejkk0/WAw88oDVr1tj3gahJUFCQFixYoBtvvFGdO3dWUVGR9uzZoz179jRoJbZ//etf+uGHH3TppZcqKSlJhYWFioqK0tixY/Xqq69q/vz5dT45aYybb75Z//3vf3XVVVepT58+Cg8Pt09kHzVqlGbPnq3Vq1fbJ/wD8A2TYbC7DAAAAAD38MQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4jcQCAAAAgNtILAAAAAC4LcjXAQANZRiGLBaLbDabr0MBAMArAgICZDabZTKZfB0K4DISCzQbVqtVR48eVUFBgSwWi6/DAQDAq8xms6KiopSYmKjAwEBfhwPUy2QYhuHrIID6WK1W7du3T2VlZYqJiVFkZKQCAwP5JAcA0OIYhiGr1arCwkLl5eUpJCREqampJBfweyQWaBaysrKUm5urjh07KiwszNfhAADQJEpKSrR3717FxsYqKSnJ1+EAdWLyNvyeYRgqKChQTEwMSQUAoFUJCwtTdHS0CgoKxGfB8HckFvB7FotFFotFkZGRvg4FAIAmFxUVZf9/IeDPSCzg9ypXf2JsKQCgNar8/x+rIcLfkVig2WCiNgCgNeL/f2guSCwAAAAAuI3EAgAAAIDbSCwAAAAAuI3EAmghTjrpJJlMpmpfkZGR6t+/v+6//35lZ2fX287tt99uP/eLL77weJwzZsyQyWTSmDFj6qy3aNEiexw1Wb16tW644QadfPLJCgsLU3h4uDp16qSRI0fqnnvu0fz58z0eO5pWbX3a+WvOnDnVzqlahuZr+/btmjJlinr16qWIiAiFhoaqQ4cOGjp0qKZMmaKPPvrI7WtU/ltT379JnjJnzhyZTCZdd911TXI9oCkF+ToAAJ41cuRInXzyyZKOryBy8OBBLVmyRP/4xz/0xhtv6KefflKXLl1qPLesrExvv/22/fjVV1/V+eef3yRxN8Szzz6rO++8UzabTe3bt9fYsWMVFxenI0eOaPXq1VqyZIkWLVqkCRMm+DpUeEDVPl2Tul5D8/Xxxx9r8uTJKisrU0JCgkaOHKk2bdooJydHa9euVUZGht577z1deumlvg4VwP+QWAAtzI033ljtk7BDhw5p9OjR2rZtm/7yl7/oww8/rPHcTz75RMeOHVNKSooyMzM1b948ZWVl+dVur+vWrbMnFf/+97912223OSxFbLPZ9PPPP+vnn3/2YZTwpJr6NFq2rKwspaWlqaysTHfffbceeeQRhYaGOtRZtWpVrf+WAfANhkIBrUBycrLuvfdeSdKCBQtqrffKK69Iku644w6NHj1aFRUVeuONN5okRld98MEHstlsGjFihO68885q+5sEBATojDPO0AMPPOCjCAG4a968eSosLFRKSoqefPLJakmFJA0ePFizZs3yQXQAasMTCzRrNpuhnOJyX4fhtrjwYAUEeHed8uTkZElSRUVFja/v3r1bCxYsUFBQkK699lqlpKRo0aJFevXVV+1JSVU7d+7UoEGDlJ+fry+//FLnnHOOw+sHDx7UgAEDdOTIEb333nuaNGmSR+4jKytLktS2bVuPtOd3bDap5Jivo3BPWLwUwOdWDWUzbMoty/V1GG6JDYlVgMn9333l33mbNm0afO6xY8f05JNP6rPPPtOuXbsUGBiobt26adKkSbrtttsUFhZW67nFxcV65JFH9P7772v//v2Kj4/XOeeco7/97W9q3759jeds2bJFjz/+uH744QcdOnRIERERGjhwoP7v//5PV1xxRYPjB5ozEgs0aznF5Rr8yPe+DsNtqx4cr4TIEK9eY/ny5ZKk3r171/j6q6++KsMwdO655yo5OVmXXnqppkyZoi1btmjJkiU67bTTHOp36dJFr776qi699FJde+21WrNmjf6/vfuPibqO/wD+hDsQDsyTxY8SIuYcGC7HWaDpVPALY+HMWU4E5wXGzIwoyzF3VFuxfkDZXORMBAfOBC0HFgcZCCTyQ/c9foygMjBFSD0kYHAg3PH+/uHuEu9Q9z2UX8/H9tnYfd7vz+d1n93xudfn/Xm/Pp6engAAg8GAyMhIaLVavPHGG+OWVADAU089BeD2yEtjYyMWLVo0btueFAa6gNT5Ex2FdXa3AE6PT3QUU073rW6syl010WFYpXxTOVwcXKzejvF73tjYiJKSEqxZs+aB+rW2tiIkJASXL1+Gq6srXnzxRQwPD6O0tBSJiYnIzc1FcXEx5s6da9Z3aGgIa9asQUNDA1avXg2FQoGKigpkZmZCrVbj119/xYIFC0b1KSgowCuvvILBwUH4+vpiw4YNuHHjBsrLy3HmzBn8/PPPppFgopmAl5SIprGRkRG0t7cjLS0NKSkpkEgkSEpKstjOWEUnNjYWAODo6IjIyEgAGPPEuGHDBiQkJKCzsxORkZGm0RCVSoWzZ89CoVBg79694/qelEolZs+ejb6+PgQEBCAiIgIpKSkoLi5GT0/PuO6LiCbG+vXrMW/ePBgMBoSGhiI4OBjJyclQq9XQarVj9ouKisLly5exbt06XLp0Cd9//z3y8/PR0tIChUIBjUaDN99802LfqqoqdHZ2orm5GQUFBTh+/DhaW1vx8ssv49q1a9i6deuo9tevX0d0dDQGBweRnJyM5uZmHDt2DCUlJaiursbcuXORmZmJ9PT0cT02RJMZEwuiaSYmJsZUhlMikcDT0xPx8fF49tlnUV5ejrVr15r1OX36NNra2uDu7o6IiAjT69u2bQMAHD9+HH19fRb3l5qaiqCgIJw7dw4qlQpqtRopKSmYM2cOTpw4gVmzxnckxsvLC6dPn4afnx/0ej3UajUSExMRGhoKFxcXLF++HLm5ueO6T5pYd36mLS3d3d0THSKNM2dnZ5SUlCAoKAhCCJSVleH9999HREQE3NzcEBAQgAMHDsBgMJj6VFRUoKamBjKZDAcPHoSTk5NpnaurKw4ePAgAyMnJwdWrVy3u94svvjCNlgCAg4MD9u/fD5lMhurqalRWVprWpaeno6enB0uWLIFKpRpVGvu5556DSqUCcPt/JNFMwVuhiKaZu0tzdnZ2oqGhARcuXMA777yDo0ePmg3nHzp0CACwdetWSKX//Vt4/vnnsWjRIjQ2NiI3N9eUaNzJzs4Oubm5UCgUSE1NxYEDByCEQEZGxphlba21dOlS/PbbbygvL0dRUREuXLgAjUaDnp4eVFZWorKyEoWFhXyWwTRxv3Kz9vb2jzAaelR8fX1RXV2N8+fPo6CgADU1NdBoNNBqtairq8OOHTvwww8/oKCgAPb29igrKwMAhIeHW6xkt2TJEixevBj19fUoLy9HdHT0qPVyuRzr1q0z6+fm5obw8HCcPHkSZWVlpttCjftTKpUW49+2bRvee+89XLx4ER0dHXjyySetOBpEUwMTC5rS5srs8b9J/zPRYVhtrmz8fhhZKs2p1+vxwQcf4NNPP8WqVavwxx9/YPbs2QAArVaLU6dOAfjvNqg7xcbGYteuXcjMzLSYWACAt7c3vv76a0RHR6O3txc7duwYs7a88aqeEOKe7+N+621tbREcHIzg4GAAt+d1VFVV4aOPPsIvv/yCrKwsREREYOPGjffczqTj6HJ7jsJU5mj9PfZ3minlZuWz5CjfVD7RYVhFPks+7tsMDAxEYGAggNv/F2pra5GamoqcnBwUFxdj37592L17N9rb2wEAPj4+Y25r/vz5qK+vN7W9k/HhipYYt3nnSMf99ieXy+Hi4oKuri5cvXqViQXNCEwsaEqztbV56JOepwOpVIrk5GSkp6fjn3/+QXZ2Nnbu3AkAOHLkCIaHhyGVSvHaa6+Z9TXeAlVZWYnff/8dfn5+Zm2EEKMerKfRaDA8PAw7OzuztsbbE/r7++8Zs3G/zs7OD/QeJRIJVqxYgcLCQgQGBkKj0SAvL2/qJRa2tpz4PEPZ2tiOy8Tn6czGxgYKhQLHjh2DTqfDqVOnkJeXZ7Fy3cNwvwseRDMd51gQzRC2trZ4+umnAQDNzc2m140Ts/V6Pc6dO2e21NfXm7W92+effw61Wo2FCxdi2bJlqKmpQWJiosW2xvuXW1pa7nmSvnjx4qj2D0oikSAkJATA7dvAiGh6CgsLA/Df99xYDra1tXXMPsZ1lkrH/v3332P2M64zVr57kP319PSgq6trzP0RTUdMLIhmiJGREdPJ0TgKUFVVhaamJsyaNQv//vsvhBAWF7VaDeD26Mbdz8E4e/YskpKSIJPJcOLECeTm5sLFxQVfffUV8vPzzeJYuXIlpFIpuru7cebMmTHjNT5R15gkGD3IFcMrV64AGP0jgIimjv/P93z16tUAgKKiItNzMO5UW1uLuro600M079bd3Y0ff/zR7HWtVouioqJR+7jz76ysLIvxZWZmAgAWLFjAxIJmDCYWRDOAXq9HUlKS6cqecYKicQTipZdeglwuH7N/WFgYPDw8cP36dfz000+m17VaLTZv3gyDwYBvvvkG/v7+8PLyQlZWFmxsbBATE2N2FdDDw8M02fH111/Hn3/+aRbrhx9+iKqqKjg4OCAhIWHUepVKhfj4eDQ0NFh8n99++60pKTGWyyWiqWX//v1QKpWjqjAZCSFw8uRJpKWlAfjve75ixQoEBQVhYGAA27dvh06nM/Xp7OzE9u3bTe29vLws7vfdd98dNY/i1q1b2LlzJ/r7+xEYGIjly5eb1sXFxeGxxx6DRqPBJ598MioZqq2tRXJyMgA8stu0iCYDzrEgmmYOHTpkqlYCADdv3kR9fT3a2toA3P5h/sILL6Cvr89UlnWsqiZGEokEUVFR2Lt3LzIyMrB+/XqMjIxgy5YtaG9vh1KpHDW5du3atdi1axe+/PJLbNq0CRUVFaPmW+zbtw8tLS0oKyuDv78/goKC4O3tDZ1Oh/Pnz6OjowOOjo7Izs42qwak0+mQlpaGtLQ0zJs3D4sXL4ZcLje9z2vXrgEA9uzZg9DQUGsOJU0Sd3+m7xYWFoaoqKhHFxA9dMPDw8jOzkZ2djZcXV0REBCAxx9/HN3d3WhqajJdsNiyZcuoohLfffcdQkJCkJ+fDx8fH6xcudL0gLze3l4oFApTQnK3ZcuWYWRkBL6+vggJCYFMJkNFRQU6Ojrg5uaG7OzsUe3d3d1x9OhRbNy4ESqVCkeOHEFAQIDpAXl6vR4xMTGIi4t7aMeJaNIRRJPcwMCAaGpqEgMDAxMdyqTm7e0tAJgt9vb2wtvbW2zatEmUlpaa2mdkZAgAwsPDQ+j1+vtuv66uTgAQEolEtLe3i48//lgAEM8884zo7+83az80NCSWLl0qAIi3337bbL1erxdZWVkiPDxcuLm5CalUKpydnYW/v7946623xF9//WUxjs7OTpGTkyPi4uKEQqEQTzzxhJBKpcLJyUn4+fmJ2NhYUVlZ+eAHjiatsT7Tdy8JCQlmfQ4fPjxhcZP1ent7RV5enoiPjxeBgYHC09NT2NnZCUdHRzF//nyxefNmUVhYaLHvzZs3xZ49e8TChQuFg4ODkMlkIiAgQHz22WdCp9OZtS8tLRUAxKpVq0RfX5/YvXu38PHxEfb29sLd3V28+uqr4sqVK2PG2tTUJJRKpSlGuVwugoODRU5OjsX2hw8fFgCEUql84OPB8yBNFTZCsMQBTW6Dg4O4dOkSfHx84ODgMNHhEBERPVI8D9JUwTkWRERERERkNSYWRERERERkNSYWRERERERkNSYWRERERERkNSYWRERERERkNSYWRERERERkNSYWRERERERkNSYWNGXwkStERDQT8fxHUwUTC5r0pFIpAODWrVsTHAkREdGjZzz/Gc+HRJMVEwua9KRSKZycnNDV1QWDwTDR4RARET0yBoMBXV1dcHJyYmJBk56N4PgaTQE6nQ5tbW2QSCSYM2cOHB0dIZFIYGNjM9GhERERjSshBAwGAwYGBtDT04ORkRF4eXnB0dFxokMjuicmFjRlDA0N4caNG9DpdBy5ICKiaU8ikUAmk8HNzQ329vYTHQ7RfTGxoClHCIHh4WGMjIxMdChEREQPha2tLezs7DgyT1MKEwsiIiIiIrIaJ28TEREREZHVmFgQEREREZHVmFgQEREREZHVmFgQEREREZHVmFgQEREREZHVmFgQEREREZHVmFgQEREREZHV/g9ba/daksjFTQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "%matplotlib inline\n", - "\n", - "names = [\"BAxUS\", \"EI\", \"Sobol\"]\n", - "runs = [Y_baxus, Y_ei, Y_Sobol]\n", - "fig, ax = plt.subplots(figsize=(8, 6))\n", - "\n", - "for name, run in zip(names, runs):\n", - " fx = np.maximum.accumulate(run.cpu())\n", - " plt.plot(-fx + branin.optimal_value, marker=\"\", lw=3)\n", - "\n", - "plt.ylabel(\"Regret\", fontsize=18)\n", - "plt.xlabel(\"Number of evaluations\", fontsize=18)\n", - "plt.title(f\"{dim}D Embedded Branin\", fontsize=24)\n", - "plt.xlim([0, len(Y_baxus)])\n", - "plt.yscale(\"log\")\n", - "\n", - "plt.grid(True)\n", - "plt.tight_layout()\n", - "plt.legend(\n", - " names + [\"Global optimal value\"],\n", - " loc=\"lower center\",\n", - " bbox_to_anchor=(0, -0.08, 1, 1),\n", - " bbox_transform=plt.gcf().transFigure,\n", - " ncol=4,\n", - " fontsize=16,\n", - ")\n", - "plt.show()" + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAKBCAYAAADKqj3oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAACWPklEQVR4nOzdd3hUVf7H8c+k90YJhCYovYMBERUQULEiFkRXUVfdnwYbu67o2thVsa9ls+LasIu9IKsiRSxIL9JR6SQBAiE9mczc3x9sxtyZlAkzkztJ3q/nyWPumTPnfic5wfnOaTbDMAwBAAAAgA9CrA4AAAAAQONHYgEAAADAZyQWAAAAAHxGYgEAAADAZyQWAAAAAHxGYgEAAADAZyQWAAAAAHxGYgEAAADAZyQWAAAAAHxGYgEAzciOHTtks9lks9l03HHHWR2O3wTqdR133HGudnfs2OG3dv2pqf5OG5tFixa5fg8jR460OhzAEiQWQBCZNWuW639M3n5dd9119brH/PnzddVVV6lbt26KjY1VSkqK+vXrpzvuuEObN28+prg3bdqkO+64Q/369VNKSopiY2PVrVs3TZ48WfPnz/e6ndpeZ1xcnNLS0tSzZ0+deeaZuvvuu/Xhhx8qPz//mGL2xtVXX13v3wdvLNDYVX2DXNNXRESEWrVqpaFDh+rWW2/VihUrrA4bQBAIszoAAA0jPz9fN9xwg2bPnm0qLy4u1uHDh/Xzzz/rmWee0fTp03XXXXd53e5DDz2k6dOny263m8q3bdumbdu26fXXX9ekSZP0wgsvKD4+/pjjLyoqUlFRkbKysrR582Z9/fXXkqTY2Fhddtllmjp1qnr16nXM7QPwnt1u18GDB3Xw4EEtW7ZMzz77rCZOnKgXX3zRp79zAI0biQUQpHr06KHRo0fXWe/kk0+us47dbteFF16oBQsWuMr69OmjQYMGqbS0VN99952ysrJkt9t19913y26367777quz3fvuu0//+Mc/XNdt27bVqaeeqqioKK1cuVIbNmyQJL3zzjvKzc3VF198obAw7/7ZGT9+vNq1a+e6rqio0OHDh5Wbm6vVq1fr0KFDko4mHC+//LLefPNNPfzww7r99ttls9m8ukd9ePv7qNS1a1e/xwBYISMjw6OspKREe/bs0Y8//qjCwkJJ0uzZs7Vnzx4tWrTI679zAE2MASBovPrqq4YkQ5IxefJkv7V77733utqNiooy3nnnHdPjZWVlxh133OGqY7PZjEWLFtXa5jfffOOqL8m44447jLKyMlOdt99+24iKinLVmT59eq1tVm1v4cKFtdbdsGGDcfPNNxuxsbGm52VkZNT6vPqYPHlyQH4fVtq+fbvrNXXq1MnqcPwmUK+rU6dOrna3b9/ut3b9yd+vfeHChaa/qdoUFhYaU6ZMMdXPzMz0OQYAjRNrLIAmbv/+/Xrqqadc108//bQuu+wyU52IiAg99thjmjhxoiTJMIw6p0NVffyyyy7TY489poiICFOdSZMm6Z///Kfr+oknntDBgweP+bVU1atXLz377LNas2aN+vbt6yrPzMzU888/75d7AKhdbGysnnvuOZ155pmusrfeesvCiABYicQCaOJee+01FRUVSZK6deumG264oca6jz32mEJCjv6zsGTJEq1evbraesuXL9fy5cslSSEhIXrsscdqbPNPf/qTa1pQQUGB3njjjWN6HTU54YQTtHDhQnXo0MFVdvfddwd0UTcAs8svv9z1/caNGy2MBICVSCyAJu6TTz5xfV+5y1FNOnbsqNNPP911/fHHH9fZ5pgxY0xv6t3ZbDZNnjy5zjZ90aJFC7388suu67y8PGVmZvr9Pv4wcuRI1846ixYtkiRlZWVp+vTpGjhwoFJSUhQVFaUePXpo2rRprrUkVe3Zs0d33323Bg4cqOTkZMXHx2vAgAF6+OGHVVJSckxxzZ8/X5MmTdLxxx+v6OhotWrVSqeeeqr+9a9/qaysrF5tFRUV6fnnn9d5552nTp06KSYmRvHx8eratauuvfZa01ofb2RlZelvf/ub+vXrp4SEBCUkJKh37966/fbbtWXLlnq1VamsrEzPPfecTj31VLVq1UrR0dE6/vjjdfnll2vhwoXH1KbUOF57ILRt29b1feUHGdWputParFmzJB39e33mmWd02mmnqV27dgoLC5PNZlNeXp7pufv379err76qyZMnu/5WwsPDlZSUpB49euiaa67RV1995VW8DzzwgCuOBx54QNLRdVyvv/66xowZo3bt2ikyMlJt27bV+PHjNWfOnDrb9Ga72Zq2Bl6xYoWuu+46devWTTExMUpOTtaQIUP08MMP1/rzBIKO1XOxAPzO32ssSkpKjJCQEFebP/74Y53Peeihh1z1hw8fXm2dYcOGueo8/PDDdbb5ww8/uOqHhoYapaWl1daTvF9jUZ2+ffu6nt+vX796P99dINZYjBgxwvQav/rqK6NFixam1171q1OnTsaOHTtcz3/55ZeNyMjIGuv37t3b2L9/f433d5+PX15ebtxwww01tifJ6Nmzp7FlyxavXt97771ntGnTptb2JBnnnnuukZeXV2d7H330kZGUlFRjO5GRkcaLL75Yr3UGGzduNLp3715rfP/3f/9nlJeX12uNRWN47d6ozxqLSq+//rqrfocOHWqsV/Vv6tVXXzW+//57o0OHDtW+vsOHD7ue98wzzxihoaF1/mwlGaeffrpx8ODBWuO9//77XfXvv/9+Y8+ePcbJJ59ca7vXXHON4XA4vPq5jRgxoto67r8rp9Np3HfffaZ/p92/OnfubPz666+1vh4gWLBtAxCk8vLy9P7772vDhg06cuSIEhISlJaWpmHDhqlv375e7Xy0ZcsWOZ1OSUdHDgYOHFjncwYNGuT6ftOmTdXWqVpetX5Nqt7X4XBo69atpnUR/nLJJZfo559/liStX79eeXl5SkpK8vt9/GXNmjW6++67VVJSovbt22v48OGKj4/X1q1b9d1338kwDO3cuVPjxo3Tzz//rNmzZ+uPf/yjpKO7Tg0ZMkRRUVH6+eeftWzZMknShg0bdOWVV+rLL7/0KoY777xT//nPfyRJ/fr104ABA2QYhlauXOma0rJp0yadfvrpWrJkSa2jU//85z/15z//WYZhSJISEhI0bNgwtW/fXg6HQxs2bNCKFStkGIbmzJmjkSNH6ocfflBMTEy17X3xxRe69NJLVVFRIenotLvhw4erW7duKiws1OLFi5WVlaXrr79ezz77rFevd+fOnRo9erSysrJcZb1799agQYNks9m0atUqrV+/XjNnzqwxrsb62gOp6jbWp556qlfP+eWXX3TbbbfpyJEjio+P12mnnaa0tDQdPnxYixcvNtXdt2+fHA6HJKlLly7q2bOnWrVqpaioKOXl5ennn3927UK3YMECjRkzRj/99JMiIyPrjKOwsFBnnXWW1q9fr5iYGJ166qnq0KGDCgoKtHDhQu3fv1+S9Oqrr6p79+668847vXp93pg+fbr+/ve/S5IGDBigvn37Kjw8XGvWrNGqVaskSdu3b9f48eO1atUqdttC8LMyqwFgVnXEoravrl27Gi+99JLhdDprbW/27Nmu56SmpnoVw4YNG0z3cv/0Oycnx/T4pk2bvGq3VatWrue899571dap2u6xjFh89dVXpja++uqrerdRVaBHLCIjI43w8HAjMzPT45PQRYsWmXa8evjhh424uDgjISHB+OCDDzzanT17tukT3W+//bba+1f9xDQ8PNyQZLRo0aLan9Vnn31mJCQkuOqfeeaZNb6ub775xvWpa0REhPHII48YRUVFHvVWr15t9OrVy9XmjTfeWG17Bw8eNFq3bu2q17dvX2Pjxo2mOg6Hw3j00UcNm81mREREePWp/ejRo131EhMTjc8//9yjzty5c43k5GTTz0i1jFg0ltfurfqMWBQXFxt/+ctfXHXDwsKMlStX1li/6t9UWFiYIR3dya2goMBUr7y83PQ38fLLLxvPPfecsWfPnhrbXrt2rXHiiSe62v/HP/5RY92qIxaVI4CTJ082cnNzTfWKioqMSZMmuerGxcUZhYWF1bZZ3xGLiIgIw2azGccff7yxdOlSj7rvvfeeqf+99tprNb4eIFiQWABBxNvEovLr3HPPrfF/coZhGP/+979ddb2dGpSbm2u6x+bNm02Pb9y40fT4oUOHvGq36jSlmTNnVlvH18Rix44dpjZef/31erdRVdU3QT169DAyMjK8/tq6dWu1bVZNLCQZL730Uo33f/DBB011bTabMX/+/BrrX3fddXW+aa36xkaSERISYvzwww81tjlv3jxT/eru73A4jK5du7rqfPTRRzW2ZxiGkZWVZaSmprreuO/evdujzt13321KinNycmpsz/3nVNOb66+//tr0s1ywYEGNbS5evNiw2WymdqtLLBrLa68P98Siuv593XXXGWeddZYp8UxISDC+/PLLWtuu+jclybjuuut8jreqvLw813S0tm3bGhUVFdXWq5pYSDImTZpUY5slJSWm6VrvvvtutfXqm1hUJvV79+6t8d5Vk7azzjqr5hcOBAkSCyCIvPrqq0bHjh2NP//5z8bcuXON3bt3G6WlpUZRUZGxZcsW49///rfRo0cP0/+Yzj///Brn/T722GOuekOHDvUqhuLiYlP7K1asMD2+bNky0+MlJSVetTtkyBDXc5544olq6/iaWBw+fNjUxjPPPFPvNqpyfxNUn6+a4q+aWPTv37/W+//666+mNsePH19r/fnz57vqDh48uNo67m9srrzyyjp/DhMmTHDVv+yyyzwe/+STT7yOsdKMGTNcz3nyySdNjzmdTtNahbrORXBfC1HTm+tLL73UVeeSSy6pM8bLL7+8zsSisbz2+nBPLLz5uuKKK+pc12AY5r+pqKgorz+YqI8bb7zRdY9169ZVW6dqYhEREWFkZWXV2uZf//pXV/2pU6dWW+dYEgv337+7qh/ktGjRota6QDBgVyggiIwfP17bt2/XE088oXHjxql9+/aKjIxUTEyMunXrphtvvFFr167VNddc43rOZ599prfffrva9kpLS13fu58xURP3OcnuuwxVbfNY2z3WnYvqEhcXZ7ouKCgIyH385eKLL6718S5duig2Ntbr+n369HF9v337dq9iuOqqq+qsU3VXr+p2TJo7d67r+6rbjtam6u5j33//vemxTZs2KTs7W5IUFhZWZ5vh4eFe3bdq7PV93TVpLK890N566y0NHDiwXru+nXHGGUpOTq73vfbv36/PPvtMjz76qKZNm6abb75ZU6ZMcX2tWLHCVXfNmjV1tnfKKaeoTZs2tdapuk5sx44d9Y65Jpdcckmtj/fo0UPR0dGSpNzc3KD/Nw1gFRAQRLxZaBwREaGXXnpJv/zyi7777jtJ0qOPPqo//OEPHnWjoqJc35eXl3sVg/vWopX/U6uuzcp23cvqate9TX9x/59uQkKC39qePHmya3tMf6maCNQkKSnJtd1k7969a62bkpLi+t6bczxsNpuGDh1aZ71hw4a5vs/JyVFWVpZpe9ElS5a4vv/www/17bff1tnmkSNHXN/v3r3b9FjV81N69Ojh1d9F1Rirs3fvXh04cMB1fdJJJ9XZ5kknnSSbzeZakF2dxvDafVXd63c4HMrNzdXKlSs1c+ZMffbZZ9q9e7cmTJigF154odbzcioNHjy4XnFs3LhRd955p/773/+6FnLXxZsDOb3ZSKJFixau7/11Rk5iYmKtmyFIR/9Gk5OTXR/G5OfnKz4+3i/3BwKBxAJohEJCQnT//fdrzJgxko7ugLRnzx61b9/eVK/qJ/jejhK413MfBXC/Likp8SqxqNquexv+UvUNm2R+ox2MEhMT66xTdReYuupXrVu5m1BtKs/AqEvl7juVo1UHDhwwJRb79u1zfV91dyBvHT582HRdNQHo2LGjV23UVa9qmzExMWrZsmWdbSYkJCgxMdHjPIWqGsNrD4TQ0FC1bt1a48aN07hx4/TAAw9o+vTpkqRbbrlFI0eOVLdu3Wpto1WrVl7f76uvvtIFF1xQ7zNVvPmE35u/w/DwcNf3dru9XjH4ct9A3RsIFKZCAY3UaaedZvofTnVbw1b9lC0nJ8erdiunYVRyf3Netc1jbTdQb/g3b95suq5reoPVvNky2Jf6danPdqpVp2S5v1lzT+jqyz0JKiwsdH3vbYxV46vOsbTpTbuN4bU3hHvuuceV4JSVlXm1Ba63I5cHDhzQxIkTXUlFp06dNGPGDH3//ffat2+fiouL5XQ6ZRxdN6r777/f9dzK7bZr4++/K29ZdV8gkEgsgEYqPDzc9KlrdUP+3bt3d32/f/9+j/UR1dm1a5fr+5SUFI9PFVu3bm2anrFz58462ywtLTV9EtujR486n3Msli5d6vo+NDRU6enpAblPU1FcXOx13aqn/7qPclR9Y7tq1SrXGzxvv9znrFcd0fI2xrpOJz6WNr1ptzG89oYQFham0aNHu67nz5/vt7ZffPFFVwLXv39/rVu3TtOmTdPw4cPVtm1bRUdHm96ksw4BsA6JBdCIVX1DUd2nlt27d1dIyNE/c8MwvFrIWHkokyT17Nmz2jpVy6vOCfemzdDQ0DqnSByrDz74wPV9//79/brGoik6fPiw6RPymhw8eNCUlLpPI0pNTXV97z7idSyqJrNVE93auK9VqK3N4uJi5ebm1tlmQUFBnSMSjeG1N5Sq0+O8+cDBW1WTlHvuuafOv2t/3htA/ZBYAI3Ub7/9ZlpEmJaW5lEnKirKtEh10aJFdbZbdfFp1d1rqho1atQxt3nyySd7dRpufX399ddav3696/qyyy7z+z2aGsMwTKM8Nam6QDk1NdWjr1VdAP7DDz/4HFfVHXg2b97s1XSjqjFWp127dqY37T/99FOdbf7000+1LtyWGsdrbyhVR1gqP9Dwh6rrWOpaaO1wOPzyewBwbEgsgEbqlVdecX2fmJioAQMGVFtv/Pjxru/r2tVo9+7dpk8Hqz63pja/+eYb7dmzp9Z2q963pjZ9kZubq+uuu8513aJFC914441+v09T9MYbb9RZ5/XXX3d9XzWprHTuuee6vn/llVe8mnJXmx49erjWx1RUVOidd96ptb43dSRz7PV93TVpLK+9IVQdmWzXrp3f2q2apNQ1PeyTTz7xy8gRgGNDYgEECW+mpFT68ccf9eSTT7quL7vsMtOOQFVNnjzZNU1qy5Yteumll2ps984773Rt4zhs2DANGjSo2nrp6emu9QsOh0PTpk2rsc3//Oc/2rp1q6Sjc/O9OT+gPn755Redfvrppukgjz32WMB2nmpq3nzzzVpHLRYuXKgPP/zQdV01gat00UUX6YQTTpAkZWVl6aabbqrzk/5KhYWFHmsEQkJCdO2117qup0+fblqj4+6JJ57w6tyOqrG/9957Wrx4cY11f/jhhxrPh6mqsbz2QFuxYoVr+2tJpvUWvurSpYvr+88++6zGegcOHNDtt9/ut/sCqD8SCyBIfPDBBxoyZIhef/31Gqc/lJaW6tlnn9WYMWNcn4wmJSWZdkFx17p1a02dOtV1fcstt+i9994z1bHb7Zo2bZrpk88ZM2bUGm/Vx9966y1NmzbNYyvE9957T7fddpvr+i9/+YtX23x6Y9OmTbr11ls1YMAArVu3zlU+depU0xsz1Cw8PFwOh0PnnnuuvvnmG4/Hv/jiC1144YWuN8pjx46t9g1jaGionn/+eYWGhkqSXn31VZ1zzjnV7lRWac2aNbrzzjvVoUOHat8Y33777a6+kp2drbFjx3rs+uV0OvXkk0/qb3/7m1cHNY4dO9Y1amEYhsaPH2864K7S119/rfPPP19Op9O081p1GstrD6RvvvlGF1xwgaufhIWF6eabb/Zb++edd57r+xkzZujNN9/0qLNq1SqNGDFCu3fvDopdsoDminMsgCCyfPlyTZ48WWFhYerRo4d69Oih5ORkORwO7d27V0uWLDGtq4iOjtann35qWjRZnXvvvVc//PCDFixYoJKSEk2cOFEPPvigBg0apNLSUi1evFhZWVmu+tOnT9eIESNqbXP06NG655579OCDD0o6ekjfG2+8oVNPPVVRUVFauXKlac3D2LFjdffdd3v9s3jmmWdMi7ErKiqUl5en3NxcrV692mPxbXR0tB599FFNmTLF63vUx9KlS+vd9uOPPx6wwwD9IS0tTRdeeKGefvppjR07Vv3799eAAQNkGIZWrlypDRs2uOq2bdtWL774Yo1tjRkzRs8//7xuvPFGORwO/fe//9WXX36pXr16qV+/fkpISFBxcbGysrK0du3aWj+Fl44uEH/55Zc1YcIEORwOrV27Vr1799Ypp5yibt26qbCwUIsXL3bNv3/88cd166231vmaX375ZQ0bNkw5OTk6fPiwzjnnHPXp00eDBg2SzWbT6tWrXYnq1KlT9eGHH9a5GLixvPZjVV2/dzgcOnTokFauXKlff/3V9NgTTzxR48YPx2Ly5Ml68skntXXrVpWVlenKK6/Uww8/rP79+ysqKkrr1693nbbdv39/nXnmmXrsscf8dn8A9WAACAqvvvqqIcnrryFDhhgbN270uv28vDzj0ksvrbXN8PBw46GHHvK6TafTafzjH/8wwsPDa233sssuM44cOVJne/V5/ZVfcXFxxvXXX29s3rzZ67i9NXny5GOKqfLr8OHDHm2OGDHC9fjChQvrjKFTp06u+tu3b6+zftX7V2f79u2uxzt16mSUl5cbf/zjH2t9Hd27dzc2bdpU570NwzAWLFhgdO3a1eufUe/evY29e/fW2N77779vJCYm1vj8yMhI44UXXvB4XbVZv359nTFef/31Rnl5eb1+/o3htXtj4cKFx9TfU1JSjDfeeKPWtqv+Tb366qtex7RlyxajS5cutd5/+PDhxp49e4z777/fVXb//fdX2543dWr6mYwYMeKY6xzL76q+/wYAVmLEAggSkyZNUrdu3fTjjz/qp59+0q+//qqDBw8qNzdXTqdTiYmJ6ty5s0466SRdfPHFOuWUU+rVfmJiombPnq3rr79er732mpYsWaKsrCyFh4erQ4cOOvPMM/XHP/6xXp802mw23XPPPbrooov00ksv6euvv9bu3btlt9vVtm1bDRs2TJMnT3adEO6L6OhoJSYmKiEhQZ06ddLgwYOVnp6usWPHenV6NKoXHh6ul156SZdccolefvllLV++XFlZWYqNjVXPnj01ceJE3XDDDV7v5DVq1Cht2rRJn3zyib744gv99NNPys7OVn5+vmJiYpSamqoePXro5JNP1rhx42rcdKDSxRdfrJNPPlnPPfecPv/8c+3cuVM2m03t27fXmDFjdOONN6pnz54e50HUpnfv3lq3bp3+85//aPbs2dq8ebOKi4vVtm1bpaen67rrrtPYsWO9bq8xvXZ/sdlsio+PV+vWrTVgwACdeeaZuuyyywK2tqlbt25avXq1MjMz9dFHH2nLli0qLy9XmzZt1LdvX11++eW69NJLXVPSAFjDZhherjIDAAAAgBqweBsAAACAz0gsAAAAAPiMxAIAAACAz0gsAAAAAPiMxAIAAACAz9hu1k+cTqf27dun+Ph42Ww2q8MBAAAATAzDUEFBgdLS0hQS4v/xBRILP9m3b586dOhgdRgAAABArXbv3q327dv7vV0SCz+pPKBr+/btSklJsTgaBBu73a6vv/5aZ5xxhsLDw60OB0GIPoK60EdQF/oI6nLo0CF17tw5YAfLklj4SeX0p/j4eCUkJFgcDYKN3W5XTEyMEhIS+Mce1aKPoC70EdSFPoK62O12SQrYtH0Wb/soMzNTvXr1Unp6utWhAAAAAJYhsfBRRkaGNm7cqOXLl1sdCgAAAGAZEgsAAAAAPiOxAAAAAOAzEgsAAAAAPiOx8BGLtwEAAAASC5+xeBsAAAAgsQAAAADgByQWAAAAAHxGYgEAAADAZyQWAAAAAHxGYgEAAADAZyQWPmK7WQAAAIDEwmdsNwsAAACQWAAAAADwAxILAAAAAD4jsQAAAADgMxILP9uTV2x1CAAAAECDI7Hws//+vN/qEAAAAIAGR2LhZ3PXZ8kwDKvDAAAAABoUiYWP3M+x2JFbpHV7jlgcFQAAANCwSCx85H6OxQlRq/Tx6r0WRwUAAAA0LBILP2sVu1qfr90nu8NpdSgAAABAgyGx8LOSmCzlFpXr+20HrQ4FAAAAaDAkFn62I7pC0bZ8fcR0KAAAADQjJBZ+VmGzqUfcd/p6Q7YKSu1WhwMAAAA0CBKLAIiP3aCyCqe+XJ9tdSgAAABAgyCxCIADsbmSpE/WMB0KAAAAzQOJRQDsibCpddgO/fhrrrKOlFgdDgAAABBwJBYBcnz8DzIM6bM1+6wOBQAAAAg4EosACYn9RZI4LA8AAADNAomFjzIzM9WrVy+lp6ebynfFFMmmCm3OLtCmrHyLogMAAAAaBomFjzIyMrRx40YtX77cVJ4XGqITotZIYtQCAAAATR+JRQC1iTuabHy6Zq8cTsPiaAAAAIDAIbEIoJLYPZKknPwyLfk11+JoAAAAgMAhsQig7VEVirIVSGI6FAAAAJo2Egs/CzV+n/JUHmJTj9gfJElfrs9SSbnDqrAAAACAgCKx8LNeRoTpOiFuvSSpqNyheZtyrAgJAAAACDgSCz87MeEE03VuzAHX96yzAAAAQFNFYuFn6cedYbreFWlTy7DdkqR1e/IsiAgAAAAIPBILP+vZ9VzFuW0te0Lc95KkLdkFKrWzzgIAAABND4mFn4WFRWlIWKKpLDR2mySpwmlowz5O4QYAAEDTQ2IRAMNaDTJd74kplOSUxHQoAAAANE0kFgFwcq+JputDYSHqErVWkrRuzxErQgIAAAACisQiADq0P1nt3JZSpMUtkyStZcQCAAAATRCJRQDYQkI0LCbNVFYec3RnqN8OFOlIid2KsAAAAICACbM6gMYuMzNTmZmZcjjMQxQntx+hD359x3X9W7Rdw1NmyZBNb86do1bxkXW23aFFD6X3u1ph4VF+jxsAAADwJxILH2VkZCgjI0P5+flKTPx9N6ghff+gkF/eltNmkySVhdi0LnWzJOnnQkmFXjSeNV8jN7yuZ//wvWwhDC4BAAAgePFuNUASEzuqjxHuczuLjAJt3PKxHyICAAAAAofEIoAmtD/dL+2s+O1Lv7QDAAAABApToQJowujHVfJVkRZkr9DhsnLTY7GRYQr53zQpd/ucZcoJ/f2xlbkbNTmgkQIAAAC+IbEIIFtIiP4wbqYmOQ31e+ArFZX/vsD7xatO1NheqdU+7/OF9+juXZ+6rlc7jsjpqFBIKL8uAAAABCemQjWA0BCb+rZPNJXVdgL34O7jTdd5ITZt37EwAJEBAAAA/kFi0UD6t08yXa+t5QTutLQT1cZhmMpW/vpFIMICAAAA/ILEooH0c0ss1u3Jk2EY1VeWNCiypel61YG1gQgLAAAA8AsSiwbSv4N5KlResV27DhXXWH9wy/6m61VlBwISFwAAAOAPJBYNpF1StFrERpjKapsONeiEc0zXWaE2Ze1bGZDYAAAAAF+RWDQQm82mfm4LuNfuzquxfpfjTlei022dxdZPAhAZAAAA4DsSiwbUv0OS6bq2naFCQsM0MDTBVLYqe3kAogIAAAB8R2LRgNx3hlq/N18VDmeN9Qen9DJdryreF4iwAAAAAJ+RWDQg96lQJXaHtu0vrLH+oM5jTde/hhrKO7w9ILEBAAAAviCxaEAt4iLVPjnaVFbbdKie3c5TtNs6i1Wb3g9EaAAAAIBPSCwaWH0OygsPj1G/EHMismrvj4EICwAAAPAJiUUDq8/OUJI0KOEE0/Wqwp3+DgkAAADwGYlFA3M/gXtLdoFK7Y4a6w/qONJ0vclmV3HxwQBEBgAAABw7EosG1rd9omy2368rnIY2ZuXXWL9fz4sUZvy+zqLCZtO6TR8GMkQAAACg3kgsGlhcZJhOaBVnKqttOlRMTEv1NMJNZat2LQpAZAAAAMCxI7Go4sILL1RycrIuvvjigN7HfTrUuloWcEvSoLhOputV+b/4OyQAAADAJyQWVdx66616/fXXA36fAR3cFnDXsuWsJA1qd7Lpep2zRHZ7sb/DAgAAAI5ZmNUBBJORI0dq0aJFAb+P+4jFbweKNODvX9dYP97WVur4+3VJiE2XPvGItttPCVCE3muTEKUpp5+gc/ulWR0KAAAALNRkRiwWL16s8847T2lpabLZbPrkk0886mRmZuq4445TVFSUhg4dqmXLljV8oJJ6tI1XeKjNVJZXbK/xa3dRitqXmw/KiwtfXetzGuprc3aBpr63VrtyGUEBAABozprMiEVRUZH69++va6+9VhMmTPB4fPbs2Zo6dapmzpypoUOH6umnn9aZZ56pLVu2qHXr1vW+X1lZmcrKylzX+flHd3ay2+2y2+21PjdEUt92iVq1K8/r+7UsSdSeiN93j7JH76tXvIFUXuHUByt36eZRx1sdStCq7BN19Q00X/QR1IU+grrQR1CXQPeNJpNYjBs3TuPGjavx8aeeekrXX3+9rrnmGknSzJkz9cUXX+iVV17RtGnT6n2/GTNmaPr06R7lCxcuVExMTJ3PPynWpp9DQmR32uqsK0nlRV2kxDWu693RpbKpQkaQ/Arf/fEXdSneYtpKF57mzZtndQgIcvQR1IU+grrQR1CT4uLAzjAJjnelAVZeXq6VK1fqrrvucpWFhIRozJgxWrJkyTG1edddd2nq1Kmu6/z8fHXo0EGjRo1SixYt6nz+2ZKuKbbrlwOFchpGnfULDkfojk1rfr9faIieOadMLdqcdCzh+2zXoRLd9fEG1/X+UpuOG3iKeqclWBJPsLPb7Zo3b57Gjh2r8PDwup+AZoc+grrQR1AX+gjqkpubG9D2m0VicfDgQTkcDqWmpprKU1NTtXnzZtf1mDFjtHbtWhUVFal9+/Z6//33NWzYsGrbjIyMVGRkpEd5eHi413/MrRLD1Sqx7tGNo8bqqfWGsqqszSgt/kHDu17q5fP962TDUOai37TncImr7Iv1ORrQqe6kqjmrT/9A80QfQV3oI6gLfQQ1CXS/aBaJhbe++eYbq0Oo1aDIVvqi4qDr+tW9C/TDG8Mti6d7SoU6xFW4rjfutOnW1yMlL6ZDHW+L0nWhrRRjCw1ghD5qcbx0ylQpOsnqSAAAAIJes0gsWrZsqdDQUOXk5JjKc3Jy1KZNG5/azszMVGZmphwOh0/teGNQq/76Imu+63pPqLTHmV/LMwIs4n9fVRllUt0zu7RA+dp9ZLsePxDYITmf7d8kXfG+1VEAAAAEvSaz3WxtIiIiNHjwYM2f//ubcqfTqfnz59c41clbGRkZ2rhxo5YvX+5rmHU6set5Ab9HQ/o2JtqbHMRav3wjVZRbHQUAAEDQazIjFoWFhfrll19c19u3b9eaNWuUkpKijh07aurUqZo8ebJOPPFEDRkyRE8//bSKiopcu0Q1Bl06j9bo7xM038pRCj8qCQlRfkiIEp1Oq0OpmeGUCrKk5E5WRwIAABDUmkxisWLFCo0aNcp1Xblj0+TJkzVr1ixNnDhRBw4c0H333afs7GwNGDBAX375pceC7mD36CVztXhFpn7L3WR1KJKkUrtDK3YeNpX1apuglFj3OVJHZ0j9O2+NaZQie9j/KTEiJbBB1td3T0oVvy9KV/5eEgsAAIA6NJnEYuTIkTLq2LZ1ypQpmjJlil/v25BrLCQpMipRY0+5u0Hu5a2Lnv9RK6skF8lpabrrgoHV1p393igdLPl9AXp2j7PUvcOIgMdYLz+/Jx3c+vv1kb3WxQIAANBINIs1FoHUkGssgtUFA9JM1/M25qiorKLaum1izIvls4uyAxbXMUtoZ74+stuaOAAAABoREgv47Oy+bRUa8vsesyV2h77ZlFNt3TaxbolFcRAmFoluiUU+IxYAAAB1IbGAz1rGRWr4CS1NZZ+t2VdtXY/EIihHLNqbr5kKBQAAUCcSC/jFBf3N06G+3XpAh4s8t2ltFIlFoltikb/HmjgAAAAaERILH2VmZqpXr15KT0+3OhRLndE7VZFhv3enCqehueuzPOqlxpp34QrOxMJ9jQWJBQAAQF1ILHzE4u2j4qPCNbpna1PZp9VMh3JfvJ1TnFPnbl4Nzn0qVMlhqbzYmlgAAAAaCRIL+M35/c2f9C/bfkj78kpMZe5ToexOuw6VHgp4bPXiPmIhsYAbAACgDiQW8JuR3VspPsp8NMqcdeZRi1bRrRRqCzWVBd3OUBGxUnSyuYzpUAAAALUisYDfRIWHalwf84jEJ6v3maY6hYaEqmW0eQepoFxn4bEzFIkFAABAbUgs4FcXDDBPI9qYla8ff801lTWOnaE4ywIAAKA+SCx8xK5QZid1aaF2SdGmsie/3mIatXBPLHKKqj9Mz1Iep28zYgEAAFAbEgsfsSuUWWiITRmjTjCVrdqVp0VbD7iu3XeGYsQCAACg8SOxgN9dcmJ7dUgxj1r8c95W16iFx1SoYFu8LUmJHczXjFgAAADUisQCfhceGqJbTu9qKlu354jmbTw65alxToXaKwXbeRsAAABBhMQCAXHhwHbq0jLWVPbUvK1yOg2PxGJ/8X45nI6GDK9u7lOh7EVSaZ4loQAAADQGJBYIiLDQEN06xjxqsTm7QP9dn+2RWFQYFcotNe8cZbn4NEk2c9kR1lkAAADUhMQCAXNuvzR1bR1nKvvnN1uVGJGssBDzQXpBt4A7LEKKSzWXsYAbAACgRiQWPmK72ZqFhth0+9huprJf9hfqi3XZSo0xv2kPusRC8pwOdWS3NXEAAAA0AiQWPmK72dqd1buNerZNMJU9M3+bWjeGxKK6BdwAAACoFokFAiokxKapbqMW2w8WyV5mTjaCc8vZ9uZrpkIBAADUKKzuKoBvxvRsrf7tE7V2zxFX2ba9oVKV5Rd7C7JUXF5RazuhITZFhoUGKkxPjFgAAAB4jcQCAWezHV1rcfWrv08XO1IYp6gqicW8rVv06ddf1dGONPz4lnp20kClxEYEKtzfuY9YsMYCAACgRkyFQoMY0a2VBndKdl077Ymmx21hR9yf4sEwpO9/Oaip761xneIdUB5TofZJTmfg7wsAANAIkVigQdhsNv25yloLwyOxKJDk3SF5i7Yc0OtLdvozvOq5T4Vy2qWiA4G/LwAAQCNEYoEGc/IJLXXz6ScoPNQmoyLJ9JjNZvwvufDOQ3M3aWuO9/WPSVxrye28DeXvCew9AQAAGinWWPgoMzNTmZmZcji8+7S9ufvzGd31fyOO16GiMo3/4jGVO8tcj828uot6t+hf7fPW7Tmim95a5bour3DqlndW65OM4YoKD9CC7pBQKSFNytv1e9mRPVK7wYG5HwAAQCPGiIWPOMei/mIjw9QhJVZt49qYyp2heWqfHFPt19l92+ra4Z1N9TdnF+ixL7cENtgE9wXc7AwFAABQHRILWKa+p2//9azu6tEm3lT2yg/b9e3WAK57cD99m7MsAAAAqkViAcu0iTWPWNSVWESFh+rZSQMVGWbutn95f61yC8tqeJaPPM6yYI0FAABAdUgsYJn6jlhIUrfUeN19dk9T2YGCMt354brAbEHrcZYFiQUAAEB1SCxgGfcRi5ziHK+ed9WwThrVvZWp7JtN+/XW0l01PMMHHmdZMBUKAACgOiQWsEx9p0JVstlseuzi/moZZz59+8EvNmr3oWK/xSfJcypUQbbksPv3HgAAAE0AiQUs455Y5JbmqtxR7tVzW8VH6vGLzVvTltqdevMnPx+c5z5iIUMqyPLvPQAAAJoAEgtYxj2xkLyfDiVJo3q01qQhHU1lH6/eK4fTj2stopOlsGhzGessAAAAPJBYwDLx4fGKCYsxlXk7HarStcOPM13vLyjTD78c9DW039ls1SzgZp0FAACAOxILWMZmsx3zOotKXVPj1a99oqnso1V+HlHwOMuCEQsAAAB3JBY+yszMVK9evZSenm51KI2S+5az9ZkKVWnCQPMb/y83ZKuwrMKnuEw4fRsAAKBOJBY+ysjI0MaNG7V8+XKrQ2mUfB2xkKTz+qcpLMTmui61O/Xfn/24wJrTtwEAAOpEYgFLeZxlUVT/EYsWcZEa2b21qeyjVX588+9x+vZu/7UNAADQRJBYwFIeIxbF9R+xkKSLBpnf/C/5LVd7DvvpTAsWbwMAANSJxAKWahPj+1QoSTq9Z2slRIWZyj5ds++Y4zJxTyxKDknlfj6IDwAAoJEjsYCl3Ecs8sryVFJRUu92IsNCdV7/NFPZh6v2yDD8cKaF+1QoScr3U9ICAADQRJBYwFLVHpJ3DOssJGnCIPPIwm8HirR2z5FjasskMk6KMm9pyzoLAAAAMxILWComPEbxEfGmsmNdZzGoY5KOa2E+cM9vZ1okdjBfszMUAACACYkFLOePLWelowfuuY9afLZ2n8ornMccm4vHzlAkFgAAAFWRWMBy/lrALUkXuh2Wl1ds18It+4+5PRdO3wYAAKgViQUslxrr++nblTqkxGhI5xRTmV+mQ3mMWJBYAAAAVEViAcv5c8RC8jzTYsHm/TpcVO5Tm5xlAQAAUDsSC1jOX2ssKo3r21aRYb93bbvD0Jx1Pm4P655Y5O+V/LGVLQAAQBNBYgHLuScWx7rdbKWEqHCd0dvc5oerfBxhcJ8KVV4olfphK1sAAIAmgsQClnNPLArsBSqyF/nU5gS36VBrdudpS3bBsTeYkOZZxjoLAAAAFxILH2VmZqpXr15KT0+3OpRGKzUm1aPM1+lQp57QUi3jIk1lL3z767E3GBYpxbY2l3GWBQAAgAuJhY8yMjK0ceNGLV++3OpQGq2osCglRyabynxNLMJCQ3T5EPOhdp+u3adducXH3qjHAm5GLAAAACqRWCAo+HsBtyRdPbyzosNDXdcOp6GZi30YtfA4y4IRCwAAgEphVgcASEfPsth0aJPrem/hXpVWlPrUZkykNHFIG836cYer7IOV2/V/IzuodXxU/RuMbyvZbL9fr3pD2vmDV091GobSDx2S/WCmHFXbCBCbpEgF/j7wn1DD0CmHDin04L/N/Qz4n6DqIxFx0sA/SL3HWxsHgKBCYoGg4L7O4sWfX9SLP7/ol7bje5ivz/nMh8aO6+BWUI9tbFtLUpYPN6+fTna7Ht1/UL3L7Q12Txy7EEktJMm3fQvQhAVdH/llnpSyWGrb3+pIAAQJpkIhKLhPhYLvdoaH69EWyXVXBIBjte1rqyMAEERILBAUuiV3szqEJmldZKRKrZ4yAaDp2r+p7joAmg2mQiEoDEsbprGdxmreznlWh9KkOGw2bR75Zw2IqeYcDgQVh8Oh9Rs2qE/v3goNDa37CWh2gqKP7FkhrX379+ucjdbEASAokVggKISHhOupkU/pcOlhFdoL/d7+joNFuuqVZaayP512vK44qaPf71WdiooKLVq4SCNHjVRYWGD/7KbMn6Lfjvzmut7QuosG9LwioPeE75x2u3bkzFWvwWcrNDzc6nAQhIKij6T2NicWudukinIpLMKaeAAEFRILBJXkqGQlR/l/XUCHeOnMbkX6csPv29h+sLRYt41MU1R44D/5s9vtSglNUfu49goP8BuCvi37mhOLgxsCej8AzUgrt90wnBVHk4vU3tbEAyCosMYCzUbGqBNM1wcLy/Xusl0WRRM4fVr2MV2vz11vUSQAmpzoJCnB7bBQ1lkA+B8SCzQbfdsn6rRurUxl/1n8m8ornBZFFBi9W5g/OdxxZIcKy/0/vQxAM9W6p/k6h1FRAEeRWKBZmeI2arHvSKk+Wd20TtDuntJdYSG/z3I0ZJgOHwQAn6T2Ml8zYgHgf0gs0KwM6ZyiIcelmMqe//ZXOZ2GRRH5X0RohLomdTWVsc4CgN+0dk8s+PcFwFEkFmh2bhp1vOl6+8EiLd1+yKJoAoN1FgACxj2xyNsllRVYEwuAoEJigWZnRLdW6p4abyr7dE3Tmg7lvs5i/UESCwB+0rKbZHPbTW//ZmtiARBUSCzQ7NhsNo0f2M5UNvfnLJVVOCyKyP/cRyz2Fu5VXmmeNcEAaFrCo6QW5pFf7eegPAAkFmimzh9gPok6v7RCi7YcsCga/+uS1EWRoZGmsg25zIMG4CfuO0ORWAAQiQWaqXZJ0RrS2byIuylNhwoPCVePFPNBViQWAPymtduBeCQWAERigWbsArdRi2827Vd+qd2iaPzPYwE36ywA+IvHWRYkFgBILEzmzJmj7t27q2vXrnrppZesDgcBdk7ftgoPtbmuyyuc+mp9toUR+Zf7Am62nAXgN6luIxbFB6XCpjOdFMCxIbH4n4qKCk2dOlULFizQ6tWr9fjjjys3N9fqsBBASTERGtGttans0zX7LIrG/3q3NP+Pf3/Jfu0v3m9RNACalOTjpLBocxnnWQDNHonF/yxbtky9e/dWu3btFBcXp3Hjxunrr7+2OiwE2PiB5ulQP/56UPvzSy2Kxr+OSzhOseGxpjJGLQD4RUio1Kq7uYwTuIFmr8kkFosXL9Z5552ntLQ02Ww2ffLJJx51MjMzddxxxykqKkpDhw7VsmXLXI/t27dP7dr9vgVpu3bttHdv01nMi+qN7pGq2Ijf92N3GtJna5vGqEWILUS9WpgPsmIBNwC/cT8oL4d/X4DmLszqAPylqKhI/fv317XXXqsJEyZ4PD579mxNnTpVM2fO1NChQ/X000/rzDPP1JYtW9S6detqWqxdWVmZysrKXNf5+fmSJLvdLru96SwAburCbNIZvVrr4zVZrrJP1+zV5JM6+PU+lX2ioftGj+QeWp693HX984Gf6Z9Byqo+gsYj2PpISMvuqnpMnjNnoxxBEltzFWx9BMEn0H2jySQW48aN07hx42p8/KmnntL111+va665RpI0c+ZMffHFF3rllVc0bdo0paWlmUYo9u7dqyFDhtTY3owZMzR9+nSP8oULFyomJsaHV4KG1qbMJlX53+PPe/M168O5ah1d83OO1bx58/zfaC3Ky8tN12uy1+iLL76QzWar4RmwWkP3ETQ+wdJHWuUX6OQq187s9Zr7xRzJ1mQmQzRawdJHEHyKi4sD2r7NMAwjoHewgM1m08cff6zx48dLOvrmKiYmRh988IGrTJImT56svLw8ffrpp6qoqFDPnj21aNEiJSYmavDgwfrxxx/VokWLau9R3YhFhw4dlJWVVeNzEJwqHE6d8vhi5Rb9/iZ8ysguunX0CX67h91u17x58zR27FiFh4f7rd267Cnco/M/O99UNuf8OUqLS6vhGbCKVX0EjUfQ9ZGCLIU/29dUZM9YKSV1siggBF0fQdDJzc1V27ZtdeTIESUkJPi9/SYzYlGbgwcPyuFwKDU11VSempqqzZs3S5LCwsL05JNPatSoUXI6nfrrX/9aa4IQGRmpyMhIj/Lw8HD+mBuZ8HDpvP5pmvXjDlfZnJ+z9ecze/j9k/2G7h/HJR2nxMhEHSk74irbfGSzOiXzP/5gxb8hqEvQ9JHkDlJUklSa5yoKz90qtfLfhzI4NkHTRxB0At0vGK+s4vzzz9fWrVv1yy+/6IYbbrA6HDSg8QPbma535BZr7Z4jNdRuPGw2m+d5FizgBuAPNpvneRacwA00a80isWjZsqVCQ0OVk5NjKs/JyVGbNm18ajszM1O9evVSenq6T+3AWv3bJ+q4Fua1MZ+sbhq7gnFQHoCAcT+Bm8QCaNaaRWIRERGhwYMHa/78+a4yp9Op+fPna9iwYT61nZGRoY0bN2r58uV1V0bQstlsOn+AedRizrp9qnA4LYrIf/q07GO63pi7UU6j8b8uAEHAfctZzrIAmrUmk1gUFhZqzZo1WrNmjSRp+/btWrNmjXbt2iVJmjp1ql588UW99tpr2rRpk2688UYVFRW5dokCxg8wL2g+WFiuH39t/Kevu49YFNoLtTN/p0XRAGhS3BOLg1ulivLq6wJo8prM4u0VK1Zo1KhRruupU6dKOrrz06xZszRx4kQdOHBA9913n7KzszVgwAB9+eWXHgu60Xx1aRWnfu0Tta7K2opP1uzVad1aWRiV71JjU9UqupUOlBxwla0/uF6dEztbGBWAJsF9KpSzQsr9RUrtVX19AE1akxmxGDlypAzD8PiaNWuWq86UKVO0c+dOlZWVaenSpRo6dKh1ASMond/fPGoxb2OOnM7GvyOz+6jFxlzmQQPwg+gkKcE8jZR1FkDz1WQSC6uweLtpOauPeTF/QWmF9uaVWBSN//RuaU4s1h9cb1EkAJocj3UWJBZAc0Vi4SMWbzct7ZKiFR9lniG4bX+BRdH4j/uIxeZDm1XhrLAoGgBNivt0qBwSC6C5IrEAqrDZbOraOs5Uti2n0KJo/Md9xKLUUapf8361KBoATQpnWQD4nyazeBvwl26p8Vq1K891vbUJJBYpUSlKi03TvqJ9rrL7frxPLaNbWhgVqnI6nTpQeEBfLvpSISF85lOX8JBwndruVE3oOkE2m83qcJo39xGLvJ1SWYEUGW9NPAAsQ2Lho8zMTGVmZsrhcFgdCvyka6r5f4ZNYSqUdHTUompiwQLu4LRl3xarQ2g05u+ar8iwSJ3b5VyrQ2neWnaXbCFS1fNxDmyR2p9oXUwALMHHYj5ijUXT0y3VcypUU9gZqm/LvlaHAPjd4j2LrQ4B4VFSyvHmspwN1sQCwFIkFoCbbm4jFiV2R5PYGerszmcrISLB6jAAv8ouyrY6BEie51ZwAjfQLDEVCnDTOj5SCVFhyi/9fdekbfsL1CElxsKofJcam6oPzvtAi/YsUrG92Opw4MbpdGrz5s3q0aMHayxqsSN/hz755RPXdVZRlnXB4Hete0kbP/39ej8jFkBzRGIBuLHZbOqaGq+VOw+7yrbmFOr0Ho3/lPa2cW01qcckq8NANex2u+bumKuze52t8PBwq8MJWhtzN5oSiwPFB1ThrFBYCP87s5T7Au6sddKKV6yJpRkLcTjU6eB6hazaL4WGWh0OgsGAP0hhEQ12O/4lBqrRLTXOLbFoGgu4gcaubWxb07XDcOhgyUG1iW1TwzPQIFq7bTlbmifNud2SUJqzUEkDJGm3tXEgiPS5uEETC8bbfcTJ201T19ZuO0M1gS1ngaYgKTJJUaFRpjKmQwWBlM5SWFTd9QA0aSQWPmJXqKbJfQH3L/ubxs5QQGNns9k8RieyCkksLBcSKp0wxuooAFiMqVBANdy3nC2xO7TncIk6tmjcC7iBpqBNbBvtyN/hus4uZmeooHD+c1JcKidvW8hpGDp86JCSU1IUwsGRkI4m/Q2IxAKoRqv4SCVGh+tIid1Vtm1/AYkFEATc11kwYhEkYlKkc5+yOopmzWG36/u5c3X22WcrhE0gYAGmQgHVsNlsHqMWW1lnAQQF96lQnGUBAMGBxAKowQkeC7jZGQoIBh4jFizeBoCgQGIB1MBjxGI/iQUQDDxGLFhjAQBBgcTCR2w323SxMxQQnNwTiyNlRzhNHgCCAImFj9hutunq6jZiUWp3as/hEouiAVCpusPwWGcBANYjsQBq0CouUkkx5l01OIEbsF50WLSSI5NNZSQWAGA9EgugBjabTd3cFnCzzgIIDh6H5LGAGwAsR2IB1OIEt+lQ29hyFggKJBYAEHxILIBadGvtfpYFIxZAMGDLWQAIPiQWQC2q2xnKwc5QgOXcE4ucohyLIgEAVCKxAGrR1S2xKKtwas9htrUErMZUKAAIPiQWPuIci6atZVyEkj12hmKdBWA1j0PyirJlGIwmAoCVSCx8xDkWTZvNZvMYtWCdBWA996lQ5c5yHSo9ZFE0AACJxAKoU9fW7jtDkVgAVmsZ3VJhtjBTGWdZAIC1SCyAOrgv4GYqFGC90JBQtY5pbSpjnQUAWIvEAqhDV7ezLH49wM5QQDBgATcABBcSC6AO7iMWZRVO7T7EzlCA1drGmddZMBUKAKxFYgHUoWVcpFJiI0xlLOAGrNcmhhELAAgmJBaAFzwWcO9nnQVgNfedoRixAABrkVgAXnBfZ8GIBWA996lQjFgAgLVILAAvsDMUEHxSY1JN1wdLDqrcUW5RNAAAEgvAC11bmxMLdoYCrOc+YiFJOcU5FkQCAJBILHyWmZmpXr16KT093epQEEDd3KZClVc4tYudoQBLxYfHKzY81lTGOgsAsI7PiUWXLl100kkneV3/1FNP1fHHH+/rbYNGRkaGNm7cqOXLl1sdCgKoRVykWrAzFBBUbDYbC7gBIIj4nFjs2LFDu3bt8rr+nj17tGPHDl9vCzQ49wXc20gsAMulxprXWbCAGwCs0+BToSoqKhQSwgwsND7u6yxYwA1Yz33EgsQCAKzToO/wS0pKtH//fsXHx9ddGQgy7ussNmXlWxQJgEokFgAQPMLq+4Rdu3Z5TGUqLy/Xd999J8OofpccwzCUl5ent956S3a7XX379j2mYAEr9W6XaLretr9QecXlSoqJqOEZAAKtTaz59O2cInaFAgCr1DuxePXVV/X3v//dVHb48GGNHDmyzucahiGbzaY//elP9b0tYLk+aYmKDAtRWYXTVbZy52GN7play7MABJL7iMW+wn2u/9cAABrWMU2FMgzD9WWz2UzX1X1JUkJCgoYPH67XX39dl19+uV9fBNAQIsJCNKBDkqls2Y5D1gQDQJLniEVxRbEK7GysAABWqHdicf/998vpdLq+DMNQmzZtTGXuXw6HQ4cPH9Z3332nK664IhCvA2gQQzqnmK6XbyexAKyUGpMqm8yjE2w5CwDWqPdUKHdXXXWVkpKS/BAKEPzSjzMnFj/vPaJSu0NR4aEWRQQ0bxGhEWoR3UIHSw66yrKLstUtuZuFUQFA8+RzYjFr1iw/hAE0DgM7JinEJjn/t0+B3WFoze48ndSlhbWBAc1Y29i2psQiq5CdoQDACn7dbtbpdGr58uX64IMP9Prrr/uzaSAoxEeFq1dagqlsBessAEu5r7Ngy1kAsIbfEovnnntObdu21UknnaSJEyfqmmuuMT1++PBh9enTRz169FBODtsBovE6sZN5OtSyHYctigSA5JlYZBezxgIArOCXxCIjI0O33XabDhw4oPj4+Gq3+UtOTtagQYO0bds2vf/++/64LWAJ9wXcq3YelsNZ/RkuAALP45A8pkIBgCV8Tiy+/PJLPf/884qLi9PHH3+svLw8tWrVqtq6l19+uQzD0DfffOPrbQHLnHhcsum6sKyCU7gBC7knFuwKBQDW8DmxmDlzpmw2m/7+97/rggsuqLXusGHDJEk///yzr7cNGpmZmerVq5fS09OtDgUNpHV8lI5rEWMqW846C8Ay7lOh9hfvl8PpsCgaAGi+fE4sli5dKkm69tpr66ybmJiohIQEZWc3nU+TMjIytHHjRi1fvtzqUNCA3LedJbEArOOeWFQYFaZdogAADcPnxOLQoUNKTExUfHy8dzcMCZHT6fT1toClPBOLw65T5gE0rJSoFEWERJjK2BkKABqez4lFQkKC8vPzZbfb66x76NAhHTlyRC1btvT1toCl0t0WcB8oKNPO3GKLogGatxBbiOfOUKyzAIAG53Ni0bdvXxmG4ZoSVZt33nlHhmHoxBNP9PW2gKWOaxGjlnGRprJlTIcCLENiAQDW8zmxuPjii2UYhh544IFapzitXbtW99xzj2w2myZNmuTrbQFL2Ww2pbvtDsVBeYB1OCQPAKznc2Jx/fXXq1evXlq4cKHGjh2rOXPmyOE4uhvHtm3bNG/ePN1yyy06+eSTdeTIEZ100km65JJLfA4csFp16ywAWMPjLAsSCwBocGG+NhAeHq4vvvhCZ511lhYuXKhFixa5HuvRo4fre8Mw1LdvX3344YfVHqAHNDbuB+VtP1ikAwVlahUfWcMzAAQKU6EAwHp+OXm7U6dOWrlypaZPn66OHTvKMAzTV1pamh544AH9+OOPatOmTd0NAo1Ajzbxio0INZUxHQqwBofkAYD1fB6xqBQTE6N7771X9957r/bt26d9+/bJ4XCoTZs26tSpk79uAwSNsNAQDeqUrO+2/b5f/rIdhzSub9tangUgENwTi8Nlh1VSUaLosGiLIgKA5sfnxOL000+XzWbTf/7zHx1//PGSpLS0NKWlpfkcHBDs0o9LMSUWK1hnAVjCfSqUdHTUonNiZwuiAYDmyefE4vvvv1d4eLgrqQCaE/cF3Bv2HVFhWYXiIv02GAjACzHhMUqISFB+eb6r7C/f/kVx4XEWRuVfTsOpwwWH9cG8DxRi88tMZjQx9BG4e37M84oJj2mw+/n87ic1NVWFhYX+iAVodAZ0SFJ4qE12x9FTt52GtGrnYZ3WrZXFkQHNT9vYtqbEYuvhrRZGEzg7D+y0OgQEOfoIKjmNmo+CCASf09nTTjtN+fn52rZtmz/iARqV6IhQ9WmXaCpjATdgjY4JHa0OAQCaNZ8Ti7/85S8KCwvTn//8ZxmG4Y+YgEZliNt0KE7gBqxxSbdLFGZjGiIAWMXnf4EHDhyod955R1dffbWGDx+uO+64QyeffLJat27NeRVoFtKPS9ELi39zXa/ZnafyCqciwpjfCjSkYWnD9OEFH2rN/jWqcFZYHY7fORwOrV+/Xn369FFoaGjdT0CzQx+Bu8jQhj1by+fEomrHXbp0qS6++OI6n2Oz2VRR0fT+0UfzNLhTsum61O7UM/O3qnV8lKvM4XBoQ5ZNuT/tqvc/9q3iIzWqe2tFR/A/CaAuXRK7qEtiF6vDCAi73a6YbTE6u+vZCg8PtzocBCH6CKzmc2LB9Cc0d8mxEeqWGqetOb9vYpC58Ndqaobqwx2bj+keQzun6N0bTmIUEAAABC2fE4uFCxf6Iw6gUTvxuBRTYuFvS7cf0i/7C9U1NT5g9wAAAPCFz4nFiBEj/BEH0KhdPLi93lm2S4EcwNtzuITEAgAABC22z6jiwgsv1KJFizR69Gh98MEHVoeDRmRQx2T9+/JB+mj1XhWU2j0eNwxDubm5atGihdfTmTbszVdB2e9rkfYXlPotXgAAAH8jsaji1ltv1bXXXqvXXnvN6lDQCI3r21bj+rat9jG73a65c+fq7LPTvV5Qd+2s5Vqweb/ren9+mV/iBAAACASfE4vFixfXq35UVJSSkpJ0/PHHB91WaCNHjtSiRYusDgOQJLWON28Rt7+AxAIAAAQvnxOLkSNHHtNONVFRURo9erT++te/6pRTTqmz/uLFi/X4449r5cqVysrK0scff6zx48eb6mRmZurxxx9Xdna2+vfvr+eee05Dhgypd2xAMPBMLJgKBQAAgpdfTvAyDKPeXyUlJZozZ45GjhypZ555ps57FBUVqX///srMzKz28dmzZ2vq1Km6//77tWrVKvXv319nnnmm9u//fSrJgAED1KdPH4+vffv2+ePHAPhVq4Qo0zUjFgAAIJj5PGLhdDo1Z84cTZ48Wampqbrjjjs0cuRItWvXTpK0d+9eLVq0SE888YRycnL0+uuv6+STT9by5cv16KOPasGCBfrzn/+sU089VYMGDarxPuPGjdO4ceNqfPypp57S9ddfr2uuuUaSNHPmTH3xxRd65ZVXNG3aNEnSmjVrfH25LmVlZSor+/2NXn5+vqSjc+ntds/Fu2jeKvtEffpGi2jzn+f+/FL6VhN2LH0EzQt9BHWhj6Auge4bPicWq1ev1iWXXKJTTjlFc+bMUWSkefpG586d1blzZ11xxRU655xzdPHFF2vp0qUaO3asxo4dq3PPPVdz585VZmamXn755WOKoby8XCtXrtRdd93lKgsJCdGYMWO0ZMkSn15fTWbMmKHp06d7lC9cuFAxMTEBuScav3nz5nldd0eBVPVPNPtIib74Yq44I69pq08fQfNEH0Fd6COoSXFxcUDb9zmxmDFjhsrLy5WZmemRVFQVERGhf/3rX+rZs6dmzJiht99+W5I0ffp0zZ07t96LwKs6ePCgHA6HUlNTTeWpqanavNn7k47HjBmjtWvXqqioSO3bt9f777+vYcOGVVv3rrvu0tSpU13X+fn56tChg0aNGqUWLVoc2wtBk2W32zVv3jyNHTvW612h9uWV6J/rv3NdOwybho8aq6QY756PxuVY+giaF/oI6kIfQV1yc3MD2r7PicX333+vhIQEdevWrc663bt3V2JiomnnpcGDBysqKioo1jl88803XteNjIysNpEKDw/njxk1qk//aJvsuWva4VKHWiUyItaU8W8I6kIfQV3oI6hJoPuFz4u3Dx8+rLKyMhleHDnsdDpVWlqqw4cPm8qjo6OPaWepSi1btlRoaKhycnJM5Tk5OWrTps0xtwtYKSIsRMluoxOcZQEAAIKVz4lFWlqaysrK9Pnnn9dZd86cOSorK1NaWpqrrDLRaNWq1THHEBERocGDB2v+/PmuMqfTqfnz59c4lclfMjMz1atXL6Wnpwf0PmieWse77wzFlrMAACA4+ZxYnH/++TIMQ9dff71+/PHHGustWbJEN9xwg2w2m84//3xX+YYNGyRJXbp0qfU+hYWFWrNmjWtnp+3bt2vNmjXatWuXJGnq1Kl68cUX9dprr2nTpk268cYbVVRU5NolKlAyMjK0ceNGLV++PKD3QfPUOoFD8gAAQOPg8xqLe+65R++9956ysrJ02mmn6bTTTtOIESOUlpYmm82mffv2adGiRVq8eLGcTqfatm2re+65x/X8N998U5I0evToWu+zYsUKjRo1ynVduXB68uTJmjVrliZOnKgDBw7ovvvuU3Z2tgYMGKAvv/zSY0E30Ji0cj8kj6lQAAAgSPmcWLRo0UILFy7UxRdfrPXr12vRokX69ttvTXUq11/07t1bH3zwgWnXpAsuuEAjR47USSedVOt9Ro4cWec6jilTpmjKlCnH+EqA4MNUKAAA0Fj4nFhIUrdu3bRq1SrNnj1b77//vlatWqUDBw5Iklq1aqVBgwbp4osv1sSJEz1Wo48cOdIfIVgmMzNTmZmZcjgcVoeCJqi1+4gFU6EAAECQ8ktiIUlhYWG64oordMUVV/iryUYhIyNDGRkZys/PV2JiotXhoIlxX2NxgMQCAAAEKZ8XbwMIHI+pUPlMhQIAAMHJbyMWlQ4cOKCdO3equLhYp512mr+bB5oV96lQReUOFZVVKDbS73+6AAAAPvHbiMVnn32mQYMGqU2bNho6dKhOP/100+OHDx/WWWedpbPOOktHjhzx122BJs19KpTEOgsAABCc/JJYPPLII7rwwgu1Zs0aGYbh+qoqOTlZ0dHRmjdvnj744AN/3DYocEAeAikmIkxxbqMTTIcCAADByOfE4qefftLf/vY3hYWF6Z///KcOHjxY49kRf/jDH2QYhubNm+frbYMGB+Qh0NgZCgAANAY+JxbPPPOMJOmuu+7SrbfeqpSUlBrrjhgxQpK0evVqX28LNBseh+SRWAAAgCDkc2Lxww8/SJJXB9O1bNlSsbGx2rdvn6+3BZqN1gkckgcAAIKfz4nF/v37FR8fr5YtW3pVPzIyUuXl5b7eFmg23KdCHchnxAIAAAQfnxOL2NhYFRcXe3XydGFhofLy8mqdLgXAjDUWAACgMfA5sejevbscDofWrVtXZ91PPvlETqdTAwYM8PW2QYNdoRBo7lvOMhUKAAAEI58Ti/PPP1+GYWjGjBm11tuzZ4+mTZsmm82miy66yNfbBg12hUKgeZy+zYgFAAAIQj4nFlOmTFG7du304Ycf6qqrrtL69etdj9ntdm3btk1PPfWUBg8erH379qlbt26aPHmyr7cFmg33qVB5xXaVVdQ99RAAAKAhhdVdpXZxcXH6/PPPdeaZZ+rNN9/UW2+95XosKur3T1oNw1BaWpo++eQThYeH+3pboNlwH7GQpAMFZWqfHGNBNAAAANXzy8nbAwYM0Nq1a3XNNdcoMjLSdPq2YRgKDw/X1VdfrRUrVqh79+7+uCXQbCREhykizPynynQoAAAQbHwesajUpk0bvfzyy/r3v/+tlStXat++fXI4HGrTpo3S09MVE3P001W73a4XXnjBq3MvAEg2m02t4yO153CJq2w/W84CAIAg47fEolJkZKROPvlkj3KHw6GXX35ZDz30kPbu3dtkEovMzExlZmZ6td0ucKzcE4sD7AwFAACCjE9ToYqLi7V27VqtWrVKhw8frraOYRiaNWuWunXrphtvvFG7d++WYRi+3DaosCsUGgI7QwEAgGB3TInFkSNHNHnyZLVo0UKDBg1Senq6WrVqpQkTJigrK8tVb9GiRerXr5/++Mc/avv27ZKkCy64QEuXLvVP9EAz4XGWBVOhAABAkKn3VKiKigqNHTtWK1euNI08GIahTz/9VFu3btWqVav03HPP6c4775TT6VRoaKgmTpyou+66S7179/brCwCaA/ctZ3OYCgUAAIJMvROL1157TStWrJAknX766TrrrLNkGIa++uorLViwQJs2bdKf/vQnvfbaa7LZbLrqqqt03333qUuXLn4PHmguPKZCMWIBAACCTL0Ti/fff182m03XX3+9Zs6c6Sq/4447dMMNN+ill17S66+/ruTkZH300UcaMWKEXwMGmqNW7lOhWGMBAACCTL3XWPz888+SpHvuucfjsXvvvdf1/SOPPEJSAfiJ+1So3KIyVTicFkUDAADgqd6JRW5urmJiYtS+fXuPxzp06OA6r+L888/3PToAkjynQhmGlFtUblE0AAAAnuqdWJSXlys+Pr7GxysfS01NPfaoAJi0iI1QaIjNVMY6CwAAEEx8OscCRw/I69Wrl9LT060OBU1YSIhNLeMiTGX72RkKAAAEERILH3FAHhoKh+QBAIBgVu9doSQpJydHoaGhtdap7XGbzaaKiopjuTXQbLkv4GYqFAAACCbHlFhUPRgPQMPwOH2bqVAAACCI1DuxuP/++wMRB4A6tGIqFAAACGIkFkAj4TEVisQCAAAEERZvA42Ee2JxIJ+pUAAAIHiQWACNROsE81SoA4VlrHcCAABBg8QCaCTcRyzsDkOHi+0WRQMAAGBGYgE0Ei3jIj3K2BkKAAAECxILH3HyNhpKRFiIUmLdTt/mLAsAABAkSCx8xMnbaEjsDAUAAIIViQXQiLTySCyYCgUAAIIDiQXQiLR2PySPqVAAACBIkFgAjUjrBLezLJgKBQAAggSJBdCIeK6xYCoUAAAIDiQWQCPiMRWKEQsAABAkSCyARsR9KtT+fE7fBgAAwYHEAmhE3KdCldgdKiyrsCgaAACA35FYAI2I+1QoielQAAAgOJBYAI1IdESo4iPDTGVsOQsAAIIBiQXQyLRyX2fBzlAAACAIkFgAjYz7OgvOsgAAAMGAxAJoZNhyFgAABCMSC6CR8TgkL5+pUAAAwHokFj7KzMxUr169lJ6ebnUoaCY8zrJgxAIAAAQBEgsfZWRkaOPGjVq+fLnVoaCZYCoUAAAIRiQWQCPDVCgAABCMSCyARsZ9KlR+aYV2Hyq2KBoAAICjSCyARqZTi1jFR5kPyXt72S6LogEAADiKxAJoZMJDQ3TRoPamsveW71Z5hdOiiAAAAEgsgEbp8qEdTde5ReX6ckO2RdEAAACQWACNUrfUeA3pnGIqe/OnnRZFAwAAQGIBNFp/OKmT6XrZ9kPallNgUTQAAKC5I7EAGqmzerdRi9gIU9lbS1nEDQAArEFiATRSEWEhujS9g6nsw5V7VFxeYVFEAACgOSOxABqxy4d0lM32+3VBWYU+X7vPuoAAAECzRWIBNGIdUmI0slsrU9mbPzEdCgAANDwSC6CRu2KoeRH3z3uPaO3uPGuCAQAAzRaJBdDIjerRWu2Sok1lbD0LAAAaGokF0MiFhtg0aYh5Effn6/bpSLHdoogAAEBzRGIBNAGXpndQWMjvq7hL7U59uGqPhREBAIDmhsQCaAJax0fpzN5tTGVvLd0pwzAsiggAADQ3YVYHAMA/rjipo774Oct1/euBIt345irFRIaa6oWF2DTs+BYaP6CdbFX3qgUAAPABiQXQRAzr0kJdWsXqtwNFrrIvN2RXW/e9FXuUW1iu607t0lDhAQCAJo6pUEATYbPZPLaerc0z32xTXnF5ACMCAADNCYnF/+zevVsjR45Ur1691K9fP73//vtWhwTU26UnttdxLWK8qltQVqH/LP4twBEBAIDmgqlQ/xMWFqann35aAwYMUHZ2tgYPHqyzzz5bsbGxVocGeC0+Klyz/zRMn6/dpwMFZR6PL9txSKt35bmuZ/24Q9ee0lkt4yIbMEoAANAUkVj8T9u2bdW2bVtJUps2bdSyZUsdOnSIxAKNTmpCVI1rJ349UKixT30r5/82iyoud2jmol91z7m9GjBCAADQFDWaqVCLFy/Weeedp7S0NNlsNn3yyScedTIzM3XccccpKipKQ4cO1bJly47pXitXrpTD4VCHDh3qrgw0Ise3itOEQe1NZW/8tFPZR0otiggAADQVjWbEoqioSP3799e1116rCRMmeDw+e/ZsTZ06VTNnztTQoUP19NNP68wzz9SWLVvUunVrSdKAAQNUUVHh8dyvv/5aaWlpkqRDhw7pqquu0osvvlhrPGVlZSor+32qSX5+viTJbrfLbufEY5hV9olg6Bs3jThOn6zeq4r/DVuUVTj13PyteuC8nhZH1rwFUx9BcKKPoC70EdQl0H3DZjTCE7RsNps+/vhjjR8/3lU2dOhQpaen61//+pckyel0qkOHDrr55ps1bdo0r9otKyvT2LFjdf311+vKK6+ste4DDzyg6dOne5S//fbbionxbvEsYJX3fgvRDzm/D1iG2gzdM9ChFJZaAADQZBUXF+vyyy/XkSNHlJCQ4Pf2G82IRW3Ky8u1cuVK3XXXXa6ykJAQjRkzRkuWLPGqDcMwdPXVV+v000+vM6mQpLvuuktTp051Xefn56tDhw4aNWqUWrRoUf8XgSbNbrdr3rx5Gjt2rMLDw60ORwOPlGrM09+rvMIpSXIYNm20ddLDZ/e2OLLmK9j6CIIPfQR1oY+gLrm5uQFtv0kkFgcPHpTD4VBqaqqpPDU1VZs3b/aqjR9++EGzZ89Wv379XOs33njjDfXt27fa+pGRkYqM9Px4Nzw8nD9m1ChY+kfHluG6YmhHvfrDDlfZR6v36aZRXdW5JRsWWClY+giCF30EdaGPoCaB7hdNIrHwh1NOOUVOp9PqMIAGc+PI4/Xust0qsTskSQ6noWe+2aqnLxtocWQAAKAxajS7QtWmZcuWCg0NVU5Ojqk8JydHbdq0Cei9MzMz1atXL6Wnpwf0PoC/tY6P0uSTjzOVfbp2n7blFFgTEAAAaNSaRGIRERGhwYMHa/78+a4yp9Op+fPna9iwYQG9d0ZGhjZu3Kjly5cH9D5AIPzptC6Ki/x94NIwpL/P2ag56/Z5fC3asl9FZZ67qgEAAEiNaCpUYWGhfvnlF9f19u3btWbNGqWkpKhjx46aOnWqJk+erBNPPFFDhgzR008/raKiIl1zzTUWRg0Et+TYCP3xlM56Zv42V9l32w7qu20Hq61/fKtYfZwxXAlRzN0FAABmjSaxWLFihUaNGuW6rtyRafLkyZo1a5YmTpyoAwcO6L777lN2drYGDBigL7/80mNBNwCzP57aWbN+3KEjJXXvbf3rgSIt3LxfFwxo1wCRAQCAxqTRJBYjR45UXUduTJkyRVOmTGmgiI7KzMxUZmamHA5Hg94X8JeEqHBljDpeD8/1bge1nHxO6QYAAJ4aTWIRrDIyMpSRkaH8/HwlJiZaHQ5wTK47pYtKyp2atylbZXbz7mjZR0pVUGVtRUEp6ywAAIAnEgsACgmx6dYxXXXrmK4ej935wTrNXrHbdU1iAQAAqtMkdoUCEDhxUebPHwrZGQoAAFSDxAJAreLdEouC0roXeQMAgOaHxMJHHJCHpq7qORcSIxYAAKB6JBY+4oA8NHXuIxaFrLEAAADVILEAUKt4t8PwWLwNAACqQ2IBoFbuU6EKmAoFAACqQWIBoFYeu0IxYgEAAKpBYgGgVgluiUWJ3SG7w1lDbQAA0FyRWPiIXaHQ1MVFhnuUFTEdCgAAuCGx8BG7QqGpc58KJbGAGwAAeCKxAFCr2IhQ2WzmMhILAADgjsQCQK1sNhuH5AEAgDqRWACoU7xHYmG3KBIAABCsSCwA1IlD8gAAQF1ILADUyX0BN4kFAABwR2LhI7abRXPAGgsAAFAXEgsfsd0smoN4jxEL1lgAAAAzEgsAdXJPLAqZCgUAANyQWACok/tUqAKmQgEAADckFgDqxK5QAACgLiQWAOrksXibxAIAALghsQBQJ/ftZtkVCgAAuCOxAFCnBHaFAgAAdSCx8BHnWKA5iIs0r7FgxAIAALgjsfAR51igOfA8x4LEAgAAmJFYAKiT+xqLsgqnyiucFkUDAACCEYkFgDrFu+0KJTEdCgAAmJFYAKiT+zkWElvOAgAAMxILAHWKCg9RaIjNVJbPzlAAAKAKEgsAdbLZbJ6H5DEVCgAAVEFiAcAr7jtDMRUKAABURWIBwCvuIxYFZUyFAgAAvyOxAOAVRiwAAEBtSCwAeMV9Z6h8EgsAAFAFiYWPMjMz1atXL6Wnp1sdChBQLN4GAAC1IbHwUUZGhjZu3Kjly5dbHQoQUO6nbzMVCgAAVEViAcAr7mssCjjHAgAAVEFiAcAr8UyFAgAAtSCxAOAVj+1mmQoFAACqILEA4BX3XaFILAAAQFUkFgC84rF4m6lQAACgChILAF5hjQUAAKgNiQUAr3hOhbLLMAyLogEAAMGGxAKAV9ynQtkdhsoqnBZFAwAAgg2JBQCvuO8KJTEdCgAA/I7EAoBX3A/Ik9gZCgAA/I7EAoBXIsNCFB5qM5UVklgAAID/IbEA4BWbzeZ5SF6Z3aJoAABAsCGxAOA1DskDAAA1IbHwUWZmpnr16qX09HSrQwECzn3EgqlQAACgEomFjzIyMrRx40YtX77c6lCAgOP0bQAAUBMSCwBeS3BLLApKWWMBAACOIrEA4DXPxduMWAAAgKNILAB4zWMqFGssAADA/5BYAPAau0IBAICakFgA8JrHrlBMhQIAAP9DYgHAa+6Lt5kKBQAAKpFYAPCa+xqLfHaFAgAA/0NiAcBrcZHmNRZMhQIAAJVILAB4LZ4D8gAAQA1ILAB4zeMci9IKGYZhUTQAACCYkFgA8Jr7iIXDaajU7rQoGgAAEExILAB4zf0cC0kqKGMBNwAAILEAUA+xkaEeZRySBwAAJBILAPUQGRaqiDDzPxucZQEAACQSCwD15H5IHiMWAABAIrEAUE/uO0MVssYCAACIxAJAPbmfvs2IBQAAkEgsANRTvNvp2yQWAABAIrFwycvL04knnqgBAwaoT58+evHFF60OCQhK7iMWnL4NAAAkKazuKs1DfHy8Fi9erJiYGBUVFalPnz6aMGGCWrRoYXVoQFCJ91hjQWIBAAAYsXAJDQ1VTEyMJKmsrEyGYcgwDIujAoKP++nbBaUs3gYAAI0osVi8eLHOO+88paWlyWaz6ZNPPvGok5mZqeOOO05RUVEaOnSoli1bVq975OXlqX///mrfvr3uuOMOtWzZ0k/RA00Hi7cBAEB1Gs1UqKKiIvXv31/XXnutJkyY4PH47NmzNXXqVM2cOVNDhw7V008/rTPPPFNbtmxR69atJUkDBgxQRYXnm6Cvv/5aaWlpSkpK0tq1a5WTk6MJEybo4osvVmpqarXxlJWVqayszHWdn58vSbLb7bLb+QQXZpV9oin0jWi3A/IKSujz/tCU+ggCgz6CutBHUJdA9w2b0Qjn+9hsNn388ccaP368q2zo0KFKT0/Xv/71L0mS0+lUhw4ddPPNN2vatGn1vsdNN92k008/XRdffHG1jz/wwAOaPn26R/nbb7/tmlIFNEXfZ9v0/vZQ13XneEO39XFYGBEAAPBGcXGxLr/8ch05ckQJCQl+b7/RjFjUpry8XCtXrtRdd93lKgsJCdGYMWO0ZMkSr9rIyclRTEyM4uPjdeTIES1evFg33nhjjfXvuusuTZ061XWdn5+vDh06aNSoUSz4hge73a558+Zp7NixCg8Pr/sJQaxibZbe3/6z6zo8Ol5nn32yhRE1DU2pjyAw6COoC30EdcnNzQ1o+00isTh48KAcDofHtKXU1FRt3rzZqzZ27typG264wbVo++abb1bfvn1rrB8ZGanIyEiP8vDwcP6YUaOm0D+SYs39vqjc0ehfUzBpCn0EgUUfQV3oI6hJoPtFk0gs/GHIkCFas2aN1WEAQS8+yvyPUj67QgEAADWiXaFq07JlS4WGhionJ8dUnpOTozZt2gT03pmZmerVq5fS09MDeh8gWMRVc45FI1yqBQAA/KxJJBYREREaPHiw5s+f7ypzOp2aP3++hg0bFtB7Z2RkaOPGjVq+fHlA7wMEC/dzLAxDKi5n8TYAAM1do5kKVVhYqF9++cV1vX37dq1Zs0YpKSnq2LGjpk6dqsmTJ+vEE0/UkCFD9PTTT6uoqEjXXHONhVEDTY97YiEdPcsiNrLR/HMCAAACoNG8E1ixYoVGjRrluq7ckWny5MmaNWuWJk6cqAMHDui+++5Tdna2BgwYoC+//LLGcygAHJvqEojCMrukqIYPBgAABI1Gk1iMHDmyznncU6ZM0ZQpUxooIqB5Cg8NUVR4iErtTlcZp28DAIAmscbCSizeRnPkvjMUiQUAACCx8BGLt9EcxVezMxQAAGjeSCwA1Fuc2wLuQkYsAABo9kgsANSb+85QHJIHAABILADUW3WH5AEAgOaNxMJHLN5Gc+S+eJupUAAAgMTCRyzeRnPkPmLBrlAAAIDEAkC9ua+xYCoUAAAgsQBQb+6JRQGJBQAAzR6JBYB6i4t0PyCPXaEAAGjuSCx8xOJtNEecYwEAANyRWPiIxdtojlhjAQAA3JFYAKi3eHaFAgAAbkgsANSbx1Sosgo5nYZF0QAAgGBAYgGg3twPyJOkonJGLQAAaM5ILADUm/sBeRLToQAAaO5ILADUW3WJBQu4AQBo3kgsfMR2s2iOQkNsio0INZVxlgUAAM0biYWP2G4WzZX7Am6mQgEA0LyRWAA4Ju7ToZgKBQBA80ZiAeCYuO8MxYgFAADNG4kFgGPicfo2iQUAAM0aiQWAY+I+FaqAqVAAADRrJBYAjon7iAW7QgEA0LyRWAA4JnGR5jUWTIUCAKB5I7HwEedYoLly326WXaEAAGjeSCx8xDkWaK4SOMcCAABUQWIB4JiweBsAAFRFYgHgmHhMhWLxNgAAzRqJBYBjwgF5AACgKhILAMfEfSoUi7cBAGjeSCwAHBP3cyyKyx1yOA2LogEAAFYjsQBwTNwTC4mzLAAAaM483xkAgBfcp0JJ0to9eWqdEGlBNI1fhb1C+4qkLdkFCgvnn2Z4oo+gLvQRuOvaOl6hIbYGux+9DsAxiY0Ik80mGVVmP131yjLrAmoSwvTouiVWB4GgRh9BXegj+N3PD5zhsdlKIDEVCsAxCQmxKS6CzyYAAMBRJBY+yszMVK9evZSenm51KECD69E23uoQAABAkODjRh9lZGQoIyND+fn5SkxMtDocoEH9/YI++vN7a7U5O9/qUJoEwzBkszXcXFg0PvQR1IU+AiuRWAA4Zj3bJmjuradaHUaTYLfbNXfuXJ199tkKD2+4+bBoPOgjqAt9BFZjKhQAAAAAn5FYAAAAAPAZiQUAAAAAn5FYAAAAAPAZiQUAAAAAn5FYAAAAAPAZiQUAAAAAn5FYAAAAAPAZiQUAAAAAn5FYAAAAAPAZiQUAAAAAn5FYAAAAAPAZiYWPMjMz1atXL6Wnp1sdCgAAAGAZEgsfZWRkaOPGjVq+fLnVoQAAAACWIbEAAAAA4DMSCwAAAAA+I7EAAAAA4DMSCwAAAAA+C7M6gKbCMAxJUkFBgcLDwy2OBsHGbreruLhY+fn59A9Uiz6CutBHUBf6COpSUFAg6ff3rf5GYuEnubm5kqTOnTtbHAkAAABQs9zcXCUmJvq9XRILP0lJSZEk7dq1KyC/KDRu+fn56tChg3bv3q2EhASrw0EQoo+gLvQR1IU+grocOXJEHTt2dL1v9TcSCz8JCTm6XCUxMZE/ZtQoISGB/oFa0UdQF/oI6kIfQV0q37f6vd2AtAoAAACgWSGxAAAAAOAzEgs/iYyM1P3336/IyEirQ0EQon+gLvQR1IU+grrQR1CXQPcRmxGo/aYAAAAANBuMWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWPhBZmamjjvuOEVFRWno0KFatmyZ1SHBIjNmzFB6erri4+PVunVrjR8/Xlu2bDHVKS0tVUZGhlq0aKG4uDhddNFFysnJsShiWOmRRx6RzWbTbbfd5iqjf2Dv3r36wx/+oBYtWig6Olp9+/bVihUrXI8bhqH77rtPbdu2VXR0tMaMGaNt27ZZGDEaksPh0L333qvOnTsrOjpaxx9/vP7xj3+o6l489JHmZfHixTrvvPOUlpYmm82mTz75xPS4N/3h0KFDuuKKK5SQkKCkpCT98Y9/VGFhYb1jIbHw0ezZszV16lTdf//9WrVqlfr3768zzzxT+/fvtzo0WODbb79VRkaGfvrpJ82bN092u11nnHGGioqKXHVuv/12ff7553r//ff17bffat++fZowYYKFUcMKy5cv1wsvvKB+/fqZyukfzdvhw4c1fPhwhYeH67///a82btyoJ598UsnJya46jz32mJ599lnNnDlTS5cuVWxsrM4880yVlpZaGDkayqOPPqrnn39e//rXv7Rp0yY9+uijeuyxx/Tcc8+56tBHmpeioiL1799fmZmZ1T7uTX+44oortGHDBs2bN09z5szR4sWLdcMNN9Q/GAM+GTJkiJGRkeG6djgcRlpamjFjxgwLo0Kw2L9/vyHJ+Pbbbw3DMIy8vDwjPDzceP/99111Nm3aZEgylixZYlWYaGAFBQVG165djXnz5hkjRowwbr31VsMw6B8wjDvvvNM45ZRTanzc6XQabdq0MR5//HFXWV5enhEZGWm88847DREiLHbOOecY1157ralswoQJxhVXXGEYBn2kuZNkfPzxx65rb/rDxo0bDUnG8uXLXXX++9//Gjabzdi7d2+97s+IhQ/Ky8u1cuVKjRkzxlUWEhKiMWPGaMmSJRZGhmBx5MgRSVJKSookaeXKlbLb7aY+06NHD3Xs2JE+04xkZGTonHPOMfUDif4B6bPPPtOJJ56oSy65RK1bt9bAgQP14osvuh7fvn27srOzTX0kMTFRQ4cOpY80EyeffLLmz5+vrVu3SpLWrl2r77//XuPGjZNEH4GZN/1hyZIlSkpK0oknnuiqM2bMGIWEhGjp0qX1ul+Yf8Jung4ePCiHw6HU1FRTeWpqqjZv3mxRVAgWTqdTt912m4YPH64+ffpIkrKzsxUREaGkpCRT3dTUVGVnZ1sQJRrau+++q1WrVmn58uUej9E/8Ntvv+n555/X1KlTdffdd2v58uW65ZZbFBERocmTJ7v6QXX/36GPNA/Tpk1Tfn6+evToodDQUDkcDj300EO64oorJIk+AhNv+kN2drZat25tejwsLEwpKSn17jMkFkCAZGRkaP369fr++++tDgVBYvfu3br11ls1b948RUVFWR0OgpDT6dSJJ56ohx9+WJI0cOBArV+/XjNnztTkyZMtjg7B4L333tNbb72lt99+W71799aaNWt02223KS0tjT4CyzEVygctW7ZUaGiox44tOTk5atOmjUVRIRhMmTJFc+bM0cKFC9W+fXtXeZs2bVReXq68vDxTffpM87By5Urt379fgwYNUlhYmMLCwvTtt9/q2WefVVhYmFJTU+kfzVzbtm3Vq1cvU1nPnj21a9cuSXL1A/6/03zdcccdmjZtmi677DL17dtXV155pW6//XbNmDFDEn0EZt70hzZt2nhsOlRRUaFDhw7Vu8+QWPggIiJCgwcP1vz5811lTqdT8+fP17BhwyyMDFYxDENTpkzRxx9/rAULFqhz586mxwcPHqzw8HBTn9myZYt27dpFn2kGRo8erZ9//llr1qxxfZ144om64oorXN/TP5q34cOHe2xRvXXrVnXq1EmS1LlzZ7Vp08bUR/Lz87V06VL6SDNRXFyskBDz27fQ0FA5nU5J9BGYedMfhg0bpry8PK1cudJVZ8GCBXI6nRo6dGj9bujT0nMY7777rhEZGWnMmjXL2Lhxo3HDDTcYSUlJRnZ2ttWhwQI33nijkZiYaCxatMjIyspyfRUXF7vq/N///Z/RsWNHY8GCBcaKFSuMYcOGGcOGDbMwalip6q5QhkH/aO6WLVtmhIWFGQ899JCxbds246233jJiYmKMN99801XnkUceMZKSkoxPP/3UWLdunXHBBRcYnTt3NkpKSiyMHA1l8uTJRrt27Yw5c+YY27dvNz766COjZcuWxl//+ldXHfpI81JQUGCsXr3aWL16tSHJeOqpp4zVq1cbO3fuNAzDu/5w1llnGQMHDjSWLl1qfP/990bXrl2NSZMm1TsWEgs/eO6554yOHTsaERERxpAhQ4yffvrJ6pBgEUnVfr366quuOiUlJcZNN91kJCcnGzExMcaFF15oZGVlWRc0LOWeWNA/8Pnnnxt9+vQxIiMjjR49ehj/+c9/TI87nU7j3nvvNVJTU43IyEhj9OjRxpYtWyyKFg0tPz/fuPXWW42OHTsaUVFRRpcuXYy//e1vRllZmasOfaR5WbhwYbXvPSZPnmwYhnf9ITc315g0aZIRFxdnJCQkGNdcc41RUFBQ71hshlHlqEYAAAAAOAassQAAAADgMxILAAAAAD4jsQAAAADgMxILAAAAAD4jsQAAAADgMxILAAAAAD4jsQAAAADgMxILAAAAAD4jsQCAABg5cqRsNpseeOABq0OxVHFxse6991717NlT0dHRstlsstlsWrNmjdWhBcwDDzwgm82mkSNHWh3KMbn66qtls9l09dVXWx0KgEaGxAJAg6l8w2Wz2RQTE6N9+/bVWHfHjh2uuosWLWq4IOFXEydO1IMPPqjNmzfLZrMpNTVVqampCg8Ptzq0ZmfRokV64IEHNGvWLKtDAdBEkVgAsERJSYmmT59udRgIoM2bN2vOnDmSpNmzZ6u4uFjZ2dnKzs5W7969LY6u+Vm0aJGmT59eZ2LRtm1bde/eXW3btm2YwAA0GSQWACzzyiuvaOvWrVaHgQD5+eefJUktWrTQpZdeanE08NaMGTO0efNmzZgxw+pQADQyJBYAGlyHDh3Ur18/VVRU6O6777Y6HARIcXGxJCkuLs7iSAAADYHEAkCDCwkJcX0a+uGHH2rZsmX1en7V9Rc7duyosd5xxx0nm83mMfXD/fk7d+7U9ddfr44dOyoqKkrHH3+87rnnHhUVFbmes379ev3hD39Qhw4dFBUVpa5du+rBBx+U3W6vM97y8nI98sgj6tevn2JjY5WcnKyxY8fqv//9b53PXb9+vW644QZ17dpVMTExiouLU79+/fS3v/1NBw8erPY57ouHP/zwQ51xxhlq3bq1QkJC6r2gvLS0VE8//bROPvlkJScnKyoqSp06ddJVV11V7SLsyvtXLv7duXOn6+d9rIuCf/jhB/3hD39Qp06dFBUVpcTERA0ZMkSPPvqoCgsLTXXtdrtatmwpm82mZ599ttZ2X3nlFdlsNiUkJLgSIUnKzs7Wc889pwsuuEA9e/ZUYmKioqOjdcIJJ+i6667Thg0b6v0aJO8W9de2+Pvw4cN6+eWXdemll6pv375KSUlx/T4uv/xy/fTTTx7PqezvlVMPv/32W9Pvw/1vxJvF24sWLdIll1yidu3aKTIyUi1bttTo0aP16quvyuFwePW65s+fr3POOUetWrVSVFSUevbsqenTp6u0tLTG+3711VeaMGGC2rdvr4iICCUkJKhLly4644wz9MQTT+jQoUM1PhdAAzAAoIHcf//9hiSjU6dOhmEYxogRIwxJxqhRozzqbt++3ZBkSDIWLlxY42Pbt2+v8X6dOnUyJBmvvvpqjc//8MMPjaSkJEOSkZCQYISGhroeO/XUU43y8nJjzpw5RkxMjCHJSExMNGw2m6vOxIkTq7135Wu76667jFNPPdWQZISFhbnuVfl1//331xj/o48+aoSEhLjqxsTEGBEREa7rtm3bGqtWrarx5zxixAhj6tSphiTDZrMZycnJRmhoaK33dLdnzx6jT58+rnuGh4cbiYmJruuQkBDj2WefNT3n8ccfN1JTU42EhARXndTUVNfXLbfc4vX9HQ6Hccstt5h+ZnFxcabfU/fu3Y0dO3aYnpeRkWFIMk488cRa2x85cqQhybj66qtN5ZMnT3a1HxYWZqSkpBhhYWGussjISOODDz6ots2qP393lf2itt9Bbc+vfEySERoaaiQnJxuRkZGuMpvNZjzzzDOm5+zatctITU01YmNjXb/Dqr+P1NRU49133/V47ZMnT642vttvv910v6SkJNPv4/TTTzfy8/NrfV2PPfaYYbPZXM+v+jc1atQoo6KiwuP506dPN/WDmJgYIy4uzlTm/m8FgIZFYgGgwbgnFkuWLHG9Ifjvf/9rqttQiUVSUpIxevRoY8OGDYZhGEZxcbHx7LPPut4o3XPPPUZiYqIxceJE15vXgoIC429/+5urjXnz5nncu/INZGJiohEZGWnMnDnTKCkpMQzj6Bu9iy++2PX8Tz/91OP5L730kutN9EMPPWRkZWUZhmEYFRUVxooVK4zTTz/dkGS0b9/eKCgoqPbnXPmm68477zT2799vGIZhlJaWerwJr0lFRYUxdOhQ1+t48803jbKyMsMwDOPXX381zj33XNeby7lz53o8/9VXXzX9vo/FPffcY0gyWrdubWRmZhq5ubmGYRhGeXm5sXDhQmPgwIGGJGPQoEGGw+FwPW/p0qWun++mTZuqbXvnzp2uN7QLFiwwPfaPf/zDePzxx42ff/7ZsNvthmEcTXLWr19vXHHFFYYkIzY21ti7d69Hu4FMLF544QXj/vvvN1asWOH6XTidTuO3334zbr31VsNmsxmhoaF1Jpy1qS2xeO6551w/1xtuuMHVLwsLC41//vOfruSruoS78v5JSUlGSEiIcddddxkHDhwwDMMwjhw5Ytx3332utl9++WXTc3fs2OFKsqdOnWr6uefl5RnfffedcdNNNxkrVqyo9bUBCCwSCwANxj2xMAzDuPDCCw1JxoABAwyn0+kqb6jEonfv3kZpaanHc6+88kpXnbFjx5piq1Q5EvHHP/7R47HKN5DVvUkyjKNvUk877TRXDFXl5+e7Rja+/PLLal+b3W43Bg8ebEgy/vnPf5oeq/qp9tSpU6t9vjfeffddVztfffVVtTFUJh59+vTxeNzXxGL79u1GaGioER0dbaxZs6baOvn5+Ub79u0NScbHH39seqx79+6uUaPqPPzww4Yko2PHjtX+fmtzzjnnGJKMf/zjHx6PBTKxqEvlSE11fdLXxKK4uNhISUkxJBmTJk2q9rnPPvusq8+4v8mv2i9rev0TJkwwJBljxowxlc+ePduQZHTr1q3W2AFYizUWACz18MMPKzQ0VGvWrNE777zT4Pe//fbbFRkZ6VF+5plnur6fNm2abDZbjXXWrVtXY/sdOnTQNddc41EeEhKie+65R5K0YcMG1w5K0tE1EXl5eRo4cKApjqrCwsI0adIkSUfnnVcnJCREd955Z42x1WX27NmSpGHDhumMM86oNob7779f0tG1IFVfgz/MmjVLDodDZ511lvr3719tnfj4eI0fP16S58/hyiuvlCS99dZbMgzD47lvvPGGJOmKK66o9vdbm3POOUeS9P3339freYEWyLjmzZvnWsNQ0xqRm266ybVN7dtvv11tncjISP3lL3+p9rELLrhAkuffVFJSkiSpoKDAtPYJQHAhsQBgqR49erjeeN97771eLYb2pyFDhlRbnpqa6vo+PT291jqHDx+usf3KxbrVOfXUUxUWFiZJWrFihav8hx9+kCRt2rRJbdq0qfHr73//u6Sji6Orc8IJJ6h169Y1xlaXypjGjBlTY51Ro0YpNDTU4zX4Q+XP4euvv6715/Dqq69K8vw5XHnllbLZbNq1a5e+/fZb02MrV67Upk2bJElXXXVVtfdfu3atbrrpJvXr108JCQkKCQlxLXa+6aabJEl79uzx62v2xm+//aa//OUvGjx4sJKSkhQaGuqK6+yzzw5YXJW/3w4dOqhbt27V1gkNDdXpp59uqu+ud+/eNe4UlpaWJkkei7CHDBmili1bKisrS0OHDtW//vUvbd68udqEEYB1wqwOAAAeeOABvfXWW/rtt980c+ZM3XzzzQ127/j4+GrLK9/we1OntmSoXbt2NT4WFRWlFi1aKCcnR/v373eVV55IXlpaWusOOZWq7mZUlS9JhSRXTHW9hpYtW3q8Bn+o/DkUFRV59Sm1+8+hY8eOGjFihBYtWqQ33njDtMtS5WhFenq6evTo4dHWv/71L916661yOp2SJJvNpsTERNfoVklJifLz8xv80/OPP/5YkyZNUllZmassISFBUVFRstlsKi8v1+HDhwMSlzf9QZLat29vqu+upr8n6fe/qYqKClN5UlKS3nnnHV1++eXasGGD69+IxMREnXbaabr00ks1ceJETnQHLMaIBQDLtWvXzvVG4cEHH/TYPrS5qdyuc+LEiTKOroWr9aumLXcrRxIaq8qfw5133unVz2HRokUebVSORnzwwQcqKSmRdPRNa+W0u8rpUlVt2rRJt912m5xOpy655BItW7ZMpaWlOnz4sOvk8KeeekqSGvQT89zcXF199dUqKyvT6aefrkWLFqm4uFhHjhxRTk6OsrOz9f777zdYPA1tzJgx2r59u15//XVNnjxZXbt21ZEjR/T555/ryiuv1MCBA7V3716rwwSaNRILAEFh2rRpSk5O1v79+/Xkk0/WWrfqaEJtn+gfOXLEb/Edq9re6JSVlSk3N1eSeXShTZs2kmqe4tRQKmOqbVpNaWlpta/BH/zxc7j44osVHR2t/Px8ffrpp5KOTq3av3+/wsPDXetUqvrggw/kcDjUs2dPvfvuu0pPT1dERISpTnZ29jHFU9l3j6Xfzp07V/n5+UpOTtbnn3+uESNGKDo62i9xecOb/lD1cX/3B0mKjY3VlVdeqVmzZmnr1q3as2ePHn30UUVFRZlGMgBYg8QCQFBITk7WtGnTJElPPvmkDhw4UGvdSrt37662ztatW5WXl+fXGI/Ft99+W+On2t99951ryseJJ57oKh8+fLiko+sAsrKyAh9kDSpjmj9/fo11Fi1a5HoNNa1FOVaVP4dvvvnGqylh1am6uLty+lPlf8eNG6eWLVt6PKeyT/Xv318hIdX/b/Kbb745pngq+25N/VaSli5dWm155XO6d++umJiYesdV+VqOdZSlsj/s2bNHW7durbaOw+HQwoULJfm/P1SnXbt2+utf/6o///nPko4uMAdgHRILAEHj5ptvVvv27VVQUKB//OMfNdaLjY3V8ccfL+noDkrVeeihhwISY33t2rVLr732mke50+nUww8/LEnq1auX+vbt63rskksuUVJSkux2u6ZOnVrrG0Gn0xmwBOqyyy6TJC1ZskRff/21x+MVFRWuBeR9+vRRnz59/Hr/a6+9VmFhYTp48KBr96malJeX1ziFrnI61Ndff61t27a5Ri5qWrSdmJgoSfr555+r/dn/97//rXbalTcqd7f66quvql0HsWDBAi1ZsqTWuLZu3VptorVmzZoad2KSjq7FkHTM/WXs2LFq0aKFpJp3hXrhhRdca2OqGw06VlXXlFSncuSmpkQQQMPgLxBA0IiOjna9Yfn8889rrVv5puWVV17Rv//9b9f8+d27d+u6667T7Nmza/xUtyElJibqxhtv1Isvvuh6M7h7925NmjTJ9cnugw8+aHpOUlKSnn76aUnSu+++q3POOUdLly51LSR2Op3atGmTnnzySfXu3Vtz5swJSOwXXXSRhg4dKkm69NJL9fbbb7sWqm/fvl0XXXSR603wY4895vf7H3/88br33ntd7V911VVav3696/GKigqtWbNGf//733XCCSdozZo11bYzduxYtWnTRhUVFbr88stVUlKi5ORknXvuudXWP+ussyQd3QY4IyPDtUNRUVGRXnjhBV188cWuN9j1demllyokJES5ubmaNGmSa9pQSUmJXnvtNV144YVKSUmp9rlnnHGGQkJCdOjQIV1xxRWuaXbl5eV67733dMYZZ9S6MLoy8duwYYN+/PHHesde9e/znXfe0f/93/8pJydH0tGF888++6xuu+02SUfXBw0ePLje96jJo48+qnHjxumNN94wTcUqKyvTe++9p8cff1zS79vtArBIg52YAaDZq+6APHcVFRVGjx49XAdpqZoD8gzj6OnXvXr1ctUJCQlxHSoXHh5uvPPOO14dkFfTAXsLFy501alJbQfAVR6EdtdddxmnnHKKK67k5GTTa7vnnntqbP/55583IiIiXHUjIyONFi1aGOHh4aY23nzzTdPzfDlgzd2ePXuM3r17u+4VERHh+jlX/tyfeeaZap/rj5O3nU6nce+997pOyJZkREdHGy1atHCdjl759f3339fYztSpU011//SnP9V638suu8xUPykpyXW/wYMHu06gru611fXzr3rCtP53qnnlidXjx493nTZe3fPvvPNOj+dW9ofOnTsbb731Vo391m63uw4NlGQkJycbnTp1Mjp16mS8//77rnq1nbxtGIZx++23u9qw2WxGcnKyK35JxqhRo4z8/Px6/1wMo+a/u6qH61X2gZSUFFO/6Nmzp+skcADWYMQCQFAJDQ11TRGqTVxcnL7//ntNnTpVnTt3VlhYmMLDw12foldO47FaRESE5s+fr4cffljdu3dXWVmZEhMTNXr0aH3xxRe1Tvn6v//7P23ZskV/+ctf1L9/f0VGRiovL09xcXE68cQTdfPNN2vevHl+nXLirl27dlqxYoWeeuopnXTSSYqOjlZxcbE6dOigK6+8UitXrtQtt9wSsPvbbDb9/e9/17p163TTTTepZ8+eCg0N1ZEjR5ScnKyTTz5Zd9xxh3788UfXmozquE97qmkaVKW33npLTz/9tPr166fIyEg5HA717dtXM2bM0A8//FDjOQzemD59ut544w2ddNJJio2NlcPh0IABAzRz5kx99NFHte7m9cgjj+j111/XkCFDFB0dLbvdrhNOOEF33323Vq9e7ToHojphYWGaP3++rrvuOnXu3FlFRUXauXOndu7cWa+d2J566iktWLBAF110kVJTU1VYWKj4+HiNGjVKr7zyiubNm1fryMmxuOGGG/Sf//xHkyZNUp8+fRQTE+NayH7qqafq6aef1qpVq1wL/gFYw2YYnC4DAAAAwDeMWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+RWAAAAADwGYkFAAAAAJ+FWR0AUF+GYchut8vpdFodCgAAARESEqLw8HDZbDarQwG8RmKBRsPhcOjgwYMqKCiQ3W63OhwAAAIqPDxc8fHxatmypUJDQ60OB6iTzTAMw+oggLo4HA7t3r1bZWVlSkxMVFxcnEJDQ/kkBwDQ5BiGIYfDocLCQh05ckSRkZHq0KEDyQWCHokFGoWcnBzl5eWpY8eOio6OtjocAAAaRElJiXbt2qWkpCSlpqZaHQ5QKxZvI+gZhqGCggIlJiaSVAAAmpXo6GglJCSooKBAfBaMYEdigaBnt9tlt9sVFxdndSgAADS4+Ph41/8LgWBGYoGgV7n7E3NLAQDNUeX//9gNEcGOxAKNBgu1AQDNEf//Q2NBYgEAAADAZyQWAAAAAHxGYgEAAADAZyQWQBNx3HHHyWazeXzFxcWpf//+uuuuu5Sbm1tnO7fccovruZ9//rnf43zggQdks9k0cuTIWustWrTIFUd1Vq1apWuvvVYnnHCCoqOjFRMTo06dOmn48OH6y1/+onnz5vk9djSsmvq0+9esWbM8nlO1DI3Xtm3bNGXKFPXq1UuxsbGKiopS+/btlZ6erilTpujDDz/0+R6V/9bU9W+Sv8yaNUs2m01XX311g9wPaEhhVgcAwL+GDx+uE044QdLRHUT27dunH3/8UY888ohef/11fffdd+rSpUu1zy0rK9Nbb73lun7llVd03nnnNUjc9fHcc8/ptttuk9PpVLt27TRq1CglJyfrwIEDWrVqlX788UctWrRIY8eOtTpU+EHVPl2d2h5D4/XRRx/p8ssvV1lZmVq0aKHhw4erVatWOnz4sNasWaPMzEy9++67uuiii6wOFcD/kFgATcx1113n8UlYdna2RowYoa1bt+qvf/2rPvjgg2qf+/HHH+vQoUNKS0tTVlaW5syZo5ycnKA67XXdunWupOKf//ynbr75ZtNWxE6nU99//72+//57C6OEP1XXp9G05eTkaPLkySorK9Of//xnPfjgg4qKijLVWblyZY3/lgGwBlOhgGagTZs2uuOOOyRJ8+fPr7Heyy+/LEm69dZbNWLECFVUVOj1119vkBi99f7778vpdGrYsGG67bbbPM43CQkJ0Wmnnaa7777boggB+GrOnDkqLCxUWlqannjiCY+kQpIGDx6sGTNmWBAdgJowYoFGzek0dLi43OowfJYcE6GQkMDuU96mTRtJUkVFRbWP79ixQ/Pnz1dYWJiuuuoqpaWladGiRXrllVdcSUlVv/32mwYNGqT8/Hx98cUXGjdunOnxffv2acCAATpw4IDeffddTZw40S+vIycnR5LUunVrv7QXdJxOqeSQ1VH4JjpFCuFzq/pyGk7lleVZHYZPkiKTFGLz/Xdf+XfeqlWrej/30KFDeuKJJ/Tpp59q+/btCg0NVbdu3TRx4kTdfPPNio6OrvG5xcXFevDBB/Xee+9pz549SklJ0bhx4/T3v/9d7dq1q/Y5mzdv1qOPPqoFCxYoOztbsbGxGjhwoP70pz/p0ksvrXf8QGNGYoFG7XBxuQY/+I3VYfhs5T1j1CIuMqD3WLZsmSSpd+/e1T7+yiuvyDAMnX322WrTpo0uuugiTZkyRZs3b9aPP/6ok08+2VS/S5cueuWVV3TRRRfpqquu0urVq9W+fXtJksPh0GWXXaYDBw7opptu8ltSIUkdO3aUdHTkZf369erTp4/f2g4KJYekx4+3Ogrf3PGrFNvS6iganbyyPI2YPcLqMHzy7cRvlRKV4nM7lX/n69ev1/z58zV69Givnvfbb7/p9NNP186dO9WqVSudffbZstvtWrhwoe68807Nnj1b33zzjZKTkz2eW15ertGjR2vdunUaOXKkBg0apO+//16vvPKK5s6dq8WLF6tr166m53zxxRe6+OKLVVpaqu7du2vChAnav3+/vv32Wy1YsEBfffWVayQYaA74SAlowpxOp/bu3at//etfeuyxxxQaGqp77rmn2nqVu+hce+21kqTo6GhddtllklTj/xgnTJigW2+9VQcPHtRll13mGg3529/+pu+++06DBg3SU0895dfXNHnyZMXHx6uwsFADBw7UOeeco8cee0zffPONjhw54td7AbDG+PHj1a5dOzkcDo0dO1ajRo3Sgw8+qLlz5+rAgQM1Pu/yyy/Xzp07df7552v79u364IMP9Omnn+rXX3/VoEGDtGrVKk2ZMqXa5y5ZskQHDx7Upk2b9MUXX+i9997Tb7/9posuukjZ2dm66qqrTPVzcnJ0xRVXqLS0VA8++KA2bdqkd955R/Pnz9dPP/2k5ORkvfLKK3rxxRf9+rMBghmJBdDEXHPNNa5tOENDQ9W+fXvdfPPN6tevn7799lude+65Hs/5+uuvtXv3bqWmpuqcc85xlf/xj3+UJL333nsqLCys9n6PP/64hg4dqh9++EF/+9vfNHfuXD322GNKTEzU+++/r8hI/47EdOjQQV9//bV69OihiooKzZ07V3feeafGjh2rlJQUDR8+XLNnz/brPWGtqn26uq+8vDyrQ4SfxcXFaf78+Ro6dKgMw9CiRf/f3r2HNPX+cQB/7+Jys77tK7XsIiYRVgah1bpSKb9EKKp/Qkpp3aREuhcRWkFFF7oRWXSzcFEpUdhNhaKUbGbxmxpl9wuVdtFMJWfltuf3h+zk3Kz4rdTp+wUHZOc553x2cDv7nPM8nycX69evx5QpU6DT6RAWFoZDhw7BZrNJ2+Tn56OwsBAajQZHjhyBn5+ftK5nz544cuQIACA9PR1v3751e9xdu3ZJT0sAwNfXFwcPHoRGo8Ht27dhMpmkdUePHkVNTQ2GDx+OpKQkp9LYI0aMQFJSEoDG70iizoJdoYg6mOalOSsrK3Hv3j3cvXsXK1aswKlTp1we5x87dgwAMGfOHCiVP74WRo4ciaFDh+L+/fvIyMiQEo2mfHx8kJGRgfDwcOzcuROHDh2CEAKpqaktlrX11OjRo/HgwQPk5eUhJycHd+/ehdlsRk1NDUwmE0wmE7KzszmXQQfxq3KzKpWqFaOh1hISEoLbt2/jzp07uHLlCgoLC2E2m1FRUYHi4mIkJCTg3LlzuHLlClQqFXJzcwEA0dHRbivZDR8+HMOGDUNJSQny8vIQGxvrtF6r1WLatGku2+l0OkRHR+P8+fPIzc2VuoU6jmcwGNzGv2DBAqxevRpPnz5FeXk5+vTp48HZIPIOTCzIq/2rUeG/yf9p6zA89q/mz/0wclea02q1YsOGDdi2bRsmTpyIx48fo1u3bgCAiooKXLx4EcCPblBNzZ8/HytXrsTx48fdJhYAEBQUhP379yM2Nha1tbVISEhosba8466eEOKn7+NX6+VyOSIiIhAREQGgcVxHQUEBNm3ahKtXryItLQ1TpkzBzJkzf7qfdkft3zhGwZupPe9j31RnKTer7aJFXkxeW4fhEW0X7R/fp16vh16vB9D4vVBUVISdO3ciPT0d165dw759+7BmzRqUlZUBAIKDg1vc14ABA1BSUiK1bcoxuaI7jn02fdLxq+NptVr4+/ujqqoKb9++ZWJBnQITC/Jqcrnsrw967giUSiW2bNmCo0eP4t27dzAajUhMTAQAnDx5Eg0NDVAqlVi4cKHLto4uUCaTCY8ePcKgQYNc2gghnCbWM5vNaGhogI+Pj0tbR/eEurq6n8bsOG7Xrl1/6z0qFAqMHz8e2dnZ0Ov1MJvNyMzM9L7EQi7nwOdOSi6T/5GBzx2ZTCZDeHg4zpw5A4vFgosXLyIzM9Nt5bq/4Vc3PIg6O46xIOok5HI5+vfvDwB4+PCh9LpjYLbVasWtW7dclpKSEpe2ze3YsQNZWVkYPHgwxowZg8LCQqxdu9ZtW0f/5efPn//0Iv306VOn9r9LoVAgMjISQGM3MCLqmKKiogD8+Jw7ysG+ePGixW0c69yVjn316lWL2znWOSrf/c7xampqUFVV1eLxiDoiJhZEnYTdbpcujo6nAAUFBSgtLUWXLl3w+fNnCCHcLllZWQAan240nwfj5s2bSE5OhkajwdmzZ5GRkQF/f3/s3bsXFy5ccIljwoQJUCqVqK6uxvXr11uM1zGjriNJcPidO4avX78G4PwjgIi8x//zOZ80aRIAICcnR5oHo6mioiIUFxdLk2g2V11djUuXLrm8XlFRgZycHKdjNP07LS3NbXzHjx8HAAwcOJCJBXUaTCyIOgGr1Yrk5GTpzp5jgKLjCcT06dOh1Wpb3D4qKgoBAQH48OEDLl++LL1eUVGBWbNmwWaz4cCBAwgNDUVgYCDS0tIgk8kwb948l7uAAQEB0mDHxYsX48mTJy6xbty4EQUFBfD19cWyZcuc1iclJWHJkiW4d++e2/d5+PBhKSlxlMslIu9y8OBBGAwGpypMDkIInD9/HikpKQB+fM7Hjx+PUaNGob6+HosWLYLFYpG2qaysxKJFi6T2gYGBbo+7atUqp3EU3759Q2JiIurq6qDX6zFu3DhpXXx8PP755x+YzWZs3brVKRkqKirCli1bAKDVumkRtQccY0HUwRw7dkyqVgIAnz59QklJCd68eQOg8Yf52LFj8eXLF6ksa0tVTRwUCgVmz56NPXv2IDU1FTNmzIDdbkdcXBzKyspgMBicBtdOnToVK1euxO7duxETE4P8/Hyn8Rb79u3D8+fPkZubi9DQUIwaNQpBQUGwWCy4c+cOysvLoVarYTQaXaoBWSwWpKSkICUlBX379sWwYcOg1Wql9/n+/XsAwLp16zB58mRPTiW1E83/p5uLiorC7NmzWy8g+usaGhpgNBphNBrRs2dPhIWFoUePHqiurkZpaal0wyIuLs6pqMTp06cRGRmJCxcuIDg4GBMmTJAmyKutrUV4eLiUkDQ3ZswY2O12hISEIDIyEhqNBvn5+SgvL4dOp4PRaHRq36tXL5w6dQozZ85EUlISTp48ibCwMGmCPKvVinnz5iE+Pv6vnSeidkcQtXP19fWitLRU1NfXt3Uo7VpQUJAA4LKoVCoRFBQkYmJixI0bN6T2qampAoAICAgQVqv1l/svLi4WAIRCoRBlZWVi8+bNAoAYMmSIqKurc2n//ft3MXr0aAFALF++3GW91WoVaWlpIjo6Wuh0OqFUKkXXrl1FaGioWLp0qXj27JnbOCorK0V6erqIj48X4eHhonfv3kKpVAo/Pz8xaNAgMX/+fGEymX7/xFG71dL/dPNl2bJlLtucOHGizeImz9XW1orMzEyxZMkSodfrRb9+/YSPj49Qq9ViwIABYtasWSI7O9vttp8+fRLr1q0TgwcPFr6+vkKj0YiwsDCxfft2YbFYXNrfuHFDABATJ04UX758EWvWrBHBwcFCpVKJXr16iblz54rXr1+3GGtpaakwGAxSjFqtVkRERIj09HS37U+cOCEACIPB8Nvng9dB8hYyIVjigNq3r1+/4uXLlwgODoavr29bh0NERNSqeB0kb8ExFkRERERE5DEmFkRERERE5DEmFkRERERE5DEmFkRERERE5DEmFkRERERE5DEmFkRERERE5DEmFkRERERE5DEmFuQ1OOUKERF1Rrz+kbdgYkHtnlKpBAB8+/atjSMhIiJqfY7rn+N6SNReMbGgdk+pVMLPzw9VVVWw2WxtHQ4REVGrsdlsqKqqgp+fHxMLavdkgs/XyAtYLBa8efMGCoUC3bt3h1qthkKhgEwma+vQiIiI/ighBGw2G+rr61FTUwO73Y7AwECo1eq2Do3op5hYkNf4/v07Pn78CIvFwicXRETU4SkUCmg0Guh0OqhUqrYOh+iXmFiQ1xFCoKGhAXa7va1DISIi+ivkcjl8fHz4ZJ68ChMLIiIiIiLyGAdvExERERGRx5hYEBERERGRx5hYEBERERGRx5hYEBERERGRx5hYEBERERGRx5hYEBERERGRx5hYEBERERGRx/4HwokjdsvGjRIAAAAASUVORK5CYII=", + "text/plain": [ + "
" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "python3", - "language": "python", - "name": "python3" + }, + "metadata": {}, + "output_type": "display_data" } + ], + "source": [ + "%matplotlib inline\n", + "\n", + "names = [\"BAxUS\", \"EI\", \"Sobol\"]\n", + "runs = [Y_baxus, Y_ei, Y_Sobol]\n", + "fig, ax = plt.subplots(figsize=(8, 6))\n", + "\n", + "for name, run in zip(names, runs):\n", + " fx = np.maximum.accumulate(run.cpu())\n", + " plt.plot(-fx + branin.optimal_value, marker=\"\", lw=3)\n", + "\n", + "plt.ylabel(\"Regret\", fontsize=18)\n", + "plt.xlabel(\"Number of evaluations\", fontsize=18)\n", + "plt.title(f\"{dim}D Embedded Branin\", fontsize=24)\n", + "plt.xlim([0, len(Y_baxus)])\n", + "plt.yscale(\"log\")\n", + "\n", + "plt.grid(True)\n", + "plt.tight_layout()\n", + "plt.legend(\n", + " names + [\"Global optimal value\"],\n", + " loc=\"lower center\",\n", + " bbox_to_anchor=(0, -0.08, 1, 1),\n", + " bbox_transform=plt.gcf().transFigure,\n", + " ncol=4,\n", + " fontsize=16,\n", + ")\n", + "plt.show()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, - "nbformat": 4, - "nbformat_minor": 4 + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 }