diff --git a/README.md b/README.md index 36a3365e5..d9e9347b2 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ For information on use cases and background material on causal inference and het - [Interpretability](#interpretability) - [Causal Model Selection and Cross-Validation](#causal-model-selection-and-cross-validation) - [Inference](#inference) + - [Policy Learning](#policy-learning) - [For Developers](#for-developers) - [Running the tests](#running-the-tests) - [Generating the documentation](#generating-the-documentation) @@ -162,6 +163,25 @@ To install from source, see [For Developers](#for-developers) section below. +
+ Dynamic Double Machine Learning (click to expand) + + ```Python + from econml.dynamic.dml import DynamicDML + # Use defaults + est = DynamicDML() + # Or specify hyperparameters + est = DynamicDML(model_y=LassoCV(cv=3), + model_t=LassoCV(cv=3), + cv=3) + est.fit(Y, T, X=X, W=None, groups=groups, inference="auto") + # Effects + treatment_effects = est.effect(X_test) + # Confidence intervals + lb, ub = est.effect_interval(X_test, alpha=0.05) + ``` +
+
Causal Forests (click to expand) diff --git a/doc/reference.rst b/doc/reference.rst index c7f9f3ca7..76b528925 100644 --- a/doc/reference.rst +++ b/doc/reference.rst @@ -104,6 +104,21 @@ Sieve Methods econml.iv.sieve.HermiteFeatures econml.iv.sieve.DPolynomialFeatures +.. _dynamic_api: + +Estimators for Dynamic Treatment Regimes +---------------------------------------- + +.. _dynamicdml_api: + +Dynamic Double Machine Learning +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. autosummary:: + :toctree: _autosummary + + econml.dynamic.dml.DynamicDML + .. _policy_api: Policy Learning diff --git a/doc/spec/estimation/dml.rst b/doc/spec/estimation/dml.rst index 3032f72c2..0ffe0718a 100644 --- a/doc/spec/estimation/dml.rst +++ b/doc/spec/estimation/dml.rst @@ -50,6 +50,7 @@ characteristics :math:`X` of the treated samples, then one can use this method. .. testsetup:: + # DML import numpy as np X = np.random.choice(np.arange(5), size=(100,3)) Y = np.random.normal(size=(100,2)) @@ -71,8 +72,9 @@ Most of the methods provided make a parametric form assumption on the heterogene linear on some pre-defined; potentially high-dimensional; featurization). These methods include: :class:`.DML`, :class:`.LinearDML`, :class:`.SparseLinearDML`, :class:`.KernelDML`. -For fullly non-parametric heterogeneous treatment effect models, checkout the :class:`.NonParamDML` -and the :class:`.CausalForestDML`. For more options of non-parametric CATE estimators, +For fullly non-parametric heterogeneous treatment effect models, check out the :class:`.NonParamDML` +and the :class:`.CausalForestDML`. +For more options of non-parametric CATE estimators, check out the :ref:`Forest Estimators User Guide ` and the :ref:`Meta Learners User Guide `. diff --git a/doc/spec/estimation/dynamic_dml.rst b/doc/spec/estimation/dynamic_dml.rst new file mode 100644 index 000000000..b92f319d8 --- /dev/null +++ b/doc/spec/estimation/dynamic_dml.rst @@ -0,0 +1,95 @@ +.. _dynamicdmluserguide: + +=============================== +Dynamic Double Machine Learning +=============================== + +What is it? +================================== + +Dynamic Double Machine Learning is a method for estimating (heterogeneous) treatment effects when +treatments are offered over time via an adaptive dynamic policy. It applies to the case when +all potential dynamic confounders/controls (factors that simultaneously had a direct effect on the adaptive treatment +decision in the collected data and the observed outcome) are observed, but are either too many (high-dimensional) for +classical statistical approaches to be applicable or their effect on +the treatment and outcome cannot be satisfactorily modeled by parametric functions (non-parametric). +Both of these latter problems can be addressed via machine learning techniques (see e.g. [Lewis2021]_). + + +What are the relevant estimator classes? +======================================== + +This section describes the methodology implemented in the class +:class:`.DynamicDML`. +Click on each of these links for a detailed module documentation and input parameters of each class. + + +When should you use it? +================================== + +Suppose you have observational (or experimental from an A/B test) historical data, where multiple treatment(s)/intervention(s)/action(s) +:math:`T` were offered over time to each of the units and some final outcome(s) :math:`Y` was observed and all the variables :math:`W` that could have +potentially gone into the choice of :math:`T`, and simultaneously could have had a direct effect on the outcome :math:`Y` (aka controls or confounders) are also recorder in the dataset. + +If your goal is to understand what was the effect of the treatment on the outcome as a function of a set of observable +characteristics :math:`X` of the treated samples, then one can use this method. For instance call: + +.. testsetup:: + + # DynamicDML + import numpy as np + groups = np.repeat(a=np.arange(100), repeats=3, axis=0) + W_dyn = np.random.normal(size=(300, 1)) + X_dyn = np.random.normal(size=(300, 1)) + T_dyn = np.random.normal(size=(300, 2)) + y_dyn = np.random.normal(size=(300, )) + +.. testcode:: + + from econml.dynamic.dml import DynamicDML + est = DynamicDML() + est.fit(y_dyn, T_dyn, X=X_dyn, W=W_dyn, groups=groups) + + +Class Hierarchy Structure +================================== + +In this library we implement variants of several of the approaches mentioned in the last section. The hierarchy +structure of the implemented CATE estimators is as follows. + + .. inheritance-diagram:: econml.dynamic.dml.DynamicDML + :parts: 1 + :private-bases: + :top-classes: econml._OrthoLearner, econml._cate_estimator.LinearModelFinalCateEstimatorMixin + +Below we give a brief description of each of these classes: + + * **DynamicDML.** The class :class:`.DynamicDML` is an extension of the Double ML approach for treatments assigned sequentially over time periods. + This estimator will adjust for treatments that can have causal effects on future outcomes. The data corresponds to a Markov decision process :math:`\{X_t, W_t, T_t, Y_t\}_{t=1}^m`, + where :math:`X_t, W_t` corresponds to the state at time :math:`t`, :math:`T_t` is the treatment at time :math:`t` and :math:`Y_t` is the observed outcome at time :math:`t`. + + The model makes the following structural equation assumptions on the data generating process: + + .. math:: + + XW_t =~& A \cdot T_{t-1} + B \cdot XW_{t-1} + \eta_t\\ + T_t =~& p(T_{t-1}, XW_t, \zeta_t) \\ + Y_t =~& \theta_0(X_0)'T_t + \mu'XW_t + \epsilon_t + + where :math:`XW` is the concatenation of the :math:`X` and :math:`W` variables. + For more details about this model and underlying assumptions, see [Lewis2021]_. + + To learn the treatment effects of treatments in the different periods on the last period outcome, one can simply call: + + .. testcode:: + + from econml.dynamic.dml import DynamicDML + est = DynamicDML() + est.fit(y_dyn, T_dyn, X=X_dyn, W=W_dyn, groups=groups) + + + +Usage FAQs +========== + +See our FAQ section in :ref:`DML User Guide ` diff --git a/doc/spec/estimation_dynamic.rst b/doc/spec/estimation_dynamic.rst new file mode 100644 index 000000000..6e7b47cc0 --- /dev/null +++ b/doc/spec/estimation_dynamic.rst @@ -0,0 +1,11 @@ +Estimation Methods for Dynamic Treatment Regimes +================================================ + +This section contains methods for estimating (heterogeneous) treatment effects, +even when treatments are offered over time and the treatments were chosen based on a dynamic +adaptive policy. This is referred to as the dynamic treatment regime (see e.g. [Hernan2010]_) + +.. toctree:: + :maxdepth: 2 + + estimation/dynamic_dml diff --git a/doc/spec/references.rst b/doc/spec/references.rst index 0692af351..5f0213ac9 100644 --- a/doc/spec/references.rst +++ b/doc/spec/references.rst @@ -113,4 +113,14 @@ References .. [Lundberg2017] Lundberg, S., Lee, S. (2017). A Unified Approach to Interpreting Model Predictions. - URL https://arxiv.org/abs/1705.07874 \ No newline at end of file + URL https://arxiv.org/abs/1705.07874 + +.. [Lewis2021] + Lewis, G., Syrgkanis, V. (2021). + Double/Debiased Machine Learning for Dynamic Treatment Effects. + URL https://arxiv.org/abs/2002.07285 + +.. [Hernan2010] + Hernán, Miguel A., and James M. Robins (2010). + Causal inference. + URL https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/ diff --git a/doc/spec/spec.rst b/doc/spec/spec.rst index 693854193..649dba98c 100644 --- a/doc/spec/spec.rst +++ b/doc/spec/spec.rst @@ -19,6 +19,7 @@ The EconML Python SDK, developed by the ALICE team at MSR New England, incorpora comparison estimation estimation_iv + estimation_dynamic inference interpretability references diff --git a/econml/_cate_estimator.py b/econml/_cate_estimator.py index d1f43e2aa..d879608ae 100644 --- a/econml/_cate_estimator.py +++ b/econml/_cate_estimator.py @@ -563,7 +563,7 @@ def effect(self, X=None, *, T0, T1): """ Calculate the heterogeneous treatment effect :math:`\\tau(X, T0, T1)`. - The effect is calculatred between the two treatment points + The effect is calculated between the two treatment points conditional on a vector of features on a set of m test samples :math:`\\{T0_i, T1_i, X_i\\}`. Since this class assumes a linear effect, only the difference between T0ᵢ and T1ᵢ matters for this computation. diff --git a/econml/_ortho_learner.py b/econml/_ortho_learner.py index 9ccdd88b3..86e0404bf 100644 --- a/econml/_ortho_learner.py +++ b/econml/_ortho_learner.py @@ -685,7 +685,8 @@ def fit(self, Y, T, X=None, W=None, Z=None, *, sample_weight=None, freq_weight=N nuisances=nuisances, sample_weight=sample_weight, freq_weight=freq_weight, - sample_var=sample_var) + sample_var=sample_var, + groups=groups) return self @@ -770,18 +771,19 @@ def _fit_nuisances(self, Y, T, X=None, W=None, Z=None, sample_weight=None, group return nuisances, fitted_models, fitted_inds, scores def _fit_final(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, - freq_weight=None, sample_var=None): + freq_weight=None, sample_var=None, groups=None): self._ortho_learner_model_final.fit(Y, T, **filter_none_kwargs(X=X, W=W, Z=Z, nuisances=nuisances, sample_weight=sample_weight, freq_weight=freq_weight, - sample_var=sample_var)) + sample_var=sample_var, + groups=groups)) self.score_ = None if hasattr(self._ortho_learner_model_final, 'score'): self.score_ = self._ortho_learner_model_final.score(Y, T, **filter_none_kwargs(X=X, W=W, Z=Z, nuisances=nuisances, - sample_weight=sample_weight) - ) + sample_weight=sample_weight, + groups=groups)) def const_marginal_effect(self, X=None): X, = check_input_arrays(X) @@ -816,7 +818,7 @@ def effect_inference(self, X=None, *, T0=0, T1=1): return super().effect_inference(X, T0=T0, T1=T1) effect_inference.__doc__ = LinearCateEstimator.effect_inference.__doc__ - def score(self, Y, T, X=None, W=None, Z=None, sample_weight=None): + def score(self, Y, T, X=None, W=None, Z=None, sample_weight=None, groups=None): """ Score the fitted CATE model on a new data set. Generates nuisance parameters for the new data set based on the fitted nuisance models created at fit time. @@ -840,6 +842,8 @@ def score(self, Y, T, X=None, W=None, Z=None, sample_weight=None): Instruments for each sample sample_weight: optional(n,) vector or None (Default=None) Weights for each samples + groups: (n,) vector, optional + All rows corresponding to the same group will be kept together during splitting. Returns ------- @@ -862,7 +866,7 @@ def score(self, Y, T, X=None, W=None, Z=None, sample_weight=None): for i, models_nuisances in enumerate(self._models_nuisance): # for each model under cross fit setting for j, mdl in enumerate(models_nuisances): - nuisance_temp = mdl.predict(Y, T, **filter_none_kwargs(X=X, W=W, Z=Z)) + nuisance_temp = mdl.predict(Y, T, **filter_none_kwargs(X=X, W=W, Z=Z, groups=groups)) if not isinstance(nuisance_temp, tuple): nuisance_temp = (nuisance_temp,) @@ -876,7 +880,8 @@ def score(self, Y, T, X=None, W=None, Z=None, sample_weight=None): nuisances[it] = np.mean(nuisances[it], axis=0) return self._ortho_learner_model_final.score(Y, T, nuisances=nuisances, - **filter_none_kwargs(X=X, W=W, Z=Z, sample_weight=sample_weight)) + **filter_none_kwargs(X=X, W=W, Z=Z, + sample_weight=sample_weight, groups=groups)) @property def ortho_learner_model_final_(self): diff --git a/econml/data/dynamic_panel_dgp.py b/econml/data/dynamic_panel_dgp.py new file mode 100644 index 000000000..82b842912 --- /dev/null +++ b/econml/data/dynamic_panel_dgp.py @@ -0,0 +1,460 @@ +import numpy as np +from econml.utilities import cross_product +from statsmodels.tools.tools import add_constant +import pandas as pd +import scipy as sp +from scipy.stats import expon +from sklearn.linear_model import LinearRegression +import matplotlib.pyplot as plt +import joblib +import os + + +dir = os.path.dirname(__file__) + +# covariance matrix + + +def new_cov_matrix(cov): + p = cov.shape[0] + # get eigen value and eigen vectors + e_val, e_vec = sp.linalg.eigh(cov) + start = [0, 35, 77, 86] + end = [35, 77, 86, p] + e_val_new = np.array([]) + for i, j in zip(start, end): + e_val_new = np.append(e_val_new, linear_approximation(i, j, e_val)) + # simulate eigen vectors + e_vec_new = np.zeros_like(e_vec) + for i in range(p): + w = np.zeros(p) # , np.random.normal(0.01, 0.01, size=p) + w[np.random.choice(p, 6)] += np.random.normal(0.01, 0.06, size=(6)) + e_vec_new[:, i] = w / np.linalg.norm(w) + # keep the top 4 eigen value and corresponding eigen vector + e_vec_new[:, -4:] = e_vec[:, -4:] + e_val_new[-4:] = e_val[-4:] + # replace the negative eigen values + e_val_new[np.where(e_val_new < 0)] = e_val[np.where(e_val_new < 0)] + # generate a new covariance matrix + cov_new = e_vec_new.dot(np.diag(e_val_new)).dot(e_vec_new.T) + return cov_new + +# get linear approximation of eigen values + + +def linear_approximation(start, end, e_val): + est = LinearRegression() + X = np.arange(start, end).reshape(-1, 1) + est.fit(X, e_val[start:end]) + pred = est.predict(X) + return pred + + +# coefs +def generate_coefs(index, columns): + simulated_coefs_df = pd.DataFrame(0, index=index, columns=columns) + # get the indices of each group of features + ind_demo = [columns.index(col) for col in columns if "demo" in col] + ind_proxy = [columns.index(col) for col in columns if "proxy" in col] + ind_investment = [columns.index(col) + for col in columns if "investment" in col] + + for i in range(7): + outcome_name = simulated_coefs_df.index[i] + if "proxy" in outcome_name: + ind_same_proxy = [ + ind for ind in ind_proxy if outcome_name in columns[ind]] + # print(ind_same_proxy) + random_proxy_name = np.random.choice( + [proxy for proxy in index[:4] if proxy != outcome_name] + ) + ind_random_other_proxy = [ + ind for ind in ind_proxy if random_proxy_name in columns[ind] + ] + # demo + simulated_coefs_df.iloc[ + i, np.random.choice(ind_demo, 2) + ] = np.random.uniform(0.004, 0.05) + # same proxy + simulated_coefs_df.iloc[i, ind_same_proxy] = sorted( + np.random.choice(expon.pdf(np.arange(10)) * + 5e-1, 6, replace=False) + ) + simulated_coefs_df.iloc[i, ind_random_other_proxy] = sorted( + np.random.choice(expon.pdf(np.arange(10)) * + 5e-2, 6, replace=False) + ) + elif "investment" in outcome_name: + ind_same_invest = [ + ind for ind in ind_investment if outcome_name in columns[ind] + ] + random_proxy_name = np.random.choice(index[:4]) + ind_random_other_proxy = [ + ind for ind in ind_proxy if random_proxy_name in columns[ind] + ] + simulated_coefs_df.iloc[ + i, np.random.choice(ind_demo, 2) + ] = np.random.uniform(0.001, 0.05) + simulated_coefs_df.iloc[i, ind_same_invest] = sorted( + np.random.choice(expon.pdf(np.arange(10)) * + 5e-1, 6, replace=False) + ) + simulated_coefs_df.iloc[i, ind_random_other_proxy] = sorted( + np.random.choice(expon.pdf(np.arange(10)) * + 1e-1, 6, replace=False) + ) + return simulated_coefs_df + + +# residuals + + +def simulate_residuals(ind): + n, n_pos, n_neg = joblib.load(os.path.join(dir, f"input_dynamicdgp/n_{ind}.jbl")) + # gmm + est = joblib.load(os.path.join(dir, f"input_dynamicdgp/gm_{ind}.jbl")) + x_new = est.sample(n - n_pos - n_neg)[0].flatten() + + # log normal on outliers + if n_pos > 0: + # positive outliers + s, loc, scale = joblib.load(os.path.join(dir, f"input_dynamicdgp/lognorm_pos_{ind}.jbl")) + fitted_pos_outliers = sp.stats.lognorm( + s, loc=loc, scale=scale).rvs(size=n_pos) + else: + fitted_pos_outliers = np.array([]) + # negative outliers + if n_neg > 0: + s, loc, scale = joblib.load(os.path.join(dir, f"input_dynamicdgp/lognorm_neg_{ind}.jbl")) + fitted_neg_outliers = - \ + sp.stats.lognorm(s, loc=loc, scale=scale).rvs(size=n_neg) + else: + fitted_neg_outliers = np.array([]) + x_new = np.concatenate((x_new, fitted_pos_outliers, fitted_neg_outliers)) + return x_new + + +def simulate_residuals_all(res_df): + res_df_new = res_df.copy(deep=True) + for i in range(res_df.shape[1]): + res_df_new.iloc[:, i] = simulate_residuals(i) + # demean the new residual again + res_df_new = res_df_new - res_df_new.mean(axis=0) + return res_df_new + +# generate data + + +def get_prediction(df, coef_matrix, residuals, thetas, n, intervention, columns, index, counterfactual): + data_matrix = df[columns].values + # sample residuals + sample_residuals = residuals + preds = np.matmul(data_matrix, coef_matrix.T) + + # get prediction for current investment + if counterfactual: + pred_inv = np.zeros(preds[:, 4:].shape) + else: + pred_inv = preds[:, 4:] + sample_residuals[:, 4:] + intervention + df[index[4:]] = pd.DataFrame(pred_inv, index=df.index) + + # get prediction for current proxy + pred_proxy = preds[:, :4] + sample_residuals[:, :4] + \ + np.matmul(pred_inv, thetas.T) + df[index[:4]] = pd.DataFrame(pred_proxy, index=df.index) + return df + + +def generate_dgp( + cov_matrix, + n_tpid, + t_period, + coef_matrix, + residual_matrix, + thetas, + intervention, + columns, + index, + counterfactual +): + df_all = pd.DataFrame() + # get first period prediction + m = cov_matrix.shape[0] + x = np.random.multivariate_normal(np.repeat(0, m), cov_matrix, size=n_tpid) + df = pd.DataFrame( + np.hstack( + (np.arange(n_tpid).reshape(-1, 1), + np.repeat(1, n_tpid).reshape(-1, 1), x) + ), + columns=["id", "datetime"] + columns, + ) + df = get_prediction(df, coef_matrix, residual_matrix[0], + thetas, n_tpid, intervention, columns, index, False) + df_all = pd.concat([df_all, df], axis=0) + + # iterate the step ahead contruction + for t in range(2, t_period + 1): + # prepare new x + new_df = df.copy(deep=True) + new_df["datetime"] = np.repeat(t, n_tpid) + for name in index: + for i in range(-6, -1): + new_df[f"{name}_{i}"] = df[f"{name}_{i+1}"] + new_df[f"{name}_-1"] = df[name] + df = get_prediction(new_df, coef_matrix, residual_matrix[t - 1], + thetas, n_tpid, [0, 0, 0], columns, index, counterfactual) + df_all = pd.concat([df_all, df]) + df_all = df_all.sort_values(["id", "datetime"]) + return df_all + + +class AbstracDynamicPanelDGP: + + def __init__(self, n_periods, n_treatments, n_x): + self.n_periods = n_periods + self.n_treatments = n_treatments + self.n_x = n_x + return + + def create_instance(self, *args, **kwargs): + pass + + def _gen_data_with_policy(self, n_units, policy_gen, random_seed=123): + pass + + def static_policy_data(self, n_units, tau, random_seed=123): + def policy_gen(Tpre, X, period): + return tau[period] + return self._gen_data_with_policy(n_units, policy_gen, random_seed=random_seed) + + def adaptive_policy_data(self, n_units, policy_gen, random_seed=123): + return self._gen_data_with_policy(n_units, policy_gen, random_seed=random_seed) + + def static_policy_effect(self, tau, mc_samples=1000): + Y_tau, _, _, _ = self.static_policy_data(mc_samples, tau) + Y_zero, _, _, _ = self.static_policy_data( + mc_samples, np.zeros((self.n_periods, self.n_treatments))) + return np.mean(Y_tau[np.arange(Y_tau.shape[0]) % self.n_periods == self.n_periods - 1]) - \ + np.mean(Y_zero[np.arange(Y_zero.shape[0]) % + self.n_periods == self.n_periods - 1]) + + def adaptive_policy_effect(self, policy_gen, mc_samples=1000): + Y_tau, _, _, _ = self.adaptive_policy_data(mc_samples, policy_gen) + Y_zero, _, _, _ = self.static_policy_data( + mc_samples, np.zeros((self.n_periods, self.n_treatments))) + return np.mean(Y_tau[np.arange(Y_tau.shape[0]) % self.n_periods == self.n_periods - 1]) - \ + np.mean(Y_zero[np.arange(Y_zero.shape[0]) % + self.n_periods == self.n_periods - 1]) + + +class DynamicPanelDGP(AbstracDynamicPanelDGP): + + def __init__(self, n_periods, n_treatments, n_x): + super().__init__(n_periods, n_treatments, n_x) + + def create_instance(self, s_x, sigma_x, sigma_y, conf_str, epsilon, Alpha_unnormalized, + hetero_strength=0, hetero_inds=None, + autoreg=.5, state_effect=.5, random_seed=123): + random_state = np.random.RandomState(random_seed) + self.s_x = s_x + self.conf_str = conf_str + self.sigma_x = sigma_x + self.sigma_y = sigma_y + self.hetero_inds = hetero_inds.astype( + int) if hetero_inds is not None else hetero_inds + self.hetero_strength = hetero_strength + self.autoreg = autoreg + self.state_effect = state_effect + self.random_seed = random_seed + self.endo_inds = np.setdiff1d( + np.arange(self.n_x), hetero_inds).astype(int) + # The first s_x state variables are confounders. The final s_x variables are exogenous and can create + # heterogeneity + self.Alpha = Alpha_unnormalized + self.Alpha /= np.linalg.norm(self.Alpha, axis=1, ord=1, keepdims=True) + self.Alpha *= state_effect + if self.hetero_inds is not None: + self.Alpha[self.hetero_inds] = 0 + + self.Beta = np.zeros((self.n_x, self.n_x)) + for t in range(self.n_x): + self.Beta[t, :] = autoreg * np.roll(random_state.uniform(low=4.0**(-np.arange( + 0, self.n_x)), high=4.0**(-np.arange(1, self.n_x + 1))), t) + if self.hetero_inds is not None: + self.Beta[np.ix_(self.endo_inds, self.hetero_inds)] = 0 + self.Beta[np.ix_(self.hetero_inds, self.endo_inds)] = 0 + + self.epsilon = epsilon + self.zeta = np.zeros(self.n_x) + self.zeta[:self.s_x] = self.conf_str / self.s_x + + self.y_hetero_effect = np.zeros(self.n_x) + self.x_hetero_effect = np.zeros(self.n_x) + if self.hetero_inds is not None: + self.y_hetero_effect[self.hetero_inds] = random_state.uniform(.5 * hetero_strength, + 1.5 * hetero_strength) /\ + len(self.hetero_inds) + self.x_hetero_effect[self.hetero_inds] = random_state.uniform(.5 * hetero_strength, + 1.5 * hetero_strength) / \ + len(self.hetero_inds) + + self.true_effect = np.zeros((self.n_periods, self.n_treatments)) + self.true_effect[0] = self.epsilon + for t in np.arange(1, self.n_periods): + self.true_effect[t, :] = (self.zeta.reshape( + 1, -1) @ np.linalg.matrix_power(self.Beta, t - 1) @ self.Alpha) + + self.true_hetero_effect = np.zeros( + (self.n_periods, (self.n_x + 1) * self.n_treatments)) + self.true_hetero_effect[0, :] = cross_product(add_constant(self.y_hetero_effect.reshape(1, -1), + has_constant='add'), + self.epsilon.reshape(1, -1)) + for t in np.arange(1, self.n_periods): + self.true_hetero_effect[t, :] = cross_product(add_constant(self.x_hetero_effect.reshape(1, -1), + has_constant='add'), + self.zeta.reshape(1, -1) @ + np.linalg.matrix_power(self.Beta, t - 1) @ self.Alpha) + + return self + + def hetero_effect_fn(self, t, x): + if t == 0: + return (np.dot(self.y_hetero_effect, x.flatten()) + 1) * self.epsilon + else: + return (np.dot(self.x_hetero_effect, x.flatten()) + 1) *\ + (self.zeta.reshape(1, -1) @ np.linalg.matrix_power(self.Beta, t - 1) + @ self.Alpha).flatten() + + def _gen_data_with_policy(self, n_units, policy_gen, random_seed=123): + random_state = np.random.RandomState(random_seed) + Y = np.zeros(n_units * self.n_periods) + T = np.zeros((n_units * self.n_periods, self.n_treatments)) + X = np.zeros((n_units * self.n_periods, self.n_x)) + groups = np.zeros(n_units * self.n_periods) + for t in range(n_units * self.n_periods): + period = t % self.n_periods + if period == 0: + X[t] = random_state.normal(0, self.sigma_x, size=self.n_x) + T[t] = policy_gen(np.zeros(self.n_treatments), X[t], period, random_state) + else: + X[t] = (np.dot(self.x_hetero_effect, X[t - 1]) + 1) * np.dot(self.Alpha, T[t - 1]) + \ + np.dot(self.Beta, X[t - 1]) + \ + random_state.normal(0, self.sigma_x, size=self.n_x) + T[t] = policy_gen(T[t - 1], X[t], period, random_state) + Y[t] = (np.dot(self.y_hetero_effect, X[t]) + 1) * np.dot(self.epsilon, T[t]) + \ + np.dot(X[t], self.zeta) + \ + random_state.normal(0, self.sigma_y) + groups[t] = t // self.n_periods + + return Y, T, X, groups + + def observational_data(self, n_units, gamma, s_t, sigma_t, random_seed=123): + """ Generated observational data with some observational treatment policy parameters + + Parameters + ---------- + n_units : how many units to observe + gamma : what is the degree of auto-correlation of the treatments across periods + s_t : sparsity of treatment policy; how many states does it depend on + sigma_t : what is the std of the exploration/randomness in the treatment + """ + Delta = np.zeros((self.n_treatments, self.n_x)) + Delta[:, :s_t] = self.conf_str / s_t + + def policy_gen(Tpre, X, period, random_state): + return gamma * Tpre + (1 - gamma) * np.dot(Delta, X) + \ + random_state.normal(0, sigma_t, size=self.n_treatments) + return self._gen_data_with_policy(n_units, policy_gen, random_seed=random_seed) + + +class SemiSynthetic: + + def create_instance(self): + # get new covariance matrix + self.cov_new = joblib.load(os.path.join(dir, f"input_dynamicdgp/cov_new.jbl")) + + # get coefs + self.index = ["proxy1", "proxy2", "proxy3", "proxy4", + "investment1", "investment2", "investment3", ] + self.columns = [f"{ind}_{i}" for ind in self.index for i in range(-6, 0)] +\ + [f"demo_{i}" for i in range(47)] + + self.coef_df = generate_coefs(self.index, self.columns) + self.n_proxies = 4 + self.n_treatments = 3 + + # get residuals + res_df = pd.DataFrame(columns=self.index) + self.new_res_df = simulate_residuals_all(res_df) + + def gen_data(self, n, n_periods, thetas, random_seed): + random_state = np.random.RandomState(random_seed) + n_proxies = self.n_proxies + n_treatments = self.n_treatments + coef_matrix = self.coef_df.values + residual_matrix = self.new_res_df.values + n_x = len(self.columns) + # proxy 1 is the outcome + outcome = "proxy1" + + # make fixed residuals + all_residuals = [] + for t in range(n_periods): + sample_residuals = [] + for i in range(7): + sample_residuals.append( + random_state.choice(residual_matrix[:, i], n)) + sample_residuals = np.array(sample_residuals).T + all_residuals.append(sample_residuals) + all_residuals = np.array(all_residuals) + + fn_df_control = generate_dgp(self.cov_new, n, n_periods, + coef_matrix, all_residuals, thetas, + [0, 0, 0], self.columns, self.index, False) + + fn_df_cf_control = generate_dgp(self.cov_new, n, n_periods, + coef_matrix, all_residuals, thetas, + [0, 0, 0], self.columns, self.index, True) + true_effect = np.zeros((n_periods, n_treatments)) + for i in range(n_treatments): + intervention = [0, 0, 0] + intervention[i] = 1 + fn_df_treated = generate_dgp(self.cov_new, n, n_periods, + coef_matrix, all_residuals, thetas, + intervention, self.columns, self.index, True) + for t in range(n_periods): + ate_control = fn_df_cf_control.loc[ + fn_df_control["datetime"] == t + 1, outcome + ].mean() + ate_treated = fn_df_treated.loc[ + fn_df_treated["datetime"] == t + 1, outcome + ].mean() + true_effect[t, i] = ate_treated - ate_control + + new_index = ["proxy1", "proxy2", "proxy3", "proxy4"] + new_columns = [f"{ind}_{i}" for ind in new_index for i in range(-6, 0)] +\ + [f"demo_{i}" for i in range(47)] + panelX = fn_df_control[new_columns].values.reshape(-1, n_periods, len(new_columns)) + panelT = fn_df_control[self.index[n_proxies:] + ].values.reshape(-1, n_periods, n_treatments) + panelY = fn_df_control[outcome].values.reshape(-1, n_periods) + panelGroups = fn_df_control["id"].values.reshape(-1, n_periods) + return panelX, panelT, panelY, panelGroups, true_effect + + def plot_coefs(self): + coef_df = self.coef_df + plt.figure(figsize=(20, 20)) + for i in range(7): + outcome = coef_df.index[i] + plt.subplot(2, 4, i + 1) + coef_list = coef_df.iloc[i] + coef_list = coef_list[coef_list != 0] + plt.plot(coef_list) + plt.xticks(rotation=90) + plt.title(f"outcome:{outcome}") + plt.show() + + def plot_cov(self): + plt.imshow(self.cov_new) + plt.colorbar() + plt.show() diff --git a/econml/data/input_dynamicdgp/cov_new.jbl b/econml/data/input_dynamicdgp/cov_new.jbl new file mode 100644 index 000000000..8d0bb38d9 Binary files /dev/null and b/econml/data/input_dynamicdgp/cov_new.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_0.jbl b/econml/data/input_dynamicdgp/gm_0.jbl new file mode 100644 index 000000000..e4e262d9d Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_0.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_1.jbl b/econml/data/input_dynamicdgp/gm_1.jbl new file mode 100644 index 000000000..02c721f90 Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_1.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_2.jbl b/econml/data/input_dynamicdgp/gm_2.jbl new file mode 100644 index 000000000..91737d308 Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_2.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_3.jbl b/econml/data/input_dynamicdgp/gm_3.jbl new file mode 100644 index 000000000..4c3ce9289 Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_3.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_4.jbl b/econml/data/input_dynamicdgp/gm_4.jbl new file mode 100644 index 000000000..77af014fa Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_4.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_5.jbl b/econml/data/input_dynamicdgp/gm_5.jbl new file mode 100644 index 000000000..68279106b Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_5.jbl differ diff --git a/econml/data/input_dynamicdgp/gm_6.jbl b/econml/data/input_dynamicdgp/gm_6.jbl new file mode 100644 index 000000000..fbdfd1f70 Binary files /dev/null and b/econml/data/input_dynamicdgp/gm_6.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_neg_0.jbl b/econml/data/input_dynamicdgp/lognorm_neg_0.jbl new file mode 100644 index 000000000..21e84be89 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_neg_0.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_neg_1.jbl b/econml/data/input_dynamicdgp/lognorm_neg_1.jbl new file mode 100644 index 000000000..0ca173d59 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_neg_1.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_neg_2.jbl b/econml/data/input_dynamicdgp/lognorm_neg_2.jbl new file mode 100644 index 000000000..f4f1cc696 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_neg_2.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_neg_3.jbl b/econml/data/input_dynamicdgp/lognorm_neg_3.jbl new file mode 100644 index 000000000..a13e9d795 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_neg_3.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_neg_4.jbl b/econml/data/input_dynamicdgp/lognorm_neg_4.jbl new file mode 100644 index 000000000..e159c5665 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_neg_4.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_neg_5.jbl b/econml/data/input_dynamicdgp/lognorm_neg_5.jbl new file mode 100644 index 000000000..9ad1efe43 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_neg_5.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_0.jbl b/econml/data/input_dynamicdgp/lognorm_pos_0.jbl new file mode 100644 index 000000000..0ee019744 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_0.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_1.jbl b/econml/data/input_dynamicdgp/lognorm_pos_1.jbl new file mode 100644 index 000000000..3e1fa8236 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_1.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_2.jbl b/econml/data/input_dynamicdgp/lognorm_pos_2.jbl new file mode 100644 index 000000000..e94f429f7 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_2.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_3.jbl b/econml/data/input_dynamicdgp/lognorm_pos_3.jbl new file mode 100644 index 000000000..4ff1b40dd Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_3.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_4.jbl b/econml/data/input_dynamicdgp/lognorm_pos_4.jbl new file mode 100644 index 000000000..472844028 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_4.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_5.jbl b/econml/data/input_dynamicdgp/lognorm_pos_5.jbl new file mode 100644 index 000000000..67d145e51 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_5.jbl differ diff --git a/econml/data/input_dynamicdgp/lognorm_pos_6.jbl b/econml/data/input_dynamicdgp/lognorm_pos_6.jbl new file mode 100644 index 000000000..ebed90e54 Binary files /dev/null and b/econml/data/input_dynamicdgp/lognorm_pos_6.jbl differ diff --git a/econml/data/input_dynamicdgp/n_0.jbl b/econml/data/input_dynamicdgp/n_0.jbl new file mode 100644 index 000000000..0d72cf99b Binary files /dev/null and b/econml/data/input_dynamicdgp/n_0.jbl differ diff --git a/econml/data/input_dynamicdgp/n_1.jbl b/econml/data/input_dynamicdgp/n_1.jbl new file mode 100644 index 000000000..a1690f6a6 Binary files /dev/null and b/econml/data/input_dynamicdgp/n_1.jbl differ diff --git a/econml/data/input_dynamicdgp/n_2.jbl b/econml/data/input_dynamicdgp/n_2.jbl new file mode 100644 index 000000000..d9a912dda Binary files /dev/null and b/econml/data/input_dynamicdgp/n_2.jbl differ diff --git a/econml/data/input_dynamicdgp/n_3.jbl b/econml/data/input_dynamicdgp/n_3.jbl new file mode 100644 index 000000000..e15e46d30 Binary files /dev/null and b/econml/data/input_dynamicdgp/n_3.jbl differ diff --git a/econml/data/input_dynamicdgp/n_4.jbl b/econml/data/input_dynamicdgp/n_4.jbl new file mode 100644 index 000000000..de5e2691a Binary files /dev/null and b/econml/data/input_dynamicdgp/n_4.jbl differ diff --git a/econml/data/input_dynamicdgp/n_5.jbl b/econml/data/input_dynamicdgp/n_5.jbl new file mode 100644 index 000000000..e60b88978 Binary files /dev/null and b/econml/data/input_dynamicdgp/n_5.jbl differ diff --git a/econml/data/input_dynamicdgp/n_6.jbl b/econml/data/input_dynamicdgp/n_6.jbl new file mode 100644 index 000000000..e9f7e305d Binary files /dev/null and b/econml/data/input_dynamicdgp/n_6.jbl differ diff --git a/econml/dml/__init__.py b/econml/dml/__init__.py index a88d3b693..a83428cf7 100644 --- a/econml/dml/__init__.py +++ b/econml/dml/__init__.py @@ -32,7 +32,6 @@ .. [ortholearner] Dylan Foster, Vasilis Syrgkanis (2019). Orthogonal Statistical Learning. ACM Conference on Learning Theory. ``_ - """ from .dml import (DML, LinearDML, SparseLinearDML, @@ -45,4 +44,4 @@ "KernelDML", "NonParamDML", "ForestDML", - "CausalForestDML", ] + "CausalForestDML"] diff --git a/econml/dml/_rlearner.py b/econml/dml/_rlearner.py index beca5085f..ebdbe1fda 100644 --- a/econml/dml/_rlearner.py +++ b/econml/dml/_rlearner.py @@ -91,7 +91,8 @@ class _ModelFinal: def __init__(self, model_final): self._model_final = model_final - def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, + sample_weight=None, freq_weight=None, sample_var=None, groups=None): Y_res, T_res = nuisances self._model_final.fit(X, T, T_res, Y_res, sample_weight=sample_weight, freq_weight=freq_weight, sample_var=sample_var) @@ -100,7 +101,7 @@ def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, def predict(self, X=None): return self._model_final.predict(X) - def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None): + def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, groups=None): Y_res, T_res = nuisances if Y_res.ndim == 1: Y_res = Y_res.reshape((-1, 1)) diff --git a/econml/dml/causal_forest.py b/econml/dml/causal_forest.py index 4c08df103..efe040c4c 100644 --- a/econml/dml/causal_forest.py +++ b/econml/dml/causal_forest.py @@ -53,7 +53,7 @@ def _ate_and_stderr(self, drpreds, mask=None): stderr = (np.nanstd(drpreds, axis=0) / np.sqrt(nonnan)).reshape(self._d_y + self._d_t) return point, stderr - def fit(self, X, T, T_res, Y_res, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, X, T, T_res, Y_res, sample_weight=None, freq_weight=None, sample_var=None, groups=None): # Track training dimensions to see if Y or T is a vector instead of a 2-dimensional array self._d_t = shape(T_res)[1:] self._d_y = shape(Y_res)[1:] diff --git a/econml/dml/dml.py b/econml/dml/dml.py index ce6e95e0b..e87d956ca 100644 --- a/econml/dml/dml.py +++ b/econml/dml/dml.py @@ -134,7 +134,7 @@ def _combine(self, X, T, fitting=True): F = np.ones((T.shape[0], 1)) return cross_product(F, T) - def fit(self, X, T, T_res, Y_res, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, X, T, T_res, Y_res, sample_weight=None, freq_weight=None, sample_var=None, groups=None): # Track training dimensions to see if Y or T is a vector instead of a 2-dimensional array self._d_t = shape(T_res)[1:] self._d_y = shape(Y_res)[1:] diff --git a/econml/dr/_drlearner.py b/econml/dr/_drlearner.py index 2986439f9..41e206263 100644 --- a/econml/dr/_drlearner.py +++ b/econml/dr/_drlearner.py @@ -126,7 +126,8 @@ def __init__(self, model_final, featurizer, multitask_model_final): self._multitask_model_final = multitask_model_final return - def fit(self, Y, T, X=None, W=None, *, nuisances, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, Y, T, X=None, W=None, *, nuisances, + sample_weight=None, freq_weight=None, sample_var=None, groups=None): Y_pred, propensities = nuisances self.d_y = Y_pred.shape[1:-1] # track whether there's a Y dimension (must be a singleton) self.d_t = Y_pred.shape[-1] - 1 # track # of treatment (exclude baseline treatment) @@ -163,7 +164,7 @@ def predict(self, X=None): preds = np.array([mdl.predict(X).reshape((-1,) + self.d_y) for mdl in self.models_cate]) return np.moveaxis(preds, 0, -1) # move treatment dim to end - def score(self, Y, T, X=None, W=None, *, nuisances, sample_weight=None): + def score(self, Y, T, X=None, W=None, *, nuisances, sample_weight=None, groups=None): if (X is not None) and (self._featurizer is not None): X = self._featurizer.transform(X) Y_pred, _ = nuisances diff --git a/econml/dynamic/__init__.py b/econml/dynamic/__init__.py new file mode 100755 index 000000000..8e4ecd538 --- /dev/null +++ b/econml/dynamic/__init__.py @@ -0,0 +1,4 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +__all__ = ["dml"] diff --git a/econml/dynamic/dml/__init__.py b/econml/dynamic/dml/__init__.py new file mode 100755 index 000000000..a95579da7 --- /dev/null +++ b/econml/dynamic/dml/__init__.py @@ -0,0 +1,20 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Double Machine Learning for Dynamic Treatment Effects. + +A Double/Orthogonal machine learning approach to estimation of heterogeneous +treatment effect in the dynamic treatment regime. For the theoretical +foundations of these methods see: [dynamicdml]_. + +References +---------- + +.. [dynamicdml] Greg Lewis and Vasilis Syrgkanis. + Double/Debiased Machine Learning for Dynamic Treatment Effects. + ``_, 2021. +""" + +from ._dml import DynamicDML + +__all__ = ["DynamicDML"] diff --git a/econml/dynamic/dml/_dml.py b/econml/dynamic/dml/_dml.py new file mode 100644 index 000000000..9453b16e5 --- /dev/null +++ b/econml/dynamic/dml/_dml.py @@ -0,0 +1,795 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +import abc +import numpy as np +from warnings import warn +from sklearn.base import clone +from sklearn.model_selection import GroupKFold +from scipy.stats import norm +from sklearn.linear_model import (ElasticNetCV, LassoCV, LogisticRegressionCV) +from ...sklearn_extensions.linear_model import (StatsModelsLinearRegression, WeightedLassoCVWrapper) +from ...sklearn_extensions.model_selection import WeightedStratifiedKFold +from ...dml.dml import _FirstStageWrapper, _FinalWrapper +from ..._cate_estimator import TreatmentExpansionMixin, LinearModelFinalCateEstimatorMixin +from ..._ortho_learner import _OrthoLearner +from ...utilities import (_deprecate_positional, add_intercept, + broadcast_unit_treatments, check_high_dimensional, + cross_product, deprecated, fit_with_groups, + hstack, inverse_onehot, ndim, reshape, + reshape_treatmentwise_effects, shape, transpose, + get_feature_names_or_default, check_input_arrays, + filter_none_kwargs) + + +def _get_groups_period_filter(groups, n_periods): + group_counts = {} + group_period_filter = {i: [] for i in range(n_periods)} + for i, g in enumerate(groups): + if g not in group_counts: + group_counts[g] = 0 + group_period_filter[group_counts[g]].append(i) + group_counts[g] += 1 + return group_period_filter + + +class _DynamicModelNuisance: + """ + Nuisance model fits the model_y and model_t at fit time and at predict time + calculates the residual Y and residual T based on the fitted models and returns + the residuals as two nuisance parameters. + """ + + def __init__(self, model_y, model_t, n_periods): + self._model_y = model_y + self._model_t = model_t + self.n_periods = n_periods + + def fit(self, Y, T, X=None, W=None, sample_weight=None, groups=None): + """Fit a series of nuisance models for each period or period pairs.""" + assert Y.shape[0] % self.n_periods == 0, \ + "Length of training data should be an integer multiple of time periods." + period_filters = _get_groups_period_filter(groups, self.n_periods) + self._model_y_trained = {} + self._model_t_trained = {j: {} for j in np.arange(self.n_periods)} + for t in np.arange(self.n_periods): + self._model_y_trained[t] = clone(self._model_y, safe=False).fit( + self._index_or_None(X, period_filters[t]), + self._index_or_None( + W, period_filters[t]), + Y[period_filters[self.n_periods - 1]]) + for j in np.arange(t, self.n_periods): + self._model_t_trained[j][t] = clone(self._model_t, safe=False).fit( + self._index_or_None(X, period_filters[t]), + self._index_or_None(W, period_filters[t]), + T[period_filters[j]]) + return self + + def predict(self, Y, T, X=None, W=None, sample_weight=None, groups=None): + """Calculate nuisances for each period or period pairs. + + Returns + ------- + Y_res : (n, d_y) matrix or vector of length n + Y residuals for each period in panel format. + This shape is required for _OrthoLearner's crossfitting. + T_res : (n, d_t, n_periods) matrix + T residuals for pairs of periods (t, j), where the data is in panel format for t + and in index form for j. For example, the residuals for (t, j) can be retrieved via + T_res[np.arange(n) % n_periods == t, ..., j]. For t < j, the entries of this + matrix are np.nan. + This shape is required for _OrthoLearner's crossfitting. + """ + assert Y.shape[0] % self.n_periods == 0, \ + "Length of training data should be an integer multiple of time periods." + period_filters = _get_groups_period_filter(groups, self.n_periods) + Y_res = np.full(Y.shape, np.nan) + T_res = np.full(T.shape + (self.n_periods, ), np.nan) + shape_formatter = self._get_shape_formatter(X, W) + for t in np.arange(self.n_periods): + Y_slice = Y[period_filters[self.n_periods - 1]] + Y_pred = self._model_y_trained[t].predict( + self._index_or_None(X, period_filters[t]), + self._index_or_None(W, period_filters[t])) + Y_res[period_filters[t]] = Y_slice\ + - shape_formatter(Y_slice, Y_pred) + for j in np.arange(t, self.n_periods): + T_slice = T[period_filters[j]] + T_pred = self._model_t_trained[j][t].predict( + self._index_or_None(X, period_filters[t]), + self._index_or_None(W, period_filters[t])) + T_res[period_filters[j], ..., t] = T_slice\ + - shape_formatter(T_slice, T_pred) + return Y_res, T_res + + def score(self, Y, T, X=None, W=None, sample_weight=None, groups=None): + assert Y.shape[0] % self.n_periods == 0, \ + "Length of training data should be an integer multiple of time periods." + period_filters = _get_groups_period_filter(groups, self.n_periods) + if hasattr(self._model_y, 'score'): + Y_score = np.full((self.n_periods, ), np.nan) + for t in np.arange(self.n_periods): + Y_score[t] = self._model_y_trained[t].score( + self._index_or_None(X, period_filters[t]), + self._index_or_None(W, period_filters[t]), + Y[period_filters[self.n_periods - 1]]) + else: + Y_score = None + if hasattr(self._model_t, 'score'): + T_score = np.full((self.n_periods, self.n_periods), np.nan) + for t in np.arange(self.n_periods): + for j in np.arange(t, self.n_periods): + T_score[j][t] = self._model_t_trained[j][t].score( + self._index_or_None(X, period_filters[t]), + self._index_or_None(W, period_filters[t]), + T[period_filters[j]]) + else: + T_score = None + return Y_score, T_score + + def _get_shape_formatter(self, X, W): + if (X is None) and (W is None): + return lambda x, x_pred: np.tile(x_pred.reshape(1, -1), (x.shape[0], 1)).reshape(x.shape) + return lambda x, x_pred: x_pred.reshape(x.shape) + + def _index_or_None(self, X, filter_idx): + return None if X is None else X[filter_idx] + + +class _DynamicModelFinal: + """ + Final model at fit time, fits a residual on residual regression with a heterogeneous coefficient + that depends on X, i.e. + + .. math :: + Y - E[Y | X, W] = \\theta(X) \\cdot (T - E[T | X, W]) + \\epsilon + + and at predict time returns :math:`\\theta(X)`. The score method returns the MSE of this final + residual on residual regression. + Assumes model final is parametric with no intercept. + """ + # TODO: update docs + + def __init__(self, model_final, n_periods): + self._model_final = model_final + self.n_periods = n_periods + self._model_final_trained = {k: clone(self._model_final, safe=False) for k in np.arange(n_periods)} + + def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, sample_var=None, groups=None): + # NOTE: sample weight, sample var are not passed in + period_filters = _get_groups_period_filter(groups, self.n_periods) + Y_res, T_res = nuisances + self._d_y = Y.shape[1:] + for t in np.arange(self.n_periods - 1, -1, -1): + Y_adj = Y_res[period_filters[t]].copy() + if t < self.n_periods - 1: + Y_adj -= np.sum( + [self._model_final_trained[j].predict_with_res( + X[period_filters[0]] if X is not None else None, + T_res[period_filters[j], ..., t] + ) for j in np.arange(t + 1, self.n_periods)], axis=0) + self._model_final_trained[t].fit( + X[period_filters[0]] if X is not None else None, T[period_filters[t]], + T_res[period_filters[t], ..., t], Y_adj) + + return self + + def predict(self, X=None): + """ + Return shape: m x dy x (p*dt) + """ + d_t_tuple = self._model_final_trained[0]._d_t + d_t = d_t_tuple[0] if d_t_tuple else 1 + x_dy_shape = (X.shape[0] if X is not None else 1, ) + \ + self._model_final_trained[0]._d_y + preds = np.zeros( + x_dy_shape + + (self.n_periods * d_t, ) + ) + for t in range(self.n_periods): + preds[..., t * d_t: (t + 1) * d_t] = \ + self._model_final_trained[t].predict(X).reshape( + x_dy_shape + (d_t, ) + ) + return preds + + def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, sample_var=None, groups=None): + assert Y.shape[0] % self.n_periods == 0, \ + "Length of training data should be an integer multiple of time periods." + Y_res, T_res = nuisances + scores = np.full((self.n_periods, ), np.nan) + period_filters = _get_groups_period_filter(groups, self.n_periods) + for t in np.arange(self.n_periods - 1, -1, -1): + Y_adj = Y_res[period_filters[t]].copy() + if t < self.n_periods - 1: + Y_adj -= np.sum( + [self._model_final_trained[j].predict_with_res( + X[period_filters[0]] if X is not None else None, + T_res[period_filters[j], ..., t] + ) for j in np.arange(t + 1, self.n_periods)], axis=0) + Y_adj_pred = self._model_final_trained[t].predict_with_res( + X[period_filters[0]] if X is not None else None, + T_res[period_filters[t], ..., t]) + if sample_weight is not None: + scores[t] = np.mean(np.average((Y_adj - Y_adj_pred)**2, weights=sample_weight, axis=0)) + else: + scores[t] = np.mean((Y_adj - Y_adj_pred) ** 2) + return scores + + +class _LinearDynamicModelFinal(_DynamicModelFinal): + """Wrapper for the DynamicModelFinal with StatsModelsLinearRegression final model. + + The final model is a linear model with (d_t*n_periods) coefficients. + This model is defined after the coefficients and covariance are calculated. + """ + + def __init__(self, model_final, n_periods): + super().__init__(model_final, n_periods) + self.model_final_ = StatsModelsLinearRegression(fit_intercept=False) + + def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, sample_var=None, groups=None): + super().fit(Y, T, X=X, W=W, Z=Z, nuisances=nuisances, + sample_weight=sample_weight, sample_var=sample_var, groups=groups) + # Compose final model + cov = self._get_cov(nuisances, X, groups) + coef = self._get_coef_() + self.model_final_._n_out = self._d_y[0] if self._d_y else 0 + self.model_final_._param_var = cov / (Y.shape[0] / self.n_periods) + self.model_final_._param = coef.T if self.model_final_._n_out else coef + + def _get_coef_(self): + period_coefs = np.array([self._model_final_trained[t]._model.coef_ for t in range(self.n_periods)]) + if self._d_y: + return np.array([ + np.array([period_coefs[k, i, :] for k in range(self.n_periods)]).flatten() + for i in range(self._d_y[0]) + ]) + return period_coefs.flatten() + + def _get_cov(self, nuisances, X, groups): + if self._d_y: + return np.array( + [self._fit_single_output_cov((nuisances[0][:, i], nuisances[1]), X, i, groups) + for i in range(self._d_y[0])] + ) + return self._fit_single_output_cov(nuisances, X, -1, groups) + + def _fit_single_output_cov(self, nuisances, X, y_index, groups): + """ Calculates the covariance (n_periods*n_treatments) + x (n_periods*n_treatments) matrix for a single outcome. + """ + Y_res, T_res = nuisances + # Calculate auxiliary quantities + period_filters = _get_groups_period_filter(groups, self.n_periods) + # X ⨂ T_res + XT_res = np.array([ + [ + self._model_final_trained[0]._combine( + X[period_filters[0]] if X is not None else None, + T_res[period_filters[t], ..., j], + fitting=False + ) + for j in range(self.n_periods) + ] + for t in range(self.n_periods) + ]) + d_xt = XT_res.shape[-1] + # sum(model_final.predict(X, T_res)) + Y_diff = np.array([ + np.sum([ + self._model_final_trained[j].predict_with_res( + X[period_filters[0]] if X is not None else None, + T_res[period_filters[j], ..., t] + ) for j in np.arange(t, self.n_periods)], + axis=0 + ) + for t in np.arange(self.n_periods) + ]) + J = np.zeros((self.n_periods * d_xt, + self.n_periods * d_xt)) + Sigma = np.zeros((self.n_periods * d_xt, + self.n_periods * d_xt)) + for t in np.arange(self.n_periods): + res_epsilon_t = (Y_res[period_filters[t]] - + (Y_diff[t][:, y_index] if y_index >= 0 else Y_diff[t]) + ).reshape(-1, 1, 1) + resT_t = XT_res[t][t] + for j in np.arange(self.n_periods): + # Calculating the (t, j) block entry (of size n_treatments x n_treatments) of matrix Sigma + res_epsilon_j = (Y_res[period_filters[j]] - + (Y_diff[j][:, y_index] if y_index >= 0 else Y_diff[j]) + ).reshape(-1, 1, 1) + resT_j = XT_res[j][j] + cov_resT_tj = resT_t.reshape(-1, d_xt, 1) @ resT_j.reshape(-1, 1, d_xt) + sigma_tj = np.mean((res_epsilon_t * res_epsilon_j) * cov_resT_tj, axis=0) + Sigma[t * d_xt:(t + 1) * d_xt, + j * d_xt:(j + 1) * d_xt] = sigma_tj + if j >= t: + # Calculating the (t, j) block entry (of size n_treatments x n_treatments) of matrix J + m_tj = np.mean( + XT_res[j][t].reshape(-1, d_xt, 1) @ resT_t.reshape(-1, 1, d_xt), + axis=0) + J[t * d_xt:(t + 1) * d_xt, + j * d_xt:(j + 1) * d_xt] = m_tj + return np.linalg.inv(J) @ Sigma @ np.linalg.inv(J).T + + +class _DynamicFinalWrapper(_FinalWrapper): + + def predict_with_res(self, X, T_res): + fts = self._combine(X, T_res, fitting=False) + prediction = self._model.predict(fts) + if self._intercept is not None: + prediction -= self._intercept + return reshape(prediction, (prediction.shape[0],) + self._d_y) + + +class DynamicDML(LinearModelFinalCateEstimatorMixin, _OrthoLearner): + """CATE estimator for dynamic treatment effect estimation. + + This estimator is an extension of the Double ML approach for treatments assigned sequentially + over time periods. + + The estimator is a special case of an :class:`_OrthoLearner` estimator, so it follows the two + stage process, where a set of nuisance functions are estimated in the first stage in a crossfitting + manner and a final stage estimates the CATE model. See the documentation of + :class:`._OrthoLearner` for a description of this two stage process. + + Parameters + ---------- + model_y: estimator or 'auto', optional (default is 'auto') + The estimator for fitting the response to the features. Must implement + `fit` and `predict` methods. + If 'auto' :class:`.WeightedLassoCV`/:class:`.WeightedMultiTaskLassoCV` will be chosen. + + model_t: estimator or 'auto', optional (default is 'auto') + The estimator for fitting the treatment to the features. + If estimator, it must implement `fit` and `predict` methods; + If 'auto', :class:`~sklearn.linear_model.LogisticRegressionCV` will be applied for discrete treatment, + and :class:`.WeightedLassoCV`/:class:`.WeightedMultiTaskLassoCV` + will be applied for continuous treatment. + + featurizer : :term:`transformer`, optional, default None + Must support fit_transform and transform. Used to create composite features in the final CATE regression. + It is ignored if X is None. The final CATE will be trained on the outcome of featurizer.fit_transform(X). + If featurizer=None, then CATE is trained on X. + + fit_cate_intercept : bool, optional, default True + Whether the linear CATE model should have a constant term. + + linear_first_stages: bool + Whether the first stage models are linear (in which case we will expand the features passed to + `model_y` accordingly) + + discrete_treatment: bool, optional (default is ``False``) + Whether the treatment values should be treated as categorical, rather than continuous, quantities + + categories: 'auto' or list, default 'auto' + The categories to use when encoding discrete treatments (or 'auto' to use the unique sorted values). + The first category will be treated as the control treatment. + + cv: int, cross-validation generator or an iterable, optional (Default=2) + Determines the cross-validation splitting strategy. + Possible inputs for cv are: + + - None, to use the default 3-fold cross-validation, + - integer, to specify the number of folds. + - :term:`CV splitter` + - An iterable yielding (train, test) splits as arrays of indices. + Iterables should make sure a group belongs to a single split. + + For integer/None inputs, :class:`~sklearn.model_selection.GroupKFold` is used + + Unless an iterable is used, we call `split(X, T, groups)` to generate the splits. + + mc_iters: int, optional (default=None) + The number of times to rerun the first stage models to reduce the variance of the nuisances. + + mc_agg: {'mean', 'median'}, optional (default='mean') + How to aggregate the nuisance value for each sample across the `mc_iters` monte carlo iterations of + cross-fitting. + + random_state: int, :class:`~numpy.random.mtrand.RandomState` instance or None, optional (default=None) + If int, random_state is the seed used by the random number generator; + If :class:`~numpy.random.mtrand.RandomState` instance, random_state is the random number generator; + If None, the random number generator is the :class:`~numpy.random.mtrand.RandomState` instance used + by :mod:`np.random`. + + Examples + -------- + A simple example with default models: + + .. testcode:: + :hide: + + import numpy as np + np.set_printoptions(suppress=True) + + .. testcode:: + + from econml.dynamic.dml import DynamicDML + + np.random.seed(123) + + n_panels = 100 # number of panels + n_periods = 3 # number of time periods per panel + n = n_panels * n_periods + groups = np.repeat(a=np.arange(n_panels), repeats=n_periods, axis=0) + X = np.random.normal(size=(n, 1)) + T = np.random.normal(size=(n, 2)) + y = np.random.normal(size=(n, )) + est = DynamicDML() + est.fit(y, T, X=X, W=None, groups=groups, inference="auto") + + >>> est.const_marginal_effect(X[:2]) + array([[-0.336..., -0.048..., -0.061..., 0.042..., -0.204..., + 0.00667271], + [-0.101..., 0.433..., 0.054..., -0.217..., -0.101..., + -0.159...]]) + >>> est.effect(X[:2], T0=0, T1=1) + array([-0.601..., -0.091...]) + >>> est.effect(X[:2], T0=np.zeros((2, n_periods*T.shape[1])), T1=np.ones((2, n_periods*T.shape[1]))) + array([-0.601..., -0.091...]) + >>> est.coef_ + array([[ 0.112...], + [ 0.231...], + [ 0.055...], + [-0.125...], + [ 0.049...], + [-0.079...]]) + >>> est.coef__interval() + (array([[-0.035...], + [ 0.029...], + [-0.087... ], + [-0.366... ], + [-0.090...], + [-0.233...]]), + array([[0.260...], + [0.433... ], + [0.198...], + [0.116...], + [0.189...], + [0.074...]])) + """ + + def __init__(self, *, + model_y='auto', model_t='auto', + featurizer=None, + fit_cate_intercept=True, + linear_first_stages=False, + discrete_treatment=False, + categories='auto', + cv=2, + mc_iters=None, + mc_agg='mean', + random_state=None): + self.fit_cate_intercept = fit_cate_intercept + self.linear_first_stages = linear_first_stages + self.featurizer = clone(featurizer, safe=False) + self.model_y = clone(model_y, safe=False) + self.model_t = clone(model_t, safe=False) + super().__init__(discrete_treatment=discrete_treatment, + discrete_instrument=False, + categories=categories, + cv=GroupKFold(cv) if isinstance(cv, int) else cv, + mc_iters=mc_iters, + mc_agg=mc_agg, + random_state=random_state) + + def _gen_featurizer(self): + return clone(self.featurizer, safe=False) + + def _gen_model_y(self): + if self.model_y == 'auto': + model_y = WeightedLassoCVWrapper(random_state=self.random_state) + else: + model_y = clone(self.model_y, safe=False) + return _FirstStageWrapper(model_y, True, self._gen_featurizer(), + self.linear_first_stages, self.discrete_treatment) + + def _gen_model_t(self): + if self.model_t == 'auto': + if self.discrete_treatment: + model_t = LogisticRegressionCV(cv=WeightedStratifiedKFold(random_state=self.random_state), + random_state=self.random_state) + else: + model_t = WeightedLassoCVWrapper(random_state=self.random_state) + else: + model_t = clone(self.model_t, safe=False) + return _FirstStageWrapper(model_t, False, self._gen_featurizer(), + self.linear_first_stages, self.discrete_treatment) + + def _gen_model_final(self): + return StatsModelsLinearRegression(fit_intercept=False) + + def _gen_ortho_learner_model_nuisance(self, n_periods): + return _DynamicModelNuisance( + model_t=self._gen_model_t(), + model_y=self._gen_model_y(), + n_periods=n_periods) + + def _gen_ortho_learner_model_final(self, n_periods): + wrapped_final_model = _DynamicFinalWrapper( + StatsModelsLinearRegression(fit_intercept=False), + fit_cate_intercept=self.fit_cate_intercept, + featurizer=self.featurizer, + use_weight_trick=False) + return _LinearDynamicModelFinal(wrapped_final_model, n_periods=n_periods) + + def _prefit(self, Y, T, *args, groups=None, only_final=False, **kwargs): + u_periods = np.unique(np.unique(groups, return_counts=True)[1]) + if len(u_periods) > 1: + raise AttributeError( + "Imbalanced panel. Method currently expects only panels with equal number of periods. Pad your data") + self._n_periods = u_periods[0] + # generate an instance of the final model + self._ortho_learner_model_final = self._gen_ortho_learner_model_final(self._n_periods) + if not only_final: + # generate an instance of the nuisance model + self._ortho_learner_model_nuisance = self._gen_ortho_learner_model_nuisance(self._n_periods) + TreatmentExpansionMixin._prefit(self, Y, T, *args, **kwargs) + + def _postfit(self, Y, T, *args, **kwargs): + super()._postfit(Y, T, *args, **kwargs) + # Set _d_t to effective number of treatments + self._d_t = (self._n_periods * self._d_t[0], ) if self._d_t else (self._n_periods, ) + + def _strata(self, Y, T, X=None, W=None, Z=None, + sample_weight=None, sample_var=None, groups=None, + cache_values=False, only_final=False, check_input=True): + # Required for bootstrap inference + return groups + + @_deprecate_positional("X, and should be passed by keyword only. In a future release " + "we will disallow passing X and W by position.", ['X', 'W']) + def fit(self, Y, T, X=None, W=None, *, sample_weight=None, sample_var=None, groups, + cache_values=False, inference='auto'): + """Estimate the counterfactual model from data, i.e. estimates function :math:`\\theta(\\cdot)`. + + The input data must contain groups with the same size corresponding to the number + of time periods the treatments were assigned over. + + The data should be preferably in panel format, with groups clustered together. + If group members do not appear together, the following is assumed: + + * the first instance of a group in the dataset is assumed to correspond to the first period of that group + * the second instance of a group in the dataset is assumed to correspond to the + second period of that group + + ...etc. + + Only the value of the features X at the first period of each unit are used for + heterogeneity. The value of X in subseuqnet periods is used as a time-varying control + but not for heterogeneity. + + Parameters + ---------- + Y: (n, d_y) matrix or vector of length n + Outcomes for each sample (required: n = n_groups * n_periods) + T: (n, d_t) matrix or vector of length n + Treatments for each sample (required: n = n_groups * n_periods) + X: optional(n, d_x) matrix or None (Default=None) + Features for each sample (Required: n = n_groups * n_periods). Only first + period features from each unit are used for heterogeneity, the rest are + used as time-varying controls together with W + W: optional(n, d_w) matrix or None (Default=None) + Controls for each sample (Required: n = n_groups * n_periods) + sample_weight: optional(n,) vector or None (Default=None) + Weights for each samples + sample_var: optional(n,) vector or None (Default=None) + Sample variance for each sample + groups: (n,) vector, required + All rows corresponding to the same group will be kept together during splitting. + If groups is not None, the `cv` argument passed to this class's initializer + must support a 'groups' argument to its split method. + cache_values: bool, default False + Whether to cache inputs and first stage results, which will allow refitting a different final model + inference: string,:class:`.Inference` instance, or None + Method for performing inference. This estimator supports 'bootstrap' + (or an instance of :class:`.BootstrapInference`) and 'auto' + (or an instance of :class:`.LinearModelFinalInference`). + + Returns + ------- + self: DynamicDML instance + """ + if sample_weight is not None or sample_var is not None: + warn("This CATE estimator does not yet support sample weights and sample variance. " + "These inputs will be ignored during fitting.", + UserWarning) + return super().fit(Y, T, X=X, W=W, + sample_weight=None, sample_var=None, groups=groups, + cache_values=cache_values, + inference=inference) + + def score(self, Y, T, X=None, W=None, sample_weight=None, *, groups): + """ + Score the fitted CATE model on a new data set. Generates nuisance parameters + for the new data set based on the fitted residual nuisance models created at fit time. + It uses the mean prediction of the models fitted by the different crossfit folds. + Then calculates the MSE of the final residual Y on residual T regression. + + If model_final does not have a score method, then it raises an :exc:`.AttributeError` + + Parameters + ---------- + Y: (n, d_y) matrix or vector of length n + Outcomes for each sample (required: n = n_groups * n_periods) + T: (n, d_t) matrix or vector of length n + Treatments for each sample (required: n = n_groups * n_periods) + X: optional(n, d_x) matrix or None (Default=None) + Features for each sample (Required: n = n_groups * n_periods) + W: optional(n, d_w) matrix or None (Default=None) + Controls for each sample (Required: n = n_groups * n_periods) + groups: (n,) vector, required + All rows corresponding to the same group will be kept together during splitting. + + Returns + ------- + score: float + The MSE of the final CATE model on the new data. + """ + if not hasattr(self._ortho_learner_model_final, 'score'): + raise AttributeError("Final model does not have a score method!") + Y, T, X, W, groups = check_input_arrays(Y, T, X, W, groups) + self._check_fitted_dims(X) + X, T = super()._expand_treatments(X, T) + n_iters = len(self._models_nuisance) + n_splits = len(self._models_nuisance[0]) + + # for each mc iteration + for i, models_nuisances in enumerate(self._models_nuisance): + # for each model under cross fit setting + for j, mdl in enumerate(models_nuisances): + nuisance_temp = mdl.predict(Y, T, **filter_none_kwargs(X=X, W=W, groups=groups)) + if not isinstance(nuisance_temp, tuple): + nuisance_temp = (nuisance_temp,) + + if i == 0 and j == 0: + nuisances = [np.zeros((n_iters * n_splits,) + nuis.shape) for nuis in nuisance_temp] + + for it, nuis in enumerate(nuisance_temp): + nuisances[it][i * n_iters + j] = nuis + + for it in range(len(nuisances)): + nuisances[it] = np.mean(nuisances[it], axis=0) + return self._ortho_learner_model_final.score(Y, T, nuisances=nuisances, + **filter_none_kwargs(X=X, W=W, + sample_weight=sample_weight, groups=groups)) + + def cate_treatment_names(self, treatment_names=None): + """ + Get treatment names for each time period. + + If the treatment is discrete, it will return expanded treatment names. + + Parameters + ---------- + treatment_names: list of strings of length T.shape[1] or None + The names of the treatments. If None and the T passed to fit was a dataframe, + it defaults to the column names from the dataframe. + + Returns + ------- + out_treatment_names: list of strings + Returns (possibly expanded) treatment names. + """ + slice_treatment_names = super().cate_treatment_names(treatment_names) + treatment_names_out = [] + for k in range(self._n_periods): + treatment_names_out += [f"({t})$_{k}$" for t in slice_treatment_names] + return treatment_names_out + + def cate_feature_names(self, feature_names=None): + """ + Get the output feature names. + + Parameters + ---------- + feature_names: list of strings of length X.shape[1] or None + The names of the input features. If None and X is a dataframe, it defaults to the column names + from the dataframe. + + Returns + ------- + out_feature_names: list of strings or None + The names of the output features :math:`\\phi(X)`, i.e. the features with respect to which the + final constant marginal CATE model is linear. It is the names of the features that are associated + with each entry of the :meth:`coef_` parameter. Not available when the featurizer is not None and + does not have a method: `get_feature_names(feature_names)`. Otherwise None is returned. + """ + if self._d_x is None: + # Handles the corner case when X=None but featurizer might be not None + return None + if feature_names is None: + feature_names = self._input_names["feature_names"] + if self.original_featurizer is None: + return feature_names + return get_feature_names_or_default(self.original_featurizer, feature_names) + + def _expand_treatments(self, X, *Ts): + # Expand treatments for each time period + outTs = [] + base_expand_treatments = super()._expand_treatments + for T in Ts: + if ndim(T) == 0: + one_T = base_expand_treatments(X, T)[1] + one_T = one_T.reshape(-1, 1) if ndim(one_T) == 1 else one_T + T = np.tile(one_T, (1, self._n_periods, )) + else: + assert (T.shape[1] == self._n_periods if self.transformer else T.shape[1] == self._d_t[0]), \ + f"Expected a list of time period * d_t, instead got a treatment array of shape {T.shape}." + if self.transformer: + T = np.hstack([ + base_expand_treatments( + X, T[:, [t]])[1] for t in range(self._n_periods) + ]) + outTs.append(T) + return (X,) + tuple(outTs) + + @property + def bias_part_of_coef(self): + return self.ortho_learner_model_final_._model_final._fit_cate_intercept + + @property + def fit_cate_intercept_(self): + return self.ortho_learner_model_final_._model_final._fit_cate_intercept + + @property + def original_featurizer(self): + # NOTE: important to use the _ortho_learner_model_final_ attribute instead of the + # attribute so that the trained featurizer will be passed through + return self.ortho_learner_model_final_._model_final_trained[0]._original_featurizer + + @property + def featurizer_(self): + # NOTE This is used by the inference methods and has to be the overall featurizer. intended + # for internal use by the library + return self.ortho_learner_model_final_._model_final_trained[0]._featurizer + + @property + def model_final_(self): + # NOTE This is used by the inference methods and is more for internal use to the library + # We need to use the _ortho_learner's copy to retain the information from fitting + return self.ortho_learner_model_final_.model_final_ + + @property + def model_final(self): + return self._gen_model_final() + + @model_final.setter + def model_final(self, model): + if model is not None: + raise ValueError("Parameter `model_final` cannot be altered for this estimator!") + + @property + def models_y(self): + return [[mdl._model_y for mdl in mdls] for mdls in super().models_nuisance_] + + @property + def models_t(self): + return [[mdl._model_t for mdl in mdls] for mdls in super().models_nuisance_] + + @property + def nuisance_scores_y(self): + return self.nuisance_scores_[0] + + @property + def nuisance_scores_t(self): + return self.nuisance_scores_[1] + + @property + def residuals_(self): + """ + A tuple (y_res, T_res, X, W), of the residuals from the first stage estimation + along with the associated X and W. Samples are not guaranteed to be in the same + order as the input order. + """ + if not hasattr(self, '_cached_values'): + raise AttributeError("Estimator is not fitted yet!") + if self._cached_values is None: + raise AttributeError("`fit` was called with `cache_values=False`. " + "Set to `True` to enable residual storage.") + Y_res, T_res = self._cached_values.nuisances + return Y_res, T_res, self._cached_values.X, self._cached_values.W diff --git a/econml/iv/dml/_dml.py b/econml/iv/dml/_dml.py index 8599b8fad..7ad1886d0 100644 --- a/econml/iv/dml/_dml.py +++ b/econml/iv/dml/_dml.py @@ -33,7 +33,8 @@ def __init__(self): self._model_final = _FinalWrapper(LinearRegression(fit_intercept=False), fit_cate_intercept=True, featurizer=None, use_weight_trick=False) - def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, + sample_weight=None, freq_weight=None, sample_var=None, groups=None): Y_res, T_res, Z_res = nuisances if Z_res.ndim == 1: Z_res = Z_res.reshape(-1, 1) @@ -49,7 +50,7 @@ def predict(self, X=None): # TODO: allow the final model to actually use X? return self._model_final.predict(X=None) - def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None): + def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, groups=None): Y_res, T_res, Z_res = nuisances if Y_res.ndim == 1: Y_res = Y_res.reshape((-1, 1)) @@ -384,7 +385,8 @@ class _BaseDMLIVModelFinal: def __init__(self, model_final): self._model_final = clone(model_final, safe=False) - def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, + sample_weight=None, freq_weight=None, sample_var=None, groups=None): Y_res, T_res = nuisances self._model_final.fit(X, T, T_res, Y_res, sample_weight=sample_weight, freq_weight=freq_weight, sample_var=sample_var) @@ -393,7 +395,7 @@ def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, def predict(self, X=None): return self._model_final.predict(X) - def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None): + def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, groups=None): Y_res, T_res = nuisances if Y_res.ndim == 1: Y_res = Y_res.reshape((-1, 1)) diff --git a/econml/iv/dr/_dr.py b/econml/iv/dr/_dr.py index 7ecd69fd0..48a68627b 100644 --- a/econml/iv/dr/_dr.py +++ b/econml/iv/dr/_dr.py @@ -75,7 +75,8 @@ def _effect_estimate(self, nuisances): self._cov_clip, np.inf) return prel_theta + (res_y - prel_theta * res_t) * res_z / clipped_cov, clipped_cov - def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, Y, T, X=None, W=None, Z=None, nuisances=None, + sample_weight=None, freq_weight=None, sample_var=None, groups=None): self.d_y = Y.shape[1:] self.d_t = nuisances[1].shape[1:] self.d_z = nuisances[3].shape[1:] @@ -115,7 +116,7 @@ def predict(self, X=None): X = self._featurizer.transform(X) return self._model_final.predict(X).reshape((-1,) + self.d_y + self.d_t) - def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None): + def score(self, Y, T, X=None, W=None, Z=None, nuisances=None, sample_weight=None, groups=None): theta_dr, clipped_cov = self._effect_estimate(nuisances) if (X is not None) and (self._featurizer is not None): diff --git a/econml/policy/_drlearner.py b/econml/policy/_drlearner.py index d7a41833a..7280be32c 100644 --- a/econml/policy/_drlearner.py +++ b/econml/policy/_drlearner.py @@ -14,7 +14,8 @@ class _PolicyModelFinal(_ModelFinal): - def fit(self, Y, T, X=None, W=None, *, nuisances, sample_weight=None, freq_weight=None, sample_var=None): + def fit(self, Y, T, X=None, W=None, *, nuisances, + sample_weight=None, freq_weight=None, sample_var=None, groups=None): if sample_var is not None: warn('Parameter `sample_var` is ignored by the final estimator') sample_var = None @@ -38,7 +39,7 @@ def predict(self, X=None): return pred[:, np.newaxis, :] return pred - def score(self, Y, T, X=None, W=None, *, nuisances, sample_weight=None): + def score(self, Y, T, X=None, W=None, *, nuisances, sample_weight=None, groups=None): return 0 diff --git a/econml/tests/dgp.py b/econml/tests/dgp.py new file mode 100644 index 000000000..403783447 --- /dev/null +++ b/econml/tests/dgp.py @@ -0,0 +1,219 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import abc +import numpy as np +from econml.utilities import cross_product +from statsmodels.tools.tools import add_constant +try: + import matplotlib + import matplotlib.pyplot as plt +except ImportError as exn: + from .utilities import MissingModule + + # make any access to matplotlib or plt throw an exception + matplotlib = plt = MissingModule("matplotlib is no longer a dependency of the main econml package; " + "install econml[plt] or econml[all] to require it, or install matplotlib " + "separately, to use the tree interpreters", exn) + + +class _BaseDynamicPanelDGP: + + def __init__(self, n_periods, n_treatments, n_x): + self.n_periods = n_periods + self.n_treatments = n_treatments + self.n_x = n_x + return + + @abc.abstractmethod + def create_instance(self, *args, **kwargs): + pass + + @abc.abstractmethod + def _gen_data_with_policy(self, n_units, policy_gen, random_seed=123): + pass + + def static_policy_data(self, n_units, tau, random_seed=123): + def policy_gen(Tpre, X, period): + return tau[period] + return self._gen_data_with_policy(n_units, policy_gen, random_seed=random_seed) + + def adaptive_policy_data(self, n_units, policy_gen, random_seed=123): + return self._gen_data_with_policy(n_units, policy_gen, random_seed=random_seed) + + def static_policy_effect(self, tau, mc_samples=1000): + Y_tau, _, _, _ = self.static_policy_data(mc_samples, tau) + Y_zero, _, _, _ = self.static_policy_data( + mc_samples, np.zeros((self.n_periods, self.n_treatments))) + return np.mean(Y_tau[np.arange(Y_tau.shape[0]) % self.n_periods == self.n_periods - 1]) - \ + np.mean(Y_zero[np.arange(Y_zero.shape[0]) % + self.n_periods == self.n_periods - 1]) + + def adaptive_policy_effect(self, policy_gen, mc_samples=1000): + Y_tau, _, _, _ = self.adaptive_policy_data(mc_samples, policy_gen) + Y_zero, _, _, _ = self.static_policy_data( + mc_samples, np.zeros((self.n_periods, self.n_treatments))) + return np.mean(Y_tau[np.arange(Y_tau.shape[0]) % self.n_periods == self.n_periods - 1]) - \ + np.mean(Y_zero[np.arange(Y_zero.shape[0]) % + self.n_periods == self.n_periods - 1]) + + +class DynamicPanelDGP(_BaseDynamicPanelDGP): + + def __init__(self, n_periods, n_treatments, n_x): + super().__init__(n_periods, n_treatments, n_x) + + def create_instance(self, s_x, sigma_x=.8, sigma_y=.1, conf_str=5, hetero_strength=.5, hetero_inds=None, + autoreg=.25, state_effect=.25, random_seed=123): + np.random.seed(random_seed) + self.s_x = s_x + self.conf_str = conf_str + self.sigma_x = sigma_x + self.sigma_y = sigma_y + self.hetero_inds = hetero_inds.astype( + int) if hetero_inds is not None else hetero_inds + self.endo_inds = np.setdiff1d( + np.arange(self.n_x), hetero_inds).astype(int) + # The first s_x state variables are confounders. The final s_x variables are exogenous and can create + # heterogeneity + self.Alpha = np.random.uniform(-1, 1, + size=(self.n_x, self.n_treatments)) + self.Alpha /= np.linalg.norm(self.Alpha, axis=1, ord=1, keepdims=True) + self.Alpha *= state_effect + if self.hetero_inds is not None: + self.Alpha[self.hetero_inds] = 0 + + self.Beta = np.zeros((self.n_x, self.n_x)) + for t in range(self.n_x): + self.Beta[t, :] = autoreg * np.roll(np.random.uniform(low=4.0**(-np.arange( + 0, self.n_x)), high=4.0**(-np.arange(1, self.n_x + 1))), t) + if self.hetero_inds is not None: + self.Beta[np.ix_(self.endo_inds, self.hetero_inds)] = 0 + self.Beta[np.ix_(self.hetero_inds, self.endo_inds)] = 0 + + self.epsilon = np.random.uniform(-1, 1, size=self.n_treatments) + self.zeta = np.zeros(self.n_x) + self.zeta[:self.s_x] = self.conf_str / self.s_x + + self.y_hetero_effect = np.zeros(self.n_x) + self.x_hetero_effect = np.zeros(self.n_x) + if self.hetero_inds is not None: + self.y_hetero_effect[self.hetero_inds] = np.random.uniform(.5 * hetero_strength, 1.5 * hetero_strength) / \ + len(self.hetero_inds) + self.x_hetero_effect[self.hetero_inds] = np.random.uniform(.5 * hetero_strength, 1.5 * hetero_strength) / \ + len(self.hetero_inds) + + self.true_effect = np.zeros((self.n_periods, self.n_treatments)) + # Invert indices to match latest API + self.true_effect[self.n_periods - 1] = self.epsilon + for t in np.arange(self.n_periods - 2, -1, -1): + self.true_effect[t, :] = (self.zeta.reshape( + 1, -1) @ np.linalg.matrix_power(self.Beta, (self.n_periods - 1 - t) - 1) @ self.Alpha) + + self.true_hetero_effect = np.zeros( + (self.n_periods, (self.n_x + 1) * self.n_treatments)) + self.true_hetero_effect[self.n_periods - 1, :] = cross_product( + add_constant(self.y_hetero_effect.reshape(1, -1), has_constant='add'), + self.epsilon.reshape(1, -1)) + for t in np.arange(self.n_periods - 2, -1, -1): + # Invert indices to match latest API + self.true_hetero_effect[t, :] = cross_product( + add_constant(self.x_hetero_effect.reshape(1, -1), has_constant='add'), + self.zeta.reshape(1, -1) @ np.linalg.matrix_power( + self.Beta, (self.n_periods - 1 - t) - 1) @ self.Alpha) + return self + + def hetero_effect_fn(self, t, x): + if t == self.n_periods - 1: + return (np.dot(self.y_hetero_effect, x.flatten()) + 1) * self.epsilon + else: + return (np.dot(self.x_hetero_effect, x.flatten()) + 1) *\ + (self.zeta.reshape(1, -1) @ np.linalg.matrix_power(self.Beta, (self.n_periods - 1 - t) - 1) + @ self.Alpha).flatten() + + def _gen_data_with_policy(self, n_units, policy_gen, random_seed=123): + np.random.seed(random_seed) + Y = np.zeros(n_units * self.n_periods) + T = np.zeros((n_units * self.n_periods, self.n_treatments)) + X = np.zeros((n_units * self.n_periods, self.n_x)) + groups = np.zeros(n_units * self.n_periods) + for t in range(n_units * self.n_periods): + period = t % self.n_periods + if period == 0: + X[t] = np.random.normal(0, self.sigma_x, size=self.n_x) + const_x0 = X[t][self.hetero_inds] + T[t] = policy_gen(np.zeros(self.n_treatments), X[t], period) + else: + X[t] = (np.dot(self.x_hetero_effect, X[t - 1]) + 1) * np.dot(self.Alpha, T[t - 1]) + \ + np.dot(self.Beta, X[t - 1]) + \ + np.random.normal(0, self.sigma_x, size=self.n_x) + # The feature for heterogeneity stays constant + X_t = X[t].copy() + X_t[self.hetero_inds] = const_x0 + T[t] = policy_gen(T[t - 1], X[t], period) + Y[t] = (np.dot(self.y_hetero_effect, X_t if period != 0 else X[t]) + 1) * np.dot(self.epsilon, T[t]) + \ + np.dot(X[t], self.zeta) + \ + np.random.normal(0, self.sigma_y) + groups[t] = t // self.n_periods + + return Y, T, X[:, self.hetero_inds] if (self.hetero_inds is not None) else None, X[:, self.endo_inds], groups + + def observational_data(self, n_units, gamma=0, s_t=1, sigma_t=0.5, random_seed=123): + """Generate observational data with some observational treatment policy parameters. + + Parameters + ---------- + n_units : how many units to observe + gamma : what is the degree of auto-correlation of the treatments across periods + s_t : sparsity of treatment policy; how many states does it depend on + sigma_t : what is the std of the exploration/randomness in the treatment + """ + Delta = np.zeros((self.n_treatments, self.n_x)) + Delta[:, :s_t] = self.conf_str / s_t + + def policy_gen(Tpre, X, period): + return gamma * Tpre + (1 - gamma) * np.dot(Delta, X) + \ + np.random.normal(0, sigma_t, size=self.n_treatments) + return self._gen_data_with_policy(n_units, policy_gen, random_seed=random_seed) + + +# Auxiliary function for adding xticks and vertical lines when plotting results +# for dynamic dml vs ground truth parameters. +def add_vlines(n_periods, n_treatments, hetero_inds): + locs, labels = plt.xticks([], []) + locs += [- .501 + (len(hetero_inds) + 1) / 2] + labels += ["\n\n$\\tau_{{{}}}$".format(0)] + locs += [qx for qx in np.arange(len(hetero_inds) + 1)] + labels += ["$1$"] + ["$x_{{{}}}$".format(qx) for qx in hetero_inds] + for q in np.arange(1, n_treatments): + plt.axvline(x=q * (len(hetero_inds) + 1) - .5, + linestyle='--', color='red', alpha=.2) + locs += [q * (len(hetero_inds) + 1) - .501 + (len(hetero_inds) + 1) / 2] + labels += ["\n\n$\\tau_{{{}}}$".format(q)] + locs += [(q * (len(hetero_inds) + 1) + qx) + for qx in np.arange(len(hetero_inds) + 1)] + labels += ["$1$"] + ["$x_{{{}}}$".format(qx) for qx in hetero_inds] + locs += [- .501 + (len(hetero_inds) + 1) * n_treatments / 2] + labels += ["\n\n\n\n$\\theta_{{{}}}$".format(0)] + for t in np.arange(1, n_periods): + plt.axvline(x=t * (len(hetero_inds) + 1) * + n_treatments - .5, linestyle='-', alpha=.6) + locs += [t * (len(hetero_inds) + 1) * n_treatments - .501 + + (len(hetero_inds) + 1) * n_treatments / 2] + labels += ["\n\n\n\n$\\theta_{{{}}}$".format(t)] + locs += [t * (len(hetero_inds) + 1) * + n_treatments - .501 + (len(hetero_inds) + 1) / 2] + labels += ["\n\n$\\tau_{{{}}}$".format(0)] + locs += [t * (len(hetero_inds) + 1) * n_treatments + + qx for qx in np.arange(len(hetero_inds) + 1)] + labels += ["$1$"] + ["$x_{{{}}}$".format(qx) for qx in hetero_inds] + for q in np.arange(1, n_treatments): + plt.axvline(x=t * (len(hetero_inds) + 1) * n_treatments + q * (len(hetero_inds) + 1) - .5, + linestyle='--', color='red', alpha=.2) + locs += [t * (len(hetero_inds) + 1) * n_treatments + q * + (len(hetero_inds) + 1) - .501 + (len(hetero_inds) + 1) / 2] + labels += ["\n\n$\\tau_{{{}}}$".format(q)] + locs += [t * (len(hetero_inds) + 1) * n_treatments + (q * (len(hetero_inds) + 1) + qx) + for qx in np.arange(len(hetero_inds) + 1)] + labels += ["$1$"] + ["$x_{{{}}}$".format(qx) for qx in hetero_inds] + plt.xticks(locs, labels) + plt.tight_layout() diff --git a/econml/tests/test_dynamic_dml.py b/econml/tests/test_dynamic_dml.py new file mode 100644 index 000000000..7539c18f9 --- /dev/null +++ b/econml/tests/test_dynamic_dml.py @@ -0,0 +1,298 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. +import unittest +import pytest +import pickle +import numpy as np +from contextlib import ExitStack +from sklearn.preprocessing import OneHotEncoder, FunctionTransformer, PolynomialFeatures +from sklearn.linear_model import (LinearRegression, LassoCV, Lasso, MultiTaskLasso, + MultiTaskLassoCV, LogisticRegression) +from econml.dynamic.dml import DynamicDML +from econml.dynamic.dml._dml import _get_groups_period_filter +from econml.inference import BootstrapInference, EmpiricalInferenceResults, NormalInferenceResults +from econml.utilities import shape, hstack, vstack, reshape, cross_product +import econml.tests.utilities # bugfix for assertWarns +from econml.tests.dgp import DynamicPanelDGP + + +@pytest.mark.dml +class TestDynamicDML(unittest.TestCase): + + def test_cate_api(self): + """Test that we correctly implement the CATE API.""" + n_panels = 100 # number of panels + n_periods = 3 # number of time periods per panel + n = n_panels * n_periods + groups = np.repeat(a=np.arange(n_panels), repeats=n_periods, axis=0) + + def make_random(n, is_discrete, d): + if d is None: + return None + sz = (n, d) if d >= 0 else (n,) + if is_discrete: + return np.random.choice(['a', 'b', 'c'], size=sz) + else: + return np.random.normal(size=sz) + + for d_t in [2, 1, -1]: + for is_discrete in [True, False] if d_t <= 1 else [False]: + # for is_discrete in [False]: + for d_y in [3, 1, -1]: + for d_x in [2, None]: + for d_w in [2, None]: + W, X, Y, T = [make_random(n, is_discrete, d) + for is_discrete, d in [(False, d_w), + (False, d_x), + (False, d_y), + (is_discrete, d_t)]] + T_test = np.hstack([(T.reshape(-1, 1) if d_t == -1 else T) for i in range(n_periods)]) + for featurizer, fit_cate_intercept in\ + [(None, True), + (PolynomialFeatures(degree=2, include_bias=False), True), + (PolynomialFeatures(degree=2, include_bias=True), False)]: + + d_t_final = (2 if is_discrete else max(d_t, 1)) * n_periods + + effect_shape = (n,) + ((d_y,) if d_y > 0 else ()) + effect_summaryframe_shape = (n * (d_y if d_y > 0 else 1), 6) + marginal_effect_shape = ((n,) + + ((d_y,) if d_y > 0 else ()) + + ((d_t_final,) if d_t_final > 0 else ())) + marginal_effect_summaryframe_shape = (n * (d_y if d_y > 0 else 1) * + (d_t_final if d_t_final > 0 else 1), 6) + + # since T isn't passed to const_marginal_effect, defaults to one row if X is None + const_marginal_effect_shape = ((n if d_x else 1,) + + ((d_y,) if d_y > 0 else ()) + + ((d_t_final,) if d_t_final > 0 else())) + const_marginal_effect_summaryframe_shape = ( + (n if d_x else 1) * (d_y if d_y > 0 else 1) * + (d_t_final if d_t_final > 0 else 1), 6) + + fd_x = featurizer.fit_transform(X).shape[1:] if featurizer and d_x\ + else ((d_x,) if d_x else (0,)) + coef_shape = Y.shape[1:] + (d_t_final, ) + fd_x + + coef_summaryframe_shape = ( + (d_y if d_y > 0 else 1) * (fd_x[0] if fd_x[0] > + 0 else 1) * (d_t_final), 6) + intercept_shape = Y.shape[1:] + (d_t_final, ) + intercept_summaryframe_shape = ( + (d_y if d_y > 0 else 1) * (d_t_final if d_t_final > 0 else 1), 6) + + all_infs = [None, 'auto', BootstrapInference(2)] + est = DynamicDML(model_y=Lasso() if d_y < 1 else MultiTaskLasso(), + model_t=LogisticRegression() if is_discrete else + (Lasso() if d_t < 1 else MultiTaskLasso()), + featurizer=featurizer, + fit_cate_intercept=fit_cate_intercept, + discrete_treatment=is_discrete) + + # ensure we can serialize the unfit estimator + pickle.dumps(est) + + for inf in all_infs: + with self.subTest(d_w=d_w, d_x=d_x, d_y=d_y, d_t=d_t, + is_discrete=is_discrete, est=est, inf=inf): + + if X is None and (not fit_cate_intercept): + with pytest.raises(AttributeError): + est.fit(Y, T, X=X, W=W, groups=groups, inference=inf) + continue + + est.fit(Y, T, X=X, W=W, groups=groups, inference=inf) + + # ensure we can pickle the fit estimator + pickle.dumps(est) + + # make sure we can call the marginal_effect and effect methods + const_marg_eff = est.const_marginal_effect(X) + marg_eff = est.marginal_effect(T_test, X) + self.assertEqual(shape(marg_eff), marginal_effect_shape) + self.assertEqual(shape(const_marg_eff), const_marginal_effect_shape) + + np.testing.assert_allclose( + marg_eff if d_x else marg_eff[0:1], const_marg_eff) + + assert len(est.score_) == n_periods + for score in est.nuisance_scores_y[0]: + assert score.shape == (n_periods, ) + for score in est.nuisance_scores_t[0]: + assert score.shape == (n_periods, n_periods) + + T0 = np.full_like(T_test, 'a') if is_discrete else np.zeros_like(T_test) + eff = est.effect(X, T0=T0, T1=T_test) + self.assertEqual(shape(eff), effect_shape) + + self.assertEqual(shape(est.coef_), coef_shape) + if fit_cate_intercept: + self.assertEqual(shape(est.intercept_), intercept_shape) + else: + with pytest.raises(AttributeError): + self.assertEqual(shape(est.intercept_), intercept_shape) + + if inf is not None: + const_marg_eff_int = est.const_marginal_effect_interval(X) + marg_eff_int = est.marginal_effect_interval(T_test, X) + self.assertEqual(shape(marg_eff_int), + (2,) + marginal_effect_shape) + self.assertEqual(shape(const_marg_eff_int), + (2,) + const_marginal_effect_shape) + self.assertEqual(shape(est.effect_interval(X, T0=T0, T1=T_test)), + (2,) + effect_shape) + self.assertEqual(shape(est.coef__interval()), + (2,) + coef_shape) + if fit_cate_intercept: + self.assertEqual(shape(est.intercept__interval()), + (2,) + intercept_shape) + else: + with pytest.raises(AttributeError): + self.assertEqual(shape(est.intercept__interval()), + (2,) + intercept_shape) + + const_marg_effect_inf = est.const_marginal_effect_inference(X) + T1 = np.full_like(T_test, 'b') if is_discrete else T_test + effect_inf = est.effect_inference(X, T0=T0, T1=T1) + marg_effect_inf = est.marginal_effect_inference(T_test, X) + # test const marginal inference + self.assertEqual(shape(const_marg_effect_inf.summary_frame()), + const_marginal_effect_summaryframe_shape) + self.assertEqual(shape(const_marg_effect_inf.point_estimate), + const_marginal_effect_shape) + self.assertEqual(shape(const_marg_effect_inf.stderr), + const_marginal_effect_shape) + self.assertEqual(shape(const_marg_effect_inf.var), + const_marginal_effect_shape) + self.assertEqual(shape(const_marg_effect_inf.pvalue()), + const_marginal_effect_shape) + self.assertEqual(shape(const_marg_effect_inf.zstat()), + const_marginal_effect_shape) + self.assertEqual(shape(const_marg_effect_inf.conf_int()), + (2,) + const_marginal_effect_shape) + np.testing.assert_array_almost_equal( + const_marg_effect_inf.conf_int()[0], + const_marg_eff_int[0], decimal=5) + const_marg_effect_inf.population_summary()._repr_html_() + + # test effect inference + self.assertEqual(shape(effect_inf.summary_frame()), + effect_summaryframe_shape) + self.assertEqual(shape(effect_inf.point_estimate), + effect_shape) + self.assertEqual(shape(effect_inf.stderr), + effect_shape) + self.assertEqual(shape(effect_inf.var), + effect_shape) + self.assertEqual(shape(effect_inf.pvalue()), + effect_shape) + self.assertEqual(shape(effect_inf.zstat()), + effect_shape) + self.assertEqual(shape(effect_inf.conf_int()), + (2,) + effect_shape) + np.testing.assert_array_almost_equal( + effect_inf.conf_int()[0], + est.effect_interval(X, T0=T0, T1=T1)[0], decimal=5) + effect_inf.population_summary()._repr_html_() + + # test marginal effect inference + self.assertEqual(shape(marg_effect_inf.summary_frame()), + marginal_effect_summaryframe_shape) + self.assertEqual(shape(marg_effect_inf.point_estimate), + marginal_effect_shape) + self.assertEqual(shape(marg_effect_inf.stderr), + marginal_effect_shape) + self.assertEqual(shape(marg_effect_inf.var), + marginal_effect_shape) + self.assertEqual(shape(marg_effect_inf.pvalue()), + marginal_effect_shape) + self.assertEqual(shape(marg_effect_inf.zstat()), + marginal_effect_shape) + self.assertEqual(shape(marg_effect_inf.conf_int()), + (2,) + marginal_effect_shape) + np.testing.assert_array_almost_equal( + marg_effect_inf.conf_int()[0], marg_eff_int[0], decimal=5) + marg_effect_inf.population_summary()._repr_html_() + + # test coef__inference and intercept__inference + if X is not None: + self.assertEqual( + shape(est.coef__inference().summary_frame()), + coef_summaryframe_shape) + np.testing.assert_array_almost_equal( + est.coef__inference().conf_int() + [0], est.coef__interval()[0], decimal=5) + + if fit_cate_intercept: + cm = ExitStack() + # ExitStack can be used as a "do nothing" ContextManager + else: + cm = pytest.raises(AttributeError) + with cm: + self.assertEqual(shape(est.intercept__inference(). + summary_frame()), + intercept_summaryframe_shape) + np.testing.assert_array_almost_equal( + est.intercept__inference().conf_int() + [0], est.intercept__interval()[0], decimal=5) + + est.summary() + est.score(Y, T, X, W, groups=groups) + # make sure we can call effect with implied scalar treatments, + # no matter the dimensions of T, and also that we warn when there + # are multiple treatments + if d_t > 1: + cm = self.assertWarns(Warning) + else: + # ExitStack can be used as a "do nothing" ContextManager + cm = ExitStack() + with cm: + effect_shape2 = (n if d_x else 1,) + ((d_y,) if d_y > 0 else()) + eff = est.effect(X) if not is_discrete else est.effect( + X, T0='a', T1='b') + self.assertEqual(shape(eff), effect_shape2) + + def test_perf(self): + np.random.seed(123) + n_units = 1000 + n_periods = 3 + n_treatments = 1 + n_x = 100 + s_x = 10 + s_t = 10 + hetero_strength = .5 + hetero_inds = np.arange(n_x - n_treatments, n_x) + + def lasso_model(): + return LassoCV(cv=3) + + # No heterogeneity + dgp = DynamicPanelDGP(n_periods, n_treatments, n_x).create_instance( + s_x, random_seed=12345) + Y, T, X, W, groups = dgp.observational_data(n_units, s_t=s_t, random_seed=12345) + est = DynamicDML(model_y=lasso_model(), model_t=lasso_model(), cv=3) + # Define indices to test + groups_filter = _get_groups_period_filter(groups, 3) + shuffled_idx = np.array([groups_filter[i] for i in range(n_periods)]).flatten() + test_indices = [np.arange(n_units * n_periods), shuffled_idx] + for test_idx in test_indices: + est.fit(Y[test_idx], T[test_idx], X=X[test_idx] if X is not None else None, W=W[test_idx], + groups=groups[test_idx], inference="auto") + np.testing.assert_allclose(est.intercept_, dgp.true_effect.flatten(), atol=0.2) + np.testing.assert_array_less(est.intercept__interval()[0], dgp.true_effect.flatten()) + np.testing.assert_array_less(dgp.true_effect.flatten(), est.intercept__interval()[1]) + + # Heterogeneous effects + dgp = DynamicPanelDGP(n_periods, n_treatments, n_x).create_instance( + s_x, hetero_strength=hetero_strength, hetero_inds=hetero_inds, random_seed=12) + Y, T, X, W, groups = dgp.observational_data(n_units, s_t=s_t, random_seed=1) + hetero_strength = .5 + hetero_inds = np.arange(n_x - n_treatments, n_x) + for test_idx in test_indices: + est.fit(Y[test_idx], T[test_idx], X=X[test_idx], W=W[test_idx], groups=groups[test_idx], inference="auto") + np.testing.assert_allclose(est.intercept_, dgp.true_effect.flatten(), atol=0.2) + np.testing.assert_allclose(est.coef_, dgp.true_hetero_effect[:, hetero_inds + 1], atol=0.2) + np.testing.assert_array_less(est.intercept__interval()[0], dgp.true_effect.flatten()) + np.testing.assert_array_less(dgp.true_effect.flatten(), est.intercept__interval()[1]) + np.testing.assert_array_less(est.coef__interval()[0] - .05, dgp.true_hetero_effect[:, hetero_inds + 1]) + np.testing.assert_array_less(dgp.true_hetero_effect[:, hetero_inds + 1] - .05, est.coef__interval()[1]) diff --git a/notebooks/CustomerScenarios/Case Study - Long-Term Return-on-Investment at Microsoft via Short-Term Proxies.ipynb b/notebooks/CustomerScenarios/Case Study - Long-Term Return-on-Investment at Microsoft via Short-Term Proxies.ipynb new file mode 100644 index 000000000..4c6ca7658 --- /dev/null +++ b/notebooks/CustomerScenarios/Case Study - Long-Term Return-on-Investment at Microsoft via Short-Term Proxies.ipynb @@ -0,0 +1,1090 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "# Long-Term Return-on-Investment at Microsoft via Short-Term Proxies\n", + "\n", + "\n", + "Policy makers typically face the problem of wanting to estimate the treatment effect of some new incentives on long-run downstream interests. However, we only have historical data of older treatment options, and we haven't seen the long-run play out yet. We assume access to a long-term dataset where only past treatments were administered and a short-term dataset where novel treatments have been administered. We propose a surrogate based approach where we assume that the long-term effect is channeled through a multitude of available short-term proxies. Our work combines three major recent techniques in the causal machine learning literature: **surrogate indices**, **dynamic treatment effect estimation** and **double machine learning**, in a unified\n", + "pipeline. For more details, see this paper [here](https://arxiv.org/pdf/2103.08390.pdf).\n", + "\n", + "In this case study, we will show you how to apply this unified pipeline to a ROI estimation problem at Microsoft. These methodologies have already been implemented into our [EconML](https://aka.ms/econml) library and you could do it with only a few lines of code." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Summary\n", + "\n", + "1. [Background](#Background)\n", + "2. [Data](#Data)\n", + "3. [Do Dynamic Adjustment with EconML](#Do-Dynamic-Adjustment-with-EconML)\n", + "4. [Train Surrogate Index](#Train-Surrogate-Index)\n", + "5. [Run DML to Learn ROI with EconML](#Run-DML-to-Learn-ROI-with-EconML)\n", + "6. [Model Evaluation](#Model-Evaluation)\n", + "7. [Extensions -- Including Heterogeneity in Effect](#Extensions----Including-Heterogeneity-in-Effect)\n", + "8. [Conclusions](#Conclusions)" + ] + }, + { + "attachments": { + "causal_graph.PNG": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAHQCAYAAAB6Jl+NAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAHYcAAB2HAY/l8WUAAHHJSURBVHhe7b1brGVnea6Zfb+VwF1uog1S3/VNuG1tofgulxt1pERqqRsuctPZvQGJqCNtoYrYocPe2mAiQA4SUSEOCh0S1QZyEcAtBzDY2AXGsR1X2YXLJ3wuV+FDFXUws/2uWq/rq9/fGHPMucbpH+N5pEdrrXEeY8655vfO///H/I0NAAAAAABAZRBkAAAAAACgOggyAAAAAABQHQQZAAAAAACoDoIMAAAAAABUB0EGAAAAAACqgyADAAAAAADVQZABAAAAAIDqIMgAAAAAAEB1EGQAAAAAAKA6CDIAMFvOX7q4OXH6gc2Hv/uNzXu+8KnNuz/78c07P/nRzb/5+EcQJ1fPRT0n3/f1L24+8K2vbY7ff8/hMxcMr2Gcu3pO3vKV2w5ew7f++PubsxfOHT57oQYIMgAwO+54/MxB4UPBgzWqYLP2YkivYQUXXsNYo3oN88FEHRBkAGA26NNbfSqWvbEg1qaey2sLNPc994uDIjC7Hoi1qTBOC828IcgAwCzQJ7h8eotLVF2r1oC65fAaxiWq5zbME4IMAEyO3iSyNw/Epbj0QujY976dnjfiUtRzHOYHQQYAJoUQg2txqWGGEINrUWM3YV4QZABgMjSYMnuzQFyqS+tmxgcRuDa5CcC8IMgAwCRoACX96XFt6jm/lMHDvIZxreqmFjAPCDIAMAncnQzXqr6zYgnoPLLzQ1y6upsZzAOCDACMju5Qlr05IK7F2j/RVRe57LwQ1+Ja7kY4dwgyADA6fM8Erl29BmqG1zCuXVpl5gFBBgBGRV96Sb96xI9UO1ZGx52dD+LaXMp4t5ohyADAqNAlBfG6td6OmbsNIl536d8PVQMEGQAYFQb5I1631u5lvIYRr1t7F9ElQJABgFGhbz3idd/92Y8fvirqgruVIV5X3aRhWggyADAqKt6yNwTEtVlrEcRrGPG6BJnpIcgAwKgw0B/xhjXCaxjxhjAtBBkAGJXsjQBxrdZIdh6IaxWmhSADAKOSvREgrtUayc4Dca3CtBBkAGBUsjcCxLVaI9l5IK5VmBaCDACMSvZGgLhWayQ7D8S1CtNCkAGAUcneCBDXao1k54G4VmFaCDIAMCrZGwHiWq2R7DwQ1ypMC0EGAEYleyNAXKs1kp0H4lqFaSHIAMCoZG8EiGu1RrLzQFyrMC0EGQAYleyNAHGt1kh2HohrFaaFIAMAo5K9ESCu1RrJzgNxrcK0EGQAYFSyNwLEtVoj2XkgrlWYFoIMAIxK9kawBt/12Y9vvvzATzaPnHvx8EpsNq9fubK56+nHN+/90ufSdXD51kh2Hmvzj058+fBqNPPB7/yPdF1cljAtBBkAGJXsjWDpquh56eLrh1dgs3nqlQsHgSZO+8y9d6brLlkVeuL+559J56/BGsnOY236uSv0Ws7U6z5bdxe/8ciDB/vQhyDZfJxemBaCDACMSvZGsGTV2qKWF6EAU7a+qCDSdBGnr0EXgyr6svlrsEay81ibMchk8/vSQUY/s/k4vTAtBBkAGJXsjWDJqkgXCivqXpYto+m3n30knbdkCTIEmVolyKCFaSHIAMCoZG8ES1WtL+ZjP/huukymupGIpi5XGlcjyu4mLnpUZEl3XdNPL+Ngpd+1vluLtE0vI3Xsmub5QutmXeBc1Gn/CmX66X1rfU8vl29iTWOGaiQ7j7W5a5Dx6yjrbub/E1rG0/xazoivZ/2PEE1dz/w6LPcr9Hr2hyimfH3rPL0PoWPU35oel1uzMC0EGQAYleyNYKk6kMQCpYsuYppaKhxGtFyc7vX8U/vVsnH/cV2hQkfT4r5U9GidON/d30QZelzUqcBx4aTl4zrl9uM2fZy2qeVqidZIdh5rc9cgY7IAkG1LgUKvBb+e4us0vv70tyj/F1hT7ld4e8Kv17gdHYPRPC3r4xFrHNeXCdNCkAGAUcneCJaqP8lUAZDNb9Iho2k9Fx9l8eL1hAqPLBB4XVEGEql1HGLKVqQYcGJhFAsxzY/ztA1Tfirs9Xa9PkuyRrLzWJvxOZ/NLzXxtWHbtuXXdPlat03/C6wp9xtpa2VVcClftw44eq2v6UOHJmFaCDIAMCrZG8FSdZGx6/gXFy9NBX5T8eL12goMr9u0bbciNR2z9xH37aJH+y2LHumWl/J4vV7TsazBvjh+/z2b85cuHv41LNl5rM0YPvT8LdWHGHF5UwYKGbdVzsteb1Htq22+KfdrmtbzhzBNXWK93+x81iZMC0EGAEYleyNYqtuKjCZdvGj9bH7Tdr1e2/62FSAuYGJXlqi7luh3r+NCLE6LNh3XtvXWYF8c+963N+/5wqdGCTPZeaxNP3ebUKiPy5vsdRe3Vc7b9prWa6dtvin3a+K0qI5fuEtZqec37XdNwrQQZABgVLI3gqWqN3yhn9n8Jl28NK3n7ZZFhNdrKy68blZQxfnbiC02LsSajrfpuLattwb7QkFG21OYGZryHNZoW/jINNnrrm1b217TTf8LrCn3a+K0aFd2uYnJUoVpIcgAwKhkbwRL1UWIWjGy+U16vaYCv6l48XpNRY30ullBFee3baPUhVjT8TYd17b11mBfOMjIW75y2+HUYYjHv1bbwkemyV53bdva9pre9no15X5NnBY12fHizcK0EGQAYFSyN4Kl2jbQvU0XNk0FflPxsq3okV63qUDZVhhlbjvepuPatt4a7IsYZOSQYSbuZ636uSuy+aVtr7u2bW17TW+bb8r9mjgtarLjxZtt4qErr0/mv77p6asXNz+/emnzxNVfbZ67dnnz6q+vHR7ZsiDIAMCoZG8ESzYbU5KprloeoO/CpuxnL7WMt1kWL9uKGtlWUEkP9t+lFcnHS5DZ3b4og4wcKsyU+1mjfu6KbH6pX3fZ9734NSfKedte055f3lxA6sMTU77eTZwW9Vi57M6GeLNNZAFjSk9duXgQcJ649qvNhTeubt44PM7aIcgAwKhkbwRLNrbKqDgov+xRBYbv6uV5CismFj6a72VFWdxsK3rktiCjfXggr4qY8u5nKo40PZ7HtkDSdFyxGGy6y9rS7YssyMgPf/cbh0v0R7aftRmfu9n8Un/ppD4giM/1GGJEXEf6taPXfTlPNrX66vj8Ohbl693EadG43Sx8aT4h57pNZGFiLp66enHz8Js/z70ZaGqHIAMAo5K9ESxdfe9CLCp8J6AYSsrvcnDhI7y80Hb8aWkZDJoCQ3RbkJH+ngijdaRbgkRZNAktE7dj247L29R5aX39LMPeki254/Eze/mBb30t3b5UyOmTbB9rc9cgEz8g8HPdz30FApOtZ7S814vLxP8jmu+/9dO/7xpkZDwuH7Nfo56Wrbc2m8gCxNw86IJ25eLmwhv1djsjyADAqGRvBGtQBYnCSSw6VJCUrRtRFf6x0Pey/hS3/KTUgSH7BNWqEBHbxuy45cX7Fzp2TSuLoqMEGe3HxySatrFUS7Jl+lDfM9MX2fbXpp/zZahoU891fwgh9HryBxh6fTcFA7V+xNdh+TpSC49elw4YWlbLaLr3V77eRZcgon1rG9620GtU/8ua/m+tzSay4DBXH34zzJy9emlzbfPrw6OvB4IMAIxK9kaAuFZLsmX6sq8wk20bca02kQWGuatxNJd+XdfoGYIMAIxK9kaAuFZLsmX69MTpBw73tD/ZdhHXahNZUKjBf72qrmb1jJ0hyADAqGRvBIhrtSRbpm81puYoZNtEXKtNZCGhJmsJMwQZABiV7I0Aca2WZMsM4X3P/eJwj7uTbQ9xrTaRhYPavFhBNzOCDACMSvZGgLhWS7JlhvCdn/zo3mEm2x7iWm0iCwa1qS/VnPsNAAgyADAq2RsB4lotyZYZyn3DTLYtxLXaRBYMalR3M5szBBkAGJXsjQBxrZZkywzpuz9748tXu5JtB3GtNpGFghrVd83M+XtmCDIAMCrZGwHiWi3JlhlKWmQQj24TWSioVX1p5lwhyADAqGRvBIhrtSRbZggVYva9FXO2PcS12kQWCGpVrTLnZnoXM4IMAIxK9kaAuFZLsmX69ighRmTbRFyrTTx/7fJkPvOmT1z91cEXXMosnOzqw286x3uYEWQAYFSyNwLEtVqSLdOnRw0xItsu4lqdOy+/cXVz5uqlg+5hWUDpqlplfvnr+Y2VIcgAwKhkbwSIa7UkW6ZPj99/z+Ge9ifbLuJarYXnr105aFXJQkpXn7x2+XBr84EgAwCjkr0RIK7VkmyZvrz1x98/3MvRyLaNuFZr4qU3jhZm1CozNwgyADAq2RsB4lotyZbpww9/9xuHezg62fYR12ptPHft8t7dzBRkXp1Z9zKCDACMSvZGgLhWS+54/MxefuBbX0u3L/sMMSLbB+JarZGfX72UBpUuPjuz7mUEGQAYleyNYEq/8ciDB8d119OPp/MRh7Qvjn3v2+n2+w4xItvPlPIaximtkXPXrux9NzPdDW1OEGQAYFSyN4IpdRH0yLkX0/m1qvMRf3Tiy+n8mvVj9uUHfpLOr8m+yIKMWmnOX+r/i+zK/Uwtr+H6/OB3/sfBud3//DPp/JqslVNX9wsyj169dLiFeUCQAYBRyd4IpnTpRZAKhmx+zfox089sfk32RRlkbvnKbYOEGBH3Mwd5Ddeng8wSHrNaefLar9Kgsk215MwJggwAjEr2RjClFEH1SZB5OzHIvOcLnxosxIh4/HOQ13B9EmSm57k3rqRBZZtzu3MZQQYARiV7I5jSpiLI0/WG+94vfe6g//3rV64cTHvp4uubz9x7503Lu+ho6u6kbXj9OP1dn/34wb60TfPUKxcOpmleXFaqm4m6Y8Rj0bKe7+PIiGMIvJx+1zlqn0bbd3cWHUN57h/7wXff2k5pea2E9lVeL+lixucar4PWL6+B/m5C68Vt6xjjtdD5tR33VPaFg8zQIUaU5zC1fl7o8Y7Td31++flSy2tY6wj/j9I2Tfl8j8fnc/e80iFfw16+Ce3by67tNTw2z1+7nAaVLs4JggwAjEr2RjCleqMVZRHk6fENXcvEYkXLeHkVP0JvtnE75fzYJ1xv8FpeaB/afvnGHYuAWEjpOLSs/3bxdfvZR26arm14u7FA8360vPD+vZ5+qujw35oXzz0rKFSgeXkfn89PxCJMuqjRNfG2tXxcR9vw8iqk4nF4HzJuW8dmfP5mbuMN+kJBRiHm7IVzh1OGIzuPKW16De/6/KrtNdzlf5ReCz4+/fTvQvvxtuzQr2FtP25T+9Lf1tdqja/hsSHIAADsQfZGMKXbgozQG2n8pDAW/56m+SYua/3GHT/V9JtzWVDE4iiGJRUR5TQtq7/jNOltq9CI08v5Qtt1ARELLaHfY+HgY1DR4mlS63u9MuTE4igej4sgoflxXixkysLFj015ztYFVdyezkvFVtP1mMq+0Df2jxFiRHYeU+rnQ/ka3vX5Vdtr2Oct4v8obS++vkU8Zv1u4nmO+Rr2euVjZtf4Gh4bggwAwB5kbwRT6mKgfEP1dBUIeoOP8/S3iW+qLh7KgsQFkt7ovS29sQttPy5rszd6bz8WJU162aY3fc8vP2GVMaiVBYjPRcTp/rQ6+5RX+nrGa+NzzPYjs0JQZtuKmlikzdUayc5jSv18iK8Vuc/zy6+L8rk1x9ewzzv7HxVDRLavLCiM+RrOrk3U8BoeDoIMAMAeZG8EU+o35/INNXvTjmZFhj/pLFsrXCDE0OBpKgC0rVIXACJbZ1shpG2IpiKobX7TNbEmrqtPSoXO3ecQdeGk373OtmKm6THY9tj4WHQNm85/LtZIdh5T2vR83ef5VdNruO114HMX5Tzpbcd1x3wNb1uP1/DwEGQAAPYgeyOYUr/Rlm+oTW/AVsuL+CYbW2rip5MuaGJ3DW9/G1rX60h3TREqhvTpafapZXZ8Xec3XRNr4rre3jbip71aXzTtp+kxaJpu9TjEIlIFmArI8lPrOVgj2XlMadPzdZ/nV02v4ez4rc9dlPOktx3X9bRt9PEa3rYer+HhIcgAAOxB9kYwpX6jLd9Qm96Ard/0yyLDRYrf7N39pPyEt2m/XdQ2tR8VQab8dLfp+LrM33ZsJq7r7TVdr8xtxUzTY9A0vVRFpz/ZFbpeWfeXKa2R7Dym1M+H8nm07/Orltdw0/FLn7so50lvO66bTdvmvtd423qW1/BwEGQAAPYgeyOYUr/Rlm+oTW/A1m/6ZZHhvukuelQMifgppmza7y7q00kXXXqDj/Oajq/L/G3HZuK63l7T9crcVsw0PQZN05vUp93+dFdFUbbMVNZIdh5T6udD+Tza9/lVy2u46filz12U86S3HdfNpm1z32u8bb1SXsP9Q5ABANiD7I1gSv1GW76hNr0BW7/pZ0WGP2VVQaRiSJSfIvpTXnHUAa0mTms7vm3zm66JNXFd9/0vP7Vuc1sx0/QYbHtsMrftayprJDuPKW16vu77/JI1vIbbjt/nLsp50tuO6475Gt7n9bjPOmNYKwQZAIA9yN4IptRvtOWbY9MbsHUhkBUZ/gTXnyCWfeRtnF8WQvpb+1Yh5WkqMMruJ/70uOnT3PJT5HJ+dvxN18SauK6O18WfPmEu+7Kr6NP0eJ7bCpOmx8DTs+uq/Wh6eV5+TPg09+hk5zGlTc/XfZ9fsobXcNvx+9xFOU9623HdMV/D8fiy/fAaHh6CDADAHmRvBFPqN9ryjbjpDdi6ECjfbGX8pFbok85yGS/nwkHozVvbdXEk4rr+ZFjrtC0nVSwZLSdVhHh+2/E3XRNrynXjPoXWlz5uoXP28i5mmvbT9BiokDLatvfRNs/E/c/BGsnOY0qbnq/7Pr9kDa/htuP3uYtyntS+RbnuWK9hWV4L/dTrl9fwOBBkAAD2IHsjmFK/0cYCIU7P3oCl3lxFWcxbFyh+c86Wkfo0Up80xoJGb9r61LH85Fbb0XHGN3UdR/zEN6pjd5Gln3G5tuNXQSWaPvn0NrOCQtPKY3QBVu7rKEWQziXuIy6j7erY47nr77kVQLJGsvOYUj9PytfwUZ5fcu6v4bbj13NdaJ1yntSxiSygjfUa1n78f0jEbfAaHh6CDADAIdeuvnFgF7I3AsS1WiPZeSCu1VohyADAKlBAuXzp2ubiq1c2F168tHnhqdc2T595ZfPoz85tTp18afPQ3S9s/uXO5zePPXj+cI12sjcCxLVaI9l5IK7VWiHIAED1dA0p9/3zs1slyCDubo1k54G4VmuFIAMAVaLgopDSNaB0VaGnC9kbAeJarZHsPBDXaq0QZACgWtQCo+CRBZJ9VTDqQvZGgLhWayQ7D8S1WisEGQCoGnUrUxeyLJTsa5cB/9kbAeJarZHsPBDXaq0QZABgEWhMTBZK9lEtPdvI3ggQ12qNZOeBuFZrhSADAItBg/yzYLKrr56/fLjFZrI3AsS1WiPZeSCu1VohyADAotBdx7JwsosKRNvI3ggQ12qNZOeBuFZrhSADAIuhrxaZZ86+crjFZrI3AsS1WiPv/ORH03NBXKO1QpABgEWg7mB93Yq5y3fJUAQh3rBGeA0j3rBWCDIAUD36Tpk+v0+GIIO4mzXyni98Kj0XxLWp10KtEGQAoGqGuP1yly/FpAhCvG6tRdCHv/uN9HwQ1+b7vv7Fw1dFfRBkAKBqnjh1IQ0jR7HLl2JSBCFet9Yi6Pj996Tng7g2j33v24evivogyABAtfQ1uD9z25dinjj9QPqGgLg2FQhq5Pyli+n5IK7N+577xeGroj4IMgBQJRdevNTruJjSbV+KSRGEeN2zF84dvirq45av3JaeE+JarHl8jCDIAEB19D24P7PLl2J+4FtfS98YENdizX3rBS2ruHZrbVE1BBkAqIohBvdndvlSTDXHZ28MiGvxjsfPHL4a6oUbd+BaVYtk7RBkAKAq+h7c//SZVw4sp3f5UkyhT6SzNwjEpVt7a4xR17js/BCXrL5CYAkfRBBkAKAa+h7cr1DkQf3ltjWvC4yVwTWqIqjmAcIl3MEM12bNdyqLEGQAoAr6Htx/6uRLb7szmQb4ex9dvhTT0M8e1+atP/7+4bN/OTDmDdeivj5gKRBkAGD29D24X9tquiuZwo1CTpcvxYzo063sDQNxaS6pCCrh+6Fw6eo5rp4ES4EgAwCzxsEiCyT7qtadbXQdIxMhzODSXXKIMWptys4dsWbVHXSJLakEGQCYNerilYWRfdXA/iGhrz0u0aUWQU1o/A93M8OlqLuTLWlMW4QgAwCzZYjB/WOgNwz62+NSXHIRtA2Nf+NLM7FW9dzVBxBL6kpWQpABgFkyxuD+odGtLbk9M9aoWmBUBC3h9qx9oCCnglCv53d/9uPpNUOcUr1m1Yqo56i6Oa/lwweCDADMjnj3sD5sG9w/Bvo0zEUQ3VVwjq61CAKAuiHIAMCsmGpwPwAAANQFQQYAZkVtg/sBAABgGggyADAbah3cDwAAAONDkAGAWXDu2YvVD+4HAACA8SDIAMDkLG1wPwAAAAwPQQYAJoXB/QAAALAPBBkAmJS+B/c/c5bB/dAv3IoYAGCeEGQAYDIY3A9zR98BpC+G1HesEGgAAOYFQQYAJkGD+7Mwsq8M7och+PB3v3HTF0fqi00BAGAeEGQAYHQY3A81cOL0AzeFGKsWGrXUAADAtBBkAGBUGNwPNaCg8s5PfjQNMlLzFHQAAGA6CDIAMCoM7ocaeN/Xv5gGmFJ1PQMAgGkgyADAaDC4H2pA42Cy0NKkbgRw9sK5w7UBAGAsCDIAMAoM7oca0J3J2rqUtcmNAAAAxoUgAwCDw+B+qAGNi1HrShZSuqouaQAAMA4EGQAYlL4H9yvEMLgfhqC81fK+qkXnjsfPHG4VAACGgiADAIPC4H6ogeP335OGkqN47HvfPtw6AAAMAUEGAAaDwf1QAxqov++4mG1yIwAAgOEgyADAIDC4H2pA42K63mr5KKrFBwAA+oUgAwC9w+B+qIVdb7V8FD/wra8d7hUAAPqAIAMAvcLgfqiFo9xqeV/V1Uz7BQCAo0OQAYBe6Xtwv8bZAAzFmC0yUb5zBgDg6BBkAKA3dEexLIzs69NnuEMZDI9aSN792Y+ngWNIuREAAMDRIMgAQC8wuB9qR2NYssAxpOraduL0A4dHAAAAu0CQAYAjM8Tg/suXrh1uHWA8hvg+mS7qyzgBAGA3CDIAcCQY3A9LQ7dkVrevLHAMKTcCAADYDYIMAOyNQgyD+2Gp6Jv5s8AxtNwIAACgGwQZANgbBvfD0pnqRgC3fOW2g5YhAABohiADAHvB4H5YE1PdCOCOx88cHgEAAJQQZABgZxjcD2tkqu+c4UYAAAA5BBkA2AkG98OaUVezqW4EwHfOAADcDEEGADrD4H6A66iVJAscQ6vbQwMAwHUIMgDQGQ3Gz8LIvjK4H2pG41c0jiULHEOq8ToAAECQAYCOqOUkCyP7yuB+WAK6s9j7vv7FNHAMqe6kxnfOAMDaIcgAwFYY3A/QzlQ3AtB33QAArBWCDAC0wuB+gG5M9Z0z3AgAANYKQQYAGmFwP8DuTHEjAI3VOXH6gcMjAABYBwQZAGiEwf0A+6FQkQWOoeU7ZwBgTRBkACCFwf0AR0M3ArjlK7elgWNI1dWMGwEAwBogyADA22BwP0B/aEB+FjiGVjcggN05f/Ha5sSDv9x8+FvPbt7zV2c27/7E6c07//zhzb/5vx9EnFQ9D/V8fN+Xntx84O+e3hw/+fLhs3a9EGQA4CbUavLQ3S+kgWQfGdwPMN2NANQipJYh2M4dP3/tILwQWrA2FWzOvnz58Jm8LggyAPAWDO4HGBZ9mWUWOIaUGwG0oxYYfbqdFYiINann8doCDUEGAN6Cwf0AwzPVd85wI4C3o1YYWmBwaapr5FogyADAAQzuBxgPdTXToPwscAwp3zlzg1t/8FJaBCIuQT2/1wBBBgB6H9yvMTYM7gfYzhTfOSOP33/P4RGsE0IMrsE1hBmCDMDKYXA/wLRMdSMAjddZI7rTU1b0IS7RpXczI8gArBgG9wPMA91Z7H1f/2IaOIZUNwJY03fOaCA0Y2JwTer5vuQbABBkAFYMg/sB5sVUNwLQd92sAe5Ohmv0ls+fPXwFLA+CDMBKYXA/wDzhRgDDoDuUZUUe4hq87xfL7PJNkAFYIUMM7ifEAPQLNwLoF31pYFbgIa5BPf+XCEEGYGUMMbj/1fPr/EZhgKG54/EzB+NYssAxpEv7zhl96SVjY3DtLnGsDEEGYEUwuB+gPnQjgFu+clsaOIZUXc2WciMA3bkpK+wQ1+QSb8dMkAFYEQzuB6iXqW4EoP3WDoP8EZfZvYwgA7ASGNwPUD9TfeeMWoTUMlQrjI9BfHDz7k+cPnxFLAeCDMAKYHA/wLLQl1lmgWNINVbnxOkHDo+gLlTAZYUd4prUOLGlQZABWDiXL11jcD/AAtHdxbLAMbQ13giAgf6I110aBBmABcPgfoBlo+994TtntpMVdIhrdGkQZAAWDIP7AdaBvpk/CxxDW8uNALKCDnGNLg2CDMBC6Xtw/6M/O8e4GIAZM9WNAN739S8eHsF8yQo6xDW6NAgyAAuEwf0A60R3FlOwyALHkOpGAPryzrmSFXSIa3RpEGQAFgaD+wFgqu+cURe3OZIVdIhrdGkQZABmzvk77tj87Pd+7/CvdtRqoi5gMYj89B9Pb374739/c89fHL9pelfPPVvvd0cArBl1NeNGANfJCjrENbo0CDIAM+bZ48c3d77jHZt//o3f6BRmysH9DjFaX/74zz590/xtMrgfoH50u+QscAytbg89F7KCDnGNLg2CDMBMeerWW98KMbYtzJSD+xVifvT7f3jT+rJrmGFwP8By0PgVjWPJAseQ6os750BW0CGu0aVBkAGYIVmIsVmYyQb3ZyHG3v0fP3bTsqUM7gdYHlPdCEBdzdTNbUqygg5xjS4NggzAzFCIycJHNIYZDe7fJcTYpjDD4H6AZTPVjQCm/M6ZrKCr3Q9+89mDc7v/mUvpfMTMpUGQAZgRXUKMPfOhD6WD++96/0fS5TO1bFxXMrgfYPlM9Z0zU90IICvoatdB5pEXL6fzETOXBkEGYCacPXYsDRtt/uz9/+dNIUStLNlybcYww+B+gHWhMSxZ4BhSjdU5cfqBwyMYh6ygm6uf+eG5g3BiXnr92uauJy5u3vvXj920HEEG93FpEGQAZoDuTpaFjC46iOwTYqzWZXA/wDrR3cWywDG0upvaPqhFZ9dWnaygm6PqJmaeunDlIKS8fuX6/2X9jMvOMch846HrH4Z9+afn0/ldFOW5Yn8uDYIMwAy4ev785uTv/m4aMroYb7G8qz/4n/7nzYO3P0qIAVgxuhHAVN85s8uNAHSct3zltp3H22QF3dxUS4xQEf9HX33qbfPUMhOnzTnI6Gc2v4smm4dHd2kQZABmwlHDzD4qxNz/T48wuB8ADtA382eBY2i7BhMfn37uQlbQzU23xtz+6Gvp/FKCDO7j0iDIAMyIMcOMQsxPvnYPg/sB4CamuhGAWlrU4tJE/C4c3UZ6F7KCbm46yGg8TDa/tAwyCg9qtRFq1dHf7/rE6betZ8uxOELH8LHbX0iXF1pe21TYMtqOA0xG2ZLUZBvaR1xW5x674el89bemx+Wkj03zpK9RPK64jMYixW2ri1+8Jtl19rwaXBoEGYCZMUaY+f5v/85BiGFwPwA0MdWNABRYShRwYrjScruQFXRzU+NKTFm4Z6roFiq0Zfm7aGqtUVgyWl7LuTgXWZgSWs7hx/tSIe9QFEOCl+0azOK2hf+WMaBoXyY79vLaOaT4p8KH1tHPchkdq6aLcrsKM762PnfTtRVtDi4NggzADFGYuftd70pDyFH93r/9zYMQ89iD5xkXAwCtTPWdM+WNALIv8tyFrKCbm2rpiMVxWcCXOsgIFd9xWRXdphxvEwv6cvsxTJUtM5GmoBVDQza/iyab53NWwMjGEQmdV2yJ8jEJXd+slapcxneI07J6HCLx3GOoKu8qN1eXBkEGYKZcOnu28dv991Uh5uQXbueb+wGgM+pqNtWNAHR3sqYwtcudy7KCbo6qcI7dmoQK66y7l4t6Fe5lUS8dispQ4RaHpjDi1hodR5xu2kKKA0HbMts02Txfm+x6SIeOGNB8TGXAiXqZLOjEUJhdM7falKFwri4NggzAjDkIM7/VT5hRiLn30//AN/cDwF6olSQLFEPrcTGlu3wXTVbQzVkVxbH7lyi7aDnIqHiP020WKryOiMtGm5YxcVpptk+rEKBjLS3DgYnTrEOYAke2Lc+P+287Jtu2zLZrpv2Ktu3PyaVBkAGYMWo1+df/93tpMNlFhZh7/uL4wffNMLgfAPYlDrif2l1uwZwVdDVYDjyPxfJRgoyCQFw2OlSQiTcIiJTLmjitnLeN2GLTdky2bRmCzLwhyADMmCdOXTgIHz/+s0+nAaWr+sJLvrkfAPpAA++zMStju8udy7KCriYdZtTi4Gm1BZmumrZ5OsZsfmaXY2pbhiAzbwgyADPlhadeOwgfGtOiFpUsoHRV65/6++8zLgYAemOqGwFY3cWsK1lBV5NZMe1p+wQZEZeNekxIDE3SxGmlbYGgq6ZtHkFmf5cGQQZghmgMi8ayKMToVslZONlV3Tjg1fvuO9wDAMDRmeo7Z+Qut2DOCrqa9N3ENLDc0/YJMhrIbpoGzHtsTjkmx8RppW2BoKsmm+eWqfLY2uxyTG3LEGTmDUEGYGZcvnTtIMToFsl9hRhLmAGAIZjqRgBd71yWFXRzUy0gKobLu2apkM4Gse8TZKSDirZZ3u0s3n65nGfitFLvs63r2jZNFrTiHcR0rNn8MuQ0XYdo2zIEmXlDkAGYEer69ejPzm1++o+nD755PwsjR1VhRt9TAwDQJ7qLWBY2hjT78syMrKCbmw4rwnfl0k9TFuj7BhkFpbhd76vtSyWlKadHdXMCo+15u9myTTpoCa8fjyfO1zXTMtLXTz/j9tpCim1bhiAzbwgyADNCg/uHDDFWX7ZJmAGAvtGNAG75ym1p6BjCrncuywq6uakQoDt7uTA26k6VBYt9g4xUmNH0GGgUABQSsu+kkV4mmxdVq0gMRbsW+Dq2GFa0rfLLJrUPXReHF6HroOtXLuvrkLXg2LZrpeshms7d3d3atj8nlwZBBmAmaHC/QswP//3vp+Gjb0/+7u8SZgBgEI5979tp8Ojbrncuywo6xDW6NAgyADNAg/vv/6dHRgsxVmEGAGAIxrgRgFp/upAVdIhrdGkQZAAmRoP7jxpi7vzd/yWd3kXCDAAMyQe+9bU0hPRh1zuXZQUd4hpdGgQZgAnx4H59YWUWMrp41x/88cH3zehnNr+LT9166+ERAQD0z/H770mDSB9qXM42soIOcY0uCXWPf/iT/z0NKV0Uqn8unT178PuUEGSgSvzN/fsGEYcY+5P/9X9Ll2vzzIc+xFgZABgcdTV7zxc+lYaRo9jlzmVZQYe4RpeCvkpCNy5SHXPvR/9zGlS2qRCj9efw1RQEGagOf3N/dJcwo2V1cwCvq1CkFp5T739/unymQgwAwJj0/Z0zXe5clhV0iGt0CcQQY3/yhc+nYaXJk5/8bzetP3WYIchAVVx48dLBl17GEGPvev9HbnpxZf7o9//wphBz6uRLByHGdAkzhBgAmIo+u5opGG0jK+gQ12jtKGwodGR1TdcwoxDzvXf81tvWnzLMEGSgGvzN/TG8lLaFmTLEaFsXX73xXQDmwf/wH9L15dljxw6XAgAYF41p6bOLWZdbMGcFHeIarZnzd9zRGGLstjCj+VmIiWo/Y0OQgSo46Pp18qWbQkuT2Q0AdHeyGGKkWnea+Nnv/d7btkGIAYApUfDIAsm+drlzWVbQIa7RWukSYuzPvvtPaYjR9G0hxr544sThnseBIANV8NiD528KIdu899P/8NaLSt/2X4aYp89c/zbiNmKYefb48cOpAADjo/EsWRjZRwUYtewoGG27c1lW0CGu0RpR7dI1xNgyzNx/712dQ4wds2YiyMDsyQb3d1FhJgsxGtzfFYUZQgwATInuWqbwkYWSNmNg0XiYE6cfONhWl9sum6ygQ1yjtaE7i+0aYqRCi8LLviHGjlU7EWRg1rQN7t/HcnA/AMCcUejQt/BnQcXGwHLse98+uK3yroGliaygQ1yjtaHveNknyEiFF42J+cG/+3fp/G1qv2ONlyHIwGzpMrh/F5sG9wMAzJV4y2UHFn3jf9+BpYmsoENcozWiO4llQWNIFWLGHCdDkIFZssvg/q62De4HAJgbZy+cOxgbo8Ci36cgK+gQ12itqGUkCxxDOHaIEQQZmCW7Du7fZpfB/QAAcDNZQYe4RmtmjDAzRYgRBBmYHfsO7m9yl8H9AABwg6ygm7Pv+sTpzTceemXz1IUb3Yj1u6ZpXrYOYhdrZ+gwM9WNkQgyMCvOPXuRwf0AADMhK+jmqoLKS69fOzzyzeaRFy8faO5/5lK6Ht5QgU98+afn0/lrdgmoxSQLIUdVd0ibCoIMzAYNxGdwPwDAfMgKurl61xPXb3qgFpjY+uJWGoLMdh1k9DObv2aXgkJHFkb29cyHPnS45WkgyMAsYHA/AMD8yAq6ufr6leut7x+7/YV0Pm6XINPskjh77FgaSnZ16hAjCDIwCxjcDwAwP7KCbq46yHzmh+fS+aXqPiWaWmrcwlN2s3J3Nf2ued6vlt9lGfnBbz57sH/PF2pRauvapfOLY4BK4vmoNUrbisur+52OI7ZaOcBkaHkv521q+diNr20ckqZpf15e51ruvwaXxs/+j/89DSddnUOIEQQZmBwG9wMAzJOsoJurKuCFCuU/+upT6TJRF+8KHdl8h5GydSJOFyrQNS1up8syCiTG82Pg0O9lsX/7o68dzn37GCCdt/728b73rx97KyB5Xlxe2/d2dSya57ARjzeGLx2Pj7Fpm+Uxl8t7H7V19VsaD3/yv6cBpasEGYA30eD+LIzsK4P7AQD6Iyvo5qrCiwt3oQJcxXy2rDxqkBGxyN9lGbXEmLL1RfN8Hgounq5zMTGoxfOO56vpCg1lC1Vcvpzna1Kes/V5xeOSMeDEdR3Wymus6QSZ6ehrnMyUg/wNQQYmg8H9AADzJivo5qyKdBfURgVzLPztUYNM03py2zJuPWoKQg4AChyeprE/IgsAbqlpCiCl2m+2vK9Jth1dQ6HrW86TDmfxnL29pvOsyaXQ92D/qW67bAgyMAkM7gcAmD9ZQVeDCgIOE6ap9aEpbHj9sqj3dBXucXp02zKmbRvGyzjc9BFkfO7l8k3TpVqOhMKVzq80Bkiv4/AjtM3axsVEl4BCh764MgskR3HKMEOQgUnoe3D/M2ev//MFAID+yAq6mlQIiAV2DA4u2lWEx3WspouyqPf0thCybRnTVtgbb8NdyxQkYguTfndXsazlSevrHHRM1uNUynPzNSmnx3nbKFtsHMCMWmfart1crZ2hQoyd4lv9BUEGRofB/QAAdZAVdLWpsJANMHdhrsI+Lm8dRsqi3tPHDjLS2xX6Pf5ddt/StuP8jPLcfE3K6XGetlnO26ZCmNb34yBq625WM6/ed9+gIcaev+OOwz2OB0EGRoXB/QAA9ZAVdDWaFeEKCOW0qENAWdR7eh9Bpmm+QohxK0s8Xo+xEWoByW7X7HEwmq/xNTE0+XqU59Y0Pc5rul5ddRc1kbUgzdVaGSvEWO1vTAgyMBoM7gcAqIusoKvRbHC9g0EcUG9jK05Z1PcRZNzdrbz7l3V3LB2Dp3kcTNt+oz6G7AtCmwJL03Sp0GHa7gbXxS7XcG7WyNghRmp/Y4YZggyMQt+D+xViGNwPADAsWUE3R1UQq+gvB/TL2AIQC+fY6hFbNFSkx3E1QwSZOG6kPGat4zEv8bgcZLp2yfIxlMtrf95+U5Apx7lYXxf9LMOMu4/F4KQAqWmxNUjLtY3pmau1cens2c3d73pXGja2+b13/Nbm5Cf/28HPbP42xwwzBBkYBQb3AwDUR1bQzdHYWiBUxMs4JiMLOQ4HQsW5i38V2m7FGSLISHf9EjpOrRMDVBlAYgCIaJrW1bnEcOHbNQtv39fD+ynPTeubuI7n6zrHY/A1i8cdw1c8Ry0nTRyvVIO1ocH9WcjYpsLLT//+a5uHrry+uf/eu9Jlunj22LHDIxkWggwMDoP7AQDqJCvo5qpCgwrnWFSr6Na0tkChYt4FvpdXQe+WnFiYSxfjba0JXZaRClexuBcq8LPuYGrV8Lk5QMjyfMswE+frd52Pg18ZZLyOr4col9FxKDTF7Wp5HXcZFrWs1i+PIdvv3K0R3UksCxlNxhBjf/bdf0qXbXPML8okyMCgMLgfAKBesoIOp9GBpwxWMoacGkNCDdaK7iSWhY1ShZiffOHzN4WYXcOMupSN/Z0yBBkYDAb3AwDUTVbQ4TSabJ5UgBEEmWGsmS5hpinE2G1hRiFmiu+SIcjAIDC4HwCgnVfP39ylaI5kBR1Oo8nG+sQbBGTz8ejWjgbfZwFE3vvR/5yGl1KFnWz9qUKMIMjAIPQ9uF/jbAAAlsTTZ15563/coz87d/B/UzcyUZdchZzLl26MU5iKrKDDaYw3JlBo8RiZOJ6lvEEA9ucSyG7H/OP/60/S0NKk7mYW158yxAiCDPSO3ohjCDmqerMHAFgiCjDZ/73oQ3e/cBBydKMT/X8dM+RkBR1Op1pbNKjerS9Cv2cD7bFfl0IMM7uGGHvmQx96K8RM8W3+EYIM9AqD+wEAuqP/bwoq2f+/LpYhx605fZEVdIhrdElcPX9+8/An/3saUroodHtlfVfN1BBkoDcY3A8AsDsKHtn/wKPax7jCrKBDXKNL4/lrl9OQ0sU5QZCBXmBwPwDA/vTdmi37aJnJCjrENbo0CDIAhyjEMLgfAOBo9P1/tI9xNFlBh7hGlwZBBuAQBvcDABwdBY++uudqO32QFXSIa3RpEGQA3oTB/QAAR0fjAftskdH/0j54558/nBZ1iGtzaRBkYPUMMbh/Dt+bAAAwJhrL0uf/Uqm7mPUBQQbxukuDIAOrhsH9AABHQ/9H45di9mlfXXTf81dn0qIOcU3qdbA0CDKwWhjcDwBwNIb4PxpVt98++PC3nk0LO8Q1+b4vPXn4ilgOBBlYLX1/gsjgfgBYE+pKdpQvwexiXy3cJx78ZVrYIa7J4ydfPnxFLAeCDKwStZxkb5r7yuB+AFgL7krW93iYzL7GG56/eC0t7BDX5NmXj/6dTHODIAOrg8H9AAD7MXRXstI++cDfPZ0Wd4hrcIndygRBBlaF3oQZ3A8AsDtjdCWL9nXrZXPfLy6lBR7iGrzj58scw0uQgdUwxCeJDO4HgDWg/3VjdCWLPvqzc4d77w99Kp0VeYhLdqmtMYIgA6uBwf0AALsxdley6BD/Yxkrg2tT36Gk1silQpCBVcDgfgCA3Ri7K1lpX7deLuEOZrgmb/1Bv1005wZBBhYPg/sBAHZjiq5kpUOOPzz2nefTog9xSer7k5YOQQYWjVpN+vxEkcH9ALBk9D/ziVMX0v9/Yzv0B0aEGVyyawgxgiADi4XB/QAA3VHr9dhdyTSgP5sux0BfEJgVgYi1qjExS+9OFiHIwGJhcD8AQDem6Eqm/6n6wCn7X61ANRYaCM13zOASvOXzZxc9sD+DIAOLhMH9AADbmaIrmQJTOZC//H6vIW69vA19zwa3Z8baVAuMAsxSvydmGwQZWBx9D+7XJ4MM7geApaH/lX1+QXAXtb/s/6kCVfy/rXA1FbpFs7rmKNS856/OpMUj4lQquOh5qeenxnmtrQWmhCADi0JvhgzuBwBoRy0iU3Ula0K3e/ayz5ylKy8AbIcgA4tBb5AM7gcAaEb/J/seP7jNrCtZEz42PkACgC4QZGAx9P3mrO0BACyFOXUla0PjY3SsAADbIMjAImBwPwBAM+q2NbeuZE1oHcYlAkAXCDJQPQzuBwDImaorGd1yAWAMCDJQNXqTZnA/AMDb0QcyfY8b3KZas+kWBgBjQZCBalGIYXA/AMDbmaIrmf4f79OVDABgXwgyUC0M7gcAuBl3JRszxNCVDACmgiADVcLgfgCAmxmilXqb+t+p1h8AfbN89gWOu7rWb6iH/SDIQHUMMbifEAMANaMw0ed4wS7qNsn87wRDkIEpIMhAVehNs+/B/XyaCAA1o2/BpysZTA1BBqaAIAPVMES3Cd6MAaBWpuhKphDDnR0hgyADU0CQgWpgcD8AwHXoSgZzgyADU0CQgSroe3A/b8gAUCv6fzhFVzL+Z0IbBBmYAoIMzB4G9wMA0JUM5g1BBqaAIAOzRt9MzeB+AFg7+kBniq5k+h8M0AWCDEwBQQZmyxCfPjK4HwBqY+yuZFJ3QqPlGnaBIANTQJCB2cLgfgBYMwoST5y6kP4/G0q6ksG+EGRgCggyMEsY3A8Aa4auZFAbBBmYAoIMzA4G9wPAmpmiK5larPk/CUeBIANTQJCBWaFPA/t8A9e2GNwPADWgINF3l9pt6n/kuWcvHh4BwP4QZGAKCDIwG/Qmrq4N2ZvtvvIGDQA1oJboUydfSv+PDaX2R1cy6AuCDEwBQQZmA4P7AWCNaHA9XcmgdggyMAUEGZgFDO4HgLVBVzJYEgQZmAKCDEyOxrD0+Wkkg/sBYO6oK1nfXWm3qa5k2i/AEBBkYAoIMjApDO4HgLXR94c3XdT30fABDxyFY995Pg0eY/vhbz17eEQABBmYEL2pMrgfANbCVF3J1HUXoA+mDjOEGCghyMBkMLgfANaCQsxjD55P/3cNJV3JYAimCjPaL0AJQQYmgcH9ALAWpuhKptDE/0QYilt/8FIaNoaSEANNEGRgdBjcDwBrwF3JxgwxdCWDsTjx4C/T0NG3Ck0ATRBkYFQY3A8Aa2CqrmT6ThqAsejrTmVNHj/58uGeAHIIMjAaemNncD8AzJ2r589vnrr11sO/dkcfrpz88F9sfvK1e9L/W0NI91qYiiHCzDv//GFCDHSCIAOjodt/Zm/A+8rgfgDom1fvu29z97vetfnn3/iNzdljxw6ndueZs69s7vlP/+Vg/e/929/cnPzC7en/r76kKxnMgT7DDCEGdoEgA6PQ9+B+BrICQN/EEGOfPX78cG477kp293/82E3rDxlmFGLoSgZzoY8wQ4iBXSHIwOAwuB8A5o5CzJ3veMdNIcRuCzP6H6f/SwoxCi7l+kOEGXUl05hDgDlxlDCjEKMbCADsAkEGBkddLbI34n1kcD8A9M35O+5oDDG2KcyotVn/l378Z59OQ0z03k//Q/p/bRfdlYwPc2Cu7BNmCDGwLwQZGIW+upbRjQIA+qRLiLFa1ihIeNyfAsq2EGPv+Yvjb/u/1lW6kkEt3PeLS2lgySTEwFEgyMBo6Bumj9LFjMH9ANAnamXpGmKswoz+l6krmf4vqctY1xBj1XpT/n/bJl3JoDa6hBlCDBwVggyMij7F1HcdZG/UbTK4HwD6RLdX3jXEyB/85m9t7jv+/x38X9onxNhdwow+xOH/H9SIwozCCiEGhoIgA5Owy62YGdwPAH1z6ezZvYKMVHhREPn+b/9OOn+bWr/LeBm6ksESyMIMIQb6giADk9HlJgB6I1c3DgCAvtGdyrKgMaQKMV3GydCVDJbE2ZcvvxVm9FPhBqAPCDIwKdtuzcynkQAwJBrzkgWOIewaYuhKBktEYebdnzhNiIFeIcjA5OhTx2zcDIP7AWAMxggzXUKMPtQ59+zFw6MCAIBtEGRgNmhAv9/QNYaGTyQBYCyGDjPbBvfrwxy6kgEA7AZBBmaFxs3oDZ0QAwBj8+KJE2kIOar6xv8svFi6kgEA7AdBBgAAVo+ChAKFQkcWRvb1rj/44zS8SLqSAQAcDYIMAACsGt0ZMXZtvev9H0lDya62hRi1PHNHRgCAo0GQAQCA1dJ058Qf/f4fpuGkq20hhjGAdXL+4rXNHT9/bXPsO89v3velJzfv+aszB3fh0u2Ey+9JQRxDPe/0HJR6Tuq5qefomiDIjMil169uHn/4wub7J57Y/P1nHt78zbH7Np/703s3n/qTuw/8fz5wJyLOWL1O9ZqVeg3rtazXNNSHu5JlQUMetYtZFmQUmF54al1FRu04vKhIJKxgTeo5e/zky4fP5OVCkBkYhxcVPYQVxOWq1/j9P3ju8JUPc0YhJnYlK+1rnEwc5E9XsrpQgNGn2/qkOysSEWvyw9969vCZvTwIMgOhAKNPa/XJbVb0IOJy/e7fPnb4nwDmhrqSPXT3CzcFl2jfg/1122V9Sz9dyepBLTAEGFyaalFcYgsNQWYA1AJDgEFct2qBpYVmPrgrWTYexip06Isrs0ByFJ89fvzwKGDOqBVG3XGyIhBxKWpsl57rS4Eg0yNqhVH3kqyoQcR1qrFw+t8A07GtK5kcKsRYfUcNzJf7fnGJVhhcjWqdWcpNAQgyPfHcE6/RCoOIqWqd4aYA07CtK5k8+YXbBw0x9vwddxweFcwJFXQM5Mc1euLBXx6+CuqFINMDKlAYyI+I2zz9k5cO/2vAGOgOYW1dyeRYIca+et99h0cHc0CFHCEG12ztYYYgc0RUmBBiELGrhJnh6dKVTI4dYuSd73gHYWYmqDsZIQbxwaq7mRFkjoC6kxFiEHFX6WY2HF26ksmffO2ezfd/+3fSsLFNhR/d3WzfEESYmR4NdmZMDOJ1FejPvnz58NVRFwSZPdHg3TmOiRHnnruYzsPx1WMhsnm4XvUByIUXLx08N6A/unQlsxrcn4WMbSq83PMXxw+2oRadbJkunj127PCoYQq4Oxnizd7y+bOHr466IMjsSR93J/vcR+7dPHLfuc0rIQVfufzG5vknX9t856s/T9fZpphrkNE56dx0jkbH+sCPXji4Ftk6tUuQwSa/8l8fOHhuwNFRV7InTl14W1jZpgJJFjKajCFGgUlhVAP4s2XbfOrWWw+PHKZAYwKyQg5x7db4PTMEmT1QH/esMNnFE7edeltBLy++euM2rfrOg3I9hQGh4r+cJ8Ucg4zOxegcdYwxwCnQZesNofata5/N61vtS2TzEPmemaOjb8vv0pWsyXs//Q9p2ChViFErjtbRF1xevnTjexi6hhl1KeM7ZaaFLmWIzaqLWW3fMUOQ2ZE+upR9+S//5a0QowK/bI2485tPvjW/LPAdZJoKfzG3IKNjFTqnsqXJrVInb3/mpulDOma4IMhgm+pixnfM7M8uXcna7BJmHGL0Pzv7lv5tYUYhhu+SmZ5j33k+LeAQ8bp6jdQEQWZHvn/iibQg2UW3TqibVTZfKsyIsuWgtiCj0GbKEDOVBBmck/qfAruxb1eyNtvGu9z1/o+81ZWsDbW2ZOsTYuYBrTGI21WrTE0QZHagrwH+bm1RkZ/Nt+5mplCjv5uI3cyEime1dCgweV+apu5sXi6qgKFQZbTfLCi5ZUXbOfvQ+YPftf2m7Uodm2gLbZlqoYldz3RM2mc2lkZo+7qesQtbec4+/pIYFruco/fjayu0fz9O0SzIuBXKj6/QcuXzQduX2TnjclSrDHRHXclOnXwpDSNHNbsd811/8McH+4tdydrQ+Je4PiFmPjA2BrGbNY2VIcjsQB9jY9yioiI9mx91Ie1QoZ8u1FX46m8Zi2yhAtnGdbKiWIHB87Q/Le8CW+vFZTVPOGDop5Zta2lxIb9L1zEfr49fxn2W5yC0rM6h7Zx1nPH8vO0YBPW3aDpHXWttU2gf3penleeZBRmHRj+Guu5xH9LPE9F2fXEZMlamG+eevdhLV7I2Y5hRiNHrO+tK1saZD33orRDDt/nPB+5UhtjNmu5gRpDZgT7uVOYCVUVsNj/qolo/Pc3rx2lRozffON2FfSza1QLg4r8MBy7Am1o0ugYTB4Kuxbi71GWBxQGgPHfT5ZxlFi7stnP0+ZTzdJ1EDE4y25fQduK0TG2r3B4uU+5g1o6ChF7PWfAYwp/+4+nNPf/pvxwEp33R7ZUvna3zdqZLRN3K+PJLxO7WMuifINMRdSvr48svXSgPGWQUTMrpXi8W++72VRb6UoW6iPvx8agFIS7bpukaZBxWsm5aCl5CxX2cLrqes+wSZLJzdFhpCiFZcGoKMll4xHXLoP+cIbuSNblLVzKoA7qVIe5mLd3LCDId6aNbmXSLw5BBJtu214vzHBpUgGt7URflcT/6XXQNJVIF+y7rKKSIbJ7M5ouu5yz1t4jTbNs5OvjFa7JtfrYvdxnUtVFgJNCgpHvZ29G39A/dlaxU//t27UoG8+fWH7yUFmuImFvL3csIMh2559u/SIuPXXVx3aVrUfYJv9dvKqZF16LeRXYbcd/7BBnvI2thyTTZPJkFA9H1nGW2Ddt2jp7XdO2zx6ZpX1rGoUw/m7aJ65G7l91g7K5kUoFJt3OGZcL4GMTd1GumBggyHeljfIzUp+9m2yfxbs2I41SyYjkquhb1bpHJivZM7XOX5aVbH1SUZPNLTTZP+prEaaLrOcumcCHbztHzmq591h2vbV9S6/icmraL61D/Y2Bz0KXrsQfPp2FjKNWVTF3YYLm8m9suI+5kLbdhJsh05HM93HbZurhtK1zdTUlFbpzu4rxpXdG1qNc2RGx1adPL7xJkvF8RA1mTvjbZsh4jU14T0fWcpfcRp9m2c/T2FADLedItaPFGAG37sj6v7BxwPXIb5mm6kik00ZVs+RBkEHeTIDMuVQWZWNxnIcIhRpQFtddtauEQXYv6GAyy1qFy/MY+QUa65Uf7KdfV9rUfXwefe3YO3k55zUTXc5b6W+j843S57RzdHaycH+9aFqd7X3FaaVOQ0bbktpY7XIZrDjLuSjZmiKEr2brgjmWIu1sDBJmO9HHHsqi7IQkVqypipQtlET/ZtypqvYy6bcm4nCgLYtlU1Lvrl7apQkKFvKYpdIjYMrJvkNEx+7bFQtv2+ZoYzLysltM+pY8naw0Ru5yzz0P70e9xm9vO0TdrEL5e+il0Dcv1fI7+W/N1Ltm1juOIfOyi6Vhwea4RhZgpupKp9QfWQ1akIWK7NUCQ6UhWdBxVFagqaF3IChe5WWuBVcHrdVQ8xwJYZEW9WwyyIKD1Y6hwqCkL6G1F/jYVuMr96G+1sMRWB/2ufcXrotBRtsRYscs5a/sOHyKu2+UcNc+tQ8LXK+sO5/P133pcNU3rGP0dH0OrZWS8Nrhs14bCxEN3v5CGjaF89Gfn6Eq2QrIiDRHbrQGCTEeyogMRsU/XxDNn6UoG45EVaYjYbg0QZDqSFR2IiH26BqboSqYQc+HFS4dHAGskK9IQsd0aIMh0JCs6EBH7dOnQlQymIivSELHdGiDIdCQrOhAR+3TJqFvXFF3JCDEgsiINEdutAYJMR7KiAxGxT5cIXclgDmRFGiK2WwMEmY5kRQciYp8uDX1b/hRdyS5funZ4BADXyYo0RGy3BggyHcmKDkTEPl0SY3clk7oTGl3JICMr0hCx3RogyHQkKzoQEft0CShIPHHqQho0hpKuZLCNrEhDxHZrgCDTkazoQETs09qhKxnMlaxIQ8R2a4Ag05Gs6EBE7NOamaIr2dNn6EoG3ciKNERstwYIMh3Jig5ExD6tEQUJBYosaAylAtO5Zy8eHgHAdrIiDRHbrQGCTEeyogMRsU9rQ13JTp18KQ0bQ6n90ZUMdiUr0hCx3RogyHQkKzr29YEfvXC41RtcufzG5vknX9t856s/T9eZ0nPPXf/kM5uHiP1ZExpcT1cyqIWsSEPEdmuAINORrOjY10fuO3ewTb0p63ep383J259J1+tD7/vEbafS+Zm1BxmhkJjNq0U9Bgq72TxcjjVAVzKokaxIQ8R2a4Ag05Gs6NhXh4my9UV/CxWsn/vIvTfN68umfbe5hCCjc8jm1SKtYutw7qgrme4SloWNoVRXMu0X4ChkRRoitlsDBJmOZEXHvraFiVdevtw4rw8JMnVKkFmHc+bV85dH70qm76OhKxn0QVakIWK7NUCQ6UhWdOxrW5hwwRrn6Xd1jTIXX716sI24nlQrjqZrvtH2vvyX/3KwjSa2dTPLimhvT+N99LuXEep2EluURFO3KI8XiueTnYfOPztOdcNz+BP6/c5vPnkwLx5TJHYz0zJuATv70PmD34W24/3pp7el+VrO60d3OW7P02MTuxVqP3F5bS8jXk9vw8eun/EcsR7nyFRdyXQ7Z4C+yIo0RGy3BggyHcmKjn11cRrDinVR7mJWRbNweImFcllQq3gVKoa1nOZrWe1Hxa6muSBXYeLtxdCR6XXiNAcZH6/36WX108t6GQeMaHm+OpZymy70VaDHIt8hKF4braufmq+Qo9/LZeJx+Hi1nkNK3J/X97E0XftdjlsKbUvzfGxxeT8mus5xv/pd6ty9X8/zY6qf2kbcH9bh3FCIeezB82nYGEq6ksEQZEUaIrZbAwSZjmRFx76q2BRlkFHhLVSYeprCiYtWq+JVhWpZrAoV03FaadO+23SxH6c5yIjy5gQurF28+7xUYMflFK5EPGYHt/Kcvb/Y0uDjctHfpNCy2TxvQ8cQt+NQIeJx+9oLT5O7HLc05TXxfsvtZI+B9PbLYIV1Oiem6Eqm0ERXMhiCrEhDxHZrgCDTkazo2FeHCRWt+l26UFWRXH56n5kVtkIhoq2w7zvIlIW49D4ccHQ8ogxeKtZFLNqzgGYdkPy3j2vb9RJaNpvnbShUxek+vywYep14DXc5biliYLVN1zV7DKSXpyvZMpwD7ko2ZoihKxkMTVakIWK7NUCQ6UhWdOyrC/2IiloVD2VBbVWsq+B38MmKY7cKaJ5CRBZo+g4y2l6cLr2POM/d3mK3LnfF8jnrHIWOX+uWluesbQkFCM1vunZiW5App/v8svW8jq/hrsctRbbtpv02Haf0ddQyWfc9rMepmaormb6TBmBIsiINEdutAYJMR7KiY19V3IouYUJFsgvhjHJ5bVuFvXCBX84XXfZtsyLaBXe5fel9xHll97KsW5m32UbZ6qHCPV4fbb8McCILDbIpIDQFCul1fA33OW6Rbbtpv03HKXW+Om+j60GgqdMpUVeyh+5+IQ0bQ6lbOdOVDMYgK9JwN196/drBtfyjrz6Vzl+6dz1x/X34yz89n85fojVAkOlIVnTs6y5hwkW6WmNigd5W2EoFB68bA8Uu+7bZvlxwx21b7yPO07GrmHdB725l7n4m3bKRFfjb1PH4OMv1s2k2OzfZFCik1/E13Oe4m5Zv2q/3GaeV6hrrmusaCx8f1uNUPHOWrmSwbLIiDXfTfPCbz940XX9/46FXFh9wHnnxeu8HnWs2f4nWAEGmI1nRsa9dw0SXYrqcHnWrR1y/676j2b58bDGsWO+jnOdWA7UWuDtU1npStl7sosNbnCayayibrmOXax+vodjluEW27ab9Nh1nplu/sscG5+3YTNGVTCGGrmQwNlmRtgYVLowK8WyZrpoyyLjAP+r25y5BZp4QZDqSFR372jVM+JP+csC51vOn7nF6aVuQiS0h28yKaBfcWbHsfZTzPKbF28sGqDvsxBsAWJ3Pti5TTUFG0+M02xQQmgKF9Drx8dv1uEW27ab9ep/aVpyemQUZt4Bxd7N5OyZTdSW7fOl69xSAMcmKtDXo7lDmvX/9WLpcF00ZZLyP2x997abpQytev/JGOm8ICTLzhCDTkazo2FcX+rEQbjIO5NZ6LpjLIKNtqVjXfC2ngtVFfSyiXShrfW9PgcnzM11Ex2neTiyWrc8vm+fjFlmYUqHuZXTu2ob0zQJi+NFy+tvL+FqVxbqP38vG+dm5yaZAIb1OfPx2OW4psm037VfbEt62t6ffNU3npN/1eOo4ZAw9Xr88DpyXY6FuXVN0JWM8DExFVqStQRX60mHjKEW4KYPMVJps3hASZOYJQaYjWdGxry4qYyHcpAdyu0hWONGn696Gl1PRquLXywn9nbVgaN24vW2f8rtwj9Pagow//c9aJ1RwC+2/7FZmfc4OYkLFurYX11FRXi6THY+Cms9BxGWyc5NuDcsKf00TZQDsetxSaN9xmmzar7dtvK4e33huuq5atzw2X/ddWuJwfIdGQeKJUxfSsDGUdCWDOZAVaUv3Mz+8XicoxHzs9uvvyxqwny3bRUOQIcjMCYJMR7KiA7EGFYIUcBSqsvk4H4dE35ZPVzJYK1mRtnTvf+b6BwgKMfpbLTOiLYi86xOnDwp136FM66jLmKabcn0tL8q7eXl/2U0A1MVNaJlyno7XoUE8deHKW+cg21B4i9vSsfo6CO1Pf7ddA21D+zRaXsdLkJknBJmOZEUHYg26haxLCyBO61CM3ZVMqgWRrmQwF7IibclmQcHdy/QzLmsVVmIBr8Ldf8fpTUGmLPBNFho0zcTpbjkS2mcMNA5Emhan+28Z9+UWKeFtOaCJMvRIXyMR96/r6HUJMvOCINORrOhArEF1dSvHDeE87Ru6kgFcJyvSlqxaR0QMLQ4JWSuIdCBRwR5vCqD1tI4ZMsg4LMR1dCxZK4qJ06y3r+2VLUIOODonhbdyHRH3pfVjkCPIzAuCTEeyogMRsU/7RF3J9K35WdgYSrqSwVzJirQl68I7dsmSDiRZa0TbvHgb5zJQ9BlkTAxSTZpsXtmtrtQtLfHYvE7WYqXAQ4vMPCHIdCQrOhAR+7Qvzj17ka5kAIGsSFuqDh1Zy4vGuwgV7XF6DCpxetQMGWQcJhTEsvWiJpvnUObuYaWeH4/Z05r2q/UEQWZeEGQ6khUdiIh9elQUJBQosqAxlApMCk4AcyYr0paqw0rWshADS2z1aAoWUVMW+n0GmXKcjlpB1E0udgGzppwe520jttgYgswNa4Ag05Gs6EBE7NOjMEVXMu2PrmRQA1mRtlS3tUaYeJexuQQZq4Dh1hmhc1IIi8uYOK2c1xRKMk3TOr52BJl5QZDpSFZ0ICL26b5ocD1dyQCayYq0JaoA0BW1dni9bcFCmrLQbwoyLvyzYNBlf1KtRm6hKbvDmTitnNcUSjJN0zoEmXlCkOlIVnQgIvbprtCVDKAbWZG2RNsGrNvYvSy2cphsgHxcZ9cgU36/jPRd1UQ5r9ShR9uL002cZrtch1Ifr7rmlfMY7D9fCDIdyYoORMQ+3QV1JXvswfNp2BhKdSXTfgFqIyvSlqaKbVN2wyp1K0cs9PW70Lw4JkXBxt3VRNcg47E6CgBxezHECE/XMWvf5fabblBgsuAVW6ayIKX5ZciJt2WO18/HZQgy84Ig05Gs6EBE7NNdUHeyLGwMpb6Phq5kUCtZkbY0HRBil7EmvayKdk9TNy4HFv1UC4VbIVTIu5jvGmTatufQJOLyRsvF5UUZzuI2vGy8dXSc7/3LeExxezIGFi3rv7WsW3kIMvOCINORrOhAROzTXRmjRUZdyV546vonogC1khVpS1OFt8haIErVQuKCPgYThQkFAM9TOHDh7kK+DBRNQUZqWa8nFAwcNrSPMkzoWLS896+f+rvcp9Q5xLCiY9Xxx2XU8hK3J3Sd1MpTLiu1TZ2HA5TW0z403eGvy/VdijVAkOlIVnQgYnfFuecupvMyv/PVn2++/Jf/ks5bqvsw5J3K6EoGSyEr0rAf24IM1m0NEGQ6khUdR/VzH7l388h95zavvHz9UxRx8dWrB4N3VcRl6+B+Xrl849MYo2nPP/na5uTtz6TrYL+KrkHmgR9d79+8S/BZgvug2x9nIeSo6lv66UoGSyEr0rAf3SpCkFmeNUCQ6UhWdBzFE7eduqm4VsEm47RsPdxPoZCo4GgVYny9de0VLLN1sR99nbN5pXd+88mD5RXqs/lLdV/6HC9DVzJYIlmRhkdXXa7cDSsbdI91WwMEmY5kRce+qmB2Aa1CrSyg1RqjIjtOw6MpsiJa117Xumk+9ifXeLtHQYPxs2Cyiw/d/cJBKAJYGlmRhvurMSsaa+IQo5/Zcli3NUCQ6UhWdOyrwougqBvPtuutMKPWGkE3s+EUPOfbPSpHGS9DVzJYMlmRhvurIGMUaLLB+Fi/NUCQ6UhWdOyrW2PUvSybn6lBzwpAseuZWhLUBadcVt2mhLavsQYu0vXThbqKdwcqoW1lLUNCP7Utt1wI/Z4df3mc+pm1OsVj1DH5GLW85sVlhebHadZjKcp1SkVbEa1jEDqvbF4cx6Tfs+uu7ev4da5nHzr/1jXQ8r5W+qnlhOZruXI7UtvQOfm6CG1H55stXz4+JR40H6+79i10HD6+bL9Nj7XQslonPuZN10fo3MvlNa3cvp972eOqbZePR3lduj5mc/OoKIjs+g3/7kpGiIElkxVpiNhuDRBkOpIVHfvoAq2pMM9UkeeiTwWgirtYCDqcWBerLuS0bCysNV+/6xj0uwtrLR+342PV+kLLxeW1DRWlXl6Foqdrf23bzo5R01xA63cv621khajXzwrtqNB2snlS52HidJ+7r1W8juV1j+fqa+D19bfPWcvFc9VycTs6Fp+Xl9UyXl7bjMsrpGj78bp7Wa/vx8nH4O3rp5bVY+3tCD8e8fgdhqxQyNE8Xx8vL7TNcnktZ8vtx+dSU5Dx8Xt96XPx/hxuy2X0M25rjvbBq+cvp4ElUyGGrmSwBrIiDRHbrQGCTEeyomMfXey3FdWlLtTKwlnFuyiLQBVsnh4LfK1vtM24jovwWHy6mBTlJ97+9D8W4VpGhWncrvS247E0HaOKZU/3NB+3tu1pcVmdS5yeKbZdc+O//ViVrTQ6Px1fPEbp8yyvbSzu4zl4O8LTpK6pKM9Xy6s4F/Fx8vLx+eFro+U9Tfq6i+z5pGMvA4vXKZ8DpnzMHSTKx8WU5+XrE7efBZl4TnF/UufiaX4cymVqsC90TbPgElVXMt3xDGANZEUaIrZbAwSZjmRFxz66KNxWVFuHlaZiPSsCvY+yUJWmLPK8TiwcXUyWhbzcdlxRbzsW323H6ELUBbWOVZTBwQVzWWBnil2DjMNaDFrW1z2eU3nc1tcxu1ZeJ27H4SYrxHW9RHxMvI24XNN0X/cYQLfp44/PDSmyUCGzcxBlsJLefgw42T597Nseb5939rjN3T5pGy/zzNlX6EoGqyIr0hCx3RogyHQkKzr20YWoiq1sfqmL9bKIbJvvgi8Wx9aU071O3E5TAWtNOV2FvM5T68msWPf+smPMlndwiN3LFAxEGRwyxbZrbvy3i3GfRzQ7Rk/z39bXMdt/uR0HxKZjzeY37TebrmMX2XW3Ch+6zj5XX3v9HpcTTcfpfcf9iGz57Pp4Wtxnts1Mt6Tp8dP6XZ4fc7FPsvEydCWDtZIVaYjYbg0QZDqSFR376AKtS0uGVCEmyiLSZgWf18kKPlNOz/aTbTtq/LcKYBebGfF42o4xK1hdnPpTexWnout1FFkRbb292OrThfipv4/bf1tfx2z/5bm2LWvL+e5aFq9X0/Vpu+7S28oonwei6TjL85IiWz47Z0+L+8y22aSeL+6GJ/S8yVqO5mbfxPEydCWDNZMVaYjYbg0QZDqSFR37qGLKdPmk2IVnWURat/DE+V4nK/hMOT3bT1ZMRo3/9if3+hnPLTuetmNsKlhjdyW3RGVd0zJFVkRbby922fL+4nJt+rjL6b6O2f7Lc21bVjqgxPmxBUJBRNfWRbzmxfXbrrufSwo/MaD5mMrngWg6zvK8pMiWz84522e2zW1qWa+X7XtuDoFej5KuZLBmsiINEdutAYJMR7KiY19dWGVjT0pd0DUt6/AQC/q2YtWU071OLBy9b+0jLiuzLk4mLiez42k7xqaCNQ5qd7eyrp+yi6ZCVttwaIn79HHEor5NL19O93XM9p+dq4gtQ1GHjfiY6FoouGiaz0PTyhAj97nuPv743JAiG/MisxAosmuQXZ9snz72bWNkMh3ssnlzEgCGISvSELHdGiDIdCQrOvbVIUCoOC+Lcc1XUerpWZHt5URZ9LYVq6ac7nVi4ehiUpQtHx4IH4tKH2c8H7UgOHTE42k7xqaC2ufr7XUJglY0FdFN23NoyPajcyyviY87TpO+jtn+s3P1tY2PhdQ+XZDHcCV0DuXzKLPtunu/MQBpmw7L5fGYMuhmrVtSND0G5TxPi/v0469rUJ6rjrmthTMLMnq+yi4to2MJAMOQFWmIQ/rev37s4Ln3+pU30vk1WAMEmY5kRcdRdJFsVMRJF1zCBZa7DgkVjSruXFyqECuL0rZi1ZTTvU4sHF1MutDX8Wm+fgpNj9twi4nOQcvpbxeLIh5P2zF6+9m8eH3KINGm0HH4Oksfl1DRnQUBn7v2625buvY+r7isjztOk76Oml/O8zrxXPW4+9g0P15LUbZIODiUeN14Xm3X3c8z7cfnqvP2fvV3XF54vh/z+LwsA4LIrkF2fTyt3Ke37/1JP0Y+J+1bj2c5X+cUt2ViKJxaABiGrEhDHNIPfvPZw2dfvc+/GiDIdCQrOo6qCj0VZirKjIouTSsLTf3tT8yFijUtlxVhKt5EuQ2p9WQ53cVwLJJjManfXRB631nhr2V9PlpOxaML5Hg8bcfo88zOzcepbWf7b1LLl+g4ta/sGKKxGBZaT+dVFuoOJXGadEuC9lXOazpXPzficWvZ2FpivV8tr2OVOj4/DjEgtF13qe3Hc9U+dSxC68Zlhbat+T4Pod+zx07EY7HZ9dHxiXKfUtPia0bbjM9bbSfO1/mU29FzR2i5OH1qAWAYsiKtL9/1idObbzz0yuapC1cO97Y5+F3TNC9bB6fxkRevv7995ofn0vl9SpAZB4JMR7KiY+m2FZNT6VCkoj2bvza3XQ8X9Nm8oyqyYFKDek6LLBhOKQAMQ1ak9aGCykuv37gboAplF8vi/mcupevhDRX4xJd/ej6d36d+bLTPbH6fEmTGgSDTkazoWLpzDDL+5H9uBehUuoWq7DYl1eqgFh1ZzutDUWuQ0TWZ47EDwDBkRVof3vXE9RZxtcDE1he30hBktusgM0a4IMjsZg0QZDqSFR1Ld25Bxl2c5tYdaEp1Tdz9TCFPj5WM3dJ2GUu0i6LGIKProWuja5fNn1IAGIasSOtDDeQWH7v9hXQ+bpcgM19rgCDTkazoWLpzCzJufZhTC9EcVEGu4BLHhahQV7DRY5it04ei1haZuQoAw5AVaX3oINN1zIVaaERTNyp3U/ujrz5103QX4Ppd63q/ahHyMkLLqTXo9kdvjFssj00Fto7D2xBqUWrr2qVtxDFAJbHlSfvXtuLyOi8da2y1coDJ0PJeztvU8rEb3z7jkJqCjI9F10Z3G9Ox+vpon22Pb7a8zn9bkPHjYLS+/tb0uJy3Wx6z9HHH69+nNUCQ6UhWdCAi9ikADENWpPWhC1EVm2X4yNzWImDKYjauJ1Qsa5qM63qaUKHvYt/LqCA33kYMHPq9DAYxFMXtC523/vY+VNS78Pa8uLy27+3qWDTP4SSeUwxoOh4fY9M2u4YZr1def1/XGEi0rI9NlOtIPebl+XqdeF3L9eLjoOXKfcXgpNY+oe3r+np6vL1znN6nNUCQ6UhWdCAi9ikADENWpPVhLGSFCuG2orKpkLamKciIWORHI1kLQmwhKFtfNM/noeDi6S6WRQxq8bzj+Wq6CvJy/3H5cp5DRNM18bnH45Ix4DStW9p0/X0MQtuM5+Qgp+OP60hvT4E2hqm4PRHX8eOg6xSvqXTA0b7i9rwf/SyndT33fayB565dSUPKNv/1TecEQQYRqxcAhiEr0vpSxWj89F2osC2LVLmt+DRNQSYWsqWmadtuPWoKQrGI9jS3BmRdl1zgdy2ktd9sef2dTZe6hkLXt5wnHQrarku06fr7GLSfsnVHf5v4uPjYRBZeFRZNnO7HoWlclY8x7kvbdxDUet5203Xpyxp48tqv0qCyzVNXrz8f5wJBBhGrFwCGISvS+lZBwEWoKVsfmgppa5qCTDk9arJ50nTZhpdxuOkjyDgsNIWIbDsu2FXE6xqUxgBZrpupdcQuxyC9Xrx2PjbNi8taLWvidAcSHbvPI+r55bHEa+FlsrDcp2Pz6hs3utd15dGrl9Kgsk2tNycIMohYvQAwDFmRNpQqYGOBHYtfFaqiqWA2cR3p9crpUZPNk6ZscYga78ddy1Q4x6JZv7cV01pf5xgLdI8BKc9df2fT47xtdG2ZaLr+bccgs+vvdTQvLmu1rInTu5K12MTnVdnVbgjH5JU3Q4wCxi786tdvbB4uAkpXzxJkEBH7FQCGISvShlRhwYV7bM1oKqStKQNLVkiXmmyeNLsEGel9C/0e/y67qWnbcX7GLiFiW1jYVR/bLscgvV6fQabtscyMzynR1zVpcywcYuQuPHn1V28GmYs3BZSu/uJa+/N0bAgyiFi9ADAMWZE2tFmhu61gNmWRmxXSpSabJ03TNlQoG7eyuBjX/j22Q6hlQF2dym14HIzmq0Uhhqamc2+7Jp7XV9Hu67jLMcjs+m87Nl87Eaebtscy01359Di4Nazsuti3YxBDjOzK+Teubk6H9XZRrTgX3lx/ThBkZqy//0U/s/mIeF0AGIasSBtaF/6x1cLFbzbmROHBlEVuVkiXmmyedLekpi5JHg+jT/09zcVz16Lbx5l1i2oKC03TZbwm2YD6Xe0zyOh3k7VyeUyLiNOz58U2vS8FGO0rjpfJ9t2XQ1OGGNmFX/361wdhZN9uZbpj2fUoOB+qCjL6Qkcx5JcVzkmfb5cvsvS30Uf0RY9a93MfuTddB3EpAsAwZEXaUVVxqaI/+1Q8FrGx+FWBb+LYEi3jT9lFXEf2EWQcVER5zHH/saXFQaZr0e3jLJfX/rz9phDRNM7FAUw/yzCjv7V+Fpwy+wwy0l29yvP19kycF58DWauW5sftKah4P/Fx8zFlobgvhyQLMXIbv3xzvdNXLh6EkWz9Lj42s/ExgiAzY3cJMsLBxfob6195+TJhBhctAAxDVqQd1dhaIFRYyjiOIQs5LsyFlo+Fun8fIshIFchGxxn3L8qCPN72N6JpWldBJ4aLWKR7+74e3k8ZFrS+iet4vq5zPAZtpzzuLBBkaj3RV5DJztfHGq91XEfGeb6WcV399LI+Ns2P22hrwevLoWgKMbKJX7+pxrWcejPE7NsSIxWAXnzjxnNnLhBkZqzPVz+z+VFx7rmLb5uuaYLuabhkAWAYsiKtD1VAqiiNRbWKUE1rKi71Cbvmu2hVAaxiVdPd7Si21kgX0uX0qIgFcJMKV96e0X6zVg0dk8/NAUKW51uGmThfvytouPDOwoLWceAR5TI6DoWmuF0tr+POwmKTPu9yHYeF7Nik18seU03z4yZ0jL6WujZNj4mW0Xp+HgjtJ4ZDnbfQMtlj7xYz7bOc14dD0BZi5Gtvzrda9twbVzdPXfvVQQBRiMnW2cV/fXMbl39945rPBYLMjO0jyNz5zScb5yEuRQAYhqxIw+26gM9aPGLIaQoAWLd9sy3EOKxET19VC8zRA4zU9p9+MxTNkUUGmS//5b9snj7zylvjRvRTf2fdq07cdmrz/JPXk3mGtrXPsrscg6adfej8W8uqS5gCSB9BRteqbV48H3dNi8sIHVecZn0zgriOzkV/u1ub0D507eK6Pi5tQ7+75Uhk18nLZ9ei7aYIJ29/5qBrndHvurblcli3ADAMWZGG2zXZPLmtJQPrtk+2hZgxVJC5dtBJbX4sLsi4BULFt8KB1nGRrCI2LquwoeXisi7AtY7+dkG9y7Iq2rWcUFFeHkNZpLvQ9na8rLevaXH5TKH1yuk6VqGfcboKfJGdj47Zy/nYsuLf8xxSdF7luWhbQvuJYcbBpOncy3NpCzKals3zvnVemqfz9OOi84/LYt0CwDBkRRpu12Tdt9Slyt2idunehfXYF/MIMRc3z8/su2Miiwsy+lQ++0TfBXIspl3kx6JWgUWo+PW0XZctC3zr448tBw4UZeHu6aIs0DNF3IbO3y0VKt7j9XAo03Fvu04+jhhupM9d5+ppvkZly4hDiFpmymkiXlPpQJUFn+xa+LrGeQ60cZ9S56tzl3E61i0ADENWpOF2PQZDKLSoq5mM41k03idbF+u3D+YQYqS6qc2Z1YyRydZ10R6Xa5redVkV36IsoKWLf63jae7albV4OBhkxXtpE9qX9huXdcApA4d0cPE+VfiLsvDPttEWEBxO/LeDSRmQpB+rGHB2DTK+rmWYlG6p2ed5hPMUAIYhK9Kwm2ptKQel6/ddB9pjfR6VuYQY+fqbxzJnFhtkVLyrENY60oFj6CDjAl/Tve+o53n5ssCPenn9zOZHhbbl/cQuW+WyLvJV0Ht56yJfv5fLx7Dl7TskOcDFY4g2BRnN8zSraeW8XZdXoPK0Uj9mBJnlCADDkBVpiNjuUZhLiNG4mJdn9i3+GYsLMmpBcKGaEdd1i0eclnWZ2mVZH2MbsbXG+O+ot6Wf2fyoKEOLA0i5ftv1MbGlpexelp23rss2YmuNl8/OLTvvXZfvQtZag3UKAMOQFWmI2O6+zCXE6G5nz8/wO2MyFhdk3KKgn24tkNm62Y0B3HJQdvXquqxbZGJR3WbZUhH1MXfZliiDjAOHjjmOhXHAabuO0XJcic8xdv1yi0x5DE0OHWR0rCIuh8sVAIYhK9IQsd19mcfg/tc3z1yd7+D+ksUFGVNOz9ZVi4KChEKPC19NK0PMLsu64I6tLm26dSQ7p13HyGQhwsEuhg5fi9jqsk1vR+er8xYxHEnhsLPNXYOJl9dxxGVldp18XWl1WYcAMAxZkYaI7R6FqcPMS5W0xJjFBRmHjFhkq2XCxXdcV2h6WZBn7rKsW1myIlr7j9PdulEGH3fnElmxXyqyIOMAoGPyNLfUaFp2Ptp3Od0tUg4IWVBz2MkCkvYZQ5+PKzs3P85xXtMxN10nT8+OU+vHYCf9OCgUxelYhwAwDFmRhojtHpWxw4xaYdSd7LVfz3tgf0aVQUaBQgV11AWrP51Xwavl9bfCjQNODDIuXku0Pa0bC+ZdlnXRL3Rcmi8dpmKhr/U83dvST+FgoGlevkmh9bJ5Dlbx3H2ddF20H+1D07xsFsJ8DUUZBKTChpfROfm8dQ1EDBW7Bhnp7fix3XadfF21vM5N87Wsnw9xWe8zHiPWIwAMQ1akIWK7fTBWmFGIeerarzZXfn2jxquJqoJMU5gQsTBVUeqCXNNVxDpcxGI+FsJaR8ZiPgaDXZaVCgIqil3YC/2dBQCFGRfjQtvScbrYj8GnSVEeg9WxCv2M03VNfF5Cx6rjiNcoqvP1cjG4RX0uvi5CgULnENdpCzJ+nMvz9rZ9TbXdbddJ23egEQ41Cl1xOZ9b9vjg/AWAYciKNERsty+GDDMKMI9fvbS5VGmAMVUFmT51sFFhnM13Ia7fd1kW61MBSeFIgSebj/MXAIYhK9IQsd0+6TPMqPvYw2/+fOra5Sq7kWWsNsj4U399El/Oc2Erd10W69OPb1NLFM5fABiGrEhDxHb7ZluYUeuKvoHfKrBomuadunpxc+ZNn3vjyuy/3HIfVhtk4ngOdflSFyQZuy65m9Euy2J9urtZNg/rEACGISvSELHdIdgWZl57c75Vd7Frm18frrlsVhtkpAKKwkgcz6FgorBSfjq/y7KIOK4AMAxZkYaI7Q5FW5hZK6sOMoi4DAFgGLIiDRHbHZKmMLNWCDKIWL0AMAxZkYaI7Q5NFmbWCkEGEasXAIYhK9IQsd0xKMPMWiHIIGL1AsAwZEUaIrY7FjHMrBWCDCJWLwAMQ1akIWK7Y+Iws1YIMohYvQAwDFmRhojtjs2rC/lyy30gyCBi9QLAMGRFGiK2C+NBkEHE6gWAYciKNERsF8aDIDNjH/jRCwfXUD+z+bWpLw7VF4tm8xCPIgAMQ1akIWK7MB5VBZlH7jt3sM21fJO+z1c/s/nRK5ffOFg28srLlw/W/dxH7k3XGVOHsnPPXUznIx5FABiGd/75w2mhhojNwngMHmQ+9Sd3p4XHPhJkmhUXX716sKzV30KBZuowc+c3nzw4lqfPvJLOr0GFMAXGbB5OKwAMA0EGcTf1moHxIMjMWJ+vfmbzoyJr7dA00WUb2K6vZTYPp1P/YwBgGN7zV2fSYg0Rc9/9idOHrx4Yg8GDzN8cuy8tPvaRINOsyIKMW0KyebibBJl5+rk/vffgcQGA/vnA3z2dFmuImHvL588evnpgDAYPMv/4N4+kxcc+dg0yGlCuLkweN6Kf+jvrXnXitlOb55987WC5jDg4fZdldzkGTTv70Pm3llWXMAWQPoKMrlU5z9vV+Wi/QvvW316mPH6hc9dxeRmp+TKeu/Q+fKMCH0d2LpoXr6u7yJXL6TppurvMCZ1Xue9d7PI4+VxKtGzclq6NuvEZ/b6UGzXM2a/81wcOrzgA9M2Hv/VsWqwhYq5eMzAegweZ7/7tY2nxsY8uKNuCjFsgVGSqSNc6/iRdhWVcVkWslovLukjWOvrbBe0uyyoQaDmhorg8hlgkSxe/3o6X9fY1LS6fKbReOd3XQ0HB07Q94f3qp/bl65odfyz2T97+zFvbygbx+1ppm57WFGS0LZFdV+0zLuuw4+uk5eNx72rXx0nbj8el32UMKfpbaBnP9/Xd9/iwm/ofAwDDcOLBX6bFGiLm6jUD4zF4kDn9k5fS4mMfXSy2FYYqLlWUlmHBBWpsdVAhLGJhriJcxCJ812VdwMZ9SR9/LIBdyJchxNOF1ovzMkW5DV0DH0s8bh+HiNNtto7U+QgV//H6enm31vhax8dJv4t4LjHwdHm8hPYVlzuKuzxO0scUp8n4PCjPQ9ewnIb9qv8xADAM5y9eS4s1RMw9e1hbwDgMHmQuvX5zkX8UXWDu8wl3tm5TYZpN77qsi/3YAmJd8GodT3MrQ9llSzo8xeK/SRFbA7RubG2Iy/paaJk4Xfr4mwKDtiVike+Qov07gJX7zIKMtiHKwCC9nbi8yMLCPu76OMmm54CvZ3YeOLwXXrx0cP0BYBje96Un04INEW+W8THjM3iQEX//mYfTAmRXXTB2CTIqRlUMax3pInToIOPiXNO976jneXkV5sJ/R728fmbzoxnaT9mqIr3d7Dr6+Jv22TTfoUuULTYyCzIOce7WFXVgist7Hw5MRwk0uz5OUn+LOC1O7/K8xH5lfAzA8Bw/+XJatCHizR77zvOHrxoYi1GCzP0/eC4tQnbVBWZbwaji1oVlRlzXhXGc5k/jyxaJrsv6GNuIrQDGf0e9Lf3M5kdFWXg32XYdt+1T62Tz3cIhslaObL22x8mUrRxa3y1N+tl0nNvUetsoz4MgMz+/f+KJg2sPAMOh7mXv/sTptHBDxOvq+2P0WoFxGSXIqHvZ5/706N2BXHy2FYz+JF8/FTQ8PVs3uzGAW0jKrl5dl93WolHaZ4vMGEEm6/IlXcw7ZJTb1t8irucWmbbHs0kdh69d07G2uevjJAky81LfH6P/LQAwPPqkOSveEPG6tMZMwyhBRuiT06wY2UUVnaKtYDTl9GxdtaSoGFbocQGuadl4la7LumDPWiUy24pgtwJ1KbZFH0Fm2/E7KMYuaw55muffYyuV9Hbjufg4ylaXrjaNZeniro+TbAoyRz0P3E9aYwDGg1YZxGZpjZmO0YJMH60ybQW4dciI4ydU8KqwFnFdoeldxlrssqxbCsq7YUntP053y0BZULvlQ8Tiv0nRR5CRTa0q7j6m+Z6m66G/pa+NC/5Y2GtbIp6Lg4iuV3Zdt42DyYKMj7EMUpm7PE7S5xVb+qT3mZ2Hgp2X109dJy0Xl8HdpTUGYHy4FTNi7q0/4O6ZUzFakBFHvRWzC3AVqSoqow4CbsVQsajl9bcLbRGLc4eIEm1P68aidJdl3SohdFyaLx2mYoGv9Tzd29JP4dYPTfPyTQqtl80r1fZEU5CJx69j0PI+Fl3HuJ6vdzwnBwwt6yI+CzLS62tZ70vTypCh9TUtW0bH6+15P9qepzW5y+MkNU9ovn6P4dPXx8+7uB1fLx+b8Hq4n/d8+xeHVxIAxkR3ZcoKOcS1qtcErTHTMWqQEbrLUFaYdLEpTIhYuKqIdJGr6Sp6XbS6qJQxMLj4jAVyDAa7LCtVgKvQ1f6N/o5dsqzCjAthoW3pOF34lgV1piiPoUkdu4jXolTzdLzGQSO2UrglQgV7XFd6H7pG+tvnounlsnpsfH2F9xWPT4FIy8Trqb9jiPG2hNaP05s86uMU5+vc/HwQml8GPO1Hy8T1cDf1P4TWGIBpUMGmbjRZQYe4NvVauO8XfAXAlIweZFSAqFtIVqCM6baC1wWpft9lWZxWhx23BOGy1P+O5564EbABYHzu+PlraVGHuCYVYvgW/+kZPciIxx++kBYpY+rWHbcYRPXJu4phueuyOJ1uIcpafbB+FWL4Fn+AeUCYwbWr71eC6ZkkyIipw4y7+Qh1JVLxK9Xq4unuXrTLsjidejyybm64DPV9VAAwHwgzuEbVEkOImQ+TBRkxhzCj4jeOa1AwUVgpx4/ssiwi9qdaYggxAPNEYYYxM7gW6U42PyYNMkJhZg5jZhBxftKdDGD+6AYA3M0Ml66e42cP70YK82HyICN0A4Cj3M0MEZen/idceJG7wQDUgr5LIysAEWtWrTB6bnOL5XkyiyBj9N0QWUGDiOtRrTD6X8AtlgHqhECDS1AB5th3nqcVZubMKsgYAg3i+lSA+f6JJ2iFAVgICjR0OcPa1HOWFph6mGWQMQo0dDlDXLZ6jdMCA7BsNEBan26rSHz3J05zgwCcXD0H9Vx835ee3Hz4W88e3ImM8FIfsw4yEQ341ae1Kno+96f3coMAxMrUa1av3b//zMOb7/7tYwd3IiO8AAAAwL5UE2QAAAAAAAAMQQYAAAAAAKqDIAMAAAAAANVBkAEAAAAAgOogyAAAAAAAQHUQZAAAAAAAoDoIMgAAAAAAUB0EGQAAAAAAqA6CDAAAAAAAVAdBBgAAAAAAqoMgAwAAAAAA1UGQAQAAAACA6iDIAAAAAABAdRBkAAAAAACgOggyAAAAAABQHQQZAAAAAACoDoIMAAAAAABUB0EGAAAAAAAqY7P5/wEz3eso7DDMWQAAAABJRU5ErkJggg==" + }, + "pipeline.PNG": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABTcAAAIICAYAAAC7P9pVAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAHYcAAB2HAY/l8WUAANDKSURBVHhe7P190HdXWeeJMnapIKLS2t1jC9ojNW1NdZ2Zrmqquq0zf4xdZ6TqVPc5w0xNUVNd54/TbYv0mZ5SZ2whiI2gJk8CSsBBZADpRsGgMq3ADBJIC+HFJEqiJg+BkCfEGEjMCy/SggPW7zzfO/f38XquXGu/3b+Xvffv86n61n3f632tvfZa17p+e//uJ2wAAAAAAAAAAAAAFgjOTQAAAAAAAAAAAFgkODcBAAAAAAAAAABgkeDcBAAAAAAAAAAAgEWCcxMAAAAAAAAAAAAWCc5NAAAAAAAAAAAAWCQ4NwEAAAAAAAAAAGCR4NwEAAAAAAAAAACARYJzEwAAAAAAAAAAABYJzk0AAAAAAAAAAABYJDg3AQAAAAAAAAAAYJHg3AQAAAAAAAAAAIBFgnMTAAAAAAAAAAAAFgnOTQAAAAAAAAAAAFgkODcBAAAAAAAAAABgkeDcBAAAAOjgWc961uYJT3jC5rbbbjsNAQAAAACAuYBzEwAAAKADOzdvvPHG0xAAAAAAAJgLODcBAGCvvOUtb9k85znP2Tz1qU89cRhJch6dO3du8/DDD5+mWg533333SdvtALOe+cxnbl7zmtecpjo7Ki+WX4knC3cDzk0AAAAAgPmCcxNG8853vvPSQVoOiqXD64a7hzEG87znPe/S+lFJTsLIFVdccSlum47CbSHHZmx/JTklt0FVdhbOtzZn2btwbgIAwFmYuz0DALB0cG7CaOR88OasA1/EB0A9kaWD5BKYemiVYbJE40QORl3DfToacQyA0BObXjtays5Nz50qbg5oTsf2t7SNtaIqN4sPENp07V19sIYBwDaRjaw1aYlvK8A05m7PAAAsndk7N7Xp61AYNwQ5zvS3DspLRk/8xKeY1K9dEZ9YcV2qfwpdB8RYx1I27imH1ujQWNph1/3N126XTBljWB+eB5KeZvQapHVea5Ti87oR88xxTZEz8RnPeMbJPuX+6Gd+hXwb91ssj4PReLr2rj48D1nDYI6s7as+ou0f+6S1VnbzVPs1ozpcdkvb+GCqwuWzlh8P3ke47gAAu2HWzk0ZN33fMRYPGjJ2bATt0lF4VtSv+GpC1K6Q0ZvrmmqwdR0QvXFr/Pf55GZ03u7jdUM5NNRHaaqR7XHaN3aoaw7uiyljDOsjGvZ986/vKc+8fmkNyOuc/q7mnNdf339KE/Nq3znrgTZ/oJTXSqGDuuKGrpWxvKEHI/dNfXXeLgeB91zl0V7l9UL5czs1RnGPVpoux4PCFe9+S9VY21ET06ns1vVUOzUesS363WOkeh1eSeX20bWGVfV7LKonaVWG0iiP87qvyufwCo1NrCdrSF9gXfgebcn3gVnCq7G6D2IfshQ/1faKaGxcnu7xSkPX54zLbTF2/d8n6rfaxpsA28XjKuX7EgAAzs6snZt9BpsUDxo+MFhzpcto2wU6JLn8WLcMqynYGJS0Uc+Bs7TJxkZ1aN0lbu8xcKgxhnkRDXup62Ad7+lK8WAw9nAf29HlRD3LBwDZuak2ZhyX29ciljckT3RitKT+RxwuB1p2orlO7SldDjbtM/nA7g+EqvSS06uOKj4ql93VFvUv2wZZQ/aM1hqWx0LppNjXPMZuT3Tg5vGu2hSvp+vx36pPfw+dS7AOutYvK8+JOG/mOl9i+1uq1tSxeL2p7rez4nYuEc8RbLbtsoR7DwBgycx61/UGIMmI8ZMM+rRWBp0OBXHjzQeYOWJHozY49SEfPneBHAguX3XGQ9eUT2VtDEq7MAincJY2HcqIc3uPgUONMcyL6iCudVz3r9d303dot2M0rm9dinPP83GIpj65kh2u1dwf++ROLK/vYNQ3fpb2g/gEVJXG8pjLKVfFR6lcX1P9jPtOJfdnyLWRE9D09VPl7vLJTYerTXEc1WfPgTzGKsN1Ky6WGZ3ice7FPsTw6DSOdcBxEO+XOAc1/zSXFJ/Xipinbx05FJrTso/jXM/3uvpxVtT/bZWVcTuXiOdItW/BdJZw7wEALJnFODf7Dn8+JFaSkZTRIS0+JaE0Oojkw4H+9sFBeWQwakNyfYrTwWjMoSKmtWFltbBRFw91Q4n9VPvjwbTrySS1U2ndf/VZ7YhtzgZhrCsapfGg7wNyxHXoZxwfj3csV7/bKFBah1cae2hVm+P4KK5l3Cle7c34QOEy1N44fx1eSeMbUf80dnF+63ddN41NxtdGbZacTz+N01TXQWXm+tR+z32j3/M9pDzKW7UrjjEcN/kDHUv3ku/rSLyXqvg4V+Marp9xfsa1LpYpqVzPW83rHDcWzfNYxpB1aAixzErxPs/3sO8937sxX1wLYniM03gqb15zFe+xUx1eyyWve9kpEetTGrXV7dN4q73K42spFB7LMDFc19+oPF3nWFdMq7gxVGuY9guXF9sa8XWI7VAZCtNYxX3SeN7GuadxUlg1l3xPxfRwHMS1rMueE/k+zIpzVGhu6p6K97TvzYznoG0izfFoSylfLn8Kvjek6h72/RZtri68JoxZD1R2HHe1Kdbn8Epx7FxGXgMU5vZozNwnjW20sSp7tVpPhNYnXaO8L+Q1I/YrK66vQmXm+aH81fzoQ3li3e5r7I/3LqWLdapPSpvX4Djfq3Uzjl2eL7k9rsNjbzTPFR7npdQah1gm6zUAwPZZjHNTG1k8VGRi2kpGG1PehKJUT9zkfAiRZBi08ipf63DThQ0rq0VMNwa1yfnUdhE3fLW7QgZFNB6iYrg26khMF69X34beytd1rdSPeH0q5fZVuG0yUPRT/VNY7Gc19xwX0dxxuNoe+20jTWEx3H9LsZ54DdymmE9h2ZD1PPHP2BcT00Sq+uL4u22aUzldbJfnWcTx1TjC8aF5EOdMlO7DSExXzdmYNx884v2ockwss3IIxHkf8w0h3keSysrtmorL7JLI41Ldd17vpNjHmK9yRijM8dW9bieH5OsVD5HVIXMI6oPLkEzczyT1q7XOeO2Txl5Xz5lYtsvrKqtK47608jmPx0+4n9X4ecxzepWfpXBYD7qmuvZWlwPR86qlar61pHojsR1deav1dgiyO3L7s0NKOC72pQuXmfvTYps2ntPk9UphsWw51vy7y/MY2waLNlk+i8Q2u6y4x8Vrot9jeW6HFOdW3OeqMvM+3kXci7JUplEdVRor9z3OF+XNxPLifIl7WFbsl9fxLuX53qoTAAC2wzhP2Z6pNjxtdJVxrs01p7W04Zl4yGpJ6X0YHbJ5WVMObXHzlVrYkIkb/RDiJm3DRH1zmFQZiNFI6VI2GGJcZcRJ1YZe5eszrlVOdN5WGnJNYts053zt9dNxVTnOE/E8jH1XG5U/hokqv1HdNhxjm4TK8/XJ8yHOJ8XFfMZp8nVwmVV9mkc23PVT/cz3YTR2c5zHMY8BHDeaM9U6P3TtULqYr0sqx3SVKeK6GfP1Ee8BSb/7vtkGLrcl3Zcij0tFXCtiH2O+6n6N+frkse0b74zq1bzo2ociVbpqjWr1eQjuQxwTl5cPsBGnifX5+rTa4DxxrLzX5TkV51wMj3M4KpYJy6eykzT3dZ3z/t9nU9lG1Dyq4rPiXIr3eJ+GromtMjXfW05c22CVXVvhe01lqr6svIa4/LgOTLHxhMrPZQnnk2I/o5NSijaefkYbLqIyVFeuJ84H9SHSaptQXV5z8jjH9ajKm/H4d8lUcVlxLY5lqz8Z91HyXPY621IsZ0jbpXgfVnUCAMD2aO+6M0AbQtwIomRg5E1VG2lMk8mbljZ8bzrK6w1Zctm5TBkP3rBVXj5UjSVvjtvGhpgUjRcZYg7PhlA2bO3skjRmcZyywRDzRcOmb0Ov8sWxiW1UvMqLRl9Mm9vUh9uWx0FEYzJThTssG4oVVX6jvilO168izuV4UPA46BpFgyriNPE6+Jp35RuCxjCXLTzGcU4AmLzmxPnTtXZoPsV8XYr3d1eZYopzMx7qJN278d7cBi5bqtpt8rhUKL/jYx9jvup+jfn6ZMdA33hHWk65rIzqivudFetr9XkI7kMcE5e3D+emiOOo3+Pf1f4Fx0HrntF6lOeQiPOmivc+LsnG9Tom2yDHmVimpHJtS9ieiXFDyGVasl+z/T8VtaWqw8ptdfhZbTzh/uV11vmyY1X4zKGf2Vazs1LlDsVrZm5Dq23C17O17nlMh1znuGarTx5X/VT5sS/63dfefdfPeJ6J6eO1rcbEfZTcVq/NUrS/1R7NfdVllFZplDfu9bEMKY5hVScAAGyP9q47I1qHFilu/nlDyUQDKxplJhqI1UYnZWMiOr+kuIkNIW6+0jaJbc/9tRFkxX7FcZJxnIltzgaDw6UxG3qVL7dRhkVrfLva1Ifb1irb5WaqcBtZGu9WeaZVrnCbqrEyVRqPQ1e+Ko3Dxo5dplV/3xgDeI7k+dMKF5pPjpPy+twillkd0HT/Oj4eZlpkx6byD23LGFy+lMcikselcrLGA2HsY8xX3a++x6VqH62I493lhIsf2mg8owMj96mF+hr7Fvew2Ha1aQzuQxwTl9dVlu2KOMbuSyufy43XOOaJ/dM10J4Nx43mh+doVr7nYro4x0xcy7ITUeua4yQTy+xbU1vzPtPqj9W1lgxlyD0c8b2n/sS1oMLtbOH+5XK68jlPdd28RrTyVrTa0AoXHgOdy5Quy+c1/d6F1mqls6p9agi+hrnOVrhRmOM9nnEPcr54zhxKLCOOYVUnAABsj+E74AyQkRU3BklGmA+RcWOXMnGj65M3nb4ys6FXGQJd5DZtExl+sewuxc07tqkyCLriHS6N2dBb+aJBbMlwysZGX5u7cNta187lZqpwzYfYZrVVB8/K0VHlN25TPlhEnCaOp8ehGmNTpRmSL6PxUnq1w7JRm8txW8feH7AedGjQeq2Db5wHujfiBypSvL/jh066t5ReUnki5tOhKx6QlEZla17GOj0fLc1X36OxPqnPceR+Ob3buAtiu/ru1dwmj4vapryxrDjeMby6XxUW06gsXwuhejSGqt/k+uKYel9XG2LZumYuVz+jQ08yKlv541qp9FVapYnhLn/Iobpaw2J5rWvuNTGOsfupMis8XvppPC+rawJgNJcr2y/Om7j+xTlmYr4+mb4y47ramvdd6F7VPRTXNems94PvtaFt0n2+DRtPeMxyH7ryOU81xl5XWnk1froOKsPyeOY2tNomHNcn1dVFbK80BPVBe4HX1Sy1zfja5nAT+xHHM85VS+OkNPk6a156THMeK45hq04AANgOw3aTmZEPwt44+jbKuNH1yQeRvjK10cX4uIkNIbdpm2QjsEtjDIKueIdLYzb0Vj6ha1EZMrGcvjZ34ba1rp3LzbTChQ698TCua5EP0F353aZ4YM84TTUO1RibKs2QfEZzPl7PSrmcvjGG9aNrn+dJJd3r8QDhuZnlOVYdRCq11qMu6b6tDq2RVvsq5TXA61rXfR6JZfXdq0PHRW2IxLjW/RoP910y2dlYSf3R+FRxlUzf+Mf+teag5kMfrTXM1zA7tfW7nUx5TrsdrXrdp3iNfT238bQarJ98L8W5FNe/GG5ivi7Fe6uvzLgeDbnfWsgedDmtusag/FPadFYbT3jM8prSlc95qn7H9S2idnmdaim3odU20dWGMcT2Sn0M2XfidfS1zeHG/ZByX9S2GG/FcuKHW12KY9hVJwAAnJ3+3WSmVBtH30YZNzptkkPIZeaDbt7csnHTR2yTtC2yAThEfooltkkGWybGZ4PB4VJrQ5eRG9GYtvJFNLbZmDRdberDbWvV63IzrfCIxtQGmdoe6crvNqlfLVxufArK49CVr0rjsCFj5wO76tf8j/dEq/6+MYb1M8TBpQNYXkNb+TzHNP+GHHpiuZ6PUldef8jVhef8EOX57/B8v7SIZfXlGTIuXQdyqWstVt6YNivvHX17ksc6XpusWKfpG//sOK4O+UPWvdYalsdC6SSH6WceY5XRVa/7FK+x7oNYj6UwlaN91Xs4gPCcleJcaoUbx0n5/mkRy8w2nohrUbaFxjAX56aZauMJj1lrX6hwnqrfXldyXq95al+uq9WGVrjoasMYYnulrvUrXneNd1xTfQ0ltc3EcOXJxDnZ6ovaFMuRXHfcS/LTuzF9HEOPnXTW8QMAgMfT3nUPjDYybRzaMOImpo0mv3LjDTE7yXxYcv68kWpjiZup0sko02HB5DzaDL1RKX3c3GK+oeRNc1tEJ2BrA83jZSdZ7rPGW2klpYkHrGhIiJgvbujxU3vld5x+RgMj5lO7VX40rnW9YlqTncy+rnHutLCxEdsbcZmZVnhG5Spda6wqPF6VQSbiOMQ57PnUuuaiSuPx07WJBlqFx6s69LTq7xtjOA50/bWe5Hte8yMfDiKaazGPfs/3ttZ7z7OYTvXleRfTaa4qPpZfHQJbqN3O16dcpveP6l6qiPuN97c+WuOiflfjHceha/1UXpUR00uqq+VoU/81tnkP8d4jXG7sq9JojLz3x722aofyKm3VfoXF/VFph4ylx7AqU31VfbHN+l3jUI2xxkFpVGaF+qN4/TQqx33UT+WVYr81LtW4wzrRtdY11zyLa4vmSl6X4hyP9pjmj9JLnjue646P65PSeU2J8zPmkRSntCLWJ8X7vULlq179jPNZ7YhrhzTk3u3C95raP5XW/ew2tvCY5X2hK5/zxLE3bkfO6zBfj0irDa1w4bmlNe6suG2Srrmvt36qjwoTvk6S1m/3Re1TGsfFa6D54nDJ805lxz1A8niqPO8JrkM/Yx0ek5jfYUob2xrjhMdVqq4hAACcjX7PzIHIm0NL2qAiVRrJxA2qS0abUhVfaehGlY2zSvHwJrTRKtwbfRfaXGNZXQfUfMgz8ZDWpWhIiBgXN3S3f4icr28OxPa2rlNuX0WXESdcViaHa5x1fXI5Nuxbc7VybMj4crwOzRHFeR7nMj1mXXOxlcbXXPXZqBM21twvj1dul66x53Yuu2+MAfaJ52M1VwHmiOds5RiKh2/m8/HQsnuytLfHPV1zpErnuZOdQi3FuRbX1C7JRohtqWi1L6uyh23HVHZVhetS+/vYpo0nWnaR81U4Txx7E+dDxGHREWy7znG5Da5HfcvIBrWtl+1FoXFSuNL1oXRuQ0siO+tbitcx2tF98ngOuafcryFnOSmOrcdVqq4hAACcjXr3nAFDnGHaJPKm2toojTbdvg1J8SZvdD5AZCm8z2AzVf5KkWiE9BGNgOgArMjjbEdol3EbxzgaEiKmy8ZSa+zUxuhkdb7Y50rZYLRRG5XbV2FjI7fXuKxMDo+GlNqicmObPLYmj6PSRuMzXhvNSaVxW6VqznnMuoymVpp4zav6fKiO6XI/fY1z2S6nNcYA+yTO6657BWAueL62aK3rsF6GOG+0N2fbo5Uvzp0hTqfoaI9rasvWk6KN0yLb3ZWqfgnHD70PfN9EmyfLbd62jacwxWW7yHkqnKfqXxy3SH5SV2X4HORrldsQ7U+lkdQfE+Mllek+Oqy6PhnZsF3zReU5nducFcPVhkh+athSnnh9PJ59cy86ez13KsU2xbH19ZOGzlEAABhO21KeAdo85fSKm6U2DIVFAyGiDVAbljcW/cyfPCqNNpW8oWrTUVp/KifyRqe8SuPy1TaVpfChxE2vJaWJ2JBQm/uIjsK+zTMbDNFYVd9jWarb4+48io84rZSNpXxt4tjZIas4j7/C83VSHpVRGU0Ki+1V2tY8idjYyO01Li+jsHydPGbuo37q76q9Hg+Xr/bGuSdyeZLGI16niMZLaVrxoiuNxzDWp/FRWrXXyMEZr4vbpPz6O887j3E1DgD7hgMGLA3P12pP0z7hNXvIngfrQddedkTcj6Vq345Ue3jenyv7Q3aKwvI8y2uq8sbylUdhQ6n6pXaoHtXd6pfap7T5w+8WGiOX31LcI/KY6Kf+nmLjtewip69QXYpr2W+KU5sy6oPHRlI5GiP3v7o2yhP7mcdU9al/sVxdL4WNudYaJ9UVr7XKVDlxbDR2CqvG3m3Q35nYd+VRGSrL4yXFvml+xfnseqo5pbSx3fpd9SncYbEP0dlaXUMAADgbs3ZuzgFt0N6IJOgmjtUY4wYAYF/kgzjA3ImHYh22NYel6FjQoR3gELCmAgAAwKHBW9cDzs3hxE9BJZybADBHeHoCloieBtITRH5ySfJTRYoDOBQ4NwEAAODQ4K3rAedmP9mpyVgBAAAAHAc4NwEAAODQ4IHqAedmP3mMJD0ZBQAAAADrBucmAAAAHBq8dT3oS6f9Cph+wuPRl4HHL+vGsQkAAABwHPBVHwAAAHBocG4CAAAAAAAAAADAIsG5CQAAAAAAAAAAAIsE5yYAAAAAAAAAAAAsEpybAAAAAAAAAAAAsEhwbgIAAAAAAAAAAMAiwbkJAAAAAAAAAAAAiwTnJgAAAMAZefjhhzfnzp3bvPOd7zwNAQAAAACAfYBzEwAAAOCMyLH5hCc84UQAAAAAALA/sMAXjg5RT33qU0//Ajg7evroiiuuOJlXml/PeMYzNjfeeONp7GZz9913b57znOdcOsQ/85nPPMkDsAvsMHrNa15zGgIwT97ylrdcWhMB5oLWUO3jmpva1/V3RE8aa856T9f+vma0l8TxYG8BAABYBzg3F46NUTgsMo7XYiDLsWmj/1nPetbJISAehnwIUrjilS46PwG2iZ2b+UAOAADdeP3MMtq7q/i17uly5LqPsl9kz+hnF8qjceRDXAAAgHmDV2zh2EiDwxEPB2s4EHT1xX2VQxNDH/YBzk0AgGlor/aeHmXiWxhRa3VuypGp/o3ZTzwm7EEAAADzBq/YwrHRBYfjtttuOzlASHple8lE52WFHU2HeG3NdfMK2XHg611JTw1H/I9c/KqhpCdyFFY54X3AFZpPdgA873nPOwkTMY3ui/zapu57ofKVz2WoDWP+oYz7qTok92FqH11e1z3qvkQHxlnGUOVEJ4niYtlGY634Vts0jopv3eN67TteB7VVecasu75Ovn4RlaO4av3TNXWfJbUjX2f1WXG5fw73WMZxVl2tMRa5z1ld1xmOG83xOFc8X+Pc9/0g6a0NUd0buyDO633VOaU+36tj1vW501qr1oLX6n3NKwAAmAd4xRaODTWAbWCDV4ZhhQ7hitfPfXPIumH/yKmjeeiDpX7qbyk6IeUU8iFZB3Wn0d+S4rLjyPGeU7Fs4zT5axrsDNBP3S/+W3FuqzT0IBzntct1PWZMH3WYc3jlMHP81PKN41vjI2l8Iu6j0lW4zOoet+PT+WNd+jn0EOsyctuE1z8pomvpcI2F2ynFelvrp8PlSPAcUTke8yqP8Ng6Ptbr8a7GCkDE+SxVxPjqntglh6jb9R07rbVqLXit3PecBgCAw8IOv3Aw1GCb9Bm80RGzbw5ZNxyOvuvuQ4yfOjLRYZfzOo8UHaWRnMbOPT3dFx142bFmJ5ycWENw/6TKiSjG9tEONDmIM3aYxX5vYwzdbv10XH4qyH1VfIXz5bpivjw+Hu9WmRmllapDr9c/KeLxjHn8j9ViWGv9jOVqvsQ80XEa55GfIs3h+t3zb8wTq3B8xHknVcT4OC/3wSHqdn3HjufG0HVzaXgv2fecBgCAw7L1Hd6HEG0okg8F+aAno1yHknhI1GaUD2PeoFqvqcWDZkQHILXF9Us6pCksH46E6xf5NTC1szpE6GCj+FbbXHc8mJix7WvhvEKHpNhutS/X7biqTUJ9V/wQg8fp1N74hEm+hnk81efWmMb5o3iPsaQy4tNQcfw0B/R3C8+3ON76Xe3O4+341hj5MBrHSH+rDZmYbui8Eup/7HtWVVeLXJby6m+FG/0ey49S+31dKsVxnzKvNc5qj+9jSX97HnXVrXpg3fj6x3lmNHcUpzlW4Xmd1zP9XYVHnEb3acbrneZsXid0TytOGoL7p7Kqe2RKH11m1T/fm77/zzqG1fhEh12kq13CZeZr7bWhtYZUdbVwWvc/4r7mshzWWq9Na6wcXs0X4X0h9ttjqLUw4/mXxwnWj+ZF3M89R+J8jmtQJdmscY/OynNO6T1HJc3jLvtF4dneUn7byopzeKXqHqmo6tHvXXZdpT68LuV2OVxo/ON1UVy8JnENaNG6r3PZKiNf84jyx+ultG6729FSvKb622tZnDO+/h5jlZ373nUNx5wJxoyxcPpKqsOo7XHu5D4NQWnzWLucqv8ee+Vx3li/wwEAYDrDTgMj0OLsxdsLtjYb/TRa9PW34rWwKz5uDnED0oaqMMVXOF4bntHm4PJcf9zwFJc3EIc7ndvlPCon53G8+lrhvHnzndK+Fs6jMdNPt9vjq59xk7VhEMc44ngboV0oXWy72i3F8XC7YtpW24TnjfI5nfLYAJBk3Hv8XKfjZBxm4nzTz9hmh1VjVJUlqjFyWRmFqX2ub8i8io4B53X7Jf0d53sXMiJzWXEs7UBU/xXnsYzjpHFQupjX/ZBcxpR5HdvnMmP7nCaGx7pb8xjWQ9xTMl7/43yL8nz0XDKKU1hemyNdadwmpalwnV3lm67+iSl91CHRYfHAqPtcYbqHzK7GsMrTN24uM46F2+x8lRw/ZLy70irM8RGv+RqLrjqcX22KtMJNNQe8NlZrfcsJAusm2lOVPB/iPK7k+daS52nc0ytpzZC9Eon2ViWlj/dspa57zMR64trlMhSmNCbH+2+pD+fL7XK470e3I/Y/5nF4HjPj+NjuaCPpWqj8aCPZ/jJui6S0vn4qW6jsHK6/rWijKd516nfbXvpd0u9un8txH/SzclaOPRO4vqFjrHQx3u2Xos2e++8xHWpb53vDdcR25Wvj+1J1uD6VkcsBAIDpXG7Bb4FoNGnBzs4M/e3Fv8so8mbVOqQZbwpxE9HmoDBtcpG4GeVDgeuQ4gaoOt2mGC5cT+uA4fK6DKJIV/tauA4pti+WFQ+ydpypTxnlcVnVWGecVsqbuPBcUH9VdsQGTt7I8/xxO5Tf42bFOm1gSbHtyufrpzpjO5TOY6SfxmXFcTOtMXJYxuHS0Hllo6d1PauxrrAhpfKyweg+qg1xTJwnXxfj61PNz7HzumsM1N5sZHbVDeul67o7rk/x/haeq3ltjnSlcb2t+8T1dpVvuvonpvbR91y8t3xAjPforsbQ+SJ94+Yy41h4TeqT1pK4Jrdw+qrdsa5IXMMkraka17h2CufP/WuFG49L7Lf6ojD1K67f0U7K6zqsF803XfM+aa557rSksmxnVPLeq59VfFS0IfTTc7MlzfE+J23fvI71DLXrjOsYQ2utc7gU26Gfjot2jPutnxnb5tHu9Lox1IaLa0YcE5WTx6JvTRKKt+I+4rZaKjv23eOf+xnX/9g+4bHJ7Rk7xsZx+ZoJj12uS+FVWRUuX32N+47a5b7oOsQ4j7njYtvimLKuAwBMZ2fOzby5Ghto2QFiKiPfm0gME9VGrk1BYZVRI1obusKkynHkA2HO02qXcZlxA5vavhauo2q3x0eKm6XGS2HZudza8Fu47Fb/XU81D4TzR3z9o7Fk4uZf9dfGehxvz7doMEaqMVK9bnssS7TGyGVkHD50Xqluhan+jPs/9Pr4YJKvs1E5io997Jt/vj75mk+Z1742Q/vTqhvWTdd1d9zQOWSquZ/pStNXr+L6yjdd/RNT++i1Kt6TXtfigWtXY6g4KdJXl8uMY+G1I5c1FZdVtbuvLq2l0eGj8Yx7q/Pn/rXCjccl9lt4PJw3/l05SGC9RGekrr3vYf3UPe64aFvH+SxVxPh4T0TbSNJ+bZtM6byWSLYxvObEPEZp1IdYR0wbw/uYYtcZh4/B911uo8Ore9H2WqxL+fW3xi7bt3aIxTVgrA3n8lvjEnFaldFC8VJlv3rOVbZ6y072nMnpjeuLjB1jk8cm4vV26hpqe1eKe2nE92u8B+L1z/NSeEzzPgAAAMMZt8MPoGWkG2/WWvi1+WR5Q9Dvxhtl3rBt4MQNymHaPHLZUjQCI1WYaRkB+lvhrb66zLi5Tm1fi760Hs9onNipljd2X5vKkKnoqjtu/lU/JcfH8emaP74OUoXLjHmrsEyVxobm0DFSmJRphYtqXtko1/zI2JCL6buwIak5pTxZjo/9bs11o7Q5j5gyrxVeldWiVTesm67r7jjNpTF47sW1J9OVpq9exfWVb7r6J6b20R+USFpXvH7oXozsagxdd6RvfXGZcSycJ5c1la52D61L4+k1TXuCafWvr9/VHIh5vO9IqjcemGH9RHtKys6h6OSJcyzOZ6kixiu98Z4u5TVD2I6UPG/jPI33RQunlWLdffgejvdLppXG9Y3BZeU2tsJNVVdlkwvbY9FZNsWGc5jGv+V4E54bKqeF4qUK5ct1G5cd8049Ezh8zBiLrnyxLWp/y9naQnmUV3W0qNL0jbnzVGMKAADDGLfDD6BvcfaG06f46XM8pMVPu3y4iJ9quv4+ZWPN4RWtDcl9afXVZcbNdWr7Wjh9i6qNcWP3pm6nWgzrw+krPGZ9kiEWDTCPTzWmscyKqq8Oi3MkU+XzYUHtM3Ee5jFyeKYVLlrzqjJ+VZ/n+9CDrevuUxybVptM6/o4vE9xXnvc4/3RRatuWDdd1z2uZV0HucyQudeVxm1q3Sdu05C53Tevp/ZR2Nmg/dQf2OT1Y1dj6DIjSquwuK4arXFe+/JY+MA+9IO3Ltzuah2NDp0+3Jc4B6ow0Qo31Ryw86g1vnA8eP4MUTUfrYoYH+ea5+QQed763ophXcQyxsxz1zPWrhOubwwuK7exFW6qunxfR+ev7c3WuaRPcRy0nnu9lNTGat303FB8C5dR0Rpf4bJj3hjWpXwmmDLGoi+fxsR5Je2PrbQZ3xvxnJpxmji+HoPWmDtPNaYAADCMcTv8APoW564NsQsfzLyZ+ECWn+Z0/a3No4XySBWtDamvLy4zbphT29fCdbRotdGOMhs9PtTlJxW76KrbY9aKb+Hxqca0r8yqrw6bYgT7oO28XWOkcCnTCheteeUxkHSdFG9jVX+PdT7H+ddHq02mdX0c3spX4XEf2r5W3bBufN3zwc94LdPP7JzT38qf7/8hc68rTd98V1xf+WbIvJ7SR6Ew5dNa5jWkWj92MYaKkyKq2+HRuag63AYpj4UdAupDrk9lah/rGr+Iy9KYxLGIjk3JyNZQ23K9lZNCaRSW50Ur3Kjtio99cPlj9mRYJ54/QxTnS85XEePjHPecHCLbkV4TpCH3Yywj1t2H66nWPOM0uR2ubwwuK7exFW6qurTWOdzrjz+Eyk5IpxszNsJrYlxT9Xtc71SmwtWHFs5b0Rpf4bJj3ipsCFPGWPTlE97bbOtLQ9Zb3xs+j1Y4TRzfvjF3nmpMAQBgGON2mQH0Lc5dG2IX8ZAmbPjnzcX1tzaPFsojVbQ2pL6+usy4uU5tXwvX0SI76IwPcm6HjaAuYzHTVbfHrBXfomtM+8qs5lYVlnHf42Fb5MNl1xgpXMq0wkU1r2R86gCv66Z67ZDQ32pPNE77cN1x/vVRtSnSuj4Ob+WrGHJtIq26Yd3Ew6DuA80b7wMiP6mi+1RpfL9K+d723Ou6N7rS9M131zvk3hsyr6f00cR80REX2cUYOl/G66rkevS76vchP4+F1r3YFs8D55VafctoPrmv+qkyVJ7+1prr8kxr/jlMY2da62cr3FRzILYzym3WOCoNrB/PH2uoHZDzVcT4eC97Tkq694ageek8Q5xETivFuvtwPV1rpteLvG65vjG4vtzGVrhp1eW2yQGpa+l0+bo6fMzYZJTX60gcC4UrTH1o4foruq6By455q7AhTB3jvnwZjY3Limt6he+NrrHzPjfkwy/jcrvmNQAAdDNulxlA3+LsDSQeTofiDVrOJR8u8iakvxUujTH8naeitSG5r9WhKrYjbq5T29fCZeVPfIXbXdUVDSqn0/iOwflb+HpVbWvRNX9ifypszMS8NjBaxrnGxWXmMYrXyr+3xsjpMq1wUc0rO/Fb988Y7CgYcsgwVZsireszZV772rTqyrTqhvUT1/xqDmg903zKDjDdA9X647VC87aF0+ieyHgfaznUvPZ1lW88r/MhPDO2j8b3mdT14dXUMazGR7iMCvXZ11NjpTVK64bHtTUWCne9zqu/FZ4dA13ounh9lNRn91Fl5nVefVR6X1f91N/5+nr9VFxkyrqq/vha6KfySvH6qB1D11tYNr7mUp57vnd0T8X70fPOqojxXXk1N+NcU/1aL+K94nlsxftYa4/mb1xL4ly2naI6+u5lr2nKX6EyXG6+Pxw+BrVbeeL4iFa4adWlcVG42q/x0O95zRAKU9wYG66iWl98fdWHFoqXKtz3WKaJcyfi9bNrv8pMHeO+fBVD89hOl1pz1ftb7GvfmFfXCQAAxjFuhx9A3+IsQ8MbnDbsvDHIYPJBJxMNGv+siPG5HP2ttuVDntJLFa0NKW5w0dBUevdRyhvllPa1cB1SNCTjwai1kdpw8ias8R2D623h66WxyGOg9mnTz/Oka/74OrTqtGES82o8nScbiIrzGFWGpYjXSj9bY+Q6Mq1wUc0rzynV12fg9xHnZ5wbRvF5TKo2RbquTxyrIfM6XpvcPsXl9cF1q3wAgDXj/axau+P+Xq3FsD5sT/Up2lrRZpIqYnzMKzzH+mTint5SnK+e41m5HZlYz1i7zvnG4HbmdrXCTasu3b+Oc1srm3+sDac06rPGwKiuai2Jc6Nlazq+wmVW608sOzLlTDB1jJ2vstk1Rqon9ltj5nNb/tCqwucmXb9Yjn7XNVGc0sQ4j4vaVqE2Kb4aUwAAGMa4HX4AQxZnf1JpaaGXvFlI1eaisJiv2uyF0kXnojYflR8NtZzX4RVdG1IsM9ahn/49b8pT2tfC+Z1XY6iyXL5+RkMnEg0naciGHnG+FtrUY5/cNslh2fjsmj++DlKFy81543zTeOQ2qI3RAInoOjid1Bojx2da4aI1r+KYRSmdjKY8n7qwkSXFvsf5Eema66Lr+kyZ1/HA5vkR08fxjgcap9VPAIC14bWuRddaDOsj21MtxT0z2kxSRYzPtkXe0ytlGyLb91mKN620Q2ycqXad48fgcnO7WuGmqy7Zvo7PYxgZY8PFMbH95XSylfJ4KCyWq5/xvOCyKtz3av2J8y4y5UzguLFjnMdCsiM4jmlf/S3yveFyHKaf8V4UHhelq2BNBwA4O+N2+AF4ce5zzmnR1wbjzVXy5tPaxIQ3xrwJZ7SJynGSN1JtXNG4Mk5T0bUhqR612Rua6tAYKNzGS97gxNj2tYjtUr0eT7VH7eoaI+F2qx1jcT19aC6ojUrvPPpb4RqHSNfmrnHsqtPjXc09XUPFu7+S+tw3TzV+Q8ao1S7XVdGaVza8dC0VZ7ksqfqUv4XS5r6rPM2/PD88xrlNRuOl+Or6iCnzWu2L/VM7W+mV1nNcarUDAGDJeI2r1kHtHV7Px9gLsHx0vbM9oP22sp29n0uVfSKiXdCyVbXPxj1datkQorK3lL6yt1R23NOVL9uFLabYdU47Bo93Hh+H53E3blNFfLhA49jFGBsuzw+NreZGNabqT0yr3yMOr1B7FFeNteed2luhPLFepdPfCs/tdLopY6y55THTT42jqOa0fh9rT2rsNbZx/up3XZdqvNUHpcnjbFS/4se2AwAA/pJxOzysji4DBfaPjRtdlwo7PlvGEQAALB8dkLXWSz78S/Egrf0AAAAAAABwbh41+tTRh6Tq03fYP32fUvd98gsAAOtAT2HlJ7b0u8J4YhMAAAAA4C/BuXnE6GlNHZb0OgbMAzs3qydy5ID2azQ8sQMAAAAAAAAAgHPzqLGjjFfS50P8HibJryLm7wYa+p1UAAAAAAAAAABrBufmkeJX0vWKG6+kzwu9ep6/pFySk7P6wnUAAAAAAAAAgGMF5yYAAAAAAAAAAAAsEpybAAAAAAAAAAAAsEhwbgIAAAAAAAAAAMAiwbkJAAAAAAAAAAAAiwTnJgAAAAAAAAAAACwSnJsAADArnvWsZ22e8IQnbG677bbTEAAAAAAAAIAanJsAADAr7Ny88cYbT0MAAAAAAAAAanBuAgDArMC5CQAAAAAAAENZlHNTB91nPvOZJ4de6XnPe95pzDp4+OGHN1dcccXmqU996kn/nvGMZ3C4B4CjY47OTbVHa/NUzp07d1LGa17zmtMQgMdgbsA+kW2p+XasX/tx7P2H7cFcAgCYF4txbt59992XnH5ycPrwuybk2FSf1E/1T5umDj3Hzjvf+c6TcZDzFwDWz1ydm2fZc+zAYk2HDHMD9onXsry+vuUtbzkKW6vVf4CxMJcAAObFYryDNv516F0rbJI1HhcOfgDHAc5NOCaYG9CHnur1GnTWeeJy8vq6rfLnTqv/Xcjx631J0sMHentMD17AfPBauq83+6bMJQAA2B2LcW7aqFjra1vaGNW/s7z2WOFxW/IrE37tQ09wrgVf7+c85zmnIetiF3MZlo+eCNKhw/e05oj+zk8KReem1i7dJ/pbUlzXQUKHzViHpN/1ZHz1RJIPQypTcj79FC6jkg68fbj8Sq7DqH1KH9uuNxUUVrVd8RoPxfnJfym2y2mE9k+Xnce+Guex+4baGb86RuXlMlxHay93+3I+t2lIX7vSiOyokNSurj1GeWLfspTfaA6q/uo6RnY5N3w9dZ1dXq4flkO87nlujMXl5HXUdezb1lKd+7QXWv1v4TVL0j2ntcP31T7bvVQ0zhqruEbuCq+pukb7wPNi6FwCAIDdsjjn5lo3EG/+296Q1z5uS2VX13suqG8SQMTOIR0INfd9mM6HHq9bcr7F9D5QStWaJodOPHQqj8tyWHaa+TDkn7Eukcvw39KQddWONPdVP51f/TNyUOXxkVyv4rITK7bHaaToxHK407h+/e78aqN+j33330OfTIrORJUR+xJx3S1Hm8vIY+tynb/V1740nlOOV1pfGyleE5P75vIlj5nrkGPIcR5r1eMwlWV2NTfi1/i4TP/N93ouD61ZunZaJz1XzmLTKf9Zy9gmbs++GNP/uDbmvUNxuh7QjcZZY6h1aNdoHd5XXWLMXAIAgN2Dc3Mm7GrzX/u4LZV9GnuHQH2TAIwPiXnOK7zl3JTk5LHjRj8dl/Mozg6cmEfI2WPnkH5GfBhyXMwXcZqpuJ7oaIu4X9H5JdQetz3ndZskjWNFTBMdW9EJJ8W+xzorZ19G46u0Gv84flrn8ni7n61xcHvynuVwaUhfqzS+BmpnLl9j47zxyTX3TYrOjehIjw5glaM+5vI9/3N6se25oWuWw5RWf7fqgPni66k5ZEf7kPuyhfJLeY4eCrdnX4zpv/YZpc33HgxH46wx1Dq2a7S+7asuMWYuAQDA7pm998FGfSVvJt7M9LcUn4KI6EAhg9DxTiOjJR7IjDdkla94pfNhRvniIUGHIR8yJNVTlZlxHZW8OftTe9Vd4UNTPOx3jVs0it2f/Im0iAfWSBwXpXFdSld9sh3HReOm+vPhrguXn8tWmMco15PrcJtbYyh8aIjXVYztg/LH9LoubnvX9ZZc5q7mnsKVV2U4vfK6nozHXqhNNvQlxSks4rhKGkc4XjTHNA/i+tPC865KG51yETunNLcrdG85X1xL3C7dX11rtvNOxfXoZ0btUZzuxQqvB17vjNtUlWmcprr/vG7oZ+670isu11nh9rXGPuJr22qz4qTW2jKkr600XkNba5Hmm+LjXur5FsNMa89o4XU39035W+VMmRseY9bcdeB5q3vU86FvvVKc5pPnnNLb1tXfUp6HnjdxfYz1VXidqO4P3TsuU9IcVphxeKU8d9XWaH+oPfo798FM6X+F6xyybwnvQ9V4CK8x8YMm4TVAbZLc7rimDkkjtNepHsc7jftfofLi+Gbl66/rU12POHdUZiwjK9vQKtN7kqQ2qx8tW7vCYxTXQ+G2KL6aGw6vmDqXfB2U3unUrjy3PY65zcLtzuMPAAA1s3duagPRgu/NQRuf/pa8iXoz80+ldR6jtC7D8ZL+dljclIU3FW083tRUf9x8VaeNFaWJZSpdH6pTeVxmbJv6LqKBUOF+K4/pGrdoVLncamNu1etw1eHyVa76H8vxuDg+tqca7xbKpzy5jQpzv/R7Hn/VEY0V1x2N64jjY7vG9kFjEtPH6yq6rrfk9nqMtzn3VHZVb8wTx0s43v1yPv10vnhdcpn+W8rXD44LzX3PC83dPNcinkOtOeNyIs6jsltUabx+duUTVZ3Gczwr0lWPD8R5PbDifR+pwjJdaVS24qo2eQ3qK994TdCa1XUY7apTuM587R3eRVeaIf2p0nQ5b7wu9s0d477nvm17bsQ8+SANy6Kaf7YJWtc27vVSnCcxPM9DpcvhffeN567yRuKHUKrTZUu2mxQWw/23FNvgMZBclsdAyuMwtf8VvpekIfdSazyMwhWf7/W4Buin73f9NEPSaGz1d4x3nQ7Ldmt1rVyGpL/j/MtjG9Prp9f/ofau2NZ5wWOj/BHP4y6bOucRU+dSvA62z2N69ddovJw2n0/c1ta5BQAALqdt5c8MbQxa4CtjxJuZpM0jH5r1tzcObSgxXptK3Kgi3gwl5Y91RyeWFDequKkNMYaE66o219iOitZmLrrGTbjcKr5VbwxXPyujI7YpXw8bMVV7K1p9cBuk6LCNhkIMd73xWhkbdzIkzNg+qF6Fqe6YXu1uza3WGOQxjn2fOvc8jsofUVt9D6jPEedxPe6XfjquOvQ7D0AkHlIlzal8XwvPrSpOOH/EeboOAU4T57nv8zz3M1WdQveC47IiXfU4rk95HXF4F11pqvEwcQ0aQlx7JJVd7X9ddQrnz9fe4V10pXF/8hhGqj7HdT3udbG/1R6ovmutVX8tp89928Xc8B4lqV61RX2BZaH9Vdcw3ku2Aaq9V3jOyJ6J11xrY7xH8zz0vRnDq3si4rqUN2KnTCxLbVGbc71d5bt+lZfvM+8n6lO0uab2vyLaR5L62ZWvNR7GY6x0EeeTVF/sj+lLo7/dv2ivCY2D+5HXCl+raC/Hfud1XOnVllh+TB9tUuFr2BqTOGaxTOF1rJU3E8uKuA2SxiheQ9v/Up5jLm/MXFIfHJftkbhvxDyuR3Eeg1ZfAACgTfdJYUZocc+bgak2hYg/edXmVKENS/FS3Ni8GarcvOGpHufJG7nwhqy2DcF1VZuY46SKrg2wa9yEy63iW/XG8Fa53ryr6yGqclu0+uAyqgO0jf84Jm53NU+q6zW2Dy6/Nc8iTltdMxHbuo25pzIUlo1a02qPx76qJxqEmVY4gNZbzU0fqKQ8v1r3vKnml/Pkw0TEaeK94fUzhlVUdY6hqx7H5fuvjyFt6kpTjYfxmtBXfkRrk9ZjH3Il/R7X0K46hfPla+/wLrrSuD+tNVC0+uw2S/o9/p3nrtbaOLcr5b7tYm4ItUXt814mVfslzJO418d7yPu5FB0uxte7utYxb56HntcxvHVPmNb8dJ6qfZmu8u3cba3rVZun9r+Fxt7tsLSOVG3qu1/d3nyvO5/aHq91pC/NlLOO55jKzNjGa/Ulo/Gu0nsOtcrx9Wr1W3HSEFrj7zaoLvc94j0rXxe3bcxc8nXIDxIYtzHX5TYon8tW/UPuIQAAeIxhu8UMsEFQGSOtjcI4byteVGn6NmTFSV1tauXNdNXlOKmiqy73q2qjcLlVfKverraKuOErTSXHt9oVcfqc1mVUtNroQ2c2VGzA2IiY2geXI0O4yyDpG8O++KpuU80HG1tqn9sfZaNKiiiuVY+o8ohWOEDE81KKB44p8855utZ5z3PVa3y/dOUTVZ1j6KrHcerDGIa0qStN15h5Deorv4Xyez0cM96uM197h3fRlWZIf3yYV7uN82msooNDcyn2y3iPqZ5Qa83rrjFx3Ni5EZHTwB96xb7BvPH6qLmUqdYyEW2XFo4fMj89/1vltean7xW1M9eT6Srfa4jKUR1Zjve9c5b+96H0vo+s/OFG3/2qcMXne935cnikL02r7EhOY4dntS6MdW56ruT0rXCx7fOCx0j5Il1tENXYTp1LnvvaC2I/LO8R+j0S63Oaao8BAIA27RV7ZmgT0EJfbW7VphRx3tYnv8JpYhl9m6HipK42tfJmuupynFTRVZf7VbVRuNwqvlVvV1tFzNclGVNDPpFs9cHlVLTaWL3OZQNOxrOZ2gcZJza2JdVffeI7dAxb8S4/j4mo5oPD+hTHQKgMhVf1COfLtMIBMtUcmzLvfG/nOWx8iJPiPet7Qz+7cN6pdNUTDzVD1kQzpE1daTzOVZu8BvWV30XVZ4dVDps4DvnaO7yLrjR+Qklq2QJ2XESHhedVay5mXEf1FFJrXlfjZKbOjQqXA8tAa5muV2VD2PEpJ0hkyH3r+DwPq/nZV57nrvJGNP/dfkntVJur+6KrfMf1yff0Wfo/FN2HdmBJ1fqWx8O01lzny+GRvjQue+xZx460OM/i9ascbBoDlaFxUJmS0+v3iK9JDhfxenVp6HnBYzSmDaIa26lzSXU4vEvVk53eb6SWLQMAAG3aK/bM8GYRNxBTbUoR58W5+XhcbhXfqrerraKvvWNp9aGrjlYbZRw5n41sG6nRsDtLH1SuyoqGvX6PRv3QMWzFu9w8JqKaD2Pno1H6Vj3C7ci0wuF40X2meRjvA92P/jBAThwzZd7Fezs/TaM434/ZqeZ7Qz+7cNld+0gXrqd1YHH79DMf4vS38ue63aYuutJ4nKu+ew3qK1/o8Ktxje3WdXb58XCsPrjceM1VX/xgKF97h3fRl8bOS9UT6xZ2FkkxzofNPKdauIzsKPD1l3Lftj035KzIDjGPu/oO8yc6tXUfZXlOSPk+cngLx+d5qHJzeF95nrvKW6F5ZxtLqu69rvIdl9va4iz9H4v7Fe+pvvHwGCtdxPlyeKQvjcvO+0Skqt/lSppXSuO1WH/HPVtEB1yl3HdfkxwuhlyvMbgvY9ogqrGdOpdURy5rKPleyWMPAADdbGc32QPeLOIGYqpNKTJko7GhGA9hfZuh4qSuNrXyZobWVdFVl/tetVF0xbtNud6+trbyTaXVxq46utroa63DnwwHlxONiG31QeXYSBwzt/ri3Taly1TzYex8NErfqke4HZlWOBwvdixJmleeW1J2OE6dd7qnHaf7LtdTHdR8b+hnF7n9lQOpCzmhnF95XYbRgd9rheRDptcrKa4hwuFddKXx2FR99xrUV76I4+52uy/qYx7z2KfYR/307/naO30XfWnUjli3fvd1cFi+prpu8bpYnl866EeHY37yJo5Fq2/bnhvuj9vYSgfzpc+BFBUd70PuW8fneai5UoV3lTfUttAc9zzM631X+Y7LbWpxlv6PparLYa3x8BjnNXfIPtSXplV2xNfA64DWRK0TWjM0j7zO6G/Nwbx2e61XOv0e175W37vGpBrDs+AxGtMGUY3tkLY5XmnNkOtQ4Q+gvAfo96EfqgEAwGNsZzfZA17o4wZiqk0pYiNRm3pFPFgM2aiN83S1qZU3M7Su/Im38Cd9Vd6ucROOrw48CnO9kb62ChtIYw7/LVp9qNpmutrofmk+2FDLxrbYVh+q+dk3hn3xipPymIhq7mneOE+c4320xt64zEwrHI4XHZI0N6OjRb/H+8JMnXdCeXQ/+/51PS2nju+XVrxR+6ODUweQMfeS0OHFjicp9111aL+KY6T06k+1Dile/ezC5VR476j67jWjr3yj9vm6SWq3xisfjoXH0tdIaTUWCneb8l43pC1D0qiOPA+VR+2p9leld1r9VB+lnD/OBZUfr7P6pGvvvaea19ucG2qL+hPLU5u7nuiCeeF7o8v+8HzKc97XvLre0RbI87C17jp9dX/4flXePlRuldblV7h8zeehuLyx/R+Lx1/3mXEfq3VI97DvyXx/6+8qPNKXZspZR2PUV2/EbVBdmdb1bYWbIXN9KG7f2Da0xlZh0pi5VM2LPjQ3PA4qa5vzFADgmFiM90EbUmuRb21KJm7o2UBSnA8LMqIifZuhy+xqUytvpq+uqo3aDNUft6PK63GrDBFhY0ibsMoz3pytSF9bhcvVZp3HR/XIiGldr4z7kMup2ma62qj6ndfjWhkuY/ugMdP1sdEolM7tV7xx+6Q47qar/cJ5c7tEa+65r/oZ2yj0t/LlcWiNvXE7Mg6vxhUAYClU67fR2u11deh+BtCHHU6VgywSbZnoGLJdqLkZ7QuVaweKlPf11n7vOT7U/pRjRnlyObapsq3tMip7wWMhVfeg4rNdP7X/FUqvezuWI5TXZcV7P16T2N541sh5hP6uwiN9aVSHy89jEuuP4+/xzWPVwm3I6TUeLj/bnopTuFTVsc3zgtvXakMON62xnTKXNNaOU/7cZ90fClc6o2vi9MZtyuczAABocxTOTaHNUWkkbToqz2VK1cbetxk6b1ebWnkzfXVFA08bndLFzbOVN/ZbfZTi5hk3YY+LP1l2uVKkr61CY2lDR3KbJYdlA7eF8+RxdjkVfW20ISGp3xVj+5DHWuk8tpVx4nH2uOunjZ2+9ruePCaiNfdkULk9ktsY+5gPD4pXeFWPcL5MnDsqQ32Nhy8AgCXQWuPMEPsDYAyVo6OF00ZbpMuu037vPT/v6639fqz9qfpzetcvyRaJ9NkLMd79iW3Qz8jU/le4HOdVWc4vVdfIzrqYR7+rLF+vvF4MWUeGpJly1on9iVI+9S+OUzW2zu+fCst4/J1HP23vbvO84DHKbVAfqnDTGtupcyleB8n9cV7J94HbpvLjtdHvTt91zQEA4C9ZjHPTBkFljHhTqj7VjSivyvFGJWljauXr2wxdTjbUhMpU3NANWWV01SVkYEYDQGWrjc7bqkvjEzdnlRNRfo+vpDpsWCq9FPG4DOmbxkF9ctkqS38rPBtYLZw/j7PLrOi7dtFYlyHaxZg+aNxiWhkmMg6rvqo/MW1s667mntqh/sZ5pDYqbTxMGLdP7alwGRnVEw8kqsOGLADAUvAaVq2PWhe9FlfxAFPwnKv294xtBSmi/VZ7sOen9mA7SGzv5fK79vux9qfiFOb69VN/V30aYi+o/liepPbKnqlsiyn9r1A5qsNjY6mMrntedalOpVUb1BaVZftMPyNDzjFD0og89lLXWcdjr/aqn5bzSvHcoHHzGDqfxkjXUXUqb0Z5YplVGrUvpnFZCq9s6AqPkfoU8X1S1SucTz8zU+eSwpTP80DSdVBYvMd8X1XzKZ5Vho4BAMAxsxjnJgAAAMA+0aHdh0sftqV4YM0HaYAlYsdSdLzAurFTT066Cq1tim85BQEAAOYEzk0AAACABnqiJj8Jpd8VxhObsBbssMe5eTz0ObT7nngEAACYEzg3AQAAAACOFL1Ca8c9r78eD3ZuVk+f63VsvzLN0+kAALAEcG4CAAAAABwZevJYDi4/lYwT67iI3+koaS5I8ftV9TsObwAAWAI4NwEAAAAAjgz9sxY7seTYxIl1fOjVc137+D3CkpycY/6ZDwAAwKHBuQkAAAAAAAAAAACLBOcmAAAAAAAAAAAALBKcmwAAAAAAAAAAALBIcG4CAAAAAAAAAADAIsG5CQAAAAAAAAAAAIsE5yYAAAAAAAAAAAAsEpybAAAAAAAAAAAAsEhwbgIAAAAAAAAAAMAiwbkJAAAAAAAAAAAAiwTnJgAAAAAAAAAAACwSnJsAAAAAAAAAAACwSHBuAgAAAAAAAAAAwCLBuQkAAAAAAAAAAACLBOcmAAAAAAAAAAAALBKcmwAAAAAAAAAAALBIcG4CAAAAAAAAAADAIsG5CQAAAAAAAAAAAIsE5yYAAEzm/INf2rzkPX+y+Ye/cM/mu675+Oavvvijm7/y/Ns3/9GPIoQq6f74lp/46Mn9ovvm6t9+6OQ+AgAAAACAaeDcBACA0bz4+gc3zzj3sc3Xv+j85okvPF86cRBCw/R1P3Z+8+1Xfmxz7rcfOr3DAAAAAABgKDg3AQBgMK+7+TOb//ilH9089cU4NBHatvRhwV9/yUdP7jMAAAAAABjGap2bvCqJ1ipeaYRD8LkvfmXzPRfn2zf8OE5NhHYt3We633TfAewabGaEdi/sd5glXzi/2dz9ks3mlu/ZbD7wtzebG5662fzWV13UE9AidfHa3fDNj11LXdN7rn7sGh8JF0dgXfCqJDpG8Uoj7JJbP/XFzXdcnF9PfOEd5fxDCO1GT//pj53cfwC7AJsZocMK+x0OxidevNm8/zs3m/c8ebO5/msLJxlaja5/0mbzvqdvNvecO7346+Vib9cBr0oixCuNsH3kWPnaK+7YPPni3KrmHEJot9L9h4MTtgk2M0LzEvY77I0/ft1m8+//xmbz3m+qHWFovZIj+4a/9tgcWCkXe7lseFUSoceLVxphG2j+6MkxHJsIHVZP+6k7Wc/hzGAzIzRvYb/Dzvjy5zabW/6rzea9T6kdX+h4pDmguaA5sTIu9m658KokQt3ilUY4C9/7uk+yviI0E+nACzAVbGaEliPsd9gqn791s3nft2827+b1cxT0vqc9NjdWxMVeLRMt+LwqiVC/eKURpqBXo/RPJao5hRDav570wjt4ZREmgc2M0PKE/Q5bQc6r679ms3nP19UOLnTcevfFubEiB+fFHi0PXpVEaJx4pRHGou9+quYSQuhw0n0JMAZsZoSWK+x3OBN67VhP5+HYRF367W9bzSvqF3uzPHhVEqHx4pVGGIr+gy7/aAKh+elrrriD/6oLo8BmRmjZwn6Hyfzuf82r6GiY9B2cK+BiT5YFr0oiNE280ghD+VtXfqycQwihw+vbL96fAEPAZkZo+cJ+h0noP2Lf8NTakYVQ1vVPXMV/Ub/Yk2XBq5IITRevNEIf5x/80ubrfoynNhGaq3R/6j4F6AObGaF1CPsdRnPDX6udWAi1pDmzcC72YjnwqiRCZxOvNEIfV1+cH9/w46yzCM1VT3zh+ZP7FKALbGaE1iPsdxjFJ1682bz3m2oHFkItvfurN5t7zp1OomVysRfLgVclETq7eKURuviHv3BPOW8QQvOR7lOALrCZEVqXsN9hMO/7jtp5hVCf3vf000m0TC72YBnwqiRC2xGvNEIX33XNx8t5gxCaj/7WVRxyoQ02M0LrE/Y7DOIL5zeb659UO64Q6pPmjubQQrnYg2XAq5IIbUe80ghd8M8nEJq/dJ8CtMBmRmh9wn6HQdxz9Wbz3qfUjiuE+nT91z42hxbKxR4sA16VRGh74pVGaPFXnl/PGYTQfKT7FKAFNjNC6xT2O/Ryy/fUTiuEhkpzaKFcbP0y4FVJhLYnXmmcD1989PSXmVDNF4TQ/ATQApsZoXUK+x16+cDfrh1WCA2VvrN1oVxs/TLgVUmEtideaZwP939gs/niI6d/zIBqviCE5ieAFtjMCK1T2O/Qyw1PrR1WCA2V5tBCudj6ZcCrkghtT7zSOB/k3JyTg7OaLwih+QmgBTYzQusU9jv08ltf9XhnFUKjdHEOLZSLrV8G1QKPEJoumAdybH7gVz81GwdnNVcQQvMTQItqviCE1iGATkpnFUIjtVAW0/JqcUcITRfMAzk1/483fmI2Ds5qriCE5ieAFtV8QQitQwCdVI4qhMZqoSym5dXijhCaLpgHdm7OxcFZzRWE0PwE82JO/xyumi8IoXUIoJPKUYXQWC2UxbS8WtwRQtMF8yA6N+fg4KzmCkJofoJ5MYcn7001XxBC6xBAJ5WjCqGxWiiLaXm1uCOEpgvmQXZuHtrBWc0VhND8BPNCa/ZcHJzVfEEIrUMwL+b01P4JlaMKobFaKItpebW4I4SmC+ZB5dw8pIOzmisIofkJ5oXWa747GSG0a8G8mMuHWpeoHFUIjdVCWUzLq8UdITRdMA9azk3pEAflaq4cg777VXdtPv/5z28eeORzZfwx6MIDnzkZg2e/8UIZj+YlmBdey+fg4KzmC0JoHYJ5ofX+0Gv+ZVSOKoTGaqEspuXV4o4Qmi6YBz4Qt7Tvg3I1V45Bz/3Ve08ce1IMf/pPfXTz6hs/faIYvka5/xqLKh7NSzAv4lp+aAdnNV8QQusQzAut9XN5av+EylGF0FgtlMW0vFrc0Tqkw/QxPy11KME8iAfilvZpNFVzZc6S09FOuTfd/ECZZohazs0Yvnan39h+3nLhkZP0V77n/jIe7VYwL/JafsjDbjVflqR33f7QydpyDB8qVTr2/qNuwbzw2j8bB2flqEJorBbKYlpeLe5zlV7p0yF7ia/2HeIJJR+oqzi0O8E8yAfilvZlNFVzZc7yq9SSfq/SDFHLual1XB++SHp1PcbtUm7PjR9/pIzfhdz/sc7NsziV0XTBvKjW8kMddqv5siS11hatTUu1r8eItRV1CeZFXPtn4eCsHFUIjdVCWUzLq8V9V5IxIaNCn5xW8ZYOu0qXjSwbJfoZw+eueLgferjdhlxnFVfJ6ffpaFijYB5UB+KW9mE0VXNlrtLaq7VADkA7OaeuXXH9q+L3Lbdnn/uI+z90DDmAH1YwL1pr+SEOu9V82bW8Hm9j3WqtLQ7f57p4CE1ZW7Vuay/02UQ/9ffUPRHNVzAv8tp/cAdn5ahaif7sj952sr5V+tNHLmz+w6dv3Pxfv//9ZV7rKx/6ByflfOHBW8r8X/rYlWW+L//uf7P5088+5iPST/1dpVuNFspiWl4t7ruSnZt9xpNvhmw4+HWSt932J5eFD5Xr3/cTlId6QsnjWMVVcnoMtrMJ5kHrQNzSro2maq7MVV5r9Vq01lv93vehVEtaT7y2VPH7ltuzz0O8+z90bZ1yAEfbE8yLrrV834fdar7sWl6PrbPYka215az29VS5Pft6YnTs2qrzgtJL+qBP+eNbDftq95KlcdL5p4qbm2BeVGv/QR2claNqJcoOyZb+wx+/q8z/Z/e8qUyf9YWH7tj8xb9/2mV55TSNafqcqIvXQllMy6vFfVc6q3PzrHL9x3Jg9DhWcZV2Ne7HJpgHXQfilnZpNFVzZa7yEyr6pz9+akhh+rtKL+nArQOy8+oAqINhl3PT5cawWF8Mt+RwVXz1WrnW9jvu+8uDp9L48BnbUSk7DFRPLEv9Uf9ajgWNjer3wVftl6NA4S5jV85NlRufLJLU9tYHed4Lla+6bq3v+sx9rLTvDw93IZgXfWv5Pg+71XzZtXRvSnZADl0XKo1dW3Ytt2fo2nhWjem/1kallfKaqPaqrH21e8nyGFZxcxPMi9bafzAHZ+WoWomGOjel7Hwc6ti0soMT5+YyWEzLq8V9V/KBSgZBFW95cm/baHD9czHqdi2PYxVXaVfjfmyCedB3IG5pV0ZTNVfmqMp5aGdWy+klB6KdY/qpNd55ooMw56vCtf600kutfcRPmDrO9ao9ilcbc7j+tqLj1k4ElyXF/tlhailv7KfS++8YPnRtVX6lH7JX+XpJGvNYt6TfY98kj2F0aipfdFpWdbtcj12sx3W/4B33PS7f0gTzYshavq/DbjVfdinf37pXdW/5XqvSDtGYtWUfcnv2ZXeO6b+f2tQ6V8WjYdIYSlXc3ATzomvtP4iDs3JUrUTRuanfHS4n5Bc/8epLcZJePXd8dkzqtXK9fm7npX7++fnnn7yaHtPJIdoqA+fmPFlMy6vFfVfygUrGRRVveXJnY8f585MhOrjJ8PPBTAcv/e0DnfNVqoxEGZM2gCwd9FuHNsUrveqLB+zoCNDfalfMZ6ncWJ/bn/uvv+NhVFK+rnZJVVwlpx9qZI4dJ6fV76pDaZ1Pca1685NFlbKz4ZD63L0bNAMNORC3tAujqZorc5Tvy7h+eV2LDs8o39uKj460vPbGPFIVrnWglV6q9hE/YaM1ItavsvLB1OXH/FGx/FiWpHWoyus82k/ik51aC+O61VrjsjyeKreKt+JY5X1Rca47v2oar4vGJ7bZ11p5Yx47WdTHOC4xPKZfsmBeDF3L93HYrebLLuX12HaV7+m+tUT3ZfzwQeXoPm+tLV4T8jri+iobK667OU7pVafza32IdbodlbTOxrJUT2X7xj0qa2z/K9m5OXRtU5+VvhoPyWul2hLDvY6rTbGNKsfjPiSNpLVZaWLflUbjV11DqRrfrJhXbamuR7b9Y/6sfO1Upue6pLL1d98837ZgXvSt/Xt3cFaOqpWo5dzsi9dr6g6XWt+XKSenv1dTkrPTcTg3l8FiWl4t7ruSNlxNWm2CVbzlyZ03NefXzxjuTVybocqWIaK/bUBoE43h+qm/pWxE6W+lkVRuzCfl9JLCXZ7zSbGdzh/zSVV9Nhj0u9NFp2nVrsqh6Lgc3pLTDzEmpoyTxycenmN/pVy3jC7H+7rFely30sV8h9Tv/eajaCaqjKGh2rbRVM2VuUkHI99b0YHlQ5uU77WuOMkHRCnHVeFaA1rppWofcR6tDTFtJaeN+aO83mTHpqU4KYY5T3XYjuMzZG2V1Dalz3tdlg+D1Xor+TCt9sVwj6HWz9zPOAdie71uZ+eHpHJy+iUL5sVQ56a068NuNV92Ja2nuq/i/Wvbq3XPS5V95nJsP+W1xWtCDnc51b3ttVSK4ZXd5r+9fmg9ieFupxTXGK2fuSyvN1I1DlP6Xymu3V3jbbXGw/IYqz0x3Pk0Ju6r0qitHvchabR2x7FRvOT0Ut6jqmvlMZI8ft7bvQ/EuJg+nkVcv+P8t+Q2S96npKrMal/dlWBeDFn79+rgrBxVK1Gvc/OhOy7Fx+/djA7L1vdxWvmfFukfECkc5+YyWEzLq8V9V2pt7Fme3HHzkyrjy5tiLlPh+dPRKn8Vr40+1x0P6F2fTrY2YcfHMJWjsKo+xcmI8N/qi4yr7DywESdDIIZLVZ1dcvrclqyp46Rr5Di124dq/XRcvmbuXytcP2P4HFRtvmiZ2qbRVM2Vucn3b77fJB+a4sFTcp7Wuq41QvFSjqvCu9JLXn9yfT6gqe15nYxy+VV742FW8ZUc77Uv5snlWTlPn1xPa6+yhpRbpfEYtsp3/TGP19x8/SXPjZjezossjVfMO0fBvBjj3JR2edit5suu5LU12jnRboxpLd2Dis/3o+4736dSvvdba0JVlhXriuFeK2JZsvP0dy5fa0KrfOXxup7tSfXHcTFvbFMM7+t/S+6LJDu7y9HWGg9LdSpOfY7hMZ/6VK2RQ9J4LNXPuAdqHN0P5Y1xDh9qYyudwrZ1FnG/lC/3yec7tdnnhV0L5sXQtX9vDs7KUbUStZybJ/8BPX2npl4zV5ye0ozhrf+GbrWcmDg3l8FiWl4t7ruSN3ZtVD7oVPLkjoZJzB+NEoflDbhSlT9K7VJ8y3jx5p2NAIV1lSs5TQxTOX35+iQDoypbaoW35PR53LOmjpOvbXWtbLBLMdzGaDZ6ZOhU6eegauNFy9W2jKZqrsxNvt+qe9sH7Xx48bqq+zuGW/FQluOq8K70Uqs+rRFemxxf9cPlV+2NdXdJ9fhw19deyfFKW8Vnea3s2xtcbtfBr6rbY9gq3/XHPD5oao7E+hyuMXGYFB0JUUPH4JCCeTHWuSnt6rBbzZddyfdQdux12WC2Kys7S/et1m/F53u/tSYoTKruW4U5PoZ7/ajal1WtNZb3nPhBf1TV5qn975LrsVRGlb81HpbLyXtPzFeNw5A02v8cnx2PlvseP6Dalo099Szi65XnuNU1P3YhmBdj1v69ODgrR9VKFJ2bXYpPZ451SrbSjy1n8Vooi2l5tbjvStlA6FPezJw/GhVxQ1d41yGvym9FwyHHWa00ffmkrnxn3bSrsrvCW3L6rvacZZz6jJQqj42xoYaX6qgU0+xa1aaLlq1tGE3VXJmT4lpa3UPRWRXvR6+rShPLs7rWjCq8b43pqk/rgg7Usa36Pe4LLr/K31d3pSF5HK+0VXyW2qb01V4V5XK79r2q7q69UHL9ub0uS84VpYnjPMSRsRTBvJji3JR2cdit5ssu5PU4f2gg+dXg/AGyZMdna61prS2tNUFhrfIU5vgYbqek2tK3LrTWGsmOL9mBSpdl+1C/O8/U/vdJzjuNu8uXWnuLFPNaHuPYXsn5cnhUX5pW2VFVmrE2dpda6bvK8XhqLNWuLMePvV5TBfNi7Nq/cwdn5ahaifqcm/qOzPhPgKSxTslW+rHlLF4LZTEtrxb3XWnI5it5cmfjxPnzJifjyXkkfWJbGTat/JINB22wOc5yGimGV2FZU/NFydiQ0aj+xc2/Vc7Y8p2+GjvrLOPktrbKr/KorwrLn8I7PBr30RjLinl3rWrDRcvXWY2maq7MST4wD1G8H/vW9dZ6II0Nl/rqs1SvD0bxSRW3p8rf1daWhuRxvNJW8VleK6u9Kqqv3LgmVg7pVvnVWu08uvbRqak1uPXUzVJV/YMydDhNdW5K2z7sVvNlF/J6nG0fKX4QlZ/Sc3hrTWitLa01oas8hTk+x9lGk7QOqz/VE4XVWpPj+qSyncdhY/s/VFpTPVaSynNc13hIzhfzSM6Xw6P60rjsOBZZVf2+TnmeObxyoG/zLOK4Pu1rj4F5MWXt36mDs3JUrURDntzMr51np6RfV29J8TE9zs1lsZiWV4v7rtTa2LM8ubNx4vz6GcMlGU0K96eQUt6su/LbcJirc1MO3PiJcaWcpxXektPncY86yzjZ+GmVX+WJZfnTe19jjUf+tHkOqjZbtA6dxWiq5sqc5PWl60mb+ESOw+I9qkNPTC85j5Tj+sKr+9tP8/TtI1K15ru9rfxDxiFL6aXqABYdEa21L8trZbVXRdnJ2DrQ+oM/rZkxvGsvlKq12nXFdGtV9c/J0GFVrcdDtc3DbjVfdiGvQ7rvdD9mKU6KH9xIDm+tNc6b7/3WmtBVntdSKcdJWvtkh7svUl5X3Z6q/FZbu+R6xvZ/rOK+5n2qbzw8xmpDDHe+HB7Vl8Zlj3VuxjYPsbG3fRZxXOt67VswL6Y4N6WdOTgrR9VKVH3npr5vM/839Ox4jHH6h0ExLiv/QyGH49xcBotpebW470qtjT3Lkztvds6vnzE8qzI6pK78cYPPcZYOrorXxh7D+/JJVRqH9W3q8bts1PZsbDguhnWFt+T0Xe05yzjpuneVX5XrPDLYosElgzmPw1xUbbRoPZpqNFVzZS5q3bNZ8UnAeEj1vdn6QMmKcVIr3I60/GS2yneeuI9ozVfa+GSQ0nv9iA6AuIZVzlitNYrTWOS1SunV77yHuF1qdyxT46pyXF/X2hrldld7VZSdl1J2Gqgu150dIL4urfKrtdrXJNezRlX3PVq2tnXYrebLtuX1eIjyBxcOb601rbWltSZUa4GlMNeX46Li2p33mK7yh66DUW7P2P5PUVWXw2I6y2OsNsRwj2MOj+pL0yo7yntb3Fc9HkNs7F2cRRzXul77VvXkODqcpjo3pZ04OCtH1UpUOTelv/j3Tzt5Jf1S3EN3XJ4v/Bd1/ed0pY/xlv750GX/Wf3TN16Kw7m5DBbT8mpx35WGbL6SJ3fe7JxfP2N4JW/YsYyu/DK+XK8Myxwv2TjTzxjufDEsq0rjw2Jff2zUKH0VX5XdFd6S03cZGWcZp+qaRLlc/+26+ubL3FRtskvVXX/4mDGsn1X8sWqK0VTNlbnIT0Pme7aS08YDUjyM64Cke9aONa8HUixHaoUPKS+uC9HJp3Uyplf+6HCU4iHOae0YVVqvzc6vNJLDYt8l5XV9LtN1qCyX17W2RrmuWHdU3DPi+Dp9bH91Tbv2QkllKD62t+VwcZ1yoOZxXqqqex4tX9s47FbzZdsash7LseR7MDqZfO9WT/Dp/vS6lO/91prg8vIHJJLCFCfluEpV2mqtsVy+2pzjWpra/7GK4x8/VHNYdvxJvq5qYwxX36vwqL40cX1urcPuuz+gUrq+eqPcBu0vVbzrHxouDZnr+1T11Dg6rKq1fKi27uCsHFUrUcu5Kel1dMdJ0fn4xU+8+rI4OUL1+rmdnPqp/NGxKcUycG4ug8W0vFrcdyUbT30bqSd3NnYq40sbo/6Om3k8aEYDw/lbG7MPicqbDZNoxOU4h8ewrCqND+SqL/dVf9s4s0GR26V+2jDIZUut8JacPrcla+o46bp3le98/tuGV1XPnFVtsH2yEzHqs5/53ObTf/To5rYbP1Xm2YcufPSxa62fVfwxa6zRVM2Vuchzbsh95vVIyuFxPdI66w9AdA9LMb1UlWMpb3TSqWzVoTb675he66nXGEkHOa1V1UFPZcS01Z6ktSymUfv1t8KrMrUeqz6lc/3eqzwuQ9exOI6V+vruNB7/rGovjXJZca1W2903xVs+MEutvXVpqu53tA6d9bBbzZdtynaP1LdeeH2MjqFoV8b8+j2up/neb60JftpP93lc96KtJ8U8SmsHmmXnW94HvNZUzshoy1drufqk8OhcnNr/SmpbLl/S3y5LaWKcw+MarXarHNed83hPzeFRQ9J4LVYb4ljF+uN11E+F5bFqyW3I6TUecc+KeSSHV/uR54WkOVXFq+05fFeq1gy0bG3VwVk5qlaiLufmydObwTmZ4+PTm0OUX1/HubkMFtPyanHflWw8dW3Okid3PFhJlfGVDQbJf+cDoDZgx2mDV1r9dLw2+mj86HencVg22CTH5fCoVpqu+qIRGNPFfiqN08dyJafP4S05vduRZcNk6jgpjeLydbWcN4bZsM5Snbq+VT2HVrW59slOxD/WOF78XdLv7u+dH3mgzLdr4dzs1hijqZorxy7P7yoOzUda87XX6FpVh2CFOb61vi9J1b2O1qOzHHar+bJN2WkYbdOWnDbailK2F/230tkJlZ17lX0tyW72va2f0daL9nfME23YWL+UHVh2RkpKJ0VnVoyXVF5sg5TXpCn9rxQddqrPdTtM5dpRaEVnnfN4/DxeCot5hjguh6SJ67DTxvr1M4/VWBs7j62k31W2r0lML+VzmtLFcmO8yqnaHcvbpar1Ai1fW3NwVo6qlajLuSnpP6U7XooOSDk/Y/4u5f+4LuHcXAaLaXm1uO9KNp60YVbxlje0vAk7fzSOZFgoPG64+r1luMjwiEZRTleVp/Zo883tsZymirO60uT61D7VFw+JapfCPDaxTTbAYplSV52VXHZLfeMe2xTLtWwEjXFu2rBV2TZ4otEjVZ/4H1LVxtonOxHzU5r6W+F6ivO91124LG4fwrnZr6FGUzVXjlnxw6YqHs1HQw7Wfev7klTd52hdmnrYrebLNuX7KDsBK8kOsy2U7UXZZ7Z1bZsp3A7RXL7SK1w/Y7gU7UxJdp+dUypbium1tqu+aGurX/6APEt1uh/6mdOp/lye2qCwar2Z0v9KTp9tTtWtcMVX+dT+aBtr7NRO9cN/x/ReX7vORkPSSNXY63fZyVV7x9rYKkPlO95j23UWcR6Xp/aonTGNxkz5Y71qh+rOaXepaq1A69BWHJyVo2oliv84SL/n+Pz0pl5Hz2n0+rm+SzN+R6ekJzv1tKb+QVHOI8Xv49RP/V2lW40WymJaXi3uCM1BNuZkCHUZZVKOO6SqTbVPLeem9NADn23G7Vo4N4dpiNFUzZVjlg+ZOgRW8Wg+8lqsg2f14ZWvpbTPg+iuVN3jaH2actit5ssa1OXcROvUUm3sXapaJ9B6dGYHZ+WoQmisFspiWl4t7gjNQUOM7TkaXtWG2qcu5+YD9z/6uDj9re/jvPn6P9488tBjzk+9xu54PeWpMh0nyUna+sdASn/vXQ+fPCGqtMp3+02fLp2bfpq0cnh2/QOi3B61X3XENFW73c+YTtKr+nb8Svo9l7dP9RlN1Vw5RulJDD2R4evmJ4DQvBWfRNJhuHrCZy3Xsrq/0To19rBbzZc1yE/W4dw8Hi3Vxt6lqjUCrUtncnBWjiqExmqhLKbl1eKO0Bxkw0uH6vypcnzNRQftGHdoVZtpn+xE7HpyMzr49LccgHJGSnKASoqTg9B5FKay5bi0wzA6Qa2c3g5V51GY03Y5N92PHOfyqvY4TdVuf++o+hj7byeqylA6SXlzvftWl9FUzZVjlF9dk1OMg/RypDXXr2jq+llaf7UW6ymgKt8SVd3bS1fXB0/HrjGH3Wq+LF26t3Ufa360Xh1H69NSbexdqlof0Po02cFZOaoQGquFspiWV4s7QnOQXm+MTwXJAPPTQg5rvSZ5SFUbaZ/klFN/snNTTycqXE68GO7+y6Enp2CMk+NQcdmJqXR2KMZ6XIccijG9w6XoNBzr3HQ5egIzplV7FOe/3e58+HZ9Mb+dpbnvc1DLaKrmCkJofqru66GyEzFKH85o/crr+z5Vrc3oLzX0sFvNl6VKtpPsKTs2j8mJhZZrY+9S1dqwdOnBAF3LbIMfuyY5OCtHFUJjtVAW0/JqcUdoLpLxpddY4yuRkv7e9xeND1W1ifbJB085JPW7ZAdefmpRcvgH3n7vZeGSwhVfOf4qR6N+V1j1SrcdjvFAPNa56X5Ur5ZHqd1SFZef8hxa5qFUGU3VXEEIzU/VPT1UXgPjWu4n0KX4gc4+Va3N6HINOexW82WpktPK81IOrWNyYqHHtEQbe5eq1oWhmusHW7bZ8wMMaIKDs3JUITRWC2UxLa8Wd4TQdFUbaJ988IySQ0+H4sqB2TJU/Alty4ip4rPjMKo6EI91burvVvmW26W2KG9WbqMcsfpbhqPiqzE6tD7ym49cZjRVcwUhND9V9/NQaT3S2pQPs143tWYd4olzt0s/q3j0mPoOu9V8QQitQ9WaMFReY+f2wRbOzW6NcnBWjiqExmqhLKbl1eKOEJquavPsk42ifCBuqWWoDDFicrz+lmIay+3ST4e5jhhmVem7yrdcZpfyU51ycNrpKcmInMtr6jy5idByVd3TQ+U1sFrL/Z3CQ9f5bapam1GtrsNuNV8QQutQtR4MVWvtt317qA+2XD/OzbYGOzgrRxVCY7VQFtPyanFHCE1XtXH2qetAXKllqPQZMXrCMcfP6cnNKcaX2qN8U/NvWy0jqZorCKH5qbqvh8prYLWWe52Kcfpbry1qDfRarA9qYj498WPHqKR0+sqQ6qCsMMXpIO20+iCoWpu71vKuf0Ck9PGDJbU/f62J2lGlUz9jOknt8NejSMpTtWmfYh1H6PhUrQVDpTVL61e19h/ygy2v83Owj+esQQ7OylGF0FgtlMW0vFrcEULTVW2afeoyiip1GSqKy085Wv7OzXh4rg7cVtd3buYDuFSld/nVoTZKaVrtHiIfoqu4fYknfhBavqp7e6iGHHDjWqi/tXZp7ZO0Xsa13a812uEnuRz9zA5Ox6kMpfX66/Uxrs1ey2OYpbAqzuW5fK35ee1Vm3I73A/1MfZfvztcZSmtHJ3K5zSHUrWeV/MFIbQOVevAUHnNrNZ+r5sxTn9v84MtSR9GuSytqWqT1/m8plZhktdktS3HqTyX7zRr+mCr18FZOaoQGquFspiWV4s7Qmi6qg2zT11GUSWlbR3+bChkA0EGh42NaGT4CZ1syNgRmsvy058qKxpYrfSt8h3n3334rZ4UUp3ZgMpy36q4fajPKKrmCkJofqru76FqreVeH7VOxXCFSZWjUmteK65a511H3htaa7MPvTHMcj+q8vNarrYpzn/7Q668lru+mN/19K3vh1Je16v5ghBah6o1YKi8llV2vB2U0fbW39oPtvXBltddp4/59TPvC1WY5HU6x+lvh6v8tX6w1WnLV44qhMZqoSym5dXijhCarmqz7JM2dW3ylVFUyYZDFSdHoIyFaFzIcHBYPnBWxoh+6m8bJQqLeXywthHVlb4qvzKKYruVXukk1xUPxEqnv53G5atcp9mnej/tvUg1VxBC81N1jw+V1iOtRVoLvT55fdS6lZ9ecbjWvxguee2rHH/+kEl5HdaVXmuj4tQeh411brofuQ9ZalNsV1Re911P9aHWXHTpn8M9zDqO0JpV3f9D5bUs2/H7+GAr7gcxvdZqhSlO67fDpSpMqpybx/bBVtOmrxxVCI3VQllMy6vFHSE0XdVG2aeWUdSS0lZGiSVDR4drGzU2KlpGhAwUOyZdttpig6RyiMbyZYCNSa+fVXuczgdgl63yFOd0ypvTRENvnxri2BTVXEEIzU/VfT5UXsujtFZpXascmIpvreVeL6s4Kcdnx2GU2xXXSa/X1dpZpdffrfItP42jtihvVm6jxsT90Bj1OU73LZ7cROh4VK0BQ6X1zeuY1zut7QrTGrfLD7ZkIyus+oDfjtK8z1RhkveFGOd+HNMHW6VtXzmqEBqrhbKYlleLO0JouqpNEq1TQx2boporCKH5qbrXh8oHNh0Qq/isfIjMcfEwmOUDp//uSu926afDduHcdJldyodfHZjdF0m/z8HJWa3v1XxBCK1D1TowVF4zo/b1wZbq0N/VvlM5K6VW/VV6/S3FdFlas5VmLR9sSfGp/RMqRxVCY7VQFtPyanFHCE1XtUGi9WmMY1NUcwUdXg888phxnXXHfZ/ZvOnmBzZP/6mPlvnQelXd70OlQ5zmT3XIrKS0rQOu52IVJ+XDYv47yu3ST4f5EBvDrCq9/m6Vb/mA2+pTl3TY9SFdh974tP6+1Vrfq/mC1qvvftVdj83ni/tEFY/WpWotGCqvmftY+5Uvxvvvqu7KWSlVYVKVXn9LMV2W83VpKR9sSTy5iXamhbKYlleLO0JouqpNEq1LYx2boporc9dzf/Wx159u/PgjZfwaZKNazsxbLjxyogsPfOaycBycx6Xqnh+qbR5wfeirDnt+NVEOTYc5fVV313duyqEY00pV+q72RClNPsSOUddTSPtQ1/pezRe0XnkPlKp4tC5V68FQ7dO5mT/ImtOTm60+dWlOH2xJzT2gclQhNFYLZTEtrxZ3tBvJQeANIkoH6Xfd/tDJJ8RVPrQsVRslWo+mODZFNVfmLh/s5PCr4tcgr8PqawzX336qU+tzjEPrVnXfD9U2D7j+HrUq3t/JFr+vzOkVF9P6n0FI0VkZHaTxMNlK3yrfcf7dh9QYZqnOvn8gcUjnZt/6Xs0XtF/piXrNj32syzg3j0vVmjBU21z7Fa74oR9sue64Xltd37kZy7Cq9F3tiVKaJX+wJXXuAZWjakf6v37/+zd/+tm/3IulLzx4y+Yv/v3TyvTWlz525ePyffETry7TogNpoSym5dXifijpBlzzqx9yEKiPcmb6CSE9FeTFR31/9hsvlHnRclRtlmgdmurYFNVcmbuO2bkpve22x5w5WrNzHFqvqnt/qLZ5wJX0z9KURodQlS35qZ3sZJSD0ulVptL6UOpDYz782knq8rvSV+XrCU+3x+l0+NYBV2H+Z2+S64rtVjkuK6ZRPqfZl4as79V8QfuVnZv72Jdwbh6XqnVhqLR+aZ4c4oMtPzWptTh+UKVwr8W5LK/b0WHZSn8MH2xJvXtA5ajakf7snjedjEWWwqv0khyfVR45Rav06EBaKItpebW4H0q+Cau4NcjOTRlmMVxPbNrJySF6+ao2TLR8ncWxKaq5MlfFA12l/JT5le+5/7IParqeRvc6qN9ffeOny6cjYxq1JZatJ+D9IZBeF1c+l6F6X/CO+y6VM0Qut3Ju+hBdfeimuhWvOl2G2qmw+Bq7y+h6td/9i20YWr6Vxyy+KaC4qn8af8W32qaxVbzSVfFjrvuSVN3/Q7XNA66kg6rK9EFUkuOvOjw6vQ+ILlttkfR3zuf00Rk5Jr1+6sCbD61OV7VbcU6nvincaZRe+WKafWjo+l7NF7RfeU3VulbFb1NaNz03q3i0LlVrw1Bte+33uqg1UWXHfaByMnrdd3r/7Z+5Lq3FCtcarg+putJrPXZ7FK7y1/TBljRoD6gcVTtSy7n5p49cKNNLemqzyoNzc2ZaKItpebW4H0q+Cau4NciHTxlmOU4Hcvef19OXrWrTRMvWWR2boporc5Wch1qv7LiSc09/W9G5ZgeY5Hg7G/UzP42ueMX5gCqHmPPlNH5y0vXHcnXo9N+Ki07AMQ5O56mcf+5bdv6p/9XYuCzFeYzUf4dXTknHqxyHjSnfcnxrzKTcxz4ngcus9qyx131JqtYAtF6NWd+r+YL2K5ybaFeq1oehkoNO82Rbzk05FFWmHYiSHH+tD7YkpbdzUfn0FSMK19+VQzSWr58uu0rvD6zW9sGWNHgPqBxVO1J0bn7hoTsu/S7plfUqT0x32e84N+elhbKYlleL+77lm6+SngyJaWVoxCdSdIjS39XB1EaJDCE5DH1QiwevmEaHRR0MfUDTYTke6nRg9oFT0uGuOqy21HVQjAZUdSgc8oSMy1D7Y94oH3xzG5R36Lgqr9IoTvWrHXHM8jWznKbqn8pxvTlOGtO+Q6vaONFytQ3Hpqjmytyl+0v3W+sQ6bVA8XkttPMr5/U6KClNjGulcdlaJ7yOSPo9rieuU+uQw/rksuJaonrct1yH5PZpPY3haqfX6bjG2vFarY1ek+NYjC1fao2ZfjpOa2bME69fDLecL9cV87key9egVeYSVK0DaJ0au75X8wXtV611y+FjbVOpSq8n1r0HSlU+xQ+xTV1uXksltzuvz2j/qtYItG6N2gMqR9WOdJlz88FbNv/hj9916W/9ntN/+Xf/m0vxeroz58/p9Qr7n/3R207S5nzV93rKoap6s6NVZeuJ0ZxeUrjiY/kqI5Yfnzb9D5++8bL8ksIc/+fnn38pXG0/KfOzDzz298V2+7tG9bvTSfrO0dhupVM7vvKhf3BZur1poSym5dXivm/JQPEhSvLfUjQQZJg4jQ54ivehUcqGi42S6LB0HpfrNDIqXJbKjo5EGR4+sCmNynCc0sU6u+R8lXHjvqmdOc51SypDcn/0Mx68Hd56csnxMc/YcbUhFg3BnKfqo+Oy0Sf5Okg5bmz7Dq1q80TL1LYcm6KaK3OX70vdc1W87//s4LJ8j8YwldVVpuQ0WmNynJ2Bee2T/CFJrrNLTl9J7ch16G/Ftdb+asy8ZlZ99lrmdXFK+VLXmMU3A2J4V7skl5nX8ynXfUmq1gK0Pk1Z36v5gvar1rrl8LG2qdZcp9fPmEfrsPPmfGNsU6/BKl/7lMO9Z+VwdBhV6wRar0bvAZWjakfKzkk59vy3nHPZAWlnn6Tfu5ybcoTmfzoUJSdkdPzJsVmli1KdsY7YnqzonO1qpxSdo9FpGcNVnn+XXI7GKDtjozQG0WG6Ny2UxbS8WtwPJU+2Ks4HOhkO+bBpA0PGQTxsOY/jcr4qjQ+Ykg/RVjw0RmNoqHOtOiiqvcrfKisacUOekHFY1wFXY+iwKePqNkky6KJBFh0PDrOcJ46xFa9DFT6mfYfW7/3mo2gmqoyZodqmY1NUc2Xu8v0X1xhL96PvWcVXcny85x1erQNWV5q4JuY4qaqzS06fD6daV6q13d9TqXilz6oOw9HpGtdLj2Fck6eULylOYa1+V3n6xtJlxj1r6nVfkqr1AK1LU9f3ar6g/aq1bk21Tb1m6SGHlq0rxTxa2xQ2xjZ1PbHdDotrLDqcqrUCrVOT9oDKUbUjVU6/6JDMT0vGODkmW05DOfzi05otxTyxrC7Z4ar6q3irVXYMtxTm+JZzM8tPgManPluqHMU710JZTMurxf1Q8kSr4vzaR+uJRBsI8TBl4yOHRzmNjJBsoMgocf7KWWhH4lCjxG2sJAOp6pvapfiW8875/XfsT85TtXfKuNrgk/GY64hjlse8FS653VIMn9K+Q+tz927QDCSjpTJohmjbjk1RzZW5y/el7rNWXJ+0FsVD5pB7tiuN15+qTZLrHbomVOl9OJXy2uP6+6T1MeazUzL+cx4fuPXTYVPL7xtX54thfWPpMpXOYSrfZXUpX/clqVoT0Hp0lvW9mi9ov2qtWw4fY5vGD2uq9cofNkkxfIptqvJt0yufy85rOTqcqvUCrU+T94DKUbUjVU6/+DRkfIU7vtqtJxVb+XPa/OSiXt92nGSnn57c9CvreurT6fMTnf4u0BiufE4vp6eesoxt35Zz06+7S6onO1jVt8v6E5zBe396c6EspuXV4n4oeZJVcTYIZATIaMhyfHUIU3wsK6ovjeKkaKBYfQfDLKVTejky9bsPu1I83FpTn5Dxk0f5qSOPUTTgpoyr+x3DopRP8XnMFFaFS74OUgyf0r5DC+bBVOfmLhyboporc5fvS91rrbh8z/ZJZSlPtQ5YXWn61l23qav8qFZ61xOfqozhrfpbssM0HmSrNXlq+X3jqjgphvXV5TKVzmEqvyprTarWBbQOnXV9r+YL2q9a65bD43oVVa2RdjC21sDWejfVNo1P5jtNfrACHU7VmoHWpTPtAZWjakeqnH7xezUlOfAUHp9Q9BOdLadhTJtfJZfia9x2VnbJaWP67FhU/flJU2sbzs3oLLWio9YO36joKI7l7kULZTEtrxb3Q8mTrCuuT/FTVBslMjRiWVF9aVxuNIasloHVktIpfTR2ZNTYwMkOzmhUdUn54+HYTwPpk2WHaVwUFg/WUiynS3FcpxiQksuqxjL2NYY7rE+tT88PIZgHU5ybu3JsimquzF2+L6s1rnXP9qm1PkR1pelbd92mrvKjWunjkz7V+teqv6VYntbr1po8tfy+cXXdMUxpu+pymXGtd55c1ppUrQ1o+drG+l7NF7RftdZIh8f1KqpaI/vW29Z657A+Vbap1nzHVw82oMOpWjfQenTmPaByVO1ILadffKVcDrzsSPTTia380SnYp+jcVD1yCHblj+mr79zU05Jql9t4kq7RTivWF52QrXArltunKv9OtVAW0/JqcT+UPMm64loHt0p9BzepL01XvX1GUZYNq2x4ReMpOiljeEzfJ5XhfH41x6/Q5Kc5na7qX0tTDEipq65WX7vyzFUwD8Y6N3fp2BTVXJm7fF+21jh/MJPXlS611oeorjR9667i+sqP6krvr/KIHxT1vcbYJa/DOtC67PiaujS1/L5xdZkxTGkVpusYwyXtHX4LIK/1U677klStD2jZ2tb6Xs0XtF+19oAptmnffuI1UorhDmutty3FdVVq1YsOo2rtQOvQVvaAylG1I7WcfvGJRDk6499D/lFPdAp2SY5Iv4Ie/5lRl/KTnvq7qi+2p9VOK+aPTshWuBXL7VPrqdKdaaEspuXV4n4oeZJ1xY0xJGyUdBkPfWm66u0zirKUTukrw8uf5MZPcd02KaYdIpenw2d8YsjOTsvhY8Z1igHZFS61+uqwMe07tGAejHFu7tqxKaq5MnfF+zKvHZKfEpezK9+jSq/1J68TXeuA1ZWmb91VXF/5UV3pFeb46Gj0+qqf2QGpv9XG6okdP62pw60dhNW4Tim/b1wVJ8WwuDdEJ6vqcBukfA2nXPclqfrnZOiwqtbtodrm+l7NF7RftfYAh7fWnmqNbJVlxT0ghjustd625LVTH3St/UOiJapaP9DytbU9oHJU7Ugtp19+UjN+d+RlT0428l/mFCxeS6+UnxaNT146XMrOTevkqc/kbLTj9LJ2Fq+Px9fklfZS+AjnZlXuQbVQFtPyanE/lDwJqzg/8VL9Y5+WbJS0jBapL43bVBkwfUZRlg2ryvCScaM4GTsxfKrxE7+o3GXHp4+sKeM6xYCM4flJJcntlWL4lPYdWjAPhjo39+HYFNVcWYL8lInWIt3D+mmHmxxZ0QmmtErje13K605rfYjqStO37rrervKj+tK7//GDp/h1IpLGQO2JY1Gtc1LMV63J0pTy+8bV+XK4D9uS69Hvqt/rb17rp1z3Jan6B2XocBq6llfa9vpezRe0X7X2gCm2qX73mlV90LRN29R1aW1VXS7bf1d50H5VrSFo2drqHlA5qnaky5xzwTkpxe/NtOI/7pFa+WO4HKN6ajE6K/WUpp4AjXmcXrIDU3liWTFOP9WeWLZ+Vt/nmZ8KlfNU4XKI5n6qPsVJfc5NlZ/zqkzHy7kq567GIObbixbKYlpeLe6Hkidg1xMvUnVoVHw2MmxIZAMoqi+N64zGkNV3yM6yYVUZXjJsfJiNjsypT8goTvkkH0K3Na5TDEjJfdFBOBpy0XiUYp4p7Tu0YB4MORDvy7EpqrmyBMnR5ntaqtY73ZsxjdYr/a3wfGhzOpUbw6Ocplp3vV60nGdeR7vKj1JaKT8habk+raMxXP3SmpadfGpX14dR0ZlYrcnW2PK7xkxyGVWc1nI7cTV+WlM1Hu67flb5FD70ui9JMC+mOjd3sb5X8wXtV7ZBtdZU4WNtU699LVvXinFjbVOth64nrt9u05I/DFqTqqfG0WFVre1DtfU9oHJU7UjRcZidm3IaOs7KDr5Wfjn44tOeLcU8Q9JL0blZxUfZ0aifVXyl2Mc+5+ZJmuBM7VKVd6daKItpebW4H0oyBDzRtOHLEIhGQIz3IUryYVY/Y3kyYFxWDI/qS+P6qgNjy8BqSemUvmV4uX/xEC2DKB9uVY7LklpGkcKdJo9N1NhxnWpA6rAcy1S6aFTqpxTzSGPbd2jBPOg7EO/TsSmquYIQmp9gXkxxbu5qfa/mC9qvWrb3VNs0OiptY9u+3JZt2mqzPoRzGdU5A+1X1ZPj6HCasvZbO9kDKkfVjtTl3NRTkNnhGJ9KlLryV87RrPgfyGNZWbEdQ52b+XX46p8PSSpbT5H677HOTT2d2eeYVXyVd6daKItpebW4H0py5EVjQUZGfppGRoicdjYeJBkLesIlp7XjsusTUafJBofleqqngPxUy9BPXFWH0ree6olGTn7qZcoTMtFg0/hUaawx4zrVgJTUx+h0lePW46G6pZxHGtO+QwvmQZdRtG/HpqjmCkJofoJ5MfaAu8v1vZovaL+yDarzQhU+xTZVWLZNZXcq7qy2qWx0hStNdZZQWteZ49B+BfNi7Npv7WwPqBxVO1J0QEZHoxUdglV86x8NWXKGKjx+n6YcfXIa5u/VlNSe+CSkfpdTMbbT36N5KX1wQJ44Ki+2U6+hx3ItleW22KmpNqpMlxHzxv77VfZKfn0+P8WptqmM7BTeixbKYlpeLe4IoemCedAyig7h2BTVXEEIzU8wL8YccHe9vlfzBSG0DsG8GLP2WzvdAypHFUJjtVAW0/JqcUcITRfMg8ooOpRjU1RzBSE0P8G8GHrA3cf6Xs0XhNA6BPNi6Npv7XwPqBxVCI3VQllMy6vFHSE0XTAPslF0SMemqOYKQmh+gnkx5IC7r/W9mi8IoXUI5sWQtd/ayx5QOaoQGquFspiWV4s7Qmi6YB5Eo+jQjk1RzRWE0PwE86LvgLvP9b2aLwihdQjmRd/ab+1tD6gcVQiN1UJZTMurxR0hNF0wD2wUzcGxKaq5ghCan2BedB1w972+V/MFIbQOwbzoWvutve4BlaMKobFaKItpebW4I4SmC+aBjJ25ODZFNVcQQvMTzIvWAfcQ63s1XxBC6xDMi9bab+19D6gcVQiN1UJZTMurxR0hNF0wD2Tw7Pvg20U1VxBC8xPMi+qAe6gPrqr5ghBah2BeVGu/dZA9oHJUITRWC2UxLa8Wd4TQdME8mJNjU1RzBSE0P8G8yAfcQz6RX80XhNA6BPOi5dw82B5QOaoQGquFspiWV4s7Qmi6YB588dHTX2bCX3l+PV8QQvOR7lOYF/GAe0jHpqjmDEJoHYJ5UTk3D7oHVI4qhMZqoSym5dXijhCaLoCKb/mJj5bzBSE0H33Tv/7o6R0Lc8EH3EM7NkU1ZxBC6xDMi+zcPPgeUDmqEBqrhbKYlleLO0JougAqvuuaj5fzBSE0Hz3j3MdO71iYCzrMzsGxKao5gxBah2BeROfmLPaAylGF0FgtlMW0nFclEdqeeKURWvzjN95bzhmE0Hz0D37u7tM7FuaCDrRzcGwKbGaE1ins9/lh5+ZcPtza/NZXPd5RhdAoXZxDC+Vi65cBr0oitD3xSiO0uPq3H9o88YXny3mDEDq8dH++5D1/cnrHwlyYi2NTYDMjtE5hv88Prf2zcWyKG765cFYhNELv/cbTybQ8LrZ+GfCqJELbE680QovzD35p8/UvwrmJ0Fz1dT92/uQ+hXkxp38Oh82M0DqF/T4/5NSc04dbmw/87dphhdBQvf87TyfT8rjY+mXAq5IIbU+80ghd/K0rP1bOG4TQ4aX7E6ALbGaE1ins9/kxK8em+Mg/qh1WCA3V7/z908m0PC62fhnwqiRC2xGvNEIf5y6utzy9idD89DVX3LF58fUPnt6pADXYzAitT9jv82ROT+2fcM/Vm831X1s7rRDqk+bO3S85nUzL42IPlgGvSiK0HfFKIwzhr7+E72xDaG76qy/m+9agH2xmhNYn7HcYxBfObzbveXLtuEKoT9c/6bE5tFAu9mA58KokQmcXrzTCEF5382c23/DjHI4Rmoue9MI7Nq/84MOndyhAN9jMCK1L2O8wmPd9R+24QqhPmjsL5mIPlgOvSiJ0NvFKI4zhe1/3yXIeIYT2qye+8I7Nf/nzF07vTIB+sJkRWo+w32EU95zj6U00Xu/+6s3mEy8+nUTL5GIvlgWvSiI0XbzSCGP43Be/svnOc/zXXYQOrb/5k3ee3I8AY8BmRmgdwn6H0dzw12oHFkIt3fDU08mzXC72YlnwqiRC08QrjTCFWz/1xc3XXnFHOacQQruXntjRfQgwFmxmhJYv7HeYxB+/brN571NqJxZCWdc/cbO595Wnk2e5XOzJ8uBVSYTGiVca4SzIsfIdfH8bQnuV1u2n/dSdODbhTGAzI7RcYb/Dmfjd/7p2ZCEU9e6v3Wxu+r+fTpplc7E3y4NXJREaJ15phLOi+fM9v3DPyRME1RxDCG1PetpOB1rWbTgr2MwILVfY73Amvvy5zeb9/0nt0ELI+u1vfWyurICLvVkmvCqJ0DDxSiNsE73m+C0/8VH+UQVCO5DWa91fvIII2wSbGaHlCfsdtsLnb91srv+a2qmFkP6JkObISrjYo+WiBZ9XJRGqxSuNsEv0n3i//eL6q3lWzT+E0DA98YXnT+6jb/upj/HfcGFnYDMjtAxhv8PWkfPqfd9eO7fQcUqvor/vaatybIqLPVs2vCqJ0OPFK42wL84/+KXN1b/90OYf/Nzdm2ec+9jmm/71Rzd/5fn1vEQI3X5yf+g/3/6tqz62+YcX7ZeXvOdPTu4jgF2DzYzQvIX9DjtDrx3f8l899o9jKmcXOh7pH03pOzZX8ip65GLv1gGvSiLEK40AAADQDTYzQvMS9jvsDf0X9Ru+ebN5z5Nrxxdar/QKuq79Cv4reouLvVwXvCqJjk280ggAAABjwWZeh/4fP3NvGY7mLex3OCj3nNts3vf0zeb6r60dYWgd0vWVfvtvbjafePHpxV8vF3u8TnhVEq1VvNIIAAAA2wKbebmSY/P+D2xwcC5A2O8wS75wfrO55+rN5nf+/mbz/u/cbN77jZvNb33V451kaCG6eO1ueOpm877v2Gxu+Z7N5u6XPHaNj4SLIwAAAAAAAABL4YuPbE4cmx/5zUdOfupvAACAYwXnJgAAAAAAwEKwY/MDv/qpzf/xxk+c/MTBCQAAxwzOTQAAgBXDYRcAYD1kx6aFgxMAAI4ZnJsAAAArxYdgDrsAAMun5djEwQkAAMcOzk0AAIAV4kMw38cGALB8+hybFg5OAAA4RnBuwmAwkgAAlkE+BHPYBQBYLkMdmxZrPgAAHBs4N2EQNqowkgAA5k3rEMxhFwBgebTW9D6x5gMAwDGBcxN6sVHFq40AAPOm7xDMYRcAYDn0rel9Ys0HAIBjAecmdJKNKowkAIB5MvQQzDoOADB/hq7pfWLNBwCAYwDnJjRpGVUYSQAA82LsIZh1HABgvoxd0/vEmg8AAGsH5yaU9BlVGEkAAPNg6iGYdRwAYH5MXdP7xJoPAABrBucmPI6hRhVGEgDAYTnrIZh1HABgPpx1Te8Taz4AAKwVnJtwGWONKowkAIDDsK1DMOs4AMDh2daa3ifWfAAAWCM4N+ESU40qjCQAgP2y7UMw6zgAwOHY9preJ9Z8AABYGzg34YSzGlUYSQAA+2FXh2DWcQCA/bOrNb1PrPkAALAmcG7C1owqjCQAgN2y60Mw6zgAwP7Y9ZreJ9Z8AABYCzg3j5xtG1UYSQAAu2Ffh2DWcQCA3bOvNb1PrPkAALAGcG4eMbsyqjCSAAC2y74PwazjAAC7Y99rep9Y8wEAYOng3DxSdm1UYSQBAGyHQx2CWccBALbPodb0PrHmAwDAksG5eYTsy6jCSAIAOBuHPgSzjgMAbI9Dr+l9Ys0HAIClgnPzyNi3UYWRBAAwjbkcglnHAQDOzlzW9D6x5gMAwBLBuXlEHMqowkgCABjH3A7BrOMAANOZ25reJ9Z8AABYGjg3j4RDG1UYSQAAw5jrIZh1HABgPHNd0/vEmg8AAEsC5+YRMBejCiMJAKCbuR+CWccBAIYz9zW9T6z5AACwFHBurpy5GVUYSQAANUs5BLOOAwD0s5Q1vU+s+QAAsARwbq6YuRpVGEkAAJeztEMw6zgAQJulrel9Ys0HAIC5g3NzpczdqMJIAgB4jKUeglnHAQAez1LX9D6x5gMAwJzBublClmJUYSQBwLGz9EMw6zgAwF+y9DW9T6z5AAAwV3BuroylGVUYSQBwrKzlEMw6DgCwnjW9T6z5AAAwR3BuroilGlUYSQBwbKztEMw6DgDHzNrW9D6x5gMAwNzAubkSlm5UYSQBwLGw1kMw6zgAHCNrXdP7xJoPAABzAufmCliLUYWRBABrZ+2HYNZxADgm1r6m94k1HwAA5gLOzYWzNqMKIwkA1sqxHIJZxwHgGPjio4+t6WfV7/3mo+Vaui+p/qpdY6XxAAAAOBQ4NxfMWg/KHIwBYG0ci2PTYh0HAOjnc/fOw7mpdgAAACwZnJsLZe0HZQ7GALAWjs2xabGOAwB0g3MTAABgO+DcXCDHclDmYAwAS+dYHZsW6zgAQBucmwAAANsB5+bCOLaDMgdjAFgqfB/b5eL72AAALgfnJgAAwHbAubkgjvUJIBycAHCscPAFAFgvrPEAAADbAefmQjhWx6aFgxMAjhEOvgAA64U1HgAAYDvg3FwAvNp4uXi1EQCOBQ6+AADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF5Y4wEAALYDzs0jAeMJAGB5sHYDAKwX1ngAAIDtgHPzSMB4AgBYHqzdAADrhTUeAABgO+DcPBIwngAAlgdrNwDAemGNBwAA2A44N48EjCcAgOXB2g0AsF60tt7/gcOLNR4AAJYOzs0jgQMyAMDyYO0GAAAAAADoBufmkcABGQBgebB2AwAAAAAAdINz80jggAwAsDxYuwEAAAAAALrBuXkkcEAGAFgerN0AAAAAAADd4Nw8EjggAwAsD62Z1T9/2LdYuwEAAAAAYK7g3DwScG4CAAAAAAAAAMDawLl5JPD0DwAAAAAsiQ/cd8/mR97z9s1z3vamzd97/c9uvv1VP7l5yjUv3PxHP/W/IDRZmkOaS3//F689mVsvet+7Njf+0YXTWQcAAEsE5yYAAAAAAMyCt991fvNP/t0vb5768hedOKGedO4FpYMKoW3qm172Y5tvvCjNPc1BAABYFjg3AQAAAADgoLzx9285fTLzis3XXY1DEx1Ocqx/6yt+4mROAgDAMsC5CQAAAAAAB+Edd53f/N9e+zJeN0ezk+bkf/rqK0/mKAAAzBucmwAAAAAAsHf0CrBeBa4cSwjNRV9/9RWb//7X/+3prAUAgDmCczPxWxc+tvmh639j84+ve/3m777uZzZPe+VLN0++uKFVGx1CQ6U5pLn0zNe/YvPf/dq/2Tz/hneezDWAIVy4/TOb6998YfPWV5zfvO7Hb9286odv3lzz3A9vfvr/+wGEEDp6aT3UuviGn7ht8+s/99HNDb96z8m6CfPlk599dPNf/G8v33wDT2uihehrr/rRzd/5hWtO5i4AAMyPo3dufukrX978yvnbNt/75tdunnTu+SefzH31lf+q3NQQ2qa++eU/fjLn/tF1rz+Zg5qLAOIrX/6LzfmbHtq8+WW3b67+/g9trvmBD2+u+mcfLA/1CCGEHq+f+R9/52T9fOvP3nGynmpdhXnw3nvuOvnnLbKBKvsIoTlLc1dzGAAA5sVROzev+tANJ18Yzesw6NDSHJSxdO3NN57OTjhWPvTO+zYv//99ePPyf8GTmQghtA1pPX35v/idzS3X33+60sKhkFOosoMQWppwcAIAzIujdG7+8u0f2XzHz/3U5sn8J0Y0M33NVT968vq65igcF7d/+E82P/e/3LK55rkfKg/nCCGEzqZz3/fBk9fXtd7C/tHrvPogt7J/EFqa/urLX8Qr6gAAM+KonJuf+9KXNt/zSz+/+Zaf+dflJoXQXKSvR9BXJXzuz790Onthrfz5n31l80vn/nDzs//ypvIwjhBCaLu65gc+tHnLy24/WX9hf+j7CnkVHa1JmtMAADAPjsa5+ZFP//HmO171k5snXoVRhZajp73yJSdzF9bJA/f+6ebn/udbNuf+Od+niRBC+9arfujmzQOf/NPTFRl2if7TtP4hS2XrILRUaU7zX9QBAObBUTg3f+2jf7D5misxqNAypVfVNYdhXdz5uw9vrvo+nJoIIXRI6VV1rcewO95x13m+3x6tVnrbSnMcAAAOy+qdm3IKfdVP/Ui5GSG0FH3VT/8IDs4VoYP0T//T+qCNEEJoz7q4HuPg3B3/2WuuLm0bhNYizXEAADgsq3Zu6nVenthEa9FXX/mveEV9BTxw7xd4YhMhhGamq/7ZB0++KgS2yxt//5bNU655YWnXILQWyUbXXAcAgMOxWuem/hGLvq+w2oAQWqr0vbH6x1iwTPTPK/Qdb9XBGiGE0GGl70Dmnwxtl6e98qWlPYPQ2vStr/iJ01kPAACHYLXOTf2naf55EFqj9B//YZnov/Pyz4MQQmi++qVzf3i6YsNZeftd5zdPffmLSlsGobXpSeeefzLnAQDgMKzSufnLt39k8018cTlaqeS01xyHZXH7h/9k8/J/8TvlYRohhNA8pA+gtF7D2dF/ka7sGITWqn/y7375dPYDAMC+WaVzU68FVBsOQmuRXvOCZXHtD95UHqQRQgjNS6/64ZtPV244CzxogI5N33hxzgMAwGFYnXPzqg/dsHny1S8oNxyE1qKvuepHN9fefOPprIe586F33re55rkfKg/RCCGE5qVz3/fBzS3X33+6gsMUbvyjC1tzbl548IHN5z//+RNd+f53l2kQmoM05z9w3z2ndwEAAOyTVTk3v/SVL/MpMToaaa5rzsO8+cqX/4LX0RFCaGHSuq31G6bxove9a/Okc9t52MCOTelNH7mpTLNUve32Wy/17dU3vb9Ms2S94Pq3X+rfjXd/vEyzNv3Ie95+ehcAAMA+WZVz81fO37b5lp/58XKjQWht0tObmvMwb87f9NDmZ//l/l9Jv/2m+y8dKG66/t4yDUIIoVp6elPrN0zj2b/6xtJ2mSLvZdLanJu3fPLCavsmqU/un/papVmbnv1rbzy9CwAAYJ+syrn5j657fbnJoLPru99w7aXXgvRTf1fp0H7132JAzZ63/uwd5cF517r3rocuHShuvfG+Mg1CCKG2fv1VHz1dyWEsf/8Xt2cnei+TcG4uS8fo3PzPX/uy07sAAAD2yaqcm0869/xykxmj+PpE1gOPPnKyMR/j9/089+3XXTYW+rtKh/YrzXmYN1d///a+a/MXX3rryROZ99/7yKV78cFPPbq5+/yDm3e84c7L0uLcXLbGXGuE0G6k9Rum8e2v+snSbpkir4FSdADqg3bZ5grXK91Pv/YlJ/H+MF5x77rz9pNwpX/2dW+4VI7iqg/qozMuvyauv++4/77LylD5VTkKU1z8vlCdIXTOULzOEg6v5LqrPqpch+U+qvzYRr0Krn67XVEKj2VJyludc+L5SPncP+dVP/O1cfpK8RV1X7fYbv0ey1uSvu1ifwAAYP+sxrn5Wxc+tvnml5/9lfRo1HRJm64NiWOQjBSe3JyfvvrKf3Uy92GeXLj9M5uf+R+3832bb732DzePPvzZx61FUTH9HJ2b23hV/hhetx97rVGt91x316XxklO4SoNQl676vg+erOMwnqdc88LSbpmiuPZFh1f84F3Osuiki5LN7jwxTXZeStEZaSef7P3oeMtSmXZaStEhWUnxfecN93NMH+VsrOKUJ9vtfc7V/JRlbG+rHknfIar0+aGIrFh+19gu8YGSr7/6itO7AAAA9slqnJvPv+GdJ46eapMZo6HOTSkaWAgdSj90/W+c3gUwN2741Xs2V/2zD5aH5rHSU3vVOhQV08/RubmNNh3DE6ljrzWqpfnh8dK8qdIg1Kfr33zhdEWHMVT2ylTFtS/a3n0OtCg7yaJjLjo9pfhkp+SHGORUjOGV5EB0+i7nn6R29zkX7Xgd08cu2elY9bOlONZjzkdypEpVnOUnN/vGYalnLQAA2D+rcW7+4y1932bevGOcPpWNn5hmowihQ0hzH+bJW19xvjwsj5VeQ47rUnxiUU/5yYmjJ/1iHpyby9SUa41qaaw8jjg30VRpHYfxVPbKVMU1scu5KRvdTkzFRZvdzr34erUUn2iM/7nczrfspPOr4VUdfnpTTyU6LD55qN/1ZGh8TTymrRx5VR8VprjKMSjHqtqndscnIuOTktnB6/YoX8v5m89HinM7NCYxLj7FGvPlp0FzvOp2uMpW+urp2iUIAAD2z2qcm3/3dT9Tbi5jlTfvsfHa0PMnvPrbBkAlGSfRuJHhog0+Gj/xNZkYLsVPYKMhIsPGRpeMA6VzPfHVcrVNbXRa/dTfsZ5Yln4qLNarMJcXFccrGyj6OxpeKkP9rspBbX3Xz587vQtgbrzux28tD8tjlR1er/zh/v++np2Acozp1VyH6bsc9dpulVd6/29euKwM51FZuX59P6Rfo5YzTnU5r55C/L33tV85k4a8Xq72VHmtXIb6Fvsrtb6v0q+622loB6LC9HuVRuXH78OM46nxufPWT18qQ2Pgcvo05VqrLqevxtLt0E9dK4e7/RoXt9np3JchY2ONmTOW4uI4ZrleSWOjNub0qlN1O536GOOz8ivqmq/xekmqI5YZpXaoDKfXT/2tcqr0aNl6zfN/73RFhzFU9spU+b6UZFc6XPZrjMv2cXRWRsea7VkpOh+rV9Kj4y7a2Fasw22LDkLVpTQt2zba/7FvVl8fox1th6wVnZ/qm8Nj/6MjUpKD03GSw6M9r7Ls4LViO2I/Yr54DazsoNXYdZ2ZliIAANg/q3FuPu2VLy03l7GKm7CU46MREw0FKRozlSqjpStPNKJieN70s+FThUfnpaX4/Al2VOxfq45YZvXpamUoyiCKRlCWyszGFmrrW1/xE6d3AcyNV/3wzeVheayyw0uOl+ikqhSdTF3Oo+zglBMqO6iy5MyJjpzYvuj0sS5cDIt/Zw1x/ClNldeKZURnX6VcX+xvzqu4vjRRKjv335KjMNZb6azXuhrLWJ7Kz+GqI88RlzNkbKbMmVx2l5Q2j0slj29fWrdb6nOax7SS7pcqnSQndkyL1qFrf/Dm0xUdxlDZK1MV77NoS7dsU6vlWIu2tx2Ccj46THaonXf5XNAlty0+EBClNqjNboc01rmZ47vyt/LGsD45T2ssrVY7+vJJ1ZlA54foeF6aAABg/6zGufnkq68oN5exipuw5HAZOdpko7EiR6fj4ye7XYpGTa6rktPGsGwYtYyXHJ6lT3+j87FSXx3RQIzOWCk+2SnZUJQhGcMrRcMSdYv/mD5frnnuh8vD8hRV38MoR2J0VkUNdRxlh0yfY9CSM8x5+pxJf/DhPy7DrW0+uamfVXxWHLeusfJTfkPHs09DnsQ8y7We4tys5PEcMjZT5ky8nipH4yLlsuQUVXr1K4a3pDKGPrkpZ2sVnxXHtLo2UU6H1qOrv/+Dpys6jKGyV6Yq3mPRcdayTa2WYy1/sC97M9rxsm2rMvoUnXEqU3krJ2e047uck1JfH7vyt/LGsC7FhxxaY2m12tGXz9LYVWeSakyWIAAA2D+rcW5WG8sUxU24S3LkRedb3JBlFPn1E/2Mn0hGh2jMozQxj9JFI8DppGgUSS3jJYfLwHJev9YS412/DbJo1LTqyAaiy5DiU67VJ+OSjEmPo+qIRiBPbw4XzJPqoDxVcsS0HCtyQPU9FSeHjpw+lQPJDrfsFHIex+V8rjM7N+WQshMttqvPCTdEfWXEMVJ7Y/vj04nxKco8VspnZ5vzd6XJTz3G/mdna8tBGXWWa30W56Ydmuqz50Tf2EgxfuicieGxTSrX4UrjcKXRmKh/sf8Kd/pcltI6XP1wuBXboGvoctWGHOc8DpPcT6VXXWqf06F1CcZT2SpTFe+76Oxq2aZWl2Mt2ptyrEVbPdqfsYz8If5QqbxYfnx9vOUUtPr62JW/lTeGDbW1u8ZSarWjL1+WzifxIQhdpyrd3AUAAPsH52ZS3IQraZOV8zI6NltPKFrRAeiNPeexs7GlmFbGSoxrGS85vKojOlj1e3Q2RnUZV9lAdHgs2+Hxk/HKSIwO0WykobZgnlSH5LPITpTqtef82m90SvlJtVhOzGuHUHTEqTyli/mk6HSzIy07mLLzzepzwrUcena6SV1lqN6YL7c/vlIcnV1dYzUkTX5VOfc/9qv1PY5ZU691Na4xb3T+xfA4xlF9YzN1zqgsh8U2xbkZr1GXnD6Xpbq6yopjW309g+Mkh8e+6Hf1v+ozWpdgPJWtMlXxXoy2YZdtKnU51vKbR/49O9NyHSozfpAv21q2a8ynulR+tLtlB7uM2JZo96odssEl19HXR5XluGw3t/LGPKozOjhVt9qqNLG8rrGUWu2IZyDJ/fLYKK3yxjbkByEcviQBAMD+wbmZFDfvluLrKlI2Hrpkg6DPWMmKaZU3xrXKGlJHNjqs+PSp1FVWNBB38d1FqF8wT6pD8jYkZ4qcZNkZGB04Uxxefc4gqSo3OzdzHmtMm6Ji2q4ycju6FPvX166+NH39H1J+S7u61l3hUX1lK8zxsU1RVRlDXkvP9elJST1xG8vLiv3oa1vM1yfnaX3nptrtJznR+gTjqWyVqYr3WrQN++zcaHNmh1zOa2UbX4rOzy45fXT0VYp1tOxi97Ovjy2notTK27L9s2J5XWMptdrRGmeXEcutFN8iW5IAAGD/4NxMypuswuSYy9+pOWTjrmSDps9YyYpplTfGtcoaWocckdFBaUWnZFdZ2UjK4zXEiKsUnwJF3YJ5Uh2St634BJzk8D6nVMxjh9BUR1Wfc8/qa1N24FnxaceuMnI7uhRfee5rV1+avv4PKX+Itnmtu8Kj+spWmOOVNsdLrTKqp1ItvQoen4js+kc+UbEffW2L+bqkeRnzVa/bS60nV9HyBeOpbJWpivdZl/0d80jR5qwccvENIys+QWjpKcP4llKl/ORmlUZSuvjwQH5K0XI/+/rYcipKXXkruz9LtrzT941lVzuqcXYZsdxK1fVYggAAYP/g3EzKm2yMy/8Ip+X4c3iXcp5o6FSKafNGn52LDu8ziLLUbjkUowFnw6avrJhHZcRPuWN74/gqTSwDTRfMk+qQvG21HGtTHF7RGSRHTc4jRYeUX2VutSGrr01D1FVGbsdQR9OQdo2pN8ZJ2+i3NORa5//IrjGIeXytpVZ4VF/bFeb4MXMmvs4e65AjUWXmaxcd3/lVcIdLsR+xbarD4VbMl19LHyK1QY73qn9oXYLxVLbKVFVfcyTJdrb9GZ2LVuuD9ipe6rJLZSPLhs1PccpJp1fLs8NS9cV2K5/SVWcE2cmxXP3u17b7+hjPJtEZKamMWGaMk2TbK3+04dVmheUHDOJZw29pRcV25LxqR4xXHU5TjaviNX4egyUKAAD2D87NpC7npgyMGKe0jovh2sDjhqx8Mji0WcuQqPJoU7dhpJ/e6J02bvoxrYyDaJRIzjPEuam8qsvlSfHTXPexr6yYJ7Y1G2K5nFy3xi1/dxHqF8yT6pA8RXpKTM4ZOVLiq6/63sX4z2ziE2Z9TinHSXYI5e+sjK/a6md+ctBxLadblhxvTuMn86TYpz71leE4Se2N302pNHI+aZyiE6xvrPrS9PV/SPnWlGsdx0SONvdNP2MeKfa7FR7V1/apcya2S9cvllkpluG2Kp/aVMVJ+WlP1+05Efum9kQHp8rWNVCa2G+Nr/6O1yY+xdl3fdEyBeOpbBWEjkUAALB/cG4mydnmQ4qU4+NrF3LA+RPY+IXgXYrOzegQbGlMWst5hjg3Y3wlf7LaV1aOt9TunDY6P7uU86G2YJ5Uh+Qpik6YLsWnxvqcUjFfdAhlZ1hLcug4j/LHOIdnZUeUNcYh1FdGdPR1Kfa5b6z60vT1f0j51pRrLSdclaZS7HcrPGpI26fMmfgkZpbqjE5SKT4d2aXYj3xdLJWv+KGvusd+V/FRuhZOi9YjGE9lqyB0LAIAgP2DczOpz7lZPXmocDk5hzjt4hOdfXn0pKfT6unG/ISmJYdrLMd5zurcVJlON6QstTemkfIr9JLGoNUXiyc3xwnmSXVInqL4ZFhL+XXkPqdUzBsdQnqqrc+RJGdWfNouO5EcniVnVUxn9Tn8ovrKULuGONviE51DHHhdafr6P6R8a8q1llp9lhMxPj0Zr3VMF8OjhrR9ypxRWVW6KJXp69SVPtad+1E5UdUnxw8Z7+hIruIt9dHp0LoE46lsFYSORQAAsH9wbibpSUUfVKJzMSo+vZm/d0b5Y7wkJ6GeYJSDMKaVxnzXjP6O31mjPH6y0uHRIRkdoi1nodqkvNHZqPbn7+0ZUtauvrsI9QvmSXVIniI5heRgkVMmOnLkuJFzpnJMxScYo3PGcjnRgWSpPjmTssNM9VdPpsnhGMvL8VF6Wi6Wq99z/X0aUoZfKXYap6vGq2+s+tKobsdV/Y/Oxfjqc6Up19r5FO88Sq9r6PIUprj4NGQcw9Y1GDI20tg5o/pi/1rS2DmPysnXXXUq3GG5H/o7jr/GJbdHY6o0ebwVNiSt+tg1Nmj5gvFUtgpCxyIAANg/ODcRWoFgnlSHZISOXXKERudgdEjK+RqfplS6mBehQwjGU9kqCB2LAABg/+DcRGgFgnlSHZIROnbFJy311GOOj9+FqaczczxC+xaMp7JVEDoWAQDA/sG5idAKBPOkOiQjdOyKzks9mRlf1ddTnPHVc712HvMidAjBeCpbBaFjEQAA7B+cmwitQDBPqkMyQscuvZaev5uzkl5Pr/IjtG/BeCpbBaFjEQAA7B+cmwitQDBPqkMyQujyf5wUHZp6klP/sKf1j5MQOoRgPJWtgtCxCAAA9g/OTYRWIJgn1SEZIYTQsgTjqWwVhI5FAACwf3BuIrQCwTypDskIIYSWJRhPZasgdCwCAID9g3MToRUI5kl1SEYIIbQswXgqWwWhYxEAAOwfnJsIrUAwT6pDMkIIoWUJxlPZKggdiwAAYP/g3ERoBYJ5Uh2SEUIILUswnspWQehYBAAA+wfnJkIrEMyT6pCMEEJoWYLxVLYKQsciAADYPzg3EVqBYJ5Uh2SEEELLEoynslWWqmdf94bN5z//+U7dePfHy7zf/YZrN2+7/dbNHfffdynthQcf2LzpIzdtnn7tSy5L+9y3X3cpPoZLKt/5K93yyQuPy4MOJwAA2D84NxFagWCeVIdkhBBCyxKMp7JVlio7He1ErCRnZc535fvfvXng0Ucel9dh+inHqdPHemI5kvIpXI5PlxMlB2rOgw4nAADYPzg3EVqBYJ5Uh2SE0O714KcePXEEvPXaPyzjERojGE9lqyxVXU7Hll5w/dsv5XnXnbc/7ilNOz7lmHTYEOdm5URF8xMAAOwfnJsIrUAwT6pDMkJo97KD4B1vuLOMR2iMYDyVrbJUjXVuypHppzPl2KzSSHplPTorcW6uRwAAsH9wbibpk1QbFtV33iA0R8E8qQ7Ja5GeiPNaee9dD5Vp1i45zu689dOXnhKU7r/3kc37f/NCmR7tT74eQ52bd59/8FKeKF1bXeNffOmtZT50HILxVLbKVMkJ6O+clNNQr2DnJyF3qbHOzXiWGNNOnJvrEQAA7B+cm0kyGmxYVMbFkiXjMH7PT5Vm6Ypf2B6/x2jtgnlSHZLXIjl8fK9Jx+b8uf2m+y/1/dGHP3vi4JUcJifnK3/4pjIv2r18HYY6N33t5Mz0tdQ1dDm6xrzifryC8VS2yhTJOaiHDXwvWvv8jsmxzk07Ylv/ZKglnJvrEQAA7B+cm0lrdm5Go2ltfbNi/9TfKs0aBfOkOiSvRXL2SHZy3nrjfWW6NSo6Nm+6/t7L4uTQ1Fgc25jMTb4+Y52b+ZrJaW8npxyfMQ4dj2A8la0yRdl2jarS70KxDXIyZmVHqx8kGOuIxLm5HgEAwP7BuZmEc3PZiv3DuQmHpjokr0F67Vr3mByb77nurpPfj8XxI2eZ15iu18+VjtfTDydfo7M6NyXPcYnX049TMJ7KVpmiuTk3K8nxGNM7XP9UKIb3aYhzs/Xf0sfWhXYrAADYPzg3k7qcm37l2a+Z6Dt14mvQCo+vQvu1FKl6fSZ+4Xh+TVzl6EvIHS+pLtUZ01kKl3HjtDJ+4n9nrF7psWLduY8aj5g39lE/Yx+Vt8u4evVN779svFSv2qjX5XPa3I6+sVY5jqsU065RME+qQ/JU6YlAPTGopyU1p/UdgYdytPj7CeX00d9uU+VIit/NWSk6APUUpMJUfizD8lOi+WlJO6b0u+LcHqWP6dS++N2KSqe/hzrAJLdBT/NV8X1Sf91eS23wWGYpXun1u/LGV6XVltYckJMuplUd+bVqj0UeT8vfJZrzxfHW2LXq0ZxVG309VF6rn9LY6+OnZN1Opdc9onCXMfTadjk3VYbLq15Nz9dF7cnXxmWojTFvlJ8Izm0YMy7KqzSKU/15/FsOd6ep+qdyXG+Ok8a0b6mC8VS2yhQt8bV0px37IfsQ52ZLsrFzHnQ4AQDA/sG5mdTl3HSYnGzR0RYlh52ddTI0YngsS4pfOC7DrQqvJAMnliMDr0on+b80VnFRLst/9/Uxj1NUdnDKMG2VJam8nMdxQ8e6z+gba2AuTTBPqkPyVEXngTXVwXYWVY4OO/z0M6aVlF6Ooyz3ITpb7JhRfCzDcr7s/Inh+iknjutxGtWjOEnjpjg7xaSW0yfLTqCWQ7BLHqdWG6rxi2n1u/vmPGqPnHkxT3xtXmnteMvOKZeTx9NyGdlJ5XyuR+UqzGOjn8rjvxUX+1k5OMdeH/XZ/ZKU3n/H8KEONvepGgu3LY+fFK+pypDiOERnocNbDl7Hxzxjx8X3gNrl8nKeqo+Oq8ZLYY7PcWPbt1TBeCpbZaqqfyhUpduVpjo3ZStX8S0NcW6OLRMdRgAAsH9wbiYNcW72yQ5FGWMxPDvw4lOPNtT0hGHM05KNm1xHlh2hQ5/crOLHSs5IlyfFfrakNvgp0zHt8Fjz5CbMkeqQPFXVvJaGOm+2JT9dGR1xctYoTM6UmLYlp5cTJDrm7JiRgySmtxSu+OyccbhUOQjtnFF9+cm06LjKTsIs5XU9Y8fdfVM9Oa/HVMqOL4dL0aEqp7GdVzlcYbk/qlPOJ/8ttcbTcr25vXm8XU9sk6Tf43jbEajr4DBpyvXxeCqP6nW4xi+2Yeh1qsZC9al+l5cddW6D8ua5474qLodVczTeEw47y7hIut5xbKIz2mGW81Tj5XZIVfiY9i1VMJ7KVlmqxjo37Yi0jTpUODfXIwAA2D84N5PGODeVVg45ORjj04XRuRfDo5GjfA6X7HyLTjrldbjS5ziFR0MoPv2pNim9X+vOaSWHR8V4ya+5qB1yQMY4tUH15LZJLi87X1WenZhqTywzOn9jHmnIWOd8Kj/GrVkwT6pD8lTFuR011HmzLfmpuOyEazmAsuTocNrc9ugsiuFW5YCK4a18fuo1t9ly/r6xjE6esQ6bvvGx40ttjeGur8pnZ1Xsd3Q4xbSVWuNpue48Ls5XOemiAy07vOx4lWL4lOvTNZ6q1/X0XVPLdVTSWFZtcxtac8H5/bevjfLlPL7+8VpMGRffQ7pPcx36W3E5j9QKl9xuKYZv675agmA8la2yVHU5HSv5jar48MAQddWDc3NZAgCA/YNzM2moczN/t40cczHe4fHV9Nar5zG85eyTskNUYdl5KONHZcd81ljnZu5jfP1dbbaTUsrt0N8Kj/3PjkgplhkNNodJQ8daiuE4N+HQVIfkqbIjISo/ibdr2Wkk50yOs1MrO+ey3I/KMXZW52bLiWInlMZLabMc33LyWS0nT5+G5Gul6crnPOpDDHd/NNbxyb0s5VO6Vr9ddx5X58vhUt81rMoce32i89JlZDm+amMl1aP0cmTqdzvxJc3tnD62IbY1yvGxDSpfYdkp6z7G6zV2XCSPf+uaKp/i87gorAqXFOb4GD6lfUsVjKeyVZaqsc7NaBN3OSNlR8dX7HFurkcAALB/cG4mDXVuZsdZy3GYnX7VE5otp16fnCc6CC05SVVudEC22hgV43Mf49jIyIpxUpU3j2eXWuMwdKylGJ7zrVkwT6pD8lTpqSs5WewwkOOq9cTYrmQHZuWYjM6elkNNT3gpXn2o2m7HjBwjOU5SuOKzs8ThLUeW4oao9QSaFZ08rboqOZ+cQFW81HIgVWGW8+Tx0rXwPHF89YRjazwt58997RrvvmtYlemwPvn6tMYqyvFVGytVYxHHMTs4Yxu6pPzVq+HxQwDfF3l+xHK6FOetx791TVvXzmVV49Uab4f1qe++WoJgPJWtslRFu7OKr5Q/vI/2uKQHEWSrxwccuurBubksAQDA/sG5mZSdcTEuho9xuMXXqP0JbXxC0085Sg7rUzSGJNVvwycqOiG72mjF+NzHODa7cG7GJ05jeG5HVz9ieM63ZsE8qQ7JS5YdPa0ntXzvVf9sR85MP7XWcnb0OcZcR3bcOLzlyFJcV/wYuazW6+WV7Bzal3NT0nirjarTZej36FRujaflfHncusa77xpWZVZhXXK/pSpeGltmayxiXdFJOaQNlVSG8/la+GnmPKecbmgfJI9/65q2rl1XXa2+duVZm2A8la2yVEW7U/ZvpfgEppUfPnBa2fD6W2eB+J3wsZ5YjqR8Cse5uQwBAMD+wbmZtAvnZn41Pb5W3fWdkfm19CGSozT3wYZTVxutGJ/7eFbnZvVaektVWVZXP2J4zrdmwTypDslLlZ8uG6Lq+x6rJ9ay7ERpOcZaDqiWw8ZSXFf8GNlZ2NWPrJZzKMrjKwdyDO/K1zdeltLZMR0dz32OMNedx61rvF1mq01VmVVYl5TOeap4aWyZrbkl+ZrHpzeHtKEllydnphycLic6niWHD+2D1HdNW9eu65q2+uqwMe1bqmA8la2yVMmO9nxvqbKLJdmielvLDk1J9rDs4/w0p+3b/ACDZOdm66un0LwEAAD7B+dm0i6cm/nV9PgkZ/4+SRsvThcdnDKCZNQojdqpMNUrI0jhNpL0M9bhtircYZINpPipcYzPfZzi3MzjojLik6qqW59s69Nrh0kxT25HLjPGxX7LmFSY6ssG5NoE86Q6JC9VfrqseiXdiq+m63eH2zmSX9HNiulynBw/fvIzO266HDPSkLYPlRxSKqurPkn9tyMxOrBaT636H8rkNjpfDLM8Xi1HYlTl9HJY5aiN1zL3s2u8XWarTVWZU66Py6nGs6vtLblPeW5JvuZ5XtphnJ+47JPmhfLJyemyq2swZVyq6xzVunYOr566dnulGL7N+2rugvFUtgpCxyIAANg/ODeTduHclKLTMio6+qT8z3Jais7NKj4q1lHFS1V87uMU56YUHY5dGlKW1DXWrXHOZaxNME+qQ/ISFZ1z0WlZyU+lRYeHnZKV4yQq1hPTyiHqcqXsuOlytknxqdOqDYof46BxW+Tcyo4t9cEOplimfneePIbReZTjHB7DLPVXcdGRqLLkdIpOZLWpcl7FcYn1qlw77qSWI6wab/c9timqKnPK9fF46lqofzFtV9tbcp8qp6DKrxyZfhpZcbke5VHaVnlun+eS2p3TTRkXj39Vr9S6du6L7tU4nnFuSjHPlPYtVTCeylZB6FgEAAD7B+dm0q6cm3pKMsZLcvrldFL8Z0Mt+YnPXG9W/g6gVtmOj2G5j1Odm3o6M37HaKVtPblZjbOUy1ibYJ5Uh+Qlyg6O6nXzLKeVw0d/RweIHCtZ2QljJ4skx4/SuDw/KZbzOE2XI8vOMJfl+u20cnuHSM4f19kqLzvd9LsdWY5Xejt+peoJQMflcEn9VZzKcZifBJRch9uUHVdSbJPS+m/99O95XJWuCpfsXIttilJclXfs9ZHzNsYprceyq+0tuU8tp2B0pjosX1PV73Y7rHoiU/JclnLfosaOi8e/1Q+3LY9L13jGNsQ80jbvqzkLxlPZKggdiwAAYP/g3EyKzrH8nTfxCcT4KrcUv4+n+q4cvRYdv29H6vpOTTnjbrz745c5BZVfYX6d3NLf8YlF5VG6qny1Qw5Ol6uf0QHa1cf43aEqP8ZJscz8RKrqlXM0P8Wpdqv+nP4sY6164lirrbyWDoegOiQvUXaIVE9nZcnhY8eGHCjRuVmpcv7IMWPHisqSA0XOFztOczvcvr6nStUW1ef2Scorh2p80nGo5EisyquclJLGRn2LDjH3r9V2p6viNL6uM4arfo+JpLFUHdmxKSlMce6D0qqNCrcDLrfNZVeOQ1+j6rpKrqfq79jro7Cq7Yprtb0l96l17VSO25THUX2O46326G+FV2MuxftC/avSWGPGRf1XvMchy+2srp366HGTNE89Hqpbynmkbd9XcxSMp7JVEDoWAQDA/sG5idAKBPOkOiQjhBBalmA8la2C0LEIAAD2D85NhFYgmCfVIRkhhNCyBOOpbBWEjkUAALB/cG4itALBPKkOyQghhJYlGE9lqyB0LAIAgP2DcxOhFQjmSXVIRgghtCzBeCpbBaFjEQAA7B+cmwitQDBPqkMyQgihZQnGU9kqCB2LAABg/+DcRGgFgnlSHZIRQggtSzCeylZB6FgEAAD7B+cmQisQzJPqkIwQQmhZgvFUtgpCxyIAANg/ODcRWoFgnlSHZIQQQssSjKeyVRA6FgEAwP7BuYnQCgTzpDokI4QQWpZgPJWtgtCxCAAA9g/OTYRWIJgn1SEZIYTQsgTjqWwVhI5FAACwf3BuIrQCwTypDskIIYSWJRhPZasgdCwCAID9g3MToRUI5kl1SEYIIbQswXgqW2UtevVN7998/vOfP9GbPnJTmabSle9/9+bGuz++eeDRRy7l19/Pfft1ZXrpu99w7eZtt9+6ueP++y7lkfS3whVf5UOHFQAA7B+cmwitQDBPqkMyQgihZQnGU9kqa9GFBx+45GTU71WaqKdf+5LNLZ+8cFke/R0dlu+68/bH5ZMzNDpClSfnk5Qu50WHFQAA7B+cmwitQDBPqkMyQmvVg5969OSg/dZr/7CMR2ipgvFUtsoa9Ozr3nCyzumJSzs5u568lOyMlKMyp9WTlyorlyOHpcIkPaEpB2krn/SC699+WTw6rAAAYP/g3ERoBYJ5Uh2SEVqrfMh+xxvuLOMRWqpgPJWtsgbpCUutc3I+yumo36unLi29tq40cmzKMVqlkfSqu+PlyPQTm32vvdvBOeQJUrQ/AQDA/sG5OUPFT2JbahlIY77Px6/I5NdZ/Kl0l/o+pUb7FcyT6pC8Bv3iS2/d3H7T/Zv77/3LdUbS3wpXfJVPUtydt3760lN+kn6/9cb7Nq/84ZvKPJacZq28MZ3KefThz15KU+n9v3nhsjzo7PLY4txEaxOMp7JVpio+pSj7tnqScV+yfa36bS8rrNUeP92pNlfxlfydniq3io/S2CithG0+HwEAwP7BuTlD2eno7+SplL9AXEaVwm3gDPk+H6fPnwrLOHIe15fV9ekz2r9gnlSH5KVLTsHoOLz3rodOlB2dlfPwPdfddSleTknli45KlZHzWHJg5rySw2JaOdcUpnY6XRYOuO3L1+KsY+vrd/f5B8t4hPYtGE9lq0yR7Fs7CKPGOAu3Jb8qLkerw9y2/KCAFB8WGGM3y85Wnq4nQqPchjH/3AjtVgAAsH9wbs5QNmrGGClTvs+nVU90bsZwNF/BPKkOyUuWHJZeG/SEZn7SUk9lyiHlNHJmxng5KBWfv5Mxlls5RR0vZ2UuU8rONDvH5MSM4Wi38jXclnOT64fmIhhPZatMUbRJs6r0u5Rt6ejI9Kvp0eFpxf+qnuO65KdDlb+Kz7I9X7UBHUYAALB/cG7OUGOdm0qn9DKGuj4Zjt/nI+HcXI9gnlSH5KUqvuqdXwPPsoNTT1hW8ZX8BGdVtusd6jTDOXYYaczHXKeWuH5oboLxVLbKFM3FuaknSF1vfAU9Pp2Z36qyfa4HEGJ4n1xefCChS3awyq6v4tH+BQAA+wfnZiE5Af2KhwySocbFtjTWuem2jn1FB+fmegTzpDokT5Wci3pa0o4+ORC7vtty27rp+ntP6lX9VXyU2uU1ZKijS44spc/OTT+1OcbRtU3nmPvdekVa3wGqeKXLcepLfF2/empVUntVjq+tpLZXT6lK/t7SmD4r16M67HSWlFd/D70+luah+mVntMrxU7wuO5epOI1PHAvlVx8U53S+bi3F+T52zBA6i2A8la0yRXN5Ld1PYVZPR/rtqfykpZ2bY52O7uPQ84f/yRHOzfkIAAD2D87NJBsvUXoiMn8au0uNcW5O/T4fCefmegTzpDokT1V0TFld31G5bdn5KIdSFZ/V9SRmlhxcdlLl19JV39ByLDvJtuHcVL1dZXlccvvk8FO489qxp3620imN0nrspOysk4PPY6V0Ob3LiI7A+Np/VUf1VQCVdJ3cDyn2K4ZH52Zsr34qj+S0yue0csjGMmN6yY7QsWOG0FkF46lslamKX68km/wQ37dpB2b13Zo+O8gJG8Pt3JRieJ+cJztLW7I9f4hxQbUAAGD/4NxMsoGQNcTRuC25DTKS9HtWdGJGZ2wsY4hUVtU3nJvLE8yT6pA8Vb4ns/JTcruSHVTVE4qV5HBS+tYTj5acX9FBluNdjhxWciA6rSSHVn76T7JzMzvHlHaoI8+a4txUn1x/fjIx91Hjo3ZFZ6Rkp676WIXncXW4fsZwj4XKaX3XaW5nSx4LlRXbq2vj+SHFOak6lT6Pu8KdJ8e5za0xHztmCJ1VMJ7KVlmq4oME2SaX7PiUoo0e7ekxD0moTOUZ6qz0k61DnaFo9wIAgP2DczPJBkXWIZybLcXXVPyp8Njv85FcT5dzU2my+MLy+QnmSXVInirfk1nRkbRLja3PT9dlB5UcUgqT7KiUk0uOqcrBFp2ZkhxXMa/zx7z6PebJUt7s6GtpinMzOhRj2jGyg1SK4e53bn/scwz3E7+tpxnd/iHXteWMlNQe1z90jtgZGcdO8vi1xryl1pghdFbBeCpbZamSk9FrS5/ifziP39M5xvHo+vKToJWmOlDRbgUAAPsH52ZSfBLSmvNr6XZuKk8V36VWPdFQqqTxiOnR4QXzpDokT5WdVFFydFVpdyHXOdZxlR1U0QkWJUdgdnJJjlf/s0NPf9vhJmdqjosOTzm+5JRz+qFjN8W5KbketTs/YThUyi/FMI2TwvJYtJybsb9qa5bjq7GPitetipccP3SOeGxz3VOdm5LbUMUhNFUwnspWWapk92pdqV5Jt3x+yDayvw9T4fEfEWWpbD+8EJ8U7XOK+qnRKecAtDsBAMD+wblZSIaEX/GQ0TD2uyzPqinOTamK79IQ52YMR/MVzJPqkDxVcl7JgRedZtF5t2t5TRj7Wnp2OmbJaWYnlySnaIy3M6/lMHPeoY4wO866yozqK9/9zA666Hh1/uqJR0nXUeOqviud5bwxrZ3GeZwcrnkRw11Gn/q+pzKOWxUvOb4aV4VpjGL/fG3z2LkupYnhUWPGDKGzCsZT2SpL1Auuf/vJmtL3wX58SjM6QfVwhJ2jOlPYgRnj/X2iMV98WrRycCqf7XiVz1Ob8xIAAOwfnJsz1Bjn5lleR8G5uR7BPKkOyUuVHUd9zkrLjquhzlCl87oTn0p0va1yhjjdspx+SNumOjclOeDk0PSr5JJ+j07p+DRpS7HM2F+Nser3WKuc/ESn01YOxzEaMs5VXeqrx6ilsc7NsWOG0FkF46lslSXKjsf4unlLTpu/vkkPSfihCSl+p77DqvL91KflPLmsfT+EgfoFAAD7B+fmDNVyOlaa+n0+UqsenJvLE8yT6pC8VPk7NOVIq+KjoiNs6CvZcoI5T3SO+XX8llNVji7Fj3lF3/W0nqSM6nO0dTk3o1SOHXJ2qsbviFT+lmMyhrk+jUd0auoJxpxfchlxTKcoXtMqXqrq8hOluj56OjQ6du04HuPcnDJmCJ1VMJ7KVlmivKYMcSB22c+y12WnR4emnriUIzQ/zRmlODk5o0NT+VSOnvTsetUdHU4AALB/cG7OUGOcm9KU7/ORcG6uRzBPqkPyUiUnkteFvice/aRiyyFYKTrPonPMT3S2nKp2nulnFZ8V6xnieHV6ORBznBx1djD2OTel7Mxz2S3HrOIk/20H8JhxtXN46Ph0ye2pXmGP8yNePztjqzx5PCyPS9XPsWOG0DYE46lsFYSORQAAsH9wbs5QY52bU7/PB+fmegTzpDokL1l+elOqHJxyFtqZJWdgdh7KEShHVnx6T1I6O0SzEzM/qRfj7PiUYl0qo3oqU44xPz2Zv5uypfhEaexzbLMU26Z0Kj+2SeV4bFyOHXVqU3wCUfnslJRiGVX6Lsmp6HKqa6b4oY7P+BRmvIYqw+MqVc7NXEd8tbzl3JTyXBk7ZghtQzCeylZB6FgEAAD7B+fmDGWnY/xOnqz8esyU7/NxHM7N5QvmSXVIXrrs4LLkvJL8BKOk3yvnW0wjB5nyRQdhy2nnV88llZHry47M6Gir2qe/s9OsS9Gp63brd9Vjh1p00MX2Or3bpHbEumP/3Vb9rvRus9NKsS1RKkdtqZy68ZqpXNfjNulnzlNJDsSYR2W4jarffYnOzehcVdqcRz+zc1NyGtejn3YWjx0zhM4qGE9lqyB0LAIAgP2Dc3OG8lOWXcpPZ0pjv8/H6eLTnJKdm3KQxnA0X8E8qQ7Ja5CcV3KY2ZEk2Qkl51rLcahwPT1oZ5QlZ5UcXF0ORznJ4pN5qk9/R0eaJSeYnIDRCeb0lfNviNS+6HBT/1WP+qOw/FSk6on9VF7lyX3U3wpXmbFsOXnd35jejlOls2NPcn5JfY95JI9fTKd8Smun4RApbWyv+mXnpNubHdSqO14L/a7xUjr9XTk3FRfHT787buyYIXRWwXgqWwWhYxEAAOwfnJsIrUAwT6pDMkJT5Vey5VCsHMF2fOLcQ2i7gvFUtgpCxyIAANg/ODcRWoFgnlSHZISmSk84ynFZPelo4dxEaPuC8VS2CkLHIgAA2D84NxFagWCeVIdkhKbKzk291t16vV3xerIzxiGEziYYT2WrIHQsAgCA/YNzE6EVCOZJdUhGaKr0fZfxOzPl5PR3bjpM8fk7LxFCZxOMp7JVEDoWAQDA/sG5idAKBPOkOiQjdBZV/yzJjs6x/xwIITRMMJ7KVkHoWAQAAPsH5yZCKxDMk+qQjBBCaFmC8VS2CkLHIgAA2D84NxFagWCeVIdkhBBCyxKMp7JVEDoWAQDA/sG5idAKBPOkOiQjhBBalmA8la2C0LEIAAD2D85NhFYgmCfVIRkhhNCyBOOpbBWEjkUAALB/cG4itALBPKkOyQghhJYlGE9lqyB0LAIAgP2DcxOhFQjmSXVIRgghtCzBeCpbBaFjEQAA7B+cmwitQDBPqkMyQgihZQnGU9kqS9Wzr3vD5vOf/3ynbrz742VedJwCAID9g3MToRUI5kl1SEYIIbQswXgqW2Wpeu7br7vkxLzlkxdKvekjN12WR2FKL8doDEfHIQAA2D84NxFagWCeVIdkhBBCyxKMp7JVlqro3KziK9m5qbxVPFq3AABg/+DcRGgFgnlSHZIRQggtSzCeylZZqnBuorECAID9g3MToRUI5kl1SF6yHn34s5cOeJUUX+VDCKElC8ZT2SpT9d1vuPbkOy21zzzw6CObt91+6+bp176kTLsLjXFu2qlZ6V133n5ZWr2yrn6pT06jv698/7svS4eWJwAA2D84NxFagWCeVIfkJcuHr/vvfWRz710PPU53n3/wsvS33njfSfqbrr/3snCEEFqSYDyVrTJFcmJeePCBS/uPJQdnlX4XGuPcVLvk4LTD8o777zv5W3r1Te+/lE4OTJep/ik+9jOWiZYnAADYPzg3EVqBYJ5Uh+Qly4eud7zhzjI+y85N/aziEUJoCYLxVLbKFEXHYlaVfheKbbCjMqpytCpc6avX0vUkqp2f0eEp+WnOGIaWJwAA2D84NxFagWCeVIfkJcuHO5ybCKFjEoynslWmaG7OzUpyZOY8Xc5NOTRb+dA6BAAA+wfnJkIrEMyT6pA8Va/84Zs2t990/6XvvdQr4L/40lvLtLuSD3J9zk07NSs9+KlHL0urft1566dPwp1Gr73rVXbFxbQIIXQIwXgqW2WKlvZautXl3HTcmz5y0+Pi0DoEAAD7B+cmQisQzJPqkDxVcmb6cGXJCVil3ZVcb59z8/2/eeHkOzjtsNRPfy+nHJlO99Zr//CSs1Y/Fa8+Da0HIYT2IRjPU655YWmvTFH1D4WqdLvSrpybVRxavr7+6itO7wIAANgnODcRWoFgnlSH5KnywSprnw5A19n6h0I5fd9r6XZkyuEZn9LUE6l6SlXOz5geIYQOIRjPt7/qJ0t7ZYnCuYnG6NuufcnpXQAAAPsE5yZCKxDMk+qQPFU+WGUdwrnZUk7f5dyU41JxemKT188RQnMWjOfvvf5nS3tlidqVc5PX0tcpzX0AANg/ODcRWoFgnlSH5Kla0mvpVpdz03HVE58IITQnwXj+P7/x5tJeWaK27dzUa/WKU5och5av57ztTad3AQAA7JPVODeffPUV5QaD0Nr1pHPPP70LYG5c89wPlwflKar+odC+n3j04W6bzs0qDiGE5qKrv/+Dpys6jOFF73tXabMsUWdxblbfD6rvEHV5+s/pOe5dd95+8o+UHHbH/Y/tl1e+/92XpUXz1L/8rf/99C4AAIB9shrn5tNe+dJyg0Fo7frWV/zE6V0Ac+NVP3xzeVheqnwYw7mJEDoWXfuDN5+u6DCGD9x3z1b/qdAhFZ2bclpWyk5MOSKdR85JSU5Lx/vpTUn/DV5l2IkpPfu6N1xK6zD9UyWHoXnq6869YHPjH104vQsAAGCfrMa5+Xdf9zPlJoPQ2vV3fuGa07sA5sbrfvzW8rC8VPmAxWvpCKFj0Wtf+HunKzqM5Rtf9mOl3bI0ydHo/a8lOSdzPn2npv67u+L18wXXv/2yeP2tfC5DaeTAzE9o8uTmcvSUa/hP6QAAh2I1zs3/4X//pXKTQWjtetabX3t6F8Dc+Hev2d8/+9mHfADbhnPzPdfddRLHPxRCCM1Zb3nZ7acrOozlnxbfN4nQmvX/eusbTmc/AADsm9U4N3/o+t8oNxmE1q7n/Z+/dnoXwNy4/s0XysPyUjXVudn6x0cPfurRk/g7b/30ZQ5O/a68sR5936jTOgwhhHat//PffOJ0RYexvP2u8yffC17ZLgitTXpqU3MeAAAOw2qcm7914WObr+efCqEjk+a85j7Mkwu3f2ZzzQ9s758KHVp2bspZqdfJK8X0v/jSv/xOMTkyFa+fjvfTm5Ke4HQZDrvp+nsvpXW40jkMIYR2Ka3fWsdhOvpe8Mp+QWht+huvePHprAcAgEOwGuemeOJVfDqMjkua8zBvzv3zD5aH5iXK/6m9SzmPHJh+QlPKr6i/9do/PPnP77FsOTLl2IxPc/LkJkJo39L6DWfjjb9/C09votVLc/zVv8t6AQBwSFbl3Pxvf+2N5YaD0Fr1vXzf5uz59Vd9tDw0I4QQmrfezPdtboX/7DVXlzYMQmvRd/6vP3062wEA4FCsyrn5K+dv23zduReUmw5Ca5P+C6nmPMyb8zc9tLn6+z9UHpwRQgjNUy//Fx8+Wb/h7LzjrvOr+c/pCGXpK6I0xwEA4LCsyrn5pa98efNNGE/oSPSUa154Mudh3nzly39x8ZD8O+XhGSGE0Dz1sh/48Mn6Ddvhn/y7X9587VU/WtozCC1VmtP/77f+4uksBwCAQ7Iq56a49uYbN0++mqc30br1NReNqas+dMPprIe5c8v192+ueS5PbyKE0BJ07vs+uPnQO+87XcFhW/wX/9vLS5sGoaXqu37+qtPZDQAAh2Z1zk3xtFe+tNyAEFqL/vrP/uvT2Q5L4VU/fHN5iEYIITQvveJ/+p3TlRu2ySc/+yhvWKHVSG9QaU4DAMA8WKVz85dv/8jJ959UGxFCS5fm9ms/wsFradz+4T/ZXPMDPL2JEEJzltbpW3/706crN2yb995zV2nbILQ0aS4DAMB8WKVzU+i/SFcbEUJL1hOvev7mv/w3P3c6y2FpvOVlt5eHaYQQQofXuX/+wc2//ak/OF2xYVfIKcQ/GEJLleYujk0AgPmxWufm5770pc13vOony00JoaXqb177ks3n/vxLp7MclsaX/uwrm5/7n28pD9UIIYQOq1f+0M2bP7+4TsPu0eu8fAcnWpL0z4P0HZu8ig4AME9W69wUH/n0H2+++sp/VW5QCC1Nmsua07BsHrj3C5urvu+D5cEaIYTQYXTVP/vg5oFP/unpSg374r//9X/LU5xo9tJXQmmuAgDAfFm1c1P82kf/YPNVP/0j5UaF0FKkOfxv/uB3T2c1LJ07b3l4c+U/rQ/YCCGE9iutx3/wwQdPV2jYN++46/zmP331lZsnnXt+aQMhdCjpwYLv/F9/+mSOAgDAvFm9c1PIwckTnGip0tzFsbk+7vzdh0+eFKoO2gghhPYjrcM4NufBG3//ls3feMWLN0+5hn8Kig6rp778RSdz8dW/+8HT2QkAAHPnKJybQq/z6vsKqw0MoTlK/zxIc5ZX0deLXlHXd7xVB26EEEK7k/550Ct/8OaTdRjmxdvvOn/pdfWnXPPC0kZCaJt60rkXnMw1zTnNPc1BAABYFkfj3BT6Ryzf80s/f/K9KdXGhtBcpDmquap/jAXrRv+84pfO/eHmmh/4UHkARwghtF1pvdW6q3/yBvPmxj+6sHnR+961+X/+yus2f+/1P7v5tmtfgh2PzizNIc2l//y1L9s8+9feuPmR97x984H77jmddQAAsESOyrlpfvn2j2y+9RU/UW52CB1SX3PVj57Mzdd+5HdOZyscC7d/+E821/7gTeVBHCGE0Nl17vs+eLLO3vrbnz5deQEAAABgDRylc9Nce/ONm2+45oWbrzv3gtLRhNC+pDmouXjVh244nZ1wrNxy/f2blz3vw5urv58nORFCaBvSeqp19UPvvO90pQUAAACANXHUzk3xpa98efMr52/bfO+bX3vyXxp51QXtS5pr+l7Nf3Td6zevv+2mk7kIIL7y5b/YnL/poc2bX3b7yaH8mh/4cHlgRwghVEvrpr5X860/e8fmtvc/cLKuAgAAAMA6OXrnZua3Lnxs8/wb3nnyfYd/5xeuOXlFWE7PyjmF0FBpDmkufdfPn9v84+tev/mh63/jZK4BDOHC7Z/Z3PCr92x+6ao/2Lz2hb+3ufYHb95c/f38p3WEEJK0HmpdfM3zf2/z1lec31z/5gsn6yYAAAAAHAc4NwEAAAAAAAAAAGCR4NwEAAAAAAAAAACARYJzEwAAAAAAAAAAABYJzk0AAAAAAAAAAABYJDg3AQAAAAAAAAAAYJHg3AQAAAAAAAAAAIBFgnMTAAAAAAAAAAAAFgnOTQAAAAAAAAAAAFgkODcBAAAAAAAAAABgkeDcBAAAAAAAAAAAgEWCcxMAAAAAAAAAAAAWCc5NAAAAAAAAAAAAWCQ4NwEAAAAAAAAAAGCR4NwEAAAAAAAAAACARYJzEwAAAAAAAAAAABYJzk0AAAAAAAAAAABYJDg3AQAAAAAAAAAAYJHg3AQAAAAAAAAAAIBFgnMTAAAAAAAA4P/fjh3QAAAAIAzqn1pzfIMYAJAkNwEAAACAJLkJAAAAACTJTQAAAAAgSW4CAAAAAElyEwAAAABIkpsAAAAAQJLcBAAAAACS5CYAAAAAkCQ3AQAAAIAkuQkAAAAAJMlNAAAAACBJbgIAAAAASXITAAAAAEiSmwAAAABAktwEAAAAAJLkJgAAAACQJDcBAAAAgCS5CQAAAAAkyU0AAAAAIEluAgAAAABJchMAAAAASJKbAAAAAECS3AQAAAAAkuQmAAAAAJAkNwEAAACAJLkJAAAAACTJTQAAAAAgSW4CAAAAAElyEwAAAABIkpsAAAAAQJLcBAAAAACS5CYAAAAAkCQ3AQAAAIAkuQkAAAAAJMlNAAAAACBJbgIAAAAASXITAAAAAEiSmwAAAABA0HbVwqwTv41EeQAAAABJRU5ErkJggg==" + } + }, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Background\n", + "\n", + "Microsoft provides multiple montetary and resource investments to enterprice customers in support of products adoption, the sales manager would like to know which of these programs (\"investments\") are more successful than others? Specifically, we are interested in identifying the average treatment effect of each investment at some period $t$, on the cumulative outcome in the subsequent $m$ months. \n", + "\n", + "There are a few challenges to answer this question. First of all, we haven't fully observed the long-term revenue yet and we don't want to wait that long to evaluate a program. In addition, a careful causal modeling is required to correctly attribute the long-term ROI of multiple programs in a holistic manner, avoiding the biased estimate coming from confounding effect or double counting issues. \n", + "\n", + "The causal graph below shows how to frame this problem:\n", + "\n", + "![causal_graph.PNG](attachment:causal_graph.PNG)\n", + "\n", + "**Methodology:** Our proposed adjusted surrogate index approach could address all the chanllenges above by assuming the long-term effect is channeled through some short-term observed surrogates and employing a dynamic adjustment step (`DynamicDML`) to the surrogate model in order to get rid of the effect from future investment, finally applying double machine learning (`DML`) techniques to estimate the ROI. \n", + "\n", + "The pipeline below tells you how to solve this problem step by step:\n", + "![pipeline.PNG](attachment:pipeline.PNG)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 2;\n", + " var nbb_formatted_code = \"# imports\\nfrom econml.data.dynamic_panel_dgp import SemiSynthetic\\nfrom sklearn.linear_model import LassoCV, MultiTaskLassoCV\\nimport numpy as np\\nimport matplotlib.pyplot as plt\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# imports\n", + "from econml.data.dynamic_panel_dgp import SemiSynthetic\n", + "from sklearn.linear_model import LassoCV, MultiTaskLassoCV\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Data\n", + "\n", + "The **semi-synthetic data*** is comprised of 4 components:\n", + " * **Surrogates:** short-term metrics that could represent long-term revenue\n", + " * **Treatments:** different types of monetary investments to the end customers\n", + " * **Outcomes:** cumulative long-term revenue\n", + " * **Controls:** lagged surrogates and treatments, other time-invariant controls (e.g. demographics)\n", + "\n", + "To build the semi-synthetic data we estimate a series of moments from a real-world dataset: a full covariance matrix of\n", + "all surrogates, treatments, and controls in one period and a series of linear prediction models (lassoCV) of each surrogate and\n", + "treatment on a set of 6 lags of each treatment, 6 lags of each surrogate, and time-invariant controls. Using these values, we draw new parameters from distributions matching the key characteristics of each family of parameters. Finally, we use these new\n", + "parameters to simulate surrogates, treatments, and controls by drawing a set of initial values from the covariance matrix and\n", + "forward simulating to match intertemporal relationships from the transformed prediction models. We use one surrogate to be the outcome of interests. Then we consider the effect of each treatment in period $t$ on the cumulative sum of outcome from following 4 periods. We can calculate the true treatment effects in the semi-synthetic data as a function of parameters from the linear prediction models.\n", + "\n", + "The input data is in a **panel format**. Each panel corresponds to one company and the different rows in a panel correspond to different time period. \n", + "\n", + "Example:\n", + "\n", + "||Company|Year|Features|Controls/Surrogates|T1|T2|T3|AdjRev|\n", + "|---|---|---|---|---|---|---|---|---|\n", + "|1|A|2018|...|...|\\$1,000|...|...|\\$10,000|\n", + "|2|A|2019|...|...|\\$2,000|...|...|\\$12,000|\n", + "|3|A|2020|...|...|\\$3,000|...|...|\\$15,000|\n", + "|4|A|2021|...|...|\\$3,000|...|...|\\$18,000|\n", + "|5|B|2018|...|...|\\$0|...|...|\\$5,000|\n", + "|6|B|2019|...|...|\\$1,000|...|...|\\$10,000|\n", + "|7|B|2020|...|...|\\$0|...|...|\\$7,000|\n", + "|8|B|2021|...|...|\\$1,200|...|...|\\$12,000|\n", + "|9|C|2018|...|...|\\$1,000|...|...|\\$20,000|\n", + "|10|C|2019|...|...|\\$1,500|...|...|\\$25,000|\n", + "|11|C|2020|...|...|\\$500|...|...|\\$18,000|\n", + "|12|C|2021|...|...|\\$500|...|...|\\$20,000|\n", + " \n", + " **For confidentiality reason, the data used in this case study is synthetically generated and the feature distributions don't exactly correspond to real distributions.*" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 3;\n", + " var nbb_formatted_code = \"# generate historical dataset (training purpose)\\nnp.random.seed(43)\\ndgp = SemiSynthetic()\\ndgp.create_instance()\\nn_periods = 4\\nn_units = 5000\\nn_treatments = dgp.n_treatments\\nrandom_seed = 43\\nthetas = np.random.uniform(0, 2, size=(dgp.n_proxies, n_treatments))\\n\\npanelX, panelT, panelY, panelGroups, true_effect = dgp.gen_data(\\n n_units, n_periods, thetas, random_seed\\n)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# generate historical dataset (training purpose)\n", + "np.random.seed(43)\n", + "dgp = SemiSynthetic()\n", + "dgp.create_instance()\n", + "n_periods = 4\n", + "n_units = 5000\n", + "n_treatments = dgp.n_treatments\n", + "random_seed = 43\n", + "thetas = np.random.uniform(0, 2, size=(dgp.n_proxies, n_treatments))\n", + "\n", + "panelX, panelT, panelY, panelGroups, true_effect = dgp.gen_data(\n", + " n_units, n_periods, thetas, random_seed\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Outcome shape: (5000, 4)\n", + "Treatment shape: (5000, 4, 3)\n", + "Controls shape: (5000, 4, 71)\n" + ] + }, + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 4;\n", + " var nbb_formatted_code = \"# print panel data shape\\nprint(\\\"Outcome shape: \\\", panelY.shape)\\nprint(\\\"Treatment shape: \\\", panelT.shape)\\nprint(\\\"Controls shape: \\\", panelX.shape)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# print panel data shape\n", + "print(\"Outcome shape: \", panelY.shape)\n", + "print(\"Treatment shape: \", panelT.shape)\n", + "print(\"Controls shape: \", panelX.shape)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 5;\n", + " var nbb_formatted_code = \"# generate new dataset (testing purpose)\\nthetas_new = np.random.uniform(0, 2, size=(dgp.n_proxies, n_treatments))\\npanelXnew, panelTnew, panelYnew, panelGroupsnew, true_effect_new = dgp.gen_data(\\n n_units, n_periods, thetas_new, random_seed\\n)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# generate new dataset (testing purpose)\n", + "thetas_new = np.random.uniform(0, 2, size=(dgp.n_proxies, n_treatments))\n", + "panelXnew, panelTnew, panelYnew, panelGroupsnew, true_effect_new = dgp.gen_data(\n", + " n_units, n_periods, thetas_new, random_seed\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True Long-term Effect for each investment: [0.90994672 0.709811 2.45310877]\n" + ] + }, + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 6;\n", + " var nbb_formatted_code = \"# print true long term effect\\ntrue_longterm_effect = np.sum(true_effect_new, axis=0)\\nprint(\\\"True Long-term Effect for each investment: \\\", true_longterm_effect)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# print true long term effect\n", + "true_longterm_effect = np.sum(true_effect_new, axis=0)\n", + "print(\"True Long-term Effect for each investment: \", true_longterm_effect)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Do Dynamic Adjustment with EconML\n", + "From the causal graph above, we could see we want to first remove the effects of future incentives from the historical outcomes to create an **adjusted long-term revenue** as if those future incentives never happened.\n", + "\n", + "EconML's `DynamicDML` estimator is an extension of Double Machine Learning approach to **dynamically estimate the period effect of treatments assigned sequentially over time period**. In this scenario, it could help us to adjust the cumulative revenue by subtracting the period effect of all of the investments after the target investment.\n", + "\n", + "For more details about `DynamicDML`, please read this [paper](https://arxiv.org/pdf/2002.07285.pdf). " + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 7;\n", + " var nbb_formatted_code = \"# Helper function to reshape the panel data\\ndef long(x): # reshape the panel data to (n_units * n_periods, -1)\\n n_units = x.shape[0]\\n n_periods = x.shape[1]\\n return (\\n x.reshape(n_units * n_periods)\\n if np.ndim(x) == 2\\n else x.reshape(n_units * n_periods, -1)\\n )\\n\\n\\ndef wide(x): # reshape the panel data to (n_units, n_periods * d_x)\\n n_units = x.shape[0]\\n return x.reshape(n_units, -1)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Helper function to reshape the panel data\n", + "def long(x): # reshape the panel data to (n_units * n_periods, -1)\n", + " n_units = x.shape[0]\n", + " n_periods = x.shape[1]\n", + " return (\n", + " x.reshape(n_units * n_periods)\n", + " if np.ndim(x) == 2\n", + " else x.reshape(n_units * n_periods, -1)\n", + " )\n", + "\n", + "\n", + "def wide(x): # reshape the panel data to (n_units, n_periods * d_x)\n", + " n_units = x.shape[0]\n", + " return x.reshape(n_units, -1)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 8;\n", + " var nbb_formatted_code = \"# on historical data construct adjusted outcomes\\nfrom econml.dynamic.dml import DynamicDML\\n\\npanelYadj = panelY.copy()\\n\\nest = DynamicDML(\\n model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=2\\n)\\nfor t in range(1, n_periods): # for each target period 1...m\\n # learn period effect for each period treatment on target period t\\n est.fit(\\n long(panelY[:, 1 : t + 1]),\\n long(panelT[:, 1 : t + 1, :]), # reshape data to long format\\n X=None,\\n W=long(panelX[:, 1 : t + 1, :]),\\n groups=long(panelGroups[:, 1 : t + 1]),\\n )\\n # remove effect of observed treatments\\n T1 = wide(panelT[:, 1 : t + 1, :])\\n panelYadj[:, t] = panelY[:, t] - est.effect(\\n T0=np.zeros_like(T1), T1=T1\\n ) # reshape data to wide format\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# on historical data construct adjusted outcomes\n", + "from econml.dynamic.dml import DynamicDML\n", + "\n", + "panelYadj = panelY.copy()\n", + "\n", + "est = DynamicDML(\n", + " model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=2\n", + ")\n", + "for t in range(1, n_periods): # for each target period 1...m\n", + " # learn period effect for each period treatment on target period t\n", + " est.fit(\n", + " long(panelY[:, 1 : t + 1]),\n", + " long(panelT[:, 1 : t + 1, :]), # reshape data to long format\n", + " X=None,\n", + " W=long(panelX[:, 1 : t + 1, :]),\n", + " groups=long(panelGroups[:, 1 : t + 1]),\n", + " )\n", + " # remove effect of observed treatments\n", + " T1 = wide(panelT[:, 1 : t + 1, :])\n", + " panelYadj[:, t] = panelY[:, t] - est.effect(\n", + " T0=np.zeros_like(T1), T1=T1\n", + " ) # reshape data to wide format" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Train Surrogate Index\n", + "Once we have the adjusted outcome, we'd like to train any ML model to learn the relationship between short-term surrogates and long-term revenue from the historical dataset, assuming the treatment effect of investments on long-term revenue could **only** go through short-term surrogates, and the **relationship keeps the same** between the historical dataset and the new dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 9;\n", + " var nbb_formatted_code = \"# train surrogate index on historical dataset\\nXS = np.hstack(\\n [panelX[:, 1], panelYadj[:, :1]]\\n) # concatenate controls and surrogates from historical dataset\\nTotalYadj = np.sum(panelYadj, axis=1) # total revenue from historical dataset\\nadjusted_proxy_model = LassoCV().fit(\\n XS, TotalYadj\\n) # train proxy model from historical dataset\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# train surrogate index on historical dataset\n", + "XS = np.hstack(\n", + " [panelX[:, 1], panelYadj[:, :1]]\n", + ") # concatenate controls and surrogates from historical dataset\n", + "TotalYadj = np.sum(panelYadj, axis=1) # total revenue from historical dataset\n", + "adjusted_proxy_model = LassoCV().fit(\n", + " XS, TotalYadj\n", + ") # train proxy model from historical dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 10;\n", + " var nbb_formatted_code = \"# predict new long term revenue\\nXSnew = np.hstack(\\n [panelXnew[:, 1], panelYnew[:, :1]]\\n) # concatenate controls and surrogates from new dataset\\nsindex_adj = adjusted_proxy_model.predict(XSnew)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# predict new long term revenue\n", + "XSnew = np.hstack(\n", + " [panelXnew[:, 1], panelYnew[:, :1]]\n", + ") # concatenate controls and surrogates from new dataset\n", + "sindex_adj = adjusted_proxy_model.predict(XSnew)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Run DML to Learn ROI with EconML\n", + "Finally we will call `LinearDML` estimator from EconML to learn the treatment effect of multiple investments on the adjusted surrogate index in new dataset. `LinearDML` is a two stage machine learning models for estimating **(heterogeneous) treatment effects** when all potential confounders are observed, it leverages the machine learning power to deal with **high dimensional dataset** and still be able to construct **confidence intervals**. \n", + "\n", + "For more details, please read this [paper](https://arxiv.org/pdf/1608.00060.pdf). " + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True Long-term Effect for each investment: [0.90994672 0.709811 2.45310877]\n", + "Coefficient Results: X is None, please call intercept_inference to learn the constant!\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
CATE Intercept Results
point_estimate stderr zstat pvalue ci_lower ci_upper
cate_intercept|T0 0.83 0.015 57.214 0.0 0.802 0.858
cate_intercept|T1 0.677 0.028 23.767 0.0 0.621 0.733
cate_intercept|T2 2.438 0.035 69.711 0.0 2.369 2.507


A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:
$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$
where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter." + ], + "text/plain": [ + "\n", + "\"\"\"\n", + " CATE Intercept Results \n", + "=======================================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "-----------------------------------------------------------------------\n", + "cate_intercept|T0 0.83 0.015 57.214 0.0 0.802 0.858\n", + "cate_intercept|T1 0.677 0.028 23.767 0.0 0.621 0.733\n", + "cate_intercept|T2 2.438 0.035 69.711 0.0 2.369 2.507\n", + "-----------------------------------------------------------------------\n", + "\n", + "A linear parametric conditional average treatment effect (CATE) model was fitted:\n", + "$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$\n", + "where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:\n", + "$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$\n", + "where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter.\n", + "\"\"\"" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 11;\n", + " var nbb_formatted_code = \"# learn treatment effect on surrogate index on new dataset\\nfrom econml.dml import LinearDML\\n\\nadjsurr_est = LinearDML(\\n model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=3\\n)\\n# fit treatment_0 on total revenue from new dataset\\nadjsurr_est.fit(sindex_adj, panelTnew[:, 0], X=None, W=panelXnew[:, 0])\\n# print treatment effect summary\\nprint(\\\"True Long-term Effect for each investment: \\\", true_longterm_effect)\\nadjsurr_est.summary(alpha=0.05)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# learn treatment effect on surrogate index on new dataset\n", + "from econml.dml import LinearDML\n", + "\n", + "adjsurr_est = LinearDML(\n", + " model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=3\n", + ")\n", + "# fit treatment_0 on total revenue from new dataset\n", + "adjsurr_est.fit(sindex_adj, panelTnew[:, 0], X=None, W=panelXnew[:, 0])\n", + "# print treatment effect summary\n", + "print(\"True Long-term Effect for each investment: \", true_longterm_effect)\n", + "adjsurr_est.summary(alpha=0.05)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 12;\n", + " var nbb_formatted_code = \"# save the treatment effect and confidence interval\\nadjsurr_point_est = adjsurr_est.intercept_\\nadjsurr_conf_int_lb, adjsurr_conf_int_ub = adjsurr_est.intercept__interval(alpha=0.05)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# save the treatment effect and confidence interval\n", + "adjsurr_point_est = adjsurr_est.intercept_\n", + "adjsurr_conf_int_lb, adjsurr_conf_int_ub = adjsurr_est.intercept__interval(alpha=0.05)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Model Evaluation\n", + "Now we want to compare the proposed **adjusted surrogate index** approach with estimation from realized long-term outcome. Below we train another `LinearDML` model on the realized cumulative revenue directly, without any adjustment. And then we visualize the two models output, comparing with the ground truth." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True Long-term Effect for each investment: [0.90994672 0.709811 2.45310877]\n", + "Coefficient Results: X is None, please call intercept_inference to learn the constant!\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
CATE Intercept Results
point_estimate stderr zstat pvalue ci_lower ci_upper
cate_intercept|T0 2.227 0.039 56.865 0.0 2.15 2.304
cate_intercept|T1 1.561 0.226 6.911 0.0 1.118 2.004
cate_intercept|T2 4.335 0.209 20.748 0.0 3.926 4.745


A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:
$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$
where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter." + ], + "text/plain": [ + "\n", + "\"\"\"\n", + " CATE Intercept Results \n", + "=======================================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "-----------------------------------------------------------------------\n", + "cate_intercept|T0 2.227 0.039 56.865 0.0 2.15 2.304\n", + "cate_intercept|T1 1.561 0.226 6.911 0.0 1.118 2.004\n", + "cate_intercept|T2 4.335 0.209 20.748 0.0 3.926 4.745\n", + "-----------------------------------------------------------------------\n", + "\n", + "A linear parametric conditional average treatment effect (CATE) model was fitted:\n", + "$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$\n", + "where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:\n", + "$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$\n", + "where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter.\n", + "\"\"\"" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 13;\n", + " var nbb_formatted_code = \"# learn treatment effect on direct outcome\\nfrom econml.dml import LinearDML\\n\\ndirect_est = LinearDML(\\n model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=3\\n)\\n# fit treatment_0 on total revenue from new dataset\\ndirect_est.fit(np.sum(panelYnew, axis=1), panelTnew[:, 0], X=None, W=panelXnew[:, 0])\\n# print treatment effect summary\\nprint(\\\"True Long-term Effect for each investment: \\\", true_longterm_effect)\\ndirect_est.summary(alpha=0.05)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# learn treatment effect on direct outcome\n", + "from econml.dml import LinearDML\n", + "\n", + "direct_est = LinearDML(\n", + " model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=3\n", + ")\n", + "# fit treatment_0 on total revenue from new dataset\n", + "direct_est.fit(np.sum(panelYnew, axis=1), panelTnew[:, 0], X=None, W=panelXnew[:, 0])\n", + "# print treatment effect summary\n", + "print(\"True Long-term Effect for each investment: \", true_longterm_effect)\n", + "direct_est.summary(alpha=0.05)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 14;\n", + " var nbb_formatted_code = \"# save the treatment effect and confidence interval\\ndirect_point_est = direct_est.intercept_\\ndirect_conf_int_lb, direct_conf_int_ub = direct_est.intercept__interval(alpha=0.05)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# save the treatment effect and confidence interval\n", + "direct_point_est = direct_est.intercept_\n", + "direct_conf_int_lb, direct_conf_int_ub = direct_est.intercept__interval(alpha=0.05)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0.5, 0.98, 'Error bar plot of treatment effect from different models')" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABCYAAAGQCAYAAAByAGHwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABCPUlEQVR4nO3deZxU9Z3v/9dHaGw2FRSXsHtvTFBsFtHguIzLgCRqoj8yRGOMl4khZkxkzOQmZhnELMY8kosjxtFrEiGZGIwDmsRgjGhERGMikBYVHPQiskQFaZRFWrbv748q2uqmd7o4vbyej0c9us73bJ86XQXfftc53xMpJSRJkiRJkrJwUNYFSJIkSZKkjstgQpIkSZIkZcZgQpIkSZIkZcZgQpIkSZIkZcZgQpIkSZIkZcZgQpIkSZIkZcZgQpLUIiJifkRcmdG+B0VEiojOB2BfXSPigYh4OyL+q9j7a29qO34R8Z2IeDMiXj/Q+85SREyNiF/knw+IiK0R0Sk/fVRELIiILRHxfyJnRkRsioi/ZFv5gRURZ0XE2kYuW3VMJUltR9E7cJKk4ouIVcBRwO6C5pkppS9kU1HrFRFTgf+ZUvpUMzfxcXLH+vCU0q4ibL9eETEIeAUoqW3/xRYRM4G1KaVvNnMT1Y5fRAwA/hUYmFJavx91nQX8IqXUr7H7bu6+iiGltBroUdA0CXgTOCSllCLiDGAM0C+ltO1A1pb1e06S1P4ZTEhS+3FhSumRhhaKiM41/7iIiE4ppd11rVPLNpq0fFMVe/v7aSCworl/oEVEAJFS2tOyZbUZNY/fAGDj/oQS+7Hvamr7bGRoILAspZQKplc1J5RoZa9LkqR9eCmHJLVzEfG/IuLJiLg5IjYCUyNiZkTcHhEPRsQ24OyIGJK/HOOtiHghIj5asI19lq9jd/8jIv4SEZsj4jcR0btgG/8VEa/nT6NfEBEnNGX7+dq+V9f2ayz7voj4bURURMTLEfHZfPs44OvAJ/KnzT9bx/q1HouIuAGYUrD+Z2qsV+v289v6bkQ8CbwDHBsRH4yIefka/zsiJhRs5/yI+Gv+da7Jn4Wx14L8z7fy+zi1xu/4rYhYGRF/l29fExHrI+KKgu0fHBE/jIjVEfFGRNwREV3z886KiLUR8a/59V6LiIn5eZOAy4Cv5Pf9QB3Hr9bXVsvx+xwwD3hffnpmfrnREfFU/rU8mz8bYu+2e0fukoa/Re6yhl9HRHfg9wXb2RoR76tR0z6/uzo+G4dGxM8jYkNEvBoR34yIg/LbaNJxruW4DI6IxyN3ecY84IiCeVWXI+WPwxUFx/lzwE+AU/PTN+TXuSAiyvO1PBURZQXbWxURX42IpcC2/HbrO67zI+Lb+de3JSIejoi99e3znqvltU2N3Gf8F/n1n4uI4yLia/njsiYixhYsX+tnND+va+T+TdgUEcuAk2vs630RMSf/O3olIq6p43iX5uvZmH/Nz0TEUXX9fiRJGUop+fDhw4ePNv4AVgH/UMe8/wXsAr5I7ky5rsBM4G3gNHIhdU/gZXJ/VHcBzgG2AB/Ib6Pm8qW17Gc+sA4YCnQH5pA7tX7v/H/K7+dg4N+B8oJ5+7V9YBCQgM756QXAfwClwHBgA3BOft7Uwrpq2U9JA8eiofX3mZ+vfTVwQv53cCiwBpiYnx5B7rT94/PLnwWcmD8WZcAbwEW1vdYav+OJQCfgO/n93ZY/3mPzr6FHfvmbgd8CvfO/kweA7xXsexfwrfyx+Ai5MKVXwe/qO/W8/u4NvLZqxye/v7UF032Bjfn9HkTu8oWNQJ/8/LnAr4Be+fr+vrbtNOZ3Q+2fjZ8Dv8kfl0HACuAzzTnOtez/T8C0/LJn5pet6z1c7Tjn972wYHoEsB74UL6WK8j9O3Bwwb8J5UD//Otq6LjOB/4fcFx++fnATXW95+o4tpXAeflj+XNyl398I/97+izwSsHy9X1GbwKeIPf+7A88v/d3m699MbmQqQtwLLASOK/m7xj4HLn3drf8MTqJ3KUxmf+b7cOHDx8+qj88Y0KS2o9f578V3Pv4bMG8v6WUbk0p7Uopbc+3/Sal9GTKXVIwnNz17TellHaklP4I/A64tGAbVcunlCrrqOE/U0rPp9zp5v8GTIj8YH4ppbtSSltSSu+S++NhWEQc2lLb3ysi+pMLOL6aUqpMKZWT+7b503UfumpGN+JYNMfMlNILKXdK/Thyp+XPyP9O/kouaPlHgJTS/JTSc/ljsRSYBfx9A9t/Jb+93eT+cO8PfCul9G5K6WFgB/A/IyLIjV9wbUqpIqW0BbgRuKRgWzvz6+5MKT0IbAU+0MjXeUF9r60RPgU8mFJ6MP/65wGLgI9ExDHAh4GrUkqb8vU93sjt1qXqs0HuGF0CfC3/Xl0F/B/g8oLlG3Wca+4kcmNpnAz8W37ZBeT+aG6uScD/TSn9OaW0O6X0M+Bdcu/fvaanlNbkP/N1HteC5WeklFbkl7+X3L8LTfFESukP+WP5X0Afcp+jncA9wKCIOKwRn9EJwHfz7881wPSCfZxMLkz5Vv7zuRL4MdXfv3vtBA4nN+bL7pTS4pTS5ia+JknSAeAYE5LUflyU6h5jYk0Dbe8D1qTq4x68Su5b1vq2Ud82XyX3TekREfEm8F1yf5z2Afbu5whyZ0rs1/ZrLPM+YO8f3IXLjmrE9veu39CxaI7C2gcCH4qItwraOgP/CRARHyL3rfFQct8KH0zuD736vFHwfDtASqlmWw9yx78bsDiXUQAQ5L5R3mtjqj4mwTtUH5ixPvW+tkau/48RcWFBWwnwGLkQoCKltKmR22qMwt/LEfl9vVrQVvN339jjXNP7gE2p+hgRr5J7Tc0xELgiIr5Y0NYlv5+9ar7n6jquexXeFaUpv/O9ah6HN9N7Y8XsDUR70PBn9H3s+1nfayC5S3beKmjrRO4Mi5r+k9zxvSciDgN+AXwjH5RIkloRgwlJ6hhSA21/A/pHxEEFf5APIHcae33bqKnwj6wB5L6xfBP4JPAx4B/InWJ+KLCJ3B/ELbH9wva/Ab0jomfBHz4DyF0G0pj9NOZY1Keu7Re2rwEeTymNqWPZXwI/Aj6cUqqMiH/nvQCmMcepPm+S+yPxhJTSuoYWrkVD+2/otTVkDbkzYz5bc0b+jIneEXFYSumtJtZVl8L13iT3nhoILMu3Fb539sdrQK+I6F4QTgyg+XWvIXdWwXfrWabme67W49oI+/ueq6mhz+hr5D7TLxTM22sNubNW3t/QTvIBxA3ADZG7s8iDwH8DP93vVyBJalFeyiFJAvgzuW9IvxIRJflB8S4kd/p1U3wqIo6PiG7kxiiYnf/GtCe508w3kvu2/sZm1lnX9qvkT/1+CvhefvC7MuAz5L4thdy3uoP2DmhYi/09Fg1tH3KXhhwXEZfn91ESESdHxJD8/J7kvlGujIhTyAU7e20gd8bJsY2sp5p82PJj4OaIOBIgIvpGxHmN3MQbDey7odfWkF8AF0bEeRHRKf87PCsi+qWUXiM3yOV/RESv/LbPLKjr8BqXBzVJ/r10L/DdiOgZEQOBL/Hee6fZUkqvkrt04oaI6BIRp5N7XzXXj4GrIuJDkdM9coOm9qxj+TqPayP2tV/vuZoa8Rm9F/ha/nfcj9wYIHv9BdgSuYE9u+Zfy9CIqDZAJkBEnB0RJ+Yv99pMLnTqqHfDkaRWzWBCktqPB+K9OxJsjYj7G7tiSmkHuT+SPkzuW+P/AD6dUnqxiTX8J7lB+14nN6jd3tHyf07udOx15L6JfrqJ221o+zVdSm7Avr8B9wPXF1zmsveSiI0RsaTmii1wLOrdfn4fW8gNlHhJvsbXge+Tu2QD4J+Bb0XEFnKD/N1bsO475C6LeTI/lkjhmAKN9VVyA3w+HRGbgUdo/BgSPwWOz+/71814bfXK/9H6MXKDj24g9w35/+a9Psvl5P7AfJHc4I//kl/vRXJjcazM1/Y+mueLwDZyAyouJHf2yl3N3FZNnyQ3WGUFcD25z0WzpJQWkRtQ8kfkzj56mdwAmXUt39BxrW9fLfGeq6m+z+gN5P69eAV4mILLgPLh0QXkxr94hdxn9CfkzsKq6WhgNrlQYjnwOI2/pEiSdABFSi19dp4kSS0vIuaTG23/J1nXIkmSpJbjGROSJEmSJCkzBhOSJEmSJCkzXsohSZIkSZIy4xkTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpMwYTkiRJkiQpM52zLqAlHXHEEWnQoEFZlyFJUquyePHiN1NKfbKuoyOwLyJJUu3q64+0q2Bi0KBBLFq0KOsyJElqVSLi1axr6Cjsi0iSVLv6+iNeyiFJkiRJkjJjMCFJkiRJkjJjMCFJkiRJkjLTrsaYqM3OnTtZu3YtlZWVWZeiDqq0tJR+/fpRUlKSdSmSpAzYF1FrYH9EUmvW7oOJtWvX0rNnTwYNGkREZF2OOpiUEhs3bmTt2rUMHjw463IkSRmwL6Ks2R+R1Nq1+0s5KisrOfzww+0IKBMRweGHH+63ZJLUgdkXUdbsj0hq7dp9MAHYEVCmfP9Jkvy/QFnzPSipNesQwUTW3njjDT75yU9y7LHHctJJJ3Hqqady//33H9AaVq1axdChQ2tt/+Uvf9msbf77v/8777zzTtV0jx49ml2fJEkqHvsikqTWzGCihvLVm/i3Xz/P/7rrL/zbr5+nfPWm/dpeSomLLrqIM888k5UrV7J48WLuuece1q5du8+yu3bt2q99NUd9nYGG6qnZGZAkSfvPvsh77ItIUsdgMFGgfPUmps1bQcW2HRx1aCkV23Ywbd6K/eoQ/PGPf6RLly5cddVVVW0DBw7ki1/8IgAzZ87kox/9KOeccw7nnnsuFRUVXHTRRZSVlTF69GiWLl0KwNSpU/nhD39YtY2hQ4eyatUqVq1axZAhQ/jsZz/LCSecwNixY9m+fTsAixcvZtiwYQwbNozbbrut1vquu+46nnjiCYYPH87NN9+8Tz3z58/nggsuqFr+C1/4AjNnzmT69On87W9/4+yzz+bss8+umv+Nb3yDYcOGMXr0aN54441mHzdJkjoi+yL2RSSpIypaMBER/SPisYhYFhEvRMTkWpY5KyLejojy/GNKwbxxEfHfEfFyRFxXrDoLzVmyjp6lJRzStYSDIjikawk9S0uYs2Rds7f5wgsvMHLkyHqXWbJkCbNnz+bxxx/n+uuvZ8SIESxdupQbb7yRT3/60w3u46WXXuLqq6/mhRde4LDDDmPOnDkATJw4kVtvvZVnn322znVvuukmzjjjDMrLy7n22mv3qacu11xzDe973/t47LHHeOyxxwDYtm0bo0eP5tlnn+XMM8/kxz/+cYO1S5Kk99gXsS8iSR1RMc+Y2AX8a0rpeGA0cHVEHF/Lck+klIbnH98CiIhOwG3Ah4HjgUvrWLdFral4hx6l1e+g2qO0M2sqWu4Uwauvvpphw4Zx8sknV7WNGTOG3r17A7Bw4UIuv/xyAM455xw2btzI5s2b693m4MGDGT58OAAnnXQSq1at4q233uKtt97izDPPBKjaZmMU1tMUXbp0qfpGY28dkiSp8eyL7FtPU9gXkaS2qWjBRErptZTSkvzzLcByoG8jVz8FeDmltDKltAO4B/hYcSp9T//e3dhaWf1axq2Vu+jfu1uzt3nCCSewZMmSqunbbruNRx99lA0bNlS1de/evcHtdO7cmT179lRNF97u6eCDD6563qlTp/2+PrSwnvr2W1NJSUnViM8tUYckdTgzzs891GHZF9m3HvsiknRgTXxoIhMfmnhA93lAxpiIiEHACODPtcw+NSKejYjfR8QJ+ba+wJqCZdZSR6gREZMiYlFELCr8D7Y5xo/sy5bKnWzevpM9KbF5+062VO5k/MjG5in7Ouecc6isrOT222+vaqtvkKYzzjiDu+++G4D58+dzxBFHcMghhzBo0KCqTsWSJUt45ZVX6t3vYYcdxmGHHcbChQsBqrZZU8+ePdmyZUud2xk4cCDLli3j3Xff5a233uLRRx9t9LqSJKlp7Ivsy76IJLV/RQ8mIqIHMAf4l5RSzfMAlwADU0rDgFuBXzd1+ymlO1NKo1JKo/r06bNftQ4f0IsvjTmO3t278MbblfTu3oUvjTmO4QN6NXubEcGvf/1rHn/8cQYPHswpp5zCFVdcwfe///1al586dSqLFy+mrKyM6667jp/97GcAjB8/noqKCk444QR+9KMfcdxxxzW47xkzZnD11VczfPhwUkq1LlNWVkanTp0YNmwYN9988z7z+/fvz4QJExg6dCgTJkxgxIgRVfMmTZrEuHHjqg04JUmSms++iH0RSeqIoq7/JFpk4xElwO+AP6SUpjVi+VXAKOD9wNSU0nn59q8BpJS+V9/6o0aNSosWLarWtnz5coYMGdKs+qWW4vtQUqPsvYxj4twW3WxELE4pjWrRjapW9kXUmvlelNQYey/jmDFuRotut77+SDHvyhHAT4HldYUSEXF0fjki4pR8PRuBZ4D3R8TgiOgCXAL8tli1SpIkSZKkbHRueJFmOw24HHguIsrzbV8HBgCklO4APg58PiJ2AduBS1LuFI5dEfEF4A9AJ+CulNILRaxVkiRJkiRloGjBREppIRANLPMj4Ed1zHsQeLAIpUmSdMCVr97EnCXrWFPxDv17d2P8yL77NW6AJElSe3FA7sohSVJHVr56E9PmraBi2w6OOrSUim07mDZvBeWrN2VdmiRJUuYMJiRJKrI5S9bRs7SEQ7qWcFAEh3QtoWdpCXOWrHtvoaX3wtpn4NWFcPPQ3LQkSVIHYDBRmxnnvzcyuiRJ+2lNxTv0KK1+9WSP0s6sqXgnN7H0XnjgGtj9bm767TW5acOJjsu+iCSpAzGYOEB+/etfExG8+OKLdS5z1llnsfcWYx/5yEd46623mryf+fPn89RTTzV5vUGDBvHmm2/u037XXXdx4oknUlZWxtChQ/nNb37T5G0Xy4033tjkdQqPcWP16NGjyfuRpEL9e3dja+Wuam1bK3fRv3e33MSj34Kd26uvtHN7rl1qIfZFWp59EUlqGQYTNRXpVNpZs2Zx+umnM2vWrEYt/+CDD3LYYYc1eT/N7QzUZu3atXz3u99l4cKFLF26lKeffpqysrJGr79r1656p/dXczoDkpSF8SP7sqVyJ5u372RPSmzevpMtlTsZP7JvboG319a+Yl3tat/si1SxLyJJHYPBRKEinUq7detWFi5cyE9/+lPuueeeqvbt27dzySWXMGTIEC6++GK2b3/v27K93xqsWrWKoUOHVrX/8Ic/ZOrUqQBMnz6d448/nrKyMi655BJWrVrFHXfcwc0338zw4cN54okn2LBhA+PHj+fkk0/m5JNP5sknnwRg48aNjB07lhNOOIErr7yS3F1aq1u/fj09e/asSul79OjB4MGDgepp/5tvvsmgQYMAmDlzJh/96Ec555xzOPfcc/eZrqio4KKLLqKsrIzRo0ezdOlSADZs2MCYMWOq6hk4cGDVtyYXXXQRJ510EieccAJ33nknANdddx3bt29n+PDhXHbZZQD84he/4JRTTmH48OF87nOfY/fu3fX+Xnr06ME3vvENhg0bxujRo3njjTcAeOWVVzj11FM58cQT+eY3v1ltnR/84AecfPLJlJWVcf311wNw//33c+6555JS4rXXXuO4447j9ddfr3ffkjqW4QN68aUxx9G7exfeeLuS3t278KUxx713V45D+9W+Yl3tar/si1RjX8S+iKSOwWCiUJFOpf3Nb37DuHHjOO644zj88MNZvHgxALfffjvdunVj+fLl3HDDDVXtjXXTTTfx17/+laVLl3LHHXcwaNAgrrrqKq699lrKy8s544wzmDx5Mtdeey3PPPMMc+bM4corrwTghhtu4PTTT+eFF17g4osvZvXq1ftsf9iwYRx11FEMHjyYiRMn8sADDzSqriVLljB79mwef/zxfaavv/56RowYwdKlS7nxxhv59Kc/XVXPOeecwwsvvMDHP/7xavXcddddLF68mEWLFjF9+nQ2btzITTfdRNeuXSkvL+fuu+9m+fLl/OpXv+LJJ5+kvLycTp06cffdd9db57Zt2xg9ejTPPvssZ555Jj/+8Y8BmDx5Mp///Od57rnnOOaYY6qWf/jhh3nppZf4y1/+Qnl5OYsXL2bBggVcfPHFHHPMMdx222189rOf5YYbbuDoo49u1LGS1HEMH9CLb180lJn/dArfvmho9VuFnjsFSrpWX6Gka65dHYt9kWrsi9gXkdQxdG54kQ6kSKfSzpo1i8mTJwNwySWXMGvWLE466SQWLFjANddcA0BZWVmTTk3cu85ll13GRRddxEUXXVTrMo888gjLli2rmt68eTNbt25lwYIF3HfffQCcf/759OrVa591O3XqxEMPPcQzzzzDo48+yrXXXsvixYurviWpy5gxY+jdu3et0wsXLmTOnDkAnHPOOWzcuJHNmzezcOFC7r//fgDGjRtXrZ7p06dXzVuzZg0vvfQShx9+eLV9PvrooyxevJiTTz4ZyH0DdOSRR9ZbZ5cuXbjgggsAOOmkk5g3bx4ATz75ZFWNl19+OV/96leBXGfg4YcfZsSIEUDu26eXXnqJM888k1tvvZWhQ4cyevRoLr300nr3K0n7KJuQ+/mbL+S+KT+0fy6U2NuujsO+SDX2ReyLSOoYDCYKHdovd8pkbe3NVFFRwR//+Eeee+45IoLdu3cTEfzgBz9o1PqdO3dmz549VdOVlZVVz+fOncuCBQt44IEH+O53v8tzzz23z/p79uzh6aefprS0tFn1RwSnnHIKp5xyCmPGjGHixIlMnTq1Wl2FNQF079693ummmD9/Po888gh/+tOf6NatG2edddY++wNIKXHFFVfwve99r9HbLikpISKAXMen8LrTve019/G1r32Nz33uc/vMW7t2LQcddBBvvPEGe/bs4aCDPBlJUhOVTYDFP8s9nzg321qUHfsi+7AvUn0f9kUktUf+i1WoCKfSzp49m8svv5xXX32VVatWsWbNGgYPHswTTzzBmWeeyS9/+UsAnn/++aprHAsdddRRrF+/no0bN/Luu+/yu9/9Dsj9J79mzRrOPvtsvv/97/P222+zdetWevbsyZYtW6rWHzt2LLfeemvVdHl5OUC1ff/+979n06ZN++z7b3/7G0uWLKm27sCBA4Hcdad7T/ecPXt2o4/HGWecUXVa4/z58zniiCM45JBDOO2007j33tz1sw8//HBVPW+//Ta9evWiW7duvPjiizz99NNV2yopKWHnzp0AnHvuucyePZv169cDuU7Yq6++2ui6Cp122mlV198WnoJ53nnncdddd7F161YA1q1bx/r169m1axf/9E//xKxZsxgyZAjTpk1r1n4lSbIvUp19EfsikjoGg4lCZRPgwunQ6eDc9KH9c9P7cSrtrFmzuPjii6u1jR8/nlmzZvH5z3+erVu3MmTIEKZMmcJJJ51UbbmIoKSkhClTplR9S/DBD34QgN27d/OpT32KE088kREjRnDNNddw2GGHceGFF3L//fdXDTg1ffp0Fi1aRFlZGccffzx33HEHANdffz0LFizghBNO4L777mPAgAH71L5z506+/OUv88EPfpDhw4fzq1/9iltuuQWAL3/5y9x+++2MGDGi1lt71WXq1KksXryYsrIyrrvuOn72s59V1fPwww8zdOhQ/uu//oujjz6anj17Mm7cOHbt2sWQIUO47rrrGD16dNW2Jk2aVHUK6fHHH893vvMdxo4dS1lZGWPGjOG1115rdF2FbrnlFm677TZOPPFE1q1bV9U+duxYPvnJT1YNRvXxj3+cLVu2cOONN3LGGWdw+umnM23aNH7yk5+wfPnyZu1bktTB2Repxr6IfRFJHUPUNgJyWzVq1KhU877Qy5cvZ8iQIU3b0Izzcz8zOpV29+7dHHnkkbz++uuUlJRkUsOB9u6779KpUyc6d+7Mn/70Jz7/+c9XfaPSHjTrfSip4ynS/z8RsTilNKpFN6pa2Rdpu9p7XwTsj0hqnIkPTQRgxrgZLbrd+vojjjFRm4yv7d17m6qO0hEAWL16NRMmTGDPnj106dKlalRqSZI6JPsiB5x9EUnKjsFEK/Tiiy9mXcIB9/73v5+//vWvWZchSZKwLyJJOrAcY0KSJEmSJGWmQ5wxkVKq9ZZL0oHQnsZxkVRk3iZUkiR1QO3+jInS0lI2btzoH4fKREqJjRs3Nvve7ZIkSZLU3rX7Myb69evH2rVr2bBhQ9alqIMqLS2lX79+WZchSWpDijUiuiRJrVG7DyZKSkoYPHhw1mVIkiRlplOnTpx44ons3LmTzp078+lPf5prr72Wgw46iEWLFvHzn/+c6dOn7/d+brzxRr7+9a/XOm/QoEH079+fJ554oqpt+PDh7Nq1i+eff77R+zjrrLP44Q9/yKhRdd8BtzHLSJJaj3Z/KYckSVJbMnflXJZuWMqiNxYxdvZY5q7c/7FHunbtSnl5OS+88ALz5s3j97//PTfccAMAo0aNqjWU2LVrV5P3c+ONN9Y7f8uWLaxZswaA5cuXN3n7kqT2yWBCkiSplZi7ci5Tn5rKjj07AHht22tMfWpqi4QTex155JHceeed/OhHPyKlxPz587ngggsAmDp1KpdffjmnnXYal19+ORs2bGD8+PGcfPLJnHzyyTz55JMAbN26lYkTJ3LiiSdSVlbGnDlzuO6669i+fTvDhw/nsssuq3XfEyZM4Fe/+hUAs2bN4tJLL62aV1lZWbXNESNG8NhjjwGwfft2LrnkEoYMGcLFF1/M9u3bq9Z5+OGHOfXUUxk5ciT/+I//yNatW1vsOElSR1SMcLwxDCYkSZJaiVuW3ELl7spqbZW7K7llyS0tup9jjz2W3bt3s379+n3mLVu2jEceeYRZs2YxefJkrr32Wp555hnmzJnDlVdeCcC3v/1tDj30UJ577jmWLl3KOeecw0033VR1Zsbdd99d637Hjx/PfffdB8ADDzzAhRdeWDXvtttuIyJ47rnnmDVrFldccQWVlZXcfvvtdOvWjeXLl3PDDTewePFiAN58802+853v8Mgjj7BkyRJGjRrFtGnTWvQ4SVJHciDC8bq0+zEmJEmS2orXt73epPZi+OhHP0rXrl0BeOSRR1i2bFnVvM2bN7N161YeeeQR7rnnnqr2Xr16NWrbhx9+OL169eKee+5hyJAhdOvWrWrewoUL+eIXvwjABz/4QQYOHMiKFStYsGAB11xzDQBlZWWUlZUB8PTTT7Ns2TJOO+00AHbs2MGpp566H69ckjq2+sLx8489v6j7NpiQJElqJY7ufjSvbXut1vaWtHLlSjp16sSRRx65z1gP3bt3r3q+Z88enn766Ra97fUnPvEJrr76ambOnLlf20kpMWbMGGbNmtUyhUlSB5dlOO6lHJIkSa3E5JGTKe1UPQQo7VTK5JGTW2wfGzZs4KqrruILX/gCEVHvsmPHjuXWW2+tmi4vLwdgzJgx3HbbbVXtmzZtAnJ3Q9u5c2e927z44ov5yle+wnnnnVet/Ywzzqi6BGTFihWsXr2aD3zgA5x55pn88pe/BOD5559n6dKlAIwePZonn3ySl19+GYBt27axYsWKhl6+JKkOdYXgLR2O18ZgQpIkqZU4/9jzmfp3U+lyUBcAjul+DFP/bup+n0K7d1DKE044gX/4h39g7NixXH/99Q2uN336dBYtWkRZWRnHH388d9xxBwDf/OY32bRpE0OHDmXYsGFVA1VOmjSJsrKyOge/BOjZsydf/epX6dKlS7X2f/7nf2bPnj2ceOKJfOITn2DmzJkcfPDBfP7zn2fr1q0MGTKEKVOmcNJJJwHQp08fZs6cyaWXXkpZWRmnnnoqL774YnMPkSR1eAciHK9LpJSKvpMDZdSoUWnRokVZlyFJUqsSEYtTSqOyrqMjqK0vsnz5coYMGdKk7Ux8aCIAM8bNaLHapOa8FyV1LHNXzmXKk1PYsWcHx3Q/hskjJ7fY+BL19UccY0KSJKmVMZCQJGXh/GPPZ/aK2cCB/b/ISzkkSZIkSVJmDCYkSZIkSVJmDCYkSZKKrD2N6aW2yfegpNasaMFERPSPiMciYllEvBAR+wzlGRGXRcTSiHguIp6KiGEF81bl28sjwhEtJUlSm1RaWsrGjRv9w1CZSSmxceNGSktLG15YkjJQzMEvdwH/mlJaEhE9gcURMS+ltKxgmVeAv08pbYqIDwN3Ah8qmH92SunNItYoSZJUVP369WPt2rVs2LAh61LUgZWWltKvX7+sy5CkWhUtmEgpvQa8ln++JSKWA32BZQXLPFWwytOA/1pKkqR2paSkhMGDB2ddhiRJrdYBGWMiIgYBI4A/17PYZ4DfF0wn4OGIWBwRk+rZ9qSIWBQRi/wmQpIkSZKktqWYl3IAEBE9gDnAv6SUNtexzNnkgonTC5pPTymti4gjgXkR8WJKaUHNdVNKd5K7BIRRo0Z58aYkSZIkSW1IUc+YiIgScqHE3Sml++pYpgz4CfCxlNLGve0ppXX5n+uB+4FTilmrJEmSJEk68Ip5V44AfgosTylNq2OZAcB9wOUppRUF7d3zA2YSEd2BscDzxapVkiRJkiRlo5iXcpwGXA48FxHl+bavAwMAUkp3AFOAw4H/yOUY7EopjQKOAu7Pt3UGfplSeqiItUqSJEmSpAwU864cC4FoYJkrgStraV8JDCtSaZIkSZIkqZU4IHflkCRJkiRJqo3BhCRJkiRJyozBhCRJkiRJyozBhCRJkiRJyozBhCRJUgMiolNE/DUifpd1LZIktTcGE5IkSQ2bDCzPughJktojgwlJkqR6REQ/4HzgJ1nXIklSe2QwIUmSVL9/B74C7KltZkRMiohFEbFow4YNB7QwSZLaA4MJSZKkOkTEBcD6lNLiupZJKd2ZUhqVUhrVp0+fA1idJEntg8GEJElS3U4DPhoRq4B7gHMi4hfZliRJUvtiMCFJklSHlNLXUkr9UkqDgEuAP6aUPpVxWZIktSsGE5IkSZIkKTOdsy5AkiSpLUgpzQfmZ1yGJElFNWPcjAO+T8+YkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmSlaMBER/SPisYhYFhEvRMTkWpaJiJgeES9HxNKIGFkw74qIeCn/uKJYdUqSJEmSpOx0LuK2dwH/mlJaEhE9gcURMS+ltKxgmQ8D788/PgTcDnwoInoD1wOjgJRf97cppU1FrFeSJEmSJB1gRTtjIqX0WkppSf75FmA50LfGYh8Dfp5yngYOi4hjgPOAeSmlinwYMQ8YV6xaJUmSJElSNg7IGBMRMQgYAfy5xqy+wJqC6bX5trraa9v2pIhYFBGLNmzY0GI1S5IkSZKk4it6MBERPYA5wL+klDa39PZTSnemlEallEb16dOnpTcvSZIkSZKKqKjBRESUkAsl7k4p3VfLIuuA/gXT/fJtdbVLkiRJkqR2pJh35Qjgp8DylNK0Ohb7LfDp/N05RgNvp5ReA/4AjI2IXhHRCxibb5MkSZIkSe1IMe/KcRpwOfBcRJTn274ODABIKd0BPAh8BHgZeAeYmJ9XERHfBp7Jr/etlFJFEWuVJEmSJEkZKFowkVJaCEQDyyTg6jrm3QXcVYTSJEmSJElSK3FA7sohSZIkSZJUG4MJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUGYMJSZIkSZKUmc7F2nBE3AVcAKxPKQ2tZf7/Bi4rqGMI0CelVBERq4AtwG5gV0ppVLHqlCRJkiRJ2SnmGRMzgXF1zUwp/SClNDylNBz4GvB4SqmiYJGz8/MNJSRJkiRJaqeKFkyklBYAFQ0umHMpMKtYtUiSJEmSpNYp8zEmIqIbuTMr5hQ0J+DhiFgcEZMaWH9SRCyKiEUbNmwoZqmSJEmSJKmFZR5MABcCT9a4jOP0lNJI4MPA1RFxZl0rp5TuTCmNSimN6tOnT7FrlSRJkiRJLag1BBOXUOMyjpTSuvzP9cD9wCkZ1CVJkiRJkoos02AiIg4F/h74TUFb94joufc5MBZ4PpsKJUmSJElSMRXzdqGzgLOAIyJiLXA9UAKQUrojv9jFwMMppW0Fqx4F3B8Re+v7ZUrpoWLVKUmSVJeIKAUWAAeT65fMTildn21VkiS1L0ULJlJKlzZimZnkbita2LYSGFacqiRJkprkXeCclNLWiCgBFkbE71NKT2ddmCRJ7UXRgglJkqS2LqWUgK35yZL8I2VXkSRJ7U9rGPxSkiSp1YqIThFRDqwH5qWU/lxjvrculyRpPxhMSJIk1SOltDulNBzoB5wSEUNrzPfW5ZIk7QeDCUmSpEZIKb0FPAaMy7gUSZLaFYMJSZKkOkREn4g4LP+8KzAGeDHToiRJamcc/FKSJKluxwA/i4hO5L7QuTel9LuMa5IkqV0xmJAkSapDSmkpMCLrOiRJas+8lEOSJEmSJGXGYEKSJEmSJGXGYEKSJEmSJGXGYEKSJEmSJGXGYEKSJEmSJGWmUcFERPxnY9okSZJaK/szkiS1To09Y+KEwon8vbxPavlyJEmSisb+jCRJrVC9wUREfC0itgBlEbE5/9gCrAd+c0AqlCRJ2g/2ZyRJat3qDSZSSt9LKfUEfpBSOiT/6JlSOjyl9LUDVKMkSVKz2Z+RJKl1a+ylHH+JiEP3TkTEYRFxUXFKkiRJKgr7M5IktUKNDSauTym9vXcipfQWcH1RKpIkSSoO+zOSJLVCjQ0maluuc0sWIkmSVGT2ZyRJaoUaG0wsiohpEfE/8o9pwOJiFiZJktTC7M9IktQKNTaY+CKwA/gVcA9QCVxdrKIkSZKKwP6MJEmtUKNOX0wpbQOui4ju+eeSJEltiv0ZSZJap0adMRERfxcRy4Dl+elhEfEfRa1MkiSpBdmfkSSpdWrspRw3A+cBGwFSSs8CZxarKEmSpCKwPyNJUivU2GCClNKaGk27W7gWSZKkorI/I0lS69PYW2StiYi/A1JElACTyZ8GKUmS1EbYn5EkqRVq7BkTV5EbtbovsA4YjqNYS5KktsX+jCRJrVC9Z0xExPdTSl8Fzk4pXXaAapIkSWox9mckSWrdGjpj4iMREcDXDkQxkiRJRWB/RpKkVqyhMSYeAjYBPSJiMxBA2vszpXRIkeuTJEnaX/ZnJElqxRo6Y+KbKaXDgLkppUNSSj0Lfx6A+iRJkvaX/RlJklqxhoKJP+V/bm7qhiPirohYHxHP1zH/rIh4OyLK848pBfPGRcR/R8TLEXFdU/ctSZJUoNn9GUmSVHwNXcrRJSI+CfxdRPx/NWemlO6rZ92ZwI+An9ezzBMppQsKGyKiE3AbMAZYCzwTEb9NKS1roFZJkqTa7E9/RpIkFVlDwcRVwGXAYcCFNeYloM7/yFNKCyJiUDNqOgV4OaW0EiAi7gE+BhhMSJKk5mh2f0aSJBVfvcFESmkhsDAiFqWUflqE/Z8aEc8CfwO+nFJ6gdy9xdcULLMW+FBdG4iIScAkgAEDBhShREmS1JYdgP6MJEnaD/WOMRERXwFIKf00Iv6xxrwb93PfS4CBKaVhwK3Ar5uzkZTSnSmlUSmlUX369NnPkiRJUntT5P6MJEnaTw0NfnlJwfOa9/4etz87TiltTiltzT9/ECiJiCOAdUD/gkX75dskSZKao2j9GUmS2pwZ5+cerUhDY0xEHc9rm26SiDgaeCOllCLiFHIhyUbgLeD9ETGYXCBxCfDJ/dmXJEnq0IrWn5EkqTUpX72JOUvWsabiHfr37sb4kX0ZPqBX1mU1qKEzJlIdz2ubriYiZpG7PdcHImJtRHwmIq6KiKvyi3wceD4/xsR04JKUswv4AvAHYDlwb37sCUmSpOZodn9GkqS2onz1JqbNW0HFth0cdWgpFdt2MG3eCspXb3pvoaX3wtpn4NWFcPPQ3HQr0NAZE8MiYjO5bxO65p+Tny6tb8WU0qUNzP8RuduJ1jbvQeDBBmqTJElqjGb3ZyRJaivmLFlHz9ISDulaAlD1c86SdbmzJpbeCw9cA7vfza3w9prcNEDZhCxKrtLQXTk6HahCJEmSisH+jCSpI1hT8Q5HHVo9b+9R2pk1Fe/kJh79FuzcXn2lndtz7RkHEw1dyiFJkiRJklq5/r27sbVyV7W2rZW76N+7W27i7bW1r1hX+wFkMCFJkiRJUhs3fmRftlTuZPP2nexJic3bd7KlcifjR/bNLXBov9pXrKv9ADKYkCRJkiSpjRs+oBdfGnMcvbt34Y23K+ndvQtfGnPce3flOHcKlHStvlJJ11x7xhoa/FKSJEmSJLUBwwf0qvv2oHvHkfjNF3IDYB7aPxdKZDy+BBhMSJIkSZLUMZRNgMU/yz2fODfbWgoYTEiSJEmS1FG0okBiL8eYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJkiRJmTGYkCRJaqcmPjSRiQ9NzLoMSZLqZTAhSZIkSZIyYzAhSZIkSZIyYzAhSZLUDs1dOZelG5ay6I1FjJ09lrkrW9996yVJAoMJSZKkdmfuyrlMfWoqO/bsAOC1ba8x9amphhOSpFbJYEKSJKmduWXJLVTurqzWVrm7kluW3JJRRZIk1c1gQpIkqZ15fdvrTWqXJClLBhOSJEntzNHdj25SuyRJWTKYkCRJamcmj5xMaafSam2lnUqZPHJyRhVJklS3zlkXIEmSpJZ1/rHnAzDlySns2LODY7ofw+SRk6vaJUlqTQwmJEmS2qHzjz2f2StmAzBj3IyMq5EkqW4GE5IkSe2UgYQkqS1wjAlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpSZogUTEXFXRKyPiOfrmH9ZRCyNiOci4qmIGFYwb1W+vTwiFhWrRkmSpPpERP+IeCwilkXECxExOeuaJElqb4p5xsRMYFw9818B/j6ldCLwbeDOGvPPTikNTymNKlJ9kiRJDdkF/GtK6XhgNHB1RByfcU2SJLUrRQsmUkoLgIp65j+VUtqUn3wa6FesWiRJkpojpfRaSmlJ/vkWYDnQN9uqJElqX1rLGBOfAX5fMJ2AhyNicURMqm/FiJgUEYsiYtGGDRuKWqQkSeq4ImIQMAL4c412+yJSGzTxoYlMfGhi1mVIohUEExFxNrlg4qsFzaenlEYCHyZ3yuSZda2fUrozpTQqpTSqT58+Ra5WkiR1RBHRA5gD/EtKaXPhPPsikiTtn0yDiYgoA34CfCyltHFve0ppXf7neuB+4JRsKpQkSR1dRJSQCyXuTindl3U9kiS1N5kFExExALgPuDyltKKgvXtE9Nz7HBgL1HpnD0mSpGKKiAB+CixPKU3Luh5JktqjzsXacETMAs4CjoiItcD1QAlASukOYApwOPAfuf/z2ZW/A8dRwP35ts7AL1NKDxWrTkmSpHqcBlwOPBcR5fm2r6eUHsyuJEmS2peiBRMppUsbmH8lcGUt7SuBYcWqS5IkqbFSSguByLoOSZLas8wHv5QkSZIkSR2XwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZIkScqMwYQkSZKkDmXuyrks3bCURW8sYuzsscxdOTfrkqQOzWBCkiRJUocxd+Vcpj41lR17dgDw2rbXmPrUVMMJKUMGE5IkSZI6jFuW3ELl7spqbZW7K7llyS0ZVSTJYEKSJElSh/H6tteb1C6p+AwmJEmSJHUYR3c/ukntkorPYEKSJElShzF55GRKO5VWayvtVMrkkZMzqkhS56wLkCRJkqQD5fxjzwdgypNT2LFnB8d0P4bJIydXtUs68AwmJEmSJHUo5x97PrNXzAZgxrgZGVcjqaiXckTEXRGxPiKer2N+RMT0iHg5IpZGxMiCeVdExEv5xxXFrFOSJEmSJGWj2GNMzATG1TP/w8D7849JwO0AEdEbuB74EHAKcH1E9CpqpZIkSZIk6YArajCRUloAVNSzyMeAn6ecp4HDIuIY4DxgXkqpIqW0CZhH/QGHJEmSJElqg7K+K0dfYE3B9Np8W13t+4iISRGxKCIWbdiwoWiFSpIkSZKklpd1MLHfUkp3ppRGpZRG9enTJ+tyJEmSJElSE2QdTKwD+hdM98u31dUuSZIkSZLakayDid8Cn87fnWM08HZK6TXgD8DYiOiVH/RybL5NkiRJkiS1I52LufGImAWcBRwREWvJ3WmjBCCldAfwIPAR4GXgHWBifl5FRHwbeCa/qW+llOobRFOSJEmSJLVBRQ0mUkqXNjA/AVfXMe8u4K5i1CVJkiRJklqHrC/lkCRJkiRJHZjBhCRJkiRJyozBhCRJkiRJyozBhCRJkiRJyozBhCRJkiRJyozBhCRJkiRJykxRbxfaVpWv3sScJetYU/EO/Xt3Y/zIvgwf0CvrsiRJkiRJanc8Y6KG8tWbmDZvBRXbdnDUoaVUbNvBtHkrKF+9KevSJEmSJElqdwwmapizZB09S0s4pGsJB0VwSNcSepaWMGfJuqxLkyRJkiSp3TGYqGFNxTv0KK1+hUuP0s6sqXgno4okSZIkSWq/DCZq6N+7G1srd1Vr21q5i/69u2VUkSRJkiRJ7ZfBRA3jR/ZlS+VONm/fyZ6U2Lx9J1sqdzJ+ZN+sS5MkSZIkqd0xmKhh+IBefGnMcfTu3oU33q6kd/cufGnMcd6VQ5IkSZKkIvB2obUYPqCXQYQkSZIkSQeAZ0xIkiRJkqTMGExIkiRJkqTMGExIkiRJkqTMGEw014zzcw9JkiRJktRsDn7ZHEvvhbXPwO534eahcO4UKJuQdVWSJEmSGmnGuBlZlyApzzMmmmrpvfDANblQAuDtNbnppfdmW5ckSZIkSW2QwURTPfot2Lm9etvO7bl2SZIkSZLUJAYTTfX22qa1S5IkSZKkOhlMNNWh/ZrWLkmSJEmS6mQw0VTnToGSrtXbSrrm2iVJUrsSEXdFxPqIeD7rWiRJaq8MJpqqbAJcOB06HZybPrR/btq7ckiS1B7NBMZlXYQkSe2ZtwttjrIJsPhnuecT52ZbiyRJKpqU0oKIGJR1HZIktWcGE81lICFJkiRJ0n7zUg5JkqT9EBGTImJRRCzasGFD1uVIktTmGExIkiTth5TSnSmlUSmlUX369Mm6HEmS2hyDCUmSJEmSlJmijjEREeOAW4BOwE9SSjfVmH8zcHZ+shtwZErpsPy83cBz+XmrU0ofLWatkiRJNUXELOAs4IiIWAtcn1L66YHYd/nqTcxZso41Fe/Qv3c3xo/sy/ABvQ7EriVJOqCKFkxERCfgNmAMsBZ4JiJ+m1JatneZlNK1Bct/ERhRsIntKaXhxapPkiSpISmlS7PYb/nqTUybt4KepSUcdWgpFdt2MG3eCr405jjDCUlSu1PMSzlOAV5OKa1MKe0A7gE+Vs/ylwKziliPJElSmzBnyTp6lpZwSNcSDorgkK4l9CwtYc6SdVmXJklSiytmMNEXWFMwvTbfto+IGAgMBv5Y0FyaH+H66Yi4qGhVSpIktTJrKt6hR2n1E1t7lHZmTcU7GVUkSVLxFHWMiSa4BJidUtpd0DYwpbQuIo4F/hgRz6WU/l/NFSNiEjAJYMCAAQemWkmSpCLq37sbFdt2cEjXkqq2rZW76N+7W4ZVSZJUHMU8Y2Id0L9gul++rTaXUOMyjpTSuvzPlcB8qo8/Ubict+iSJEntyviRfdlSuZPN23eyJyU2b9/JlsqdjB9Z68mnkiS1acUMJp4B3h8RgyOiC7nw4bc1F4qIDwK9gD8VtPWKiIPzz48ATgOW1VxXUhsx4/zcQ5LUKMMH9OJLY46jd/cuvPF2Jb27d9l34Mul98LNQ2HqYbmfS+/NrF5JkvZH0S7lSCntiogvAH8gd7vQu1JKL0TEt4BFKaW9IcUlwD0ppVSw+hDg/0bEHnLhyU2Fd/OQ1Hp4OztJKo7hA3rV/e/p0nvhgWtg5/bc9NtrctMAZRMOTIGSJLWQoo4xkVJ6EHiwRtuUGtNTa1nvKeDEYtYmaf95OztJysij33ovlNhr5/Zcu8GEJKmNKealHJLaOW9nJ0kZeXtt09olSWrFDCYkNZu3s5OkjBzar2ntkiS1YgYTkpqtf+9ubK3cVa3N29lJ0gFw7hQo6Vq9raRrrl2SpDbGYEJSszXqdnZL74W1z8CrCx01XpJaStkEuHA6HNofiNzPC6c7voQkqU0q6uCXktq3vbezK7wrx2fPGPzewJd7R43f/W5u2lHjJanllE3w31JJUrtgMCFpv9R7OztHjZckSZLUAIMJScXjqPGSJKkIyldvqnbG5viRfb1VudSGOcaEpOJx1HhJktTCyldvYtq8FVRs28FRh5ZSsW0H0+atoHz1puoLzjg/95DU6hlMSCoeR42XJEktbM6SdfQsLeGQriUcFMEhXUvoWVrCnCXr3lvIwbelNsVgQlLx7B01vtPBuWlHjZckSftpTcU79CitfkV6j9LOrKl4JzdR1+DbhhNSq2UwIam4yiZAv5Nh4Olw7fOGEpIkab/0792NrZW7qrVtrdxF/97dchP1Db4tqVUymJAkSZLUZowf2ZctlTvZvH0ne1Ji8/adbKncyfiRfXMLOPi21OYYTEiSJElqM4YP6MWXxhxH7+5deOPtSnp378KXxhz33l05HHxbanO8XagkSZKkNmX4gF513x703Cm5MSUKL+dw8G2pVfOMCUmSJEnth4NvS22OZ0xIKr6Jc7OuQJIkdSRlEwwipDbEMyYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmDCYkSZIkSVJmIqWUdQ0tJiI2AK8ewF0eAbx5APcntWV+XqTGKcZnZWBKqU8Lb1O1yKAvAv77KtXHz4dUuyw+G3X2R9pVMHGgRcSilNKorOuQ2gI/L1Lj+FlRU/mekerm50OqXWv7bHgphyRJkiRJyozBhCRJkiRJyozBxP65M+sCpDbEz4vUOH5W1FS+Z6S6+fmQateqPhuOMSFJkiRJkjLjGROSJEmSJCkzBhOSJEmSJCkznbMuoC2IiMOBR/OTRwO7gQ356SnAD4BOwE9SSjcd+Aql1qOBz8sS4AJgfUppaAblSa1GPZ+VnsBq4CggAXemlG7JpEi1KvZHpNrZ95Dq1lb6G44x0UQRMRXYmlL6YUR0AlYAY4C1wDPApSmlZRmWKLUahZ+X/PSZwFbg53YOpPfU+L/lGOCYlNKSiOgJLAYu8v8WFbI/ItXOvodUt9bc3/BSjv1zCvBySmllSmkHcA/wsYxrklqtlNICoCLrOqTWLKX0WkppSf75FmA50DfbqtTK2R+R6mDfQ6pda+tvGEzsn77AmoLptdh5lCS1kIgYBIwA/pxxKWrd7I9IkpqtNfQ3DCYkSWqFIqIHMAf4l5TS5qzrkSRJ7U9r6W8YTOyfdUD/gul++TZJkpotIkrIdRLuTindl3U9avXsj0iSmqw19TcMJvbPM8D7I2JwRHQBLgF+m3FNkqQ2LCIC+CmwPKU0Let61CbYH5EkNUlr628YTOyHlNIu4AvAH8gNFnJvSumFbKuSWq+ImAX8CfhARKyNiM9kXZPUCp0GXA6cExHl+cdHsi5KrZf9Ealu9j2kOrWq/oa3C5UkSZIkSZnxjAlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpQZgwlJkiRJkpSZ/x+J7zx3rcWt3AAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 15;\n", + " var nbb_formatted_code = \"# plot the error bar plot of different models\\nplt.figure(figsize=(18, 6))\\nplt.subplot(1, 2, 1)\\n\\nplt.errorbar(\\n np.arange(n_treatments) - 0.04,\\n true_longterm_effect,\\n fmt=\\\"o\\\",\\n alpha=0.6,\\n label=\\\"Ground truth\\\",\\n)\\nplt.errorbar(\\n np.arange(n_treatments),\\n adjsurr_point_est,\\n yerr=(\\n adjsurr_conf_int_ub - adjsurr_point_est,\\n adjsurr_point_est - adjsurr_conf_int_lb,\\n ),\\n fmt=\\\"o\\\",\\n label=\\\"Adjusted Surrogate Index\\\",\\n)\\nplt.xticks(np.arange(n_treatments), [\\\"T0\\\", \\\"T1\\\", \\\"T2\\\"])\\nplt.ylabel(\\\"Effect\\\")\\nplt.legend()\\n\\nplt.subplot(1, 2, 2)\\nplt.errorbar(\\n np.arange(n_treatments) - 0.04,\\n true_longterm_effect,\\n fmt=\\\"o\\\",\\n alpha=0.6,\\n label=\\\"Ground truth\\\",\\n)\\nplt.errorbar(\\n np.arange(n_treatments),\\n adjsurr_point_est,\\n yerr=(\\n adjsurr_conf_int_ub - adjsurr_point_est,\\n adjsurr_point_est - adjsurr_conf_int_lb,\\n ),\\n fmt=\\\"o\\\",\\n label=\\\"Adjusted Surrogate Index\\\",\\n)\\nplt.errorbar(\\n np.arange(n_treatments) + 0.04,\\n direct_point_est,\\n yerr=(direct_conf_int_ub - direct_point_est, direct_point_est - direct_conf_int_lb),\\n fmt=\\\"o\\\",\\n label=\\\"Direct Model\\\",\\n)\\nplt.xticks(np.arange(n_treatments), [\\\"T0\\\", \\\"T1\\\", \\\"T2\\\"])\\nplt.ylabel(\\\"Effect\\\")\\nplt.legend()\\nplt.suptitle(\\\"Error bar plot of treatment effect from different models\\\")\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# plot the error bar plot of different models\n", + "plt.figure(figsize=(18, 6))\n", + "plt.subplot(1, 2, 1)\n", + "\n", + "plt.errorbar(\n", + " np.arange(n_treatments) - 0.04,\n", + " true_longterm_effect,\n", + " fmt=\"o\",\n", + " alpha=0.6,\n", + " label=\"Ground truth\",\n", + ")\n", + "plt.errorbar(\n", + " np.arange(n_treatments),\n", + " adjsurr_point_est,\n", + " yerr=(\n", + " adjsurr_conf_int_ub - adjsurr_point_est,\n", + " adjsurr_point_est - adjsurr_conf_int_lb,\n", + " ),\n", + " fmt=\"o\",\n", + " label=\"Adjusted Surrogate Index\",\n", + ")\n", + "plt.xticks(np.arange(n_treatments), [\"T0\", \"T1\", \"T2\"])\n", + "plt.ylabel(\"Effect\")\n", + "plt.legend()\n", + "\n", + "plt.subplot(1, 2, 2)\n", + "plt.errorbar(\n", + " np.arange(n_treatments) - 0.04,\n", + " true_longterm_effect,\n", + " fmt=\"o\",\n", + " alpha=0.6,\n", + " label=\"Ground truth\",\n", + ")\n", + "plt.errorbar(\n", + " np.arange(n_treatments),\n", + " adjsurr_point_est,\n", + " yerr=(\n", + " adjsurr_conf_int_ub - adjsurr_point_est,\n", + " adjsurr_point_est - adjsurr_conf_int_lb,\n", + " ),\n", + " fmt=\"o\",\n", + " label=\"Adjusted Surrogate Index\",\n", + ")\n", + "plt.errorbar(\n", + " np.arange(n_treatments) + 0.04,\n", + " direct_point_est,\n", + " yerr=(direct_conf_int_ub - direct_point_est, direct_point_est - direct_conf_int_lb),\n", + " fmt=\"o\",\n", + " label=\"Direct Model\",\n", + ")\n", + "plt.xticks(np.arange(n_treatments), [\"T0\", \"T1\", \"T2\"])\n", + "plt.ylabel(\"Effect\")\n", + "plt.legend()\n", + "plt.suptitle(\"Error bar plot of treatment effect from different models\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We could see the **adjusted surrogate index** approach does a good job overcomes a common data limitation when considering long-term effects of novel treatments and expands the surrogate approach to consider a common, and previously\n", + "problematic, pattern of serially correlated treatments." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Extensions -- Including Heterogeneity in Effect\n", + "\n", + "Finally, I will show that our EconML's `DynamicDML` and `LinearDML` estimators could not only learn Average Treatment Effect (ATE), but also **Heterogeneous Treatment Effect (CATE)**, which will return the treatment effect as a function of interested characteristics. In the example below, I will use first control variable as feature to learn effect heterogeneity, and retrain the final `LinearDML` model. Similarly, you could train `DynamicDML` with feature $X$ as well." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True Long-term Effect for each investment: [0.90994672 0.709811 2.45310877]\n", + "Average treatment effect for each investment: [0.82738185 0.71610965 2.56087599]\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
Coefficient Results
point_estimate stderr zstat pvalue ci_lower ci_upper
X0|T0 0.009 0.011 0.76 0.447 -0.014 0.031
X0|T1 0.037 0.031 1.218 0.223 -0.023 0.098
X0|T2 -0.072 0.151 -0.478 0.633 -0.369 0.224
\n", + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
CATE Intercept Results
point_estimate stderr zstat pvalue ci_lower ci_upper
cate_intercept|T0 0.827 0.015 56.625 0.0 0.799 0.856
cate_intercept|T1 0.716 0.032 22.466 0.0 0.654 0.779
cate_intercept|T2 2.56 0.237 10.82 0.0 2.096 3.024


A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:
$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$
where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter." + ], + "text/plain": [ + "\n", + "\"\"\"\n", + " Coefficient Results \n", + "===========================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "-----------------------------------------------------------\n", + "X0|T0 0.009 0.011 0.76 0.447 -0.014 0.031\n", + "X0|T1 0.037 0.031 1.218 0.223 -0.023 0.098\n", + "X0|T2 -0.072 0.151 -0.478 0.633 -0.369 0.224\n", + " CATE Intercept Results \n", + "=======================================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "-----------------------------------------------------------------------\n", + "cate_intercept|T0 0.827 0.015 56.625 0.0 0.799 0.856\n", + "cate_intercept|T1 0.716 0.032 22.466 0.0 0.654 0.779\n", + "cate_intercept|T2 2.56 0.237 10.82 0.0 2.096 3.024\n", + "-----------------------------------------------------------------------\n", + "\n", + "A linear parametric conditional average treatment effect (CATE) model was fitted:\n", + "$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$\n", + "where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:\n", + "$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$\n", + "where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter.\n", + "\"\"\"" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "application/javascript": [ + "\n", + " setTimeout(function() {\n", + " var nbb_cell_id = 16;\n", + " var nbb_formatted_code = \"# learn treatment effect on surrogate index on new dataset\\nfrom econml.dml import LinearDML\\n\\nadjsurr_est = LinearDML(\\n model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=3\\n)\\n# fit treatment_0 on total revenue from new dataset\\nadjsurr_est.fit(\\n sindex_adj, panelTnew[:, 0], X=panelXnew[:, 0, :1], W=panelXnew[:, 0, 1:]\\n)\\n# print treatment effect summary\\nprint(\\\"True Long-term Effect for each investment: \\\", true_longterm_effect)\\nprint(\\n \\\"Average treatment effect for each investment: \\\",\\n adjsurr_est.const_marginal_ate(panelXnew[:, 0, :1]),\\n)\\nadjsurr_est.summary(alpha=0.05)\";\n", + " var nbb_cells = Jupyter.notebook.get_cells();\n", + " for (var i = 0; i < nbb_cells.length; ++i) {\n", + " if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n", + " nbb_cells[i].set_text(nbb_formatted_code);\n", + " break;\n", + " }\n", + " }\n", + " }, 500);\n", + " " + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# learn treatment effect on surrogate index on new dataset\n", + "from econml.dml import LinearDML\n", + "\n", + "adjsurr_est = LinearDML(\n", + " model_y=LassoCV(max_iter=2000), model_t=MultiTaskLassoCV(max_iter=2000), cv=3\n", + ")\n", + "# fit treatment_0 on total revenue from new dataset\n", + "adjsurr_est.fit(\n", + " sindex_adj, panelTnew[:, 0], X=panelXnew[:, 0, :1], W=panelXnew[:, 0, 1:]\n", + ")\n", + "# print treatment effect summary\n", + "print(\"True Long-term Effect for each investment: \", true_longterm_effect)\n", + "print(\n", + " \"Average treatment effect for each investment: \",\n", + " adjsurr_est.const_marginal_ate(panelXnew[:, 0, :1]),\n", + ")\n", + "adjsurr_est.summary(alpha=0.05)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From the summary table above, none of the coefficient for feature $X0$ is significant, that means there is no effect heterogeneity identified, which is consistent with the data generation process." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Conclusions\n", + "\n", + "In this notebook, we have demonstrated the power of using EconML to:\n", + "\n", + "* estimate treatment effects in settings when multiple treatments are assigned over time and treatments can have a causal effect on future outcomes\n", + "* correct the bias coming from auto-correlation of the historical treatment policy\n", + "* use Machine Learning to enable estimation with high-dimensional surrogates and controls\n", + "* solve a complex problem using an unified pipeline with only a few lines of code\n", + "\n", + "To learn more about what EconML can do for you, visit our [website](https://aka.ms/econml), our [GitHub page](https://github.com/microsoft/EconML) or our [documentation](https://econml.azurewebsites.net/). " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/Dynamic Double Machine Learning Examples.ipynb b/notebooks/Dynamic Double Machine Learning Examples.ipynb new file mode 100755 index 000000000..b5d48e3a2 --- /dev/null +++ b/notebooks/Dynamic Double Machine Learning Examples.ipynb @@ -0,0 +1,778 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Dynamic Double Machine Learning: Use Cases and Examples\n", + "\n", + "Dynamic DoubleML is an extension of the Double ML approach for treatments assigned sequentially over time periods. This estimator will account for treatments that can have causal effects on future outcomes. For more details, see [this paper](https://arxiv.org/abs/2002.07285) or the [EconML docummentation](https://econml.azurewebsites.net/).\n", + "\n", + "For example, the Dynamic DoubleML could be useful in estimating the following causal effects:\n", + "* the effect of investments on revenue at companies that receive investments at regular intervals ([see more](https://arxiv.org/abs/2103.08390))\n", + "* the effect of prices on demand in stores where prices of goods change over time\n", + "* the effect of income on health outcomes in people who receive yearly income\n", + "\n", + "The preferred data format is balanced panel data. Each panel corresponds to one entity (e.g. company, store or person) and the different rows in a panel correspond to different time points. Example:\n", + "\n", + "||Company|Year|Features|Investment|Revenue|\n", + "|---|---|---|---|---|---|\n", + "|1|A|2018|...|\\$1,000|\\$10,000|\n", + "|2|A|2019|...|\\$2,000|\\$12,000|\n", + "|3|A|2020|...|\\$3,000|\\$15,000|\n", + "|4|B|2018|...|\\$0|\\$5,000|\n", + "|5|B|2019|...|\\$100|\\$10,000|\n", + "|6|B|2020|...|\\$1,200|\\$7,000|\n", + "|7|C|2018|...|\\$1,000|\\$20,000|\n", + "|8|C|2019|...|\\$1,500|\\$25,000|\n", + "|9|C|2020|...|\\$500|\\$15,000|\n", + "\n", + "(Note: when passing the data to the DynamicDML estimator, the \"Company\" column above corresponds to the `groups` argument at fit time. The \"Year\" column above should not be passed in as it will be inferred from the \"Company\" column)\n", + "\n", + "If group memebers do not appear together, it is assumed that the first instance of a group in the dataset corresponds to the first period of that group, the second instance of the group corresponds to the second period, etc. Example:\n", + "\n", + "||Company|Features|Investment|Revenue|\n", + "|---|---|---|---|---|\n", + "|1|A|...|\\$1,000|\\$10,000|\n", + "|2|B|...|\\$0|\\$5,000\n", + "|3|C|...|\\$1,000|\\$20,000|\n", + "|4|A|...|\\$2,000|\\$12,000|\n", + "|5|B|...|\\$100|\\$10,000|\n", + "|6|C|...|\\$1,500|\\$25,000|\n", + "|7|A|...|\\$3,000|\\$15,000|\n", + "|8|B|...|\\$1,200|\\$7,000|\n", + "|9|C|...|\\$500|\\$15,000|\n", + "\n", + "In this dataset, 1st row corresponds to the first period of group `A`, 4th row corresponds to the second period of group `A`, etc.\n", + "\n", + "In this notebook, we show the performance of the DynamicDML on synthetic and observational data. \n", + "\n", + "## Notebook Contents\n", + "\n", + "1. [Example Usage with Average Treatment Effects](#1.-Example-Usage-with-Average-Treatment-Effects)\n", + "2. [Example Usage with Heterogeneous Treatment Effects](#2.-Example-Usage-with-Heterogeneous-Treatment-Effects)" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "%load_ext autoreload\n", + "%autoreload 2" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "import econml" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Main imports\n", + "from econml.dynamic.dml import DynamicDML\n", + "from econml.tests.dgp import DynamicPanelDGP, add_vlines\n", + "\n", + "# Helper imports\n", + "import numpy as np\n", + "from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, LogisticRegressionCV, MultiTaskLassoCV\n", + "import matplotlib.pyplot as plt\n", + "\n", + "%matplotlib inline" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 1. Example Usage with Average Treatment Effects" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1.1 DGP\n", + "\n", + "We consider a data generating process from a markovian treatment model. \n", + "\n", + "In the example bellow, $T_t\\rightarrow$ treatment(s) at time $t$, $Y_t\\rightarrow$outcome at time $t$, $X_t\\rightarrow$ features and controls at time $t$ (the coefficients $e, f$ will pick the features and the controls).\n", + "\\begin{align}\n", + " X_t =& (\\pi'X_{t-1} + 1) \\cdot A\\, T_{t-1} + B X_{t-1} + \\epsilon_t\\\\\n", + " T_t =& \\gamma\\, T_{t-1} + (1-\\gamma) \\cdot D X_t + \\zeta_t\\\\\n", + " Y_t =& (\\sigma' X_{t} + 1) \\cdot e\\, T_{t} + f X_t + \\eta_t\n", + "\\end{align}\n", + "\n", + "with $X_0, T_0 = 0$ and $\\epsilon_t, \\zeta_t, \\eta_t \\sim N(0, \\sigma^2)$. Moreover, $X_t \\in R^{n_x}$, $B[:, 0:s_x] \\neq 0$ and $B[:, s_x:-1] = 0$, $\\gamma\\in [0, 1]$, $D[:, 0:s_x] \\neq 0$, $D[:, s_x:-1]=0$, $f[0:s_x]\\neq 0$, $f[s_x:-1]=0$. We draw a single time series of samples of length $n\\_panels \\cdot n\\_periods$." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# Define DGP parameters\n", + "np.random.seed(123)\n", + "n_panels = 5000 # number of panels\n", + "n_periods = 3 # number of time periods in each panel\n", + "n_treatments = 2 # number of treatments in each period\n", + "n_x = 100 # number of features + controls\n", + "s_x = 10 # number of controls (endogeneous variables)\n", + "s_t = 10 # treatment support size" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "# Generate data\n", + "dgp = DynamicPanelDGP(n_periods, n_treatments, n_x).create_instance(\n", + " s_x, random_seed=12345)\n", + "Y, T, X, W, groups = dgp.observational_data(n_panels, s_t=s_t, random_seed=12345)\n", + "true_effect = dgp.true_effect" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1.2 Train Estimator" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "est = DynamicDML(\n", + " model_y=LassoCV(cv=3, max_iter=1000), \n", + " model_t=MultiTaskLassoCV(cv=3, max_iter=1000), \n", + " cv=3)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "est.fit(Y, T, X=None, W=W, groups=groups)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Average effect of default policy: 1.40\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "A scalar was specified but there are multiple treatments; the same value will be used for each treatment. Consider specifyingall treatments, or using the const_marginal_effect method.\n" + ] + } + ], + "source": [ + "# Average treatment effect of all periods on last period for unit treatments\n", + "print(f\"Average effect of default policy: {est.ate():0.2f}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Effect of target policy over baseline policy: 1.40\n" + ] + } + ], + "source": [ + "# Effect of target policy over baseline policy\n", + "# Must specify a treatment for each period\n", + "baseline_policy = np.zeros((1, n_periods * n_treatments))\n", + "target_policy = np.ones((1, n_periods * n_treatments))\n", + "eff = est.effect(T0=baseline_policy, T1=target_policy)\n", + "print(f\"Effect of target policy over baseline policy: {eff[0]:0.2f}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Marginal effect of a treatments in period 1 on period 3 outcome: [0.04000235 0.0701606 ]\n", + "Marginal effect of a treatments in period 2 on period 3 outcome: [0.31611764 0.23714736]\n", + "Marginal effect of a treatments in period 3 on period 3 outcome: [0.13108411 0.60656886]\n" + ] + } + ], + "source": [ + "# Period treatment effects + interpretation\n", + "for i, theta in enumerate(est.intercept_.reshape(-1, n_treatments)):\n", + " print(f\"Marginal effect of a treatments in period {i+1} on period {n_periods} outcome: {theta}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Coefficient Results: X is None, please call intercept_inference to learn the constant!\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
CATE Intercept Results
point_estimate stderr zstat pvalue ci_lower ci_upper
cate_intercept|(T0)$_0$ 0.04 0.041 0.977 0.328 -0.027 0.107
cate_intercept|(T1)$_0$ 0.07 0.04 1.74 0.082 0.004 0.136
cate_intercept|(T0)$_1$ 0.316 0.036 8.848 0.0 0.257 0.375
cate_intercept|(T1)$_1$ 0.237 0.036 6.608 0.0 0.178 0.296
cate_intercept|(T0)$_2$ 0.131 0.003 45.665 0.0 0.126 0.136
cate_intercept|(T1)$_2$ 0.607 0.003 210.244 0.0 0.602 0.611


A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:
$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$
where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter." + ], + "text/plain": [ + "\n", + "\"\"\"\n", + " CATE Intercept Results \n", + "==============================================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "------------------------------------------------------------------------------\n", + "cate_intercept|(T0)$_0$ 0.04 0.041 0.977 0.328 -0.027 0.107\n", + "cate_intercept|(T1)$_0$ 0.07 0.04 1.74 0.082 0.004 0.136\n", + "cate_intercept|(T0)$_1$ 0.316 0.036 8.848 0.0 0.257 0.375\n", + "cate_intercept|(T1)$_1$ 0.237 0.036 6.608 0.0 0.178 0.296\n", + "cate_intercept|(T0)$_2$ 0.131 0.003 45.665 0.0 0.126 0.136\n", + "cate_intercept|(T1)$_2$ 0.607 0.003 210.244 0.0 0.602 0.611\n", + "------------------------------------------------------------------------------\n", + "\n", + "A linear parametric conditional average treatment effect (CATE) model was fitted:\n", + "$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$\n", + "where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:\n", + "$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$\n", + "where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter.\n", + "\"\"\"" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Period treatment effects with confidence intervals\n", + "est.summary()" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [], + "source": [ + "conf_ints = est.intercept__interval(alpha=0.05)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1.3 Performance Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAEzCAYAAABjbqHIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAApUklEQVR4nO3de3TV5Z3v8c/XGEiAXLiJkHAJHWTkThOp1Bs4KNg6x9RedHC1Vo8HOQU9p6elheW1q2Ori47tsHRE2nG0Z4rYUUSLzDBeoFOPtJBIhAJSaARJUAyBhABJSMJz/tgxTcKGBMiPh/3s92st1s7v+T3Z+Qj4XfvL8/x+P3POCQAAAACQ+C7wHQAAAAAA0DVo8AAAAAAgEDR4AAAAABAIGjwAAAAACAQNHgAAAAAEggYPAAAAAAJxoe8Ap6tfv35u2LBhvmMACMihugZJUmZaquckAEJCbQEQleLi4v3Ouf7xziVcgzds2DAVFRX5jgEgICV7qiRJEwZne80BICzUFgBRMbPdJzuXcA0eAHQ1PnwBiAK1BYAPXIMHAAAAAIGgwQOQ9H63o0K/21HhOwaAwFBbAPgQxBbNhoYGlZWVqa6uzncUNEtLS1Nubq5SU7mwHOe/+objviMACBC1BYAPQTR4ZWVlysjI0LBhw2RmvuMkPeecKisrVVZWpry8PN9xAAAAgKQRxBbNuro69e3bl+buPGFm6tu3LyuqAAAAwDkWRIMniebuPMOfBwAAAHDuBdPgna5bnl6nW55e12Xvl5KSogkTJmj06NEaP368Hn/8cR0/7mfvfVFRke69995Tzhk2bJjGjh2rsWPHatSoUbr//vtVX18vSdq1a5fMTA888EDL/P379ys1NVVz586VJD388MP6yU9+Et1/BHAOXZyVpouz0nzHABAYagsAH5K2wetq6enpKikp0ZYtW/T6669r1apV+sEPfuAlS0FBgRYtWtThvDVr1mjz5s1av369SktLNWvWrJZzw4cP18qVK1uO/+3f/k2jR4+OJC/g25icLI3JyfIdA0BgqC0AfEjKBm/FxnJt/LBKf/jggK549C2t2Fjepe9/0UUXacmSJXriiSfknNNVV12lkpKSlvNXXHGFNm3apIcfflh33nmnpkyZouHDh7dpygoLC5Wfn6/Ro0dryZIlLeO9evXS97//feXn52vatGlav359y/e/+uqrkqS1a9fqxhtvlCQdPnxYd9xxh8aOHatx48bppZdeOiFvr169tHjxYq1YsUIHDhyQFGtYL730UhUVFUmSXnjhBX3ta1/r0t8nAAAAAF0r6Rq8FRvLtWD5Zh1rim2fLK+q1YLlm7u8yRs+fLiOHz+uTz75RHfddZeeffZZSdKf/vQn1dfXa9y4cZKk999/X6tXr9b69ev1gx/8QA0NDZKkZ555RsXFxSoqKtKiRYtUWVkpSTpy5IimTJmi4uJiZWRk6P7779frr7+ul19+WQ8++OAJOX74wx8qKytLmzdv1qZNm3TttdfGzZuZmam8vDzt2LGjZezWW2/VsmXLVFZWppSUFA0aNKgrf4uA88aa7Z9ozfZPfMcAEBhqC9A5KzaW64pH31Le/Nf+svhSViyt/I70r1+JvZYV+46ZMCJt8MxshpltN7OdZjb/JHOmmFmJmW0xs99GmUeSFq7ertqGpjZjtQ1NWrh6e5f/LOecJOmrX/2qVq5cqYaGBj3zzDP65je/2TLni1/8orp3765+/frpoosu0r59+yRJixYt0vjx43X55Zdrz549LY1Xt27dNGPGDEnS2LFjdc011yg1NVVjx47Vrl27TsjwxhtvaM6cOS3HvXv37jDvp2bMmKHXX39dzz//vG655ZYz+j0AEkFTk1NTk+t4IgCcBmoL0LFPF1/Kq2rlFFt8+dXyl7Xv1Qek2kopc2Dsdc0jNHmdFNlz8MwsRdKTkq6TVCZpg5m96pzb2mpOtqR/kjTDOfehmV0UVZ5P7a2qPa3xM1VaWqqUlBRddNFFMjNdd911euWVV/TrX/+6ZdujJHXv3r3l65SUFDU2Nmrt2rV64403tG7dOvXo0UNTpkxpeeRAampqyx0qL7jggpbvv+CCC9TY2HhCDudcp+5oWVNTo127dumSSy5RdXW1pFgzmZ+fr3/4h3/Qli1b9Jvf/ObMf0MAAACAduItvvytW6PN+6UBg5uvYU1rfi1ZKuXmn+OEiSfKFbxJknY650qdc8ckLZN0U7s5MyUtd859KEnOucj3MQzKTj+t8TNRUVGh2bNna+7cuS3N1V133aV7771Xl112mfr06XPK76+urlbv3r3Vo0cPvf/++/r9739/xlmuv/56PfHEEy3HBw8ePGHO4cOH9a1vfUuFhYUnrPB95zvf0WOPPaa+ffuecQYAAAAgnniLLIOtQhXHurUd7J4hVe0+R6kSW5QNXo6kPa2Oy5rHWrtEUm8zW2tmxWb2jQjzSJLmTR+p9NSUNmPpqSmaN33kWb1vbW1ty2MSpk2bpuuvv14PPfRQy/n8/HxlZmbqjjvu6PC9ZsyYocbGRo0bN04PPPCALr/88jPOdf/99+vgwYMaM2aMxo8frzVr1rScmzp1qsaMGaNJkyZpyJAhevrpp0/4/tGjR+v222+P+95///d/r9zc3JZfAAAAwOmIt8iyx/VX/27H2g7W10jZQ89RqsRm7a+76rI3NvuqpOnOubuaj78uaZJz7p5Wc56QVCDpbySlS1on6YvOuT+1e69ZkmZJ0pAhQ/J3727bvW/btk2XXnppp7Ot2Fiu7724SceajisnO13zpo9U4cT2vWfX2rt3r6ZMmaL3339fF1yQHPe2Od0/F8CXbR8dkiRdOjDTcxIAIaG2AB379Bq81ts0L0v9QE8MXKUB/S+KrdzV10h1h6Sp97FFs5mZFTvnCuKdi+waPMVW7Aa3Os6VtDfOnP3OuSOSjpjZf0kaL6lNg+ecWyJpiSQVFBScdUdaODFHz6//UJL0wt2Tz/btOvTLX/5S9913nx5//PGkae6ARMKHLwBRoLYAHft0kWXh6u3aW1WrQdnpum36lzSg/+TYNXdVu2Mrd5PvobnrpChX8C5UrFH7G0nlkjZImumc29JqzqWSnpA0XVI3Sesl3eqc++PJ3regoMC1vkmJxErR+Yo/FwAAAKDreVnBc841mtlcSaslpUh6xjm3xcxmN59f7JzbZmb/IWmTpOOSfnGq5g4AovDG1tjjSaaNGuA5CYCQUFsA+BDlFk0551ZJWtVubHG744WSFkaZAwAAAACSAReEAQAAAEAgaPAAAAAAIBA0eF1k3759mjlzpoYPH678/HxNnjxZL7/88jnNsGvXLo0ZMybu+NKlS8/oPX/2s5/p6NGjLce9evU643wAAAAAokWD1wWccyosLNTVV1+t0tJSFRcXa9myZSorKzthbmNj4znPd6oGr6M87Rs8IERD+/bQ0L49fMcAEBhqCwAfIr3JynmrrLjtczUmzDyr52q89dZb6tatm2bPnt0yNnToUN1zT+yZ7s8++6xee+011dXV6ciRI3rxxRd15513qrS0VD169NCSJUs0btw4Pfzww+rVq5e++93vSpLGjBmjlStXSpJuuOEGXXnllXrnnXeUk5OjV155Renp6SouLtadd96pHj166Morr4ybb/78+dq2bZsmTJig22+/Xb17926T58EHH9RPfvKTlp81d+5cFRQU6NChQ9q7d6+mTp2qfv36ac2aNZKk++67TytXrlR6erpeeeUVDRjA3cGQ2EYMyPAdAUCAqC0AfEi+FbyyYmnNI1JtpZQ5MPa65pHY+BnasmWLPvvZz55yzrp16/Tcc8/prbfe0kMPPaSJEydq06ZN+tGPfqRvfOMbHf6MHTt2aM6cOdqyZYuys7P10ksvSZLuuOMOLVq0SOvWrTvp9z766KO66qqrVFJSom9/+9sn5DmZe++9V4MGDdKaNWtamrsjR47o8ssv13vvvaerr75aP//5zzvMDpzvGpuOq7HpuO8YAAJDbQHgQ/I1eCVLpbRMKS1Lsgtir2mZsfEuMmfOHI0fP16XXXZZy9h1112nPn36SJLefvttff3rX5ckXXvttaqsrFR1dfUp3zMvL08TJkyQJOXn52vXrl2qrq5WVVWVrrnmGklqec/OaJ3ndHTr1k033nhjmxxAolu7vUJrt1f4jgEgMNQWAD4kX4NXtVvq3m7LRPeM2PgZGj16tN59992W4yeffFJvvvmmKir+UtR79uzZ8rVz7oT3MDNdeOGFOn78L//SV1dX95eI3bu3fJ2SkqLGxkY552RmZ5S5dZ5T/dz2UlNTW37mpzkAAAAAnB+Sr8HLHirV17Qdq6+JjZ+ha6+9VnV1dXrqqadaxk51Y5Krr75av/rVryRJa9euVb9+/ZSZmalhw4a1NIrvvvuuPvjgg1P+3OzsbGVlZentt9+WpJb3bC8jI0M1NTVxz0mx6wW3bt2q+vp6VVdX68033+z09wIAAAA4fyRfgzdhplR3SKqrltzx2Gvdodj4GTIzrVixQr/97W+Vl5enSZMm6fbbb9djjz0Wd/7DDz+soqIijRs3TvPnz9dzzz0nSfryl7+sAwcOaMKECXrqqad0ySWXdPiz/+Vf/kVz5szR5MmTlZ6eHnfOuHHjdOGFF2r8+PH66U9/esL5wYMH62tf+5rGjRun2267TRMnTmw5N2vWLN1www2aOnVqZ34rAAAAAHhk8bYLns8KCgpcUVFRm7Ft27bp0ksv7fybdPFdNBHfaf+5AJ68sXWfJGnaKO4IC6DrUFsARMXMip1zBfHOJedjEnLzaegAtBjev2fHkwDgNFFbAPiQnA0eALQyvH8v3xEABIjaAsCH5LsGDwDaqWtoUl1Dk+8YAAJDbQHgQzANXqJdSxg6/jyQSN7esV9v79jvOwaAwFBbAPgQRIOXlpamyspKmorzhHNOlZWVSktL8x0FAAAASCpBXIOXm5ursrKyNg8Wh19paWnKzc31HQMAAABIKkE0eKmpqcrLy/MdAwAAAAC8CmKLJgAAAAAgkBU8ADgbIwZwK3MAXY/aAsAHGjwASW9oXx5GDKDrUVsA+MAWTQBJ70h9o47UN/qOASAw1BYAPtDgAUh66/5cqXV/rvQdA0BgqC0AfKDBAwAAAIBA0OABAAAAQCBo8AAAAAAgEDR4AAAAABAIHpMAIOn99cAM3xEABIjaAsAHGjwASS+3dw/fEQAEiNoCwIdIt2ia2Qwz225mO81sfpzzU8ys2sxKmn89GGUeAIinurZB1bUNvmMACAy1BYAPka3gmVmKpCclXSepTNIGM3vVObe13dTfOedujCoHAHRkwwcHJEnTRg3wnARASKgtAHyIcgVvkqSdzrlS59wxScsk3RThzwMAAACApBZlg5cjaU+r47LmsfYmm9l7ZvbvZjY6wjwAAAAAELQob7JiccZcu+N3JQ11zh02sy9IWiFpxAlvZDZL0ixJGjJkSBfHBAAAAIAwRLmCVyZpcKvjXEl7W09wzh1yzh1u/nqVpFQz69f+jZxzS5xzBc65gv79+0cYGQAAAAASV5QreBskjTCzPEnlkm6VNLP1BDO7WNI+55wzs0mKNZyVEWYCgBOMycnyHQFAgKgtAHyIrMFzzjWa2VxJqyWlSHrGObfFzGY3n18s6SuS/qeZNUqqlXSrc679Nk4AiNTFWWm+IwAIELUFgA+WaP1UQUGBKyoq8h0DQEAOHjkmSerds5vnJABCQm0BEBUzK3bOFcQ7F+mDzgEgERTvPqji3Qd9xwAQGGoLAB9o8AAAAAAgEDR4AAAAABAIGjwAAAAACAQNHgAAAAAEIsrn4AFAQhg/ONt3BAABorYA8IEGD0DS65/R3XcEAAGitgDwgS2aAJJeRU29KmrqfccAEBhqCwAfaPAAJL339lTpvT1VvmMACAy1BYAPNHgAAAAAEAgaPAAAAAAIBA0eAAAAAASCBg8AAAAAAsFjEgAkvfyhvX1HABAgagsAH2jwACS93j27+Y4AIEDUFgA+sEUTQNL7uLpOH1fX+Y4BIDDUFgA+sIIHIOn9sbxaknRxVprnJABCQm0B4AMreAAAAAAQCBo8AAAAAAgEDR4AAAAABIIGDwAAAAACwU1WACS9y/L6+I4AIEDUFgA+0OABSHpZ6am+IwAIELUFgA9s0QSQ9MoOHlXZwaO+YwAIDLUFgA+s4AFIeu9/VCNJyu3dw3MSACGhtgDwgRU8AAAAAAgEDR4AAAAABIIGDwAAAAACQYMHAAAAAIGItMEzsxlmtt3MdprZ/FPMu8zMmszsK1HmAYB4Jn+mryZ/pq/vGAACQ20B4ENkd9E0sxRJT0q6TlKZpA1m9qpzbmuceY9JWh1VFgA4lZ7duaEwgK5HbQHgQ5QreJMk7XTOlTrnjklaJummOPPukfSSpE8izAIAJ7W78oh2Vx7xHQNAYKgtAHyIssHLkbSn1XFZ81gLM8uR9CVJiyPMAQCntGPfYe3Yd9h3DACBobYA8CHKBs/ijLl2xz+T9H3nXNMp38hslpkVmVlRRUVFV+UDAAAAgKBEuTm8TNLgVse5kva2m1MgaZmZSVI/SV8ws0bn3IrWk5xzSyQtkaSCgoL2TSIAAAAAQNE2eBskjTCzPEnlkm6VNLP1BOdc3qdfm9mzkla2b+4AAAAAAJ0TWYPnnGs0s7mK3R0zRdIzzrktZja7+TzX3QEAAABAFzLnEmvHY0FBgSsqKvIdA0BA6hpilwGnpaZ4TgIgJNQWAFExs2LnXEG8czygBUDSS/YPX7c8vU6S9MLdkz0nAcKS7LUFgB9R3kUTABJCacVhlVZwK3MAXYvaAsAHGjwASa+04ohKK3gYMYCuRW0B4AMNHgAAAAAEggYPAAAAAAJBgwcAAAAAgaDBAwAAAIBA8JgEAElvysj+viMACBC1BYAPNHgAkt6FKWxmAND1qC0AfKDyAEh6O/bVaMe+Gt8xAASG2gLABxo8AElvd+VR7a486jsGgMBQWwD4QIMHAAAAAIGgwQMAAACAQNDgAQAAAEAgaPAAAAAAIBA8JgFA0ps2aoDvCAACRG0B4AMreAAAAAAQCBo8AElv20eHtO2jQ75jAAgMtQWADzR4AJJe+cFalR+s9R0DQGCoLQB8oMEDAAAAgEDQ4AEAAABAIGjwAAAAACAQPCYBQNJLSTHfEQAEiNoCwAcaPABJb+rIi3xHABAgagsAH9iiCQBJbMXGcm38sEp/+OCArnj0La3YWO47EgAAOAs0eACS3h/Lq/XH8mrfMc65FRvLtWD5Zh1rOi5JKq+q1YLlm2nygC6SrLUFgF+davDM7P92ZgwAEtHH1XX6uLrOd4xzbuHq7aptaGozVtvQpIWrt3tKBIQlWWsLAL86u4I3uvWBmaVIyu/6OACAc2VvVfwHMJ9sHAAAnP9O2eCZ2QIzq5E0zswONf+qkfSJpFfOSUIAQCQGZaef1jgAADj/nbLBc8792DmXIWmhcy6z+VeGc66vc25BR29uZjPMbLuZ7TSz+XHO32Rmm8ysxMyKzOzKs/hvAQCchnnTRyo9NaXNWHpqiuZNH+kpEQAAOFud3aK53syyPj0ws2wzKzzVNzRv43xS0g2SRkn6OzMb1W7am5LGO+cmSLpT0i86mQcAukz31AvUPTX57jlVODFHP755rLqlxP7bc7LT9eObx6pwYo7nZDif3fL0Ot3y9DrfMRJCstYWAH519jl4DznnXv70wDlXZWYPSVpxiu+ZJGmnc65UksxsmaSbJG1t9T6HW83vKcl1Mg8AdJmrRvT3HcGbwok5en79h5KkF+6e7DkNEJZkri0A/OnsPyvFm9dRc5gjaU+r47LmsTbM7Etm9r6k1xRbxQMAAAAAnIHOruAVmdnjim25dJLukVTcwfdYnLETVuiaVwZfNrOrJf1Q0rQT3shslqRZkjRkyJBORgaAzinZUyVJmjA422uO80pZsVSyVKraLWUPlSbMlHK5eTJwOqgtAHzo7ArePZKOSXpB0q8l1Uqa08H3lEka3Oo4V9Lek012zv2XpM+YWb8455Y45wqccwX9+7PdAUDX2l9Tr/019b5jnD/KiqU1j0i1lVLmwNjrmkdi4wA6jdoCwIdOreA5545Imm9mvdpdN3cqGySNMLM8SeWSbpU0s/UEM/srSX92zjkz+6ykbpIqO50eAND1SpZKaZlSWvO9tT59LVnKKh4AAOe5Tq3gmdnnzWyrmm+QYmbjzeyfTvU9zrlGSXMlrZa0TdKvnXNbzGy2mc1unvZlSX80sxLFtn/e4pzjRisA4FPVbql7Rtux7hmxcQAAcF7r7DV4P5U0XdKrkuSce6/5mrlTcs6tkrSq3djiVl8/JumxTqcFAEQve2hsW2Za1l/G6mti4wAA4LzW6YezOOf2tBtq6uIsAOBFj24p6tEtpeOJyWLCTKnukFRXLbnjsde6Q7FxAJ1GbQHgQ2dX8PaY2eclOTPrJulexbZdAkDC+/xfnXBvp+SWmy9Nva/tXTQn38P1d8BporYA8KGzDd5sSf+o2HPsyiT9pzq+iyYAIFHl5tPQAQCQgE7Z4JnZY86570ua6py77RxlAoBzqnj3AUlS/tA+npMACAm1BYAPHV2D9wUzS5W04FyEAQAfDh5p0MEjDb5jAAgMtQWADx1t0fwPSfsl9TSzQ5JMkvv01TmXGXE+AAAAAEAndbSCd79zLkvSa865TOdcRuvXcxEQAAAAANA5HTV465pfD0UdBAAAAABwdjraotnNzG6X9Hkzu7n9Sefc8mhiAcC5k5HW2RsKA0DnUVsA+NBR5Zkt6TZJ2ZL+tt05J4kGD0DC+9zwvr4jAAgQtQWAD6ds8Jxzb0t628yKnHP/fI4yAQAAAADOwCmvwTOz70mSc+6fzeyr7c79KMpgAHCu/KG0Un8orfQdA0BgqC0AfOjoJiu3tvq6/bPwZnRxFgDwoqauUTV1jb5jAAgMtQWADx1dg2cn+TreMQAASDZlxVLJUqlqt5Q9VJowU8rN950KAJJWRyt47iRfxzsGAADJpKxYWvOIVFspZQ6Mva55JDYOAPCioxW88WZ2SLHVuvTmr9V8nBZpMgDAOfHC3ZN9R0CiKlkqpWVKaVmx409fS5ayigcAnnR0F82UcxUEAHzp3TPVdwQgIazYWK6NH1bpWNNxXfHoW3opc7suzs1rO6l7Rmy7JqgtALzgCZwAkl7+0D6+IwDnvRUby7Vg+WYdazouSSqvqtVbR9N09YWfKHfgxX+ZWF8TuxYP1BYAXnR0DR4AAIAWrt6u2oamNmPLGq5S2UcfS3XVkjsee607FLvRCgDACxo8AEnvnZ379c7O/b5jAOe1vVW1J4xtcn+lH9fdLKX3lQ59FHudeh/X3zWjtgDwgS2aAJLe0WNNHU8Cktyg7HSVx2ny9meNlW78Xx4Snf+oLQB8YAUPAAB0aN70kUpPbXvvtfTUFM2bPtJTIgBAPKzgAQCADhVOzJEkfe/FTTrWdFw52emaN31kyzgA4PxAgwcAADqlcGKOnl//oSSenwgA5ysaPABJr19Gd98RAASI2gLABxo8AElvwuBs3xEABIjaAsAHbrICAAAAAIGgwQOQ9H63o0K/21HhOwaAwFBbAPjAFk0ASa++4bjvCAACRG0B4EOkK3hmNsPMtpvZTjObH+f8bWa2qfnXO2Y2Pso8AAAAABCyyBo8M0uR9KSkGySNkvR3Zjaq3bQPJF3jnBsn6YeSlkSVBwAAAABCF+UK3iRJO51zpc65Y5KWSbqp9QTn3DvOuYPNh7+XlBthHgAAAAAIWpTX4OVI2tPquEzS504x/79L+vcI8wBAXBdnpfmOACBA1BYAPkTZ4FmcMRd3otlUxRq8K09yfpakWZI0ZMiQrsoHAJKkMTlZviMACBC1BYAPUW7RLJM0uNVxrqS97SeZ2ThJv5B0k3OuMt4bOeeWOOcKnHMF/fv3jyQsAAAAACS6KBu8DZJGmFmemXWTdKukV1tPMLMhkpZL+rpz7k8RZgGAk1qz/ROt2f6J7xgAAkNtAeBDZFs0nXONZjZX0mpJKZKecc5tMbPZzecXS3pQUl9J/2RmktTonCuIKhMAxNPUFHf3OACcFWoLAB8ifdC5c26VpFXtxha3+vouSXdFmQEAAAAAkkWkDzoHAAAAAJw7NHgAAAAAEIhIt2gCQCLI6Z3uOwKAAFFbAPhAgwcg6V06MNN3BAABorYA8IEtmgAAAAAQCBo8AEnvja379MbWfb5jAAgMtQWADzR4AAAAABAIrsEDAACd9sLdk31HAACcAit4AAAAABAIGjwAAAAACARbNAEkvaF9e/iOACBA1BYAPtDgAUh6IwZk+I4AIEDUFgA+sEUTQNJrbDquxqbjvmMACAy1BYAPNHgAkt7a7RVau73CdwwAgaG2APCBBg8I1C1Pr9MtT6/zHQMAAADnEA0eAAAAAASCBg8AAAAAAkGDBwAAAACB4DEJAJLe8P49fUcAEKBkri0rNpZr4ert2ltVq0HZ6Zo3faQKJ+ZIZcVSyVKpareUPVSaMFPKzfcdFwgKDR6ApDe8fy/fEQAEKFlry4qN5VqwfLNqG5okSeVVtVqwfLOyD2zSlL0/l9IypcyBUm2ltOYRaep9NHlAF2KLJoCkV9fQpLrmDyIA0FWStbYsXL29pbn7VG1DkyrfeTbW3KVlSXZB7DUtM7aiB6DL0OABSHpv79ivt3fs9x0DQGCStbbsraqNO97n2EdS94y2g90zYts1AXQZGjwAAAB0mUHZ6XHHD3QbKNXXtB2sr4ldiwegy9DgAQAAoMvMmz5S6akpbcbSU1PU9/PflOoOSXXVkjsee607FLvRCoAuw01WAAAA0GUKJ+ZI0gl30ZwyMUcqu6jtXTQn38MNVoAuRoOXQG55ep0k6YW7J3tOAgAAcHKFE3NaGr02cvNp6ICI0eABSHojBiTnrcwBRIvaAsAHGjwASW9o3+R9GDGA6FBbAPjATVYAJL0j9Y06Ut/oOwaAwFBbAPgQaYNnZjPMbLuZ7TSz+XHO/7WZrTOzejP7bpRZAOBk1v25Uuv+XOk7BoDAUFsA+BDZFk0zS5H0pKTrJJVJ2mBmrzrntraadkDSvZIKo8oBAAAAAMkiyhW8SZJ2OudKnXPHJC2TdFPrCc65T5xzGyQ1RJgDAAAAAJJClA1ejqQ9rY7LmscAAAAAABGIssGzOGPujN7IbJaZFZlZUUVFxVnGAgAAAIAwRfmYhDJJg1sd50raeyZv5JxbImmJJBUUFJxRkwgAJ/PXAzN8RwAQIGoLAB+ibPA2SBphZnmSyiXdKmlmhD8PAM5Ibu8eviMACBC1BYAPkTV4zrlGM5srabWkFEnPOOe2mNns5vOLzexiSUWSMiUdN7P/LWmUc+5QVLkAoL3q2th9nrLSUz0nARASagsAH6JcwZNzbpWkVe3GFrf6+mPFtm4CgDcbPjggSZo2aoDnJABCQm0B4EOkDzoHAAAAAJw7NHgAAAAAEAgaPAAAAAAIBA0eEKAVG8u18cMq/eGDA7ri0be0YmO570gAAAA4ByK9yQqAc2/FxnItWL5Zx5qOS5LKq2q1YPlmSVLhxByf0c5bY3KyfEcAECBqCwAfWMEDArNw9XbVNjS1GattaNLC1ds9JTr/XZyVpouz0nzHABAYagsAH2jwgMDsrao9rXFIB48c08Ejx3zHABAYagsAH2jwgMAMyk4/rXFIxbsPqnj3Qd8xAASG2gLABxo8IDDzpo9UempKm7H01BQ9clm9tPI70r9+JfZaVuwpIQAAAKJCg5foyor50I42Cifm6Mc3j1W3lNj/3jnZ6XpqitOUvT+XaiulzIGx1zWP8PcFAAAgMDR4iaysOPYhnQ/taKdwYo4mDsnW5/L66P/Nv1ZTat+Q0jKltCzJLoi9pmVKJUt9RwUAAEAXosFLZCVL+dCOzqnaLXXPaDvWPSM2DgAAgGDwHLxEVrU7tnLXGh/aEU/20NgKb1qrZzLV18TGofGDs31HABAgagsAH1jBS2TZQ2Mf0lvjQzvimTBTqjsk1VVL7njste5QbBzqn9Fd/TO6+44BIDDUFgA+0OAliBUby7Xxwyr94YMDuuLRt7RiYzkf2tF5ufnS1Puk9L7SoY9ir1Pvi41DFTX1qqip9x0DQGCoLQB8YItmAlixsVwLlm/WsabjkqTyqlotWL5ZunmsCqfeF7vmrmp3bOVu8j18aEd8ufn83TiJ9/ZUSZKmjRrgNwiAoFBbAPhAg5cAFq7ertqGpjZjtQ1NWrh6uwrnX8uHdgAAAACS2KKZEPZW1Z7WOAAAAIDkRIOXAAZlp5/WOAAAAIDkRIOXAOZNH6n01JQ2Y+mpKZo3faSnRAAAAADOR1yDlwAKJ+ZIkr734iYdazqunOx0zZs+smUcwNnJH9rbdwQAAaK2APCBBi9BFE7M0fPrP5QkvXD3ZM9pgLD07tnNdwQAAaK2APCBLZoAkt7H1XX6uLrOdwwAgaG2APCBFTwASe+P5dWSpIuz0jwnARASagsAH1jBAwAAAIBA0OABAAAAQCBo8AAAAAAgEDR4AAAAABCISBs8M5thZtvNbKeZzY9z3sxsUfP5TWb22SjzAEA8l+X10WV5fXzHABAYagsAHyK7i6aZpUh6UtJ1ksokbTCzV51zW1tNu0HSiOZfn5P0VPMrAJwzWempviMACBC1BYAPUa7gTZK00zlX6pw7JmmZpJvazblJ0i9dzO8lZZvZwAgzAcAJyg4eVdnBo75jAAgMtQWAD1E2eDmS9rQ6LmseO905ABCp9z+q0fsf1fiOASAw1BYAPkTZ4FmcMXcGc2Rms8ysyMyKKioquiQcAAAAAIQmygavTNLgVse5kvaewRw555Y45wqccwX9+/fv8qAAAAAAEIIoG7wNkkaYWZ6ZdZN0q6RX2815VdI3mu+mebmkaufcRxFmAgAAAIBgRXYXTedco5nNlbRaUoqkZ5xzW8xsdvP5xZJWSfqCpJ2Sjkq6I6o8AAAAABC6yBo8SXLOrVKsiWs9trjV107SnCgzAMnqhbsn+46QMCZ/pq/vCAACRG0B4EOkDR4AJIKe3SmFALoetQWAD1FegwcACWF35RHtrjziOwaAwFBbAPjAPy0BSHo79h2WJA3t29NzEgAhobYA8IEVPAAAAAAIBA0eAAAAAASCBg8AAAAAAkGDBwAAAACB4CYrAJLelSP6+Y4AIEDUFgA+0OABSHppqSm+IwAIELUFgA9s0QSQ9EorDqu04rDvGAACQ20B4AMNHoCkV1pxRKUVPIwYQNeitgDwgS2aCeSFuyf7jgAAAADgPMYKHgAAAAAEggYPAAAAAAJBgwcAAAAAgeAaPABJb8rI/r4jAAgQtQWADzR4AJLehSlsZgDQ9agtAHyg8gBIejv21WjHvhrfMQAEhtoCwAcaPABJb3flUe2uPOo7BoDAUFsA+ECDBwAAAACBoMEDAAAAgEDQ4AEAAABAIGjwAAAAACAQ5pzzneG0mFmFpN2+cyBh9JO033cIAMGhtgCIArUFnTXUORf3YZsJ1+ABp8PMipxzBb5zAAgLtQVAFKgt6Aps0QQAAACAQNDgAQAAAEAgaPAQuiW+AwAIErUFQBSoLThrXIMHAAAAAIFgBQ8AAAAAAkGDBwAAAACBoMEDAAAAgEDQ4CEoZpZiZv9oZlvMbLOZDfedCUDio7YAiAK1BVGgwUNoFkgqdc6NlrRI0rc85wEQBmoLgChQW9DlLvQdAOgqZtZT0pecc/nNQx9I+qLHSAACQG0BEAVqC6JCg4eQTJM02MxKmo/7SHrDXxwAgaC2AIgCtQWRYIsmQjJB0oPOuQnOuQmS/lNSiZn1NLPnzOznZnab14QAEtEExa8tw83sn83sRa/pACSqCYpfWwqbP7O8YmbXe02IhESDh5D0lnRUkszsQknXS/qNpJslveic+x+S/pu/eAASVNza4pwrdc79d6/JACSyk9WWFc2fWb4p6RZ/8ZCoaPAQkj9Jurz5629Les0594GkXEl7msebfAQDkNBOVlsA4Gx0VFvul/TkOU+FhEeDh5A8L+mzZrZT0jhJ/6d5vEyxJk/i7zyA03ey2gIAZyNubbGYxyT9u3PuXZ8BkZjMOec7AxCp5rtUPSGpTtLbzrlfeY4EIABm1lfSI5Kuk/QL59yPPUcCEAAzu1fS7ZI2SCpxzi32HAkJhgYPAAAAAALBdjUAAAAACAQNHgAAAAAEggYPAAAAAAJBgwcAAAAAgaDBAwAAAIBA0OABAAAAQCBo8AAAAAAgEDR4AAAAABAIGjwAAAAACMT/B1HSZiIhWdHSAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "# Some plotting boilerplate code\n", + "plt.figure(figsize=(15, 5))\n", + "plt.errorbar(np.arange(n_periods*n_treatments)-.04, est.intercept_, yerr=(conf_ints[1] - est.intercept_,\n", + " est.intercept_ - conf_ints[0]), fmt='o', label='DynamicDML')\n", + "plt.errorbar(np.arange(n_periods*n_treatments), true_effect.flatten(), fmt='o', alpha=.6, label='Ground truth')\n", + "for t in np.arange(1, n_periods):\n", + " plt.axvline(x=t * n_treatments - .5, linestyle='--', alpha=.4)\n", + "plt.xticks([t * n_treatments - .5 + n_treatments/2 for t in range(n_periods)],\n", + " [\"$\\\\theta_{}$\".format(t) for t in range(n_periods)])\n", + "plt.gca().set_xlim([-.5, n_periods*n_treatments - .5])\n", + "plt.ylabel(\"Effect\")\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 2. Example Usage with Heterogeneous Treatment Effects on Time-Invariant Unit Characteristics\n", + "\n", + "We can also estimate treatment effect heterogeneity with respect to the value of some subset of features $X$ in the initial period. Heterogeneity is currently only supported with respect to such initial state features. This for instance can support heterogeneity with respect to time-invariant unit characteristics. In that case you can simply pass as $X$ a repetition of some unit features that stay constant in all periods. You can also pass time-varying features, and their time varying component will be used as a time-varying control. However, heterogeneity will only be estimated with respect to the initial state." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.1 DGP" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "# Define additional DGP parameters\n", + "het_strength = .5\n", + "het_inds = np.arange(n_x - n_treatments, n_x)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "# Generate data\n", + "dgp = DynamicPanelDGP(n_periods, n_treatments, n_x).create_instance(\n", + " s_x, hetero_strength=het_strength, hetero_inds=het_inds, random_seed=12)\n", + "Y, T, X, W, groups = dgp.observational_data(n_panels, s_t=s_t, random_seed=1)\n", + "ate_effect = dgp.true_effect\n", + "het_effect = dgp.true_hetero_effect[:, het_inds + 1]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.2 Train Estimator" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "est = DynamicDML(\n", + " model_y=LassoCV(cv=3), \n", + " model_t=MultiTaskLassoCV(cv=3), \n", + " cv=3)" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "est.fit(Y, T, X=X, W=W, groups=groups, inference=\"auto\")" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
Coefficient Results
point_estimate stderr zstat pvalue ci_lower ci_upper
X0|(T0)$_0$ 0.009 0.045 0.203 0.839 -0.065 0.083
X0|(T1)$_0$ 0.017 0.042 0.416 0.677 -0.051 0.086
X0|(T0)$_1$ -0.001 0.041 -0.035 0.972 -0.069 0.067
X0|(T1)$_1$ -0.031 0.041 -0.76 0.447 -0.099 0.036
X0|(T0)$_2$ -0.306 0.008 -36.667 0.0 -0.32 -0.292
X0|(T1)$_2$ 0.158 0.008 19.656 0.0 0.145 0.171
X1|(T0)$_0$ 0.017 0.044 0.378 0.706 -0.056 0.09
X1|(T1)$_0$ -0.007 0.045 -0.164 0.87 -0.082 0.067
X1|(T0)$_1$ -0.034 0.042 -0.821 0.412 -0.103 0.034
X1|(T1)$_1$ -0.025 0.042 -0.6 0.549 -0.095 0.044
X1|(T0)$_2$ -0.302 0.008 -35.72 0.0 -0.316 -0.288
X1|(T1)$_2$ 0.156 0.008 18.801 0.0 0.142 0.169
\n", + "\n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "\n", + " \n", + "\n", + "
CATE Intercept Results
point_estimate stderr zstat pvalue ci_lower ci_upper
cate_intercept|(T0)$_0$ 0.024 0.036 0.653 0.513 -0.036 0.084
cate_intercept|(T1)$_0$ -0.033 0.036 -0.929 0.353 -0.092 0.025
cate_intercept|(T0)$_1$ -0.105 0.034 -3.067 0.002 -0.162 -0.049
cate_intercept|(T1)$_1$ 0.037 0.034 1.079 0.281 -0.019 0.093
cate_intercept|(T0)$_2$ -0.743 0.005 -140.503 0.0 -0.752 -0.734
cate_intercept|(T1)$_2$ 0.48 0.005 91.061 0.0 0.472 0.489


A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:
$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$
where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter." + ], + "text/plain": [ + "\n", + "\"\"\"\n", + " Coefficient Results \n", + "==================================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "------------------------------------------------------------------\n", + "X0|(T0)$_0$ 0.009 0.045 0.203 0.839 -0.065 0.083\n", + "X0|(T1)$_0$ 0.017 0.042 0.416 0.677 -0.051 0.086\n", + "X0|(T0)$_1$ -0.001 0.041 -0.035 0.972 -0.069 0.067\n", + "X0|(T1)$_1$ -0.031 0.041 -0.76 0.447 -0.099 0.036\n", + "X0|(T0)$_2$ -0.306 0.008 -36.667 0.0 -0.32 -0.292\n", + "X0|(T1)$_2$ 0.158 0.008 19.656 0.0 0.145 0.171\n", + "X1|(T0)$_0$ 0.017 0.044 0.378 0.706 -0.056 0.09\n", + "X1|(T1)$_0$ -0.007 0.045 -0.164 0.87 -0.082 0.067\n", + "X1|(T0)$_1$ -0.034 0.042 -0.821 0.412 -0.103 0.034\n", + "X1|(T1)$_1$ -0.025 0.042 -0.6 0.549 -0.095 0.044\n", + "X1|(T0)$_2$ -0.302 0.008 -35.72 0.0 -0.316 -0.288\n", + "X1|(T1)$_2$ 0.156 0.008 18.801 0.0 0.142 0.169\n", + " CATE Intercept Results \n", + "===============================================================================\n", + " point_estimate stderr zstat pvalue ci_lower ci_upper\n", + "-------------------------------------------------------------------------------\n", + "cate_intercept|(T0)$_0$ 0.024 0.036 0.653 0.513 -0.036 0.084\n", + "cate_intercept|(T1)$_0$ -0.033 0.036 -0.929 0.353 -0.092 0.025\n", + "cate_intercept|(T0)$_1$ -0.105 0.034 -3.067 0.002 -0.162 -0.049\n", + "cate_intercept|(T1)$_1$ 0.037 0.034 1.079 0.281 -0.019 0.093\n", + "cate_intercept|(T0)$_2$ -0.743 0.005 -140.503 0.0 -0.752 -0.734\n", + "cate_intercept|(T1)$_2$ 0.48 0.005 91.061 0.0 0.472 0.489\n", + "-------------------------------------------------------------------------------\n", + "\n", + "A linear parametric conditional average treatment effect (CATE) model was fitted:\n", + "$Y = \\Theta(X)\\cdot T + g(X, W) + \\epsilon$\n", + "where for every outcome $i$ and treatment $j$ the CATE $\\Theta_{ij}(X)$ has the form:\n", + "$\\Theta_{ij}(X) = \\phi(X)' coef_{ij} + cate\\_intercept_{ij}$\n", + "where $\\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\\_intercept_{ij}$ parameter.\n", + "\"\"\"" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "est.summary()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Average effect of default policy:-0.42\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "A scalar was specified but there are multiple treatments; the same value will be used for each treatment. Consider specifyingall treatments, or using the const_marginal_effect method.\n", + "A scalar was specified but there are multiple treatments; the same value will be used for each treatment. Consider specifyingall treatments, or using the const_marginal_effect method.\n" + ] + } + ], + "source": [ + "# Average treatment effect for test points\n", + "X_test = X[np.arange(0, 25, 3)]\n", + "print(f\"Average effect of default policy:{est.ate(X=X_test):0.2f}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Effect of target policy over baseline policy for test set:\n", + " [-0.37368525 -0.30896804 -0.43030363 -0.52252401 -0.42849622 -0.48790877\n", + " -0.34417987 -0.51804937 -0.36806744]\n" + ] + } + ], + "source": [ + "# Effect of target policy over baseline policy\n", + "# Must specify a treatment for each period\n", + "baseline_policy = np.zeros((1, n_periods * n_treatments))\n", + "target_policy = np.ones((1, n_periods * n_treatments))\n", + "eff = est.effect(X=X_test, T0=baseline_policy, T1=target_policy)\n", + "print(\"Effect of target policy over baseline policy for test set:\\n\", eff)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(array([ 0.02374269, -0.03302781, -0.10526464, 0.03675719, -0.74294675,\n", + " 0.48025068]),\n", + " array([[ 0.00914226, 0.01675409],\n", + " [ 0.01732804, -0.00741467],\n", + " [-0.00143705, -0.03431712],\n", + " [-0.03136295, -0.02536834],\n", + " [-0.30581311, -0.30189654],\n", + " [ 0.15773252, 0.15564665]]))" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Coefficients: intercept is of shape n_treatments*n_periods\n", + "# coef_ is of shape (n_treatments*n_periods, n_hetero_inds).\n", + "# first n_treatment rows are from first period, next n_treatment\n", + "# from second period, etc.\n", + "est.intercept_, est.coef_" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [], + "source": [ + "# Confidence intervals\n", + "conf_ints_intercept = est.intercept__interval(alpha=0.05)\n", + "conf_ints_coef = est.coef__interval(alpha=0.05)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.3 Performance Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "# parse true parameters in array of shape (n_treatments*n_periods, 1 + n_hetero_inds)\n", + "# first column is the intercept\n", + "true_effect_inds = []\n", + "for t in range(n_treatments):\n", + " true_effect_inds += [t * (1 + n_x)] + (list(t * (1 + n_x) + 1 + het_inds) if len(het_inds)>0 else [])\n", + "true_effect_params = dgp.true_hetero_effect[:, true_effect_inds]\n", + "true_effect_params = true_effect_params.reshape((n_treatments*n_periods, 1 + het_inds.shape[0]))" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], + "source": [ + "# concatenating intercept and coef_\n", + "param_hat = np.hstack([est.intercept_.reshape(-1, 1), est.coef_])\n", + "lower = np.hstack([conf_ints_intercept[0].reshape(-1, 1), conf_ints_coef[0]])\n", + "upper = np.hstack([conf_ints_intercept[1].reshape(-1, 1), conf_ints_coef[1]])" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABDAAAAFgCAYAAABNIolGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAySklEQVR4nO3de7hdZX0v+u+bCwmXJIRLkBAwZFc93EIskepuqyAXQauwt9tocXvBosdHKPt42CK7IiBlt9pa3c2RVqkX8HjBFKqo6HZD1BZOqRhsGmSjYDFCACEEiCAkJOQ9f8xFClnJzGWurDHWnJ/P84xnzjHmWHP8Ruab75rzt8YYs9RaAwAAANBm45ouAAAAAGBrNDAAAACA1tPAAAAAAFpPAwMAAABoPQ0MAAAAoPUmNF1AN/vss0+dPXt202XAyFu3rnM7cWKzdUCShx5fmyTZZ49JDVcCQ2QkLSMnaR05SZ+75ZZbHqq17rvp8lY3MGbPnp0lS5Y0XQZAX/ubf7grSfLOl89puBKAdpKTAKOrlPKLzS13CgkAAADQehoY0IR//dfOBMBwMhKgOznJgGr1KSTQt55+uukKANpLRgJ0JycZUBoYAAAA9L1169ZlxYoVWbNmTdOlMGTy5MmZNWtWJm7jBWk1MAAAAOh7K1asyJQpUzJ79uyUUpouZ+DVWrNq1aqsWLEiBx988Db9jGtgAAAA0PfWrFmTvffeW/OiJUop2XvvvbfriBhHYEATpkxpugKA9pKRAN3JyR2medEu2/t6aGBAE/bfv+kKANpLRgJ0JycZUE4hAQAAgFHwwAMP5LTTTsucOXNy1FFH5WUve1m++tWvjmoNy5cvz+GHH/6cZbfeemvmzZuXefPmZa+99srBBx+cefPm5fjjj9/m5/zSl760cf7yyy/PWWedNaJ1JxoY0Iw77+xMAAwnIwG6k5Oj5sFfrcmCT92UBx/r/ZtLaq059dRT8/KXvzx33XVXbrnlllx55ZVZsWLFsHXXr1/f8/a2xxFHHJGlS5dm6dKled3rXpc///M/z9KlS3P99ddvU02bNjB2Fg0MaEKtnQmA4WQkQHejlZPLFiUfPzy5aM/O7bJFO3+bLbNw8Z354fKHs/D63htG3/3ud7PLLrvk3e9+98Zlz3/+8/OHf/iHSTpHLbzhDW/Ia1/72px44ol5+OGHc+qpp2bu3Ll56UtfmmXLliVJLrroonz0ox/d+ByHH354li9fnuXLl+eQQw7JO9/5zhx22GE58cQT8+STTyZJbrnllhx55JF52ctelksvvXSbaz7mmGPyR3/0R3nFK16Rv/zLv8zb3/72XHXVVRsf32OPPZIk5513Xm644YbMmzcvH//4x5Mk9913X0466aS84AUvyLnnnruD/2rPpYEBAADAcy1blHzj7GT1PUlq5/YbZw9ME+NF5387s8+7Nl/4wd2pNfnCD+7O7POuzYvO//YOP+dtt92W3/zN3+y6zk033ZQrrrgi3/3ud3PhhRfmxS9+cZYtW5Y/+ZM/yVvf+tatbuPOO+/MmWeemdtuuy177rlnrr766iTJ6aefnoULF+amm27a7rofffTR/P3f/33OOeecLa7z4Q9/OL/7u7+bpUuX5r3vfW+SZOnSpfnKV76SW2+9NV/5yldyzz33bPe2N6WBAQAAwHMtvjhZ9+Rzl617srN8ANxw7rF53byZmTyx85F58sRxOWXezNzw/mNHbBtnnnlmjjzyyLzkJS/ZuOyEE07IXnvtlSS58cYb85a3vCVJ8spXvjKrVq3K6tWruz7nM9euSJKjjjoqy5cvz+rVq/Poo4/mFa94RZJsfM5t9cY3vnG71n/Gcccdl2nTpmXy5Mk59NBD84tf/GKHnufZNDAAAAB4rtXDr8vQdXmfmTF1cqZMmpC16zdk0oRxWbt+Q6ZMmpAZUybv8HMedthh+dGPfrRx/tJLL83ixYuzcuXKjct23333jffrZk4TKqVkwoQJ2bBhw8Zla9b82/U5Jk2atPH++PHjs379+tRae/r62GfX9Oxt11rz1FNPbfHnNldLrzQwoAnTpnUmAIaTkQDdjUJO3lv33q7l/eihx9fmzb/1/Hz1Pb+dN//W87Py8bU9Pd8rX/nKrFmzJn/913+9cdkTTzyxxfVf/vKX54tf/GKS5Pvf/3722WefTJ06NbNnz97YCPnRj36Un//85123u+eee2batGm58cYbk2Tjc+6I2bNn55ZbbkmSXHPNNVm3bl2SZMqUKXnsscd2+Hm31YSdvgVguP32a7oCgPaSkQDdjUJO7vHqi/PU/3xvdqn/9qH9qTIpe7x6ME4hSZJPvWX+xvuXnHp4lzW3TSklX/va1/Le9743f/Znf5Z99903u+++ez7ykY9sdv2LLroop59+eubOnZvddtstV1xxRZLk9a9/fT7/+c9n3rx5eclLXpIXvvCFW9325z73ubzjHe/Ibrvtlle96lU7vA/vfOc7c8opp+Too4/Occcdt/HojLlz52bChAk58sgj8/a3vz3Tp0/f4W10UzZ3WEpbzJ8/vy5ZsqTpMgD62t/8w11Jkne+fE7DlQC0k5xkUC36zF/k39/9V5mZVbkve+cfD3pPFvzBli/k2Ha33357DjnkkKbLYBObe11KKbfUWudvuq4jMKAJd9zRud2GbinAwJGRAN2NUk4u3uUVWfbiV+W0ow/Kl26+OysfW5MFO3WL0J0GBgAAAMOM9CkU0CsX8QQAAABaTwMDAAAAaD0NDAAAAKD1XAMDmrCTvlYIoC/ISIDu5OTo+dxrOrenX9tsHSRxBAY0Y999OxMAw8lIgO7k5Jg1fvz4zJs3L4cddliOPPLIfOxjH8uGDRsaqWXJkiU5++yzu64ze/bsHHHEETniiCNy6KGH5vzzz8/atWuTJMuXL08pJR/84Ac3rv/QQw9l4sSJOeuss5IkF110UT760Y+OWM0aGNCEDRs6EwDDyUiA7uTk6Fi2KFnxw+QXNyYfP7wz36Ndd901S5cuzW233Zbrrrsu3/rWt/KhD31oBIrdfvPnz8/ChQu3ut73vve93Hrrrbn55ptz11135V3vetfGx+bMmZNvfvObG+f/9m//NocddthOqTcZoQZGKeWkUspPSyk/K6Wc12W9l5RSni6l/KeR2C6MWT/7WWcCYDgZCdCdnNz5li1KvnF28nTnaIOsvqczPwJNjGfMmDEjl112WT7xiU+k1prf/d3fzdKlSzc+/tu//dtZtmxZLrroorzjHe/IMccckzlz5jyn6XDqqafmqKOOymGHHZbLLrts4/I99tgj73//+3PUUUfl+OOPz80337zx57/+9a8nSb7//e/n937v95Ikjz/+eE4//fQcccQRmTt3bq6++uph9e6xxx755Cc/ma997Wt5+OGHk3QaMoccckiWLFmSJPnKV76SBQsWjNi/0aZ6bmCUUsYnuTTJyUkOTfL7pZRDt7DeR5J8p9dtAgAAwE6z+OJk3ZPPXbbuyc7yETRnzpxs2LAhDz74YM4444xcfvnlSZI77rgja9euzdy5c5MkP/nJT/Kd73wnN998cz70oQ9l3bp1SZLPfvazueWWW7JkyZIsXLgwq1atSpL8+te/zjHHHJNbbrklU6ZMyfnnn5/rrrsuX/3qV3PBBRcMq+OP//iPM23atNx6661ZtmxZXvnKV2623qlTp+bggw/OnXfeuXHZm970plx55ZVZsWJFxo8fn5kzZ47kP9FzjMQRGEcn+Vmt9a5a61NJrkxyymbW+8MkVyd5cAS2CQAAADvH6hXbt7wHtdYkyRve8IZ885vfzLp16/LZz342b3/72zeu85rXvCaTJk3KPvvskxkzZuSBBx5IkixcuDBHHnlkXvrSl+aee+7Z2FjYZZddctJJJyVJjjjiiLziFa/IxIkTc8QRR2T58uXDarj++utz5plnbpyf3uVCsc/U+4yTTjop1113Xb785S/njW984w79G2yrkWhgHJDknmfNrxhatlEp5YAk/yHJJ0dgewAAALDzTJu1fct30F133ZXx48dnxowZ2W233XLCCSfkmmuuyaJFi3LaaadtXG/SpEkb748fPz7r16/P97///Vx//fW56aab8i//8i958YtfnDVr1iRJJk6cmFJKkmTcuHEbf37cuHFZv379sDpqrRvX7+axxx7L8uXL88IXvnDjsl122SVHHXVU/uIv/iKvf/3rd+wfYhuNRANjc3tZN5n/H0neX2t9eqtPVsq7SilLSilLVq5cOQLlAQAAwHY47oJk4q7PXTZx187yEbJy5cq8+93vzllnnbWxeXDGGWfk7LPPzkte8pLstddeXX9+9erVmT59enbbbbf85Cc/yT/90z/tcC0nnnhiPvGJT2ycf+SRR4at8/jjj+c973lPTj311GFHaJxzzjn5yEc+kr333nuHa9gWI9HAWJHkwGfNz0py3ybrzE9yZSlleZL/lOSvSimnbu7Jaq2X1Vrn11rn7+urgehXe+/dmQAYTkYCdCcnd765C5LXLkzGDx35MO3Azvzc3i5Q+eSTT278GtXjjz8+J554Yi688MKNjx911FGZOnVqTj/99K0+10knnZT169dn7ty5+eAHP5iXvvSlO1zX+eefn0ceeSSHH354jjzyyHzve9/b+Nixxx6bww8/PEcffXQOOuigfOpTnxr284cddlje9ra3bfa5L7nkksyaNWvj1Iuy6fkr2/0EpUxIckeS45Lcm+SHSU6rtd62hfUvT/LNWutVW3vu+fPn12euZgrAzvE3/3BXkuSdL5/TcCUA7SQnoT/cfvvtOeSQQ7bvhz73ms7t6deOfEGbcd999+WYY47JT37yk4wbNyJfGtp6m3tdSim31Frnb7ruhF43VmtdX0o5K51vFxmf5LO11ttKKe8eetx1L2BTz5x3NqHn/4IA/UdGAnQnJ0fPKDUukuTzn/98PvCBD+RjH/vYwDQvtteIjPha67eSfGuTZZttXNRa3z4S24Qx7a7OX3LyrIvfADBERgJ0Jyf70lvf+ta89a1vbbqMVtPWAQAAYCD0egkFRtb2vh4aGAAAAPS9yZMnZ9WqVZoYLVFrzapVqzJ58uRt/hknTQEAAND3Zs2alRUrVmTlypVNl8KQyZMnb9c3k2hgAAAA0PcmTpyYgw8+uOky6IEGBjRh332brgCgvWQkQHdykgGlgQFNmD696QoA2ktGAnQnJxlQLuIJTXjqqc4EwHAyEqA7OcmA0sCAJixf3pkAGE5GAnQnJxlQGhgAAABA62lgAAAAAK2ngQEAAAC0ngYGAAAA0Hq+RhWasN9+TVcA0F4yEqA7OcmA0sCAJkyb1nQFAO0lIwG6k5MMKKeQQBPWrOlMAAwnIwG6k5MMKA0MaMLdd3cmAIaTkQDdyUkGlAYGAAAA0HoaGAAAAEDraWAAAAAAraeBAQAAALSer1GFJjzveU1XANBeMhKgOznJgNLAgCZMndp0BQDtJSMBupOTDCinkEATnniiMwEwnIwE6E5OMqA0MKAJK1Z0JgCGk5EA3clJBpQGBgAAANB6GhgAAABA62lgAAAAAK2ngQEAAAC0nq9RhSbMnNl0BQDtJSMBupOTDCgNDGjCHns0XQFAe8lIgO7kJAPKKSTQhMcf70wADCcjAbqTkwyoEWlglFJOKqX8tJTys1LKeZt5/M2llGVD0z+WUo4cie3CmHXffZ0JgOFkJEB3cpIB1XMDo5QyPsmlSU5OcmiS3y+lHLrJaj9P8opa69wkf5zksl63CwAAAAyOkTgC4+gkP6u13lVrfSrJlUlOefYKtdZ/rLU+MjT7T0lmjcB2AQAAgAExEg2MA5Lc86z5FUPLtuQPknx7Sw+WUt5VSllSSlmycuXKESgPAAAAGOtGooFRNrOsbnbFUo5Np4Hx/i09Wa31slrr/Frr/H333XcEygMAAADGupH4GtUVSQ581vysJMOuKFNKmZvk00lOrrWuGoHtwtg1y1lUAFskIwG6k5MMqJE4AuOHSV5QSjm4lLJLkjcl+fqzVyilHJTk75K8pdZ6xwhsE8a23XbrTAAMJyMBupOTI+LBX63Jgk/dlAcfW9N0KSOuX/et5wZGrXV9krOSfCfJ7UkW1VpvK6W8u5Ty7qHVLkiyd5K/KqUsLaUs6XW7MKb96ledCYDhZCRAd3JyRCxcfGd+uPzhLLz+zqZLGXH9um8jcQpJaq3fSvKtTZZ98ln3z0hyxkhsC/rCL3/ZuZ06tdk6ANpIRgJ0Jyd78qLzv51XbfiHnDthUS7e5aHc98/75OwfLsh3xr08P73k5KbL68mLzv921q7fsHH+Cz+4O1/4wd2ZNGHcmN+3ZIQaGAAAADAW3Pzah7Pr//xMdqlrkySzykP56KTP5I9PPrzhynp3w7nH5ttfXpjj7/tk9s+q3J99snjm/5mTTju76dJGxEhcAwMAAADGhGn/+KcbmxfP2KWuzbT/708bqmjkzFj+9fz+Lz+aA8qqjCvJAeWhvOmXH82Mn3996z88BmhgAAAAMDhWr9i+5WPJ4os325zJ4osbKmhkaWAAAAAwOKZt4Wtot7R8LOnn5kw0MKAZBx3UmQAYTkYCdCcne3PcBcnEXZ+7bOKuneVjXT83Z6KBAc2YPLkzATCcjAToTk72Zu6C5LULk/GTOvPTDuzMz13QbF0joZ+bM/EtJNCM1as7t9OmNVsHQBvJSIDu5GTv5i7oj4bFpp7Zp8UXd04bmTar07zok33VwIAmPPBA59YvHYDhZCRAd3KSbvq1OROnkAAAAABjgAYGAAAA0HoaGAAAAEDraWAAAAAArecintCE2bObrgCgvWQkQHdykgGlgQFJ8rnXdG5Pv3Z0trfLLqOzHYCxaLQzcrR/BwD0yntJBpRTSNh2n3vNv73JozePPNKZ6I0xCf1JRgJ0JycZUI7AGGn+isO2WLmyczt9erN1ALSRjAToTk4yoByBAQAAALSeBgZAGzk9BgAAnkMDAwBGisYTAMBOo4EBAAAAtJ6LeEIT5sxpugKA9pKRAN3JSQaUBgY0YYL/egBbJCMBupOTDCinkEATVq3qTAAMJyPZGtebYdDJSQaUBgY0wS8dgC0bzYxctihZ8cPkFzcmHz+8Mw/Qdt5LMqA0MACArevHv3gvW5R84+zk6bWd+dX3dOb7pYnRj68ZAANNAwMAGEyLL07WPfncZeue7CwHAFpHA2MkOQyVNvIXOIDNW71i+5YDAI3SwBgp/X4Yaj/TeBqbvG5Ar6bN2r7lAECjNDBGisNQx6amGk+/8RudiR2jYQj9bbQy8rgLkom7PnfZxF07ywHazHtJBpQGxkjp98NQ+/Wv3U01nsaN60zsGA1D6G+jlZFzFySvXZinx0/OhiRPT5mVvHZhZzlAm3kvyYAakVFfSjmplPLTUsrPSinnbebxUkpZOPT4slLKb47Edlulnw9D7ee/djfVeFq5sjOxY/q9YQiDbjQzcu6CXDjhvfl3a76YC+dc2VfNiwfX75YFd5+SBx9b03QpI66f9w22ifeSDKieGxillPFJLk1ycpJDk/x+KeXQTVY7OckLhqZ3JfnrXrfbOv18GGo//7W7qcbTI490JnZMPzcMh3hzPgb165FqTRiljHzR+d/O7POuzRdWH5Gaki/84O7MPu/avOj8b+/0bY+GhQ8dlR8+OTMLr7+z6VJGXD/vG2wT7yUZUCNxBMbRSX5Wa72r1vpUkiuTnLLJOqck+Xzt+Kcke5ZS9h+BbbfGixZNydm/Pj0rNuyTDbVkxYZ9cvavT8+LFk1purSebXh083/V3tLyseSch0/JE3WX5yx7ou6Scx7edAjTKv3cMEySZYsy8d5/ypUb3peJC+f6IDwW9PORan3shnOPzevmzczksi5JMnniuJwyb2ZueP+xDVfWm35uzPTzvgGwdSPRwDggyT3Pml8xtGx71xnTbjj32GTughy/7i8yZ+0Xc3z9RMrcBWP+TVCS1Kmbf6m2tHwsef/7Ppi/nfm+3Fv3zoZacm/dJ1fNfF/ef+4Hmy5txPTjX/L7uWF4zgc/kCeuPjPT86uMK8n0dQ/kiavPzDkf/EDTpdFNPx+p1sdmTJ2cKZMmZG2dkEllfdau35ApkyZkxpTJTZfWk35tzCT9vW8AbN1INDDKZpbVHVins2Ip7yqlLCmlLFk5hs7r6tc3QUky/oQL81SZ9JxlT5VJGX/ChQ1VNHJmTJ2cO2acnN9ZuzCHPHVFfuephbljxsl98bo9ox8Ps+3nhuGf7fm17Faees6y3cpT+bM9v9ZMQWwb12UZsx56fG3ePO3H+epBV+fNv/X8rHx8bdMl9ayf35P0874BsHUTRuA5ViQ58Fnzs5LctwPrJElqrZcluSxJ5s+fv9kmR1s98ybotD3/d760/7lZ2S9/8Z67IJ+/8a68/sFLs2d+lUcn7perp78jZ/TJhc769XV70fnfztr1G5IckST5wg/uzhd+cHcmTRiXn15ycrPF9Wj4G9gJffMGdvxj927Xclpi2qzOaSObW06rfeot85PPdRryl5x6eMPVjJx+/d2W9Pe+AdDdSDQwfpjkBaWUg5Pcm+RNSU7bZJ2vJzmrlHJlkt9KsrrWev8IbLtV+vVNUJKc8Z7zks/dkCSZfvq1OaPhekZSI6/bC1+40zdxw7nH5pJv3Z7/9S+/yJo6MZMnjsurDntePvCaQ3b6tkdD376B9UF4bDrugs41L559Gkk/XZdltI1CRva7fn5P8qkj70qWfyx5YG0uWfOjof9n85suC0aXnGRA9dzAqLWuL6WcleQ7ScYn+Wyt9bZSyruHHv9kkm8leXWSnyV5IsnpvW4X6G7G1Mn5nV9/N+dO/KvMLA/lvuyTf/z1ezJjyoubLm1E9O2bcx+Ex6Znjki75qzOhTynHdh5zfrkSLWN37Dy9NrON6z00771q359zbZ0wdykP/YPgK5G4giM1Fq/lU6T4tnLPvms+zXJmSOxLegLDzzQud1vv523jWWLcsqKj2TSuM6bvFl5KKes+Eiy7EBv8tqs3z8I97O5C/LgD/42Z913Qj5xxlv74pSmJM18YByNjOxn/fwhv9sFc8f6vsH2kJMMqJG4iCewvVav7kw70+KLM6k+92J0k+pa34owFsxdkMx6SfL830ne+2NvyseQfrxobiPfsDIaGflsp1/bmfpFP38rjgvmQsdo5yS0xIgcgQG0kDd5MGr6+aK5Gx5dkXGb+S6xDY+u8FeQturn/HedIICB5r0H9KstvZnzJo+mfe41namP3HDusXndvJmZXNYlSSZPHJdT5s3si6/2rVMP2K7ltEA/5/9xF3SuC/RsrhMEMDA0MKBfeZMHo2b4V/tu6J+v9j3hwjxVJj1n2VNlUsafcGFDFbFV/Zz/cxckr12YjB8ak9MO7Mw71Q5gIDiFZKT10zm07DxlM8djjzQXg4RR1bdf7Tt3QT5/4115/YOXZs/8Ko9O3C9XT39HztiZWTIaGdnP+j3/5y5Ibrmic9/7LgaVnGRAaWBAE17wgtHZjjd5MGr69qt9k5zxnvOSz92QJJl++rU5Y2dvcLQysp/Jf+hvcpIB5RQSAAAAoPUcgQFNuP/+zu3++zdbx1jnr4rQn2QkQHdykgGlgQHJ6H8Qfuyxzq1fOgDDyUiA7uQkA8opJAAAAEDrOQKDbedwfQAAABqigQEAI0WjFwBgp3EKCTRh/PjOBINm2aI8vWJJNvzixjz9F4clyxY1XRFtJCMBupOTDChHYEAT/t2/a7oCGH3LFiXfODvjn17TmX9sRfKNszv35y5ori7aR0YCdCcnGVCOwABoo9Ov7bvTEe69+r8l65587sJ1T3aWAwDAVjgCA5pw772d2wMO2Pnb6rMPwYxdM8uq7VrOABvNjAQYi+QkA0oDA5rw6183XQGMujJtVrL6ns0vh2eTkWyN5jyDTk4yoDQwABgdx12QtV89K5Pq2o2L1pZJmXTcBQ0WxTbzgXHs8ZoB0GdcAwOA0TF3QSb9h08k4yd15qcd2Jl3AU8AALaBIzAAGD1zFyS3XNG576/DAABsBw0MaMLEiU1XANBeMhKgOznJgNLAgCYcfHDTFQC0l4wE6E5OMqBcAwMAAABoPQ0MaMI993QmAIaTkQDdyUkGlFNIoAlPPtl0BQDtJSMBupOTDChHYAAAAACtp4EBAAAAtJ4GBgAAANB6roEBTZg0qekKANpLRgJ0JycZUBoY0ITnP7/pCqA5p1/bdAW0nYwE6E5OMqCcQgIAAAC0Xk8NjFLKXqWU60opdw7dTt/MOgeWUr5XSrm9lHJbKeW/9LJN6Au/+EVnAmA4GQnQnZxkQPV6BMZ5SRbXWl+QZPHQ/KbWJzmn1npIkpcmObOUcmiP24Wxbe3azgTAcDISoDs5yYDqtYFxSpIrhu5fkeTUTVeotd5fa/3R0P3Hktye5IAetwsAAAAMkF4bGPvVWu9POo2KJDO6rVxKmZ3kxUl+0GWdd5VSlpRSlqxcubLH8gAAAIB+sNVvISmlXJ/keZt56APbs6FSyh5Jrk7yf9Vaf7Wl9WqtlyW5LEnmz59ft2cbAAAAQH/aagOj1nr8lh4rpTxQStm/1np/KWX/JA9uYb2J6TQvvlhr/bsdrhb6xa67Nl0BQHvJSIDu5CQDaqsNjK34epK3Jfnw0O01m65QSilJPpPk9lrrx3rcHvSHAw9sugKA9pKRAN3JSQZUr9fA+HCSE0opdyY5YWg+pZSZpZRvDa3z20nekuSVpZSlQ9Ore9wuAAAAMEB6OgKj1roqyXGbWX5fklcP3b8xSellO9B3fv7zzu3BBzdbB0AbyUiA7uQkA6rXU0iAHbFuXdMVALSXjAToTk4yoHo9hQQAAABgp9PAAAAAAFpPAwMAAABoPdfAgCbsvnvTFQC0l4wE6E5OMqA0MKAJBxzQdAUA7SUjAbqTkwwop5AAAAA73YO/WpMFn7opDz62pulSRlw/7xu0iQYGNOFf/7UzATCcjIS+tHDxnfnh8oez8Po7my5lxI36vslJBpRTSKAJTz/ddAUA7SUjoa+86PxvZ+36DRvnv/CDu/OFH9ydSRPG5aeXnNxgZb1rbN/kJAPKERgAAMBOc8O5x+Z182Zm8sTOR4/JE8fllHkzc8P7j224st71875BG2lgAAAAO82MqZMzZdKErF2/IZMmjMva9RsyZdKEzJgyuenSetbP+wZt5BQSAABgp3ro8bV58289P6cdfVC+dPPdWdlHF7s8+P5r86NJl2bP/CqP7rpfrr7/HUmOaLos6EsaGNCEKVOargCgvWQk9J1PHXlXcs1ZydK1uWTagclxFySZ33RZvVu2KGc88j+SPJkkmb7ugc78sjnJ3AU7b7tykgGlgQFN2H//pisAaC8ZCf1l2aLkG2cnT6/tzK++pzOf7NwP+aNh8cXJuiefu2zdk53lO3Pf5CQDyjUwAACAnafbh/yxbvWK7VsO9EQDA5pw552dCYDhZCT0l37+kD9t1vYtHylykgGlgQFNqLUzATCcjIT+0tSH/NFw3AXJxF2fu2zirkPX+NiJ5CQDSgMDAADYeZr6kD8a5i5IXrswmXZgktK5fe3CsX9tD2gpF/EEAAB2nmc+zC++uHPayLRZneZFv3zIn7ugf/YFWk4DAwAA2Ll8yAdGgAYGNGHatKYrAGgvGQnQnZxkQGlgQBP226/pCgDaS0YCdCcnGVAu4gkAAAC0ngYGNOGOOzoTAMPJSIDu5CQDSgMDAAAAaD0NDAAAAKD1NDAAAACA1tPAAAAAAFrP16hCE6ZPb7oCgPaSkQDdyUkGlAYGNGHffZuuAKC9ZCRAd3KSAdXTKSSllL1KKdeVUu4cut1iK7CUMr6U8s+llG/2sk3oCxs2dCYAhpORAN3JSQZUr9fAOC/J4lrrC5IsHprfkv+S5PYetwf94Wc/60wADCcjAbqTkwyoXhsYpyS5Yuj+FUlO3dxKpZRZSV6T5NM9bg8AAAAYQL02MPartd6fJEO3M7aw3v9Icm6SrR7nVEp5VyllSSllycqVK3ssDwAAAOgHW72IZynl+iTP28xDH9iWDZRSfi/Jg7XWW0opx2xt/VrrZUkuS5L58+fXbdkGAAAA0N+22sCotR6/pcdKKQ+UUvavtd5fStk/yYObWe23k7yulPLqJJOTTC2lfKHW+p93uGoAAABgoPR6CsnXk7xt6P7bklyz6Qq11v9Wa51Va52d5E1Jvqt5wcDbe+/OBMBwMhKgOznJgNrqERhb8eEki0opf5Dk7iRvSJJSyswkn661vrrH54f+5BcOwJbJSIDu5CQDqqcGRq11VZLjNrP8viTDmhe11u8n+X4v24S+sH5953ZCrz1EgD4kIwG6k5MMqF5PIQF2xF13dSYAhpORAN3JSQaUBgYAAADQehoYAAAAQOtpYAAAAACtp4EBAAAAtJ7L1kIT9t236QoA2ktGAnQnJxlQGhjQhOnTm64AoL1kJEB3cpIB5RQSaMJTT3UmAIaTkQDdyUkGlAYGNGH58s4EwHAyEqA7OcmA0sAAAAAAWk8DAwAAAGg9DQwAAACg9TQwAAAAgNbzNarQhP32a7oCgPaSkQDdyUkGlAYGNGHatKYrAGgvGQnQnZxkQDmFBJqwZk1nAmA4GQnQnZxkQGlgQBPuvrszATCcjAToTk4yoDQwAAAAgNbTwAAAAABaTwMDAAAAaD0NDAAAAKD1fI0qNOF5z2u6AoD2kpEA3clJBpQGBjRh6tSmKwBoLxkJ0J2cZEA5hQSa8MQTnQmA4WQkQHdykgGlgQFNWLGiMwEwnIwE6E5OMqA0MAAAAIDW08AAAAAAWk8DAwAAAGg9DQwAAACg9XyNKjRh5symKwBoLxkJ0J2cZEBpYEAT9tij6QoA2ktGAnQnJxlQPZ1CUkrZq5RyXSnlzqHb6VtYb89SylWllJ+UUm4vpbysl+3CmPf4450JgOFkJEB3cpIB1es1MM5LsrjW+oIki4fmN+cvk/zPWuv/keTIJLf3uF0Y2+67rzMBMJyMBOhOTjKgem1gnJLkiqH7VyQ5ddMVSilTk7w8yWeSpNb6VK310R63CwAAAAyQXhsY+9Va70+SodsZm1lnTpKVST5XSvnnUsqnSym7b+kJSynvKqUsKaUsWblyZY/lAQAAAP1gqw2MUsr1pZQfb2Y6ZRu3MSHJbyb561rri5P8Ols+1SS11stqrfNrrfP33XffbdwEAAAA0M+2+i0ktdbjt/RYKeWBUsr+tdb7Syn7J3lwM6utSLKi1vqDofmr0qWBAQAAALCpXr9G9etJ3pbkw0O312y6Qq31l6WUe0opL6q1/jTJcUn+d4/bhbFt1qymKwBoLxkJ0J2cZED12sD4cJJFpZQ/SHJ3kjckSSllZpJP11pfPbTeHyb5YilllyR3JTm9x+3C2Lbbbk1XANBeMhKgOznJgOqpgVFrXZXOERWbLr8vyaufNb80yfxetgV95Ve/6txOndpsHQBtJCMBupOTDKhej8AAdsQvf9m59UsHYDgZCdCdnGRA9fo1qgAAAAA7nQYGAAAA0HoaGAAAAEDraWAAAAAArecintCEgw5qugKA9pKRAN3JSQaUBgY0YfLkpisAaC8ZCdCdnGRAOYUEmrB6dWcCYDgZCdCdnGRAOQIDmvDAA53badOarQOgjWQkQHdykgHlCAwAAACg9TQwAAAAgNbTwAAAAABaTwMDAAAAaD0X8YQmzJ7ddAUA7SUjAbqTkwwoDQxowi67NF0BQHvJSIDu5CQDyikk0IRHHulMAAwnIwG6k5MMKEdgQBNWruzcTp/ebB0AbSQjAbqTkwwoR2AAAAAAraeBAQAAALSeBgYAALTBskXJxw9PLtqzc7tsUdMVAbSKa2AAAEDTli1KvnF2su7JzvzqezrzSTJ3QXN1AbSIBgY0Yc6cpisAaC8ZySBafPG/NS+ese7JznINDDYlJxlQGhjQhAn+6wFskYxkEK1esX3LGWxykgHlGhjQhFWrOhMAw8lIBtG0Wdu3nMEmJxlQGhjQBL90ALZMRjKAznn4lDxRd3nOsifqLjnn4VMaqohWk5MMKA0MAABo2Pvf98H87cz35d66TzbUknvrPrlq5vvy/nM/2HRpAK3h5CkAAGjYjKmTc8eMk3PR8sOyy/hxeerpDXnzjIPy1imTmy4NoDU0MAAAoAUeenxt3vxbz89pRx+UL918d1Y+tqbpkgBaRQMDAABa4FNvmb/x/iWnHt5gJQDtpIEBTfiN32i6AoD2kpEA3clJBlRPF/EspexVSrmulHLn0O30Laz33lLKbaWUH5dSvlxKcTIfg23cuM4EwHAyEqA7OcmA6nXUn5dkca31BUkWD80/RynlgCRnJ5lfaz08yfgkb+pxuzC2rVzZmQAYTkYCdCcnGVC9NjBOSXLF0P0rkpy6hfUmJNm1lDIhyW5J7utxuzC2PfJIZwJgOBkJ0J2cZED12sDYr9Z6f5IM3c7YdIVa671JPprk7iT3J1lda/1fW3rCUsq7SilLSilLVuoqAgAAANmGBkYp5fqha1dsOp2yLRsYui7GKUkOTjIzye6llP+8pfVrrZfVWufXWufvu+++27ofAAAAQB/b6reQ1FqP39JjpZQHSin711rvL6Xsn+TBzax2fJKf11pXDv3M3yX590m+sIM1AwAAAAOm11NIvp7kbUP335bkms2sc3eSl5ZSdiullCTHJbm9x+0CAAAAA6TUWnf8h0vZO8miJAel06h4Q6314VLKzCSfrrW+emi9DyV5Y5L1Sf45yRm11rXb8Pwrk/xihwtszj5JHmq6iJ3k4CQ/b7qInWS0X7fR3J4xOTb18+vWz/tmTI69bTWxvdFiPI5N/bxvxuTY3J4xOTaN5dft+bXWYdeU6KmBweaVUpbUWuc3XcfOUEr5da1196br2BlG+3Ubze0Zk2NTn79u/bxvxuQY21YT2xstxuPY1Of7ZkyOwe0Zk2NTP75uvZ5CAgAAALDTaWAAAAAAraeBsXNc1nQBO9HfNV3ATjTar9tobs+YHJv6+XXr530zJsfetprY3mgxHsemft43Y3Jsbs+YHJv67nVzDQwAAACg9RyBAQAAALSeBgYAAADQehoYAAAAQOtpYAAAAACtp4Exwkopny2lPFhK+XHTtfSilPK9UsoJQ/cvKaUsbLqmkdLP+7apfhmPSX+/bv28b5vqlzHZz69ZP+/b5hiT7dfP+7Y5xmT79fO+bapfxmPS369bP+/bpiY0XUAfujzJJ5J8vuE6enVhkotLKTOSvDjJ6xquZyT1875t6vL0x3hM+vt16+d929Tl6Y8x2c+vWT/v2+ZcHmOy7fp53zbn8hiTbdfP+7apy9Mf4zHp79etn/ftOXyN6k5QSpmd5Ju11sObrqUXpZS/T7JHkmNqrY+VUg5NclGSVUkWJ7k5nUB7KMkdtdYPN1Xr9tqGffvfz56vtV7VUKk965fxmBiTMSZbxXjsj/GYGJNjgTE5NhmT/TEm+2U8JsZk+mBMOgKDzSqlHJFk/yQP1VofG1p8cpL/p9Z6Qynl60keTXJtrfVTpZQx05Xdxn37+03mx+R/8H5iTBqTbWI8Go9tY0wak21jTBqTbWNM9seYdA0Mhiml7J/ki0lOSfLrUsqrhh76f5O8qZTy50n2TvLPQ/PfTfK9RordTtuxb5vO0yBj0phsE+PReGwbY9KYbBtj0phsG2Oyj8ZkrdU0wlOS2Ul+3HQdO1j7bkluSnLC0PzLk9y0yTrjk1yT5L8mefnQsquarn0k921L82NxGsvjcXtfN2NybExjeUwaj/03Hof2w5hs4WRMGpNtmwZ1TI7l8bi9r5sx2f7JNTB2gn46T+zZhvbrj5LsnuSv0znE6qJ0zhF7vNb6X5uqrVeb2bcVz56vtd7YXHW96dfxmBiTzVXXm34dk8bj2GVMjj3G5NhkTI5N/ToeE2Oyuep2nAbGCCulfDnJMUn2SfJAkgtrrZ9ptCgGlvFI2xiTtI0xSdsYk7SJ8UjbaGAAAAAArecingAAAEDraWAAAAAAraeBAQAAALSeBgYAAADQehoYAAAAQOtpYAAAAACtp4EBAAAAtJ4GBgAAANB6GhgAAABA62lgAAAAAK2ngQEAAAC0ngYGAAAA0HoaGAAAAEDraWAAAAAArTeh6QJor1LK1CR/n2SXJAcnuSPJmiT/vta6ocnaGEzGJG1jTNI2xiRtYjzSNsbk2FdqrU3XQMuVUo5O8oFa6ylN1wKJMUn7GJO0jTFJmxiPtI0xOXY5hYRtcXiS256ZKaXsXkq5opTyN6WUNzdYF4Nr0zE5p5TymVLKVQ3WxGCTk7SNnKRNZCRtIyPHKA0MtsWhSX78rPn/mOSqWus7k7yumZIYcM8Zk7XWu2qtf9BgPSAnaRs5SZvISNpGRo5RGhhsi5lJfvms+VlJ7hm6//TolwPDxiQ0TU7SNnKSNpGRtI2MHKM0MNgW30nymVLKK4bmV6TziycxhmjGpmMSmiYnaRs5SZvISNpGRo5RLuLJdiul7J7kE+lcsffGWusXGy6JAVdK2TvJf09yQpJP11r/tOGSGHBykraRk7SJjKRtZOTYoYEBAAAAtJ5DtgAAAIDW08AAAAAAWk8DAwAAAGg9DQwAAACg9TQwAAAAgNbTwAAAAABaTwMDAAAAaD0NDAAAAKD1NDAAAACA1tPAAAAAAFpPAwMAAABoPQ0MAAAAoPU0MAAAAIDW08AAAAAAWk8DAwAAAGg9DQwAAACg9TQwYBSUUsaXUv6ylHJbKeXWUsqcpmsCaBM5CdCdnAQNDBgt/y3JXbXWw5IsTPKehusBaBs5CdCdnGTgTWi6AOh3pZTdk/yHWutRQ4t+nuQ1DZYE0CpyEqA7OQkdGhiw8x2f5MBSytKh+b2SXN9cOQCtIycBupOTEKeQwGiYl+SCWuu8Wuu8JP8rydJSyu6llCtKKX9TSnlzoxUCNGteNp+Tc0opnymlXNVodQDNm5fN5+SpQ+8lrymlnNhohTAKNDBg55ue5IkkKaVMSHJikm8k+Y9Jrqq1vjPJ65orD6Bxm83JWutdtdY/aLQygHbYUk5+bei95NuTvLG58mB0aGDAzndHkpcO3X9vkmtrrT9PMivJPUPLn26iMICW2FJOAtCxtZw8P8mlo14VjDINDNj5vpzkN0spP0syN8n/PbR8RTpNjMT/RWCwbSknAejYbE6Wjo8k+Xat9UdNFgijodRam64BBtLQ1aQ/kWRNkhtrrV9suCSAViml7J3kvyc5Icmna61/2nBJAK1SSjk7yduS/DDJ0lrrJxsuCXYqDQwAAACg9Ry2DgAAALSeBgYAAADQehoYAAAAQOtpYAAAAACtp4EBAAAAtJ4GBgAAANB6GhgAAABA62lgAAAAAK33/wNlmeyogN4zIgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure(figsize=(15, 5))\n", + "plt.errorbar(np.arange(n_periods * (len(het_inds) + 1) * n_treatments),\n", + " true_effect_params.flatten(), fmt='*', label='Ground Truth')\n", + "plt.errorbar(np.arange(n_periods * (len(het_inds) + 1) * n_treatments),\n", + " param_hat.flatten(), yerr=((upper - param_hat).flatten(),\n", + " (param_hat - lower).flatten()), fmt='o', label='DynamicDML')\n", + "add_vlines(n_periods, n_treatments, het_inds)\n", + "plt.legend()\n", + "plt.show()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/setup.cfg b/setup.cfg index 266aa9d16..0d9f2db65 100644 --- a/setup.cfg +++ b/setup.cfg @@ -82,6 +82,7 @@ exclude = [options.package_data] ; include all CSV files as data * = *.csv + *.jbl ; coverage configuration [coverage:run]