+
+ LipDP is a Python toolkit dedicated to robust and certifiable learning under privacy guarantees. +
-Therefore, the computation time is heavily reduced and the results on the MNIST and CIFAR10 datasets are the following : +This package is the code for the paper "*DP-SGD Without Clipping: The Lipschitz Neural Network Way*" by Louis Béthune, Thomas Massena, Thibaut Boissin, Aurélien Bellet, Franck Mamalet, Yannick Prudent, Corentin Friedrich, Mathieu Serrurier, David Vigouroux, published at the **International Conference on Learning Representations (ICLR 2024)**. The paper is available on [arxiv](https://arxiv.org/abs/2305.16202). -# Status of the repository : -- ci tests to develop. -- sensitivity.py to debug. -- requirements.txt tested on my machine, still to check by someone else. +State-of-the-art approaches for training Differentially Private (DP) Deep Neural Networks (DNN) face difficulties to estimate tight bounds on the sensitivity of the network's layers, and instead rely on a process of per-sample gradient clipping. This clipping process not only biases the direction of gradients but also proves costly both in memory consumption and in computation. To provide sensitivity bounds and bypass the drawbacks of the clipping process, we propose to rely on Lipschitz constrained networks. Our theoretical analysis reveals an unexplored link between the Lipschitz constant with respect to their input and the one with respect to their parameters. By bounding the Lipschitz constant of each layer with respect to its parameters, we prove that we can train these networks with privacy guarantees. Our analysis not only allows the computation of the aforementioned sensitivities at scale, but also provides guidance on how to maximize the gradient-to-noise ratio for fixed privacy guarantees. To facilitate the application of Lipschitz networks and foster robust and certifiable learning under privacy guarantees, we provide this Python package that implements building blocks allowing the construction and private training of such networks. -# Deel library repository template + -Ce dépôt git sert de template pour les librairies DEEL ayant vocation à être rendues publiques sur github. -Il donne la structure des répertoires d'un projet telle que celle adoptée par les librairies DEEL déjà publiques. - -A la racine du projet on trouve: - -- deel : répertoire destiné à recevoir le code de la librairie. C'est le premier mot de l'espaces de nommage de - la librairie. Ce n'est pas un module python, il ne contient donc pas de fichier __init__.py. - Il contient le module principal de la librairie du nom de cette librairie. - - Example: - - librairie **deel-lip**: - deel/deel-lip - -- docs: répertoire destiné à la documentation de la librairie - -- tests: répertoire des tests unitaires - -- .pre-commit-config.yaml : configuration de outil de contrôle avant commit (pre-commit) - -- LICENCE/headers/MIT-Clause.txt : entête licence MIT injectée dans les fichiers du projet - -- CONTRIBUTING.md: description de la procédure pour apporter une contribution à la librairie. - -- GOUVERNANCE.md: description de la manière dont la librairie est gérée. - -- LICENCE : texte de la licence sous laquelle est publiée la librairie (MIT). - -- README.md - - -# pre-commit : Conventional Commits 1.0.0 - -The commit message should be structured as follows: - -``` -
- 
--
- Libname is a Python toolkit dedicated to making people happy and fun.
+LipDP is a Python toolkit dedicated to robust and certifiable learning under privacy guarantees.
+
+
-
-
- Explore Libname docs »
-
+This package is the code for the paper "*DP-SGD Without Clipping: The Lipschitz Neural Network Way*" by Louis Béthune, Thomas Massena, Thibaut Boissin, Aurélien Bellet, Franck Mamalet, Yannick Prudent, Corentin Friedrich, Mathieu Serrurier, David Vigouroux, published at the **International Conference on Learning Representations (ICLR 2024)**. The paper is available on [arxiv](https://arxiv.org/abs/2305.16202).
+
+
+State-of-the-art approaches for training Differentially Private (DP) Deep Neural Networks (DNN) face difficulties to estimate tight bounds on the sensitivity of the network's layers, and instead rely on a process of per-sample gradient clipping. This clipping process not only biases the direction of gradients but also proves costly both in memory consumption and in computation. To provide sensitivity bounds and bypass the drawbacks of the clipping process, we propose to rely on Lipschitz constrained networks. Our theoretical analysis reveals an unexplored link between the Lipschitz constant with respect to their input and the one with respect to their parameters. By bounding the Lipschitz constant of each layer with respect to its parameters, we prove that we can train these networks with privacy guarantees. Our analysis not only allows the computation of the aforementioned sensitivities at scale, but also provides guidance on how to maximize the gradient-to-noise ratio for fixed privacy guarantees. To facilitate the application of Lipschitz networks and foster robust and certifiable learning under privacy guarantees, we provide this Python package that implements building blocks allowing the construction and private training of such networks.
-
+
+
+
-
-This project received funding from the French ”Investing for the Future – PIA3” program within the Artificial and Natural Intelligence Toulouse Institute (ANITI). The authors gratefully acknowledge the support of the DEEL project.
+The creators thank the whole [DEEL](https://deel-ai.com/) team for its support, and [Aurélien Bellet](http://researchers.lille.inria.fr/abellet/) for his guidance.
## 👨🎓 Creators
-If you want to highlight the main contributors
-
+The library has been created by [Louis Béthune](https://github.com/Algue-Rythme), [Thomas Masséna](https://github.com/massena-t) during an internsip at [DEEL](https://deel-ai.com/), and [Thibaut Boissin](https://github.com/thib-s).
## 🗞️ Citation
-If you use Libname as part of your workflow in a scientific publication, please consider citing 🗞️ [our paper](https://www.youtube.com/watch?v=dQw4w9WgXcQ):
+If you find this work useful for your research, please consider citing it:
```
-@article{rickroll,
- title={Rickrolling},
- author={Some Internet Trolls},
- journal={Best Memes},
- year={ND}
+@inproceedings{
+bethune2024dpsgd,
+title={{DP}-{SGD} Without Clipping: The Lipschitz Neural Network Way},
+author={Louis B{\'e}thune and Thomas Massena and Thibaut Boissin and Aur{\'e}lien Bellet and Franck Mamalet and Yannick Prudent and Corentin Friedrich and Mathieu Serrurier and David Vigouroux},
+booktitle={The Twelfth International Conference on Learning Representations},
+year={2024},
+url={https://openreview.net/forum?id=BEyEziZ4R6}
}
```
## 📝 License
-The package is released under MIT license.
\ No newline at end of file
+The package is released under [MIT license](../LICENSE).
diff --git a/docs/notebooks/advanced_cifar10.ipynb b/docs/notebooks/advanced_cifar10.ipynb
new file mode 100644
index 0000000..dca2a33
--- /dev/null
+++ b/docs/notebooks/advanced_cifar10.ipynb
@@ -0,0 +1,1895 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "f7bf07b9-d489-4484-acb9-175cb740dc60",
+ "metadata": {},
+ "source": [
+ "# Cifar-10 tutorial\n",
+ "\n",
+ "This notebook introduces advanced tools like MLP mixer, which involves residual connections with Lipschitz guarantees, other input space (HSB) and loss gradient clipping.\n",
+ "\n",
+ "## Imports"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "8a0eebdf-6082-4d00-aa14-b42953217a93",
+ "metadata": {},
+ "source": [
+ "The library is based on tensorflow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "91c2965e-0375-4966-bc55-776204af9d69",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "9356cd9b-6f79-45f1-8f2e-c46a526c4ae7",
+ "metadata": {},
+ "source": [
+ "### lip-dp dependencies\n",
+ "\n",
+ "The need a model `DP_Model` that handles the noisification of gradients. It is trained with a `loss`. The model is initialized with the convenience function `DPParameters`. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "1e5d58f8-386c-44c7-8c5d-e5b69b5be231",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp import losses\n",
+ "from deel.lipdp.model import DP_Model\n",
+ "from deel.lipdp.model import DPParameters"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "3a247cd3-48d6-4854-92df-01420d3bea80",
+ "metadata": {},
+ "source": [
+ "The `DP_Accountant` callback keeps track of $(\\epsilon,\\delta)$-DP values epoch after epoch. In practice we may be interested in reaching the maximum val_accuracy under privacy constraint $\\epsilon$: the convenience function `get_max_epochs` exactly does that by performing a dichotomy search over the number of epochs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "950c5c56-4b34-4653-aaf3-7d97acc1f5f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.model import DP_Accountant\n",
+ "from deel.lipdp.sensitivity import get_max_epochs"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "893d3078-5166-428c-9cb1-d29ec1f05d71",
+ "metadata": {},
+ "source": [
+ "The framework requires a control of the maximum norm of inputs. This can be ensured with input clipping for example: `bound_clip_value`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "f395c9fc-b67d-4fd2-be4b-b1c43221ebcb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.pipeline import bound_clip_value\n",
+ "from deel.lipdp.pipeline import load_and_prepare_data"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "e54a79db-24b4-4dae-b684-170fa743bc5d",
+ "metadata": {},
+ "source": [
+ "## Setup DP Lipschitz model\n",
+ "\n",
+ "Here we apply the \"global\" strategy, with a noise multiplier $2.5$. Note that for Cifar-10 the dataset size is $N=50,000$, and it is recommended that $\\delta<\\frac{1}{N}$. So we propose a value of $\\delta=10^{-5}$. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "f79ea3b0-33a6-401c-a3a3-e314939fd269",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import warnings\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "\n",
+ "dp_parameters = DPParameters(\n",
+ " noisify_strategy=\"global\",\n",
+ " noise_multiplier=4.0,\n",
+ " delta=1e-5,\n",
+ ")\n",
+ "\n",
+ "epsilon_max = 10.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ba392eec-4451-49e5-bd45-883af7aa2d40",
+ "metadata": {},
+ "source": [
+ "With many parameters, it can be interesting to use `local` strategy over `global`, since the effective noise growths as $\\mathcal{O}(\\sqrt{(D)})$ in `global` strategy. Since the privacy leakge is more important is `local` strategy, we compensate with high `noise_multiplier`.\n",
+ "\n",
+ "![DP-SGD accountant](../assets/fig_accountant.png \"DP-SGD accountant\")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "6482128c-ac2e-4cdd-9bbd-6d3172c292b1",
+ "metadata": {},
+ "source": [
+ "### Loading the data\n",
+ "\n",
+ "We clip the elementwise input upper-bound to $40.0$. The operates in `HSV` space. The train set is augmented with random left/right flips."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "a8ed0fc4-4655-4bad-a6ac-8697cd5bc7a6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-05-24 17:27:24.335576: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
+ "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
+ "2023-05-24 17:27:24.905888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 47066 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:03:00.0, compute capability: 7.5\n"
+ ]
+ }
+ ],
+ "source": [
+ "def augmentation_fct(image, label):\n",
+ " image = tf.image.random_flip_left_right(image)\n",
+ " return image, label\n",
+ "\n",
+ "input_upper_bound = 30.0\n",
+ "ds_train, ds_test, dataset_metadata = load_and_prepare_data(\n",
+ " \"cifar10\",\n",
+ " colorspace=\"HSV\",\n",
+ " batch_size=10_000,\n",
+ " drop_remainder=True, # accounting assumes fixed batch size\n",
+ " augmentation_fct=augmentation_fct,\n",
+ " bound_fct=bound_clip_value( # other strategies are possible, like normalization.\n",
+ " input_upper_bound\n",
+ " ), # clipping preprocessing allows to control input bound\n",
+ ")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "eb356c04-a836-4f49-93d7-7e0cc4c12b1d",
+ "metadata": {},
+ "source": [
+ "### Build the MLP Mixer model\n",
+ "\n",
+ "We imitate the interface of Keras. We use common layers found in deel-lip, which a wrapper that handles the bound propagation. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "be32d5d7-efc7-4cc6-91bc-1a2b9bedddca",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.layers import DP_AddBias\n",
+ "from deel.lipdp.layers import DP_BoundedInput\n",
+ "from deel.lipdp.layers import DP_ClipGradient\n",
+ "from deel.lipdp.layers import DP_Flatten\n",
+ "from deel.lipdp.layers import DP_GroupSort\n",
+ "from deel.lipdp.layers import DP_Lambda\n",
+ "from deel.lipdp.layers import DP_LayerCentering\n",
+ "from deel.lipdp.layers import DP_Permute\n",
+ "from deel.lipdp.layers import DP_QuickSpectralDense\n",
+ "from deel.lipdp.layers import DP_Reshape\n",
+ "from deel.lipdp.layers import DP_ScaledGlobalL2NormPooling2D\n",
+ "from deel.lipdp.layers import DP_ScaledL2NormPooling2D\n",
+ "from deel.lipdp.layers import DP_QuickSpectralConv2D"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "15b21796-b8e7-41d3-8718-0efdb5d92179",
+ "metadata": {},
+ "source": [
+ "The MLP Mixer uses residual connections. Residuals connections are handled with the utility function `make_residuals` that wraps the layers inside a block that handles bounds propagation.\n",
+ "\n",
+ "![Residuals Connections](../assets/residuals.png \"Residual Connections\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "0590e72d-ce2e-48c1-a8ae-e86ecd32b524",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.layers import make_residuals"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9d75f692-c66d-4318-a915-f16707ed87fa",
+ "metadata": {},
+ "source": [
+ "Now, we proceed with the creation of the environnement."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "30cf44ed-653b-4eaa-8ed9-26e4815db511",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "skip_connections = False # use skip connections, like in original MLP Mixer architecture.\n",
+ "clip_loss_gradient = 2**0.5 # elementwise gradient is clipped to value sqrt(2) - which is the maximum for CCE loss.\n",
+ "add_biases = False # Add biases after linear transformations.\n",
+ "biases_norm_max = 0.05\n",
+ "hidden_size = 64\n",
+ "mlp_seq_dim = 64\n",
+ "mlp_channel_dim = 128\n",
+ "num_mixer_layers = 2 # Two MLP Mixer blocks.\n",
+ "layer_centering = False # Centering operation (like LayerNormalization without the reducing operation). Linear 1-Lipschitz.\n",
+ "patch_size = 4 # Number of pixels in each patch.\n",
+ "\n",
+ "def create_MLP_Mixer(dp_parameters, dataset_metadata, upper_bound):\n",
+ " input_shape = (32, 32, 3)\n",
+ " layers = [DP_BoundedInput(input_shape=input_shape, upper_bound=upper_bound)]\n",
+ "\n",
+ " layers.append(\n",
+ " DP_Lambda(\n",
+ " tf.image.extract_patches,\n",
+ " arguments=dict(\n",
+ " sizes=[1, patch_size, patch_size, 1],\n",
+ " strides=[1, patch_size, patch_size, 1],\n",
+ " rates=[1, 1, 1, 1],\n",
+ " padding=\"VALID\",\n",
+ " ),\n",
+ " )\n",
+ " )\n",
+ "\n",
+ " seq_len = (input_shape[0] // patch_size) * (input_shape[1] // patch_size)\n",
+ "\n",
+ " layers.append(DP_Reshape((seq_len, (patch_size ** 2) * input_shape[-1])))\n",
+ " layers.append(\n",
+ " DP_QuickSpectralDense(\n",
+ " units=hidden_size, use_bias=False, kernel_initializer=\"identity\"\n",
+ " )\n",
+ " )\n",
+ "\n",
+ " for _ in range(num_mixer_layers):\n",
+ " to_add = [\n",
+ " DP_Permute((2, 1)),\n",
+ " DP_QuickSpectralDense(\n",
+ " units=mlp_seq_dim, use_bias=False, kernel_initializer=\"identity\"\n",
+ " ),\n",
+ " ]\n",
+ " if add_biases:\n",
+ " to_add.append(DP_AddBias(biases_norm_max))\n",
+ " to_add.append(DP_GroupSort(2))\n",
+ " if layer_centering:\n",
+ " to_add.append(DP_LayerCentering())\n",
+ " to_add += [\n",
+ " DP_QuickSpectralDense(\n",
+ " units=seq_len, use_bias=False, kernel_initializer=\"identity\"\n",
+ " ),\n",
+ " DP_Permute((2, 1)),\n",
+ " ]\n",
+ "\n",
+ " if skip_connections:\n",
+ " layers += make_residuals(\"1-lip-add\", to_add)\n",
+ " else:\n",
+ " layers += to_add\n",
+ "\n",
+ " to_add = [\n",
+ " DP_QuickSpectralDense(\n",
+ " units=mlp_channel_dim, use_bias=False, kernel_initializer=\"identity\"\n",
+ " ),\n",
+ " ]\n",
+ " if add_biases:\n",
+ " to_add.append(DP_AddBias(biases_norm_max))\n",
+ " to_add.append(DP_GroupSort(2))\n",
+ " if layer_centering:\n",
+ " to_add.append(DP_LayerCentering())\n",
+ " to_add.append(\n",
+ " DP_QuickSpectralDense(\n",
+ " units=hidden_size, use_bias=False, kernel_initializer=\"identity\"\n",
+ " )\n",
+ " )\n",
+ "\n",
+ " if skip_connections:\n",
+ " layers += make_residuals(\"1-lip-add\", to_add)\n",
+ " else:\n",
+ " layers += to_add\n",
+ "\n",
+ " layers.append(DP_Flatten())\n",
+ " layers.append(\n",
+ " DP_QuickSpectralDense(units=10, use_bias=False, kernel_initializer=\"identity\")\n",
+ " )\n",
+ "\n",
+ " layers.append(DP_ClipGradient(clip_loss_gradient))\n",
+ "\n",
+ " model = DP_Model(\n",
+ " layers,\n",
+ " dp_parameters=dp_parameters,\n",
+ " dataset_metadata=dataset_metadata,\n",
+ " name=\"mlp_mixer\",\n",
+ " )\n",
+ "\n",
+ " model.build(input_shape=(None, *input_shape))\n",
+ "\n",
+ " return model"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "09777811",
+ "metadata": {},
+ "source": [
+ "We compile the model with:\n",
+ "* any first order optimizer (e.g Adam). No adaptation is needed.\n",
+ "* a loss with known Lipschitz constant, e.g Categorical Cross-entropy with temperature."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "efd97e75-34f0-49fa-ad2c-1816247f1611",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Model: \"mlp_mixer\"\n",
+ "_________________________________________________________________\n",
+ " Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ " dp__bounded_input (DP_Bound multiple 0 \n",
+ " edInput) \n",
+ " \n",
+ " dp__lambda (DP_Lambda) multiple 0 \n",
+ " \n",
+ " dp__reshape (DP_Reshape) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense (D multiple 3072 \n",
+ " P_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_1 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort (DP_GroupSor multiple 0 \n",
+ " t) \n",
+ " \n",
+ " dp__quick_spectral_dense_2 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute_1 (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_3 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_1 (DP_GroupS multiple 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_4 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute_2 (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_5 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_2 (DP_GroupS multiple 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_6 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute_3 (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_7 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_3 (DP_GroupS multiple 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_8 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__flatten (DP_Flatten) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_9 multiple 40960 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__clip_gradient (DP_ClipG multiple 0 \n",
+ " radient) \n",
+ " \n",
+ "=================================================================\n",
+ "Total params: 93,184\n",
+ "Trainable params: 93,184\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = create_MLP_Mixer(dp_parameters, dataset_metadata, input_upper_bound)\n",
+ "model.compile(\n",
+ " # Compile model using DP loss\n",
+ " loss=losses.DP_TauCategoricalCrossentropy(256.0),\n",
+ " # this method is compatible with any first order optimizer\n",
+ " optimizer=tf.keras.optimizers.Adam(learning_rate=2e-4),\n",
+ " metrics=[\"accuracy\"],\n",
+ ")\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "28ae2da5-ed40-4131-8721-73bbc73fa68d",
+ "metadata": {},
+ "source": [
+ "Observe that the model contains only 246K parmaeters. This is an advantage of MLP Mixer architectures: the number of parameters is small. However the number of FLOPS can be quite high. Without gradient clipping, huge batch sizes can be used, which benefits to privacy/utility ratio. \n",
+ "\n",
+ "In order to control epsilon, we compute the adequate number of epochs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "dd611afd-be30-4bd3-b658-48d1961247aa",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "epoch bounds = (0, 512.0) and epsilon = 14.81894855578722 at epoch 512.0\n",
+ "epoch bounds = (256.0, 512.0) and epsilon = 9.820083418023108 at epoch 256.0\n",
+ "epoch bounds = (256.0, 384.0) and epsilon = 12.31951600358698 at epoch 384.0\n",
+ "epoch bounds = (256.0, 320.0) and epsilon = 11.069799714608529 at epoch 320.0\n",
+ "epoch bounds = (256.0, 288.0) and epsilon = 10.44494156631582 at epoch 288.0\n",
+ "epoch bounds = (256.0, 272.0) and epsilon = 10.132512492169463 at epoch 272.0\n",
+ "epoch bounds = (264.0, 272.0) and epsilon = 9.976297955096285 at epoch 264.0\n",
+ "epoch bounds = (264.0, 268.0) and epsilon = 10.054405223632873 at epoch 268.0\n",
+ "epoch bounds = (264.0, 266.0) and epsilon = 10.015351589364581 at epoch 266.0\n",
+ "epoch bounds = (265.0, 266.0) and epsilon = 9.995824772230431 at epoch 265.0\n"
+ ]
+ }
+ ],
+ "source": [
+ "num_epochs = get_max_epochs(epsilon_max, model)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "53e94244",
+ "metadata": {},
+ "source": [
+ "## Train the model\n",
+ "\n",
+ "The model can be trained, and the DP Accountant will automatically track the privacy loss."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "0ddcb192-547e-400e-87bb-2d4246185c64",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Epoch 1/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1751 - accuracy: 0.1077\n",
+ " (0.5205893807331654, 1e-05)-DP guarantees for epoch 1 \n",
+ "\n",
+ "5/5 [==============================] - 8s 547ms/step - loss: 0.1751 - accuracy: 0.1077 - val_loss: 0.1409 - val_accuracy: 0.1045\n",
+ "Epoch 2/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1243 - accuracy: 0.1061\n",
+ " (0.7169615437758403, 1e-05)-DP guarantees for epoch 2 \n",
+ "\n",
+ "5/5 [==============================] - 3s 451ms/step - loss: 0.1243 - accuracy: 0.1061 - val_loss: 0.1145 - val_accuracy: 0.1055\n",
+ "Epoch 3/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1124 - accuracy: 0.1170\n",
+ " (0.8714581783028138, 1e-05)-DP guarantees for epoch 3 \n",
+ "\n",
+ "5/5 [==============================] - 3s 386ms/step - loss: 0.1124 - accuracy: 0.1170 - val_loss: 0.1095 - val_accuracy: 0.1124\n",
+ "Epoch 4/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1051 - accuracy: 0.1178\n",
+ " (1.0041033056975341, 1e-05)-DP guarantees for epoch 4 \n",
+ "\n",
+ "5/5 [==============================] - 3s 416ms/step - loss: 0.1051 - accuracy: 0.1178 - val_loss: 0.1019 - val_accuracy: 0.1173\n",
+ "Epoch 5/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0994 - accuracy: 0.1219\n",
+ " (1.121902451763874, 1e-05)-DP guarantees for epoch 5 \n",
+ "\n",
+ "5/5 [==============================] - 3s 404ms/step - loss: 0.0994 - accuracy: 0.1219 - val_loss: 0.0973 - val_accuracy: 0.1199\n",
+ "Epoch 6/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0950 - accuracy: 0.1287\n",
+ " (1.2297900098052366, 1e-05)-DP guarantees for epoch 6 \n",
+ "\n",
+ "5/5 [==============================] - 3s 372ms/step - loss: 0.0950 - accuracy: 0.1287 - val_loss: 0.0952 - val_accuracy: 0.1274\n",
+ "Epoch 7/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0927 - accuracy: 0.1332\n",
+ " (1.3301791512711914, 1e-05)-DP guarantees for epoch 7 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0927 - accuracy: 0.1332 - val_loss: 0.0917 - val_accuracy: 0.1319\n",
+ "Epoch 8/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0896 - accuracy: 0.1396\n",
+ " (1.425115891691246, 1e-05)-DP guarantees for epoch 8 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0896 - accuracy: 0.1396 - val_loss: 0.0898 - val_accuracy: 0.1348\n",
+ "Epoch 9/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0878 - accuracy: 0.1423\n",
+ " (1.512644960027369, 1e-05)-DP guarantees for epoch 9 \n",
+ "\n",
+ "5/5 [==============================] - 2s 367ms/step - loss: 0.0878 - accuracy: 0.1423 - val_loss: 0.0876 - val_accuracy: 0.1386\n",
+ "Epoch 10/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0857 - accuracy: 0.1461\n",
+ " (1.599192443478913, 1e-05)-DP guarantees for epoch 10 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0857 - accuracy: 0.1461 - val_loss: 0.0859 - val_accuracy: 0.1469\n",
+ "Epoch 11/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0840 - accuracy: 0.1543\n",
+ " (1.6782666312983627, 1e-05)-DP guarantees for epoch 11 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0840 - accuracy: 0.1543 - val_loss: 0.0844 - val_accuracy: 0.1497\n",
+ "Epoch 12/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0829 - accuracy: 0.1556\n",
+ " (1.7566369758486253, 1e-05)-DP guarantees for epoch 12 \n",
+ "\n",
+ "5/5 [==============================] - 3s 358ms/step - loss: 0.0829 - accuracy: 0.1556 - val_loss: 0.0829 - val_accuracy: 0.1516\n",
+ "Epoch 13/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0816 - accuracy: 0.1578\n",
+ " (1.833150779023074, 1e-05)-DP guarantees for epoch 13 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0816 - accuracy: 0.1578 - val_loss: 0.0819 - val_accuracy: 0.1565\n",
+ "Epoch 14/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0806 - accuracy: 0.1618\n",
+ " (1.903546174784228, 1e-05)-DP guarantees for epoch 14 \n",
+ "\n",
+ "5/5 [==============================] - 3s 370ms/step - loss: 0.0806 - accuracy: 0.1618 - val_loss: 0.0809 - val_accuracy: 0.1592\n",
+ "Epoch 15/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0794 - accuracy: 0.1657\n",
+ " (1.9739415712927695, 1e-05)-DP guarantees for epoch 15 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0794 - accuracy: 0.1657 - val_loss: 0.0799 - val_accuracy: 0.1614\n",
+ "Epoch 16/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0788 - accuracy: 0.1654\n",
+ " (2.044336966003477, 1e-05)-DP guarantees for epoch 16 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0788 - accuracy: 0.1654 - val_loss: 0.0791 - val_accuracy: 0.1642\n",
+ "Epoch 17/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0778 - accuracy: 0.1696\n",
+ " (2.111107170532668, 1e-05)-DP guarantees for epoch 17 \n",
+ "\n",
+ "5/5 [==============================] - 3s 373ms/step - loss: 0.0778 - accuracy: 0.1696 - val_loss: 0.0783 - val_accuracy: 0.1667\n",
+ "Epoch 18/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0773 - accuracy: 0.1720\n",
+ " (2.173720558035018, 1e-05)-DP guarantees for epoch 18 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0773 - accuracy: 0.1720 - val_loss: 0.0775 - val_accuracy: 0.1713\n",
+ "Epoch 19/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0765 - accuracy: 0.1745\n",
+ " (2.236333946199693, 1e-05)-DP guarantees for epoch 19 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0765 - accuracy: 0.1745 - val_loss: 0.0768 - val_accuracy: 0.1718\n",
+ "Epoch 20/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0755 - accuracy: 0.1785\n",
+ " (2.298947335447459, 1e-05)-DP guarantees for epoch 20 \n",
+ "\n",
+ "5/5 [==============================] - 3s 351ms/step - loss: 0.0755 - accuracy: 0.1785 - val_loss: 0.0761 - val_accuracy: 0.1749\n",
+ "Epoch 21/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0751 - accuracy: 0.1809\n",
+ " (2.3615607218535017, 1e-05)-DP guarantees for epoch 21 \n",
+ "\n",
+ "5/5 [==============================] - 2s 370ms/step - loss: 0.0751 - accuracy: 0.1809 - val_loss: 0.0755 - val_accuracy: 0.1779\n",
+ "Epoch 22/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0744 - accuracy: 0.1807\n",
+ " (2.424031214499055, 1e-05)-DP guarantees for epoch 22 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0744 - accuracy: 0.1807 - val_loss: 0.0749 - val_accuracy: 0.1782\n",
+ "Epoch 23/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0737 - accuracy: 0.1829\n",
+ " (2.4794700865598074, 1e-05)-DP guarantees for epoch 23 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0737 - accuracy: 0.1829 - val_loss: 0.0744 - val_accuracy: 0.1796\n",
+ "Epoch 24/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0735 - accuracy: 0.1836\n",
+ " (2.5344857802909178, 1e-05)-DP guarantees for epoch 24 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0735 - accuracy: 0.1836 - val_loss: 0.0738 - val_accuracy: 0.1815\n",
+ "Epoch 25/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0730 - accuracy: 0.1853\n",
+ " (2.589501472054093, 1e-05)-DP guarantees for epoch 25 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0730 - accuracy: 0.1853 - val_loss: 0.0733 - val_accuracy: 0.1836\n",
+ "Epoch 26/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0726 - accuracy: 0.1884\n",
+ " (2.6445171621630954, 1e-05)-DP guarantees for epoch 26 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0726 - accuracy: 0.1884 - val_loss: 0.0729 - val_accuracy: 0.1857\n",
+ "Epoch 27/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0722 - accuracy: 0.1881\n",
+ " (2.699532854747239, 1e-05)-DP guarantees for epoch 27 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0722 - accuracy: 0.1881 - val_loss: 0.0723 - val_accuracy: 0.1882\n",
+ "Epoch 28/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0715 - accuracy: 0.1901\n",
+ " (2.754548546420506, 1e-05)-DP guarantees for epoch 28 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0715 - accuracy: 0.1901 - val_loss: 0.0718 - val_accuracy: 0.1879\n",
+ "Epoch 29/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0711 - accuracy: 0.1928\n",
+ " (2.809564239271509, 1e-05)-DP guarantees for epoch 29 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0711 - accuracy: 0.1928 - val_loss: 0.0715 - val_accuracy: 0.1915\n",
+ "Epoch 30/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0710 - accuracy: 0.1933\n",
+ " (2.8645799306976425, 1e-05)-DP guarantees for epoch 30 \n",
+ "\n",
+ "5/5 [==============================] - 2s 362ms/step - loss: 0.0710 - accuracy: 0.1933 - val_loss: 0.0710 - val_accuracy: 0.1922\n",
+ "Epoch 31/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0701 - accuracy: 0.1993\n",
+ " (2.915773408283026, 1e-05)-DP guarantees for epoch 31 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0701 - accuracy: 0.1993 - val_loss: 0.0706 - val_accuracy: 0.1940\n",
+ "Epoch 32/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0698 - accuracy: 0.1996\n",
+ " (2.9633676512735834, 1e-05)-DP guarantees for epoch 32 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0698 - accuracy: 0.1996 - val_loss: 0.0702 - val_accuracy: 0.1964\n",
+ "Epoch 33/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0695 - accuracy: 0.2004\n",
+ " (3.010961895901816, 1e-05)-DP guarantees for epoch 33 \n",
+ "\n",
+ "5/5 [==============================] - 3s 375ms/step - loss: 0.0695 - accuracy: 0.2004 - val_loss: 0.0699 - val_accuracy: 0.1984\n",
+ "Epoch 34/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0692 - accuracy: 0.1995\n",
+ " (3.0585561401091397, 1e-05)-DP guarantees for epoch 34 \n",
+ "\n",
+ "5/5 [==============================] - 3s 352ms/step - loss: 0.0692 - accuracy: 0.1995 - val_loss: 0.0696 - val_accuracy: 0.1975\n",
+ "Epoch 35/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0685 - accuracy: 0.2045\n",
+ " (3.1061503817189315, 1e-05)-DP guarantees for epoch 35 \n",
+ "\n",
+ "5/5 [==============================] - 3s 349ms/step - loss: 0.0685 - accuracy: 0.2045 - val_loss: 0.0692 - val_accuracy: 0.2009\n",
+ "Epoch 36/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0686 - accuracy: 0.2045\n",
+ " (3.1537446235861095, 1e-05)-DP guarantees for epoch 36 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0686 - accuracy: 0.2045 - val_loss: 0.0689 - val_accuracy: 0.2032\n",
+ "Epoch 37/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0684 - accuracy: 0.2033\n",
+ " (3.2013388677062005, 1e-05)-DP guarantees for epoch 37 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0684 - accuracy: 0.2033 - val_loss: 0.0686 - val_accuracy: 0.2033\n",
+ "Epoch 38/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0684 - accuracy: 0.2024\n",
+ " (3.2489331117939875, 1e-05)-DP guarantees for epoch 38 \n",
+ "\n",
+ "5/5 [==============================] - 3s 352ms/step - loss: 0.0684 - accuracy: 0.2024 - val_loss: 0.0683 - val_accuracy: 0.2046\n",
+ "Epoch 39/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0675 - accuracy: 0.2064\n",
+ " (3.296527354122463, 1e-05)-DP guarantees for epoch 39 \n",
+ "\n",
+ "5/5 [==============================] - 3s 390ms/step - loss: 0.0675 - accuracy: 0.2064 - val_loss: 0.0681 - val_accuracy: 0.2055\n",
+ "Epoch 40/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0678 - accuracy: 0.2071\n",
+ " (3.3441215974412257, 1e-05)-DP guarantees for epoch 40 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0678 - accuracy: 0.2071 - val_loss: 0.0679 - val_accuracy: 0.2061\n",
+ "Epoch 41/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0670 - accuracy: 0.2076\n",
+ " (3.391715841019588, 1e-05)-DP guarantees for epoch 41 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0670 - accuracy: 0.2076 - val_loss: 0.0676 - val_accuracy: 0.2047\n",
+ "Epoch 42/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0670 - accuracy: 0.2074\n",
+ " (3.4393100820764655, 1e-05)-DP guarantees for epoch 42 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0670 - accuracy: 0.2074 - val_loss: 0.0673 - val_accuracy: 0.2077\n",
+ "Epoch 43/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0668 - accuracy: 0.2091\n",
+ " (3.4869043257012042, 1e-05)-DP guarantees for epoch 43 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0668 - accuracy: 0.2091 - val_loss: 0.0671 - val_accuracy: 0.2098\n",
+ "Epoch 44/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0664 - accuracy: 0.2133\n",
+ " (3.5344943006583662, 1e-05)-DP guarantees for epoch 44 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0664 - accuracy: 0.2133 - val_loss: 0.0668 - val_accuracy: 0.2111\n",
+ "Epoch 45/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0662 - accuracy: 0.2116\n",
+ " (3.577278802435221, 1e-05)-DP guarantees for epoch 45 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0662 - accuracy: 0.2116 - val_loss: 0.0666 - val_accuracy: 0.2110\n",
+ "Epoch 46/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0658 - accuracy: 0.2144\n",
+ " (3.6176202954309518, 1e-05)-DP guarantees for epoch 46 \n",
+ "\n",
+ "5/5 [==============================] - 3s 363ms/step - loss: 0.0658 - accuracy: 0.2144 - val_loss: 0.0663 - val_accuracy: 0.2136\n",
+ "Epoch 47/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0660 - accuracy: 0.2136\n",
+ " (3.6579617884266824, 1e-05)-DP guarantees for epoch 47 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0660 - accuracy: 0.2136 - val_loss: 0.0662 - val_accuracy: 0.2103\n",
+ "Epoch 48/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0658 - accuracy: 0.2124\n",
+ " (3.698303280878773, 1e-05)-DP guarantees for epoch 48 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0658 - accuracy: 0.2124 - val_loss: 0.0660 - val_accuracy: 0.2126\n",
+ "Epoch 49/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0651 - accuracy: 0.2170\n",
+ " (3.7386447748463074, 1e-05)-DP guarantees for epoch 49 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0651 - accuracy: 0.2170 - val_loss: 0.0658 - val_accuracy: 0.2141\n",
+ "Epoch 50/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0650 - accuracy: 0.2147\n",
+ " (3.778986264959221, 1e-05)-DP guarantees for epoch 50 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0650 - accuracy: 0.2147 - val_loss: 0.0657 - val_accuracy: 0.2139\n",
+ "Epoch 51/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0649 - accuracy: 0.2157\n",
+ " (3.819327759198358, 1e-05)-DP guarantees for epoch 51 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0649 - accuracy: 0.2157 - val_loss: 0.0654 - val_accuracy: 0.2154\n",
+ "Epoch 52/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0646 - accuracy: 0.2177\n",
+ " (3.859669252353283, 1e-05)-DP guarantees for epoch 52 \n",
+ "\n",
+ "5/5 [==============================] - 3s 374ms/step - loss: 0.0646 - accuracy: 0.2177 - val_loss: 0.0652 - val_accuracy: 0.2159\n",
+ "Epoch 53/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0647 - accuracy: 0.2164\n",
+ " (3.900010744909916, 1e-05)-DP guarantees for epoch 53 \n",
+ "\n",
+ "5/5 [==============================] - 3s 398ms/step - loss: 0.0647 - accuracy: 0.2164 - val_loss: 0.0651 - val_accuracy: 0.2139\n",
+ "Epoch 54/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0642 - accuracy: 0.2180\n",
+ " (3.9403522382284417, 1e-05)-DP guarantees for epoch 54 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0642 - accuracy: 0.2180 - val_loss: 0.0649 - val_accuracy: 0.2165\n",
+ "Epoch 55/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0643 - accuracy: 0.2178\n",
+ " (3.9806937272852823, 1e-05)-DP guarantees for epoch 55 \n",
+ "\n",
+ "5/5 [==============================] - 3s 385ms/step - loss: 0.0643 - accuracy: 0.2178 - val_loss: 0.0648 - val_accuracy: 0.2190\n",
+ "Epoch 56/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0642 - accuracy: 0.2194\n",
+ " (4.021035219696142, 1e-05)-DP guarantees for epoch 56 \n",
+ "\n",
+ "5/5 [==============================] - 3s 358ms/step - loss: 0.0642 - accuracy: 0.2194 - val_loss: 0.0646 - val_accuracy: 0.2190\n",
+ "Epoch 57/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0641 - accuracy: 0.2193\n",
+ " (4.061376713362479, 1e-05)-DP guarantees for epoch 57 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0641 - accuracy: 0.2193 - val_loss: 0.0644 - val_accuracy: 0.2188\n",
+ "Epoch 58/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0637 - accuracy: 0.2209\n",
+ " (4.101718205195644, 1e-05)-DP guarantees for epoch 58 \n",
+ "\n",
+ "5/5 [==============================] - 3s 389ms/step - loss: 0.0637 - accuracy: 0.2209 - val_loss: 0.0643 - val_accuracy: 0.2203\n",
+ "Epoch 59/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0636 - accuracy: 0.2207\n",
+ " (4.142059698567775, 1e-05)-DP guarantees for epoch 59 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0636 - accuracy: 0.2207 - val_loss: 0.0641 - val_accuracy: 0.2217\n",
+ "Epoch 60/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0631 - accuracy: 0.2238\n",
+ " (4.182401188996273, 1e-05)-DP guarantees for epoch 60 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0631 - accuracy: 0.2238 - val_loss: 0.0639 - val_accuracy: 0.2218\n",
+ "Epoch 61/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0635 - accuracy: 0.2223\n",
+ " (4.222742681534986, 1e-05)-DP guarantees for epoch 61 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0635 - accuracy: 0.2223 - val_loss: 0.0638 - val_accuracy: 0.2214\n",
+ "Epoch 62/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0628 - accuracy: 0.2212\n",
+ " (4.263084178169554, 1e-05)-DP guarantees for epoch 62 \n",
+ "\n",
+ "5/5 [==============================] - 3s 358ms/step - loss: 0.0628 - accuracy: 0.2212 - val_loss: 0.0637 - val_accuracy: 0.2214\n",
+ "Epoch 63/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0629 - accuracy: 0.2236\n",
+ " (4.303425669322495, 1e-05)-DP guarantees for epoch 63 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0629 - accuracy: 0.2236 - val_loss: 0.0635 - val_accuracy: 0.2238\n",
+ "Epoch 64/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0628 - accuracy: 0.2244\n",
+ " (4.343767159305043, 1e-05)-DP guarantees for epoch 64 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0628 - accuracy: 0.2244 - val_loss: 0.0633 - val_accuracy: 0.2229\n",
+ "Epoch 65/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0627 - accuracy: 0.2242\n",
+ " (4.384108652677016, 1e-05)-DP guarantees for epoch 65 \n",
+ "\n",
+ "5/5 [==============================] - 3s 375ms/step - loss: 0.0627 - accuracy: 0.2242 - val_loss: 0.0632 - val_accuracy: 0.2232\n",
+ "Epoch 66/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0625 - accuracy: 0.2260\n",
+ " (4.42445014497077, 1e-05)-DP guarantees for epoch 66 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0625 - accuracy: 0.2260 - val_loss: 0.0630 - val_accuracy: 0.2248\n",
+ "Epoch 67/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0625 - accuracy: 0.2271\n",
+ " (4.4647916365799585, 1e-05)-DP guarantees for epoch 67 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0625 - accuracy: 0.2271 - val_loss: 0.0628 - val_accuracy: 0.2265\n",
+ "Epoch 68/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0622 - accuracy: 0.2292\n",
+ " (4.505133128586104, 1e-05)-DP guarantees for epoch 68 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0622 - accuracy: 0.2292 - val_loss: 0.0626 - val_accuracy: 0.2242\n",
+ "Epoch 69/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0623 - accuracy: 0.2276\n",
+ " (4.544958472325187, 1e-05)-DP guarantees for epoch 69 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0623 - accuracy: 0.2276 - val_loss: 0.0626 - val_accuracy: 0.2254\n",
+ "Epoch 70/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0619 - accuracy: 0.2288\n",
+ " (4.580253889044595, 1e-05)-DP guarantees for epoch 70 \n",
+ "\n",
+ "5/5 [==============================] - 2s 362ms/step - loss: 0.0619 - accuracy: 0.2288 - val_loss: 0.0624 - val_accuracy: 0.2272\n",
+ "Epoch 71/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0619 - accuracy: 0.2288\n",
+ " (4.613504255128257, 1e-05)-DP guarantees for epoch 71 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0619 - accuracy: 0.2288 - val_loss: 0.0623 - val_accuracy: 0.2258\n",
+ "Epoch 72/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0617 - accuracy: 0.2283\n",
+ " (4.646754619793705, 1e-05)-DP guarantees for epoch 72 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0617 - accuracy: 0.2283 - val_loss: 0.0622 - val_accuracy: 0.2262\n",
+ "Epoch 73/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0615 - accuracy: 0.2309\n",
+ " (4.680004986868141, 1e-05)-DP guarantees for epoch 73 \n",
+ "\n",
+ "5/5 [==============================] - 3s 363ms/step - loss: 0.0615 - accuracy: 0.2309 - val_loss: 0.0621 - val_accuracy: 0.2292\n",
+ "Epoch 74/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0614 - accuracy: 0.2298\n",
+ " (4.713255352027643, 1e-05)-DP guarantees for epoch 74 \n",
+ "\n",
+ "5/5 [==============================] - 3s 392ms/step - loss: 0.0614 - accuracy: 0.2298 - val_loss: 0.0619 - val_accuracy: 0.2273\n",
+ "Epoch 75/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0616 - accuracy: 0.2288\n",
+ " (4.746505714565027, 1e-05)-DP guarantees for epoch 75 \n",
+ "\n",
+ "5/5 [==============================] - 2s 346ms/step - loss: 0.0616 - accuracy: 0.2288 - val_loss: 0.0618 - val_accuracy: 0.2283\n",
+ "Epoch 76/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0613 - accuracy: 0.2314\n",
+ " (4.779756080992392, 1e-05)-DP guarantees for epoch 76 \n",
+ "\n",
+ "5/5 [==============================] - 3s 375ms/step - loss: 0.0613 - accuracy: 0.2314 - val_loss: 0.0617 - val_accuracy: 0.2285\n",
+ "Epoch 77/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0611 - accuracy: 0.2321\n",
+ " (4.813006446042454, 1e-05)-DP guarantees for epoch 77 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0611 - accuracy: 0.2321 - val_loss: 0.0615 - val_accuracy: 0.2279\n",
+ "Epoch 78/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0609 - accuracy: 0.2321\n",
+ " (4.84625681135709, 1e-05)-DP guarantees for epoch 78 \n",
+ "\n",
+ "5/5 [==============================] - 2s 366ms/step - loss: 0.0609 - accuracy: 0.2321 - val_loss: 0.0614 - val_accuracy: 0.2309\n",
+ "Epoch 79/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0608 - accuracy: 0.2326\n",
+ " (4.879507178851574, 1e-05)-DP guarantees for epoch 79 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0608 - accuracy: 0.2326 - val_loss: 0.0613 - val_accuracy: 0.2316\n",
+ "Epoch 80/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0608 - accuracy: 0.2311\n",
+ " (4.912757545677179, 1e-05)-DP guarantees for epoch 80 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0608 - accuracy: 0.2311 - val_loss: 0.0612 - val_accuracy: 0.2311\n",
+ "Epoch 81/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0607 - accuracy: 0.2333\n",
+ " (4.9460079085624, 1e-05)-DP guarantees for epoch 81 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0607 - accuracy: 0.2333 - val_loss: 0.0611 - val_accuracy: 0.2317\n",
+ "Epoch 82/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0607 - accuracy: 0.2341\n",
+ " (4.979258270989774, 1e-05)-DP guarantees for epoch 82 \n",
+ "\n",
+ "5/5 [==============================] - 2s 339ms/step - loss: 0.0607 - accuracy: 0.2341 - val_loss: 0.0610 - val_accuracy: 0.2338\n",
+ "Epoch 83/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0604 - accuracy: 0.2339\n",
+ " (5.012508634818511, 1e-05)-DP guarantees for epoch 83 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0604 - accuracy: 0.2339 - val_loss: 0.0609 - val_accuracy: 0.2318\n",
+ "Epoch 84/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0605 - accuracy: 0.2348\n",
+ " (5.045759003430268, 1e-05)-DP guarantees for epoch 84 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0605 - accuracy: 0.2348 - val_loss: 0.0608 - val_accuracy: 0.2312\n",
+ "Epoch 85/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0603 - accuracy: 0.2332\n",
+ " (5.0790093680054635, 1e-05)-DP guarantees for epoch 85 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0603 - accuracy: 0.2332 - val_loss: 0.0607 - val_accuracy: 0.2326\n",
+ "Epoch 86/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0600 - accuracy: 0.2355\n",
+ " (5.112259736439092, 1e-05)-DP guarantees for epoch 86 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0600 - accuracy: 0.2355 - val_loss: 0.0606 - val_accuracy: 0.2333\n",
+ "Epoch 87/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0600 - accuracy: 0.2357\n",
+ " (5.14551009793596, 1e-05)-DP guarantees for epoch 87 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0600 - accuracy: 0.2357 - val_loss: 0.0604 - val_accuracy: 0.2335\n",
+ "Epoch 88/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0598 - accuracy: 0.2397\n",
+ " (5.178760460033292, 1e-05)-DP guarantees for epoch 88 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0598 - accuracy: 0.2397 - val_loss: 0.0603 - val_accuracy: 0.2327\n",
+ "Epoch 89/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0596 - accuracy: 0.2377\n",
+ " (5.212010824793953, 1e-05)-DP guarantees for epoch 89 \n",
+ "\n",
+ "5/5 [==============================] - 2s 345ms/step - loss: 0.0596 - accuracy: 0.2377 - val_loss: 0.0602 - val_accuracy: 0.2333\n",
+ "Epoch 90/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0597 - accuracy: 0.2372\n",
+ " (5.24526119058743, 1e-05)-DP guarantees for epoch 90 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0597 - accuracy: 0.2372 - val_loss: 0.0601 - val_accuracy: 0.2336\n",
+ "Epoch 91/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0595 - accuracy: 0.2367\n",
+ " (5.278511560314511, 1e-05)-DP guarantees for epoch 91 \n",
+ "\n",
+ "5/5 [==============================] - 2s 361ms/step - loss: 0.0595 - accuracy: 0.2367 - val_loss: 0.0600 - val_accuracy: 0.2331\n",
+ "Epoch 92/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0598 - accuracy: 0.2373\n",
+ " (5.311761920262455, 1e-05)-DP guarantees for epoch 92 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0598 - accuracy: 0.2373 - val_loss: 0.0599 - val_accuracy: 0.2358\n",
+ "Epoch 93/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0594 - accuracy: 0.2368\n",
+ " (5.3450122912656255, 1e-05)-DP guarantees for epoch 93 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0594 - accuracy: 0.2368 - val_loss: 0.0598 - val_accuracy: 0.2346\n",
+ "Epoch 94/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0592 - accuracy: 0.2380\n",
+ " (5.37826264973137, 1e-05)-DP guarantees for epoch 94 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0592 - accuracy: 0.2380 - val_loss: 0.0597 - val_accuracy: 0.2347\n",
+ "Epoch 95/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0593 - accuracy: 0.2357\n",
+ " (5.4115130208687106, 1e-05)-DP guarantees for epoch 95 \n",
+ "\n",
+ "5/5 [==============================] - 2s 360ms/step - loss: 0.0593 - accuracy: 0.2357 - val_loss: 0.0596 - val_accuracy: 0.2348\n",
+ "Epoch 96/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0594 - accuracy: 0.2376\n",
+ " (5.444763387799843, 1e-05)-DP guarantees for epoch 96 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0594 - accuracy: 0.2376 - val_loss: 0.0595 - val_accuracy: 0.2362\n",
+ "Epoch 97/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0589 - accuracy: 0.2411\n",
+ " (5.47801375480832, 1e-05)-DP guarantees for epoch 97 \n",
+ "\n",
+ "5/5 [==============================] - 2s 363ms/step - loss: 0.0589 - accuracy: 0.2411 - val_loss: 0.0594 - val_accuracy: 0.2375\n",
+ "Epoch 98/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0590 - accuracy: 0.2404\n",
+ " (5.511264111964721, 1e-05)-DP guarantees for epoch 98 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0590 - accuracy: 0.2404 - val_loss: 0.0593 - val_accuracy: 0.2377\n",
+ "Epoch 99/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0586 - accuracy: 0.2406\n",
+ " (5.544514479570887, 1e-05)-DP guarantees for epoch 99 \n",
+ "\n",
+ "5/5 [==============================] - 2s 347ms/step - loss: 0.0586 - accuracy: 0.2406 - val_loss: 0.0593 - val_accuracy: 0.2389\n",
+ "Epoch 100/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0587 - accuracy: 0.2436\n",
+ " (5.5777648468507035, 1e-05)-DP guarantees for epoch 100 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0587 - accuracy: 0.2436 - val_loss: 0.0592 - val_accuracy: 0.2383\n",
+ "Epoch 101/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0586 - accuracy: 0.2405\n",
+ " (5.611015209476669, 1e-05)-DP guarantees for epoch 101 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0586 - accuracy: 0.2405 - val_loss: 0.0590 - val_accuracy: 0.2382\n",
+ "Epoch 102/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0586 - accuracy: 0.2409\n",
+ " (5.644265572603777, 1e-05)-DP guarantees for epoch 102 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0586 - accuracy: 0.2409 - val_loss: 0.0589 - val_accuracy: 0.2376\n",
+ "Epoch 103/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0584 - accuracy: 0.2425\n",
+ " (5.67751593629532, 1e-05)-DP guarantees for epoch 103 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0584 - accuracy: 0.2425 - val_loss: 0.0588 - val_accuracy: 0.2397\n",
+ "Epoch 104/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0583 - accuracy: 0.2422\n",
+ " (5.710766303023046, 1e-05)-DP guarantees for epoch 104 \n",
+ "\n",
+ "5/5 [==============================] - 3s 370ms/step - loss: 0.0583 - accuracy: 0.2422 - val_loss: 0.0587 - val_accuracy: 0.2384\n",
+ "Epoch 105/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0582 - accuracy: 0.2425\n",
+ " (5.7440166690784755, 1e-05)-DP guarantees for epoch 105 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0582 - accuracy: 0.2425 - val_loss: 0.0586 - val_accuracy: 0.2383\n",
+ "Epoch 106/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0583 - accuracy: 0.2411\n",
+ " (5.777267031618594, 1e-05)-DP guarantees for epoch 106 \n",
+ "\n",
+ "5/5 [==============================] - 2s 345ms/step - loss: 0.0583 - accuracy: 0.2411 - val_loss: 0.0586 - val_accuracy: 0.2387\n",
+ "Epoch 107/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0578 - accuracy: 0.2438\n",
+ " (5.8105173958576675, 1e-05)-DP guarantees for epoch 107 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0578 - accuracy: 0.2438 - val_loss: 0.0585 - val_accuracy: 0.2409\n",
+ "Epoch 108/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0582 - accuracy: 0.2442\n",
+ " (5.843767765269359, 1e-05)-DP guarantees for epoch 108 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0582 - accuracy: 0.2442 - val_loss: 0.0584 - val_accuracy: 0.2440\n",
+ "Epoch 109/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0578 - accuracy: 0.2456\n",
+ " (5.877018127929281, 1e-05)-DP guarantees for epoch 109 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0578 - accuracy: 0.2456 - val_loss: 0.0584 - val_accuracy: 0.2419\n",
+ "Epoch 110/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0580 - accuracy: 0.2440\n",
+ " (5.910268490844311, 1e-05)-DP guarantees for epoch 110 \n",
+ "\n",
+ "5/5 [==============================] - 2s 362ms/step - loss: 0.0580 - accuracy: 0.2440 - val_loss: 0.0583 - val_accuracy: 0.2429\n",
+ "Epoch 111/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0578 - accuracy: 0.2473\n",
+ " (5.943518855328065, 1e-05)-DP guarantees for epoch 111 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0578 - accuracy: 0.2473 - val_loss: 0.0583 - val_accuracy: 0.2448\n",
+ "Epoch 112/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0577 - accuracy: 0.2469\n",
+ " (5.9767692222925275, 1e-05)-DP guarantees for epoch 112 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0577 - accuracy: 0.2469 - val_loss: 0.0582 - val_accuracy: 0.2447\n",
+ "Epoch 113/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0579 - accuracy: 0.2479\n",
+ " (6.0100195891034165, 1e-05)-DP guarantees for epoch 113 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0579 - accuracy: 0.2479 - val_loss: 0.0581 - val_accuracy: 0.2453\n",
+ "Epoch 114/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0576 - accuracy: 0.2468\n",
+ " (6.043269950764723, 1e-05)-DP guarantees for epoch 114 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0576 - accuracy: 0.2468 - val_loss: 0.0580 - val_accuracy: 0.2432\n",
+ "Epoch 115/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0574 - accuracy: 0.2472\n",
+ " (6.076520315246205, 1e-05)-DP guarantees for epoch 115 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0574 - accuracy: 0.2472 - val_loss: 0.0579 - val_accuracy: 0.2441\n",
+ "Epoch 116/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0573 - accuracy: 0.2476\n",
+ " (6.109770681686705, 1e-05)-DP guarantees for epoch 116 \n",
+ "\n",
+ "5/5 [==============================] - 2s 363ms/step - loss: 0.0573 - accuracy: 0.2476 - val_loss: 0.0579 - val_accuracy: 0.2440\n",
+ "Epoch 117/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0575 - accuracy: 0.2470\n",
+ " (6.143021045607053, 1e-05)-DP guarantees for epoch 117 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0575 - accuracy: 0.2470 - val_loss: 0.0578 - val_accuracy: 0.2479\n",
+ "Epoch 118/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0572 - accuracy: 0.2481\n",
+ " (6.1762714106501475, 1e-05)-DP guarantees for epoch 118 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0572 - accuracy: 0.2481 - val_loss: 0.0576 - val_accuracy: 0.2450\n",
+ "Epoch 119/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0572 - accuracy: 0.2500\n",
+ " (6.209521499901805, 1e-05)-DP guarantees for epoch 119 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0572 - accuracy: 0.2500 - val_loss: 0.0576 - val_accuracy: 0.2446\n",
+ "Epoch 120/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0569 - accuracy: 0.2497\n",
+ " (6.241605627485653, 1e-05)-DP guarantees for epoch 120 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0569 - accuracy: 0.2497 - val_loss: 0.0575 - val_accuracy: 0.2451\n",
+ "Epoch 121/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0569 - accuracy: 0.2510\n",
+ " (6.271221812058615, 1e-05)-DP guarantees for epoch 121 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0569 - accuracy: 0.2510 - val_loss: 0.0574 - val_accuracy: 0.2445\n",
+ "Epoch 122/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0571 - accuracy: 0.2481\n",
+ " (6.298196491974402, 1e-05)-DP guarantees for epoch 122 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0571 - accuracy: 0.2481 - val_loss: 0.0574 - val_accuracy: 0.2447\n",
+ "Epoch 123/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0568 - accuracy: 0.2517\n",
+ " (6.324510712314491, 1e-05)-DP guarantees for epoch 123 \n",
+ "\n",
+ "5/5 [==============================] - 2s 345ms/step - loss: 0.0568 - accuracy: 0.2517 - val_loss: 0.0573 - val_accuracy: 0.2481\n",
+ "Epoch 124/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0570 - accuracy: 0.2505\n",
+ " (6.350824932887864, 1e-05)-DP guarantees for epoch 124 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0570 - accuracy: 0.2505 - val_loss: 0.0573 - val_accuracy: 0.2449\n",
+ "Epoch 125/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0567 - accuracy: 0.2489\n",
+ " (6.377139153079873, 1e-05)-DP guarantees for epoch 125 \n",
+ "\n",
+ "5/5 [==============================] - 2s 368ms/step - loss: 0.0567 - accuracy: 0.2489 - val_loss: 0.0572 - val_accuracy: 0.2450\n",
+ "Epoch 126/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0570 - accuracy: 0.2488\n",
+ " (6.403453374888347, 1e-05)-DP guarantees for epoch 126 \n",
+ "\n",
+ "5/5 [==============================] - 3s 349ms/step - loss: 0.0570 - accuracy: 0.2488 - val_loss: 0.0572 - val_accuracy: 0.2485\n",
+ "Epoch 127/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0566 - accuracy: 0.2539\n",
+ " (6.429767596763488, 1e-05)-DP guarantees for epoch 127 \n",
+ "\n",
+ "5/5 [==============================] - 3s 391ms/step - loss: 0.0566 - accuracy: 0.2539 - val_loss: 0.0571 - val_accuracy: 0.2452\n",
+ "Epoch 128/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0565 - accuracy: 0.2505\n",
+ " (6.4560818158974875, 1e-05)-DP guarantees for epoch 128 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0565 - accuracy: 0.2505 - val_loss: 0.0570 - val_accuracy: 0.2466\n",
+ "Epoch 129/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0566 - accuracy: 0.2522\n",
+ " (6.482396036898421, 1e-05)-DP guarantees for epoch 129 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0566 - accuracy: 0.2522 - val_loss: 0.0570 - val_accuracy: 0.2461\n",
+ "Epoch 130/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0561 - accuracy: 0.2521\n",
+ " (6.5087102545452, 1e-05)-DP guarantees for epoch 130 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0561 - accuracy: 0.2521 - val_loss: 0.0569 - val_accuracy: 0.2468\n",
+ "Epoch 131/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0562 - accuracy: 0.2534\n",
+ " (6.53502447810436, 1e-05)-DP guarantees for epoch 131 \n",
+ "\n",
+ "5/5 [==============================] - 2s 374ms/step - loss: 0.0562 - accuracy: 0.2534 - val_loss: 0.0569 - val_accuracy: 0.2470\n",
+ "Epoch 132/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0563 - accuracy: 0.2530\n",
+ " (6.5613386977335715, 1e-05)-DP guarantees for epoch 132 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0563 - accuracy: 0.2530 - val_loss: 0.0568 - val_accuracy: 0.2501\n",
+ "Epoch 133/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0561 - accuracy: 0.2564\n",
+ " (6.587652915827986, 1e-05)-DP guarantees for epoch 133 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0561 - accuracy: 0.2564 - val_loss: 0.0569 - val_accuracy: 0.2470\n",
+ "Epoch 134/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0561 - accuracy: 0.2555\n",
+ " (6.613967135260202, 1e-05)-DP guarantees for epoch 134 \n",
+ "\n",
+ "5/5 [==============================] - 3s 402ms/step - loss: 0.0561 - accuracy: 0.2555 - val_loss: 0.0568 - val_accuracy: 0.2492\n",
+ "Epoch 135/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0564 - accuracy: 0.2535\n",
+ " (6.6402813578423405, 1e-05)-DP guarantees for epoch 135 \n",
+ "\n",
+ "5/5 [==============================] - 2s 347ms/step - loss: 0.0564 - accuracy: 0.2535 - val_loss: 0.0567 - val_accuracy: 0.2499\n",
+ "Epoch 136/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0559 - accuracy: 0.2552\n",
+ " (6.666595582737012, 1e-05)-DP guarantees for epoch 136 \n",
+ "\n",
+ "5/5 [==============================] - 2s 360ms/step - loss: 0.0559 - accuracy: 0.2552 - val_loss: 0.0567 - val_accuracy: 0.2506\n",
+ "Epoch 137/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2562\n",
+ " (6.692909796982604, 1e-05)-DP guarantees for epoch 137 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0560 - accuracy: 0.2562 - val_loss: 0.0566 - val_accuracy: 0.2484\n",
+ "Epoch 138/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2538\n",
+ " (6.719224016310403, 1e-05)-DP guarantees for epoch 138 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0560 - accuracy: 0.2538 - val_loss: 0.0565 - val_accuracy: 0.2471\n",
+ "Epoch 139/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2526\n",
+ " (6.74553823900151, 1e-05)-DP guarantees for epoch 139 \n",
+ "\n",
+ "5/5 [==============================] - 3s 399ms/step - loss: 0.0560 - accuracy: 0.2526 - val_loss: 0.0565 - val_accuracy: 0.2509\n",
+ "Epoch 140/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2536\n",
+ " (6.771852459824933, 1e-05)-DP guarantees for epoch 140 \n",
+ "\n",
+ "5/5 [==============================] - 3s 493ms/step - loss: 0.0560 - accuracy: 0.2536 - val_loss: 0.0564 - val_accuracy: 0.2493\n",
+ "Epoch 141/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0557 - accuracy: 0.2555\n",
+ " (6.798166680154963, 1e-05)-DP guarantees for epoch 141 \n",
+ "\n",
+ "5/5 [==============================] - 3s 391ms/step - loss: 0.0557 - accuracy: 0.2555 - val_loss: 0.0563 - val_accuracy: 0.2511\n",
+ "Epoch 142/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0559 - accuracy: 0.2541\n",
+ " (6.824480898392123, 1e-05)-DP guarantees for epoch 142 \n",
+ "\n",
+ "5/5 [==============================] - 3s 443ms/step - loss: 0.0559 - accuracy: 0.2541 - val_loss: 0.0563 - val_accuracy: 0.2484\n",
+ "Epoch 143/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2547\n",
+ " (6.850795124433479, 1e-05)-DP guarantees for epoch 143 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0560 - accuracy: 0.2547 - val_loss: 0.0563 - val_accuracy: 0.2487\n",
+ "Epoch 144/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0556 - accuracy: 0.2545\n",
+ " (6.877109344205954, 1e-05)-DP guarantees for epoch 144 \n",
+ "\n",
+ "5/5 [==============================] - 3s 374ms/step - loss: 0.0556 - accuracy: 0.2545 - val_loss: 0.0562 - val_accuracy: 0.2487\n",
+ "Epoch 145/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0555 - accuracy: 0.2569\n",
+ " (6.903423558068683, 1e-05)-DP guarantees for epoch 145 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0555 - accuracy: 0.2569 - val_loss: 0.0562 - val_accuracy: 0.2508\n",
+ "Epoch 146/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0558 - accuracy: 0.2560\n",
+ " (6.929737777126363, 1e-05)-DP guarantees for epoch 146 \n",
+ "\n",
+ "5/5 [==============================] - 3s 387ms/step - loss: 0.0558 - accuracy: 0.2560 - val_loss: 0.0561 - val_accuracy: 0.2504\n",
+ "Epoch 147/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0557 - accuracy: 0.2556\n",
+ " (6.956052008535497, 1e-05)-DP guarantees for epoch 147 \n",
+ "\n",
+ "5/5 [==============================] - 3s 372ms/step - loss: 0.0557 - accuracy: 0.2556 - val_loss: 0.0561 - val_accuracy: 0.2509\n",
+ "Epoch 148/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0557 - accuracy: 0.2538\n",
+ " (6.982366223228706, 1e-05)-DP guarantees for epoch 148 \n",
+ "\n",
+ "5/5 [==============================] - 3s 381ms/step - loss: 0.0557 - accuracy: 0.2538 - val_loss: 0.0561 - val_accuracy: 0.2528\n",
+ "Epoch 149/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0553 - accuracy: 0.2580\n",
+ " (7.0086804403647855, 1e-05)-DP guarantees for epoch 149 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0553 - accuracy: 0.2580 - val_loss: 0.0560 - val_accuracy: 0.2530\n",
+ "Epoch 150/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0549 - accuracy: 0.2595\n",
+ " (7.034994664689931, 1e-05)-DP guarantees for epoch 150 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0549 - accuracy: 0.2595 - val_loss: 0.0560 - val_accuracy: 0.2519\n",
+ "Epoch 151/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0554 - accuracy: 0.2585\n",
+ " (7.061308885525292, 1e-05)-DP guarantees for epoch 151 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0554 - accuracy: 0.2585 - val_loss: 0.0559 - val_accuracy: 0.2531\n",
+ "Epoch 152/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0554 - accuracy: 0.2580\n",
+ " (7.087623106633284, 1e-05)-DP guarantees for epoch 152 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0554 - accuracy: 0.2580 - val_loss: 0.0558 - val_accuracy: 0.2543\n",
+ "Epoch 153/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0553 - accuracy: 0.2585\n",
+ " (7.113937323136563, 1e-05)-DP guarantees for epoch 153 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0553 - accuracy: 0.2585 - val_loss: 0.0558 - val_accuracy: 0.2537\n",
+ "Epoch 154/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0551 - accuracy: 0.2595\n",
+ " (7.140251544398778, 1e-05)-DP guarantees for epoch 154 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0551 - accuracy: 0.2595 - val_loss: 0.0558 - val_accuracy: 0.2551\n",
+ "Epoch 155/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0550 - accuracy: 0.2600\n",
+ " (7.166565767658498, 1e-05)-DP guarantees for epoch 155 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0550 - accuracy: 0.2600 - val_loss: 0.0557 - val_accuracy: 0.2569\n",
+ "Epoch 156/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0553 - accuracy: 0.2561\n",
+ " (7.192879981310637, 1e-05)-DP guarantees for epoch 156 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0553 - accuracy: 0.2561 - val_loss: 0.0556 - val_accuracy: 0.2545\n",
+ "Epoch 157/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0550 - accuracy: 0.2581\n",
+ " (7.2191942080187195, 1e-05)-DP guarantees for epoch 157 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0550 - accuracy: 0.2581 - val_loss: 0.0556 - val_accuracy: 0.2566\n",
+ "Epoch 158/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0550 - accuracy: 0.2601\n",
+ " (7.245508431022666, 1e-05)-DP guarantees for epoch 158 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0550 - accuracy: 0.2601 - val_loss: 0.0556 - val_accuracy: 0.2574\n",
+ "Epoch 159/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0548 - accuracy: 0.2599\n",
+ " (7.27182264840541, 1e-05)-DP guarantees for epoch 159 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0548 - accuracy: 0.2599 - val_loss: 0.0555 - val_accuracy: 0.2567\n",
+ "Epoch 160/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0548 - accuracy: 0.2616\n",
+ " (7.298136867745498, 1e-05)-DP guarantees for epoch 160 \n",
+ "\n",
+ "5/5 [==============================] - 2s 367ms/step - loss: 0.0548 - accuracy: 0.2616 - val_loss: 0.0554 - val_accuracy: 0.2560\n",
+ "Epoch 161/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0551 - accuracy: 0.2595\n",
+ " (7.324451088022072, 1e-05)-DP guarantees for epoch 161 \n",
+ "\n",
+ "5/5 [==============================] - 3s 349ms/step - loss: 0.0551 - accuracy: 0.2595 - val_loss: 0.0554 - val_accuracy: 0.2577\n",
+ "Epoch 162/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0548 - accuracy: 0.2606\n",
+ " (7.350765305854425, 1e-05)-DP guarantees for epoch 162 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0548 - accuracy: 0.2606 - val_loss: 0.0554 - val_accuracy: 0.2580\n",
+ "Epoch 163/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0547 - accuracy: 0.2588\n",
+ " (7.37707952170881, 1e-05)-DP guarantees for epoch 163 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0547 - accuracy: 0.2588 - val_loss: 0.0553 - val_accuracy: 0.2549\n",
+ "Epoch 164/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0546 - accuracy: 0.2585\n",
+ " (7.403393741099066, 1e-05)-DP guarantees for epoch 164 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0546 - accuracy: 0.2585 - val_loss: 0.0553 - val_accuracy: 0.2591\n",
+ "Epoch 165/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0546 - accuracy: 0.2607\n",
+ " (7.429707969366283, 1e-05)-DP guarantees for epoch 165 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0546 - accuracy: 0.2607 - val_loss: 0.0552 - val_accuracy: 0.2574\n",
+ "Epoch 166/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0547 - accuracy: 0.2598\n",
+ " (7.456022189620042, 1e-05)-DP guarantees for epoch 166 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0547 - accuracy: 0.2598 - val_loss: 0.0551 - val_accuracy: 0.2544\n",
+ "Epoch 167/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0544 - accuracy: 0.2589\n",
+ " (7.4823364015791975, 1e-05)-DP guarantees for epoch 167 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0544 - accuracy: 0.2589 - val_loss: 0.0552 - val_accuracy: 0.2570\n",
+ "Epoch 168/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0546 - accuracy: 0.2620\n",
+ " (7.508650622437409, 1e-05)-DP guarantees for epoch 168 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0546 - accuracy: 0.2620 - val_loss: 0.0551 - val_accuracy: 0.2585\n",
+ "Epoch 169/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0544 - accuracy: 0.2609\n",
+ " (7.5349648424170645, 1e-05)-DP guarantees for epoch 169 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0544 - accuracy: 0.2609 - val_loss: 0.0550 - val_accuracy: 0.2591\n",
+ "Epoch 170/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0545 - accuracy: 0.2618\n",
+ " (7.561279065737033, 1e-05)-DP guarantees for epoch 170 \n",
+ "\n",
+ "5/5 [==============================] - 3s 369ms/step - loss: 0.0545 - accuracy: 0.2618 - val_loss: 0.0551 - val_accuracy: 0.2582\n",
+ "Epoch 171/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0542 - accuracy: 0.2642\n",
+ " (7.587593290867159, 1e-05)-DP guarantees for epoch 171 \n",
+ "\n",
+ "5/5 [==============================] - 3s 372ms/step - loss: 0.0542 - accuracy: 0.2642 - val_loss: 0.0551 - val_accuracy: 0.2598\n",
+ "Epoch 172/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0543 - accuracy: 0.2640\n",
+ " (7.613907506714526, 1e-05)-DP guarantees for epoch 172 \n",
+ "\n",
+ "5/5 [==============================] - 3s 369ms/step - loss: 0.0543 - accuracy: 0.2640 - val_loss: 0.0550 - val_accuracy: 0.2604\n",
+ "Epoch 173/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0543 - accuracy: 0.2642\n",
+ " (7.640221723584304, 1e-05)-DP guarantees for epoch 173 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0543 - accuracy: 0.2642 - val_loss: 0.0549 - val_accuracy: 0.2604\n",
+ "Epoch 174/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0542 - accuracy: 0.2635\n",
+ " (7.666535950048996, 1e-05)-DP guarantees for epoch 174 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0542 - accuracy: 0.2635 - val_loss: 0.0549 - val_accuracy: 0.2628\n",
+ "Epoch 175/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0541 - accuracy: 0.2648\n",
+ " (7.692850164248792, 1e-05)-DP guarantees for epoch 175 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0541 - accuracy: 0.2648 - val_loss: 0.0548 - val_accuracy: 0.2625\n",
+ "Epoch 176/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0542 - accuracy: 0.2637\n",
+ " (7.719164393302542, 1e-05)-DP guarantees for epoch 176 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0542 - accuracy: 0.2637 - val_loss: 0.0547 - val_accuracy: 0.2621\n",
+ "Epoch 177/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0540 - accuracy: 0.2661\n",
+ " (7.745478613553454, 1e-05)-DP guarantees for epoch 177 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0540 - accuracy: 0.2661 - val_loss: 0.0546 - val_accuracy: 0.2665\n",
+ "Epoch 178/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0541 - accuracy: 0.2668\n",
+ " (7.771792822684058, 1e-05)-DP guarantees for epoch 178 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0541 - accuracy: 0.2668 - val_loss: 0.0546 - val_accuracy: 0.2659\n",
+ "Epoch 179/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0539 - accuracy: 0.2685\n",
+ " (7.7981070469012, 1e-05)-DP guarantees for epoch 179 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0539 - accuracy: 0.2685 - val_loss: 0.0545 - val_accuracy: 0.2646\n",
+ "Epoch 180/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0538 - accuracy: 0.2682\n",
+ " (7.824421268798268, 1e-05)-DP guarantees for epoch 180 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0538 - accuracy: 0.2682 - val_loss: 0.0545 - val_accuracy: 0.2656\n",
+ "Epoch 181/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0538 - accuracy: 0.2671\n",
+ " (7.850735498247861, 1e-05)-DP guarantees for epoch 181 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0538 - accuracy: 0.2671 - val_loss: 0.0545 - val_accuracy: 0.2639\n",
+ "Epoch 182/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0539 - accuracy: 0.2661\n",
+ " (7.877049711425853, 1e-05)-DP guarantees for epoch 182 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0539 - accuracy: 0.2661 - val_loss: 0.0544 - val_accuracy: 0.2645\n",
+ "Epoch 183/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0537 - accuracy: 0.2635\n",
+ " (7.903363929529842, 1e-05)-DP guarantees for epoch 183 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0537 - accuracy: 0.2635 - val_loss: 0.0544 - val_accuracy: 0.2645\n",
+ "Epoch 184/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0538 - accuracy: 0.2641\n",
+ " (7.929678153587394, 1e-05)-DP guarantees for epoch 184 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0538 - accuracy: 0.2641 - val_loss: 0.0543 - val_accuracy: 0.2641\n",
+ "Epoch 185/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2668\n",
+ " (7.955992381685565, 1e-05)-DP guarantees for epoch 185 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0535 - accuracy: 0.2668 - val_loss: 0.0543 - val_accuracy: 0.2638\n",
+ "Epoch 186/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2641\n",
+ " (7.982306589145621, 1e-05)-DP guarantees for epoch 186 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0535 - accuracy: 0.2641 - val_loss: 0.0543 - val_accuracy: 0.2654\n",
+ "Epoch 187/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0537 - accuracy: 0.2653\n",
+ " (8.008620808026855, 1e-05)-DP guarantees for epoch 187 \n",
+ "\n",
+ "5/5 [==============================] - 3s 412ms/step - loss: 0.0537 - accuracy: 0.2653 - val_loss: 0.0542 - val_accuracy: 0.2651\n",
+ "Epoch 188/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2656\n",
+ " (8.034935029136395, 1e-05)-DP guarantees for epoch 188 \n",
+ "\n",
+ "5/5 [==============================] - 3s 488ms/step - loss: 0.0535 - accuracy: 0.2656 - val_loss: 0.0542 - val_accuracy: 0.2662\n",
+ "Epoch 189/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0536 - accuracy: 0.2653\n",
+ " (8.061249248434443, 1e-05)-DP guarantees for epoch 189 \n",
+ "\n",
+ "5/5 [==============================] - 3s 444ms/step - loss: 0.0536 - accuracy: 0.2653 - val_loss: 0.0541 - val_accuracy: 0.2659\n",
+ "Epoch 190/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2676\n",
+ " (8.087563469816706, 1e-05)-DP guarantees for epoch 190 \n",
+ "\n",
+ "5/5 [==============================] - 3s 405ms/step - loss: 0.0533 - accuracy: 0.2676 - val_loss: 0.0541 - val_accuracy: 0.2663\n",
+ "Epoch 191/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2669\n",
+ " (8.113877688170744, 1e-05)-DP guarantees for epoch 191 \n",
+ "\n",
+ "5/5 [==============================] - 3s 385ms/step - loss: 0.0534 - accuracy: 0.2669 - val_loss: 0.0541 - val_accuracy: 0.2675\n",
+ "Epoch 192/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2648\n",
+ " (8.140191906358039, 1e-05)-DP guarantees for epoch 192 \n",
+ "\n",
+ "5/5 [==============================] - 3s 392ms/step - loss: 0.0535 - accuracy: 0.2648 - val_loss: 0.0540 - val_accuracy: 0.2676\n",
+ "Epoch 193/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2680\n",
+ " (8.166506132866681, 1e-05)-DP guarantees for epoch 193 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0534 - accuracy: 0.2680 - val_loss: 0.0540 - val_accuracy: 0.2676\n",
+ "Epoch 194/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2654\n",
+ " (8.192820350846777, 1e-05)-DP guarantees for epoch 194 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0533 - accuracy: 0.2654 - val_loss: 0.0540 - val_accuracy: 0.2679\n",
+ "Epoch 195/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.2681\n",
+ " (8.219134573417037, 1e-05)-DP guarantees for epoch 195 \n",
+ "\n",
+ "5/5 [==============================] - 3s 381ms/step - loss: 0.0531 - accuracy: 0.2681 - val_loss: 0.0541 - val_accuracy: 0.2654\n",
+ "Epoch 196/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0532 - accuracy: 0.2671\n",
+ " (8.24544879099129, 1e-05)-DP guarantees for epoch 196 \n",
+ "\n",
+ "5/5 [==============================] - 3s 381ms/step - loss: 0.0532 - accuracy: 0.2671 - val_loss: 0.0540 - val_accuracy: 0.2658\n",
+ "Epoch 197/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2666\n",
+ " (8.271763016196239, 1e-05)-DP guarantees for epoch 197 \n",
+ "\n",
+ "5/5 [==============================] - 3s 389ms/step - loss: 0.0535 - accuracy: 0.2666 - val_loss: 0.0540 - val_accuracy: 0.2656\n",
+ "Epoch 198/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2676\n",
+ " (8.298077232897459, 1e-05)-DP guarantees for epoch 198 \n",
+ "\n",
+ "5/5 [==============================] - 3s 415ms/step - loss: 0.0534 - accuracy: 0.2676 - val_loss: 0.0539 - val_accuracy: 0.2656\n",
+ "Epoch 199/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.2672\n",
+ " (8.324391446543665, 1e-05)-DP guarantees for epoch 199 \n",
+ "\n",
+ "5/5 [==============================] - 3s 380ms/step - loss: 0.0531 - accuracy: 0.2672 - val_loss: 0.0538 - val_accuracy: 0.2658\n",
+ "Epoch 200/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2638\n",
+ " (8.350705669706155, 1e-05)-DP guarantees for epoch 200 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0534 - accuracy: 0.2638 - val_loss: 0.0538 - val_accuracy: 0.2659\n",
+ "Epoch 201/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2672\n",
+ " (8.377019893272927, 1e-05)-DP guarantees for epoch 201 \n",
+ "\n",
+ "5/5 [==============================] - 2s 369ms/step - loss: 0.0533 - accuracy: 0.2672 - val_loss: 0.0538 - val_accuracy: 0.2678\n",
+ "Epoch 202/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2685\n",
+ " (8.403334112768452, 1e-05)-DP guarantees for epoch 202 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0533 - accuracy: 0.2685 - val_loss: 0.0537 - val_accuracy: 0.2677\n",
+ "Epoch 203/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0528 - accuracy: 0.2701\n",
+ " (8.429648329088547, 1e-05)-DP guarantees for epoch 203 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0528 - accuracy: 0.2701 - val_loss: 0.0537 - val_accuracy: 0.2669\n",
+ "Epoch 204/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0528 - accuracy: 0.2696\n",
+ " (8.455962556505566, 1e-05)-DP guarantees for epoch 204 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0528 - accuracy: 0.2696 - val_loss: 0.0537 - val_accuracy: 0.2681\n",
+ "Epoch 205/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0532 - accuracy: 0.2691\n",
+ " (8.4822767745793, 1e-05)-DP guarantees for epoch 205 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0532 - accuracy: 0.2691 - val_loss: 0.0536 - val_accuracy: 0.2692\n",
+ "Epoch 206/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.2703\n",
+ " (8.508590990133396, 1e-05)-DP guarantees for epoch 206 \n",
+ "\n",
+ "5/5 [==============================] - 3s 354ms/step - loss: 0.0531 - accuracy: 0.2703 - val_loss: 0.0535 - val_accuracy: 0.2683\n",
+ "Epoch 207/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0529 - accuracy: 0.2705\n",
+ " (8.534905221654196, 1e-05)-DP guarantees for epoch 207 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0529 - accuracy: 0.2705 - val_loss: 0.0535 - val_accuracy: 0.2661\n",
+ "Epoch 208/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0526 - accuracy: 0.2726\n",
+ " (8.56121943210842, 1e-05)-DP guarantees for epoch 208 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0526 - accuracy: 0.2726 - val_loss: 0.0535 - val_accuracy: 0.2671\n",
+ "Epoch 209/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0530 - accuracy: 0.2703\n",
+ " (8.58753364829852, 1e-05)-DP guarantees for epoch 209 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0530 - accuracy: 0.2703 - val_loss: 0.0534 - val_accuracy: 0.2691\n",
+ "Epoch 210/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0527 - accuracy: 0.2701\n",
+ " (8.613847875406321, 1e-05)-DP guarantees for epoch 210 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0527 - accuracy: 0.2701 - val_loss: 0.0534 - val_accuracy: 0.2676\n",
+ "Epoch 211/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2713\n",
+ " (8.640162093797892, 1e-05)-DP guarantees for epoch 211 \n",
+ "\n",
+ "5/5 [==============================] - 3s 350ms/step - loss: 0.0525 - accuracy: 0.2713 - val_loss: 0.0534 - val_accuracy: 0.2689\n",
+ "Epoch 212/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0527 - accuracy: 0.2711\n",
+ " (8.666476313556027, 1e-05)-DP guarantees for epoch 212 \n",
+ "\n",
+ "5/5 [==============================] - 2s 346ms/step - loss: 0.0527 - accuracy: 0.2711 - val_loss: 0.0533 - val_accuracy: 0.2679\n",
+ "Epoch 213/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0524 - accuracy: 0.2722\n",
+ " (8.692790541337777, 1e-05)-DP guarantees for epoch 213 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0524 - accuracy: 0.2722 - val_loss: 0.0532 - val_accuracy: 0.2673\n",
+ "Epoch 214/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0526 - accuracy: 0.2705\n",
+ " (8.719104752659717, 1e-05)-DP guarantees for epoch 214 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0526 - accuracy: 0.2705 - val_loss: 0.0532 - val_accuracy: 0.2675\n",
+ "Epoch 215/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0523 - accuracy: 0.2729\n",
+ " (8.745418971706883, 1e-05)-DP guarantees for epoch 215 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0523 - accuracy: 0.2729 - val_loss: 0.0532 - val_accuracy: 0.2674\n",
+ "Epoch 216/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2733\n",
+ " (8.77173318977154, 1e-05)-DP guarantees for epoch 216 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0525 - accuracy: 0.2733 - val_loss: 0.0532 - val_accuracy: 0.2662\n",
+ "Epoch 217/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2719\n",
+ " (8.798047413801395, 1e-05)-DP guarantees for epoch 217 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0525 - accuracy: 0.2719 - val_loss: 0.0531 - val_accuracy: 0.2676\n",
+ "Epoch 218/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0524 - accuracy: 0.2741\n",
+ " (8.82436163499698, 1e-05)-DP guarantees for epoch 218 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0524 - accuracy: 0.2741 - val_loss: 0.0531 - val_accuracy: 0.2671\n",
+ "Epoch 219/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2702\n",
+ " (8.850675857122916, 1e-05)-DP guarantees for epoch 219 \n",
+ "\n",
+ "5/5 [==============================] - 3s 386ms/step - loss: 0.0525 - accuracy: 0.2702 - val_loss: 0.0531 - val_accuracy: 0.2672\n",
+ "Epoch 220/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0527 - accuracy: 0.2709\n",
+ " (8.876990076626331, 1e-05)-DP guarantees for epoch 220 \n",
+ "\n",
+ "5/5 [==============================] - 3s 376ms/step - loss: 0.0527 - accuracy: 0.2709 - val_loss: 0.0531 - val_accuracy: 0.2668\n",
+ "Epoch 221/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2715\n",
+ " (8.903304291167267, 1e-05)-DP guarantees for epoch 221 \n",
+ "\n",
+ "5/5 [==============================] - 3s 363ms/step - loss: 0.0525 - accuracy: 0.2715 - val_loss: 0.0531 - val_accuracy: 0.2661\n",
+ "Epoch 222/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0524 - accuracy: 0.2721\n",
+ " (8.929618511328595, 1e-05)-DP guarantees for epoch 222 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0524 - accuracy: 0.2721 - val_loss: 0.0530 - val_accuracy: 0.2677\n",
+ "Epoch 223/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2726\n",
+ " (8.955932731489924, 1e-05)-DP guarantees for epoch 223 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0521 - accuracy: 0.2726 - val_loss: 0.0530 - val_accuracy: 0.2686\n",
+ "Epoch 224/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2727\n",
+ " (8.982246951651252, 1e-05)-DP guarantees for epoch 224 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0521 - accuracy: 0.2727 - val_loss: 0.0530 - val_accuracy: 0.2690\n",
+ "Epoch 225/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0522 - accuracy: 0.2706\n",
+ " (9.00856117181258, 1e-05)-DP guarantees for epoch 225 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0522 - accuracy: 0.2706 - val_loss: 0.0529 - val_accuracy: 0.2701\n",
+ "Epoch 226/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2715\n",
+ " (9.034875391973909, 1e-05)-DP guarantees for epoch 226 \n",
+ "\n",
+ "5/5 [==============================] - 3s 396ms/step - loss: 0.0521 - accuracy: 0.2715 - val_loss: 0.0529 - val_accuracy: 0.2690\n",
+ "Epoch 227/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2713\n",
+ " (9.061189612135239, 1e-05)-DP guarantees for epoch 227 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0521 - accuracy: 0.2713 - val_loss: 0.0529 - val_accuracy: 0.2702\n",
+ "Epoch 228/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0520 - accuracy: 0.2729\n",
+ " (9.087503832296568, 1e-05)-DP guarantees for epoch 228 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0520 - accuracy: 0.2729 - val_loss: 0.0529 - val_accuracy: 0.2698\n",
+ "Epoch 229/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2717\n",
+ " (9.113818052457898, 1e-05)-DP guarantees for epoch 229 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0518 - accuracy: 0.2717 - val_loss: 0.0528 - val_accuracy: 0.2704\n",
+ "Epoch 230/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2710\n",
+ " (9.140132272619226, 1e-05)-DP guarantees for epoch 230 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0518 - accuracy: 0.2710 - val_loss: 0.0528 - val_accuracy: 0.2693\n",
+ "Epoch 231/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0519 - accuracy: 0.2725\n",
+ " (9.166446492780555, 1e-05)-DP guarantees for epoch 231 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0519 - accuracy: 0.2725 - val_loss: 0.0528 - val_accuracy: 0.2674\n",
+ "Epoch 232/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2743\n",
+ " (9.192760712941883, 1e-05)-DP guarantees for epoch 232 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0517 - accuracy: 0.2743 - val_loss: 0.0528 - val_accuracy: 0.2685\n",
+ "Epoch 233/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0519 - accuracy: 0.2712\n",
+ " (9.219074933103212, 1e-05)-DP guarantees for epoch 233 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0519 - accuracy: 0.2712 - val_loss: 0.0527 - val_accuracy: 0.2687\n",
+ "Epoch 234/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2731\n",
+ " (9.24538915326454, 1e-05)-DP guarantees for epoch 234 \n",
+ "\n",
+ "5/5 [==============================] - 3s 370ms/step - loss: 0.0517 - accuracy: 0.2731 - val_loss: 0.0527 - val_accuracy: 0.2663\n",
+ "Epoch 235/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2720\n",
+ " (9.27170337342587, 1e-05)-DP guarantees for epoch 235 \n",
+ "\n",
+ "5/5 [==============================] - 3s 350ms/step - loss: 0.0518 - accuracy: 0.2720 - val_loss: 0.0527 - val_accuracy: 0.2663\n",
+ "Epoch 236/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2729\n",
+ " (9.298017593587199, 1e-05)-DP guarantees for epoch 236 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0515 - accuracy: 0.2729 - val_loss: 0.0526 - val_accuracy: 0.2658\n",
+ "Epoch 237/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2726\n",
+ " (9.324331813748529, 1e-05)-DP guarantees for epoch 237 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0517 - accuracy: 0.2726 - val_loss: 0.0526 - val_accuracy: 0.2650\n",
+ "Epoch 238/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2752\n",
+ " (9.350646033909857, 1e-05)-DP guarantees for epoch 238 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0515 - accuracy: 0.2752 - val_loss: 0.0525 - val_accuracy: 0.2655\n",
+ "Epoch 239/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2739\n",
+ " (9.376960254071186, 1e-05)-DP guarantees for epoch 239 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0517 - accuracy: 0.2739 - val_loss: 0.0525 - val_accuracy: 0.2665\n",
+ "Epoch 240/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2736\n",
+ " (9.403274474232514, 1e-05)-DP guarantees for epoch 240 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0515 - accuracy: 0.2736 - val_loss: 0.0525 - val_accuracy: 0.2673\n",
+ "Epoch 241/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2729\n",
+ " (9.429588694393843, 1e-05)-DP guarantees for epoch 241 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0517 - accuracy: 0.2729 - val_loss: 0.0524 - val_accuracy: 0.2674\n",
+ "Epoch 242/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2746\n",
+ " (9.455902914555171, 1e-05)-DP guarantees for epoch 242 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0518 - accuracy: 0.2746 - val_loss: 0.0525 - val_accuracy: 0.2694\n",
+ "Epoch 243/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2756\n",
+ " (9.482217134716501, 1e-05)-DP guarantees for epoch 243 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0515 - accuracy: 0.2756 - val_loss: 0.0524 - val_accuracy: 0.2699\n",
+ "Epoch 244/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0512 - accuracy: 0.2760\n",
+ " (9.50853135487783, 1e-05)-DP guarantees for epoch 244 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0512 - accuracy: 0.2760 - val_loss: 0.0524 - val_accuracy: 0.2712\n",
+ "Epoch 245/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0514 - accuracy: 0.2756\n",
+ " (9.534845575173634, 1e-05)-DP guarantees for epoch 245 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0514 - accuracy: 0.2756 - val_loss: 0.0523 - val_accuracy: 0.2700\n",
+ "Epoch 246/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0512 - accuracy: 0.2758\n",
+ " (9.561159795911662, 1e-05)-DP guarantees for epoch 246 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0512 - accuracy: 0.2758 - val_loss: 0.0523 - val_accuracy: 0.2716\n",
+ "Epoch 247/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0512 - accuracy: 0.2783\n",
+ " (9.587474015660208, 1e-05)-DP guarantees for epoch 247 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0512 - accuracy: 0.2783 - val_loss: 0.0523 - val_accuracy: 0.2719\n",
+ "Epoch 248/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0514 - accuracy: 0.2768\n",
+ " (9.613788235533915, 1e-05)-DP guarantees for epoch 248 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0514 - accuracy: 0.2768 - val_loss: 0.0522 - val_accuracy: 0.2711\n",
+ "Epoch 249/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2780\n",
+ " (9.64006012187098, 1e-05)-DP guarantees for epoch 249 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0511 - accuracy: 0.2780 - val_loss: 0.0522 - val_accuracy: 0.2720\n",
+ "Epoch 250/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2746\n",
+ " (9.665736679745127, 1e-05)-DP guarantees for epoch 250 \n",
+ "\n",
+ "5/5 [==============================] - 3s 352ms/step - loss: 0.0511 - accuracy: 0.2746 - val_loss: 0.0522 - val_accuracy: 0.2731\n",
+ "Epoch 251/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0509 - accuracy: 0.2777\n",
+ " (9.690604545814235, 1e-05)-DP guarantees for epoch 251 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0509 - accuracy: 0.2777 - val_loss: 0.0522 - val_accuracy: 0.2727\n",
+ "Epoch 252/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2793\n",
+ " (9.714604392289775, 1e-05)-DP guarantees for epoch 252 \n",
+ "\n",
+ "5/5 [==============================] - 3s 354ms/step - loss: 0.0511 - accuracy: 0.2793 - val_loss: 0.0522 - val_accuracy: 0.2701\n",
+ "Epoch 253/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2751\n",
+ " (9.737670917278972, 1e-05)-DP guarantees for epoch 253 \n",
+ "\n",
+ "5/5 [==============================] - 3s 354ms/step - loss: 0.0511 - accuracy: 0.2751 - val_loss: 0.0522 - val_accuracy: 0.2719\n",
+ "Epoch 254/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0510 - accuracy: 0.2764\n",
+ " (9.759732027763015, 1e-05)-DP guarantees for epoch 254 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0510 - accuracy: 0.2764 - val_loss: 0.0522 - val_accuracy: 0.2718\n",
+ "Epoch 255/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0510 - accuracy: 0.2765\n",
+ " (9.780707877727917, 1e-05)-DP guarantees for epoch 255 \n",
+ "\n",
+ "5/5 [==============================] - 3s 341ms/step - loss: 0.0510 - accuracy: 0.2765 - val_loss: 0.0522 - val_accuracy: 0.2708\n",
+ "Epoch 256/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2758\n",
+ " (9.80055660088896, 1e-05)-DP guarantees for epoch 256 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0511 - accuracy: 0.2758 - val_loss: 0.0521 - val_accuracy: 0.2726\n",
+ "Epoch 257/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0508 - accuracy: 0.2767\n",
+ " (9.820083418023108, 1e-05)-DP guarantees for epoch 257 \n",
+ "\n",
+ "5/5 [==============================] - 2s 360ms/step - loss: 0.0508 - accuracy: 0.2767 - val_loss: 0.0520 - val_accuracy: 0.2722\n",
+ "Epoch 258/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2740\n",
+ " (9.839610235157256, 1e-05)-DP guarantees for epoch 258 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0511 - accuracy: 0.2740 - val_loss: 0.0520 - val_accuracy: 0.2707\n",
+ "Epoch 259/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0507 - accuracy: 0.2782\n",
+ " (9.859137052291402, 1e-05)-DP guarantees for epoch 259 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0507 - accuracy: 0.2782 - val_loss: 0.0520 - val_accuracy: 0.2731\n",
+ "Epoch 260/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0510 - accuracy: 0.2761\n",
+ " (9.87866386942555, 1e-05)-DP guarantees for epoch 260 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0510 - accuracy: 0.2761 - val_loss: 0.0519 - val_accuracy: 0.2707\n",
+ "Epoch 261/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0509 - accuracy: 0.2751\n",
+ " (9.898190686559698, 1e-05)-DP guarantees for epoch 261 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0509 - accuracy: 0.2751 - val_loss: 0.0519 - val_accuracy: 0.2724\n",
+ "Epoch 262/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2766\n",
+ " (9.917717503693844, 1e-05)-DP guarantees for epoch 262 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0511 - accuracy: 0.2766 - val_loss: 0.0520 - val_accuracy: 0.2729\n",
+ "Epoch 263/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0508 - accuracy: 0.2773\n",
+ " (9.937244320827991, 1e-05)-DP guarantees for epoch 263 \n",
+ "\n",
+ "5/5 [==============================] - 2s 342ms/step - loss: 0.0508 - accuracy: 0.2773 - val_loss: 0.0520 - val_accuracy: 0.2708\n",
+ "Epoch 264/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0507 - accuracy: 0.2777\n",
+ " (9.95677113796214, 1e-05)-DP guarantees for epoch 264 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0507 - accuracy: 0.2777 - val_loss: 0.0519 - val_accuracy: 0.2733\n",
+ "Epoch 265/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0505 - accuracy: 0.2784\n",
+ " (9.976297955096285, 1e-05)-DP guarantees for epoch 265 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0505 - accuracy: 0.2784 - val_loss: 0.0519 - val_accuracy: 0.2738\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = model.fit(\n",
+ " ds_train,\n",
+ " epochs=num_epochs,\n",
+ " validation_data=ds_test,\n",
+ " callbacks=[\n",
+ " # accounting is done thanks to a callback\n",
+ " DP_Accountant(log_fn=\"logging\"), # wandb.log also available.\n",
+ " ],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8e139678-6ec6-4a2e-980b-83059c98c48b",
+ "metadata": {},
+ "source": [
+ "This final val_accuracy is compliant with results reported in other framework. For comparison, in Opacus tutorials, the Resnet 18 reaches 60% val_accuracy at $\\epsilon=47$, but 15% at $\\epsilon=13$. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "db636e0c-0334-45ee-b953-e4cc85bb7d8e",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/notebooks/basics_mnist.ipynb b/docs/notebooks/basics_mnist.ipynb
new file mode 100644
index 0000000..3020545
--- /dev/null
+++ b/docs/notebooks/basics_mnist.ipynb
@@ -0,0 +1,886 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "f7bf07b9-d489-4484-acb9-175cb740dc60",
+ "metadata": {},
+ "source": [
+ "# Mnist tutorial\n",
+ "\n",
+ "This notebook introduces the basics of usage of our library.\n",
+ "\n",
+ "## Imports"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "8a0eebdf-6082-4d00-aa14-b42953217a93",
+ "metadata": {},
+ "source": [
+ "The library is based on tensorflow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "91c2965e-0375-4966-bc55-776204af9d69",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "9356cd9b-6f79-45f1-8f2e-c46a526c4ae7",
+ "metadata": {},
+ "source": [
+ "### lip-dp dependencies\n",
+ "\n",
+ "The need a model `DP_Sequential` that handles the noisification of gradients. It is composed `layers` and trained with a loss found in `loss`. The model is initialized with the convenience function `DPParameters`. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "1e5d58f8-386c-44c7-8c5d-e5b69b5be231",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp import layers\n",
+ "from deel.lipdp import losses\n",
+ "from deel.lipdp.model import DP_Sequential\n",
+ "from deel.lipdp.model import DPParameters"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "3a247cd3-48d6-4854-92df-01420d3bea80",
+ "metadata": {},
+ "source": [
+ "The `DP_Accountant` callback keeps track of $(\\epsilon,\\delta)$-DP values epoch after epoch. In practice we may be interested in reaching the maximum val_accuracy under privacy constraint $\\epsilon$: the convenience function `get_max_epochs` exactly does that by performing a dichotomy search over the number of epochs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "950c5c56-4b34-4653-aaf3-7d97acc1f5f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.model import DP_Accountant\n",
+ "from deel.lipdp.sensitivity import get_max_epochs"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "893d3078-5166-428c-9cb1-d29ec1f05d71",
+ "metadata": {},
+ "source": [
+ "The framework requires a control of the maximum norm of inputs. This can be ensured with input clipping for example: `bound_clip_value`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "f395c9fc-b67d-4fd2-be4b-b1c43221ebcb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.pipeline import bound_clip_value\n",
+ "from deel.lipdp.pipeline import load_and_prepare_data"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "e54a79db-24b4-4dae-b684-170fa743bc5d",
+ "metadata": {},
+ "source": [
+ "## Setup DP Lipschitz model\n",
+ "\n",
+ "Here we apply the \"global\" strategy, with a noise multiplier $2.5$. Note that for Mnist the dataset size is $N=60,000$, and it is recommended that $\\delta<\\frac{1}{N}$. So we propose a value of $\\delta=10^{-5}$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "f79ea3b0-33a6-401c-a3a3-e314939fd269",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dp_parameters = DPParameters(\n",
+ " noisify_strategy=\"global\",\n",
+ " noise_multiplier=2.0,\n",
+ " delta=1e-5,\n",
+ ")\n",
+ "\n",
+ "epsilon_max = 3.0"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "6482128c-ac2e-4cdd-9bbd-6d3172c292b1",
+ "metadata": {},
+ "source": [
+ "### Loading the data\n",
+ "\n",
+ "We clip the elementwise input upper-bound to $20.0$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "a8ed0fc4-4655-4bad-a6ac-8697cd5bc7a6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-05-24 16:00:31.206597: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
+ "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
+ "2023-05-24 16:00:31.742417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 47066 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:03:00.0, compute capability: 7.5\n"
+ ]
+ }
+ ],
+ "source": [
+ "import warnings\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "\n",
+ "# data loader return dataset_metadata which allows to\n",
+ "# know the informations required for privacy accounting\n",
+ "# (dataset size, number of samples, max input bound...)\n",
+ "input_upper_bound = 20.0\n",
+ "ds_train, ds_test, dataset_metadata = load_and_prepare_data(\n",
+ " \"mnist\",\n",
+ " batch_size=1000,\n",
+ " drop_remainder=True, # accounting assumes fixed batch size\n",
+ " bound_fct=bound_clip_value( # other strategies are possible, like normalization.\n",
+ " input_upper_bound\n",
+ " ), # clipping preprocessing allows to control input bound\n",
+ ")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "eb356c04-a836-4f49-93d7-7e0cc4c12b1d",
+ "metadata": {},
+ "source": [
+ "### Build the DP model\n",
+ "\n",
+ "We imitate the interface of Keras. We use common layers found in deel-lip, which a wrapper that handles the bound propagation. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "30cf44ed-653b-4eaa-8ed9-26e4815db511",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# construct DP_Sequential\n",
+ "model = DP_Sequential(\n",
+ " # works like usual sequential but requires DP layers\n",
+ " layers=[\n",
+ " # BoundedInput works like Input, but performs input clipping to guarantee input bound\n",
+ " layers.DP_BoundedInput(\n",
+ " input_shape=dataset_metadata.input_shape, upper_bound=input_upper_bound\n",
+ " ),\n",
+ " layers.DP_QuickSpectralConv2D( # Reshaped Kernel Orthogonalization (RKO) convolution.\n",
+ " filters=32,\n",
+ " kernel_size=3,\n",
+ " kernel_initializer=\"orthogonal\",\n",
+ " strides=1,\n",
+ " use_bias=False, # No biases since the framework handles a single tf.Variable per layer.\n",
+ " ),\n",
+ " layers.DP_GroupSort(2), # GNP activation function.\n",
+ " layers.DP_ScaledL2NormPooling2D(pool_size=2, strides=2), # GNP pooling.\n",
+ " layers.DP_QuickSpectralConv2D( # Reshaped Kernel Orthogonalization (RKO) convolution.\n",
+ " filters=64,\n",
+ " kernel_size=3,\n",
+ " kernel_initializer=\"orthogonal\",\n",
+ " strides=1,\n",
+ " use_bias=False, # No biases since the framework handles a single tf.Variable per layer.\n",
+ " ),\n",
+ " layers.DP_GroupSort(2), # GNP activation function.\n",
+ " layers.DP_ScaledL2NormPooling2D(pool_size=2, strides=2), # GNP pooling.\n",
+ " \n",
+ " layers.DP_Flatten(), # Convert features maps to flat vector.\n",
+ " \n",
+ " layers.DP_QuickSpectralDense(512), # GNP layer with orthogonal weight matrix.\n",
+ " layers.DP_GroupSort(2),\n",
+ " layers.DP_QuickSpectralDense(dataset_metadata.nb_classes),\n",
+ " ],\n",
+ " dp_parameters=dp_parameters,\n",
+ " dataset_metadata=dataset_metadata,\n",
+ ")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "09777811",
+ "metadata": {},
+ "source": [
+ "We compile the model with:\n",
+ "* any first order optimizer (e.g SGD). No adaptation or special optimizer is needed.\n",
+ "* a loss with known Lipschitz constant, e.g Categorical Cross-entropy with temperature."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "efd97e75-34f0-49fa-ad2c-1816247f1611",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Model: \"dp__sequential\"\n",
+ "_________________________________________________________________\n",
+ " Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ " dp__bounded_input (DP_Bound (None, 28, 28, 1) 0 \n",
+ " edInput) \n",
+ " \n",
+ " dp__quick_spectral_conv2d ( (None, 26, 26, 32) 288 \n",
+ " DP_QuickSpectralConv2D) \n",
+ " \n",
+ " dp__group_sort (DP_GroupSor (None, 26, 26, 32) 0 \n",
+ " t) \n",
+ " \n",
+ " dp__scaled_l2_norm_pooling2 (None, 13, 13, 32) 0 \n",
+ " d (DP_ScaledL2NormPooling2D \n",
+ " ) \n",
+ " \n",
+ " dp__quick_spectral_conv2d_1 (None, 11, 11, 64) 18432 \n",
+ " (DP_QuickSpectralConv2D) \n",
+ " \n",
+ " dp__group_sort_1 (DP_GroupS (None, 11, 11, 64) 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__scaled_l2_norm_pooling2 (None, 5, 5, 64) 0 \n",
+ " d_1 (DP_ScaledL2NormPooling \n",
+ " 2D) \n",
+ " \n",
+ " dp__flatten (DP_Flatten) (None, 1600) 0 \n",
+ " \n",
+ " dp__quick_spectral_dense (D (None, 512) 819200 \n",
+ " P_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_2 (DP_GroupS (None, 512) 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_1 (None, 10) 5120 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ "=================================================================\n",
+ "Total params: 843,040\n",
+ "Trainable params: 843,040\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model.compile(\n",
+ " # Compile model using DP loss\n",
+ " loss=losses.DP_TauCategoricalCrossentropy(18.0),\n",
+ " # this method is compatible with any first order optimizer\n",
+ " optimizer=tf.keras.optimizers.SGD(learning_rate=2e-4, momentum=0.9),\n",
+ " metrics=[\"accuracy\"],\n",
+ ")\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "28ae2da5-ed40-4131-8721-73bbc73fa68d",
+ "metadata": {},
+ "source": [
+ "Note that the model contains $843$K parameters. Without gradient clipping these architectures can be trained with batch sizes as big as $1000$ on a standard GPU.\n",
+ "\n",
+ "Then, we compute the number of epochs. The maximum value of epsilon will depends on dp_parameters and the number of epochs. In order to control epsilon, we compute the adequate number of epochs"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "dd611afd-be30-4bd3-b658-48d1961247aa",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "epoch bounds = (0, 512.0) and epsilon = 7.994426666195571 at epoch 512.0\n",
+ "epoch bounds = (0, 256.0) and epsilon = 5.34128917907949 at epoch 256.0\n",
+ "epoch bounds = (0, 128.0) and epsilon = 3.631964622805248 at epoch 128.0\n",
+ "epoch bounds = (64.0, 128.0) and epsilon = 2.4829841192119444 at epoch 64.0\n",
+ "epoch bounds = (64.0, 96.0) and epsilon = 3.089635897639078 at epoch 96.0\n",
+ "epoch bounds = (80.0, 96.0) and epsilon = 2.796528753679695 at epoch 80.0\n",
+ "epoch bounds = (88.0, 96.0) and epsilon = 2.952713799856404 at epoch 88.0\n",
+ "epoch bounds = (88.0, 92.0) and epsilon = 3.0216241846349847 at epoch 92.0\n",
+ "epoch bounds = (90.0, 92.0) and epsilon = 2.987618328313939 at epoch 90.0\n",
+ "epoch bounds = (90.0, 91.0) and epsilon = 3.0046212568846444 at epoch 91.0\n"
+ ]
+ }
+ ],
+ "source": [
+ "num_epochs = get_max_epochs(epsilon_max, model)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "53e94244",
+ "metadata": {},
+ "source": [
+ "## Train the model\n",
+ "\n",
+ "The model can be trained, and the DP Accountant will automatically track the privacy loss."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "0ddcb192-547e-400e-87bb-2d4246185c64",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Epoch 1/91\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-05-24 16:00:36.621954: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8300\n",
+ "2023-05-24 16:00:37.363789: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "60/60 [==============================] - ETA: 0s - loss: 0.2020 - accuracy: 0.2324\n",
+ " (0.3227333785403041, 1e-05)-DP guarantees for epoch 1 \n",
+ "\n",
+ "60/60 [==============================] - 5s 38ms/step - loss: 0.2020 - accuracy: 0.2324 - val_loss: 0.1712 - val_accuracy: 0.3147\n",
+ "Epoch 2/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.1607 - accuracy: 0.3958\n",
+ " (0.41135036253440604, 1e-05)-DP guarantees for epoch 2 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1604 - accuracy: 0.3992 - val_loss: 0.1486 - val_accuracy: 0.5122\n",
+ "Epoch 3/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1426 - accuracy: 0.5510\n",
+ " (0.4972854400421322, 1e-05)-DP guarantees for epoch 3 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1426 - accuracy: 0.5510 - val_loss: 0.1334 - val_accuracy: 0.6108\n",
+ "Epoch 4/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1291 - accuracy: 0.6333\n",
+ " (0.5737399623472044, 1e-05)-DP guarantees for epoch 4 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1291 - accuracy: 0.6333 - val_loss: 0.1213 - val_accuracy: 0.6784\n",
+ "Epoch 5/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1182 - accuracy: 0.6883\n",
+ " (0.6418194146435952, 1e-05)-DP guarantees for epoch 5 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1182 - accuracy: 0.6883 - val_loss: 0.1109 - val_accuracy: 0.7180\n",
+ "Epoch 6/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.1088 - accuracy: 0.7247\n",
+ " (0.7042008802236781, 1e-05)-DP guarantees for epoch 6 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1087 - accuracy: 0.7247 - val_loss: 0.1024 - val_accuracy: 0.7527\n",
+ "Epoch 7/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1012 - accuracy: 0.7488\n",
+ " (0.7616059152520757, 1e-05)-DP guarantees for epoch 7 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.1012 - accuracy: 0.7488 - val_loss: 0.0955 - val_accuracy: 0.7698\n",
+ "Epoch 8/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0948 - accuracy: 0.7644\n",
+ " (0.8155744676428971, 1e-05)-DP guarantees for epoch 8 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0948 - accuracy: 0.7644 - val_loss: 0.0899 - val_accuracy: 0.7815\n",
+ "Epoch 9/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0896 - accuracy: 0.7785\n",
+ " (0.8666021691681208, 1e-05)-DP guarantees for epoch 9 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0896 - accuracy: 0.7785 - val_loss: 0.0848 - val_accuracy: 0.7936\n",
+ "Epoch 10/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0849 - accuracy: 0.7868\n",
+ " (0.9152742048884784, 1e-05)-DP guarantees for epoch 10 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0849 - accuracy: 0.7868 - val_loss: 0.0804 - val_accuracy: 0.8003\n",
+ "Epoch 11/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0810 - accuracy: 0.7967\n",
+ " (0.9617965624530973, 1e-05)-DP guarantees for epoch 11 \n",
+ "\n",
+ "60/60 [==============================] - 2s 30ms/step - loss: 0.0809 - accuracy: 0.7975 - val_loss: 0.0769 - val_accuracy: 0.8109\n",
+ "Epoch 12/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0774 - accuracy: 0.8060\n",
+ " (1.0059716506359193, 1e-05)-DP guarantees for epoch 12 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0774 - accuracy: 0.8060 - val_loss: 0.0733 - val_accuracy: 0.8179\n",
+ "Epoch 13/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0740 - accuracy: 0.8131\n",
+ " (1.049398006635733, 1e-05)-DP guarantees for epoch 13 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0740 - accuracy: 0.8131 - val_loss: 0.0704 - val_accuracy: 0.8269\n",
+ "Epoch 14/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0713 - accuracy: 0.8216\n",
+ " (1.090263192229449, 1e-05)-DP guarantees for epoch 14 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0713 - accuracy: 0.8216 - val_loss: 0.0677 - val_accuracy: 0.8309\n",
+ "Epoch 15/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0689 - accuracy: 0.8240\n",
+ " (1.131126828240101, 1e-05)-DP guarantees for epoch 15 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0689 - accuracy: 0.8240 - val_loss: 0.0656 - val_accuracy: 0.8355\n",
+ "Epoch 16/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0669 - accuracy: 0.8293\n",
+ " (1.169340908770284, 1e-05)-DP guarantees for epoch 16 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0668 - accuracy: 0.8296 - val_loss: 0.0635 - val_accuracy: 0.8398\n",
+ "Epoch 17/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0647 - accuracy: 0.8333\n",
+ " (1.2074292910030167, 1e-05)-DP guarantees for epoch 17 \n",
+ "\n",
+ "60/60 [==============================] - 2s 29ms/step - loss: 0.0646 - accuracy: 0.8335 - val_loss: 0.0615 - val_accuracy: 0.8437\n",
+ "Epoch 18/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0630 - accuracy: 0.8366\n",
+ " (1.2447047350704166, 1e-05)-DP guarantees for epoch 18 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0629 - accuracy: 0.8367 - val_loss: 0.0598 - val_accuracy: 0.8468\n",
+ "Epoch 19/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0612 - accuracy: 0.8399\n",
+ " (1.2800495944157277, 1e-05)-DP guarantees for epoch 19 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0612 - accuracy: 0.8399 - val_loss: 0.0582 - val_accuracy: 0.8508\n",
+ "Epoch 20/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0598 - accuracy: 0.8428\n",
+ " (1.3153944538284068, 1e-05)-DP guarantees for epoch 20 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0598 - accuracy: 0.8428 - val_loss: 0.0569 - val_accuracy: 0.8563\n",
+ "Epoch 21/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0584 - accuracy: 0.8468\n",
+ " (1.3507368078845663, 1e-05)-DP guarantees for epoch 21 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0584 - accuracy: 0.8466 - val_loss: 0.0557 - val_accuracy: 0.8572\n",
+ "Epoch 22/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0572 - accuracy: 0.8509\n",
+ " (1.383564204783113, 1e-05)-DP guarantees for epoch 22 \n",
+ "\n",
+ "60/60 [==============================] - 2s 30ms/step - loss: 0.0572 - accuracy: 0.8509 - val_loss: 0.0546 - val_accuracy: 0.8610\n",
+ "Epoch 23/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0561 - accuracy: 0.8519\n",
+ " (1.4161979427317832, 1e-05)-DP guarantees for epoch 23 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0562 - accuracy: 0.8518 - val_loss: 0.0537 - val_accuracy: 0.8619\n",
+ "Epoch 24/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0552 - accuracy: 0.8547\n",
+ " (1.448831680775656, 1e-05)-DP guarantees for epoch 24 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0552 - accuracy: 0.8547 - val_loss: 0.0525 - val_accuracy: 0.8657\n",
+ "Epoch 25/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0541 - accuracy: 0.8575\n",
+ " (1.4814654188092617, 1e-05)-DP guarantees for epoch 25 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0541 - accuracy: 0.8576 - val_loss: 0.0516 - val_accuracy: 0.8675\n",
+ "Epoch 26/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.8578\n",
+ " (1.512526290723161, 1e-05)-DP guarantees for epoch 26 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0531 - accuracy: 0.8578 - val_loss: 0.0506 - val_accuracy: 0.8691\n",
+ "Epoch 27/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0522 - accuracy: 0.8605\n",
+ " (1.5424804710143858, 1e-05)-DP guarantees for epoch 27 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0522 - accuracy: 0.8605 - val_loss: 0.0497 - val_accuracy: 0.8709\n",
+ "Epoch 28/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0512 - accuracy: 0.8624\n",
+ " (1.5724346510360574, 1e-05)-DP guarantees for epoch 28 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0512 - accuracy: 0.8626 - val_loss: 0.0488 - val_accuracy: 0.8730\n",
+ "Epoch 29/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0503 - accuracy: 0.8650\n",
+ " (1.6023888317992228, 1e-05)-DP guarantees for epoch 29 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0503 - accuracy: 0.8653 - val_loss: 0.0479 - val_accuracy: 0.8752\n",
+ "Epoch 30/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0495 - accuracy: 0.8665\n",
+ " (1.632343011263517, 1e-05)-DP guarantees for epoch 30 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0495 - accuracy: 0.8667 - val_loss: 0.0471 - val_accuracy: 0.8749\n",
+ "Epoch 31/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0488 - accuracy: 0.8684\n",
+ " (1.6622962394525178, 1e-05)-DP guarantees for epoch 31 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0487 - accuracy: 0.8686 - val_loss: 0.0463 - val_accuracy: 0.8779\n",
+ "Epoch 32/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0480 - accuracy: 0.8697\n",
+ " (1.689965116494089, 1e-05)-DP guarantees for epoch 32 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0480 - accuracy: 0.8697 - val_loss: 0.0457 - val_accuracy: 0.8777\n",
+ "Epoch 33/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0475 - accuracy: 0.8700\n",
+ " (1.7172705001520499, 1e-05)-DP guarantees for epoch 33 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0475 - accuracy: 0.8704 - val_loss: 0.0452 - val_accuracy: 0.8790\n",
+ "Epoch 34/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0469 - accuracy: 0.8736\n",
+ " (1.7445758842338837, 1e-05)-DP guarantees for epoch 34 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0468 - accuracy: 0.8738 - val_loss: 0.0446 - val_accuracy: 0.8806\n",
+ "Epoch 35/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0463 - accuracy: 0.8754\n",
+ " (1.7718812676250233, 1e-05)-DP guarantees for epoch 35 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0462 - accuracy: 0.8756 - val_loss: 0.0441 - val_accuracy: 0.8825\n",
+ "Epoch 36/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0456 - accuracy: 0.8763\n",
+ " (1.799186650959813, 1e-05)-DP guarantees for epoch 36 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0456 - accuracy: 0.8763 - val_loss: 0.0434 - val_accuracy: 0.8831\n",
+ "Epoch 37/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0450 - accuracy: 0.8771\n",
+ " (1.8264920346090618, 1e-05)-DP guarantees for epoch 37 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0450 - accuracy: 0.8773 - val_loss: 0.0429 - val_accuracy: 0.8846\n",
+ "Epoch 38/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0444 - accuracy: 0.8786\n",
+ " (1.8537974184156425, 1e-05)-DP guarantees for epoch 38 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0444 - accuracy: 0.8786 - val_loss: 0.0423 - val_accuracy: 0.8855\n",
+ "Epoch 39/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0439 - accuracy: 0.8800\n",
+ " (1.8807666749981604, 1e-05)-DP guarantees for epoch 39 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0439 - accuracy: 0.8802 - val_loss: 0.0419 - val_accuracy: 0.8863\n",
+ "Epoch 40/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0435 - accuracy: 0.8803\n",
+ " (1.9054738700393052, 1e-05)-DP guarantees for epoch 40 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0435 - accuracy: 0.8804 - val_loss: 0.0415 - val_accuracy: 0.8858\n",
+ "Epoch 41/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0430 - accuracy: 0.8816\n",
+ " (1.9301604511513608, 1e-05)-DP guarantees for epoch 41 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0430 - accuracy: 0.8816 - val_loss: 0.0410 - val_accuracy: 0.8884\n",
+ "Epoch 42/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0425 - accuracy: 0.8824\n",
+ " (1.9548470320035656, 1e-05)-DP guarantees for epoch 42 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0425 - accuracy: 0.8824 - val_loss: 0.0405 - val_accuracy: 0.8890\n",
+ "Epoch 43/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0421 - accuracy: 0.8837\n",
+ " (1.979533612594768, 1e-05)-DP guarantees for epoch 43 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0421 - accuracy: 0.8837 - val_loss: 0.0403 - val_accuracy: 0.8890\n",
+ "Epoch 44/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0418 - accuracy: 0.8856\n",
+ " (2.0042201936126345, 1e-05)-DP guarantees for epoch 44 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0418 - accuracy: 0.8856 - val_loss: 0.0399 - val_accuracy: 0.8908\n",
+ "Epoch 45/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0414 - accuracy: 0.8858\n",
+ " (2.0289067746857206, 1e-05)-DP guarantees for epoch 45 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0414 - accuracy: 0.8856 - val_loss: 0.0393 - val_accuracy: 0.8926\n",
+ "Epoch 46/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0408 - accuracy: 0.8872\n",
+ " (2.053593355232055, 1e-05)-DP guarantees for epoch 46 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0408 - accuracy: 0.8872 - val_loss: 0.0388 - val_accuracy: 0.8951\n",
+ "Epoch 47/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0405 - accuracy: 0.8882\n",
+ " (2.078279935996221, 1e-05)-DP guarantees for epoch 47 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0404 - accuracy: 0.8887 - val_loss: 0.0385 - val_accuracy: 0.8959\n",
+ "Epoch 48/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0400 - accuracy: 0.8882\n",
+ " (2.1029665168498504, 1e-05)-DP guarantees for epoch 48 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0400 - accuracy: 0.8882 - val_loss: 0.0381 - val_accuracy: 0.8952\n",
+ "Epoch 49/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0397 - accuracy: 0.8890\n",
+ " (2.127653097450219, 1e-05)-DP guarantees for epoch 49 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0398 - accuracy: 0.8888 - val_loss: 0.0379 - val_accuracy: 0.8943\n",
+ "Epoch 50/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0396 - accuracy: 0.8887\n",
+ " (2.151531383398666, 1e-05)-DP guarantees for epoch 50 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0395 - accuracy: 0.8889 - val_loss: 0.0375 - val_accuracy: 0.8946\n",
+ "Epoch 51/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0391 - accuracy: 0.8893\n",
+ " (2.1736284198821467, 1e-05)-DP guarantees for epoch 51 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0391 - accuracy: 0.8895 - val_loss: 0.0372 - val_accuracy: 0.8968\n",
+ "Epoch 52/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0387 - accuracy: 0.8908\n",
+ " (2.195725456202997, 1e-05)-DP guarantees for epoch 52 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0387 - accuracy: 0.8908 - val_loss: 0.0368 - val_accuracy: 0.8967\n",
+ "Epoch 53/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0385 - accuracy: 0.8905\n",
+ " (2.217822492103547, 1e-05)-DP guarantees for epoch 53 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0385 - accuracy: 0.8905 - val_loss: 0.0366 - val_accuracy: 0.8991\n",
+ "Epoch 54/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0382 - accuracy: 0.8913\n",
+ " (2.2399195284840734, 1e-05)-DP guarantees for epoch 54 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0382 - accuracy: 0.8913 - val_loss: 0.0365 - val_accuracy: 0.8992\n",
+ "Epoch 55/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0380 - accuracy: 0.8924\n",
+ " (2.2620165646623547, 1e-05)-DP guarantees for epoch 55 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0380 - accuracy: 0.8921 - val_loss: 0.0362 - val_accuracy: 0.8994\n",
+ "Epoch 56/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0377 - accuracy: 0.8925\n",
+ " (2.2841136015562187, 1e-05)-DP guarantees for epoch 56 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0377 - accuracy: 0.8925 - val_loss: 0.0358 - val_accuracy: 0.8999\n",
+ "Epoch 57/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0374 - accuracy: 0.8930\n",
+ " (2.3062106367493893, 1e-05)-DP guarantees for epoch 57 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0374 - accuracy: 0.8930 - val_loss: 0.0356 - val_accuracy: 0.9004\n",
+ "Epoch 58/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0371 - accuracy: 0.8938\n",
+ " (2.3283076739544244, 1e-05)-DP guarantees for epoch 58 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0372 - accuracy: 0.8939 - val_loss: 0.0354 - val_accuracy: 0.9010\n",
+ "Epoch 59/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0369 - accuracy: 0.8951\n",
+ " (2.3504047095381226, 1e-05)-DP guarantees for epoch 59 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0369 - accuracy: 0.8951 - val_loss: 0.0351 - val_accuracy: 0.9010\n",
+ "Epoch 60/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0365 - accuracy: 0.8963\n",
+ " (2.3725017457248683, 1e-05)-DP guarantees for epoch 60 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0365 - accuracy: 0.8963 - val_loss: 0.0347 - val_accuracy: 0.9037\n",
+ "Epoch 61/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0363 - accuracy: 0.8968\n",
+ " (2.3945987822094885, 1e-05)-DP guarantees for epoch 61 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0363 - accuracy: 0.8968 - val_loss: 0.0346 - val_accuracy: 0.9024\n",
+ "Epoch 62/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0360 - accuracy: 0.8979\n",
+ " (2.4166958179233653, 1e-05)-DP guarantees for epoch 62 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0360 - accuracy: 0.8981 - val_loss: 0.0343 - val_accuracy: 0.9041\n",
+ "Epoch 63/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0358 - accuracy: 0.8986\n",
+ " (2.438792853624178, 1e-05)-DP guarantees for epoch 63 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0358 - accuracy: 0.8987 - val_loss: 0.0340 - val_accuracy: 0.9068\n",
+ "Epoch 64/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0355 - accuracy: 0.8995\n",
+ " (2.4608898896847116, 1e-05)-DP guarantees for epoch 64 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0356 - accuracy: 0.8992 - val_loss: 0.0338 - val_accuracy: 0.9072\n",
+ "Epoch 65/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0352 - accuracy: 0.9005\n",
+ " (2.4829841192119444, 1e-05)-DP guarantees for epoch 65 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0353 - accuracy: 0.9000 - val_loss: 0.0336 - val_accuracy: 0.9059\n",
+ "Epoch 66/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0351 - accuracy: 0.8996\n",
+ " (2.5034880893370737, 1e-05)-DP guarantees for epoch 66 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0351 - accuracy: 0.8996 - val_loss: 0.0334 - val_accuracy: 0.9070\n",
+ "Epoch 67/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0350 - accuracy: 0.9003\n",
+ " (2.523024133549594, 1e-05)-DP guarantees for epoch 67 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0349 - accuracy: 0.9003 - val_loss: 0.0333 - val_accuracy: 0.9069\n",
+ "Epoch 68/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0348 - accuracy: 0.9005\n",
+ " (2.542560178527111, 1e-05)-DP guarantees for epoch 68 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0348 - accuracy: 0.9005 - val_loss: 0.0332 - val_accuracy: 0.9071\n",
+ "Epoch 69/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0346 - accuracy: 0.9006\n",
+ " (2.5620962223364145, 1e-05)-DP guarantees for epoch 69 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0347 - accuracy: 0.9007 - val_loss: 0.0329 - val_accuracy: 0.9081\n",
+ "Epoch 70/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0345 - accuracy: 0.9015\n",
+ " (2.5816322672410785, 1e-05)-DP guarantees for epoch 70 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0345 - accuracy: 0.9014 - val_loss: 0.0327 - val_accuracy: 0.9069\n",
+ "Epoch 71/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0343 - accuracy: 0.9017\n",
+ " (2.601168310806795, 1e-05)-DP guarantees for epoch 71 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0343 - accuracy: 0.9019 - val_loss: 0.0326 - val_accuracy: 0.9090\n",
+ "Epoch 72/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0342 - accuracy: 0.9021\n",
+ " (2.620704354996593, 1e-05)-DP guarantees for epoch 72 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0342 - accuracy: 0.9022 - val_loss: 0.0324 - val_accuracy: 0.9089\n",
+ "Epoch 73/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0340 - accuracy: 0.9018\n",
+ " (2.640240400625916, 1e-05)-DP guarantees for epoch 73 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0339 - accuracy: 0.9020 - val_loss: 0.0322 - val_accuracy: 0.9096\n",
+ "Epoch 74/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0339 - accuracy: 0.9018\n",
+ " (2.659776444789028, 1e-05)-DP guarantees for epoch 74 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0338 - accuracy: 0.9022 - val_loss: 0.0320 - val_accuracy: 0.9103\n",
+ "Epoch 75/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0335 - accuracy: 0.9024\n",
+ " (2.679312488654814, 1e-05)-DP guarantees for epoch 75 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0335 - accuracy: 0.9024 - val_loss: 0.0318 - val_accuracy: 0.9088\n",
+ "Epoch 76/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0333 - accuracy: 0.9025\n",
+ " (2.69884853278786, 1e-05)-DP guarantees for epoch 76 \n",
+ "\n",
+ "60/60 [==============================] - 2s 29ms/step - loss: 0.0333 - accuracy: 0.9023 - val_loss: 0.0315 - val_accuracy: 0.9098\n",
+ "Epoch 77/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0332 - accuracy: 0.9033\n",
+ " (2.7183845763895516, 1e-05)-DP guarantees for epoch 77 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0332 - accuracy: 0.9033 - val_loss: 0.0314 - val_accuracy: 0.9125\n",
+ "Epoch 78/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0330 - accuracy: 0.9046\n",
+ " (2.737920620600221, 1e-05)-DP guarantees for epoch 78 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0330 - accuracy: 0.9048 - val_loss: 0.0313 - val_accuracy: 0.9119\n",
+ "Epoch 79/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0328 - accuracy: 0.9053\n",
+ " (2.7574566653298858, 1e-05)-DP guarantees for epoch 79 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0328 - accuracy: 0.9053 - val_loss: 0.0311 - val_accuracy: 0.9115\n",
+ "Epoch 80/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0328 - accuracy: 0.9052\n",
+ " (2.7769927101097007, 1e-05)-DP guarantees for epoch 80 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0327 - accuracy: 0.9056 - val_loss: 0.0310 - val_accuracy: 0.9118\n",
+ "Epoch 81/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0325 - accuracy: 0.9056\n",
+ " (2.796528753679695, 1e-05)-DP guarantees for epoch 81 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0325 - accuracy: 0.9056 - val_loss: 0.0308 - val_accuracy: 0.9114\n",
+ "Epoch 82/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0324 - accuracy: 0.9057\n",
+ " (2.816064798903292, 1e-05)-DP guarantees for epoch 82 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0324 - accuracy: 0.9057 - val_loss: 0.0307 - val_accuracy: 0.9114\n",
+ "Epoch 83/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0323 - accuracy: 0.9053\n",
+ " (2.8356008431856474, 1e-05)-DP guarantees for epoch 83 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0322 - accuracy: 0.9057 - val_loss: 0.0305 - val_accuracy: 0.9117\n",
+ "Epoch 84/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0320 - accuracy: 0.9063\n",
+ " (2.8551368864333964, 1e-05)-DP guarantees for epoch 84 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0320 - accuracy: 0.9063 - val_loss: 0.0303 - val_accuracy: 0.9117\n",
+ "Epoch 85/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0318 - accuracy: 0.9064\n",
+ " (2.8746729305801413, 1e-05)-DP guarantees for epoch 85 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0318 - accuracy: 0.9064 - val_loss: 0.0302 - val_accuracy: 0.9121\n",
+ "Epoch 86/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0317 - accuracy: 0.9074\n",
+ " (2.894208975473722, 1e-05)-DP guarantees for epoch 86 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0316 - accuracy: 0.9076 - val_loss: 0.0299 - val_accuracy: 0.9132\n",
+ "Epoch 87/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9078\n",
+ " (2.9137450193835823, 1e-05)-DP guarantees for epoch 87 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0314 - accuracy: 0.9076 - val_loss: 0.0298 - val_accuracy: 0.9123\n",
+ "Epoch 88/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0313 - accuracy: 0.9086\n",
+ " (2.9332810632263646, 1e-05)-DP guarantees for epoch 88 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0313 - accuracy: 0.9086 - val_loss: 0.0299 - val_accuracy: 0.9133\n",
+ "Epoch 89/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0313 - accuracy: 0.9087\n",
+ " (2.952713799856404, 1e-05)-DP guarantees for epoch 89 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0313 - accuracy: 0.9087 - val_loss: 0.0298 - val_accuracy: 0.9140\n",
+ "Epoch 90/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0312 - accuracy: 0.9097\n",
+ " (2.970615400210975, 1e-05)-DP guarantees for epoch 90 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0312 - accuracy: 0.9097 - val_loss: 0.0298 - val_accuracy: 0.9127\n",
+ "Epoch 91/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0312 - accuracy: 0.9091\n",
+ " (2.987618328313939, 1e-05)-DP guarantees for epoch 91 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0312 - accuracy: 0.9093 - val_loss: 0.0297 - val_accuracy: 0.9132\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = model.fit(\n",
+ " ds_train,\n",
+ " epochs=num_epochs,\n",
+ " validation_data=ds_test,\n",
+ " callbacks=[\n",
+ " # accounting is done thanks to a callback\n",
+ " DP_Accountant(log_fn=\"logging\"), # wandb.log also available.\n",
+ " ],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e1cbeee4-c204-454f-8f6f-20273b0169b7",
+ "metadata": {},
+ "source": [
+ "The model can be further improved by tuning various hyper-parameters, by adding layers (see `advanced_cifar10.ipynb` tutorial). "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fedc70ab-ccd5-4239-9d62-416d680af324",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/notebooks/demo_fake.ipynb b/docs/notebooks/demo_fake.ipynb
deleted file mode 100644
index 061deb7..0000000
--- a/docs/notebooks/demo_fake.ipynb
+++ /dev/null
@@ -1,101 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## This is a fake demo notebook"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Hello World!\n"
- ]
- }
- ],
- "source": [
- "print(\"Hello World!\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This print is such a classic!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "1 + 2 = 3\n"
- ]
- }
- ],
- "source": [
- "a = 1\n",
- "b = 2\n",
- "\n",
- "c = np.sum([a, b])\n",
- "\n",
- "print(f\"{a} + {b} = {c}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This one is a little bit more fancy!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.8"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 0000000..6edbb83
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,16 @@
+# Tutorials
+
+This folder contains two notebooks illustrating the usage of the library on Mnist and
+ Cifar10 datasets.
+
+The notebooks are self-contained and can be run as is.
+
+The requirements are given in the `requirements.txt` file.
+
+## Mnist
+
+The notebook `basics_mnist.ipynb` is intended to be a quick start guide to the library.
+
+## Cifar10
+
+The notebook `advanced_cifar10.ipynb` is intended to show more advanced features of the library, such as residual connections and loss gradient clipping.
diff --git a/examples/advanced_cifar10.ipynb b/examples/advanced_cifar10.ipynb
new file mode 100644
index 0000000..f37b750
--- /dev/null
+++ b/examples/advanced_cifar10.ipynb
@@ -0,0 +1,1895 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "f7bf07b9-d489-4484-acb9-175cb740dc60",
+ "metadata": {},
+ "source": [
+ "# Cifar-10 tutorial\n",
+ "\n",
+ "This notebook introduces advanced tools like MLP mixer, which involves residual connections with Lipschitz guarantees, other input space (HSB) and loss gradient clipping.\n",
+ "\n",
+ "## Imports"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "8a0eebdf-6082-4d00-aa14-b42953217a93",
+ "metadata": {},
+ "source": [
+ "The library is based on tensorflow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "91c2965e-0375-4966-bc55-776204af9d69",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "9356cd9b-6f79-45f1-8f2e-c46a526c4ae7",
+ "metadata": {},
+ "source": [
+ "### lip-dp dependencies\n",
+ "\n",
+ "The need a model `DP_Model` that handles the noisification of gradients. It is trained with a `loss`. The model is initialized with the convenience function `DPParameters`. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "1e5d58f8-386c-44c7-8c5d-e5b69b5be231",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp import losses\n",
+ "from deel.lipdp.model import DP_Model\n",
+ "from deel.lipdp.model import DPParameters"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "3a247cd3-48d6-4854-92df-01420d3bea80",
+ "metadata": {},
+ "source": [
+ "The `DP_Accountant` callback keeps track of $(\\epsilon,\\delta)$-DP values epoch after epoch. In practice we may be interested in reaching the maximum val_accuracy under privacy constraint $\\epsilon$: the convenience function `get_max_epochs` exactly does that by performing a dichotomy search over the number of epochs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "950c5c56-4b34-4653-aaf3-7d97acc1f5f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.model import DP_Accountant\n",
+ "from deel.lipdp.sensitivity import get_max_epochs"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "893d3078-5166-428c-9cb1-d29ec1f05d71",
+ "metadata": {},
+ "source": [
+ "The framework requires a control of the maximum norm of inputs. This can be ensured with input clipping for example: `bound_clip_value`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "f395c9fc-b67d-4fd2-be4b-b1c43221ebcb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.pipeline import bound_clip_value\n",
+ "from deel.lipdp.pipeline import load_and_prepare_data"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "e54a79db-24b4-4dae-b684-170fa743bc5d",
+ "metadata": {},
+ "source": [
+ "## Setup DP Lipschitz model\n",
+ "\n",
+ "Here we apply the \"global\" strategy, with a noise multiplier $2.5$. Note that for Cifar-10 the dataset size is $N=50,000$, and it is recommended that $\\delta<\\frac{1}{N}$. So we propose a value of $\\delta=10^{-5}$. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "f79ea3b0-33a6-401c-a3a3-e314939fd269",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import warnings\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "\n",
+ "dp_parameters = DPParameters(\n",
+ " noisify_strategy=\"global\",\n",
+ " noise_multiplier=4.0,\n",
+ " delta=1e-5,\n",
+ ")\n",
+ "\n",
+ "epsilon_max = 10.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ba392eec-4451-49e5-bd45-883af7aa2d40",
+ "metadata": {},
+ "source": [
+ "With many parameters, it can be interesting to use `local` strategy over `global`, since the effective noise growths as $\\mathcal{O}(\\sqrt{(D)})$ in `global` strategy. Since the privacy leakge is more important is `local` strategy, we compensate with high `noise_multiplier`.\n",
+ "\n",
+ "![DP-SGD accountant](fig_accountant.png \"DP-SGD accountant\")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "6482128c-ac2e-4cdd-9bbd-6d3172c292b1",
+ "metadata": {},
+ "source": [
+ "### Loading the data\n",
+ "\n",
+ "We clip the elementwise input upper-bound to $40.0$. The operates in `HSV` space. The train set is augmented with random left/right flips."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "a8ed0fc4-4655-4bad-a6ac-8697cd5bc7a6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-05-24 17:27:24.335576: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
+ "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
+ "2023-05-24 17:27:24.905888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 47066 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:03:00.0, compute capability: 7.5\n"
+ ]
+ }
+ ],
+ "source": [
+ "def augmentation_fct(image, label):\n",
+ " image = tf.image.random_flip_left_right(image)\n",
+ " return image, label\n",
+ "\n",
+ "input_upper_bound = 30.0\n",
+ "ds_train, ds_test, dataset_metadata = load_and_prepare_data(\n",
+ " \"cifar10\",\n",
+ " colorspace=\"HSV\",\n",
+ " batch_size=10_000,\n",
+ " drop_remainder=True, # accounting assumes fixed batch size\n",
+ " augmentation_fct=augmentation_fct,\n",
+ " bound_fct=bound_clip_value( # other strategies are possible, like normalization.\n",
+ " input_upper_bound\n",
+ " ), # clipping preprocessing allows to control input bound\n",
+ ")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "eb356c04-a836-4f49-93d7-7e0cc4c12b1d",
+ "metadata": {},
+ "source": [
+ "### Build the MLP Mixer model\n",
+ "\n",
+ "We imitate the interface of Keras. We use common layers found in deel-lip, which a wrapper that handles the bound propagation. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "be32d5d7-efc7-4cc6-91bc-1a2b9bedddca",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.layers import DP_AddBias\n",
+ "from deel.lipdp.layers import DP_BoundedInput\n",
+ "from deel.lipdp.layers import DP_ClipGradient\n",
+ "from deel.lipdp.layers import DP_Flatten\n",
+ "from deel.lipdp.layers import DP_GroupSort\n",
+ "from deel.lipdp.layers import DP_Lambda\n",
+ "from deel.lipdp.layers import DP_LayerCentering\n",
+ "from deel.lipdp.layers import DP_Permute\n",
+ "from deel.lipdp.layers import DP_QuickSpectralDense\n",
+ "from deel.lipdp.layers import DP_Reshape\n",
+ "from deel.lipdp.layers import DP_ScaledGlobalL2NormPooling2D\n",
+ "from deel.lipdp.layers import DP_ScaledL2NormPooling2D\n",
+ "from deel.lipdp.layers import DP_QuickSpectralConv2D"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "15b21796-b8e7-41d3-8718-0efdb5d92179",
+ "metadata": {},
+ "source": [
+ "The MLP Mixer uses residual connections. Residuals connections are handled with the utility function `make_residuals` that wraps the layers inside a block that handles bounds propagation.\n",
+ "\n",
+ "![Residuals Connections](residuals.png \"Residual Connections\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "0590e72d-ce2e-48c1-a8ae-e86ecd32b524",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.layers import make_residuals"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9d75f692-c66d-4318-a915-f16707ed87fa",
+ "metadata": {},
+ "source": [
+ "Now, we proceed with the creation of the environnement."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "30cf44ed-653b-4eaa-8ed9-26e4815db511",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "skip_connections = False # use skip connections, like in original MLP Mixer architecture.\n",
+ "clip_loss_gradient = 2**0.5 # elementwise gradient is clipped to value sqrt(2) - which is the maximum for CCE loss.\n",
+ "add_biases = False # Add biases after linear transformations.\n",
+ "biases_norm_max = 0.05\n",
+ "hidden_size = 64\n",
+ "mlp_seq_dim = 64\n",
+ "mlp_channel_dim = 128\n",
+ "num_mixer_layers = 2 # Two MLP Mixer blocks.\n",
+ "layer_centering = False # Centering operation (like LayerNormalization without the reducing operation). Linear 1-Lipschitz.\n",
+ "patch_size = 4 # Number of pixels in each patch.\n",
+ "\n",
+ "def create_MLP_Mixer(dp_parameters, dataset_metadata, upper_bound):\n",
+ " input_shape = (32, 32, 3)\n",
+ " layers = [DP_BoundedInput(input_shape=input_shape, upper_bound=upper_bound)]\n",
+ "\n",
+ " layers.append(\n",
+ " DP_Lambda(\n",
+ " tf.image.extract_patches,\n",
+ " arguments=dict(\n",
+ " sizes=[1, patch_size, patch_size, 1],\n",
+ " strides=[1, patch_size, patch_size, 1],\n",
+ " rates=[1, 1, 1, 1],\n",
+ " padding=\"VALID\",\n",
+ " ),\n",
+ " )\n",
+ " )\n",
+ "\n",
+ " seq_len = (input_shape[0] // patch_size) * (input_shape[1] // patch_size)\n",
+ "\n",
+ " layers.append(DP_Reshape((seq_len, (patch_size ** 2) * input_shape[-1])))\n",
+ " layers.append(\n",
+ " DP_QuickSpectralDense(\n",
+ " units=hidden_size, use_bias=False, kernel_initializer=\"identity\"\n",
+ " )\n",
+ " )\n",
+ "\n",
+ " for _ in range(num_mixer_layers):\n",
+ " to_add = [\n",
+ " DP_Permute((2, 1)),\n",
+ " DP_QuickSpectralDense(\n",
+ " units=mlp_seq_dim, use_bias=False, kernel_initializer=\"identity\"\n",
+ " ),\n",
+ " ]\n",
+ " if add_biases:\n",
+ " to_add.append(DP_AddBias(biases_norm_max))\n",
+ " to_add.append(DP_GroupSort(2))\n",
+ " if layer_centering:\n",
+ " to_add.append(DP_LayerCentering())\n",
+ " to_add += [\n",
+ " DP_QuickSpectralDense(\n",
+ " units=seq_len, use_bias=False, kernel_initializer=\"identity\"\n",
+ " ),\n",
+ " DP_Permute((2, 1)),\n",
+ " ]\n",
+ "\n",
+ " if skip_connections:\n",
+ " layers += make_residuals(\"1-lip-add\", to_add)\n",
+ " else:\n",
+ " layers += to_add\n",
+ "\n",
+ " to_add = [\n",
+ " DP_QuickSpectralDense(\n",
+ " units=mlp_channel_dim, use_bias=False, kernel_initializer=\"identity\"\n",
+ " ),\n",
+ " ]\n",
+ " if add_biases:\n",
+ " to_add.append(DP_AddBias(biases_norm_max))\n",
+ " to_add.append(DP_GroupSort(2))\n",
+ " if layer_centering:\n",
+ " to_add.append(DP_LayerCentering())\n",
+ " to_add.append(\n",
+ " DP_QuickSpectralDense(\n",
+ " units=hidden_size, use_bias=False, kernel_initializer=\"identity\"\n",
+ " )\n",
+ " )\n",
+ "\n",
+ " if skip_connections:\n",
+ " layers += make_residuals(\"1-lip-add\", to_add)\n",
+ " else:\n",
+ " layers += to_add\n",
+ "\n",
+ " layers.append(DP_Flatten())\n",
+ " layers.append(\n",
+ " DP_QuickSpectralDense(units=10, use_bias=False, kernel_initializer=\"identity\")\n",
+ " )\n",
+ "\n",
+ " layers.append(DP_ClipGradient(clip_loss_gradient))\n",
+ "\n",
+ " model = DP_Model(\n",
+ " layers,\n",
+ " dp_parameters=dp_parameters,\n",
+ " dataset_metadata=dataset_metadata,\n",
+ " name=\"mlp_mixer\",\n",
+ " )\n",
+ "\n",
+ " model.build(input_shape=(None, *input_shape))\n",
+ "\n",
+ " return model"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "09777811",
+ "metadata": {},
+ "source": [
+ "We compile the model with:\n",
+ "* any first order optimizer (e.g Adam). No adaptation is needed.\n",
+ "* a loss with known Lipschitz constant, e.g Categorical Cross-entropy with temperature."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "efd97e75-34f0-49fa-ad2c-1816247f1611",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Model: \"mlp_mixer\"\n",
+ "_________________________________________________________________\n",
+ " Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ " dp__bounded_input (DP_Bound multiple 0 \n",
+ " edInput) \n",
+ " \n",
+ " dp__lambda (DP_Lambda) multiple 0 \n",
+ " \n",
+ " dp__reshape (DP_Reshape) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense (D multiple 3072 \n",
+ " P_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_1 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort (DP_GroupSor multiple 0 \n",
+ " t) \n",
+ " \n",
+ " dp__quick_spectral_dense_2 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute_1 (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_3 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_1 (DP_GroupS multiple 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_4 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute_2 (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_5 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_2 (DP_GroupS multiple 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_6 multiple 4096 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__permute_3 (DP_Permute) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_7 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_3 (DP_GroupS multiple 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_8 multiple 8192 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__flatten (DP_Flatten) multiple 0 \n",
+ " \n",
+ " dp__quick_spectral_dense_9 multiple 40960 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ " dp__clip_gradient (DP_ClipG multiple 0 \n",
+ " radient) \n",
+ " \n",
+ "=================================================================\n",
+ "Total params: 93,184\n",
+ "Trainable params: 93,184\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = create_MLP_Mixer(dp_parameters, dataset_metadata, input_upper_bound)\n",
+ "model.compile(\n",
+ " # Compile model using DP loss\n",
+ " loss=losses.DP_TauCategoricalCrossentropy(256.0),\n",
+ " # this method is compatible with any first order optimizer\n",
+ " optimizer=tf.keras.optimizers.Adam(learning_rate=2e-4),\n",
+ " metrics=[\"accuracy\"],\n",
+ ")\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "28ae2da5-ed40-4131-8721-73bbc73fa68d",
+ "metadata": {},
+ "source": [
+ "Observe that the model contains only 246K parmaeters. This is an advantage of MLP Mixer architectures: the number of parameters is small. However the number of FLOPS can be quite high. Without gradient clipping, huge batch sizes can be used, which benefits to privacy/utility ratio. \n",
+ "\n",
+ "In order to control epsilon, we compute the adequate number of epochs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "dd611afd-be30-4bd3-b658-48d1961247aa",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "epoch bounds = (0, 512.0) and epsilon = 14.81894855578722 at epoch 512.0\n",
+ "epoch bounds = (256.0, 512.0) and epsilon = 9.820083418023108 at epoch 256.0\n",
+ "epoch bounds = (256.0, 384.0) and epsilon = 12.31951600358698 at epoch 384.0\n",
+ "epoch bounds = (256.0, 320.0) and epsilon = 11.069799714608529 at epoch 320.0\n",
+ "epoch bounds = (256.0, 288.0) and epsilon = 10.44494156631582 at epoch 288.0\n",
+ "epoch bounds = (256.0, 272.0) and epsilon = 10.132512492169463 at epoch 272.0\n",
+ "epoch bounds = (264.0, 272.0) and epsilon = 9.976297955096285 at epoch 264.0\n",
+ "epoch bounds = (264.0, 268.0) and epsilon = 10.054405223632873 at epoch 268.0\n",
+ "epoch bounds = (264.0, 266.0) and epsilon = 10.015351589364581 at epoch 266.0\n",
+ "epoch bounds = (265.0, 266.0) and epsilon = 9.995824772230431 at epoch 265.0\n"
+ ]
+ }
+ ],
+ "source": [
+ "num_epochs = get_max_epochs(epsilon_max, model)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "53e94244",
+ "metadata": {},
+ "source": [
+ "## Train the model\n",
+ "\n",
+ "The model can be trained, and the DP Accountant will automatically track the privacy loss."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "0ddcb192-547e-400e-87bb-2d4246185c64",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Epoch 1/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1751 - accuracy: 0.1077\n",
+ " (0.5205893807331654, 1e-05)-DP guarantees for epoch 1 \n",
+ "\n",
+ "5/5 [==============================] - 8s 547ms/step - loss: 0.1751 - accuracy: 0.1077 - val_loss: 0.1409 - val_accuracy: 0.1045\n",
+ "Epoch 2/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1243 - accuracy: 0.1061\n",
+ " (0.7169615437758403, 1e-05)-DP guarantees for epoch 2 \n",
+ "\n",
+ "5/5 [==============================] - 3s 451ms/step - loss: 0.1243 - accuracy: 0.1061 - val_loss: 0.1145 - val_accuracy: 0.1055\n",
+ "Epoch 3/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1124 - accuracy: 0.1170\n",
+ " (0.8714581783028138, 1e-05)-DP guarantees for epoch 3 \n",
+ "\n",
+ "5/5 [==============================] - 3s 386ms/step - loss: 0.1124 - accuracy: 0.1170 - val_loss: 0.1095 - val_accuracy: 0.1124\n",
+ "Epoch 4/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.1051 - accuracy: 0.1178\n",
+ " (1.0041033056975341, 1e-05)-DP guarantees for epoch 4 \n",
+ "\n",
+ "5/5 [==============================] - 3s 416ms/step - loss: 0.1051 - accuracy: 0.1178 - val_loss: 0.1019 - val_accuracy: 0.1173\n",
+ "Epoch 5/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0994 - accuracy: 0.1219\n",
+ " (1.121902451763874, 1e-05)-DP guarantees for epoch 5 \n",
+ "\n",
+ "5/5 [==============================] - 3s 404ms/step - loss: 0.0994 - accuracy: 0.1219 - val_loss: 0.0973 - val_accuracy: 0.1199\n",
+ "Epoch 6/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0950 - accuracy: 0.1287\n",
+ " (1.2297900098052366, 1e-05)-DP guarantees for epoch 6 \n",
+ "\n",
+ "5/5 [==============================] - 3s 372ms/step - loss: 0.0950 - accuracy: 0.1287 - val_loss: 0.0952 - val_accuracy: 0.1274\n",
+ "Epoch 7/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0927 - accuracy: 0.1332\n",
+ " (1.3301791512711914, 1e-05)-DP guarantees for epoch 7 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0927 - accuracy: 0.1332 - val_loss: 0.0917 - val_accuracy: 0.1319\n",
+ "Epoch 8/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0896 - accuracy: 0.1396\n",
+ " (1.425115891691246, 1e-05)-DP guarantees for epoch 8 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0896 - accuracy: 0.1396 - val_loss: 0.0898 - val_accuracy: 0.1348\n",
+ "Epoch 9/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0878 - accuracy: 0.1423\n",
+ " (1.512644960027369, 1e-05)-DP guarantees for epoch 9 \n",
+ "\n",
+ "5/5 [==============================] - 2s 367ms/step - loss: 0.0878 - accuracy: 0.1423 - val_loss: 0.0876 - val_accuracy: 0.1386\n",
+ "Epoch 10/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0857 - accuracy: 0.1461\n",
+ " (1.599192443478913, 1e-05)-DP guarantees for epoch 10 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0857 - accuracy: 0.1461 - val_loss: 0.0859 - val_accuracy: 0.1469\n",
+ "Epoch 11/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0840 - accuracy: 0.1543\n",
+ " (1.6782666312983627, 1e-05)-DP guarantees for epoch 11 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0840 - accuracy: 0.1543 - val_loss: 0.0844 - val_accuracy: 0.1497\n",
+ "Epoch 12/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0829 - accuracy: 0.1556\n",
+ " (1.7566369758486253, 1e-05)-DP guarantees for epoch 12 \n",
+ "\n",
+ "5/5 [==============================] - 3s 358ms/step - loss: 0.0829 - accuracy: 0.1556 - val_loss: 0.0829 - val_accuracy: 0.1516\n",
+ "Epoch 13/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0816 - accuracy: 0.1578\n",
+ " (1.833150779023074, 1e-05)-DP guarantees for epoch 13 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0816 - accuracy: 0.1578 - val_loss: 0.0819 - val_accuracy: 0.1565\n",
+ "Epoch 14/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0806 - accuracy: 0.1618\n",
+ " (1.903546174784228, 1e-05)-DP guarantees for epoch 14 \n",
+ "\n",
+ "5/5 [==============================] - 3s 370ms/step - loss: 0.0806 - accuracy: 0.1618 - val_loss: 0.0809 - val_accuracy: 0.1592\n",
+ "Epoch 15/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0794 - accuracy: 0.1657\n",
+ " (1.9739415712927695, 1e-05)-DP guarantees for epoch 15 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0794 - accuracy: 0.1657 - val_loss: 0.0799 - val_accuracy: 0.1614\n",
+ "Epoch 16/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0788 - accuracy: 0.1654\n",
+ " (2.044336966003477, 1e-05)-DP guarantees for epoch 16 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0788 - accuracy: 0.1654 - val_loss: 0.0791 - val_accuracy: 0.1642\n",
+ "Epoch 17/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0778 - accuracy: 0.1696\n",
+ " (2.111107170532668, 1e-05)-DP guarantees for epoch 17 \n",
+ "\n",
+ "5/5 [==============================] - 3s 373ms/step - loss: 0.0778 - accuracy: 0.1696 - val_loss: 0.0783 - val_accuracy: 0.1667\n",
+ "Epoch 18/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0773 - accuracy: 0.1720\n",
+ " (2.173720558035018, 1e-05)-DP guarantees for epoch 18 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0773 - accuracy: 0.1720 - val_loss: 0.0775 - val_accuracy: 0.1713\n",
+ "Epoch 19/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0765 - accuracy: 0.1745\n",
+ " (2.236333946199693, 1e-05)-DP guarantees for epoch 19 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0765 - accuracy: 0.1745 - val_loss: 0.0768 - val_accuracy: 0.1718\n",
+ "Epoch 20/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0755 - accuracy: 0.1785\n",
+ " (2.298947335447459, 1e-05)-DP guarantees for epoch 20 \n",
+ "\n",
+ "5/5 [==============================] - 3s 351ms/step - loss: 0.0755 - accuracy: 0.1785 - val_loss: 0.0761 - val_accuracy: 0.1749\n",
+ "Epoch 21/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0751 - accuracy: 0.1809\n",
+ " (2.3615607218535017, 1e-05)-DP guarantees for epoch 21 \n",
+ "\n",
+ "5/5 [==============================] - 2s 370ms/step - loss: 0.0751 - accuracy: 0.1809 - val_loss: 0.0755 - val_accuracy: 0.1779\n",
+ "Epoch 22/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0744 - accuracy: 0.1807\n",
+ " (2.424031214499055, 1e-05)-DP guarantees for epoch 22 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0744 - accuracy: 0.1807 - val_loss: 0.0749 - val_accuracy: 0.1782\n",
+ "Epoch 23/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0737 - accuracy: 0.1829\n",
+ " (2.4794700865598074, 1e-05)-DP guarantees for epoch 23 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0737 - accuracy: 0.1829 - val_loss: 0.0744 - val_accuracy: 0.1796\n",
+ "Epoch 24/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0735 - accuracy: 0.1836\n",
+ " (2.5344857802909178, 1e-05)-DP guarantees for epoch 24 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0735 - accuracy: 0.1836 - val_loss: 0.0738 - val_accuracy: 0.1815\n",
+ "Epoch 25/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0730 - accuracy: 0.1853\n",
+ " (2.589501472054093, 1e-05)-DP guarantees for epoch 25 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0730 - accuracy: 0.1853 - val_loss: 0.0733 - val_accuracy: 0.1836\n",
+ "Epoch 26/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0726 - accuracy: 0.1884\n",
+ " (2.6445171621630954, 1e-05)-DP guarantees for epoch 26 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0726 - accuracy: 0.1884 - val_loss: 0.0729 - val_accuracy: 0.1857\n",
+ "Epoch 27/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0722 - accuracy: 0.1881\n",
+ " (2.699532854747239, 1e-05)-DP guarantees for epoch 27 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0722 - accuracy: 0.1881 - val_loss: 0.0723 - val_accuracy: 0.1882\n",
+ "Epoch 28/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0715 - accuracy: 0.1901\n",
+ " (2.754548546420506, 1e-05)-DP guarantees for epoch 28 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0715 - accuracy: 0.1901 - val_loss: 0.0718 - val_accuracy: 0.1879\n",
+ "Epoch 29/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0711 - accuracy: 0.1928\n",
+ " (2.809564239271509, 1e-05)-DP guarantees for epoch 29 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0711 - accuracy: 0.1928 - val_loss: 0.0715 - val_accuracy: 0.1915\n",
+ "Epoch 30/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0710 - accuracy: 0.1933\n",
+ " (2.8645799306976425, 1e-05)-DP guarantees for epoch 30 \n",
+ "\n",
+ "5/5 [==============================] - 2s 362ms/step - loss: 0.0710 - accuracy: 0.1933 - val_loss: 0.0710 - val_accuracy: 0.1922\n",
+ "Epoch 31/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0701 - accuracy: 0.1993\n",
+ " (2.915773408283026, 1e-05)-DP guarantees for epoch 31 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0701 - accuracy: 0.1993 - val_loss: 0.0706 - val_accuracy: 0.1940\n",
+ "Epoch 32/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0698 - accuracy: 0.1996\n",
+ " (2.9633676512735834, 1e-05)-DP guarantees for epoch 32 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0698 - accuracy: 0.1996 - val_loss: 0.0702 - val_accuracy: 0.1964\n",
+ "Epoch 33/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0695 - accuracy: 0.2004\n",
+ " (3.010961895901816, 1e-05)-DP guarantees for epoch 33 \n",
+ "\n",
+ "5/5 [==============================] - 3s 375ms/step - loss: 0.0695 - accuracy: 0.2004 - val_loss: 0.0699 - val_accuracy: 0.1984\n",
+ "Epoch 34/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0692 - accuracy: 0.1995\n",
+ " (3.0585561401091397, 1e-05)-DP guarantees for epoch 34 \n",
+ "\n",
+ "5/5 [==============================] - 3s 352ms/step - loss: 0.0692 - accuracy: 0.1995 - val_loss: 0.0696 - val_accuracy: 0.1975\n",
+ "Epoch 35/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0685 - accuracy: 0.2045\n",
+ " (3.1061503817189315, 1e-05)-DP guarantees for epoch 35 \n",
+ "\n",
+ "5/5 [==============================] - 3s 349ms/step - loss: 0.0685 - accuracy: 0.2045 - val_loss: 0.0692 - val_accuracy: 0.2009\n",
+ "Epoch 36/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0686 - accuracy: 0.2045\n",
+ " (3.1537446235861095, 1e-05)-DP guarantees for epoch 36 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0686 - accuracy: 0.2045 - val_loss: 0.0689 - val_accuracy: 0.2032\n",
+ "Epoch 37/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0684 - accuracy: 0.2033\n",
+ " (3.2013388677062005, 1e-05)-DP guarantees for epoch 37 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0684 - accuracy: 0.2033 - val_loss: 0.0686 - val_accuracy: 0.2033\n",
+ "Epoch 38/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0684 - accuracy: 0.2024\n",
+ " (3.2489331117939875, 1e-05)-DP guarantees for epoch 38 \n",
+ "\n",
+ "5/5 [==============================] - 3s 352ms/step - loss: 0.0684 - accuracy: 0.2024 - val_loss: 0.0683 - val_accuracy: 0.2046\n",
+ "Epoch 39/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0675 - accuracy: 0.2064\n",
+ " (3.296527354122463, 1e-05)-DP guarantees for epoch 39 \n",
+ "\n",
+ "5/5 [==============================] - 3s 390ms/step - loss: 0.0675 - accuracy: 0.2064 - val_loss: 0.0681 - val_accuracy: 0.2055\n",
+ "Epoch 40/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0678 - accuracy: 0.2071\n",
+ " (3.3441215974412257, 1e-05)-DP guarantees for epoch 40 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0678 - accuracy: 0.2071 - val_loss: 0.0679 - val_accuracy: 0.2061\n",
+ "Epoch 41/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0670 - accuracy: 0.2076\n",
+ " (3.391715841019588, 1e-05)-DP guarantees for epoch 41 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0670 - accuracy: 0.2076 - val_loss: 0.0676 - val_accuracy: 0.2047\n",
+ "Epoch 42/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0670 - accuracy: 0.2074\n",
+ " (3.4393100820764655, 1e-05)-DP guarantees for epoch 42 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0670 - accuracy: 0.2074 - val_loss: 0.0673 - val_accuracy: 0.2077\n",
+ "Epoch 43/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0668 - accuracy: 0.2091\n",
+ " (3.4869043257012042, 1e-05)-DP guarantees for epoch 43 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0668 - accuracy: 0.2091 - val_loss: 0.0671 - val_accuracy: 0.2098\n",
+ "Epoch 44/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0664 - accuracy: 0.2133\n",
+ " (3.5344943006583662, 1e-05)-DP guarantees for epoch 44 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0664 - accuracy: 0.2133 - val_loss: 0.0668 - val_accuracy: 0.2111\n",
+ "Epoch 45/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0662 - accuracy: 0.2116\n",
+ " (3.577278802435221, 1e-05)-DP guarantees for epoch 45 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0662 - accuracy: 0.2116 - val_loss: 0.0666 - val_accuracy: 0.2110\n",
+ "Epoch 46/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0658 - accuracy: 0.2144\n",
+ " (3.6176202954309518, 1e-05)-DP guarantees for epoch 46 \n",
+ "\n",
+ "5/5 [==============================] - 3s 363ms/step - loss: 0.0658 - accuracy: 0.2144 - val_loss: 0.0663 - val_accuracy: 0.2136\n",
+ "Epoch 47/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0660 - accuracy: 0.2136\n",
+ " (3.6579617884266824, 1e-05)-DP guarantees for epoch 47 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0660 - accuracy: 0.2136 - val_loss: 0.0662 - val_accuracy: 0.2103\n",
+ "Epoch 48/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0658 - accuracy: 0.2124\n",
+ " (3.698303280878773, 1e-05)-DP guarantees for epoch 48 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0658 - accuracy: 0.2124 - val_loss: 0.0660 - val_accuracy: 0.2126\n",
+ "Epoch 49/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0651 - accuracy: 0.2170\n",
+ " (3.7386447748463074, 1e-05)-DP guarantees for epoch 49 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0651 - accuracy: 0.2170 - val_loss: 0.0658 - val_accuracy: 0.2141\n",
+ "Epoch 50/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0650 - accuracy: 0.2147\n",
+ " (3.778986264959221, 1e-05)-DP guarantees for epoch 50 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0650 - accuracy: 0.2147 - val_loss: 0.0657 - val_accuracy: 0.2139\n",
+ "Epoch 51/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0649 - accuracy: 0.2157\n",
+ " (3.819327759198358, 1e-05)-DP guarantees for epoch 51 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0649 - accuracy: 0.2157 - val_loss: 0.0654 - val_accuracy: 0.2154\n",
+ "Epoch 52/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0646 - accuracy: 0.2177\n",
+ " (3.859669252353283, 1e-05)-DP guarantees for epoch 52 \n",
+ "\n",
+ "5/5 [==============================] - 3s 374ms/step - loss: 0.0646 - accuracy: 0.2177 - val_loss: 0.0652 - val_accuracy: 0.2159\n",
+ "Epoch 53/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0647 - accuracy: 0.2164\n",
+ " (3.900010744909916, 1e-05)-DP guarantees for epoch 53 \n",
+ "\n",
+ "5/5 [==============================] - 3s 398ms/step - loss: 0.0647 - accuracy: 0.2164 - val_loss: 0.0651 - val_accuracy: 0.2139\n",
+ "Epoch 54/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0642 - accuracy: 0.2180\n",
+ " (3.9403522382284417, 1e-05)-DP guarantees for epoch 54 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0642 - accuracy: 0.2180 - val_loss: 0.0649 - val_accuracy: 0.2165\n",
+ "Epoch 55/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0643 - accuracy: 0.2178\n",
+ " (3.9806937272852823, 1e-05)-DP guarantees for epoch 55 \n",
+ "\n",
+ "5/5 [==============================] - 3s 385ms/step - loss: 0.0643 - accuracy: 0.2178 - val_loss: 0.0648 - val_accuracy: 0.2190\n",
+ "Epoch 56/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0642 - accuracy: 0.2194\n",
+ " (4.021035219696142, 1e-05)-DP guarantees for epoch 56 \n",
+ "\n",
+ "5/5 [==============================] - 3s 358ms/step - loss: 0.0642 - accuracy: 0.2194 - val_loss: 0.0646 - val_accuracy: 0.2190\n",
+ "Epoch 57/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0641 - accuracy: 0.2193\n",
+ " (4.061376713362479, 1e-05)-DP guarantees for epoch 57 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0641 - accuracy: 0.2193 - val_loss: 0.0644 - val_accuracy: 0.2188\n",
+ "Epoch 58/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0637 - accuracy: 0.2209\n",
+ " (4.101718205195644, 1e-05)-DP guarantees for epoch 58 \n",
+ "\n",
+ "5/5 [==============================] - 3s 389ms/step - loss: 0.0637 - accuracy: 0.2209 - val_loss: 0.0643 - val_accuracy: 0.2203\n",
+ "Epoch 59/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0636 - accuracy: 0.2207\n",
+ " (4.142059698567775, 1e-05)-DP guarantees for epoch 59 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0636 - accuracy: 0.2207 - val_loss: 0.0641 - val_accuracy: 0.2217\n",
+ "Epoch 60/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0631 - accuracy: 0.2238\n",
+ " (4.182401188996273, 1e-05)-DP guarantees for epoch 60 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0631 - accuracy: 0.2238 - val_loss: 0.0639 - val_accuracy: 0.2218\n",
+ "Epoch 61/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0635 - accuracy: 0.2223\n",
+ " (4.222742681534986, 1e-05)-DP guarantees for epoch 61 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0635 - accuracy: 0.2223 - val_loss: 0.0638 - val_accuracy: 0.2214\n",
+ "Epoch 62/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0628 - accuracy: 0.2212\n",
+ " (4.263084178169554, 1e-05)-DP guarantees for epoch 62 \n",
+ "\n",
+ "5/5 [==============================] - 3s 358ms/step - loss: 0.0628 - accuracy: 0.2212 - val_loss: 0.0637 - val_accuracy: 0.2214\n",
+ "Epoch 63/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0629 - accuracy: 0.2236\n",
+ " (4.303425669322495, 1e-05)-DP guarantees for epoch 63 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0629 - accuracy: 0.2236 - val_loss: 0.0635 - val_accuracy: 0.2238\n",
+ "Epoch 64/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0628 - accuracy: 0.2244\n",
+ " (4.343767159305043, 1e-05)-DP guarantees for epoch 64 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0628 - accuracy: 0.2244 - val_loss: 0.0633 - val_accuracy: 0.2229\n",
+ "Epoch 65/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0627 - accuracy: 0.2242\n",
+ " (4.384108652677016, 1e-05)-DP guarantees for epoch 65 \n",
+ "\n",
+ "5/5 [==============================] - 3s 375ms/step - loss: 0.0627 - accuracy: 0.2242 - val_loss: 0.0632 - val_accuracy: 0.2232\n",
+ "Epoch 66/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0625 - accuracy: 0.2260\n",
+ " (4.42445014497077, 1e-05)-DP guarantees for epoch 66 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0625 - accuracy: 0.2260 - val_loss: 0.0630 - val_accuracy: 0.2248\n",
+ "Epoch 67/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0625 - accuracy: 0.2271\n",
+ " (4.4647916365799585, 1e-05)-DP guarantees for epoch 67 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0625 - accuracy: 0.2271 - val_loss: 0.0628 - val_accuracy: 0.2265\n",
+ "Epoch 68/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0622 - accuracy: 0.2292\n",
+ " (4.505133128586104, 1e-05)-DP guarantees for epoch 68 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0622 - accuracy: 0.2292 - val_loss: 0.0626 - val_accuracy: 0.2242\n",
+ "Epoch 69/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0623 - accuracy: 0.2276\n",
+ " (4.544958472325187, 1e-05)-DP guarantees for epoch 69 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0623 - accuracy: 0.2276 - val_loss: 0.0626 - val_accuracy: 0.2254\n",
+ "Epoch 70/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0619 - accuracy: 0.2288\n",
+ " (4.580253889044595, 1e-05)-DP guarantees for epoch 70 \n",
+ "\n",
+ "5/5 [==============================] - 2s 362ms/step - loss: 0.0619 - accuracy: 0.2288 - val_loss: 0.0624 - val_accuracy: 0.2272\n",
+ "Epoch 71/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0619 - accuracy: 0.2288\n",
+ " (4.613504255128257, 1e-05)-DP guarantees for epoch 71 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0619 - accuracy: 0.2288 - val_loss: 0.0623 - val_accuracy: 0.2258\n",
+ "Epoch 72/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0617 - accuracy: 0.2283\n",
+ " (4.646754619793705, 1e-05)-DP guarantees for epoch 72 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0617 - accuracy: 0.2283 - val_loss: 0.0622 - val_accuracy: 0.2262\n",
+ "Epoch 73/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0615 - accuracy: 0.2309\n",
+ " (4.680004986868141, 1e-05)-DP guarantees for epoch 73 \n",
+ "\n",
+ "5/5 [==============================] - 3s 363ms/step - loss: 0.0615 - accuracy: 0.2309 - val_loss: 0.0621 - val_accuracy: 0.2292\n",
+ "Epoch 74/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0614 - accuracy: 0.2298\n",
+ " (4.713255352027643, 1e-05)-DP guarantees for epoch 74 \n",
+ "\n",
+ "5/5 [==============================] - 3s 392ms/step - loss: 0.0614 - accuracy: 0.2298 - val_loss: 0.0619 - val_accuracy: 0.2273\n",
+ "Epoch 75/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0616 - accuracy: 0.2288\n",
+ " (4.746505714565027, 1e-05)-DP guarantees for epoch 75 \n",
+ "\n",
+ "5/5 [==============================] - 2s 346ms/step - loss: 0.0616 - accuracy: 0.2288 - val_loss: 0.0618 - val_accuracy: 0.2283\n",
+ "Epoch 76/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0613 - accuracy: 0.2314\n",
+ " (4.779756080992392, 1e-05)-DP guarantees for epoch 76 \n",
+ "\n",
+ "5/5 [==============================] - 3s 375ms/step - loss: 0.0613 - accuracy: 0.2314 - val_loss: 0.0617 - val_accuracy: 0.2285\n",
+ "Epoch 77/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0611 - accuracy: 0.2321\n",
+ " (4.813006446042454, 1e-05)-DP guarantees for epoch 77 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0611 - accuracy: 0.2321 - val_loss: 0.0615 - val_accuracy: 0.2279\n",
+ "Epoch 78/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0609 - accuracy: 0.2321\n",
+ " (4.84625681135709, 1e-05)-DP guarantees for epoch 78 \n",
+ "\n",
+ "5/5 [==============================] - 2s 366ms/step - loss: 0.0609 - accuracy: 0.2321 - val_loss: 0.0614 - val_accuracy: 0.2309\n",
+ "Epoch 79/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0608 - accuracy: 0.2326\n",
+ " (4.879507178851574, 1e-05)-DP guarantees for epoch 79 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0608 - accuracy: 0.2326 - val_loss: 0.0613 - val_accuracy: 0.2316\n",
+ "Epoch 80/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0608 - accuracy: 0.2311\n",
+ " (4.912757545677179, 1e-05)-DP guarantees for epoch 80 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0608 - accuracy: 0.2311 - val_loss: 0.0612 - val_accuracy: 0.2311\n",
+ "Epoch 81/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0607 - accuracy: 0.2333\n",
+ " (4.9460079085624, 1e-05)-DP guarantees for epoch 81 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0607 - accuracy: 0.2333 - val_loss: 0.0611 - val_accuracy: 0.2317\n",
+ "Epoch 82/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0607 - accuracy: 0.2341\n",
+ " (4.979258270989774, 1e-05)-DP guarantees for epoch 82 \n",
+ "\n",
+ "5/5 [==============================] - 2s 339ms/step - loss: 0.0607 - accuracy: 0.2341 - val_loss: 0.0610 - val_accuracy: 0.2338\n",
+ "Epoch 83/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0604 - accuracy: 0.2339\n",
+ " (5.012508634818511, 1e-05)-DP guarantees for epoch 83 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0604 - accuracy: 0.2339 - val_loss: 0.0609 - val_accuracy: 0.2318\n",
+ "Epoch 84/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0605 - accuracy: 0.2348\n",
+ " (5.045759003430268, 1e-05)-DP guarantees for epoch 84 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0605 - accuracy: 0.2348 - val_loss: 0.0608 - val_accuracy: 0.2312\n",
+ "Epoch 85/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0603 - accuracy: 0.2332\n",
+ " (5.0790093680054635, 1e-05)-DP guarantees for epoch 85 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0603 - accuracy: 0.2332 - val_loss: 0.0607 - val_accuracy: 0.2326\n",
+ "Epoch 86/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0600 - accuracy: 0.2355\n",
+ " (5.112259736439092, 1e-05)-DP guarantees for epoch 86 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0600 - accuracy: 0.2355 - val_loss: 0.0606 - val_accuracy: 0.2333\n",
+ "Epoch 87/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0600 - accuracy: 0.2357\n",
+ " (5.14551009793596, 1e-05)-DP guarantees for epoch 87 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0600 - accuracy: 0.2357 - val_loss: 0.0604 - val_accuracy: 0.2335\n",
+ "Epoch 88/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0598 - accuracy: 0.2397\n",
+ " (5.178760460033292, 1e-05)-DP guarantees for epoch 88 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0598 - accuracy: 0.2397 - val_loss: 0.0603 - val_accuracy: 0.2327\n",
+ "Epoch 89/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0596 - accuracy: 0.2377\n",
+ " (5.212010824793953, 1e-05)-DP guarantees for epoch 89 \n",
+ "\n",
+ "5/5 [==============================] - 2s 345ms/step - loss: 0.0596 - accuracy: 0.2377 - val_loss: 0.0602 - val_accuracy: 0.2333\n",
+ "Epoch 90/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0597 - accuracy: 0.2372\n",
+ " (5.24526119058743, 1e-05)-DP guarantees for epoch 90 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0597 - accuracy: 0.2372 - val_loss: 0.0601 - val_accuracy: 0.2336\n",
+ "Epoch 91/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0595 - accuracy: 0.2367\n",
+ " (5.278511560314511, 1e-05)-DP guarantees for epoch 91 \n",
+ "\n",
+ "5/5 [==============================] - 2s 361ms/step - loss: 0.0595 - accuracy: 0.2367 - val_loss: 0.0600 - val_accuracy: 0.2331\n",
+ "Epoch 92/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0598 - accuracy: 0.2373\n",
+ " (5.311761920262455, 1e-05)-DP guarantees for epoch 92 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0598 - accuracy: 0.2373 - val_loss: 0.0599 - val_accuracy: 0.2358\n",
+ "Epoch 93/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0594 - accuracy: 0.2368\n",
+ " (5.3450122912656255, 1e-05)-DP guarantees for epoch 93 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0594 - accuracy: 0.2368 - val_loss: 0.0598 - val_accuracy: 0.2346\n",
+ "Epoch 94/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0592 - accuracy: 0.2380\n",
+ " (5.37826264973137, 1e-05)-DP guarantees for epoch 94 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0592 - accuracy: 0.2380 - val_loss: 0.0597 - val_accuracy: 0.2347\n",
+ "Epoch 95/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0593 - accuracy: 0.2357\n",
+ " (5.4115130208687106, 1e-05)-DP guarantees for epoch 95 \n",
+ "\n",
+ "5/5 [==============================] - 2s 360ms/step - loss: 0.0593 - accuracy: 0.2357 - val_loss: 0.0596 - val_accuracy: 0.2348\n",
+ "Epoch 96/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0594 - accuracy: 0.2376\n",
+ " (5.444763387799843, 1e-05)-DP guarantees for epoch 96 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0594 - accuracy: 0.2376 - val_loss: 0.0595 - val_accuracy: 0.2362\n",
+ "Epoch 97/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0589 - accuracy: 0.2411\n",
+ " (5.47801375480832, 1e-05)-DP guarantees for epoch 97 \n",
+ "\n",
+ "5/5 [==============================] - 2s 363ms/step - loss: 0.0589 - accuracy: 0.2411 - val_loss: 0.0594 - val_accuracy: 0.2375\n",
+ "Epoch 98/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0590 - accuracy: 0.2404\n",
+ " (5.511264111964721, 1e-05)-DP guarantees for epoch 98 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0590 - accuracy: 0.2404 - val_loss: 0.0593 - val_accuracy: 0.2377\n",
+ "Epoch 99/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0586 - accuracy: 0.2406\n",
+ " (5.544514479570887, 1e-05)-DP guarantees for epoch 99 \n",
+ "\n",
+ "5/5 [==============================] - 2s 347ms/step - loss: 0.0586 - accuracy: 0.2406 - val_loss: 0.0593 - val_accuracy: 0.2389\n",
+ "Epoch 100/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0587 - accuracy: 0.2436\n",
+ " (5.5777648468507035, 1e-05)-DP guarantees for epoch 100 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0587 - accuracy: 0.2436 - val_loss: 0.0592 - val_accuracy: 0.2383\n",
+ "Epoch 101/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0586 - accuracy: 0.2405\n",
+ " (5.611015209476669, 1e-05)-DP guarantees for epoch 101 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0586 - accuracy: 0.2405 - val_loss: 0.0590 - val_accuracy: 0.2382\n",
+ "Epoch 102/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0586 - accuracy: 0.2409\n",
+ " (5.644265572603777, 1e-05)-DP guarantees for epoch 102 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0586 - accuracy: 0.2409 - val_loss: 0.0589 - val_accuracy: 0.2376\n",
+ "Epoch 103/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0584 - accuracy: 0.2425\n",
+ " (5.67751593629532, 1e-05)-DP guarantees for epoch 103 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0584 - accuracy: 0.2425 - val_loss: 0.0588 - val_accuracy: 0.2397\n",
+ "Epoch 104/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0583 - accuracy: 0.2422\n",
+ " (5.710766303023046, 1e-05)-DP guarantees for epoch 104 \n",
+ "\n",
+ "5/5 [==============================] - 3s 370ms/step - loss: 0.0583 - accuracy: 0.2422 - val_loss: 0.0587 - val_accuracy: 0.2384\n",
+ "Epoch 105/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0582 - accuracy: 0.2425\n",
+ " (5.7440166690784755, 1e-05)-DP guarantees for epoch 105 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0582 - accuracy: 0.2425 - val_loss: 0.0586 - val_accuracy: 0.2383\n",
+ "Epoch 106/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0583 - accuracy: 0.2411\n",
+ " (5.777267031618594, 1e-05)-DP guarantees for epoch 106 \n",
+ "\n",
+ "5/5 [==============================] - 2s 345ms/step - loss: 0.0583 - accuracy: 0.2411 - val_loss: 0.0586 - val_accuracy: 0.2387\n",
+ "Epoch 107/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0578 - accuracy: 0.2438\n",
+ " (5.8105173958576675, 1e-05)-DP guarantees for epoch 107 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0578 - accuracy: 0.2438 - val_loss: 0.0585 - val_accuracy: 0.2409\n",
+ "Epoch 108/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0582 - accuracy: 0.2442\n",
+ " (5.843767765269359, 1e-05)-DP guarantees for epoch 108 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0582 - accuracy: 0.2442 - val_loss: 0.0584 - val_accuracy: 0.2440\n",
+ "Epoch 109/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0578 - accuracy: 0.2456\n",
+ " (5.877018127929281, 1e-05)-DP guarantees for epoch 109 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0578 - accuracy: 0.2456 - val_loss: 0.0584 - val_accuracy: 0.2419\n",
+ "Epoch 110/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0580 - accuracy: 0.2440\n",
+ " (5.910268490844311, 1e-05)-DP guarantees for epoch 110 \n",
+ "\n",
+ "5/5 [==============================] - 2s 362ms/step - loss: 0.0580 - accuracy: 0.2440 - val_loss: 0.0583 - val_accuracy: 0.2429\n",
+ "Epoch 111/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0578 - accuracy: 0.2473\n",
+ " (5.943518855328065, 1e-05)-DP guarantees for epoch 111 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0578 - accuracy: 0.2473 - val_loss: 0.0583 - val_accuracy: 0.2448\n",
+ "Epoch 112/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0577 - accuracy: 0.2469\n",
+ " (5.9767692222925275, 1e-05)-DP guarantees for epoch 112 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0577 - accuracy: 0.2469 - val_loss: 0.0582 - val_accuracy: 0.2447\n",
+ "Epoch 113/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0579 - accuracy: 0.2479\n",
+ " (6.0100195891034165, 1e-05)-DP guarantees for epoch 113 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0579 - accuracy: 0.2479 - val_loss: 0.0581 - val_accuracy: 0.2453\n",
+ "Epoch 114/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0576 - accuracy: 0.2468\n",
+ " (6.043269950764723, 1e-05)-DP guarantees for epoch 114 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0576 - accuracy: 0.2468 - val_loss: 0.0580 - val_accuracy: 0.2432\n",
+ "Epoch 115/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0574 - accuracy: 0.2472\n",
+ " (6.076520315246205, 1e-05)-DP guarantees for epoch 115 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0574 - accuracy: 0.2472 - val_loss: 0.0579 - val_accuracy: 0.2441\n",
+ "Epoch 116/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0573 - accuracy: 0.2476\n",
+ " (6.109770681686705, 1e-05)-DP guarantees for epoch 116 \n",
+ "\n",
+ "5/5 [==============================] - 2s 363ms/step - loss: 0.0573 - accuracy: 0.2476 - val_loss: 0.0579 - val_accuracy: 0.2440\n",
+ "Epoch 117/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0575 - accuracy: 0.2470\n",
+ " (6.143021045607053, 1e-05)-DP guarantees for epoch 117 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0575 - accuracy: 0.2470 - val_loss: 0.0578 - val_accuracy: 0.2479\n",
+ "Epoch 118/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0572 - accuracy: 0.2481\n",
+ " (6.1762714106501475, 1e-05)-DP guarantees for epoch 118 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0572 - accuracy: 0.2481 - val_loss: 0.0576 - val_accuracy: 0.2450\n",
+ "Epoch 119/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0572 - accuracy: 0.2500\n",
+ " (6.209521499901805, 1e-05)-DP guarantees for epoch 119 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0572 - accuracy: 0.2500 - val_loss: 0.0576 - val_accuracy: 0.2446\n",
+ "Epoch 120/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0569 - accuracy: 0.2497\n",
+ " (6.241605627485653, 1e-05)-DP guarantees for epoch 120 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0569 - accuracy: 0.2497 - val_loss: 0.0575 - val_accuracy: 0.2451\n",
+ "Epoch 121/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0569 - accuracy: 0.2510\n",
+ " (6.271221812058615, 1e-05)-DP guarantees for epoch 121 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0569 - accuracy: 0.2510 - val_loss: 0.0574 - val_accuracy: 0.2445\n",
+ "Epoch 122/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0571 - accuracy: 0.2481\n",
+ " (6.298196491974402, 1e-05)-DP guarantees for epoch 122 \n",
+ "\n",
+ "5/5 [==============================] - 2s 359ms/step - loss: 0.0571 - accuracy: 0.2481 - val_loss: 0.0574 - val_accuracy: 0.2447\n",
+ "Epoch 123/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0568 - accuracy: 0.2517\n",
+ " (6.324510712314491, 1e-05)-DP guarantees for epoch 123 \n",
+ "\n",
+ "5/5 [==============================] - 2s 345ms/step - loss: 0.0568 - accuracy: 0.2517 - val_loss: 0.0573 - val_accuracy: 0.2481\n",
+ "Epoch 124/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0570 - accuracy: 0.2505\n",
+ " (6.350824932887864, 1e-05)-DP guarantees for epoch 124 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0570 - accuracy: 0.2505 - val_loss: 0.0573 - val_accuracy: 0.2449\n",
+ "Epoch 125/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0567 - accuracy: 0.2489\n",
+ " (6.377139153079873, 1e-05)-DP guarantees for epoch 125 \n",
+ "\n",
+ "5/5 [==============================] - 2s 368ms/step - loss: 0.0567 - accuracy: 0.2489 - val_loss: 0.0572 - val_accuracy: 0.2450\n",
+ "Epoch 126/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0570 - accuracy: 0.2488\n",
+ " (6.403453374888347, 1e-05)-DP guarantees for epoch 126 \n",
+ "\n",
+ "5/5 [==============================] - 3s 349ms/step - loss: 0.0570 - accuracy: 0.2488 - val_loss: 0.0572 - val_accuracy: 0.2485\n",
+ "Epoch 127/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0566 - accuracy: 0.2539\n",
+ " (6.429767596763488, 1e-05)-DP guarantees for epoch 127 \n",
+ "\n",
+ "5/5 [==============================] - 3s 391ms/step - loss: 0.0566 - accuracy: 0.2539 - val_loss: 0.0571 - val_accuracy: 0.2452\n",
+ "Epoch 128/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0565 - accuracy: 0.2505\n",
+ " (6.4560818158974875, 1e-05)-DP guarantees for epoch 128 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0565 - accuracy: 0.2505 - val_loss: 0.0570 - val_accuracy: 0.2466\n",
+ "Epoch 129/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0566 - accuracy: 0.2522\n",
+ " (6.482396036898421, 1e-05)-DP guarantees for epoch 129 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0566 - accuracy: 0.2522 - val_loss: 0.0570 - val_accuracy: 0.2461\n",
+ "Epoch 130/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0561 - accuracy: 0.2521\n",
+ " (6.5087102545452, 1e-05)-DP guarantees for epoch 130 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0561 - accuracy: 0.2521 - val_loss: 0.0569 - val_accuracy: 0.2468\n",
+ "Epoch 131/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0562 - accuracy: 0.2534\n",
+ " (6.53502447810436, 1e-05)-DP guarantees for epoch 131 \n",
+ "\n",
+ "5/5 [==============================] - 2s 374ms/step - loss: 0.0562 - accuracy: 0.2534 - val_loss: 0.0569 - val_accuracy: 0.2470\n",
+ "Epoch 132/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0563 - accuracy: 0.2530\n",
+ " (6.5613386977335715, 1e-05)-DP guarantees for epoch 132 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0563 - accuracy: 0.2530 - val_loss: 0.0568 - val_accuracy: 0.2501\n",
+ "Epoch 133/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0561 - accuracy: 0.2564\n",
+ " (6.587652915827986, 1e-05)-DP guarantees for epoch 133 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0561 - accuracy: 0.2564 - val_loss: 0.0569 - val_accuracy: 0.2470\n",
+ "Epoch 134/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0561 - accuracy: 0.2555\n",
+ " (6.613967135260202, 1e-05)-DP guarantees for epoch 134 \n",
+ "\n",
+ "5/5 [==============================] - 3s 402ms/step - loss: 0.0561 - accuracy: 0.2555 - val_loss: 0.0568 - val_accuracy: 0.2492\n",
+ "Epoch 135/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0564 - accuracy: 0.2535\n",
+ " (6.6402813578423405, 1e-05)-DP guarantees for epoch 135 \n",
+ "\n",
+ "5/5 [==============================] - 2s 347ms/step - loss: 0.0564 - accuracy: 0.2535 - val_loss: 0.0567 - val_accuracy: 0.2499\n",
+ "Epoch 136/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0559 - accuracy: 0.2552\n",
+ " (6.666595582737012, 1e-05)-DP guarantees for epoch 136 \n",
+ "\n",
+ "5/5 [==============================] - 2s 360ms/step - loss: 0.0559 - accuracy: 0.2552 - val_loss: 0.0567 - val_accuracy: 0.2506\n",
+ "Epoch 137/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2562\n",
+ " (6.692909796982604, 1e-05)-DP guarantees for epoch 137 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0560 - accuracy: 0.2562 - val_loss: 0.0566 - val_accuracy: 0.2484\n",
+ "Epoch 138/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2538\n",
+ " (6.719224016310403, 1e-05)-DP guarantees for epoch 138 \n",
+ "\n",
+ "5/5 [==============================] - 2s 349ms/step - loss: 0.0560 - accuracy: 0.2538 - val_loss: 0.0565 - val_accuracy: 0.2471\n",
+ "Epoch 139/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2526\n",
+ " (6.74553823900151, 1e-05)-DP guarantees for epoch 139 \n",
+ "\n",
+ "5/5 [==============================] - 3s 399ms/step - loss: 0.0560 - accuracy: 0.2526 - val_loss: 0.0565 - val_accuracy: 0.2509\n",
+ "Epoch 140/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2536\n",
+ " (6.771852459824933, 1e-05)-DP guarantees for epoch 140 \n",
+ "\n",
+ "5/5 [==============================] - 3s 493ms/step - loss: 0.0560 - accuracy: 0.2536 - val_loss: 0.0564 - val_accuracy: 0.2493\n",
+ "Epoch 141/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0557 - accuracy: 0.2555\n",
+ " (6.798166680154963, 1e-05)-DP guarantees for epoch 141 \n",
+ "\n",
+ "5/5 [==============================] - 3s 391ms/step - loss: 0.0557 - accuracy: 0.2555 - val_loss: 0.0563 - val_accuracy: 0.2511\n",
+ "Epoch 142/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0559 - accuracy: 0.2541\n",
+ " (6.824480898392123, 1e-05)-DP guarantees for epoch 142 \n",
+ "\n",
+ "5/5 [==============================] - 3s 443ms/step - loss: 0.0559 - accuracy: 0.2541 - val_loss: 0.0563 - val_accuracy: 0.2484\n",
+ "Epoch 143/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0560 - accuracy: 0.2547\n",
+ " (6.850795124433479, 1e-05)-DP guarantees for epoch 143 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0560 - accuracy: 0.2547 - val_loss: 0.0563 - val_accuracy: 0.2487\n",
+ "Epoch 144/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0556 - accuracy: 0.2545\n",
+ " (6.877109344205954, 1e-05)-DP guarantees for epoch 144 \n",
+ "\n",
+ "5/5 [==============================] - 3s 374ms/step - loss: 0.0556 - accuracy: 0.2545 - val_loss: 0.0562 - val_accuracy: 0.2487\n",
+ "Epoch 145/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0555 - accuracy: 0.2569\n",
+ " (6.903423558068683, 1e-05)-DP guarantees for epoch 145 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0555 - accuracy: 0.2569 - val_loss: 0.0562 - val_accuracy: 0.2508\n",
+ "Epoch 146/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0558 - accuracy: 0.2560\n",
+ " (6.929737777126363, 1e-05)-DP guarantees for epoch 146 \n",
+ "\n",
+ "5/5 [==============================] - 3s 387ms/step - loss: 0.0558 - accuracy: 0.2560 - val_loss: 0.0561 - val_accuracy: 0.2504\n",
+ "Epoch 147/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0557 - accuracy: 0.2556\n",
+ " (6.956052008535497, 1e-05)-DP guarantees for epoch 147 \n",
+ "\n",
+ "5/5 [==============================] - 3s 372ms/step - loss: 0.0557 - accuracy: 0.2556 - val_loss: 0.0561 - val_accuracy: 0.2509\n",
+ "Epoch 148/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0557 - accuracy: 0.2538\n",
+ " (6.982366223228706, 1e-05)-DP guarantees for epoch 148 \n",
+ "\n",
+ "5/5 [==============================] - 3s 381ms/step - loss: 0.0557 - accuracy: 0.2538 - val_loss: 0.0561 - val_accuracy: 0.2528\n",
+ "Epoch 149/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0553 - accuracy: 0.2580\n",
+ " (7.0086804403647855, 1e-05)-DP guarantees for epoch 149 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0553 - accuracy: 0.2580 - val_loss: 0.0560 - val_accuracy: 0.2530\n",
+ "Epoch 150/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0549 - accuracy: 0.2595\n",
+ " (7.034994664689931, 1e-05)-DP guarantees for epoch 150 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0549 - accuracy: 0.2595 - val_loss: 0.0560 - val_accuracy: 0.2519\n",
+ "Epoch 151/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0554 - accuracy: 0.2585\n",
+ " (7.061308885525292, 1e-05)-DP guarantees for epoch 151 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0554 - accuracy: 0.2585 - val_loss: 0.0559 - val_accuracy: 0.2531\n",
+ "Epoch 152/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0554 - accuracy: 0.2580\n",
+ " (7.087623106633284, 1e-05)-DP guarantees for epoch 152 \n",
+ "\n",
+ "5/5 [==============================] - 2s 355ms/step - loss: 0.0554 - accuracy: 0.2580 - val_loss: 0.0558 - val_accuracy: 0.2543\n",
+ "Epoch 153/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0553 - accuracy: 0.2585\n",
+ " (7.113937323136563, 1e-05)-DP guarantees for epoch 153 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0553 - accuracy: 0.2585 - val_loss: 0.0558 - val_accuracy: 0.2537\n",
+ "Epoch 154/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0551 - accuracy: 0.2595\n",
+ " (7.140251544398778, 1e-05)-DP guarantees for epoch 154 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0551 - accuracy: 0.2595 - val_loss: 0.0558 - val_accuracy: 0.2551\n",
+ "Epoch 155/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0550 - accuracy: 0.2600\n",
+ " (7.166565767658498, 1e-05)-DP guarantees for epoch 155 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0550 - accuracy: 0.2600 - val_loss: 0.0557 - val_accuracy: 0.2569\n",
+ "Epoch 156/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0553 - accuracy: 0.2561\n",
+ " (7.192879981310637, 1e-05)-DP guarantees for epoch 156 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0553 - accuracy: 0.2561 - val_loss: 0.0556 - val_accuracy: 0.2545\n",
+ "Epoch 157/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0550 - accuracy: 0.2581\n",
+ " (7.2191942080187195, 1e-05)-DP guarantees for epoch 157 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0550 - accuracy: 0.2581 - val_loss: 0.0556 - val_accuracy: 0.2566\n",
+ "Epoch 158/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0550 - accuracy: 0.2601\n",
+ " (7.245508431022666, 1e-05)-DP guarantees for epoch 158 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0550 - accuracy: 0.2601 - val_loss: 0.0556 - val_accuracy: 0.2574\n",
+ "Epoch 159/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0548 - accuracy: 0.2599\n",
+ " (7.27182264840541, 1e-05)-DP guarantees for epoch 159 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0548 - accuracy: 0.2599 - val_loss: 0.0555 - val_accuracy: 0.2567\n",
+ "Epoch 160/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0548 - accuracy: 0.2616\n",
+ " (7.298136867745498, 1e-05)-DP guarantees for epoch 160 \n",
+ "\n",
+ "5/5 [==============================] - 2s 367ms/step - loss: 0.0548 - accuracy: 0.2616 - val_loss: 0.0554 - val_accuracy: 0.2560\n",
+ "Epoch 161/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0551 - accuracy: 0.2595\n",
+ " (7.324451088022072, 1e-05)-DP guarantees for epoch 161 \n",
+ "\n",
+ "5/5 [==============================] - 3s 349ms/step - loss: 0.0551 - accuracy: 0.2595 - val_loss: 0.0554 - val_accuracy: 0.2577\n",
+ "Epoch 162/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0548 - accuracy: 0.2606\n",
+ " (7.350765305854425, 1e-05)-DP guarantees for epoch 162 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0548 - accuracy: 0.2606 - val_loss: 0.0554 - val_accuracy: 0.2580\n",
+ "Epoch 163/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0547 - accuracy: 0.2588\n",
+ " (7.37707952170881, 1e-05)-DP guarantees for epoch 163 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0547 - accuracy: 0.2588 - val_loss: 0.0553 - val_accuracy: 0.2549\n",
+ "Epoch 164/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0546 - accuracy: 0.2585\n",
+ " (7.403393741099066, 1e-05)-DP guarantees for epoch 164 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0546 - accuracy: 0.2585 - val_loss: 0.0553 - val_accuracy: 0.2591\n",
+ "Epoch 165/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0546 - accuracy: 0.2607\n",
+ " (7.429707969366283, 1e-05)-DP guarantees for epoch 165 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0546 - accuracy: 0.2607 - val_loss: 0.0552 - val_accuracy: 0.2574\n",
+ "Epoch 166/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0547 - accuracy: 0.2598\n",
+ " (7.456022189620042, 1e-05)-DP guarantees for epoch 166 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0547 - accuracy: 0.2598 - val_loss: 0.0551 - val_accuracy: 0.2544\n",
+ "Epoch 167/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0544 - accuracy: 0.2589\n",
+ " (7.4823364015791975, 1e-05)-DP guarantees for epoch 167 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0544 - accuracy: 0.2589 - val_loss: 0.0552 - val_accuracy: 0.2570\n",
+ "Epoch 168/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0546 - accuracy: 0.2620\n",
+ " (7.508650622437409, 1e-05)-DP guarantees for epoch 168 \n",
+ "\n",
+ "5/5 [==============================] - 2s 343ms/step - loss: 0.0546 - accuracy: 0.2620 - val_loss: 0.0551 - val_accuracy: 0.2585\n",
+ "Epoch 169/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0544 - accuracy: 0.2609\n",
+ " (7.5349648424170645, 1e-05)-DP guarantees for epoch 169 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0544 - accuracy: 0.2609 - val_loss: 0.0550 - val_accuracy: 0.2591\n",
+ "Epoch 170/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0545 - accuracy: 0.2618\n",
+ " (7.561279065737033, 1e-05)-DP guarantees for epoch 170 \n",
+ "\n",
+ "5/5 [==============================] - 3s 369ms/step - loss: 0.0545 - accuracy: 0.2618 - val_loss: 0.0551 - val_accuracy: 0.2582\n",
+ "Epoch 171/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0542 - accuracy: 0.2642\n",
+ " (7.587593290867159, 1e-05)-DP guarantees for epoch 171 \n",
+ "\n",
+ "5/5 [==============================] - 3s 372ms/step - loss: 0.0542 - accuracy: 0.2642 - val_loss: 0.0551 - val_accuracy: 0.2598\n",
+ "Epoch 172/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0543 - accuracy: 0.2640\n",
+ " (7.613907506714526, 1e-05)-DP guarantees for epoch 172 \n",
+ "\n",
+ "5/5 [==============================] - 3s 369ms/step - loss: 0.0543 - accuracy: 0.2640 - val_loss: 0.0550 - val_accuracy: 0.2604\n",
+ "Epoch 173/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0543 - accuracy: 0.2642\n",
+ " (7.640221723584304, 1e-05)-DP guarantees for epoch 173 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0543 - accuracy: 0.2642 - val_loss: 0.0549 - val_accuracy: 0.2604\n",
+ "Epoch 174/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0542 - accuracy: 0.2635\n",
+ " (7.666535950048996, 1e-05)-DP guarantees for epoch 174 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0542 - accuracy: 0.2635 - val_loss: 0.0549 - val_accuracy: 0.2628\n",
+ "Epoch 175/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0541 - accuracy: 0.2648\n",
+ " (7.692850164248792, 1e-05)-DP guarantees for epoch 175 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0541 - accuracy: 0.2648 - val_loss: 0.0548 - val_accuracy: 0.2625\n",
+ "Epoch 176/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0542 - accuracy: 0.2637\n",
+ " (7.719164393302542, 1e-05)-DP guarantees for epoch 176 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0542 - accuracy: 0.2637 - val_loss: 0.0547 - val_accuracy: 0.2621\n",
+ "Epoch 177/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0540 - accuracy: 0.2661\n",
+ " (7.745478613553454, 1e-05)-DP guarantees for epoch 177 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0540 - accuracy: 0.2661 - val_loss: 0.0546 - val_accuracy: 0.2665\n",
+ "Epoch 178/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0541 - accuracy: 0.2668\n",
+ " (7.771792822684058, 1e-05)-DP guarantees for epoch 178 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0541 - accuracy: 0.2668 - val_loss: 0.0546 - val_accuracy: 0.2659\n",
+ "Epoch 179/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0539 - accuracy: 0.2685\n",
+ " (7.7981070469012, 1e-05)-DP guarantees for epoch 179 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0539 - accuracy: 0.2685 - val_loss: 0.0545 - val_accuracy: 0.2646\n",
+ "Epoch 180/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0538 - accuracy: 0.2682\n",
+ " (7.824421268798268, 1e-05)-DP guarantees for epoch 180 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0538 - accuracy: 0.2682 - val_loss: 0.0545 - val_accuracy: 0.2656\n",
+ "Epoch 181/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0538 - accuracy: 0.2671\n",
+ " (7.850735498247861, 1e-05)-DP guarantees for epoch 181 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0538 - accuracy: 0.2671 - val_loss: 0.0545 - val_accuracy: 0.2639\n",
+ "Epoch 182/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0539 - accuracy: 0.2661\n",
+ " (7.877049711425853, 1e-05)-DP guarantees for epoch 182 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0539 - accuracy: 0.2661 - val_loss: 0.0544 - val_accuracy: 0.2645\n",
+ "Epoch 183/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0537 - accuracy: 0.2635\n",
+ " (7.903363929529842, 1e-05)-DP guarantees for epoch 183 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0537 - accuracy: 0.2635 - val_loss: 0.0544 - val_accuracy: 0.2645\n",
+ "Epoch 184/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0538 - accuracy: 0.2641\n",
+ " (7.929678153587394, 1e-05)-DP guarantees for epoch 184 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0538 - accuracy: 0.2641 - val_loss: 0.0543 - val_accuracy: 0.2641\n",
+ "Epoch 185/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2668\n",
+ " (7.955992381685565, 1e-05)-DP guarantees for epoch 185 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0535 - accuracy: 0.2668 - val_loss: 0.0543 - val_accuracy: 0.2638\n",
+ "Epoch 186/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2641\n",
+ " (7.982306589145621, 1e-05)-DP guarantees for epoch 186 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0535 - accuracy: 0.2641 - val_loss: 0.0543 - val_accuracy: 0.2654\n",
+ "Epoch 187/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0537 - accuracy: 0.2653\n",
+ " (8.008620808026855, 1e-05)-DP guarantees for epoch 187 \n",
+ "\n",
+ "5/5 [==============================] - 3s 412ms/step - loss: 0.0537 - accuracy: 0.2653 - val_loss: 0.0542 - val_accuracy: 0.2651\n",
+ "Epoch 188/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2656\n",
+ " (8.034935029136395, 1e-05)-DP guarantees for epoch 188 \n",
+ "\n",
+ "5/5 [==============================] - 3s 488ms/step - loss: 0.0535 - accuracy: 0.2656 - val_loss: 0.0542 - val_accuracy: 0.2662\n",
+ "Epoch 189/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0536 - accuracy: 0.2653\n",
+ " (8.061249248434443, 1e-05)-DP guarantees for epoch 189 \n",
+ "\n",
+ "5/5 [==============================] - 3s 444ms/step - loss: 0.0536 - accuracy: 0.2653 - val_loss: 0.0541 - val_accuracy: 0.2659\n",
+ "Epoch 190/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2676\n",
+ " (8.087563469816706, 1e-05)-DP guarantees for epoch 190 \n",
+ "\n",
+ "5/5 [==============================] - 3s 405ms/step - loss: 0.0533 - accuracy: 0.2676 - val_loss: 0.0541 - val_accuracy: 0.2663\n",
+ "Epoch 191/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2669\n",
+ " (8.113877688170744, 1e-05)-DP guarantees for epoch 191 \n",
+ "\n",
+ "5/5 [==============================] - 3s 385ms/step - loss: 0.0534 - accuracy: 0.2669 - val_loss: 0.0541 - val_accuracy: 0.2675\n",
+ "Epoch 192/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2648\n",
+ " (8.140191906358039, 1e-05)-DP guarantees for epoch 192 \n",
+ "\n",
+ "5/5 [==============================] - 3s 392ms/step - loss: 0.0535 - accuracy: 0.2648 - val_loss: 0.0540 - val_accuracy: 0.2676\n",
+ "Epoch 193/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2680\n",
+ " (8.166506132866681, 1e-05)-DP guarantees for epoch 193 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0534 - accuracy: 0.2680 - val_loss: 0.0540 - val_accuracy: 0.2676\n",
+ "Epoch 194/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2654\n",
+ " (8.192820350846777, 1e-05)-DP guarantees for epoch 194 \n",
+ "\n",
+ "5/5 [==============================] - 2s 356ms/step - loss: 0.0533 - accuracy: 0.2654 - val_loss: 0.0540 - val_accuracy: 0.2679\n",
+ "Epoch 195/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.2681\n",
+ " (8.219134573417037, 1e-05)-DP guarantees for epoch 195 \n",
+ "\n",
+ "5/5 [==============================] - 3s 381ms/step - loss: 0.0531 - accuracy: 0.2681 - val_loss: 0.0541 - val_accuracy: 0.2654\n",
+ "Epoch 196/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0532 - accuracy: 0.2671\n",
+ " (8.24544879099129, 1e-05)-DP guarantees for epoch 196 \n",
+ "\n",
+ "5/5 [==============================] - 3s 381ms/step - loss: 0.0532 - accuracy: 0.2671 - val_loss: 0.0540 - val_accuracy: 0.2658\n",
+ "Epoch 197/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0535 - accuracy: 0.2666\n",
+ " (8.271763016196239, 1e-05)-DP guarantees for epoch 197 \n",
+ "\n",
+ "5/5 [==============================] - 3s 389ms/step - loss: 0.0535 - accuracy: 0.2666 - val_loss: 0.0540 - val_accuracy: 0.2656\n",
+ "Epoch 198/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2676\n",
+ " (8.298077232897459, 1e-05)-DP guarantees for epoch 198 \n",
+ "\n",
+ "5/5 [==============================] - 3s 415ms/step - loss: 0.0534 - accuracy: 0.2676 - val_loss: 0.0539 - val_accuracy: 0.2656\n",
+ "Epoch 199/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.2672\n",
+ " (8.324391446543665, 1e-05)-DP guarantees for epoch 199 \n",
+ "\n",
+ "5/5 [==============================] - 3s 380ms/step - loss: 0.0531 - accuracy: 0.2672 - val_loss: 0.0538 - val_accuracy: 0.2658\n",
+ "Epoch 200/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.2638\n",
+ " (8.350705669706155, 1e-05)-DP guarantees for epoch 200 \n",
+ "\n",
+ "5/5 [==============================] - 2s 358ms/step - loss: 0.0534 - accuracy: 0.2638 - val_loss: 0.0538 - val_accuracy: 0.2659\n",
+ "Epoch 201/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2672\n",
+ " (8.377019893272927, 1e-05)-DP guarantees for epoch 201 \n",
+ "\n",
+ "5/5 [==============================] - 2s 369ms/step - loss: 0.0533 - accuracy: 0.2672 - val_loss: 0.0538 - val_accuracy: 0.2678\n",
+ "Epoch 202/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0533 - accuracy: 0.2685\n",
+ " (8.403334112768452, 1e-05)-DP guarantees for epoch 202 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0533 - accuracy: 0.2685 - val_loss: 0.0537 - val_accuracy: 0.2677\n",
+ "Epoch 203/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0528 - accuracy: 0.2701\n",
+ " (8.429648329088547, 1e-05)-DP guarantees for epoch 203 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0528 - accuracy: 0.2701 - val_loss: 0.0537 - val_accuracy: 0.2669\n",
+ "Epoch 204/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0528 - accuracy: 0.2696\n",
+ " (8.455962556505566, 1e-05)-DP guarantees for epoch 204 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0528 - accuracy: 0.2696 - val_loss: 0.0537 - val_accuracy: 0.2681\n",
+ "Epoch 205/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0532 - accuracy: 0.2691\n",
+ " (8.4822767745793, 1e-05)-DP guarantees for epoch 205 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0532 - accuracy: 0.2691 - val_loss: 0.0536 - val_accuracy: 0.2692\n",
+ "Epoch 206/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.2703\n",
+ " (8.508590990133396, 1e-05)-DP guarantees for epoch 206 \n",
+ "\n",
+ "5/5 [==============================] - 3s 354ms/step - loss: 0.0531 - accuracy: 0.2703 - val_loss: 0.0535 - val_accuracy: 0.2683\n",
+ "Epoch 207/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0529 - accuracy: 0.2705\n",
+ " (8.534905221654196, 1e-05)-DP guarantees for epoch 207 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0529 - accuracy: 0.2705 - val_loss: 0.0535 - val_accuracy: 0.2661\n",
+ "Epoch 208/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0526 - accuracy: 0.2726\n",
+ " (8.56121943210842, 1e-05)-DP guarantees for epoch 208 \n",
+ "\n",
+ "5/5 [==============================] - 2s 351ms/step - loss: 0.0526 - accuracy: 0.2726 - val_loss: 0.0535 - val_accuracy: 0.2671\n",
+ "Epoch 209/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0530 - accuracy: 0.2703\n",
+ " (8.58753364829852, 1e-05)-DP guarantees for epoch 209 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0530 - accuracy: 0.2703 - val_loss: 0.0534 - val_accuracy: 0.2691\n",
+ "Epoch 210/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0527 - accuracy: 0.2701\n",
+ " (8.613847875406321, 1e-05)-DP guarantees for epoch 210 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0527 - accuracy: 0.2701 - val_loss: 0.0534 - val_accuracy: 0.2676\n",
+ "Epoch 211/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2713\n",
+ " (8.640162093797892, 1e-05)-DP guarantees for epoch 211 \n",
+ "\n",
+ "5/5 [==============================] - 3s 350ms/step - loss: 0.0525 - accuracy: 0.2713 - val_loss: 0.0534 - val_accuracy: 0.2689\n",
+ "Epoch 212/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0527 - accuracy: 0.2711\n",
+ " (8.666476313556027, 1e-05)-DP guarantees for epoch 212 \n",
+ "\n",
+ "5/5 [==============================] - 2s 346ms/step - loss: 0.0527 - accuracy: 0.2711 - val_loss: 0.0533 - val_accuracy: 0.2679\n",
+ "Epoch 213/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0524 - accuracy: 0.2722\n",
+ " (8.692790541337777, 1e-05)-DP guarantees for epoch 213 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0524 - accuracy: 0.2722 - val_loss: 0.0532 - val_accuracy: 0.2673\n",
+ "Epoch 214/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0526 - accuracy: 0.2705\n",
+ " (8.719104752659717, 1e-05)-DP guarantees for epoch 214 \n",
+ "\n",
+ "5/5 [==============================] - 3s 355ms/step - loss: 0.0526 - accuracy: 0.2705 - val_loss: 0.0532 - val_accuracy: 0.2675\n",
+ "Epoch 215/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0523 - accuracy: 0.2729\n",
+ " (8.745418971706883, 1e-05)-DP guarantees for epoch 215 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0523 - accuracy: 0.2729 - val_loss: 0.0532 - val_accuracy: 0.2674\n",
+ "Epoch 216/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2733\n",
+ " (8.77173318977154, 1e-05)-DP guarantees for epoch 216 \n",
+ "\n",
+ "5/5 [==============================] - 3s 362ms/step - loss: 0.0525 - accuracy: 0.2733 - val_loss: 0.0532 - val_accuracy: 0.2662\n",
+ "Epoch 217/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2719\n",
+ " (8.798047413801395, 1e-05)-DP guarantees for epoch 217 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0525 - accuracy: 0.2719 - val_loss: 0.0531 - val_accuracy: 0.2676\n",
+ "Epoch 218/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0524 - accuracy: 0.2741\n",
+ " (8.82436163499698, 1e-05)-DP guarantees for epoch 218 \n",
+ "\n",
+ "5/5 [==============================] - 2s 348ms/step - loss: 0.0524 - accuracy: 0.2741 - val_loss: 0.0531 - val_accuracy: 0.2671\n",
+ "Epoch 219/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2702\n",
+ " (8.850675857122916, 1e-05)-DP guarantees for epoch 219 \n",
+ "\n",
+ "5/5 [==============================] - 3s 386ms/step - loss: 0.0525 - accuracy: 0.2702 - val_loss: 0.0531 - val_accuracy: 0.2672\n",
+ "Epoch 220/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0527 - accuracy: 0.2709\n",
+ " (8.876990076626331, 1e-05)-DP guarantees for epoch 220 \n",
+ "\n",
+ "5/5 [==============================] - 3s 376ms/step - loss: 0.0527 - accuracy: 0.2709 - val_loss: 0.0531 - val_accuracy: 0.2668\n",
+ "Epoch 221/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0525 - accuracy: 0.2715\n",
+ " (8.903304291167267, 1e-05)-DP guarantees for epoch 221 \n",
+ "\n",
+ "5/5 [==============================] - 3s 363ms/step - loss: 0.0525 - accuracy: 0.2715 - val_loss: 0.0531 - val_accuracy: 0.2661\n",
+ "Epoch 222/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0524 - accuracy: 0.2721\n",
+ " (8.929618511328595, 1e-05)-DP guarantees for epoch 222 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0524 - accuracy: 0.2721 - val_loss: 0.0530 - val_accuracy: 0.2677\n",
+ "Epoch 223/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2726\n",
+ " (8.955932731489924, 1e-05)-DP guarantees for epoch 223 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0521 - accuracy: 0.2726 - val_loss: 0.0530 - val_accuracy: 0.2686\n",
+ "Epoch 224/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2727\n",
+ " (8.982246951651252, 1e-05)-DP guarantees for epoch 224 \n",
+ "\n",
+ "5/5 [==============================] - 2s 350ms/step - loss: 0.0521 - accuracy: 0.2727 - val_loss: 0.0530 - val_accuracy: 0.2690\n",
+ "Epoch 225/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0522 - accuracy: 0.2706\n",
+ " (9.00856117181258, 1e-05)-DP guarantees for epoch 225 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0522 - accuracy: 0.2706 - val_loss: 0.0529 - val_accuracy: 0.2701\n",
+ "Epoch 226/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2715\n",
+ " (9.034875391973909, 1e-05)-DP guarantees for epoch 226 \n",
+ "\n",
+ "5/5 [==============================] - 3s 396ms/step - loss: 0.0521 - accuracy: 0.2715 - val_loss: 0.0529 - val_accuracy: 0.2690\n",
+ "Epoch 227/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0521 - accuracy: 0.2713\n",
+ " (9.061189612135239, 1e-05)-DP guarantees for epoch 227 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0521 - accuracy: 0.2713 - val_loss: 0.0529 - val_accuracy: 0.2702\n",
+ "Epoch 228/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0520 - accuracy: 0.2729\n",
+ " (9.087503832296568, 1e-05)-DP guarantees for epoch 228 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0520 - accuracy: 0.2729 - val_loss: 0.0529 - val_accuracy: 0.2698\n",
+ "Epoch 229/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2717\n",
+ " (9.113818052457898, 1e-05)-DP guarantees for epoch 229 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0518 - accuracy: 0.2717 - val_loss: 0.0528 - val_accuracy: 0.2704\n",
+ "Epoch 230/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2710\n",
+ " (9.140132272619226, 1e-05)-DP guarantees for epoch 230 \n",
+ "\n",
+ "5/5 [==============================] - 3s 371ms/step - loss: 0.0518 - accuracy: 0.2710 - val_loss: 0.0528 - val_accuracy: 0.2693\n",
+ "Epoch 231/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0519 - accuracy: 0.2725\n",
+ " (9.166446492780555, 1e-05)-DP guarantees for epoch 231 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0519 - accuracy: 0.2725 - val_loss: 0.0528 - val_accuracy: 0.2674\n",
+ "Epoch 232/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2743\n",
+ " (9.192760712941883, 1e-05)-DP guarantees for epoch 232 \n",
+ "\n",
+ "5/5 [==============================] - 3s 361ms/step - loss: 0.0517 - accuracy: 0.2743 - val_loss: 0.0528 - val_accuracy: 0.2685\n",
+ "Epoch 233/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0519 - accuracy: 0.2712\n",
+ " (9.219074933103212, 1e-05)-DP guarantees for epoch 233 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0519 - accuracy: 0.2712 - val_loss: 0.0527 - val_accuracy: 0.2687\n",
+ "Epoch 234/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2731\n",
+ " (9.24538915326454, 1e-05)-DP guarantees for epoch 234 \n",
+ "\n",
+ "5/5 [==============================] - 3s 370ms/step - loss: 0.0517 - accuracy: 0.2731 - val_loss: 0.0527 - val_accuracy: 0.2663\n",
+ "Epoch 235/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2720\n",
+ " (9.27170337342587, 1e-05)-DP guarantees for epoch 235 \n",
+ "\n",
+ "5/5 [==============================] - 3s 350ms/step - loss: 0.0518 - accuracy: 0.2720 - val_loss: 0.0527 - val_accuracy: 0.2663\n",
+ "Epoch 236/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2729\n",
+ " (9.298017593587199, 1e-05)-DP guarantees for epoch 236 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0515 - accuracy: 0.2729 - val_loss: 0.0526 - val_accuracy: 0.2658\n",
+ "Epoch 237/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2726\n",
+ " (9.324331813748529, 1e-05)-DP guarantees for epoch 237 \n",
+ "\n",
+ "5/5 [==============================] - 2s 353ms/step - loss: 0.0517 - accuracy: 0.2726 - val_loss: 0.0526 - val_accuracy: 0.2650\n",
+ "Epoch 238/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2752\n",
+ " (9.350646033909857, 1e-05)-DP guarantees for epoch 238 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0515 - accuracy: 0.2752 - val_loss: 0.0525 - val_accuracy: 0.2655\n",
+ "Epoch 239/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2739\n",
+ " (9.376960254071186, 1e-05)-DP guarantees for epoch 239 \n",
+ "\n",
+ "5/5 [==============================] - 3s 368ms/step - loss: 0.0517 - accuracy: 0.2739 - val_loss: 0.0525 - val_accuracy: 0.2665\n",
+ "Epoch 240/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2736\n",
+ " (9.403274474232514, 1e-05)-DP guarantees for epoch 240 \n",
+ "\n",
+ "5/5 [==============================] - 3s 356ms/step - loss: 0.0515 - accuracy: 0.2736 - val_loss: 0.0525 - val_accuracy: 0.2673\n",
+ "Epoch 241/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0517 - accuracy: 0.2729\n",
+ " (9.429588694393843, 1e-05)-DP guarantees for epoch 241 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0517 - accuracy: 0.2729 - val_loss: 0.0524 - val_accuracy: 0.2674\n",
+ "Epoch 242/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0518 - accuracy: 0.2746\n",
+ " (9.455902914555171, 1e-05)-DP guarantees for epoch 242 \n",
+ "\n",
+ "5/5 [==============================] - 3s 359ms/step - loss: 0.0518 - accuracy: 0.2746 - val_loss: 0.0525 - val_accuracy: 0.2694\n",
+ "Epoch 243/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0515 - accuracy: 0.2756\n",
+ " (9.482217134716501, 1e-05)-DP guarantees for epoch 243 \n",
+ "\n",
+ "5/5 [==============================] - 2s 357ms/step - loss: 0.0515 - accuracy: 0.2756 - val_loss: 0.0524 - val_accuracy: 0.2699\n",
+ "Epoch 244/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0512 - accuracy: 0.2760\n",
+ " (9.50853135487783, 1e-05)-DP guarantees for epoch 244 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0512 - accuracy: 0.2760 - val_loss: 0.0524 - val_accuracy: 0.2712\n",
+ "Epoch 245/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0514 - accuracy: 0.2756\n",
+ " (9.534845575173634, 1e-05)-DP guarantees for epoch 245 \n",
+ "\n",
+ "5/5 [==============================] - 3s 360ms/step - loss: 0.0514 - accuracy: 0.2756 - val_loss: 0.0523 - val_accuracy: 0.2700\n",
+ "Epoch 246/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0512 - accuracy: 0.2758\n",
+ " (9.561159795911662, 1e-05)-DP guarantees for epoch 246 \n",
+ "\n",
+ "5/5 [==============================] - 2s 344ms/step - loss: 0.0512 - accuracy: 0.2758 - val_loss: 0.0523 - val_accuracy: 0.2716\n",
+ "Epoch 247/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0512 - accuracy: 0.2783\n",
+ " (9.587474015660208, 1e-05)-DP guarantees for epoch 247 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0512 - accuracy: 0.2783 - val_loss: 0.0523 - val_accuracy: 0.2719\n",
+ "Epoch 248/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0514 - accuracy: 0.2768\n",
+ " (9.613788235533915, 1e-05)-DP guarantees for epoch 248 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0514 - accuracy: 0.2768 - val_loss: 0.0522 - val_accuracy: 0.2711\n",
+ "Epoch 249/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2780\n",
+ " (9.64006012187098, 1e-05)-DP guarantees for epoch 249 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0511 - accuracy: 0.2780 - val_loss: 0.0522 - val_accuracy: 0.2720\n",
+ "Epoch 250/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2746\n",
+ " (9.665736679745127, 1e-05)-DP guarantees for epoch 250 \n",
+ "\n",
+ "5/5 [==============================] - 3s 352ms/step - loss: 0.0511 - accuracy: 0.2746 - val_loss: 0.0522 - val_accuracy: 0.2731\n",
+ "Epoch 251/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0509 - accuracy: 0.2777\n",
+ " (9.690604545814235, 1e-05)-DP guarantees for epoch 251 \n",
+ "\n",
+ "5/5 [==============================] - 3s 357ms/step - loss: 0.0509 - accuracy: 0.2777 - val_loss: 0.0522 - val_accuracy: 0.2727\n",
+ "Epoch 252/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2793\n",
+ " (9.714604392289775, 1e-05)-DP guarantees for epoch 252 \n",
+ "\n",
+ "5/5 [==============================] - 3s 354ms/step - loss: 0.0511 - accuracy: 0.2793 - val_loss: 0.0522 - val_accuracy: 0.2701\n",
+ "Epoch 253/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2751\n",
+ " (9.737670917278972, 1e-05)-DP guarantees for epoch 253 \n",
+ "\n",
+ "5/5 [==============================] - 3s 354ms/step - loss: 0.0511 - accuracy: 0.2751 - val_loss: 0.0522 - val_accuracy: 0.2719\n",
+ "Epoch 254/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0510 - accuracy: 0.2764\n",
+ " (9.759732027763015, 1e-05)-DP guarantees for epoch 254 \n",
+ "\n",
+ "5/5 [==============================] - 3s 365ms/step - loss: 0.0510 - accuracy: 0.2764 - val_loss: 0.0522 - val_accuracy: 0.2718\n",
+ "Epoch 255/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0510 - accuracy: 0.2765\n",
+ " (9.780707877727917, 1e-05)-DP guarantees for epoch 255 \n",
+ "\n",
+ "5/5 [==============================] - 3s 341ms/step - loss: 0.0510 - accuracy: 0.2765 - val_loss: 0.0522 - val_accuracy: 0.2708\n",
+ "Epoch 256/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2758\n",
+ " (9.80055660088896, 1e-05)-DP guarantees for epoch 256 \n",
+ "\n",
+ "5/5 [==============================] - 3s 364ms/step - loss: 0.0511 - accuracy: 0.2758 - val_loss: 0.0521 - val_accuracy: 0.2726\n",
+ "Epoch 257/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0508 - accuracy: 0.2767\n",
+ " (9.820083418023108, 1e-05)-DP guarantees for epoch 257 \n",
+ "\n",
+ "5/5 [==============================] - 2s 360ms/step - loss: 0.0508 - accuracy: 0.2767 - val_loss: 0.0520 - val_accuracy: 0.2722\n",
+ "Epoch 258/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2740\n",
+ " (9.839610235157256, 1e-05)-DP guarantees for epoch 258 \n",
+ "\n",
+ "5/5 [==============================] - 3s 348ms/step - loss: 0.0511 - accuracy: 0.2740 - val_loss: 0.0520 - val_accuracy: 0.2707\n",
+ "Epoch 259/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0507 - accuracy: 0.2782\n",
+ " (9.859137052291402, 1e-05)-DP guarantees for epoch 259 \n",
+ "\n",
+ "5/5 [==============================] - 3s 367ms/step - loss: 0.0507 - accuracy: 0.2782 - val_loss: 0.0520 - val_accuracy: 0.2731\n",
+ "Epoch 260/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0510 - accuracy: 0.2761\n",
+ " (9.87866386942555, 1e-05)-DP guarantees for epoch 260 \n",
+ "\n",
+ "5/5 [==============================] - 3s 353ms/step - loss: 0.0510 - accuracy: 0.2761 - val_loss: 0.0519 - val_accuracy: 0.2707\n",
+ "Epoch 261/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0509 - accuracy: 0.2751\n",
+ " (9.898190686559698, 1e-05)-DP guarantees for epoch 261 \n",
+ "\n",
+ "5/5 [==============================] - 3s 379ms/step - loss: 0.0509 - accuracy: 0.2751 - val_loss: 0.0519 - val_accuracy: 0.2724\n",
+ "Epoch 262/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0511 - accuracy: 0.2766\n",
+ " (9.917717503693844, 1e-05)-DP guarantees for epoch 262 \n",
+ "\n",
+ "5/5 [==============================] - 2s 352ms/step - loss: 0.0511 - accuracy: 0.2766 - val_loss: 0.0520 - val_accuracy: 0.2729\n",
+ "Epoch 263/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0508 - accuracy: 0.2773\n",
+ " (9.937244320827991, 1e-05)-DP guarantees for epoch 263 \n",
+ "\n",
+ "5/5 [==============================] - 2s 342ms/step - loss: 0.0508 - accuracy: 0.2773 - val_loss: 0.0520 - val_accuracy: 0.2708\n",
+ "Epoch 264/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0507 - accuracy: 0.2777\n",
+ " (9.95677113796214, 1e-05)-DP guarantees for epoch 264 \n",
+ "\n",
+ "5/5 [==============================] - 3s 366ms/step - loss: 0.0507 - accuracy: 0.2777 - val_loss: 0.0519 - val_accuracy: 0.2733\n",
+ "Epoch 265/265\n",
+ "5/5 [==============================] - ETA: 0s - loss: 0.0505 - accuracy: 0.2784\n",
+ " (9.976297955096285, 1e-05)-DP guarantees for epoch 265 \n",
+ "\n",
+ "5/5 [==============================] - 3s 378ms/step - loss: 0.0505 - accuracy: 0.2784 - val_loss: 0.0519 - val_accuracy: 0.2738\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = model.fit(\n",
+ " ds_train,\n",
+ " epochs=num_epochs,\n",
+ " validation_data=ds_test,\n",
+ " callbacks=[\n",
+ " # accounting is done thanks to a callback\n",
+ " DP_Accountant(log_fn=\"logging\"), # wandb.log also available.\n",
+ " ],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8e139678-6ec6-4a2e-980b-83059c98c48b",
+ "metadata": {},
+ "source": [
+ "This final val_accuracy is compliant with results reported in other framework. For comparison, in Opacus tutorials, the Resnet 18 reaches 60% val_accuracy at $\\epsilon=47$, but 15% at $\\epsilon=13$. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "db636e0c-0334-45ee-b953-e4cc85bb7d8e",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/basics_mnist.ipynb b/examples/basics_mnist.ipynb
new file mode 100644
index 0000000..3020545
--- /dev/null
+++ b/examples/basics_mnist.ipynb
@@ -0,0 +1,886 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "f7bf07b9-d489-4484-acb9-175cb740dc60",
+ "metadata": {},
+ "source": [
+ "# Mnist tutorial\n",
+ "\n",
+ "This notebook introduces the basics of usage of our library.\n",
+ "\n",
+ "## Imports"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "8a0eebdf-6082-4d00-aa14-b42953217a93",
+ "metadata": {},
+ "source": [
+ "The library is based on tensorflow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "91c2965e-0375-4966-bc55-776204af9d69",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "9356cd9b-6f79-45f1-8f2e-c46a526c4ae7",
+ "metadata": {},
+ "source": [
+ "### lip-dp dependencies\n",
+ "\n",
+ "The need a model `DP_Sequential` that handles the noisification of gradients. It is composed `layers` and trained with a loss found in `loss`. The model is initialized with the convenience function `DPParameters`. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "1e5d58f8-386c-44c7-8c5d-e5b69b5be231",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp import layers\n",
+ "from deel.lipdp import losses\n",
+ "from deel.lipdp.model import DP_Sequential\n",
+ "from deel.lipdp.model import DPParameters"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "3a247cd3-48d6-4854-92df-01420d3bea80",
+ "metadata": {},
+ "source": [
+ "The `DP_Accountant` callback keeps track of $(\\epsilon,\\delta)$-DP values epoch after epoch. In practice we may be interested in reaching the maximum val_accuracy under privacy constraint $\\epsilon$: the convenience function `get_max_epochs` exactly does that by performing a dichotomy search over the number of epochs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "950c5c56-4b34-4653-aaf3-7d97acc1f5f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.model import DP_Accountant\n",
+ "from deel.lipdp.sensitivity import get_max_epochs"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "893d3078-5166-428c-9cb1-d29ec1f05d71",
+ "metadata": {},
+ "source": [
+ "The framework requires a control of the maximum norm of inputs. This can be ensured with input clipping for example: `bound_clip_value`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "f395c9fc-b67d-4fd2-be4b-b1c43221ebcb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deel.lipdp.pipeline import bound_clip_value\n",
+ "from deel.lipdp.pipeline import load_and_prepare_data"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "e54a79db-24b4-4dae-b684-170fa743bc5d",
+ "metadata": {},
+ "source": [
+ "## Setup DP Lipschitz model\n",
+ "\n",
+ "Here we apply the \"global\" strategy, with a noise multiplier $2.5$. Note that for Mnist the dataset size is $N=60,000$, and it is recommended that $\\delta<\\frac{1}{N}$. So we propose a value of $\\delta=10^{-5}$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "f79ea3b0-33a6-401c-a3a3-e314939fd269",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dp_parameters = DPParameters(\n",
+ " noisify_strategy=\"global\",\n",
+ " noise_multiplier=2.0,\n",
+ " delta=1e-5,\n",
+ ")\n",
+ "\n",
+ "epsilon_max = 3.0"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "6482128c-ac2e-4cdd-9bbd-6d3172c292b1",
+ "metadata": {},
+ "source": [
+ "### Loading the data\n",
+ "\n",
+ "We clip the elementwise input upper-bound to $20.0$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "a8ed0fc4-4655-4bad-a6ac-8697cd5bc7a6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-05-24 16:00:31.206597: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
+ "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
+ "2023-05-24 16:00:31.742417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 47066 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:03:00.0, compute capability: 7.5\n"
+ ]
+ }
+ ],
+ "source": [
+ "import warnings\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "\n",
+ "# data loader return dataset_metadata which allows to\n",
+ "# know the informations required for privacy accounting\n",
+ "# (dataset size, number of samples, max input bound...)\n",
+ "input_upper_bound = 20.0\n",
+ "ds_train, ds_test, dataset_metadata = load_and_prepare_data(\n",
+ " \"mnist\",\n",
+ " batch_size=1000,\n",
+ " drop_remainder=True, # accounting assumes fixed batch size\n",
+ " bound_fct=bound_clip_value( # other strategies are possible, like normalization.\n",
+ " input_upper_bound\n",
+ " ), # clipping preprocessing allows to control input bound\n",
+ ")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "eb356c04-a836-4f49-93d7-7e0cc4c12b1d",
+ "metadata": {},
+ "source": [
+ "### Build the DP model\n",
+ "\n",
+ "We imitate the interface of Keras. We use common layers found in deel-lip, which a wrapper that handles the bound propagation. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "30cf44ed-653b-4eaa-8ed9-26e4815db511",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# construct DP_Sequential\n",
+ "model = DP_Sequential(\n",
+ " # works like usual sequential but requires DP layers\n",
+ " layers=[\n",
+ " # BoundedInput works like Input, but performs input clipping to guarantee input bound\n",
+ " layers.DP_BoundedInput(\n",
+ " input_shape=dataset_metadata.input_shape, upper_bound=input_upper_bound\n",
+ " ),\n",
+ " layers.DP_QuickSpectralConv2D( # Reshaped Kernel Orthogonalization (RKO) convolution.\n",
+ " filters=32,\n",
+ " kernel_size=3,\n",
+ " kernel_initializer=\"orthogonal\",\n",
+ " strides=1,\n",
+ " use_bias=False, # No biases since the framework handles a single tf.Variable per layer.\n",
+ " ),\n",
+ " layers.DP_GroupSort(2), # GNP activation function.\n",
+ " layers.DP_ScaledL2NormPooling2D(pool_size=2, strides=2), # GNP pooling.\n",
+ " layers.DP_QuickSpectralConv2D( # Reshaped Kernel Orthogonalization (RKO) convolution.\n",
+ " filters=64,\n",
+ " kernel_size=3,\n",
+ " kernel_initializer=\"orthogonal\",\n",
+ " strides=1,\n",
+ " use_bias=False, # No biases since the framework handles a single tf.Variable per layer.\n",
+ " ),\n",
+ " layers.DP_GroupSort(2), # GNP activation function.\n",
+ " layers.DP_ScaledL2NormPooling2D(pool_size=2, strides=2), # GNP pooling.\n",
+ " \n",
+ " layers.DP_Flatten(), # Convert features maps to flat vector.\n",
+ " \n",
+ " layers.DP_QuickSpectralDense(512), # GNP layer with orthogonal weight matrix.\n",
+ " layers.DP_GroupSort(2),\n",
+ " layers.DP_QuickSpectralDense(dataset_metadata.nb_classes),\n",
+ " ],\n",
+ " dp_parameters=dp_parameters,\n",
+ " dataset_metadata=dataset_metadata,\n",
+ ")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "09777811",
+ "metadata": {},
+ "source": [
+ "We compile the model with:\n",
+ "* any first order optimizer (e.g SGD). No adaptation or special optimizer is needed.\n",
+ "* a loss with known Lipschitz constant, e.g Categorical Cross-entropy with temperature."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "efd97e75-34f0-49fa-ad2c-1816247f1611",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Model: \"dp__sequential\"\n",
+ "_________________________________________________________________\n",
+ " Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ " dp__bounded_input (DP_Bound (None, 28, 28, 1) 0 \n",
+ " edInput) \n",
+ " \n",
+ " dp__quick_spectral_conv2d ( (None, 26, 26, 32) 288 \n",
+ " DP_QuickSpectralConv2D) \n",
+ " \n",
+ " dp__group_sort (DP_GroupSor (None, 26, 26, 32) 0 \n",
+ " t) \n",
+ " \n",
+ " dp__scaled_l2_norm_pooling2 (None, 13, 13, 32) 0 \n",
+ " d (DP_ScaledL2NormPooling2D \n",
+ " ) \n",
+ " \n",
+ " dp__quick_spectral_conv2d_1 (None, 11, 11, 64) 18432 \n",
+ " (DP_QuickSpectralConv2D) \n",
+ " \n",
+ " dp__group_sort_1 (DP_GroupS (None, 11, 11, 64) 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__scaled_l2_norm_pooling2 (None, 5, 5, 64) 0 \n",
+ " d_1 (DP_ScaledL2NormPooling \n",
+ " 2D) \n",
+ " \n",
+ " dp__flatten (DP_Flatten) (None, 1600) 0 \n",
+ " \n",
+ " dp__quick_spectral_dense (D (None, 512) 819200 \n",
+ " P_QuickSpectralDense) \n",
+ " \n",
+ " dp__group_sort_2 (DP_GroupS (None, 512) 0 \n",
+ " ort) \n",
+ " \n",
+ " dp__quick_spectral_dense_1 (None, 10) 5120 \n",
+ " (DP_QuickSpectralDense) \n",
+ " \n",
+ "=================================================================\n",
+ "Total params: 843,040\n",
+ "Trainable params: 843,040\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model.compile(\n",
+ " # Compile model using DP loss\n",
+ " loss=losses.DP_TauCategoricalCrossentropy(18.0),\n",
+ " # this method is compatible with any first order optimizer\n",
+ " optimizer=tf.keras.optimizers.SGD(learning_rate=2e-4, momentum=0.9),\n",
+ " metrics=[\"accuracy\"],\n",
+ ")\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "28ae2da5-ed40-4131-8721-73bbc73fa68d",
+ "metadata": {},
+ "source": [
+ "Note that the model contains $843$K parameters. Without gradient clipping these architectures can be trained with batch sizes as big as $1000$ on a standard GPU.\n",
+ "\n",
+ "Then, we compute the number of epochs. The maximum value of epsilon will depends on dp_parameters and the number of epochs. In order to control epsilon, we compute the adequate number of epochs"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "dd611afd-be30-4bd3-b658-48d1961247aa",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "epoch bounds = (0, 512.0) and epsilon = 7.994426666195571 at epoch 512.0\n",
+ "epoch bounds = (0, 256.0) and epsilon = 5.34128917907949 at epoch 256.0\n",
+ "epoch bounds = (0, 128.0) and epsilon = 3.631964622805248 at epoch 128.0\n",
+ "epoch bounds = (64.0, 128.0) and epsilon = 2.4829841192119444 at epoch 64.0\n",
+ "epoch bounds = (64.0, 96.0) and epsilon = 3.089635897639078 at epoch 96.0\n",
+ "epoch bounds = (80.0, 96.0) and epsilon = 2.796528753679695 at epoch 80.0\n",
+ "epoch bounds = (88.0, 96.0) and epsilon = 2.952713799856404 at epoch 88.0\n",
+ "epoch bounds = (88.0, 92.0) and epsilon = 3.0216241846349847 at epoch 92.0\n",
+ "epoch bounds = (90.0, 92.0) and epsilon = 2.987618328313939 at epoch 90.0\n",
+ "epoch bounds = (90.0, 91.0) and epsilon = 3.0046212568846444 at epoch 91.0\n"
+ ]
+ }
+ ],
+ "source": [
+ "num_epochs = get_max_epochs(epsilon_max, model)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "53e94244",
+ "metadata": {},
+ "source": [
+ "## Train the model\n",
+ "\n",
+ "The model can be trained, and the DP Accountant will automatically track the privacy loss."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "0ddcb192-547e-400e-87bb-2d4246185c64",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Epoch 1/91\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2023-05-24 16:00:36.621954: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8300\n",
+ "2023-05-24 16:00:37.363789: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "60/60 [==============================] - ETA: 0s - loss: 0.2020 - accuracy: 0.2324\n",
+ " (0.3227333785403041, 1e-05)-DP guarantees for epoch 1 \n",
+ "\n",
+ "60/60 [==============================] - 5s 38ms/step - loss: 0.2020 - accuracy: 0.2324 - val_loss: 0.1712 - val_accuracy: 0.3147\n",
+ "Epoch 2/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.1607 - accuracy: 0.3958\n",
+ " (0.41135036253440604, 1e-05)-DP guarantees for epoch 2 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1604 - accuracy: 0.3992 - val_loss: 0.1486 - val_accuracy: 0.5122\n",
+ "Epoch 3/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1426 - accuracy: 0.5510\n",
+ " (0.4972854400421322, 1e-05)-DP guarantees for epoch 3 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1426 - accuracy: 0.5510 - val_loss: 0.1334 - val_accuracy: 0.6108\n",
+ "Epoch 4/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1291 - accuracy: 0.6333\n",
+ " (0.5737399623472044, 1e-05)-DP guarantees for epoch 4 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1291 - accuracy: 0.6333 - val_loss: 0.1213 - val_accuracy: 0.6784\n",
+ "Epoch 5/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1182 - accuracy: 0.6883\n",
+ " (0.6418194146435952, 1e-05)-DP guarantees for epoch 5 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1182 - accuracy: 0.6883 - val_loss: 0.1109 - val_accuracy: 0.7180\n",
+ "Epoch 6/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.1088 - accuracy: 0.7247\n",
+ " (0.7042008802236781, 1e-05)-DP guarantees for epoch 6 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.1087 - accuracy: 0.7247 - val_loss: 0.1024 - val_accuracy: 0.7527\n",
+ "Epoch 7/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.1012 - accuracy: 0.7488\n",
+ " (0.7616059152520757, 1e-05)-DP guarantees for epoch 7 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.1012 - accuracy: 0.7488 - val_loss: 0.0955 - val_accuracy: 0.7698\n",
+ "Epoch 8/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0948 - accuracy: 0.7644\n",
+ " (0.8155744676428971, 1e-05)-DP guarantees for epoch 8 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0948 - accuracy: 0.7644 - val_loss: 0.0899 - val_accuracy: 0.7815\n",
+ "Epoch 9/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0896 - accuracy: 0.7785\n",
+ " (0.8666021691681208, 1e-05)-DP guarantees for epoch 9 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0896 - accuracy: 0.7785 - val_loss: 0.0848 - val_accuracy: 0.7936\n",
+ "Epoch 10/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0849 - accuracy: 0.7868\n",
+ " (0.9152742048884784, 1e-05)-DP guarantees for epoch 10 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0849 - accuracy: 0.7868 - val_loss: 0.0804 - val_accuracy: 0.8003\n",
+ "Epoch 11/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0810 - accuracy: 0.7967\n",
+ " (0.9617965624530973, 1e-05)-DP guarantees for epoch 11 \n",
+ "\n",
+ "60/60 [==============================] - 2s 30ms/step - loss: 0.0809 - accuracy: 0.7975 - val_loss: 0.0769 - val_accuracy: 0.8109\n",
+ "Epoch 12/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0774 - accuracy: 0.8060\n",
+ " (1.0059716506359193, 1e-05)-DP guarantees for epoch 12 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0774 - accuracy: 0.8060 - val_loss: 0.0733 - val_accuracy: 0.8179\n",
+ "Epoch 13/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0740 - accuracy: 0.8131\n",
+ " (1.049398006635733, 1e-05)-DP guarantees for epoch 13 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0740 - accuracy: 0.8131 - val_loss: 0.0704 - val_accuracy: 0.8269\n",
+ "Epoch 14/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0713 - accuracy: 0.8216\n",
+ " (1.090263192229449, 1e-05)-DP guarantees for epoch 14 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0713 - accuracy: 0.8216 - val_loss: 0.0677 - val_accuracy: 0.8309\n",
+ "Epoch 15/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0689 - accuracy: 0.8240\n",
+ " (1.131126828240101, 1e-05)-DP guarantees for epoch 15 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0689 - accuracy: 0.8240 - val_loss: 0.0656 - val_accuracy: 0.8355\n",
+ "Epoch 16/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0669 - accuracy: 0.8293\n",
+ " (1.169340908770284, 1e-05)-DP guarantees for epoch 16 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0668 - accuracy: 0.8296 - val_loss: 0.0635 - val_accuracy: 0.8398\n",
+ "Epoch 17/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0647 - accuracy: 0.8333\n",
+ " (1.2074292910030167, 1e-05)-DP guarantees for epoch 17 \n",
+ "\n",
+ "60/60 [==============================] - 2s 29ms/step - loss: 0.0646 - accuracy: 0.8335 - val_loss: 0.0615 - val_accuracy: 0.8437\n",
+ "Epoch 18/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0630 - accuracy: 0.8366\n",
+ " (1.2447047350704166, 1e-05)-DP guarantees for epoch 18 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0629 - accuracy: 0.8367 - val_loss: 0.0598 - val_accuracy: 0.8468\n",
+ "Epoch 19/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0612 - accuracy: 0.8399\n",
+ " (1.2800495944157277, 1e-05)-DP guarantees for epoch 19 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0612 - accuracy: 0.8399 - val_loss: 0.0582 - val_accuracy: 0.8508\n",
+ "Epoch 20/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0598 - accuracy: 0.8428\n",
+ " (1.3153944538284068, 1e-05)-DP guarantees for epoch 20 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0598 - accuracy: 0.8428 - val_loss: 0.0569 - val_accuracy: 0.8563\n",
+ "Epoch 21/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0584 - accuracy: 0.8468\n",
+ " (1.3507368078845663, 1e-05)-DP guarantees for epoch 21 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0584 - accuracy: 0.8466 - val_loss: 0.0557 - val_accuracy: 0.8572\n",
+ "Epoch 22/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0572 - accuracy: 0.8509\n",
+ " (1.383564204783113, 1e-05)-DP guarantees for epoch 22 \n",
+ "\n",
+ "60/60 [==============================] - 2s 30ms/step - loss: 0.0572 - accuracy: 0.8509 - val_loss: 0.0546 - val_accuracy: 0.8610\n",
+ "Epoch 23/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0561 - accuracy: 0.8519\n",
+ " (1.4161979427317832, 1e-05)-DP guarantees for epoch 23 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0562 - accuracy: 0.8518 - val_loss: 0.0537 - val_accuracy: 0.8619\n",
+ "Epoch 24/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0552 - accuracy: 0.8547\n",
+ " (1.448831680775656, 1e-05)-DP guarantees for epoch 24 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0552 - accuracy: 0.8547 - val_loss: 0.0525 - val_accuracy: 0.8657\n",
+ "Epoch 25/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0541 - accuracy: 0.8575\n",
+ " (1.4814654188092617, 1e-05)-DP guarantees for epoch 25 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0541 - accuracy: 0.8576 - val_loss: 0.0516 - val_accuracy: 0.8675\n",
+ "Epoch 26/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.8578\n",
+ " (1.512526290723161, 1e-05)-DP guarantees for epoch 26 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0531 - accuracy: 0.8578 - val_loss: 0.0506 - val_accuracy: 0.8691\n",
+ "Epoch 27/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0522 - accuracy: 0.8605\n",
+ " (1.5424804710143858, 1e-05)-DP guarantees for epoch 27 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0522 - accuracy: 0.8605 - val_loss: 0.0497 - val_accuracy: 0.8709\n",
+ "Epoch 28/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0512 - accuracy: 0.8624\n",
+ " (1.5724346510360574, 1e-05)-DP guarantees for epoch 28 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0512 - accuracy: 0.8626 - val_loss: 0.0488 - val_accuracy: 0.8730\n",
+ "Epoch 29/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0503 - accuracy: 0.8650\n",
+ " (1.6023888317992228, 1e-05)-DP guarantees for epoch 29 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0503 - accuracy: 0.8653 - val_loss: 0.0479 - val_accuracy: 0.8752\n",
+ "Epoch 30/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0495 - accuracy: 0.8665\n",
+ " (1.632343011263517, 1e-05)-DP guarantees for epoch 30 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0495 - accuracy: 0.8667 - val_loss: 0.0471 - val_accuracy: 0.8749\n",
+ "Epoch 31/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0488 - accuracy: 0.8684\n",
+ " (1.6622962394525178, 1e-05)-DP guarantees for epoch 31 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0487 - accuracy: 0.8686 - val_loss: 0.0463 - val_accuracy: 0.8779\n",
+ "Epoch 32/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0480 - accuracy: 0.8697\n",
+ " (1.689965116494089, 1e-05)-DP guarantees for epoch 32 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0480 - accuracy: 0.8697 - val_loss: 0.0457 - val_accuracy: 0.8777\n",
+ "Epoch 33/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0475 - accuracy: 0.8700\n",
+ " (1.7172705001520499, 1e-05)-DP guarantees for epoch 33 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0475 - accuracy: 0.8704 - val_loss: 0.0452 - val_accuracy: 0.8790\n",
+ "Epoch 34/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0469 - accuracy: 0.8736\n",
+ " (1.7445758842338837, 1e-05)-DP guarantees for epoch 34 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0468 - accuracy: 0.8738 - val_loss: 0.0446 - val_accuracy: 0.8806\n",
+ "Epoch 35/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0463 - accuracy: 0.8754\n",
+ " (1.7718812676250233, 1e-05)-DP guarantees for epoch 35 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0462 - accuracy: 0.8756 - val_loss: 0.0441 - val_accuracy: 0.8825\n",
+ "Epoch 36/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0456 - accuracy: 0.8763\n",
+ " (1.799186650959813, 1e-05)-DP guarantees for epoch 36 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0456 - accuracy: 0.8763 - val_loss: 0.0434 - val_accuracy: 0.8831\n",
+ "Epoch 37/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0450 - accuracy: 0.8771\n",
+ " (1.8264920346090618, 1e-05)-DP guarantees for epoch 37 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0450 - accuracy: 0.8773 - val_loss: 0.0429 - val_accuracy: 0.8846\n",
+ "Epoch 38/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0444 - accuracy: 0.8786\n",
+ " (1.8537974184156425, 1e-05)-DP guarantees for epoch 38 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0444 - accuracy: 0.8786 - val_loss: 0.0423 - val_accuracy: 0.8855\n",
+ "Epoch 39/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0439 - accuracy: 0.8800\n",
+ " (1.8807666749981604, 1e-05)-DP guarantees for epoch 39 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0439 - accuracy: 0.8802 - val_loss: 0.0419 - val_accuracy: 0.8863\n",
+ "Epoch 40/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0435 - accuracy: 0.8803\n",
+ " (1.9054738700393052, 1e-05)-DP guarantees for epoch 40 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0435 - accuracy: 0.8804 - val_loss: 0.0415 - val_accuracy: 0.8858\n",
+ "Epoch 41/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0430 - accuracy: 0.8816\n",
+ " (1.9301604511513608, 1e-05)-DP guarantees for epoch 41 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0430 - accuracy: 0.8816 - val_loss: 0.0410 - val_accuracy: 0.8884\n",
+ "Epoch 42/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0425 - accuracy: 0.8824\n",
+ " (1.9548470320035656, 1e-05)-DP guarantees for epoch 42 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0425 - accuracy: 0.8824 - val_loss: 0.0405 - val_accuracy: 0.8890\n",
+ "Epoch 43/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0421 - accuracy: 0.8837\n",
+ " (1.979533612594768, 1e-05)-DP guarantees for epoch 43 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0421 - accuracy: 0.8837 - val_loss: 0.0403 - val_accuracy: 0.8890\n",
+ "Epoch 44/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0418 - accuracy: 0.8856\n",
+ " (2.0042201936126345, 1e-05)-DP guarantees for epoch 44 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0418 - accuracy: 0.8856 - val_loss: 0.0399 - val_accuracy: 0.8908\n",
+ "Epoch 45/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0414 - accuracy: 0.8858\n",
+ " (2.0289067746857206, 1e-05)-DP guarantees for epoch 45 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0414 - accuracy: 0.8856 - val_loss: 0.0393 - val_accuracy: 0.8926\n",
+ "Epoch 46/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0408 - accuracy: 0.8872\n",
+ " (2.053593355232055, 1e-05)-DP guarantees for epoch 46 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0408 - accuracy: 0.8872 - val_loss: 0.0388 - val_accuracy: 0.8951\n",
+ "Epoch 47/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0405 - accuracy: 0.8882\n",
+ " (2.078279935996221, 1e-05)-DP guarantees for epoch 47 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0404 - accuracy: 0.8887 - val_loss: 0.0385 - val_accuracy: 0.8959\n",
+ "Epoch 48/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0400 - accuracy: 0.8882\n",
+ " (2.1029665168498504, 1e-05)-DP guarantees for epoch 48 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0400 - accuracy: 0.8882 - val_loss: 0.0381 - val_accuracy: 0.8952\n",
+ "Epoch 49/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0397 - accuracy: 0.8890\n",
+ " (2.127653097450219, 1e-05)-DP guarantees for epoch 49 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0398 - accuracy: 0.8888 - val_loss: 0.0379 - val_accuracy: 0.8943\n",
+ "Epoch 50/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0396 - accuracy: 0.8887\n",
+ " (2.151531383398666, 1e-05)-DP guarantees for epoch 50 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0395 - accuracy: 0.8889 - val_loss: 0.0375 - val_accuracy: 0.8946\n",
+ "Epoch 51/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0391 - accuracy: 0.8893\n",
+ " (2.1736284198821467, 1e-05)-DP guarantees for epoch 51 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0391 - accuracy: 0.8895 - val_loss: 0.0372 - val_accuracy: 0.8968\n",
+ "Epoch 52/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0387 - accuracy: 0.8908\n",
+ " (2.195725456202997, 1e-05)-DP guarantees for epoch 52 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0387 - accuracy: 0.8908 - val_loss: 0.0368 - val_accuracy: 0.8967\n",
+ "Epoch 53/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0385 - accuracy: 0.8905\n",
+ " (2.217822492103547, 1e-05)-DP guarantees for epoch 53 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0385 - accuracy: 0.8905 - val_loss: 0.0366 - val_accuracy: 0.8991\n",
+ "Epoch 54/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0382 - accuracy: 0.8913\n",
+ " (2.2399195284840734, 1e-05)-DP guarantees for epoch 54 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0382 - accuracy: 0.8913 - val_loss: 0.0365 - val_accuracy: 0.8992\n",
+ "Epoch 55/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0380 - accuracy: 0.8924\n",
+ " (2.2620165646623547, 1e-05)-DP guarantees for epoch 55 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0380 - accuracy: 0.8921 - val_loss: 0.0362 - val_accuracy: 0.8994\n",
+ "Epoch 56/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0377 - accuracy: 0.8925\n",
+ " (2.2841136015562187, 1e-05)-DP guarantees for epoch 56 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0377 - accuracy: 0.8925 - val_loss: 0.0358 - val_accuracy: 0.8999\n",
+ "Epoch 57/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0374 - accuracy: 0.8930\n",
+ " (2.3062106367493893, 1e-05)-DP guarantees for epoch 57 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0374 - accuracy: 0.8930 - val_loss: 0.0356 - val_accuracy: 0.9004\n",
+ "Epoch 58/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0371 - accuracy: 0.8938\n",
+ " (2.3283076739544244, 1e-05)-DP guarantees for epoch 58 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0372 - accuracy: 0.8939 - val_loss: 0.0354 - val_accuracy: 0.9010\n",
+ "Epoch 59/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0369 - accuracy: 0.8951\n",
+ " (2.3504047095381226, 1e-05)-DP guarantees for epoch 59 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0369 - accuracy: 0.8951 - val_loss: 0.0351 - val_accuracy: 0.9010\n",
+ "Epoch 60/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0365 - accuracy: 0.8963\n",
+ " (2.3725017457248683, 1e-05)-DP guarantees for epoch 60 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0365 - accuracy: 0.8963 - val_loss: 0.0347 - val_accuracy: 0.9037\n",
+ "Epoch 61/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0363 - accuracy: 0.8968\n",
+ " (2.3945987822094885, 1e-05)-DP guarantees for epoch 61 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0363 - accuracy: 0.8968 - val_loss: 0.0346 - val_accuracy: 0.9024\n",
+ "Epoch 62/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0360 - accuracy: 0.8979\n",
+ " (2.4166958179233653, 1e-05)-DP guarantees for epoch 62 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0360 - accuracy: 0.8981 - val_loss: 0.0343 - val_accuracy: 0.9041\n",
+ "Epoch 63/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0358 - accuracy: 0.8986\n",
+ " (2.438792853624178, 1e-05)-DP guarantees for epoch 63 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0358 - accuracy: 0.8987 - val_loss: 0.0340 - val_accuracy: 0.9068\n",
+ "Epoch 64/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0355 - accuracy: 0.8995\n",
+ " (2.4608898896847116, 1e-05)-DP guarantees for epoch 64 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0356 - accuracy: 0.8992 - val_loss: 0.0338 - val_accuracy: 0.9072\n",
+ "Epoch 65/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0352 - accuracy: 0.9005\n",
+ " (2.4829841192119444, 1e-05)-DP guarantees for epoch 65 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0353 - accuracy: 0.9000 - val_loss: 0.0336 - val_accuracy: 0.9059\n",
+ "Epoch 66/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0351 - accuracy: 0.8996\n",
+ " (2.5034880893370737, 1e-05)-DP guarantees for epoch 66 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0351 - accuracy: 0.8996 - val_loss: 0.0334 - val_accuracy: 0.9070\n",
+ "Epoch 67/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0350 - accuracy: 0.9003\n",
+ " (2.523024133549594, 1e-05)-DP guarantees for epoch 67 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0349 - accuracy: 0.9003 - val_loss: 0.0333 - val_accuracy: 0.9069\n",
+ "Epoch 68/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0348 - accuracy: 0.9005\n",
+ " (2.542560178527111, 1e-05)-DP guarantees for epoch 68 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0348 - accuracy: 0.9005 - val_loss: 0.0332 - val_accuracy: 0.9071\n",
+ "Epoch 69/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0346 - accuracy: 0.9006\n",
+ " (2.5620962223364145, 1e-05)-DP guarantees for epoch 69 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0347 - accuracy: 0.9007 - val_loss: 0.0329 - val_accuracy: 0.9081\n",
+ "Epoch 70/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0345 - accuracy: 0.9015\n",
+ " (2.5816322672410785, 1e-05)-DP guarantees for epoch 70 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0345 - accuracy: 0.9014 - val_loss: 0.0327 - val_accuracy: 0.9069\n",
+ "Epoch 71/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0343 - accuracy: 0.9017\n",
+ " (2.601168310806795, 1e-05)-DP guarantees for epoch 71 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0343 - accuracy: 0.9019 - val_loss: 0.0326 - val_accuracy: 0.9090\n",
+ "Epoch 72/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0342 - accuracy: 0.9021\n",
+ " (2.620704354996593, 1e-05)-DP guarantees for epoch 72 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0342 - accuracy: 0.9022 - val_loss: 0.0324 - val_accuracy: 0.9089\n",
+ "Epoch 73/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0340 - accuracy: 0.9018\n",
+ " (2.640240400625916, 1e-05)-DP guarantees for epoch 73 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0339 - accuracy: 0.9020 - val_loss: 0.0322 - val_accuracy: 0.9096\n",
+ "Epoch 74/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0339 - accuracy: 0.9018\n",
+ " (2.659776444789028, 1e-05)-DP guarantees for epoch 74 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0338 - accuracy: 0.9022 - val_loss: 0.0320 - val_accuracy: 0.9103\n",
+ "Epoch 75/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0335 - accuracy: 0.9024\n",
+ " (2.679312488654814, 1e-05)-DP guarantees for epoch 75 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0335 - accuracy: 0.9024 - val_loss: 0.0318 - val_accuracy: 0.9088\n",
+ "Epoch 76/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0333 - accuracy: 0.9025\n",
+ " (2.69884853278786, 1e-05)-DP guarantees for epoch 76 \n",
+ "\n",
+ "60/60 [==============================] - 2s 29ms/step - loss: 0.0333 - accuracy: 0.9023 - val_loss: 0.0315 - val_accuracy: 0.9098\n",
+ "Epoch 77/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0332 - accuracy: 0.9033\n",
+ " (2.7183845763895516, 1e-05)-DP guarantees for epoch 77 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0332 - accuracy: 0.9033 - val_loss: 0.0314 - val_accuracy: 0.9125\n",
+ "Epoch 78/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0330 - accuracy: 0.9046\n",
+ " (2.737920620600221, 1e-05)-DP guarantees for epoch 78 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0330 - accuracy: 0.9048 - val_loss: 0.0313 - val_accuracy: 0.9119\n",
+ "Epoch 79/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0328 - accuracy: 0.9053\n",
+ " (2.7574566653298858, 1e-05)-DP guarantees for epoch 79 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0328 - accuracy: 0.9053 - val_loss: 0.0311 - val_accuracy: 0.9115\n",
+ "Epoch 80/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0328 - accuracy: 0.9052\n",
+ " (2.7769927101097007, 1e-05)-DP guarantees for epoch 80 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0327 - accuracy: 0.9056 - val_loss: 0.0310 - val_accuracy: 0.9118\n",
+ "Epoch 81/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0325 - accuracy: 0.9056\n",
+ " (2.796528753679695, 1e-05)-DP guarantees for epoch 81 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0325 - accuracy: 0.9056 - val_loss: 0.0308 - val_accuracy: 0.9114\n",
+ "Epoch 82/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0324 - accuracy: 0.9057\n",
+ " (2.816064798903292, 1e-05)-DP guarantees for epoch 82 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0324 - accuracy: 0.9057 - val_loss: 0.0307 - val_accuracy: 0.9114\n",
+ "Epoch 83/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0323 - accuracy: 0.9053\n",
+ " (2.8356008431856474, 1e-05)-DP guarantees for epoch 83 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0322 - accuracy: 0.9057 - val_loss: 0.0305 - val_accuracy: 0.9117\n",
+ "Epoch 84/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0320 - accuracy: 0.9063\n",
+ " (2.8551368864333964, 1e-05)-DP guarantees for epoch 84 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0320 - accuracy: 0.9063 - val_loss: 0.0303 - val_accuracy: 0.9117\n",
+ "Epoch 85/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0318 - accuracy: 0.9064\n",
+ " (2.8746729305801413, 1e-05)-DP guarantees for epoch 85 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0318 - accuracy: 0.9064 - val_loss: 0.0302 - val_accuracy: 0.9121\n",
+ "Epoch 86/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0317 - accuracy: 0.9074\n",
+ " (2.894208975473722, 1e-05)-DP guarantees for epoch 86 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0316 - accuracy: 0.9076 - val_loss: 0.0299 - val_accuracy: 0.9132\n",
+ "Epoch 87/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9078\n",
+ " (2.9137450193835823, 1e-05)-DP guarantees for epoch 87 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0314 - accuracy: 0.9076 - val_loss: 0.0298 - val_accuracy: 0.9123\n",
+ "Epoch 88/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0313 - accuracy: 0.9086\n",
+ " (2.9332810632263646, 1e-05)-DP guarantees for epoch 88 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0313 - accuracy: 0.9086 - val_loss: 0.0299 - val_accuracy: 0.9133\n",
+ "Epoch 89/91\n",
+ "59/60 [============================>.] - ETA: 0s - loss: 0.0313 - accuracy: 0.9087\n",
+ " (2.952713799856404, 1e-05)-DP guarantees for epoch 89 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0313 - accuracy: 0.9087 - val_loss: 0.0298 - val_accuracy: 0.9140\n",
+ "Epoch 90/91\n",
+ "60/60 [==============================] - ETA: 0s - loss: 0.0312 - accuracy: 0.9097\n",
+ " (2.970615400210975, 1e-05)-DP guarantees for epoch 90 \n",
+ "\n",
+ "60/60 [==============================] - 2s 28ms/step - loss: 0.0312 - accuracy: 0.9097 - val_loss: 0.0298 - val_accuracy: 0.9127\n",
+ "Epoch 91/91\n",
+ "58/60 [============================>.] - ETA: 0s - loss: 0.0312 - accuracy: 0.9091\n",
+ " (2.987618328313939, 1e-05)-DP guarantees for epoch 91 \n",
+ "\n",
+ "60/60 [==============================] - 2s 27ms/step - loss: 0.0312 - accuracy: 0.9093 - val_loss: 0.0297 - val_accuracy: 0.9132\n"
+ ]
+ }
+ ],
+ "source": [
+ "hist = model.fit(\n",
+ " ds_train,\n",
+ " epochs=num_epochs,\n",
+ " validation_data=ds_test,\n",
+ " callbacks=[\n",
+ " # accounting is done thanks to a callback\n",
+ " DP_Accountant(log_fn=\"logging\"), # wandb.log also available.\n",
+ " ],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e1cbeee4-c204-454f-8f6f-20273b0169b7",
+ "metadata": {},
+ "source": [
+ "The model can be further improved by tuning various hyper-parameters, by adding layers (see `advanced_cifar10.ipynb` tutorial). "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fedc70ab-ccd5-4239-9d62-416d680af324",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/fig_accountant.png b/examples/fig_accountant.png
new file mode 100644
index 0000000..68a0410
Binary files /dev/null and b/examples/fig_accountant.png differ
diff --git a/examples/residuals.png b/examples/residuals.png
new file mode 100644
index 0000000..4840e69
Binary files /dev/null and b/examples/residuals.png differ
diff --git a/experiments/CIFAR10/CLI_sweep.sh b/experiments/CIFAR10/CLI_sweep.sh
deleted file mode 100644
index fdcc1cc..0000000
--- a/experiments/CIFAR10/CLI_sweep.sh
+++ /dev/null
@@ -1,5 +0,0 @@
-for noise in 2.5 3.2 3.5 4. 5.:
-do
- python experiments/CIFAR10/main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="TauCategoricalCrossentropy" --cfg.opt_iterations=30 --cfg.noisify_strategy="global"
- python experiments/CIFAR10/main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="KCosineSimilarity" --cfg.opt_iterations=30 --cfg.noisify_strategy="global"
-done
\ No newline at end of file
diff --git a/experiments/CIFAR10/main.py b/experiments/CIFAR10/main.py
new file mode 100644
index 0000000..b697fb7
--- /dev/null
+++ b/experiments/CIFAR10/main.py
@@ -0,0 +1,409 @@
+# -*- coding: utf-8 -*-
+# Copyright IRT Antoine de Saint Exupéry et Université Paul Sabatier Toulouse III - All
+# rights reserved. DEEL is a research program operated by IVADO, IRT Saint Exupéry,
+# CRIAQ and ANITI - https://www.deel.ai/
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+import numpy as np
+import tensorflow as tf
+from absl import app
+from ml_collections import config_dict
+from ml_collections import config_flags
+
+import deel.lipdp.layers as DP_layers
+from deel.lip.metrics import CategoricalProvableRobustAccuracy
+from deel.lipdp import losses
+from deel.lipdp.dynamic import AdaptiveQuantileClipping
+from deel.lipdp.dynamic import LaplaceAdaptiveLossGradientClipping
+from deel.lipdp.model import DP_Accountant
+from deel.lipdp.model import DP_Model
+from deel.lipdp.model import DPParameters
+from deel.lipdp.pipeline import bound_clip_value
+from deel.lipdp.pipeline import default_delta_value
+from deel.lipdp.pipeline import load_and_prepare_images_data
+from deel.lipdp.sensitivity import get_max_epochs
+from deel.lipdp.utils import PrivacyMetrics
+from deel.lipdp.utils import SignaltoNoiseAverage
+from deel.lipdp.utils import SignaltoNoiseHistogram
+from experiments.wandb_utils import init_wandb
+from experiments.wandb_utils import run_with_wandb
+from wandb.keras import WandbCallback
+
+
+def default_cfg_cifar10():
+ cfg = config_dict.ConfigDict()
+ cfg.batch_size = 2_500 # 5% of the dataset.
+ cfg.clip_loss_gradient = None # not required for dynamic clipping.
+ cfg.depth = 1
+ cfg.dynamic_clipping = "quantiles" # can be "fixed", "laplace", "quantiles". "fixed" requires a clipping value.
+ cfg.dynamic_clipping_quantiles = (
+ 0.9 # crop to 90% of the distribution of gradient norm.
+ )
+ cfg.delta = 1e-5 # 1e-5 is the default value in the paper.
+ cfg.epsilon_max = 8.0 # budget!
+ cfg.input_bound = 3.0 # 15.0 works well in RGB non standardized.
+ cfg.learning_rate = 8e-2 # works well for vanilla SGD.
+ cfg.log_wandb = "disabled"
+ cfg.loss = "TauCategoricalCrossentropy"
+ cfg.mia = False
+ cfg.multiplicity = 0 # 0 means no multiplicity.
+ cfg.noise_multiplier = 3.0
+ cfg.noisify_strategy = "per-layer"
+ cfg.representation = "RGB_STANDARDIZED" # "RGB", "RGB_STANDARDIZED", "HSV".
+ cfg.optimizer = "SGD"
+ cfg.signal_to_noise = "histogram"
+ cfg.sweep_id = "" # useful to resume a sweep.
+ cfg.sweep_yaml_config = "" # useful to load a sweep from a yaml file.
+ cfg.tau = 20.0 # temperature for the softmax.
+ cfg.use_residuals = False # better without.
+ cfg.width_multiplier = 1
+ return cfg
+
+
+project = "ICLR_Cifar10"
+cfg = default_cfg_cifar10()
+_CONFIG = config_flags.DEFINE_config_dict(
+ "cfg", cfg
+) # for FLAGS parsing in command line.
+
+
+def create_MLP_Mixer(dataset_metadata, dp_parameters):
+ layers = [
+ DP_layers.DP_BoundedInput(
+ input_shape=dataset_metadata.input_shape,
+ upper_bound=dataset_metadata.max_norm,
+ )
+ ]
+
+ patch_size = 4
+ num_mixer_layers = cfg.depth
+ seq_len = (dataset_metadata.input_shape[0] // patch_size) * (
+ dataset_metadata.input_shape[1] // patch_size
+ )
+ multiplier = cfg.width_multiplier
+ mlp_seq_dim = multiplier * seq_len
+ mlp_channel_dim = multiplier * seq_len
+ hidden_size = multiplier * seq_len
+ use_residuals = cfg.use_residuals
+
+ layers.append(
+ DP_layers.DP_Lambda(
+ tf.image.extract_patches,
+ arguments=dict(
+ sizes=[1, patch_size, patch_size, 1],
+ strides=[1, patch_size, patch_size, 1],
+ rates=[1, 1, 1, 1],
+ padding="VALID",
+ ),
+ )
+ )
+
+ layers.append(
+ DP_layers.DP_Reshape(
+ (seq_len, (patch_size**2) * dataset_metadata.input_shape[-1])
+ )
+ )
+ layers.append(
+ DP_layers.DP_QuickSpectralDense(
+ units=hidden_size, use_bias=False, kernel_initializer="orthogonal"
+ )
+ )
+
+ for _ in range(num_mixer_layers):
+ to_add = [
+ DP_layers.DP_Permute((2, 1)),
+ DP_layers.DP_QuickSpectralDense(
+ units=mlp_seq_dim, use_bias=False, kernel_initializer="orthogonal"
+ ),
+ ]
+ to_add.append(DP_layers.DP_GroupSort(2))
+ to_add.append(DP_layers.DP_LayerCentering())
+ to_add += [
+ DP_layers.DP_QuickSpectralDense(
+ units=seq_len, use_bias=False, kernel_initializer="orthogonal"
+ ),
+ DP_layers.DP_Permute((2, 1)),
+ ]
+
+ if use_residuals:
+ layers += DP_layers.make_residuals("1-lip-add", to_add)
+ else:
+ layers += to_add
+
+ to_add = [
+ DP_layers.DP_QuickSpectralDense(
+ units=mlp_channel_dim, use_bias=False, kernel_initializer="orthogonal"
+ ),
+ ]
+ to_add.append(DP_layers.DP_GroupSort(2))
+ to_add.append(DP_layers.DP_LayerCentering())
+ to_add.append(
+ DP_layers.DP_QuickSpectralDense(
+ units=hidden_size, use_bias=False, kernel_initializer="orthogonal"
+ )
+ )
+
+ if use_residuals:
+ layers += DP_layers.make_residuals("1-lip-add", to_add)
+ else:
+ layers += to_add
+
+ layers += [
+ DP_layers.DP_Flatten(),
+ ]
+
+ layers.append(
+ DP_layers.DP_QuickSpectralDense(
+ units=dataset_metadata.nb_classes,
+ use_bias=False,
+ kernel_initializer="orthogonal",
+ )
+ )
+
+ layers.append(
+ DP_layers.DP_ClipGradient(
+ clip_value=cfg.clip_loss_gradient,
+ mode="dynamic",
+ )
+ )
+
+ model = DP_Model(
+ layers,
+ dp_parameters=dp_parameters,
+ dataset_metadata=dataset_metadata,
+ name="mlp_mixer",
+ )
+
+ model.build(input_shape=(None, *dataset_metadata.input_shape))
+
+ return model
+
+
+def get_cifar10_standardized(verbose=True):
+ (x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
+ x_train = x_train.astype("float32") / 255.0
+ x_test = x_test.astype("float32") / 255.0
+ CIFAR10_MEAN = np.array([0.4914, 0.4822, 0.4465]).reshape((1, 1, 3))
+ CIFAR10_STD_DEV = np.array([0.2023, 0.1994, 0.2010]).reshape((1, 1, 3))
+ x_train = (x_train - CIFAR10_MEAN) / CIFAR10_STD_DEV
+ x_test = (x_test - CIFAR10_MEAN) / CIFAR10_STD_DEV
+ y_train = y_train.flatten()
+ y_test = y_test.flatten()
+ cifar10 = (x_train, y_train, x_test, y_test)
+ all_norms = np.linalg.norm(x_train, axis=-1)
+ if verbose:
+ print(f"Dataset Max norm: {np.max(all_norms)}")
+ print(f"Dataset Min norm: {np.min(all_norms)}")
+ print(f"Dataset Mean norm: {np.mean(all_norms)}")
+ print(f"Dataset Std norm: {np.std(all_norms)}")
+ quantiles = [0.25, 0.5, 0.75, 0.8, 0.85, 0.9, 0.95, 0.99]
+ print(f"Dataset Quantiles: {np.quantile(all_norms, quantiles)} at {quantiles}")
+ # We assume no privacy loss to estimate the max norm, the mean pixel value and the std pixel value.
+ # This is a reasonable assumption shared by many papers. If necessary, they can be estimated privately.
+ max_norm = np.max(all_norms)
+ max_norm = max_norm.astype(np.float32)
+ return cifar10, max_norm
+
+
+def certifiable_acc_metrics(epsilons):
+ """Returns a list of metrics for certifiable accuracy at the given epsilons.
+
+ Args:
+ epsilons: list of epsilons to evaluate, assuming 8bits encoding.
+
+ Returns:
+ list of metrics.
+ """
+ metrics = []
+ for epsilon_8bit in epsilons:
+ name = f"certacc_{epsilon_8bit}"
+ epsilon = epsilon_8bit / 255.0
+ # dataset has been standardized so we take that into account:
+ if cfg.representation == "RGB_STANDARDIZED":
+ epsilon = epsilon / 0.2023 # maximum std dev: lower bound of radius.
+ else:
+ assert (
+ cfg.representation == "RGB"
+ ), "Certifiable accuracy only implemented for RGB and RGB_STANDARDIZED"
+ metric = CategoricalProvableRobustAccuracy(
+ epsilon=epsilon, disjoint_neurons=False, name=name
+ )
+ metrics.append(metric)
+ return metrics
+
+
+def train():
+ init_wandb(cfg=cfg, project=project)
+
+ ##########################
+ #### Dataset loading #####
+ ##########################
+
+ # clipping preprocessing allows to control input bound
+ input_bound = cfg.input_bound
+ if cfg.representation == "RGB_STANDARDIZED":
+ cifar10_standardized, max_norm_cifar10 = get_cifar10_standardized(verbose=True)
+ if input_bound is None:
+ input_bound = max_norm_cifar10
+ print(f"Max norm set to {input_bound}")
+ bound_fct = bound_clip_value(input_bound)
+
+ ds_train, ds_test, dataset_metadata = load_and_prepare_images_data(
+ "cifar10",
+ batch_size=cfg.batch_size,
+ colorspace=cfg.representation,
+ drop_remainder=True, # accounting assumes fixed batch size
+ bound_fct=bound_fct,
+ multiplicity=cfg.multiplicity,
+ )
+
+ ##########################
+ #### Model definition ####
+ ##########################
+
+ # declare the privacy parameters
+ dp_parameters = DPParameters(
+ noisify_strategy=cfg.noisify_strategy,
+ noise_multiplier=cfg.noise_multiplier,
+ delta=default_delta_value(dataset_metadata),
+ )
+
+ model = create_MLP_Mixer(dataset_metadata, dp_parameters)
+
+ ##########################
+ ######## Loss setup ######
+ ##########################
+
+ if cfg.loss == "TauCategoricalCrossentropy":
+ loss = losses.DP_TauCategoricalCrossentropy(cfg.tau)
+ elif cfg.loss == "MulticlassHKR":
+ alpha = 200.0
+ margin = 1.0
+ loss = losses.DP_MulticlassHKR(alpha=alpha, min_margin=margin)
+ elif cfg.loss == "KCosineSimilarity":
+ K = 0.99
+ loss = losses.DP_KCosineSimilarity(K=K)
+
+ ##########################
+ ##### Optimizer setup ####
+ ##########################
+
+ if cfg.optimizer == "Adam":
+ optimizer = tf.keras.optimizers.Adam(learning_rate=cfg.learning_rate)
+ elif cfg.optimizer == "SGD":
+ # geometric sequence: memory length ~= 1 / (1 - momentum)
+ # memory length = nb_steps_per_epochs => momentum = 1 - (1./nb_steps_per_epochs)
+ momentum = 1 - 1.0 / dataset_metadata.nb_steps_per_epochs
+ momentum = max(0.5, min(0.99, momentum)) # reasonable range
+ optimizer = tf.keras.optimizers.SGD(
+ learning_rate=cfg.learning_rate, momentum=momentum
+ )
+ else:
+ raise ValueError(f"Unknown optimizer {cfg.optimizer}")
+
+ model.compile(
+ loss=loss,
+ optimizer=optimizer,
+ metrics=[
+ "accuracy",
+ *certifiable_acc_metrics([1, 2, 4, 8, 16, 36]),
+ ], # accuracy metric is necessary for dynamic loss gradient clipping with "laplace"
+ run_eagerly=False,
+ )
+
+ callbacks = [
+ WandbCallback(save_model=False, monitor="val_accuracy"),
+ DP_Accountant(),
+ ]
+
+ if cfg.signal_to_noise == "disabled":
+ pass
+ elif cfg.signal_to_noise == "average":
+ batch_train = next(iter(ds_train))
+ callbacks.append(SignaltoNoiseAverage(batch_train))
+ elif cfg.signal_to_noise == "histogram":
+ batch_train = next(iter(ds_train))
+ callbacks.append(SignaltoNoiseHistogram(batch_train))
+ else:
+ raise ValueError(f"Unknown signal_to_noise {cfg.signal_to_noise}")
+
+ ########################
+ ### Dynamic clipping ###
+ ########################
+
+ if cfg.dynamic_clipping == "fixed":
+ assert (
+ cfg.clip_loss_gradient is not None
+ ), "Fixed mode requires a clipping value"
+ elif cfg.dynamic_clipping == "laplace":
+ adaptive = LaplaceAdaptiveLossGradientClipping(
+ ds_train=ds_train,
+ patience=1,
+ epsilon=1.0,
+ )
+ adaptive.set_model(model)
+ callbacks.append(adaptive)
+ elif cfg.dynamic_clipping == "quantiles":
+ adaptive = AdaptiveQuantileClipping(
+ ds_train=ds_train,
+ patience=1,
+ noise_multiplier=cfg.noise_multiplier * 2, # more noisy.
+ quantile=cfg.dynamic_clipping_quantiles,
+ learning_rate=1.0,
+ )
+ adaptive.set_model(model)
+ callbacks.append(adaptive)
+ else:
+ raise ValueError(f"Unknown clipping strategy {cfg.dynamic_clipping}")
+
+ ########################
+ ###### MIA attack ######
+ ########################
+
+ if cfg.mia:
+ privacy_metrics = PrivacyMetrics(cifar10_standardized)
+ callbacks.append(privacy_metrics)
+
+ ########################
+ ### Training process ###
+ ########################
+
+ if cfg.epsilon_max is None:
+ num_epochs = 1 # useful for debugging.
+ else:
+ # compute the max number of epochs to reach the budget.
+ num_epochs = get_max_epochs(cfg.epsilon_max, model, safe=True)
+
+ hist = model.fit(
+ ds_train,
+ epochs=num_epochs,
+ validation_data=ds_test,
+ callbacks=callbacks,
+ )
+
+ if cfg.mia:
+ privacy_metrics.log_report()
+
+
+def main(_):
+ run_with_wandb(cfg=cfg, train_function=train, project=project)
+
+
+if __name__ == "__main__":
+ app.run(main)
diff --git a/experiments/CIFAR10/main_template.py b/experiments/CIFAR10/main_template.py
deleted file mode 100644
index 745248c..0000000
--- a/experiments/CIFAR10/main_template.py
+++ /dev/null
@@ -1,241 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright IRT Antoine de Saint Exupéry et Université Paul Sabatier Toulouse III - All
-# rights reserved. DEEL is a research program operated by IVADO, IRT Saint Exupéry,
-# CRIAQ and ANITI - https://www.deel.ai/
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-import os
-
-import numpy as np
-import tensorflow as tf
-import yaml
-from absl import app
-from ml_collections import config_dict
-from ml_collections import config_flags
-from models_CIFAR import create_MLP_Mixer
-from models_CIFAR import create_ResNet
-from models_CIFAR import create_VGG
-from tensorflow.keras.callbacks import EarlyStopping
-from tensorflow.keras.callbacks import ReduceLROnPlateau
-
-import deel.lipdp.layers as DP_layers
-import wandb
-from deel.lipdp import losses
-from deel.lipdp.model import AdaptiveLossGradientClipping
-from deel.lipdp.model import DP_Accountant
-from deel.lipdp.model import DP_Model
-from deel.lipdp.model import DPParameters
-from deel.lipdp.pipeline import bound_clip_value
-from deel.lipdp.pipeline import load_and_prepare_data
-from deel.lipdp.sensitivity import get_max_epochs
-from wandb.keras import WandbCallback
-from wandb_sweeps.src_config.wandb_utils import init_wandb
-from wandb_sweeps.src_config.wandb_utils import run_with_wandb
-
-cfg = config_dict.ConfigDict()
-
-cfg.batch_size = 5_000
-cfg.clip_loss_gradient = 1.0
-cfg.delta = 1e-5
-cfg.epsilon_max = 10.0
-cfg.input_bound = 15.0
-cfg.K = 0.99
-cfg.learning_rate = 1e-3
-cfg.log_wandb = "disabled"
-cfg.opt_iterations = 10
-cfg.noise_multiplier = 3.0
-cfg.noisify_strategy = "global"
-cfg.representation = "HSV"
-cfg.optimizer = "Adam"
-cfg.sweep_yaml_config = ""
-cfg.tau = 8.0
-cfg.sweep_id = ""
-cfg.loss = "TauCategoricalCrossentropy"
-
-_CONFIG = config_flags.DEFINE_config_dict("cfg", cfg)
-
-
-def create_Mixer(dataset_metadata, dp_parameters):
- layers = [
- DP_layers.DP_BoundedInput(
- input_shape=dataset_metadata.input_shape,
- upper_bound=dataset_metadata.max_norm,
- )
- ]
-
- patch_size = 3
- num_mixer_layers = 2
- seq_len = (dataset_metadata.input_shape[0] // patch_size) * (
- dataset_metadata.input_shape[1] // patch_size
- )
- multiplier = 2
- mlp_seq_dim = multiplier * seq_len
- mlp_channel_dim = multiplier * seq_len
- hidden_size = multiplier * seq_len
-
- layers.append(
- DP_layers.DP_Lambda(
- tf.image.extract_patches,
- arguments=dict(
- sizes=[1, patch_size, patch_size, 1],
- strides=[1, patch_size, patch_size, 1],
- rates=[1, 1, 1, 1],
- padding="VALID",
- ),
- )
- )
-
- layers.append(
- DP_layers.DP_Reshape(
- (seq_len, (patch_size**2) * dataset_metadata.input_shape[-1])
- )
- )
- layers.append(
- DP_layers.DP_QuickSpectralDense(
- units=hidden_size, use_bias=False, kernel_initializer="identity"
- )
- )
-
- for _ in range(num_mixer_layers):
- to_add = [
- DP_layers.DP_Permute((2, 1)),
- DP_layers.DP_QuickSpectralDense(
- units=mlp_seq_dim, use_bias=False, kernel_initializer="identity"
- ),
- ]
- to_add.append(DP_layers.DP_GroupSort(2))
- to_add.append(DP_layers.DP_LayerCentering())
- to_add += [
- DP_layers.DP_QuickSpectralDense(
- units=seq_len, use_bias=False, kernel_initializer="identity"
- ),
- DP_layers.DP_Permute((2, 1)),
- ]
- layers += DP_layers.make_residuals("1-lip-add", to_add)
- to_add = [
- DP_layers.DP_QuickSpectralDense(
- units=mlp_channel_dim, use_bias=False, kernel_initializer="identity"
- ),
- ]
- to_add.append(DP_layers.DP_GroupSort(2))
- to_add.append(DP_layers.DP_LayerCentering())
- to_add.append(
- DP_layers.DP_QuickSpectralDense(
- units=hidden_size, use_bias=False, kernel_initializer="identity"
- )
- )
- layers += DP_layers.make_residuals("1-lip-add", to_add)
-
- layers.append(DP_layers.DP_Flatten())
-
- layers.append(
- DP_layers.DP_QuickSpectralDense(
- units=10, use_bias=False, kernel_initializer="identity"
- )
- )
- layers.append(DP_layers.DP_ClipGradient(clip_value=cfg.clip_loss_gradient))
-
- model = DP_Model(
- layers,
- dp_parameters=dp_parameters,
- dataset_metadata=dataset_metadata,
- name="mlp_mixer",
- )
-
- model.build(input_shape=(None, *dataset_metadata.input_shape))
-
- return model
-
-
-def train():
- init_wandb(cfg=cfg, project="CIFAR10_dynamic_clipping")
-
- # declare the privacy parameters
- dp_parameters = DPParameters(
- noisify_strategy=cfg.noisify_strategy,
- noise_multiplier=cfg.noise_multiplier,
- delta=cfg.delta,
- )
-
- ds_train, ds_test, dataset_metadata = load_and_prepare_data(
- "cifar10",
- batch_size=cfg.batch_size,
- colorspace=cfg.representation,
- drop_remainder=True, # accounting assumes fixed batch size
- bound_fct=bound_clip_value(
- cfg.input_bound
- ), # clipping preprocessing allows to control input bound
- )
-
- model = create_Mixer(dataset_metadata, dp_parameters)
-
- if cfg.loss == "TauCategoricalCrossentropy":
- loss = losses.DP_TauCategoricalCrossentropy(cfg.tau)
- elif cfg.loss == "KCosineSimilarity":
- loss = losses.DP_KCosineSimilarity(cfg.K)
-
- model.compile(
- loss=loss,
- optimizer=tf.keras.optimizers.Adam(learning_rate=cfg.learning_rate),
- # accuracy metric is necessary for dynamic loss gradient clipping
- metrics=["accuracy"],
- run_eagerly=False,
- )
-
- num_epochs = get_max_epochs(cfg.epsilon_max, model)
-
- callbacks = [
- WandbCallback(save_model=False, monitor="val_accuracy"),
- EarlyStopping(monitor="val_accuracy", min_delta=0.001, patience=15),
- ReduceLROnPlateau(
- monitor="val_accuracy", factor=0.9, min_delta=0.001, patience=8
- ),
- DP_Accountant(),
- # AdaptiveLossGradientClipping(),
- ]
-
- hist = model.fit(
- ds_train,
- epochs=num_epochs,
- validation_data=ds_test,
- callbacks=callbacks,
- )
-
- wandb.log(
- {
- "Accuracies": wandb.plot.line_series(
- xs=[
- np.linspace(0, num_epochs, num_epochs + 1),
- np.linspace(0, num_epochs, num_epochs + 1),
- ],
- ys=[hist.history["accuracy"], hist.history["val_accuracy"]],
- keys=["Train Accuracy", "Test Accuracy"],
- title="Train/Test Accuracy",
- xname="num_epochs",
- )
- }
- )
-
-
-def main(_):
- run_with_wandb(cfg=cfg, train_function=train, project="CIFAR10_dynamic_clipping")
-
-
-if __name__ == "__main__":
- app.run(main)
diff --git a/experiments/CIFAR10/models_CIFAR.py b/experiments/CIFAR10/models.py
similarity index 89%
rename from experiments/CIFAR10/models_CIFAR.py
rename to experiments/CIFAR10/models.py
index e58dd15..d988344 100644
--- a/experiments/CIFAR10/models_CIFAR.py
+++ b/experiments/CIFAR10/models.py
@@ -44,7 +44,30 @@
def create_MLP_Mixer(dp_parameters, dataset_metadata, cfg, upper_bound):
- input_shape = (32, 32, 3)
+ """Creates a MLP-Mixer network.
+
+ The cfg object must contain some information:
+ - cfg.add_biases (bool): DP_AddBias layers after each linear layer.
+ - cfg.layer_centering (bool): DP_LayerCentering layers after each activation.
+ - cfg.skip_connections (bool): skip connections in the MLP-Mixer network.
+ - cfg.num_mixer_layers (int): number of mixer layers.
+ - cfg.patch_size (int): size of the patches.
+ - cfg.hidden_size (int): size of the hidden layer.
+ - cfg.mlp_seq_dim (int): size of the hidden layer in the MLP block.
+ - cfg.mlp_channel_dim (int): size of the hidden layer in the channel block.
+ - cfg.clip_loss_gradient (float): clip the gradient of the loss to this value.
+
+ Args:
+ dp_parameters: parameters for differentially private training
+ dataset_metadata: metadata of the dataset, for privacy accounting
+ cfg: configuration containing information for DP_Sequential and MLP-Mixer
+ hyper-parameters
+ upper_bound (float): maximum norm of the input (clipped if input norm is higher)
+
+ Returns:
+ DP_Sequential: DP MLP-Mixer network
+ """
+ input_shape = dataset_metadata.input_shape
layers = [DP_BoundedInput(input_shape=input_shape, upper_bound=upper_bound)]
layers.append(
@@ -121,7 +144,9 @@ def create_MLP_Mixer(dp_parameters, dataset_metadata, cfg, upper_bound):
DP_QuickSpectralDense(units=10, use_bias=False, kernel_initializer="identity")
)
if cfg.clip_loss_gradient is not None:
- layers.append(DP_ClipGradient(cfg.clip_loss_gradient))
+ layers.append(
+ DP_ClipGradient(cfg.clip_loss_gradient, mode=cfg.dynamic_clipping)
+ )
model = DP_Model(
layers,
@@ -253,7 +278,7 @@ def VGG_factory(
layers.append(DP_SpectralDense(10, use_bias=False, kernel_initializer="orthogonal"))
layers.append(DP_AddBias(norm_max=1))
- layers.append(DP_ClipGradient(cfg.clip_loss_gradient))
+ layers.append(DP_ClipGradient(cfg.clip_loss_gradient, mode=cfg.dynamic_clipping))
# Remove DP_AddBias and DP_LayerCentering layers if required
if cfg.add_biases is False:
@@ -403,7 +428,9 @@ def create_ResNet(dp_parameters, dataset_metadata, cfg, upper_bound):
layers += [
DP_ScaledGlobalL2NormPooling2D(name="globalpool1"),
DP_SpectralDense(classes, use_bias=False, name="fc1"),
- DP_ClipGradient(cfg.clip_loss_gradient, name="clipgrad"),
+ DP_ClipGradient(
+ cfg.clip_loss_gradient, mode=cfg.dynamic_clipping, name="clipgrad"
+ ),
]
model = DP_Model(
diff --git a/experiments/CIFAR10/sweep_1.yaml b/experiments/CIFAR10/sweep_1.yaml
new file mode 100644
index 0000000..2a4e951
--- /dev/null
+++ b/experiments/CIFAR10/sweep_1.yaml
@@ -0,0 +1,36 @@
+method: bayes
+metric:
+ name: val_certacc_16
+ goal: maximize
+parameters:
+ noise_multiplier:
+ min: 0.8
+ max: 6.0
+ distribution: uniform
+ learning_rate:
+ min: 0.00001
+ max: 1.0
+ distribution: log_uniform_values
+ batch_size:
+ values: [2000, 2500, 5000]
+ distribution: categorical
+ input_bound:
+ min: 1.0
+ max: 4.5
+ distribution: uniform
+ tau:
+ min: 0.01
+ max: 100.0
+ distribution: log_uniform_values
+ epsilon_max:
+ value: 20.0
+ distribution: constant
+ width_multiplier:
+ value: 2
+ distribution: constant
+ depth:
+ value: 1
+ distribution: constant
+ multiplicity:
+ value: 0
+ distribution: constant
diff --git a/experiments/MNIST/CLI_sweep.sh b/experiments/MNIST/CLI_sweep.sh
deleted file mode 100644
index 0688a27..0000000
--- a/experiments/MNIST/CLI_sweep.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-for noise in 17.5 14.5 11.5 9.0 7.0 6.0:
-do
- python main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="TauCategoricalCrossentropy" --cfg.opt_iterations=50 --cfg.architecture="Dense" --cfg.noisify_strategy="global"
- python main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="TauCategoricalCrossentropy" --cfg.opt_iterations=50 --cfg.architecture="ConvNet" --cfg.noisify_strategy="global"
- python main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="KCosineSimilarity" --cfg.opt_iterations=50 --cfg.architecture="Dense" --cfg.noisify_strategy="global"
- python main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="KCosineSimilarity" --cfg.opt_iterations=50 --cfg.architecture="ConvNet" --cfg.noisify_strategy="global"
- python main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="MulticlassHKR" --cfg.opt_iterations=50 --cfg.architecture="Dense" --cfg.noisify_strategy="global"
- python main_template.py --cfg.log_wandb="sweep_archi" --cfg.noise_multiplier=$noise --cfg.loss="MulticlassHKR" --cfg.opt_iterations=50 --cfg.architecture="ConvNet" --cfg.noisify_strategy="global"
-done
\ No newline at end of file
diff --git a/experiments/MNIST/main_template.py b/experiments/MNIST/main.py
similarity index 52%
rename from experiments/MNIST/main_template.py
rename to experiments/MNIST/main.py
index 5a8f80b..a16fa78 100644
--- a/experiments/MNIST/main_template.py
+++ b/experiments/MNIST/main.py
@@ -28,65 +28,106 @@
from absl import app
from ml_collections import config_dict
from ml_collections import config_flags
-from models_MNIST import create_ConvNet
-from tensorflow.keras.callbacks import EarlyStopping
-from tensorflow.keras.callbacks import ReduceLROnPlateau
import wandb
+from deel.lipdp.dynamic import AdaptiveQuantileClipping
+from deel.lipdp.layers import DP_AddBias
+from deel.lipdp.layers import DP_BoundedInput
+from deel.lipdp.layers import DP_ClipGradient
+from deel.lipdp.layers import DP_Flatten
+from deel.lipdp.layers import DP_GroupSort
+from deel.lipdp.layers import DP_LayerCentering
+from deel.lipdp.layers import DP_ScaledL2NormPooling2D
+from deel.lipdp.layers import DP_SpectralConv2D
+from deel.lipdp.layers import DP_SpectralDense
from deel.lipdp.losses import *
from deel.lipdp.model import DP_Accountant
+from deel.lipdp.model import DP_Sequential
from deel.lipdp.model import DPParameters
-from deel.lipdp.pipeline import bound_clip_value
-from deel.lipdp.pipeline import load_and_prepare_data
+from deel.lipdp.pipeline import bound_normalize
+from deel.lipdp.pipeline import default_delta_value
+from deel.lipdp.pipeline import load_and_prepare_images_data
from deel.lipdp.sensitivity import get_max_epochs
+from experiments.wandb_utils import init_wandb
+from experiments.wandb_utils import run_with_wandb
from wandb.keras import WandbCallback
-from wandb_sweeps.src_config.wandb_utils import init_wandb
-from wandb_sweeps.src_config.wandb_utils import run_with_wandb
-
-
-cfg = config_dict.ConfigDict()
-
-cfg.add_biases = True
-cfg.alpha = 50.0
-cfg.architecture = "ConvNet"
-cfg.batch_size = 8_192
-cfg.condense = True
-cfg.clip_loss_gradient = 1.0
-cfg.delta = 1e-5
-cfg.epsilon_max = 3.0
-cfg.input_clipping = 0.7
-cfg.K = 0.99
-cfg.learning_rate = 1e-2
-cfg.lip_coef = 1.0
-cfg.loss = "TauCategoricalCrossentropy"
-cfg.log_wandb = "disabled"
-cfg.min_margin = 0.5
-cfg.min_norm = 5.21
-cfg.model_name = "No_name"
-cfg.noise_multiplier = 5.0
-cfg.noisify_strategy = "local"
-cfg.optimizer = "Adam"
-cfg.N = 50_000
-cfg.num_classes = 10
-cfg.opt_iterations = 10
-cfg.run_eagerly = False
-cfg.sweep_yaml_config = ""
-cfg.save = False
-cfg.save_folder = os.getcwd()
-cfg.sweep_id = ""
-cfg.tau = 1.0
-cfg.tag = "Default"
+
+def default_cfg_mnist():
+ cfg = config_dict.ConfigDict()
+ cfg.add_biases = True
+ cfg.batch_size = 2_000
+ cfg.clip_loss_gradient = None # not required for dynamic clipping.
+ cfg.dynamic_clipping = "quantiles" # can be "fixed", "laplace", "quantiles". "fixed" requires a clipping value.
+ cfg.dynamic_clipping_quantiles = (
+ 0.9 # crop to 90% of the distribution of gradient norm.
+ )
+ cfg.epsilon_max = 3.0
+ cfg.input_clipping = 0.7
+ cfg.learning_rate = 5e-3
+ cfg.loss = "TauCategoricalCrossentropy"
+ cfg.log_wandb = "disabled"
+ cfg.noise_multiplier = 1.5
+ cfg.noisify_strategy = "per-layer"
+ cfg.optimizer = "Adam"
+ cfg.opt_iterations = None
+ cfg.save = False
+ cfg.save_folder = os.getcwd()
+ cfg.sweep_yaml_config = ""
+ cfg.sweep_id = ""
+ cfg.tau = 32.0
+ return cfg
+
+
+cfg = default_cfg_mnist()
_CONFIG = config_flags.DEFINE_config_dict("cfg", cfg)
-def create_model(dp_parameters, dataset_metadata, cfg, upper_bound):
- if cfg.architecture == "ConvNet":
- model = create_ConvNet(dp_parameters, dataset_metadata, cfg, upper_bound)
- elif cfg.architecture == "Dense":
- raise NotImplementedError("Dense architecture not implemented yet")
- else:
- raise ValueError(f"Invalid architecture argument {cfg.architecture}")
+def create_ConvNet(dp_parameters, dataset_metadata):
+ norm_max = 1.0
+ all_layers = [
+ DP_BoundedInput(input_shape=(28, 28, 1), upper_bound=dataset_metadata.max_norm),
+ DP_SpectralConv2D(
+ filters=16,
+ kernel_size=3,
+ kernel_initializer="orthogonal",
+ strides=1,
+ use_bias=False,
+ ),
+ DP_AddBias(norm_max=norm_max),
+ DP_GroupSort(2),
+ DP_ScaledL2NormPooling2D(pool_size=2, strides=2),
+ DP_LayerCentering(),
+ DP_SpectralConv2D(
+ filters=32,
+ kernel_size=3,
+ kernel_initializer="orthogonal",
+ strides=1,
+ use_bias=False,
+ ),
+ DP_AddBias(norm_max=norm_max),
+ DP_GroupSort(2),
+ DP_ScaledL2NormPooling2D(pool_size=2, strides=2),
+ DP_LayerCentering(),
+ DP_Flatten(),
+ DP_SpectralDense(1024, use_bias=False, kernel_initializer="orthogonal"),
+ DP_AddBias(norm_max=norm_max),
+ DP_SpectralDense(10, use_bias=False, kernel_initializer="orthogonal"),
+ DP_AddBias(norm_max=norm_max),
+ DP_ClipGradient(
+ clip_value=cfg.clip_loss_gradient,
+ mode="dynamic",
+ ),
+ ]
+ if not cfg.add_biases:
+ all_layers = [
+ layer for layer in all_layers if not isinstance(layer, DP_AddBias)
+ ]
+ model = DP_Sequential(
+ all_layers,
+ dp_parameters=dp_parameters,
+ dataset_metadata=dataset_metadata,
+ )
return model
@@ -98,18 +139,19 @@ def compile_model(model, cfg):
optimizer = tf.keras.optimizers.Adam(learning_rate=cfg.learning_rate)
else:
print("Illegal optimizer argument : ", cfg.optimizer)
+
# Choice of loss function
if cfg.loss == "MulticlassHKR":
if cfg.optimizer == "SGD":
cfg.learning_rate = cfg.learning_rate / cfg.alpha
loss = DP_MulticlassHKR(
- alpha=cfg.alpha,
- min_margin=cfg.min_margin,
+ alpha=50.0,
+ min_margin=0.5,
reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE,
)
elif cfg.loss == "MulticlassHinge":
loss = DP_MulticlassHinge(
- min_margin=cfg.min_margin,
+ min_margin=0.5,
reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE,
)
elif cfg.loss == "MulticlassKR":
@@ -119,9 +161,8 @@ def compile_model(model, cfg):
cfg.tau, reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE
)
elif cfg.loss == "KCosineSimilarity":
- KX_min = cfg.K * cfg.min_norm
loss = DP_KCosineSimilarity(
- KX_min, reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE
+ 0.99, reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE
)
elif cfg.loss == "MAE":
loss = DP_MeanAbsoluteError(
@@ -129,6 +170,7 @@ def compile_model(model, cfg):
)
else:
raise ValueError(f"Illegal loss argument {cfg.loss}")
+
# Compile model
model.compile(
# decreasing alpha and increasing min_margin improve robustness (at the cost of accuracy)
@@ -136,7 +178,6 @@ def compile_model(model, cfg):
loss=loss,
optimizer=optimizer,
metrics=["accuracy"],
- run_eagerly=cfg.run_eagerly,
)
return model
@@ -144,34 +185,42 @@ def compile_model(model, cfg):
def train():
init_wandb(cfg=cfg, project="MNIST_ClipLess_SGD")
- ds_train, ds_test, dataset_metadata = load_and_prepare_data(
+ ds_train, ds_test, dataset_metadata = load_and_prepare_images_data(
"mnist",
cfg.batch_size,
- colorspace="RGB",
+ colorspace="grayscale",
drop_remainder=True,
- bound_fct=bound_clip_value(cfg.input_clipping),
+ bound_fct=bound_normalize(),
)
- model = create_model(
+
+ model = create_ConvNet(
DPParameters(
noisify_strategy=cfg.noisify_strategy,
noise_multiplier=cfg.noise_multiplier,
- delta=cfg.delta,
+ delta=default_delta_value(dataset_metadata),
),
dataset_metadata,
- cfg,
- upper_bound=dataset_metadata.max_norm,
)
+
model = compile_model(model, cfg)
model.summary()
+
num_epochs = get_max_epochs(cfg.epsilon_max, model)
+
+ adaptive = AdaptiveQuantileClipping(
+ ds_train=ds_train,
+ patience=1,
+ noise_multiplier=cfg.noise_multiplier * 5, # more noisy.
+ quantile=cfg.dynamic_clipping_quantiles,
+ learning_rate=1.0,
+ )
+ adaptive.set_model(model)
callbacks = [
WandbCallback(save_model=False, monitor="val_accuracy"),
- EarlyStopping(monitor="val_accuracy", min_delta=0.001, patience=15),
- ReduceLROnPlateau(
- monitor="val_accuracy", factor=0.9, min_delta=0.0001, patience=5
- ),
DP_Accountant(),
+ adaptive,
]
+
hist = model.fit(
ds_train,
epochs=num_epochs,
@@ -179,26 +228,10 @@ def train():
batch_size=cfg.batch_size,
callbacks=callbacks,
)
- wandb.log(
- {
- "Accuracies": wandb.plot.line_series(
- xs=[
- np.linspace(0, num_epochs, num_epochs + 1),
- np.linspace(0, num_epochs, num_epochs + 1),
- ],
- ys=[hist.history["accuracy"], hist.history["val_accuracy"]],
- keys=["Train Accuracy", "Test Accuracy"],
- title="Train/Test Accuracy",
- xname="num_epochs",
- )
- }
- )
- if cfg.save:
- model.save(f"{cfg.save_folder}/{cfg.model_name}.h5")
def main(_):
- run_with_wandb(cfg=cfg, train_function=train, project="MNIST_ClipLess_SGD")
+ run_with_wandb(cfg=cfg, train_function=train, project="ICLR_MNIST_acc")
if __name__ == "__main__":
diff --git a/experiments/MNIST/models_MNIST.py b/experiments/MNIST/models_MNIST.py
deleted file mode 100644
index 2bea581..0000000
--- a/experiments/MNIST/models_MNIST.py
+++ /dev/null
@@ -1,81 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright IRT Antoine de Saint Exupéry et Université Paul Sabatier Toulouse III - All
-# rights reserved. DEEL is a research program operated by IVADO, IRT Saint Exupéry,
-# CRIAQ and ANITI - https://www.deel.ai/
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-import tensorflow as tf
-
-from deel.lipdp.layers import DP_AddBias
-from deel.lipdp.layers import DP_BoundedInput
-from deel.lipdp.layers import DP_ClipGradient
-from deel.lipdp.layers import DP_Flatten
-from deel.lipdp.layers import DP_GroupSort
-from deel.lipdp.layers import DP_LayerCentering
-from deel.lipdp.layers import DP_ScaledL2NormPooling2D
-from deel.lipdp.layers import DP_SpectralConv2D
-from deel.lipdp.layers import DP_SpectralDense
-from deel.lipdp.layers import make_residuals
-from deel.lipdp.model import DP_Model
-from deel.lipdp.model import DP_Sequential
-
-
-def create_ConvNet(dp_parameters, dataset_metadata, cfg, upper_bound):
- norm_max = 1.0
- all_layers = [
- DP_BoundedInput(input_shape=(28, 28, 1), upper_bound=upper_bound),
- DP_SpectralConv2D(
- filters=16,
- kernel_size=3,
- kernel_initializer="orthogonal",
- strides=1,
- use_bias=False,
- ),
- DP_AddBias(norm_max=norm_max),
- DP_GroupSort(2),
- DP_ScaledL2NormPooling2D(pool_size=2, strides=2),
- DP_LayerCentering(),
- DP_SpectralConv2D(
- filters=32,
- kernel_size=3,
- kernel_initializer="orthogonal",
- strides=1,
- use_bias=False,
- ),
- DP_AddBias(norm_max=norm_max),
- DP_GroupSort(2),
- DP_ScaledL2NormPooling2D(pool_size=2, strides=2),
- DP_LayerCentering(),
- DP_Flatten(),
- DP_SpectralDense(1024, use_bias=False, kernel_initializer="orthogonal"),
- DP_AddBias(norm_max=norm_max),
- DP_SpectralDense(10, use_bias=False, kernel_initializer="orthogonal"),
- DP_AddBias(norm_max=norm_max),
- DP_ClipGradient(clip_value=cfg.clip_loss_gradient),
- ]
- if not cfg.add_biases:
- all_layers = [
- layer for layer in all_layers if not isinstance(layer, DP_AddBias)
- ]
- model = DP_Sequential(
- all_layers,
- dp_parameters=dp_parameters,
- dataset_metadata=dataset_metadata,
- )
- return model
diff --git a/experiments/MNIST/sweep_1.yaml b/experiments/MNIST/sweep_1.yaml
new file mode 100644
index 0000000..8b13789
--- /dev/null
+++ b/experiments/MNIST/sweep_1.yaml
@@ -0,0 +1 @@
+
diff --git a/experiments/dynamic_gradient_clipping/main_gradient_clipping.py b/experiments/dynamic_gradient_clipping/main_gradient_clipping.py
deleted file mode 100644
index 1493169..0000000
--- a/experiments/dynamic_gradient_clipping/main_gradient_clipping.py
+++ /dev/null
@@ -1,112 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright IRT Antoine de Saint Exupéry et Université Paul Sabatier Toulouse III - All
-# rights reserved. DEEL is a research program operated by IVADO, IRT Saint Exupéry,
-# CRIAQ and ANITI - https://www.deel.ai/
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-import os
-
-import numpy as np
-import tensorflow as tf
-import yaml
-from absl import app
-from ml_collections import config_dict
-from ml_collections import config_flags
-from tensorflow.keras.callbacks import EarlyStopping
-from tensorflow.keras.callbacks import ReduceLROnPlateau
-
-import deel.lipdp.layers as DP_layers
-from deel.lipdp import losses
-from deel.lipdp.model import AdaptiveLossGradientClipping
-from deel.lipdp.model import DP_Accountant
-from deel.lipdp.model import DP_Sequential
-from deel.lipdp.model import DPParameters
-from deel.lipdp.pipeline import bound_clip_value
-from deel.lipdp.pipeline import load_and_prepare_data
-from deel.lipdp.sensitivity import get_max_epochs
-
-# declare the privacy parameters
-dp_parameters = DPParameters(
- noisify_strategy="global",
- noise_multiplier=1.2,
- delta=1e-5,
-)
-
-ds_train, ds_test, dataset_metadata = load_and_prepare_data(
- "cifar10",
- batch_size=4096,
- colorspace="HSV",
- drop_remainder=True, # accounting assumes fixed batch size
- bound_fct=bound_clip_value(
- 10.0
- ), # clipping preprocessing allows to control input bound
-)
-
-layers = [
- DP_layers.DP_BoundedInput(
- input_shape=dataset_metadata.input_shape,
- upper_bound=dataset_metadata.max_norm,
- ),
- DP_layers.DP_SpectralConv2D(
- filters=80, kernel_size=3, use_bias=False, kernel_initializer="orthogonal"
- ),
- DP_layers.DP_ScaledL2NormPooling2D(pool_size=2),
- DP_layers.DP_Flatten(),
- DP_layers.DP_SpectralDense(
- units=512, use_bias=False, kernel_initializer="orthogonal"
- ),
- DP_layers.DP_LayerCentering(),
- DP_layers.DP_GroupSort(2),
- DP_layers.DP_SpectralDense(
- units=10, use_bias=False, kernel_initializer="orthogonal"
- ),
- DP_layers.DP_ClipGradient(
- clip_value=None, epsilon=1, patience=5
- ), # for fixed clipping use clip_value = cste
-]
-
-model = DP_Sequential(
- layers=layers, dp_parameters=dp_parameters, dataset_metadata=dataset_metadata
-)
-
-loss = losses.DP_TauCategoricalCrossentropy(14.5)
-
-model.compile(
- loss=loss,
- optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
- metrics=["accuracy"],
- run_eagerly=False,
-)
-
-num_epochs = get_max_epochs(8.0, model)
-
-callbacks = [
- DP_Accountant(log_fn="logging"),
- AdaptiveLossGradientClipping(
- ds_train=ds_train
- ), # DO NOT USE THIS CALLBACK WHEN mode != "dynamic_svt"
- ReduceLROnPlateau(monitor="val_accuracy", factor=0.9, min_delta=0.01, patience=3),
-]
-
-hist = model.fit(
- ds_train,
- epochs=num_epochs,
- validation_data=ds_test,
- callbacks=callbacks,
-)
diff --git a/experiments/paper_plots/histogram_monitoring.ipynb b/experiments/paper_plots/histogram_monitoring.ipynb
new file mode 100644
index 0000000..b35dad1
--- /dev/null
+++ b/experiments/paper_plots/histogram_monitoring.ipynb
@@ -0,0 +1,376 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | _type | \n", + "bins | \n", + "values | \n", + "sha256 | \n", + "path | \n", + "size | \n", + "
|---|---|---|---|---|---|---|
| val_certacc_4 | \n", + "0.0001 | \n", + "0.0001 | \n", + "0.0001 | \n", + "0.0001 | \n", + "0.0001 | \n", + "0.0001 | \n", + "
| _timestamp | \n", + "1695985453.405793 | \n", + "1695985453.405793 | \n", + "1695985453.405793 | \n", + "1695985453.405793 | \n", + "1695985453.405793 | \n", + "1695985453.405793 | \n", + "
| gradient_bounds_dp__quick_spectral_dense/kernel:0 | \n", + "4.242641 | \n", + "4.242641 | \n", + "4.242641 | \n", + "4.242641 | \n", + "4.242641 | \n", + "4.242641 | \n", + "
| certacc_2 | \n", + "0.01618 | \n", + "0.01618 | \n", + "0.01618 | \n", + "0.01618 | \n", + "0.01618 | \n", + "0.01618 | \n", + "
| certacc_4 | \n", + "0.00008 | \n", + "0.00008 | \n", + "0.00008 | \n", + "0.00008 | \n", + "0.00008 | \n", + "0.00008 | \n", + "
+ 
+ 
+ 
+ 
+ /data/Projets/dp-lipschitz/sandbox/wandb/run-20230919_154934-e3tgtj03"
+ ],
+ "text/plain": [
+ "| \n", + " | summary | \n", + "config | \n", + "name | \n", + "
|---|---|---|---|
| 0 | \n", + "{'_timestamp': 1684172153, 'median_time': 4.57... | \n", + "{'seed': 1337, 'delta': 1e-05, 'dpsgd': True, ... | \n", + "hardy-water-466 | \n", + "
| 1 | \n", + "{'mad_time': 0.006640114821493626, '_timestamp... | \n", + "{'seed': 1337, 'delta': 1e-05, 'dpsgd': True, ... | \n", + "scarlet-lake-465 | \n", + "
| 2 | \n", + "{'times': {'artifact_path': 'wandb-client-arti... | \n", + "{'seed': 1337, 'delta': 1e-05, 'dpsgd': True, ... | \n", + "dandy-sun-464 | \n", + "
| 3 | \n", + "{'times': {'sha256': 'cb4ef9a38ecaef8e82175a02... | \n", + "{'seed': 1337, 'delta': 1e-05, 'dpsgd': True, ... | \n", + "smart-serenity-463 | \n", + "
| 4 | \n", + "{'_step': 0, 'times': {'size': 233, '_type': '... | \n", + "{'seed': 1337, 'delta': 1e-05, 'dpsgd': True, ... | \n", + "desert-plant-462 | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 290 | \n", + "{'_step': 0, 'times': {'artifact_path': 'wandb... | \n", + "{'K': 0.99, 'N': 50000, 'tag': 'Default', 'tau... | \n", + "peachy-yogurt-5 | \n", + "
| 291 | \n", + "{'_timestamp': 1684003865.3701484, 'median_tim... | \n", + "{'K': 0.99, 'N': 50000, 'tag': 'Default', 'tau... | \n", + "fallen-sea-4 | \n", + "
| 292 | \n", + "{'_step': 0, 'times': {'sha256': '742bef733139... | \n", + "{'K': 0.99, 'N': 50000, 'tag': 'Default', 'tau... | \n", + "stellar-dust-3 | \n", + "
| 293 | \n", + "{'median_time': 153.4468233315274, '_step': 0,... | \n", + "{'K': 0.99, 'N': 50000, 'tag': 'Default', 'tau... | \n", + "brisk-dragon-2 | \n", + "
| 294 | \n", + "{'median_time': 117.60727335698904, '_step': 0... | \n", + "{'K': 0.99, 'N': 50000, 'tag': 'Default', 'tau... | \n", + "apricot-frost-1 | \n", + "
295 rows × 3 columns
\n", + "| \n", + " | _step | \n", + "epsilon | \n", + "accuracy | \n", + "_timestamp | \n", + "val_loss | \n", + "val_epoch | \n", + "loss | \n", + "epoch | \n", + "_runtime | \n", + "delta | \n", + "... | \n", + "EPOCHS | \n", + "EPSILON | \n", + "sweep_id | \n", + "log_wandb | \n", + "BATCH_SIZE | \n", + "MAX_GRAD_NORM | \n", + "sweep_yaml_config | \n", + "MAX_PHYSICAL_BATCH_SIZE | \n", + "num_epochs | \n", + "run_id | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "0 | \n", + "19.990826 | \n", + "0.098041 | \n", + "1.695634e+09 | \n", + "6.353306 | \n", + "1 | \n", + "36.156680 | \n", + "1 | \n", + "52.867483 | \n", + "0.00001 | \n", + "... | \n", + "1 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "2000 | \n", + "0.129804 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "1 | \n", + "6zf3ts8o | \n", + "
| 0 | \n", + "0 | \n", + "15.995284 | \n", + "0.251423 | \n", + "1.695636e+09 | \n", + "1.910835 | \n", + "1 | \n", + "2.096110 | \n", + "1 | \n", + "47.248856 | \n", + "0.00001 | \n", + "... | \n", + "2 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "1000 | \n", + "0.759089 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "2 | \n", + "rxhgpsuc | \n", + "
| 1 | \n", + "1 | \n", + "19.999911 | \n", + "0.346381 | \n", + "1.695636e+09 | \n", + "1.802558 | \n", + "2 | \n", + "1.828817 | \n", + "2 | \n", + "87.922384 | \n", + "0.00001 | \n", + "... | \n", + "2 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "1000 | \n", + "0.759089 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "2 | \n", + "rxhgpsuc | \n", + "
| 0 | \n", + "0 | \n", + "15.995284 | \n", + "0.102954 | \n", + "1.695637e+09 | \n", + "2.411215 | \n", + "1 | \n", + "3.859198 | \n", + "1 | \n", + "47.313033 | \n", + "0.00001 | \n", + "... | \n", + "2 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "1000 | \n", + "2.136437 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "2 | \n", + "crjs7cyj | \n", + "
| 1 | \n", + "1 | \n", + "19.999911 | \n", + "0.188909 | \n", + "1.695637e+09 | \n", + "1.911932 | \n", + "2 | \n", + "2.205820 | \n", + "2 | \n", + "87.703218 | \n", + "0.00001 | \n", + "... | \n", + "2 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "1000 | \n", + "2.136437 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "2 | \n", + "crjs7cyj | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 370 | \n", + "370 | \n", + "19.857968 | \n", + "0.543862 | \n", + "1.695692e+09 | \n", + "1.902890 | \n", + "371 | \n", + "1.709858 | \n", + "371 | \n", + "22835.259901 | \n", + "0.00001 | \n", + "... | \n", + "375 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "500 | \n", + "0.303976 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "375 | \n", + "5kzoad2m | \n", + "
| 371 | \n", + "371 | \n", + "19.892148 | \n", + "0.536630 | \n", + "1.695692e+09 | \n", + "1.898140 | \n", + "372 | \n", + "1.743008 | \n", + "372 | \n", + "22926.907113 | \n", + "0.00001 | \n", + "... | \n", + "375 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "500 | \n", + "0.303976 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "375 | \n", + "5kzoad2m | \n", + "
| 372 | \n", + "372 | \n", + "19.926302 | \n", + "0.538424 | \n", + "1.695692e+09 | \n", + "1.902564 | \n", + "373 | \n", + "1.741843 | \n", + "373 | \n", + "23013.055382 | \n", + "0.00001 | \n", + "... | \n", + "375 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "500 | \n", + "0.303976 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "375 | \n", + "5kzoad2m | \n", + "
| 373 | \n", + "373 | \n", + "19.960432 | \n", + "0.540294 | \n", + "1.695692e+09 | \n", + "1.904188 | \n", + "374 | \n", + "1.727001 | \n", + "374 | \n", + "23098.732067 | \n", + "0.00001 | \n", + "... | \n", + "375 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "500 | \n", + "0.303976 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "375 | \n", + "5kzoad2m | \n", + "
| 374 | \n", + "374 | \n", + "19.994536 | \n", + "0.539051 | \n", + "1.695692e+09 | \n", + "1.908609 | \n", + "375 | \n", + "1.730211 | \n", + "375 | \n", + "23189.756623 | \n", + "0.00001 | \n", + "... | \n", + "375 | \n", + "20 | \n", + "\n", + " | sweep_cifar10 | \n", + "500 | \n", + "0.303976 | \n", + "experiments/paper_plots/opacus_cifar10.yaml | \n", + "200 | \n", + "375 | \n", + "5kzoad2m | \n", + "
1702 rows × 25 columns
\n", + "| \n", + " | _step | \n", + "loss | \n", + "_runtime | \n", + "delta | \n", + "accuracy | \n", + "epoch | \n", + "epsilon | \n", + "val_epoch | \n", + "val_accuracy | \n", + "_timestamp | \n", + "... | \n", + "log_wandb | \n", + "BATCH_SIZE | \n", + "sweep_count | \n", + "dataset_name | \n", + "MAX_GRAD_NORM | \n", + "sweep_yaml_config | \n", + "MAX_PHYSICAL_BATCH_SIZE | \n", + "num_epochs | \n", + "runtime | \n", + "Algorithm | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "0 | \n", + "0.141726 | \n", + "13.243446 | \n", + "0.00001 | \n", + "0.067458 | \n", + "1 | \n", + "0.994203 | \n", + "1 | \n", + "0.002921 | \n", + "1.695826e+09 | \n", + "... | \n", + "sweep_celeba | \n", + "500 | \n", + "1 | \n", + "8_celeba | \n", + "15.641288 | \n", + "experiments/paper_plots/opacus_tabular_celeba.... | \n", + "2000 | \n", + "1 | \n", + "13.243446 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 0 | \n", + "0 | \n", + "0.282124 | \n", + "6.706519 | \n", + "0.00001 | \n", + "0.332581 | \n", + "1 | \n", + "0.990906 | \n", + "1 | \n", + "0.048418 | \n", + "1.695818e+09 | \n", + "... | \n", + "sweep_celeba | \n", + "4000 | \n", + "1 | \n", + "8_celeba | \n", + "0.212050 | \n", + "experiments/paper_plots/opacus_tabular_celeba.... | \n", + "2000 | \n", + "1 | \n", + "6.706519 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 0 | \n", + "0 | \n", + "134.203330 | \n", + "5.134475 | \n", + "0.00001 | \n", + "0.119470 | \n", + "1 | \n", + "0.992871 | \n", + "1 | \n", + "0.009124 | \n", + "1.695811e+09 | \n", + "... | \n", + "sweep_ALOI | \n", + "5000 | \n", + "1 | \n", + "1_ALOI | \n", + "0.073162 | \n", + "experiments/paper_plots/opacus_tabular_ALOI.yaml | \n", + "2000 | \n", + "1 | \n", + "5.134475 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 0 | \n", + "0 | \n", + "44.203109 | \n", + "8.024379 | \n", + "0.00001 | \n", + "0.003230 | \n", + "1 | \n", + "0.805283 | \n", + "1 | \n", + "0.000145 | \n", + "1.695826e+09 | \n", + "... | \n", + "sweep_celeba | \n", + "1000 | \n", + "1 | \n", + "8_celeba | \n", + "91.954707 | \n", + "experiments/paper_plots/opacus_tabular_celeba.... | \n", + "2000 | \n", + "2 | \n", + "12.918013 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 1 | \n", + "1 | \n", + "12.024511 | \n", + "12.918013 | \n", + "0.00001 | \n", + "0.000199 | \n", + "2 | \n", + "0.991248 | \n", + "2 | \n", + "0.000000 | \n", + "1.695826e+09 | \n", + "... | \n", + "sweep_celeba | \n", + "1000 | \n", + "1 | \n", + "8_celeba | \n", + "91.954707 | \n", + "experiments/paper_plots/opacus_tabular_celeba.... | \n", + "2000 | \n", + "2 | \n", + "12.918013 | \n", + "[Unconstrained] DP-SGD | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 181 | \n", + "181 | \n", + "1.707506 | \n", + "87.931256 | \n", + "0.00001 | \n", + "0.013245 | \n", + "182 | \n", + "0.983863 | \n", + "182 | \n", + "0.015058 | \n", + "1.695824e+09 | \n", + "... | \n", + "sweep_magic.gamma | \n", + "2000 | \n", + "1 | \n", + "22_magic.gamma | \n", + "0.070708 | \n", + "experiments/paper_plots/opacus_tabular_magic.yaml | \n", + "2000 | \n", + "186 | \n", + "89.897592 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 182 | \n", + "182 | \n", + "1.725051 | \n", + "88.413407 | \n", + "0.00001 | \n", + "0.013326 | \n", + "183 | \n", + "0.986790 | \n", + "183 | \n", + "0.009371 | \n", + "1.695824e+09 | \n", + "... | \n", + "sweep_magic.gamma | \n", + "2000 | \n", + "1 | \n", + "22_magic.gamma | \n", + "0.070708 | \n", + "experiments/paper_plots/opacus_tabular_magic.yaml | \n", + "2000 | \n", + "186 | \n", + "89.897592 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 183 | \n", + "183 | \n", + "1.915484 | \n", + "88.939903 | \n", + "0.00001 | \n", + "0.014903 | \n", + "184 | \n", + "0.989710 | \n", + "184 | \n", + "0.008823 | \n", + "1.695824e+09 | \n", + "... | \n", + "sweep_magic.gamma | \n", + "2000 | \n", + "1 | \n", + "22_magic.gamma | \n", + "0.070708 | \n", + "experiments/paper_plots/opacus_tabular_magic.yaml | \n", + "2000 | \n", + "186 | \n", + "89.897592 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 184 | \n", + "184 | \n", + "2.160897 | \n", + "89.360624 | \n", + "0.00001 | \n", + "0.015205 | \n", + "185 | \n", + "0.992623 | \n", + "185 | \n", + "0.006242 | \n", + "1.695824e+09 | \n", + "... | \n", + "sweep_magic.gamma | \n", + "2000 | \n", + "1 | \n", + "22_magic.gamma | \n", + "0.070708 | \n", + "experiments/paper_plots/opacus_tabular_magic.yaml | \n", + "2000 | \n", + "186 | \n", + "89.897592 | \n", + "[Unconstrained] DP-SGD | \n", + "
| 185 | \n", + "185 | \n", + "1.716604 | \n", + "89.897592 | \n", + "0.00001 | \n", + "0.013043 | \n", + "186 | \n", + "0.995529 | \n", + "186 | \n", + "0.014555 | \n", + "1.695824e+09 | \n", + "... | \n", + "sweep_magic.gamma | \n", + "2000 | \n", + "1 | \n", + "22_magic.gamma | \n", + "0.070708 | \n", + "experiments/paper_plots/opacus_tabular_magic.yaml | \n", + "2000 | \n", + "186 | \n", + "89.897592 | \n", + "[Unconstrained] DP-SGD | \n", + "
2904 rows × 32 columns
\n", + "| \n", + " | \n", + " | val_auc | \n", + "runtime | \n", + "||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| \n", + " | \n", + " | count | \n", + "mean | \n", + "std | \n", + "min | \n", + "25% | \n", + "50% | \n", + "75% | \n", + "max | \n", + "count | \n", + "mean | \n", + "std | \n", + "min | \n", + "25% | \n", + "50% | \n", + "75% | \n", + "max | \n", + "
| dataset_name | \n", + "Algorithm | \n", + "\n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " |
| 11_donors | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "164400.0 | \n", + "96.451578 | \n", + "9.113185 | \n", + "17.639668 | \n", + "99.460174 | \n", + "99.996766 | \n", + "99.999827 | \n", + "100.000000 | \n", + "164400.0 | \n", + "61407.731995 | \n", + "65441.461169 | \n", + "2482.987499 | \n", + "20043.537021 | \n", + "33832.470131 | \n", + "74528.666520 | \n", + "252637.987208 | \n", + "
| [Unconstrained] DP-SGD | \n", + "11500.0 | \n", + "99.703631 | \n", + "2.322741 | \n", + "78.022714 | \n", + "99.997559 | \n", + "99.999858 | \n", + "99.999898 | \n", + "100.000000 | \n", + "11500.0 | \n", + "155162.970586 | \n", + "85635.565808 | \n", + "5844.299769 | \n", + "73720.919514 | \n", + "146913.288975 | \n", + "261562.642336 | \n", + "261562.642336 | \n", + "|
| 1_ALOI | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "23500.0 | \n", + "51.823692 | \n", + "1.798945 | \n", + "44.709831 | \n", + "50.313932 | \n", + "51.813668 | \n", + "53.124815 | \n", + "55.954248 | \n", + "23500.0 | \n", + "1221.679546 | \n", + "651.378530 | \n", + "445.251513 | \n", + "767.002189 | \n", + "1106.218457 | \n", + "1365.052581 | \n", + "3481.084037 | \n", + "
| [Unconstrained] DP-SGD | \n", + "16000.0 | \n", + "51.188402 | \n", + "1.939941 | \n", + "47.029400 | \n", + "49.952227 | \n", + "51.207678 | \n", + "52.224942 | \n", + "56.539985 | \n", + "16000.0 | \n", + "11183.590906 | \n", + "6028.492954 | \n", + "440.257883 | \n", + "4164.996696 | \n", + "15527.475047 | \n", + "15914.127016 | \n", + "15914.127016 | \n", + "|
| 22_magic.gamma | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "1009500.0 | \n", + "82.857822 | \n", + "5.862948 | \n", + "36.686993 | \n", + "81.589234 | \n", + "84.793174 | \n", + "86.414188 | \n", + "89.650577 | \n", + "1009500.0 | \n", + "5948.305362 | \n", + "6491.375994 | \n", + "409.854245 | \n", + "2385.544372 | \n", + "3471.148205 | \n", + "5947.921085 | \n", + "35086.627150 | \n", + "
| [Unconstrained] DP-SGD | \n", + "20800.0 | \n", + "85.941895 | \n", + "5.492180 | \n", + "45.323727 | \n", + "82.023358 | \n", + "86.323582 | \n", + "89.948395 | \n", + "90.565502 | \n", + "20800.0 | \n", + "8358.125399 | \n", + "1711.000937 | \n", + "369.822955 | \n", + "8354.261941 | \n", + "8898.133087 | \n", + "8989.759183 | \n", + "9748.688793 | \n", + "|
| 32_shuttle | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "173600.0 | \n", + "55.800043 | \n", + "30.400548 | \n", + "0.541935 | \n", + "31.391390 | \n", + "59.078997 | \n", + "82.406007 | \n", + "99.320984 | \n", + "173600.0 | \n", + "2224.551282 | \n", + "1375.473827 | \n", + "449.407363 | \n", + "1156.689024 | \n", + "1783.021927 | \n", + "3021.826863 | \n", + "6199.828029 | \n", + "
| [Unconstrained] DP-SGD | \n", + "1500.0 | \n", + "89.333983 | \n", + "22.892754 | \n", + "7.466900 | \n", + "94.076493 | \n", + "94.711019 | \n", + "97.711854 | \n", + "98.276475 | \n", + "1500.0 | \n", + "992.853699 | \n", + "435.060059 | \n", + "411.808443 | \n", + "594.473338 | \n", + "1107.826638 | \n", + "1511.607003 | \n", + "1511.607003 | \n", + "|
| 33_skin | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "774200.0 | \n", + "98.429040 | \n", + "3.957222 | \n", + "7.629676 | \n", + "98.980165 | \n", + "99.390349 | \n", + "99.567267 | \n", + "99.750799 | \n", + "774100.0 | \n", + "10057.989374 | \n", + "8105.187836 | \n", + "776.917553 | \n", + "4437.767076 | \n", + "7315.362215 | \n", + "12576.503825 | \n", + "37232.936525 | \n", + "
| [Unconstrained] DP-SGD | \n", + "6700.0 | \n", + "88.947578 | \n", + "22.669838 | \n", + "21.916921 | \n", + "98.232144 | \n", + "98.598746 | \n", + "98.902363 | \n", + "99.973205 | \n", + "6700.0 | \n", + "47952.599370 | \n", + "27312.004767 | \n", + "1309.671617 | \n", + "9704.866052 | \n", + "65604.450512 | \n", + "65604.450512 | \n", + "65604.450512 | \n", + "|
| 47_yeast | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "437500.0 | \n", + "57.197655 | \n", + "6.753847 | \n", + "24.503489 | \n", + "52.794755 | \n", + "58.035439 | \n", + "62.113485 | \n", + "75.132811 | \n", + "437000.0 | \n", + "657.247671 | \n", + "187.621443 | \n", + "297.493792 | \n", + "509.328812 | \n", + "609.803009 | \n", + "779.524273 | \n", + "1614.569068 | \n", + "
| [Unconstrained] DP-SGD | \n", + "8000.0 | \n", + "60.191832 | \n", + "9.062512 | \n", + "42.872297 | \n", + "59.085169 | \n", + "65.025763 | \n", + "65.746868 | \n", + "66.821580 | \n", + "8000.0 | \n", + "2082.814635 | \n", + "912.533382 | \n", + "324.902797 | \n", + "782.168627 | \n", + "2207.534122 | \n", + "2846.807861 | \n", + "2846.807861 | \n", + "|
| 5_campaign | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "73400.0 | \n", + "73.598445 | \n", + "7.481093 | \n", + "35.363200 | \n", + "70.984790 | \n", + "76.568681 | \n", + "78.681868 | \n", + "82.162070 | \n", + "73400.0 | \n", + "2223.236004 | \n", + "1919.194831 | \n", + "403.837204 | \n", + "998.516965 | \n", + "1371.006036 | \n", + "2689.749789 | \n", + "7941.281772 | \n", + "
| [Unconstrained] DP-SGD | \n", + "10200.0 | \n", + "83.015317 | \n", + "5.386048 | \n", + "60.435855 | \n", + "77.882855 | \n", + "84.261916 | \n", + "87.799651 | \n", + "89.992312 | \n", + "10200.0 | \n", + "10891.452233 | \n", + "6652.474763 | \n", + "424.317646 | \n", + "4314.654589 | \n", + "15563.628340 | \n", + "16885.352564 | \n", + "16885.352564 | \n", + "|
| 8_celeba | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "30600.0 | \n", + "72.493234 | \n", + "14.873687 | \n", + "29.988620 | \n", + "64.967158 | \n", + "70.648220 | \n", + "81.512453 | \n", + "96.499097 | \n", + "30600.0 | \n", + "58119.989446 | \n", + "52693.521208 | \n", + "1133.571005 | \n", + "13389.174640 | \n", + "39128.973794 | \n", + "135084.460163 | \n", + "135084.460163 | \n", + "
| [Unconstrained] DP-SGD | \n", + "5300.0 | \n", + "87.757796 | \n", + "9.015357 | \n", + "60.557364 | \n", + "78.140055 | \n", + "92.812696 | \n", + "94.042255 | \n", + "96.631613 | \n", + "5300.0 | \n", + "12716.799540 | \n", + "9495.836684 | \n", + "670.651889 | \n", + "4107.234979 | \n", + "11284.685779 | \n", + "14371.910644 | \n", + "28718.150520 | \n", + "|
| 9_census | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "27600.0 | \n", + "82.784794 | \n", + "13.853022 | \n", + "50.000000 | \n", + "84.121956 | \n", + "90.317890 | \n", + "91.374253 | \n", + "92.492223 | \n", + "27600.0 | \n", + "29663.313305 | \n", + "36842.442326 | \n", + "3082.371330 | \n", + "8192.256904 | \n", + "10982.626510 | \n", + "26077.858949 | \n", + "104678.196144 | \n", + "
| [Unconstrained] DP-SGD | \n", + "19200.0 | \n", + "88.087718 | \n", + "5.205276 | \n", + "67.565076 | \n", + "85.682278 | \n", + "89.454925 | \n", + "91.742135 | \n", + "93.264390 | \n", + "19200.0 | \n", + "77589.148114 | \n", + "31185.888038 | \n", + "2298.387170 | \n", + "68978.514171 | \n", + "82004.339337 | \n", + "84395.474815 | \n", + "119632.448363 | \n", + "
| \n", + " | \n", + " | val_auc | \n", + "runtime | \n", + "val_auc | \n", + "runtime | \n", + "||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| \n", + " | \n", + " | count | \n", + "mean | \n", + "std | \n", + "min | \n", + "25% | \n", + "50% | \n", + "75% | \n", + "max | \n", + "count | \n", + "mean | \n", + "std | \n", + "min | \n", + "25% | \n", + "50% | \n", + "75% | \n", + "max | \n", + "iqr | \n", + "iqr | \n", + "
| dataset_name | \n", + "Algorithm | \n", + "\n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " | \n", + " |
| 11_donors | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "164400.0 | \n", + "96.451578 | \n", + "9.113185 | \n", + "17.639668 | \n", + "99.460174 | \n", + "99.996766 | \n", + "99.999827 | \n", + "100.000000 | \n", + "164400.0 | \n", + "61407.731995 | \n", + "65441.461169 | \n", + "2482.987499 | \n", + "20043.537021 | \n", + "33832.470131 | \n", + "74528.666520 | \n", + "252637.987208 | \n", + "0.539653 | \n", + "54485.129499 | \n", + "
| [Unconstrained] DP-SGD | \n", + "11500.0 | \n", + "99.703631 | \n", + "2.322741 | \n", + "78.022714 | \n", + "99.997559 | \n", + "99.999858 | \n", + "99.999898 | \n", + "100.000000 | \n", + "11500.0 | \n", + "155162.970586 | \n", + "85635.565808 | \n", + "5844.299769 | \n", + "73720.919514 | \n", + "146913.288975 | \n", + "261562.642336 | \n", + "261562.642336 | \n", + "0.002338 | \n", + "187841.722822 | \n", + "|
| 1_ALOI | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "23500.0 | \n", + "51.823692 | \n", + "1.798945 | \n", + "44.709831 | \n", + "50.313932 | \n", + "51.813668 | \n", + "53.124815 | \n", + "55.954248 | \n", + "23500.0 | \n", + "1221.679546 | \n", + "651.378530 | \n", + "445.251513 | \n", + "767.002189 | \n", + "1106.218457 | \n", + "1365.052581 | \n", + "3481.084037 | \n", + "2.810884 | \n", + "598.050392 | \n", + "
| [Unconstrained] DP-SGD | \n", + "16000.0 | \n", + "51.188402 | \n", + "1.939941 | \n", + "47.029400 | \n", + "49.952227 | \n", + "51.207678 | \n", + "52.224942 | \n", + "56.539985 | \n", + "16000.0 | \n", + "11183.590906 | \n", + "6028.492954 | \n", + "440.257883 | \n", + "4164.996696 | \n", + "15527.475047 | \n", + "15914.127016 | \n", + "15914.127016 | \n", + "2.272715 | \n", + "11749.130321 | \n", + "|
| 22_magic.gamma | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "1009500.0 | \n", + "82.857822 | \n", + "5.862948 | \n", + "36.686993 | \n", + "81.589234 | \n", + "84.793174 | \n", + "86.414188 | \n", + "89.650577 | \n", + "1009500.0 | \n", + "5948.305362 | \n", + "6491.375994 | \n", + "409.854245 | \n", + "2385.544372 | \n", + "3471.148205 | \n", + "5947.921085 | \n", + "35086.627150 | \n", + "4.824954 | \n", + "3562.376714 | \n", + "
| [Unconstrained] DP-SGD | \n", + "20800.0 | \n", + "85.941895 | \n", + "5.492180 | \n", + "45.323727 | \n", + "82.023358 | \n", + "86.323582 | \n", + "89.948395 | \n", + "90.565502 | \n", + "20800.0 | \n", + "8358.125399 | \n", + "1711.000937 | \n", + "369.822955 | \n", + "8354.261941 | \n", + "8898.133087 | \n", + "8989.759183 | \n", + "9748.688793 | \n", + "7.925038 | \n", + "635.497242 | \n", + "|
| 32_shuttle | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "173600.0 | \n", + "55.800043 | \n", + "30.400548 | \n", + "0.541935 | \n", + "31.391390 | \n", + "59.078997 | \n", + "82.406007 | \n", + "99.320984 | \n", + "173600.0 | \n", + "2224.551282 | \n", + "1375.473827 | \n", + "449.407363 | \n", + "1156.689024 | \n", + "1783.021927 | \n", + "3021.826863 | \n", + "6199.828029 | \n", + "51.014616 | \n", + "1865.137839 | \n", + "
| [Unconstrained] DP-SGD | \n", + "1500.0 | \n", + "89.333983 | \n", + "22.892754 | \n", + "7.466900 | \n", + "94.076493 | \n", + "94.711019 | \n", + "97.711854 | \n", + "98.276475 | \n", + "1500.0 | \n", + "992.853699 | \n", + "435.060059 | \n", + "411.808443 | \n", + "594.473338 | \n", + "1107.826638 | \n", + "1511.607003 | \n", + "1511.607003 | \n", + "3.635361 | \n", + "917.133665 | \n", + "|
| 33_skin | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "774200.0 | \n", + "98.429040 | \n", + "3.957222 | \n", + "7.629676 | \n", + "98.980165 | \n", + "99.390349 | \n", + "99.567267 | \n", + "99.750799 | \n", + "774100.0 | \n", + "10057.989374 | \n", + "8105.187836 | \n", + "776.917553 | \n", + "4437.767076 | \n", + "7315.362215 | \n", + "12576.503825 | \n", + "37232.936525 | \n", + "0.587103 | \n", + "8138.736749 | \n", + "
| [Unconstrained] DP-SGD | \n", + "6700.0 | \n", + "88.947578 | \n", + "22.669838 | \n", + "21.916921 | \n", + "98.232144 | \n", + "98.598746 | \n", + "98.902363 | \n", + "99.973205 | \n", + "6700.0 | \n", + "47952.599370 | \n", + "27312.004767 | \n", + "1309.671617 | \n", + "9704.866052 | \n", + "65604.450512 | \n", + "65604.450512 | \n", + "65604.450512 | \n", + "0.670218 | \n", + "55899.584460 | \n", + "|
| 47_yeast | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "437500.0 | \n", + "57.197655 | \n", + "6.753847 | \n", + "24.503489 | \n", + "52.794755 | \n", + "58.035439 | \n", + "62.113485 | \n", + "75.132811 | \n", + "437000.0 | \n", + "657.247671 | \n", + "187.621443 | \n", + "297.493792 | \n", + "509.328812 | \n", + "609.803009 | \n", + "779.524273 | \n", + "1614.569068 | \n", + "9.318730 | \n", + "270.195460 | \n", + "
| [Unconstrained] DP-SGD | \n", + "8000.0 | \n", + "60.191832 | \n", + "9.062512 | \n", + "42.872297 | \n", + "59.085169 | \n", + "65.025763 | \n", + "65.746868 | \n", + "66.821580 | \n", + "8000.0 | \n", + "2082.814635 | \n", + "912.533382 | \n", + "324.902797 | \n", + "782.168627 | \n", + "2207.534122 | \n", + "2846.807861 | \n", + "2846.807861 | \n", + "6.661699 | \n", + "2064.639235 | \n", + "|
| 5_campaign | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "73400.0 | \n", + "73.598445 | \n", + "7.481093 | \n", + "35.363200 | \n", + "70.984790 | \n", + "76.568681 | \n", + "78.681868 | \n", + "82.162070 | \n", + "73400.0 | \n", + "2223.236004 | \n", + "1919.194831 | \n", + "403.837204 | \n", + "998.516965 | \n", + "1371.006036 | \n", + "2689.749789 | \n", + "7941.281772 | \n", + "7.697079 | \n", + "1691.232824 | \n", + "
| [Unconstrained] DP-SGD | \n", + "10200.0 | \n", + "83.015317 | \n", + "5.386048 | \n", + "60.435855 | \n", + "77.882855 | \n", + "84.261916 | \n", + "87.799651 | \n", + "89.992312 | \n", + "10200.0 | \n", + "10891.452233 | \n", + "6652.474763 | \n", + "424.317646 | \n", + "4314.654589 | \n", + "15563.628340 | \n", + "16885.352564 | \n", + "16885.352564 | \n", + "9.916797 | \n", + "12570.697975 | \n", + "|
| 8_celeba | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "30600.0 | \n", + "72.493234 | \n", + "14.873687 | \n", + "29.988620 | \n", + "64.967158 | \n", + "70.648220 | \n", + "81.512453 | \n", + "96.499097 | \n", + "30600.0 | \n", + "58119.989446 | \n", + "52693.521208 | \n", + "1133.571005 | \n", + "13389.174640 | \n", + "39128.973794 | \n", + "135084.460163 | \n", + "135084.460163 | \n", + "16.545294 | \n", + "121695.285523 | \n", + "
| [Unconstrained] DP-SGD | \n", + "5300.0 | \n", + "87.757796 | \n", + "9.015357 | \n", + "60.557364 | \n", + "78.140055 | \n", + "92.812696 | \n", + "94.042255 | \n", + "96.631613 | \n", + "5300.0 | \n", + "12716.799540 | \n", + "9495.836684 | \n", + "670.651889 | \n", + "4107.234979 | \n", + "11284.685779 | \n", + "14371.910644 | \n", + "28718.150520 | \n", + "15.902200 | \n", + "10264.675665 | \n", + "|
| 9_census | \n", + "[Lipschitz] Clipless DP-SGD | \n", + "27600.0 | \n", + "82.784794 | \n", + "13.853022 | \n", + "50.000000 | \n", + "84.121956 | \n", + "90.317890 | \n", + "91.374253 | \n", + "92.492223 | \n", + "27600.0 | \n", + "29663.313305 | \n", + "36842.442326 | \n", + "3082.371330 | \n", + "8192.256904 | \n", + "10982.626510 | \n", + "26077.858949 | \n", + "104678.196144 | \n", + "7.252297 | \n", + "17885.602045 | \n", + "
| [Unconstrained] DP-SGD | \n", + "19200.0 | \n", + "88.087718 | \n", + "5.205276 | \n", + "67.565076 | \n", + "85.682278 | \n", + "89.454925 | \n", + "91.742135 | \n", + "93.264390 | \n", + "19200.0 | \n", + "77589.148114 | \n", + "31185.888038 | \n", + "2298.387170 | \n", + "68978.514171 | \n", + "82004.339337 | \n", + "84395.474815 | \n", + "119632.448363 | \n", + "6.059857 | \n", + "15416.960645 | \n", + "