Skip to content

Commit

Permalink
workspace_creator renamed to workspace_builder
Browse files Browse the repository at this point in the history 
finding values without help of any tags

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
  • Loading branch information
@ParthM-GitHub
ParthM-GitHub committed Sep 28, 2023
1 parent 20068be commit dba24af
Show file tree
Hide file tree
Showing 11 changed files with 1,192 additions and 177 deletions.
150 changes: 104 additions & 46 deletions openfl-tutorials/experimental/Workflow_Interface_101_MNIST-Test-Approach1.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
},
{
"cell_type": "markdown",
"id": "9819a58c",
"id": "f450bfaa",
"metadata": {},
"source": [
"Before we start the experiment, let's write some `nbdev` tags which enable us to convert this notebook to aggregator-based-workspace workflow.\n",
Expand All @@ -58,6 +58,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Add markdowns for cells like these\n",
"#| default_exp experiment"
]
},
Expand All @@ -71,22 +72,22 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6c45e1ca",
"id": "4dbb89b6",
"metadata": {},
"source": [
"- `#| export` specifies that code in this cell is to extracted into python script.\n",
"- `# pip requirements STARTS` specifies where requirements.txt file begins.\n",
"- `# pip requirements ENDS` specifies where requirements.txt file ends."
"First we start by installing the necessary dependencies for the workflow interface"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4dbb89b6",
"id": "5b7a3948",
"metadata": {},
"source": [
"First we start by installing the necessary dependencies for the workflow interface"
"- `#| export` specifies that code in this cell is to extracted into python script.\n",
"- `# export requirements-start` specifies where requirements.txt file begins.\n",
"- `# export requirements-ends` specifies where requirements.txt file ends."
]
},
{
Expand All @@ -96,30 +97,31 @@
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"#| export requirements\n",
"# export requirements-start\n",
"\n",
"# pip requirements STARTS\n",
"# !pip install git+https://github.com/intel/openfl.git\n",
"# !pip install -r requirements_workflow_interface.txt\n",
"# !pip install torch\n",
"# !pip install torchvision\n",
"# pip requirements ENDS\n",
"!pip install git+https://github.com/intel/openfl.git\n",
"!pip install -r requirements_workflow_interface.txt\n",
"!pip install torch\n",
"!pip install torchvision\n",
"\n",
"# Uncomment this if running in Google Colab\n",
"# !pip install -r https://raw.githubusercontent.com/intel/openfl/develop/openfl-tutorials/experimental/requirements_workflow_interface.txt\n",
"# import os\n",
"# os.environ[\"USERNAME\"] = \"colab\""
"# os.environ[\"USERNAME\"] = \"colab\"\n",
"\n",
"# export requirements-end"
]
},
{
"cell_type": "markdown",
"id": "21699cbf",
"id": "3184ac5c",
"metadata": {},
"source": [
"- `# Collaborator private attributes STARTS` specifies where collaborator_private_attributes.py begings.\n",
"- `# Collaborator private attributes ENDS` specifies where collaborator_private_attributes.py ends.\n",
"- `# Flow Class STARTS` specifies where flow.py begins.\n",
"- `# Flow Class ENDS` specifies where flow.py ends."
"- `# export collaborator-privatea-attributes-start` specifies where collaborator_private_attributes.py begings.\n",
"- `# export collaborator-privatea-attributes-ends` specifies where collaborator_private_attributes.py ends.\n",
"- `# export flow-class-start` specifies where flow.py begins.\n",
"- `# export flow-class-ends` specifies where flow.py ends."
]
},
{
Expand All @@ -134,22 +136,35 @@
{
"cell_type": "code",
"execution_count": null,
"id": "7e85e030",
"id": "7c2de9c7",
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"#| export collaborator_private_attributes\n",
"#| export flow\n",
"\n",
"# Collaborator private attributes STARTS\n",
"# Flow Class STARTS\n",
"# export collaborator-privatea-attributes-start\n",
"# export flow-class-start\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import torch.optim as optim\n",
"import torch\n",
"import torchvision\n",
"import numpy as np\n",
"# Flow Class ENDS\n",
"# export flow-class-end\n",
"# export collaborator-privatea-attributes-end"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1d163c1b",
"metadata": {},
"outputs": [],
"source": [
"#| export collaborator_private_attributes\n",
"\n",
"# export collaborator-privatea-attributes-start\n",
"n_epochs = 3\n",
"batch_size_train = 64\n",
"batch_size_test = 1000\n",
Expand Down Expand Up @@ -184,8 +199,19 @@
" ]\n",
" ),\n",
")\n",
"# Collaborator private attributes ENDS\n",
"# Flow Class STARTS\n",
"# export collaborator-privatea-attributes-end"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e85e030",
"metadata": {},
"outputs": [],
"source": [
"#| export flow\n",
"\n",
"# export flow-class-start\n",
"class Net(nn.Module):\n",
" def __init__(self):\n",
" super(Net, self).__init__()\n",
Expand Down Expand Up @@ -220,7 +246,7 @@
" 100. * correct / len(test_loader.dataset)))\n",
" accuracy = float(correct / len(test_loader.dataset))\n",
" return accuracy\n",
"# Flow Class ENDS"
"# export flow-class-end"
]
},
{
Expand All @@ -243,9 +269,9 @@
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"#| export flow\n",
"\n",
"# Flow Class STARTS\n",
"# export flow-class-start\n",
"from copy import deepcopy\n",
"\n",
"from openfl.experimental.interface import FLSpec, Aggregator, Collaborator\n",
Expand All @@ -261,7 +287,7 @@
" for state in state_dicts], dtype=object), axis=0) / len(models)\n",
" new_model.load_state_dict(state_dict)\n",
" return new_model\n",
"# Flow Class ENDS"
"# export flow-class-end"
]
},
{
Expand All @@ -288,9 +314,9 @@
"metadata": {},
"outputs": [],
"source": [
"#| export\n",
"#| export flow\n",
"\n",
"# Flow Class STARTS\n",
"# export flow-class-start\n",
"class FederatedFlow(FLSpec):\n",
"\n",
" def __init__(self, model=None, optimizer=None, rounds=3, **kwargs):\n",
Expand Down Expand Up @@ -370,16 +396,16 @@
" @aggregator\n",
" def end(self):\n",
" print(f'This is the end of the flow')\n",
"# Flow Class ENDS"
"# export flow-class-end"
]
},
{
"cell_type": "markdown",
"id": "6b182515",
"id": "b737da9e",
"metadata": {},
"source": [
"- In this example there are not aggregator private attributes but we can specify the same with `# Aggregator private attributes STARTS/ENDS` tags.\n",
"- `# n Collaborators STARTS` specifies list of collaborator names, this is required to build data.yaml file."
"- In this example there are not aggregator private attributes but we can specify the same with `# export aggregator-private-attributes-start/end` tags.\n",
"- `# n-collaborators-start/ends` specifies list of collaborator names, this is required to build data.yaml file."
]
},
{
Expand All @@ -404,11 +430,21 @@
"\n",
"aggregator_ = Aggregator()\n",
"\n",
"# n Collaborators STARTS\n",
"# n-collaborators-start\n",
"collaborator_names = [\"Portland\", \"Seattle\", \"Chandler\", \"Bangalore\"]\n",
"# n Collaborators ENDS\n",
"# n-collaborators-end"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c946da80",
"metadata": {},
"outputs": [],
"source": [
"#| export collaborator_private_attributes\n",
"\n",
"# Collaborator private attributes STARTS\n",
"# export collaborator-privatea-attributes-start\n",
"def callable_to_initialize_collaborator_private_attributes(index, n_collaborators, batch_size, train_dataset, test_dataset):\n",
" train = deepcopy(train_dataset)\n",
" test = deepcopy(test_dataset)\n",
Expand All @@ -421,7 +457,17 @@
" \"train_loader\": torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True),\n",
" \"test_loader\": torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=True),\n",
" }\n",
"# Collaborator private attributes ENDS\n",
"# export collaborator-private-attributes-end"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9825b9b0",
"metadata": {},
"outputs": [],
"source": [
"#| export \n",
"\n",
"collaborators = []\n",
"for idx, collaborator_name in enumerate(collaborator_names):\n",
Expand All @@ -441,10 +487,10 @@
},
{
"cell_type": "markdown",
"id": "428b0257",
"id": "6c165b11",
"metadata": {},
"source": [
"- `# FLSpec object STARTS` specifies object creation of flow class, this information will be used to build federated_flow section in plan.yaml."
"- `# flspec-object-start/ends` specifies object creation of flow class, this information will be used to build federated_flow section in plan.yaml."
]
},
{
Expand All @@ -465,16 +511,28 @@
"source": [
"#| export\n",
"\n",
"# FLSpec object STARTS\n",
"# flspec-object-start\n",
"model = None\n",
"best_model = None\n",
"optimizer = None\n",
"flflow = FederatedFlow(model, optimizer, checkpoint=True)\n",
"# FLSpec object ENDS\n",
"# flspec-object-end\n",
"flflow.runtime = local_runtime\n",
"flflow.run()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e40fd1cf",
"metadata": {},
"outputs": [],
"source": [
"from nbdev.export import nb_export\n",
"\n",
"nb_export(\"Workflow_Interface_101_MNIST-Test-Approach2.ipynb\", \".\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
Expand Down
Loading

0 comments on commit dba24af

Please sign in to comment.