Skip to content
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.

Commit

Permalink
switched to ACR instead of docker hub (#4)
Browse files Browse the repository at this point in the history 
* sdk.v1.0.69, plus switched to ACR push. ACR pull coming next

* full acr use, push and pull, and use in Estimator

* temp fix for dcker image bug

* fixed the az acr login --username and --password issue

* full switch to ACR for docker image storage
  • Loading branch information
@georgeAccnt-GH
georgeAccnt-GH authored Nov 7, 2019
1 parent 93ba611 commit 16652c7
Show file tree
Hide file tree
Showing 4 changed files with 680 additions and 2,167 deletions.
245 changes: 222 additions & 23 deletions ...ples/imaging/azureml_devito/notebooks/000_Setup_GeophysicsTutorial_FWI_Azure_devito.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
"User input requiring steps:\n",
" - [Fill in and save sensitive information](#dot_env_description)\n",
" - [Azure login](#Azure_login) (may be required first time the notebook is run) \n",
" - [Set __create_ACR_FLAG__ to true to trigger ACR creation and to save of ACR login info](#set_create_ACR_flag)\n",
" - [Azure CLI login ](#Azure_cli_login) (may be required once to create an [ACR](https://azure.microsoft.com/en-us/services/container-registry/)) \n",
"\n"
]
},
Expand Down Expand Up @@ -53,7 +55,7 @@
"from azureml.core import Workspace, Experiment\n",
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
"from azureml.core.compute_target import ComputeTargetException\n",
"import platform"
"import platform, dotenv"
]
},
{
Expand All @@ -65,13 +67,13 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Azure ML SDK Version: 1.0.65\n"
"Azure ML SDK Version: 1.0.69\n"
]
},
{
"data": {
"text/plain": [
"'Linux-4.15.0-1060-azure-x86_64-with-debian-10.0'"
"'Linux-4.15.0-1061-azure-x86_64-with-debian-10.0'"
]
},
"execution_count": 3,
Expand Down Expand Up @@ -266,7 +268,7 @@
"\n",
"\n",
"import project_utils\n",
"prj_consts = project_utils.project_consts()"
"prj_consts = project_utils.project_consts()\n"
]
},
{
Expand All @@ -282,11 +284,37 @@
"* The notebook tries to find and use an existing Azure resource group (rsg) defined by __crt_resource_group__. It creates a new one if needed. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='set_create_ACR_flag'></a>\n",
"\n",
"##### Create [ACR]() first time this notebook is run. \n",
"Either docker hub or ACR can be used to store the experimentation image. To create the ACR, set: \n",
"```\n",
"create_ACR_FLAG=True \n",
"```\n",
"It will create an ACR by running severral steps described below in section 2.7. __Create an [ACR]__ \n",
" \n",
" \n",
"[Back](#user_input_requiring_steps) to summary of user input requiring steps."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"create_ACR_FLAG = False #True False"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"sensitive_info = {}"
]
Expand All @@ -296,7 +324,6 @@
"metadata": {},
"source": [
"<a id='dot_env_description'></a>\n",
"\n",
"##### 2.1. Input here sensitive and configuration information\n",
"[dotenv](https://github.com/theskumar/python-dotenv) is used to hide sensitive info, like Azure subscription name/ID. The serialized info needs to be manually input once. \n",
" \n",
Expand All @@ -317,7 +344,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -328,16 +355,20 @@
"# gpu_cluster_name = \"gpuclstfwi02\"\n",
"# gpucluster_admin_user_name = \"\"\n",
"# gpucluster_admin_user_password = \"\"\n",
"\n",
"# experimentation_docker_image_name = \"fwi01_azureml\"\n",
"# experimentation_docker_image_tag = \"sdk.v1.0.60\"\n",
"# docker_container_mount_point = '/datadrive01/prj/DeepSeismic/fwi' # use project directory or a subdirectory\n",
"\n",
"# docker_login = \"georgedockeraccount\"\n",
"# docker_pwd = \"\"\n",
"# experimentation_image_tag = \"fwi01_azureml\"\n",
"# experimentation_image_version = \"sdk.v1.0.60\"\n",
"# docker_container_mount_point = '/datadrive01/prj/DeepSeismic/fwi' # use project directory or a subdirectory"
"\n",
"# acr_name=\"fwi01acr\""
]
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -349,11 +380,12 @@
"# 'GPU_CLUSTER_NAME':gpu_cluster_name,\n",
"# 'GPU_CLUSTER_ADMIN_USER_NAME':gpucluster_admin_user_name,\n",
"# 'GPU_CLUSTER_ADMIN_USER_PASSWORD':gpucluster_admin_user_password,\n",
"# 'EXPERIMENTATION_DOCKER_IMAGE_NAME':experimentation_docker_image_name,\n",
"# 'EXPERIMENTATION_DOCKER_IMAGE_TAG':experimentation_docker_image_tag,\n",
"# 'DOCKER_CONTAINER_MOUNT_POINT':docker_container_mount_point,\n",
"# 'DOCKER_LOGIN':docker_login,\n",
"# 'DOCKER_PWD':docker_pwd,\n",
"# 'EXPERIMENTATION_IMAGE_TAG':experimentation_image_tag,\n",
"# 'EXPERIMENTATION_IMAGE_VERSION':experimentation_image_version,\n",
"# 'DOCKER_CONTAINER_MOUNT_POINT':docker_container_mount_point\n",
"# 'ACR_NAME':acr_name\n",
"# }"
]
},
Expand All @@ -368,7 +400,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 11,
"metadata": {},
"outputs": [
{
Expand All @@ -377,7 +409,7 @@
"'./../not_shared/general.env'"
]
},
"execution_count": 10,
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -405,7 +437,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -439,7 +471,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -462,7 +494,7 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 14,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -518,7 +550,7 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -540,7 +572,7 @@
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 16,
"metadata": {},
"outputs": [
{
Expand All @@ -549,7 +581,7 @@
"'gpuclstfwi02'"
]
},
"execution_count": 15,
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -561,7 +593,7 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 17,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -594,9 +626,176 @@
"# gpu_cluster.wait_for_completion(show_output=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### 2.7. Create an [ACR](https://docs.microsoft.com/en-us/azure/container-registry/) if you have not done so using the [portal](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal) \n",
" - Follow the 4 ACR steps described below. \n",
" - Uncomment cells' lines as needed to login and see commands responses while you set the right subscription and then create the ACR. \n",
" - You need [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) to run the commands below. \n",
"\n",
"<a id='Azure_cli_login'></a>\n",
"##### ACR Step 1. Select ACR subscription (az cli login into Azure may be required here)\n",
"[Back](#user_input_requiring_steps) to summary of user input requiring steps."
]
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"azure-cli 2.0.75\r\n",
"\r\n",
"command-modules-nspkg 2.0.3\r\n",
"core 2.0.75\r\n",
"nspkg 3.0.4\r\n",
"telemetry 1.0.4\r\n",
"\r\n",
"Python location '/opt/az/bin/python3'\r\n",
"Extensions directory '/root/.azure/cliextensions'\r\n",
"\r\n",
"Python (Linux) 3.6.5 (default, Oct 11 2019, 09:04:03) \r\n",
"[GCC 6.3.0 20170516]\r\n",
"\r\n",
"Legal docs and information: aka.ms/AzureCliLegal\r\n",
"\r\n",
"\r\n",
"Your CLI is up-to-date.\r\n"
]
}
],
"source": [
"!az --version\n",
"if create_ACR_FLAG:\n",
" !az login\n",
" response01 = ! az account list --all --refresh -o table\n",
" response02 = ! az account set --subscription $subscription_id\n",
" response03 = ! az account list -o table\n",
" response04 = ! $cli_command\n",
"\n",
" response01\n",
" response02\n",
" response03\n",
" response04"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### ACR Step 2. Create the ACR"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'az acr create --resource-group ghiordanfwirsg01 --name fwi01acr --sku Basic'"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%dotenv $dotenv_file_path\n",
"acr_name = os.getenv('ACR_NAME')\n",
"\n",
"cli_command='az acr create --resource-group '+ crt_resource_group +' --name ' + acr_name + ' --sku Basic'\n",
"cli_command\n",
"\n",
"if create_ACR_FLAG:\n",
" !$cli_command"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### ACR Step 3. Also enable password and login via __ [--admin-enabled true](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication) __ and then use the az cli or portal to set up the credentials"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'az acr update -n fwi01acr --admin-enabled true'"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# per https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication\n",
"cli_command='az acr update -n '+acr_name+' --admin-enabled true'\n",
"cli_command\n",
"\n",
"if create_ACR_FLAG:\n",
" response = !$cli_command\n",
" response"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### ACR Step 4. Save the ACR password and login"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"if create_ACR_FLAG:\n",
" import subprocess\n",
" cli_command = 'az acr credential show -n '+acr_name\n",
"\n",
" acr_username = subprocess.Popen(cli_command+' --query username',shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE).\\\n",
" communicate()[0].decode(\"utf-8\").split()[0].strip('\\\"')\n",
"\n",
" acr_password = subprocess.Popen(cli_command+' --query passwords[0].value',shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE).\\\n",
" communicate()[0].decode(\"utf-8\").split()[0].strip('\\\"')\n",
"\n",
" dotenv.set_key(dotenv_file_path, 'ACR_PASSWORD', acr_password)\n",
" dotenv.set_key(dotenv_file_path, 'ACR_USERNAME', acr_username)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"%reload_ext dotenv\n",
"%dotenv -o $dotenv_file_path\n",
"\n",
"# print acr password and login info saved in dotenv file\n",
"if create_ACR_FLAG:\n",
" os.getenv('ACR_PASSWORD')\n",
" os.getenv('ACR_USERNAME')"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
Expand Down
Loading

0 comments on commit 16652c7

Please sign in to comment.