PermissionError: [Errno 13] Permission denied: '/root/.azureml' with Azure Databricks High Concurrency/Passthrough

[krisbock]

• Package Name: azureml-sdk[databricks]
• Package Version: SDK version: 1.0.85
• Operating System: Azure Databricks
• Python Version:

Describe the bug
Unable to initialise Azure ML Workspace in Azure Databricks (ADB) notebook, using ADB High Concurrency with Passthrough enabled. Standard clusters with/without passthrough work fine, as does High concurrency without passthrough.

To Reproduce
Steps to reproduce the behavior:

1. Create ADB Workspace.
2. Create ADB Cluster - High Concurrency, Passthrough checked.
3. Install azureml-sdk[databricks] onto cluster.
4. Create new Python notebook, attach to cluster.
5. Use following code:
   from azureml.core import Workspace
   from azureml.core.authentication import InteractiveLoginAuthentication
   interactive_auth = InteractiveLoginAuthentication(tenant_id="XXXXXXXXX")

ws = Workspace.get(name = workspace_name,
subscription_id = subscription_id,
resource_group = resource_group,
location = workspace_region,
auth=interactive_auth,
exist_ok=True)

Also same for Service Principal login.

Result:
PermissionError: [Errno 13] Permission denied: '/root/.azureml'

PermissionError Traceback (most recent call last)
in ()
4 from azureml.core import Workspace
5 from azureml.core.authentication import InteractiveLoginAuthentication
----> 6 interactive_auth = InteractiveLoginAuthentication(tenant_id="XXXXXXXXX")
7
8 ws = Workspace.get(name = workspace_name,

/databricks/python/lib/python3.7/site-packages/azureml/core/authentication.py in init(self, force, tenant_id, cloud)
408 print("Performing interactive authentication. Please follow the instructions "
409 "on the terminal.")
--> 410 perform_interactive_login(tenant=tenant_id, cloud_type=self._cloud_type)
411 print("Interactive authentication successfully completed.")
412

/databricks/python/lib/python3.7/site-packages/azureml/_base_sdk_common/common.py in perform_interactive_login(username, password, service_principal, tenant, allow_no_subscriptions, identity, use_device_code, use_cert_sn_issuer, cloud_type)
531 from azureml._vendor.azure_cli_core._environment import get_config_dir
532
--> 533 ACCOUNT.load(os.path.join(get_config_dir(), 'azureProfile.json'))
534 CONFIG.load(os.path.join(get_config_dir(), 'az.json'))
535 SESSION.load(os.path.join(get_config_dir(), 'az.sess'),

/databricks/python/lib/python3.7/site-packages/azureml/_vendor/azure_cli_core/_environment.py in get_config_dir()
16 else:
17 if sys.version_info > (3, 0):
---> 18 os.makedirs(_AUTH_FOLDER_PATH, exist_ok=True)
19 else:
20 if not os.path.exists(_AUTH_FOLDER_PATH):

/local_disk0/pythonVirtualEnvDirs/virtualEnv-bd1a0cae-0254-493b-8d51-ddadcf6a91c2/lib/python3.7/os.py in makedirs(name, mode, exist_ok)
209 if head and tail and not path.exists(head):
210 try:
--> 211 makedirs(head, exist_ok=exist_ok)
212 except FileExistsError:
213 # Defeats race condition when another thread created the path

/local_disk0/pythonVirtualEnvDirs/virtualEnv-bd1a0cae-0254-493b-8d51-ddadcf6a91c2/lib/python3.7/os.py in makedirs(name, mode, exist_ok)
219 return
220 try:
--> 221 mkdir(name, mode)
222 except OSError:
223 # Cannot rely on checking for EEXIST, since the operating system

PermissionError: [Errno 13] Permission denied: '/root/.azureml'

Expected behavior
Workspace object instantiated - Interactive authentication successfully completed

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Created new workspace, tried all variations of HC/Standard/No Passthrough/Passthrough.

[kaerm]

@krisbock thanks for reporting this, someone from our team will take a look at this

[rastala]

@krisbock Could you try granting read/write access to the /root/.azureml folder?

[krisbock]

Hi @rastala, this doesn't appear to be possible when using High-Concurrency clusters in with Storage Credential Passthrough enabled.

[luigiw]

@krisbock thx for reporting this issue, there're 2 options to solve this.

1. Upgrade your databricks runtime to be version 7.5+ (in which databricks creates per notebook home directory on the high concurrency clusters, and you'll not have the permission issue)
2. Azure ML is going to release a new version of azureml-core on 3/8/2021, which will allow you to customize the auth config folder path.

I can provide more details on how to do # 2 if # 1 is not possible for your use case.

[krisbock]

Thanks @luigiw , can confirm option 1 has resolved this.