Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deploy_model does not complete when the environment has a CRAN/GitHub package #414

Open
seed-of-apricot opened this issue Mar 16, 2021 · 4 comments
Open

deploy_model does not complete when the environment has a CRAN/GitHub package #414

seed-of-apricot opened this issue Mar 16, 2021 · 4 comments

Comments

@seed-of-apricot
Copy link

seed-of-apricot commented Mar 16, 2021 •
edited
Loading

Describe the bug
deploy_model does not complete when the environment has a CRAN/GitHub package.

To Reproduce

library(azuremlsdk)
get_current_run()

interactive_auth <- interactive_login_authentication(
    force = TRUE,
    tenant_id = <id>
)

ws <- load_workspace_from_config()

model <- register_model(ws,
    model_path = "model.rds",
    model_name = "iris_model",
    description = "Predict an Iris flower type"
)

r_env <- r_environment(
  name = "r_env",
  github_packages = list(github_package("ropensci/ghql"))
)

inference_config <- inference_config(
    entry_script = "score.r",
    source_directory = ".",
    environment = r_env
)

deployment_config <- aci_webservice_deployment_config(
  cpu_cores = 1,
  memory_gb = 1
)

local_deployment_config <- local_webservice_deployment_config()

service <- deploy_model(
    ws,
    "rservice-local",
    list(model),
    inference_config,
    local_deployment_config
)

Expected behavior
The model is deployed.

Log

> service <- deploy_model(
+     ws,
+     "rservice-local",
+     list(model),
+     inference_config,
+     local_deployment_config
+ )
Downloading model iris_model:11 to C:\Users\apricot\AppData\Local\Temp\azureml_fs2oovsv\iris_model\11
Generating Docker build context.
2021/03/16 18:23:35 Downloading source code...
2021/03/16 18:23:36 Finished downloading source code
2021/03/16 18:23:37 Creating Docker network: acb_default_network, driver: 'bridge'
2021/03/16 18:23:37 Successfully set up Docker network: acb_default_network
2021/03/16 18:23:37 Setting up Docker configuration...
2021/03/16 18:23:38 Successfully set up Docker configuration
2021/03/16 18:23:38 Logging in to registry: 177e8d3857024445bea26417e3c33c2c.azurecr.io
2021/03/16 18:23:39 Successfully logged into 177e8d3857024445bea26417e3c33c2c.azurecr.io
2021/03/16 18:23:39 Executing step ID: acb_step_0. Timeout(sec): 5400, Working directory: '', Network: 'acb_default_network'
2021/03/16 18:23:39 Scanning for dependencies...
2021/03/16 18:23:39 Successfully scanned dependencies
2021/03/16 18:23:39 Launching container with name: acb_step_0
Sending build context to Docker daemon  67.58kB
Step 1/30 : FROM mcr.microsoft.com/azureml/o16n-base/python-assets:latest AS inferencing-assets
latest: Pulling from azureml/o16n-base/python-assets
4c214b2e9432: Pulling fs layer
4c214b2e9432: Verifying Checksum
4c214b2e9432: Download complete
4c214b2e9432: Pull complete
Digest: sha256:dd6373d9d31b52869f58057a54a321d43d6adef416f03f98a578277287eda7e8
Status: Downloaded newer image for mcr.microsoft.com/azureml/o16n-base/python-assets:latest
 ---> 394986a72682
Step 2/30 : FROM mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu16.04@sha256:8bc7ffc7142fb2914e40e8d64fed7bb89f7d087b670c0cb3168d241a5e908e98
sha256:8bc7ffc7142fb2914e40e8d64fed7bb89f7d087b670c0cb3168d241a5e908e98: Pulling from azureml/base
3386e6af03b0: Pulling fs layer
49ac0bbe6c8e: Pulling fs layer
d1983a67e104: Pulling fs layer
1a0f3a523f04: Pulling fs layer
682f5066db89: Pulling fs layer
f8e72cca56c5: Pulling fs layer
0fd03f8f74b9: Pulling fs layer
926adb528adb: Pulling fs layer
7161d6556851: Pulling fs layer
99339a70e7aa: Pulling fs layer
2bdbf3988b02: Pulling fs layer
1a0f3a523f04: Waiting
682f5066db89: Waiting
f8e72cca56c5: Waiting
0fd03f8f74b9: Waiting
926adb528adb: Waiting
7161d6556851: Waiting
99339a70e7aa: Waiting
2bdbf3988b02: Waiting
d1983a67e104: Verifying Checksum
d1983a67e104: Download complete
49ac0bbe6c8e: Verifying Checksum
49ac0bbe6c8e: Download complete
1a0f3a523f04: Verifying Checksum
1a0f3a523f04: Download complete
f8e72cca56c5: Verifying Checksum
f8e72cca56c5: Download complete
3386e6af03b0: Verifying Checksum
3386e6af03b0: Download complete
0fd03f8f74b9: Verifying Checksum
0fd03f8f74b9: Download complete
7161d6556851: Verifying Checksum
7161d6556851: Download complete
99339a70e7aa: Verifying Checksum
99339a70e7aa: Download complete
2bdbf3988b02: Verifying Checksum
2bdbf3988b02: Download complete
3386e6af03b0: Pull complete
49ac0bbe6c8e: Pull complete
d1983a67e104: Pull complete
1a0f3a523f04: Pull complete
926adb528adb: Verifying Checksum
926adb528adb: Download complete
682f5066db89: Download complete
682f5066db89: Pull complete
f8e72cca56c5: Pull complete
0fd03f8f74b9: Pull complete
926adb528adb: Pull complete
7161d6556851: Pull complete
99339a70e7aa: Pull complete
2bdbf3988b02: Pull complete
Digest: sha256:8bc7ffc7142fb2914e40e8d64fed7bb89f7d087b670c0cb3168d241a5e908e98
Status: Downloaded newer image for mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu16.04@sha256:8bc7ffc7142fb2914e40e8d64fed7bb89f7d087b670c0cb3168d241a5e908e98
 ---> 6ac3db102f47
Step 3/30 : USER root
 ---> Running in 5647ffb297d6
Removing intermediate container 5647ffb297d6
 ---> 428002d3fc1e
Step 4/30 : RUN mkdir -p $HOME/.cache
 ---> Running in 9214e46e1458
Removing intermediate container 9214e46e1458
 ---> 83c9f59fd6b1
Step 5/30 : WORKDIR /
 ---> Running in df86876e3ed5
Removing intermediate container df86876e3ed5
 ---> d7d3f2b9bf2d
Step 6/30 : COPY azureml-environment-setup/99brokenproxy /etc/apt/apt.conf.d/
 ---> 93219fbfa2e8
Step 7/30 : RUN true
 ---> Running in b6b9bb4c2fc6
Removing intermediate container b6b9bb4c2fc6
 ---> 2959ed49d2e6
Step 8/30 : COPY --from=inferencing-assets /artifacts /var/
 ---> 26c45a28f8ce
Step 9/30 : RUN /var/requirements/install_system_requirements.sh
 ---> Running in 365e9cf93694
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial InRelease [17.5 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:4 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial/main amd64 Packages [8487 B]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [1969 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [15.9 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [984 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [8820 B]
Get:12 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [2478 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [16.4 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [1539 kB]
Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [26.2 kB]
Get:19 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [10.9 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [12.6 kB]
Fetched 19.2 MB in 4s (4722 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  python3-software-properties
The following NEW packages will be installed:
  software-properties-common
The following packages will be upgraded:
  python3-software-properties
1 upgraded, 1 newly installed, 0 to remove and 89 not upgraded.
Need to get 29.7 kB of archives.
After this operation, 193 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.10 [20.2 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.10 [9504 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 29.7 kB in 0s (45.3 kB/s)
(Reading database ... 17866 files and directories currently installed.)
Preparing to unpack .../python3-software-properties_0.96.20.10_all.deb ...
Unpacking python3-software-properties (0.96.20.10) over (0.96.20.9) ...
Selecting previously unselected package software-properties-common.
Preparing to unpack .../software-properties-common_0.96.20.10_all.deb ...
Unpacking software-properties-common (0.96.20.10) ...
Processing triggers for dbus (1.10.6-1ubuntu3.5) ...
Setting up python3-software-properties (0.96.20.10) ...
Setting up software-properties-common (0.96.20.10) ...
Processing triggers for dbus (1.10.6-1ubuntu3.5) ...
gpg: keyring `/tmp/tmpi6hgjufi/secring.gpg' created
gpg: keyring `/tmp/tmpi6hgjufi/pubring.gpg' created
gpg: requesting key 5234BF2B from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpi6hgjufi/trustdb.gpg: trustdb created
gpg: key 5234BF2B: public key "Launchpad PPA for Adiscon" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  dh-python distro-info-data file gir1.2-glib-2.0 iso-codes libapt-inst2.0
  libdbus-glib-1-2 libgirepository-1.0-1 libmagic1 libmpdec2 libpython3-stdlib
  libpython3.5-minimal libpython3.5-stdlib lsb-release mime-support
  powermgmt-base python-apt-common python3 python3-apt python3-dbus python3-gi
  python3-minimal python3-pycurl python3-software-properties python3.5
  python3.5-minimal unattended-upgrades
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  software-properties-common*
0 upgraded, 0 newly installed, 1 to remove and 89 not upgraded.
After this operation, 193 kB disk space will be freed.
(Reading database ... 17881 files and directories currently installed.)
Removing software-properties-common (0.96.20.10) ...
Purging configuration files for software-properties-common (0.96.20.10) ...
Processing triggers for dbus (1.10.6-1ubuntu3.5) ...
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial InRelease
Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
libunwind8 is already the newest version (1.1-4.1).
liblttng-ust0 is already the newest version (2.7.1-1).
libxml++2.6-2v5 is already the newest version (2.40.1-1).
runit is already the newest version (2.1.2-3ubuntu1).
psmisc is already the newest version (22.21-2.1ubuntu0.1).
wget is already the newest version (1.17.1-1ubuntu1.5).
nginx-light is already the newest version (1.10.3-0ubuntu0.16.04.5).
The following packages were automatically installed and are no longer required:
  dh-python distro-info-data file gir1.2-glib-2.0 iso-codes libapt-inst2.0
  libdbus-glib-1-2 libgirepository-1.0-1 libmagic1 libmpdec2 libpython3-stdlib
  libpython3.5-minimal libpython3.5-stdlib lsb-release mime-support
  powermgmt-base python-apt-common python3 python3-apt python3-dbus python3-gi
  python3-minimal python3-pycurl python3-software-properties python3.5
  python3.5-minimal unattended-upgrades
Use 'apt autoremove' to remove them.
Suggested packages:
  rsyslog-mysql | rsyslog-pgsql rsyslog-doc rsyslog-relp rsyslog-elasticsearch
  rsyslog-mmjsonparse rsyslog-imptcp rsyslog-gnutls rsyslog-openssl
  rsyslog-udpspoof rsyslog-mmrm1stspace rsyslog-mmutf8fix rsyslog-kafka
  rsyslog-omstdout apparmor zip
The following packages will be upgraded:
  libcurl3 rsyslog unzip
3 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.
Need to get 1022 kB of archives.
After this operation, 50.2 kB of additional disk space will be used.
Get:1 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial/main amd64 rsyslog amd64 8.2102.0-0adiscon1xenial1 [673 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libcurl3 amd64 7.47.0-1ubuntu2.18 [187 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 unzip amd64 6.0-20ubuntu1.1 [162 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 1022 kB in 1s (879 kB/s)
(Reading database ... 17865 files and directories currently installed.)
Preparing to unpack .../rsyslog_8.2102.0-0adiscon1xenial1_amd64.deb ...
Unpacking rsyslog (8.2102.0-0adiscon1xenial1) over (8.2001.0-0adiscon1xenial1) ...
Preparing to unpack .../libcurl3_7.47.0-1ubuntu2.18_amd64.deb ...
Unpacking libcurl3:amd64 (7.47.0-1ubuntu2.18) over (7.47.0-1ubuntu2.14) ...
Preparing to unpack .../unzip_6.0-20ubuntu1.1_amd64.deb ...
Unpacking unzip (6.0-20ubuntu1.1) over (6.0-20ubuntu1) ...
Processing triggers for systemd (229-4ubuntu21.23) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up rsyslog (8.2102.0-0adiscon1xenial1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.
Setting up libcurl3:amd64 (7.47.0-1ubuntu2.18) ...
Setting up unzip (6.0-20ubuntu1.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Removing intermediate container 365e9cf93694
 ---> 20efb61c2138
Step 10/30 : RUN cp /var/configuration/rsyslog.conf /etc/rsyslog.conf && cp /var/configuration/nginx.conf /etc/nginx/sites-available/app && ln -sf /etc/nginx/sites-available/app /etc/nginx/sites-enabled/app && rm -f /etc/nginx/sites-enabled/default
 ---> Running in 63cacea1cd34
Removing intermediate container 63cacea1cd34
 ---> 84faae4c281c
Step 11/30 : ENV SVDIR=/var/runit
 ---> Running in bdf9d906e6c9
Removing intermediate container bdf9d906e6c9
 ---> e88673ec280d
Step 12/30 : RUN if dpkg --compare-versions `conda --version | grep -oE '[^ ]+$'` lt 4.4.11; then conda install conda==4.4.11; fi
 ---> Running in 9da19291b3cb
Removing intermediate container 9da19291b3cb
 ---> c989a8ec3bde
Step 13/30 : COPY azureml-environment-setup/mutated_conda_dependencies.yml azureml-environment-setup/mutated_conda_dependencies.yml
 ---> cbd0eb1b0c07
Step 14/30 : RUN ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name __pycache__ -exec rm -rf {} + && ldconfig
 ---> Running in 9810bd03962e
Solving environment: ...working...
done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.11
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda


libstdcxx-ng-9.1.0   | 4.0 MB    | ########## | 100%
zlib-1.2.11          | 120 KB    | ########## | 100%
setuptools-50.3.0    | 891 KB    | ########## | 100%
tk-8.6.10            | 3.2 MB    | ########## | 100%
libedit-3.1          | 171 KB    | ########## | 100%
python-3.6.2         | 27.0 MB   | ########## | 100%
wheel-0.35.1         | 36 KB     | ########## | 100%
sqlite-3.23.1        | 1.5 MB    | ########## | 100%
ncurses-6.0          | 907 KB    | ########## | 100%
openssl-1.0.2u       | 3.1 MB    | ########## | 100%
certifi-2020.6.20    | 160 KB    | ########## | 100%
libgcc-ng-9.1.0      | 8.1 MB    | ########## | 100%
ca-certificates-2020 | 128 KB    | ########## | 100%
readline-7.0         | 387 KB    | ########## | 100%
pip-20.2.4           | 2.0 MB    | ########## | 100%
xz-5.2.5             | 438 KB    | ########## | 100%
libffi-3.2.1         | 52 KB     | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting azureml-defaults
  Downloading azureml_defaults-1.24.0-py3-none-any.whl (3.1 kB)
Collecting configparser==3.7.4
  Downloading configparser-3.7.4-py2.py3-none-any.whl (22 kB)
Collecting azureml-dataset-runtime[fuse]~=1.24.0
  Downloading azureml_dataset_runtime-1.24.0-py3-none-any.whl (3.4 kB)
Collecting gunicorn==19.9.0
  Downloading gunicorn-19.9.0-py2.py3-none-any.whl (112 kB)
Collecting azureml-core~=1.24.0
  Downloading azureml_core-1.24.0.post2-py3-none-any.whl (2.2 MB)
Collecting applicationinsights>=0.11.7
  Downloading applicationinsights-0.11.9-py2.py3-none-any.whl (58 kB)
Collecting werkzeug<=1.0.1,>=0.16.1
  Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting json-logging-py==0.2
  Downloading json-logging-py-0.2.tar.gz (3.6 kB)
Collecting azureml-model-management-sdk==1.0.1b6.post1
  Downloading azureml_model_management_sdk-1.0.1b6.post1-py2.py3-none-any.whl (130 kB)
Collecting flask==1.0.3
  Downloading Flask-1.0.3-py2.py3-none-any.whl (92 kB)
Collecting pyarrow<2.0.0,>=0.17.0
  Downloading pyarrow-1.0.1-cp36-cp36m-manylinux2014_x86_64.whl (17.3 MB)
Collecting azureml-dataprep<2.12.0a,>=2.11.0a
  Downloading azureml_dataprep-2.11.2-py3-none-any.whl (39.4 MB)
Collecting numpy!=1.19.3; sys_platform == "linux"
  Downloading numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
Collecting fusepy<4.0.0,>=3.0.1; extra == "fuse"
  Downloading fusepy-3.0.1.tar.gz (11 kB)
Collecting adal>=1.2.0
  Downloading adal-1.2.6-py2.py3-none-any.whl (55 kB)
Collecting requests<3.0.0,>=2.19.1
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting azure-mgmt-containerregistry>=2.0.0
  Downloading azure_mgmt_containerregistry-2.8.0-py2.py3-none-any.whl (718 kB)
Collecting azure-mgmt-keyvault<7.0.0,>=0.40.0
  Downloading azure_mgmt_keyvault-2.2.0-py2.py3-none-any.whl (89 kB)
Collecting azure-mgmt-resource<15.0.0,>=1.2.1
  Downloading azure_mgmt_resource-12.1.0-py2.py3-none-any.whl (1.1 MB)
Collecting azure-mgmt-storage<16.0.0,>=1.5.0
  Downloading azure_mgmt_storage-11.2.0-py2.py3-none-any.whl (547 kB)
Collecting SecretStorage
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting azure-mgmt-authorization<1.0.0,>=0.40.0
  Downloading azure_mgmt_authorization-0.61.0-py2.py3-none-any.whl (94 kB)
Collecting msrestazure>=0.4.33
  Downloading msrestazure-0.6.4-py2.py3-none-any.whl (40 kB)
Collecting pathspec
  Downloading pathspec-0.8.1-py2.py3-none-any.whl (28 kB)
Collecting ruamel.yaml>=0.15.35
  Downloading ruamel.yaml-0.16.13-py2.py3-none-any.whl (111 kB)
Collecting python-dateutil>=2.7.3
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting msrest>=0.5.1
  Downloading msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting azure-graphrbac<1.0.0,>=0.40.0
  Downloading azure_graphrbac-0.61.1-py2.py3-none-any.whl (141 kB)
Collecting contextlib2
  Downloading contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)
Collecting backports.tempfile
  Downloading backports.tempfile-1.0-py2.py3-none-any.whl (4.4 kB)
Collecting jsonpickle
  Downloading jsonpickle-2.0.0-py2.py3-none-any.whl (37 kB)
Collecting pytz
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting urllib3>=1.23
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting cryptography!=1.9,!=2.0.*,!=2.1.*,!=2.2.*,<4.0.0
  Downloading cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting PyJWT<3.0.0
  Downloading PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting pyopenssl<21.0.0
  Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting jmespath
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting docker
  Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)
Collecting ndg-httpsclient
  Downloading ndg_httpsclient-0.5.1-py3-none-any.whl (34 kB)
Collecting azure-common>=1.1.12
  Downloading azure_common-1.1.26-py2.py3-none-any.whl (12 kB)
Collecting six>=1.10
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting dill>=0.2.7.1
  Downloading dill-0.3.3-py2.py3-none-any.whl (81 kB)
Collecting liac-arff>=2.1.1
  Downloading liac-arff-2.5.0.tar.gz (13 kB)
Collecting pandas>=0.20.2
  Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
Collecting itsdangerous>=0.24
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting click>=5.1
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting Jinja2>=2.10
  Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting azureml-dataprep-native<31.0.0,>=30.0.0
  Downloading azureml_dataprep_native-30.0.0-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB)
Collecting cloudpickle<2.0.0,>=1.1.0
  Downloading cloudpickle-1.6.0-py3-none-any.whl (23 kB)
Collecting azure-identity<1.5.0,>=1.2.0
  Downloading azure_identity-1.4.1-py2.py3-none-any.whl (86 kB)
Collecting azureml-dataprep-rslex<1.10.0a,>=1.9.0dev0
  Downloading azureml_dataprep_rslex-1.9.1-cp36-cp36m-manylinux2010_x86_64.whl (9.0 MB)
Collecting dotnetcore2<3.0.0,>=2.1.14
  Downloading dotnetcore2-2.1.20-py3-none-manylinux1_x86_64.whl (28.7 MB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad/lib/python3.6/site-packages (from requests<3.0.0,>=2.19.1->azureml-core~=1.24.0->azureml-defaults->-r /azureml-environment-setup/condaenv.x40v5k3j.requirements.txt (line 1)) (2020.6.20)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting jeepney>=0.6
  Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10"
  Downloading ruamel.yaml.clib-0.2.2-cp36-cp36m-manylinux1_x86_64.whl (549 kB)
Collecting isodate>=0.6.0
  Downloading isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting backports.weakref
  Downloading backports.weakref-1.0.post1-py2.py3-none-any.whl (5.2 kB)
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-3.7.3-py3-none-any.whl (12 kB)
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
Collecting websocket-client>=0.32.0
  Downloading websocket_client-0.58.0-py2.py3-none-any.whl (61 kB)
Collecting pyasn1>=0.1.1
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux2010_x86_64.whl (32 kB)
Collecting msal<2.0.0,>=1.3.0
  Downloading msal-1.10.0-py2.py3-none-any.whl (60 kB)
Collecting msal-extensions~=0.2.2
  Downloading msal_extensions-0.2.2-py2.py3-none-any.whl (15 kB)
Collecting azure-core<2.0.0,>=1.0.0
  Downloading azure_core-1.12.0-py2.py3-none-any.whl (130 kB)
Collecting distro>=1.2.0
  Downloading distro-1.5.0-py2.py3-none-any.whl (18 kB)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting portalocker~=1.0; platform_system != "Windows"
  Downloading portalocker-1.7.1-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: json-logging-py, fusepy, liac-arff
  Building wheel for json-logging-py (setup.py): started
  Building wheel for json-logging-py (setup.py): finished with status 'done'
  Created wheel for json-logging-py: filename=json_logging_py-0.2-py3-none-any.whl size=3924 sha256=58da48f725b18b375adb049c020eadf7817aae8f36814bbc3a3a4ffff7a13b0e
  Stored in directory: /root/.cache/pip/wheels/e2/1d/52/535a274b9c2ce7d4064838f2bdb62013801281ef7d7f21e2ee
  Building wheel for fusepy (setup.py): started
  Building wheel for fusepy (setup.py): finished with status 'done'
  Created wheel for fusepy: filename=fusepy-3.0.1-py3-none-any.whl size=10504 sha256=d7becb3e840801932d930e44a90b0945e31ad7b0aedbda37f0f54e05337deec3
  Stored in directory: /root/.cache/pip/wheels/21/5c/83/1dd7e8a232d12227e5410120f4374b33adeb4037473105b079
  Building wheel for liac-arff (setup.py): started
  Building wheel for liac-arff (setup.py): finished with status 'done'
  Created wheel for liac-arff: filename=liac_arff-2.5.0-py3-none-any.whl size=11730 sha256=4a09a7de985de702a96b1c5ffe9c23428bb331c1c8f34f391dfea7f3a6aff91b
  Stored in directory: /root/.cache/pip/wheels/53/ba/da/8562a6a6dbb428fd1ecc21053106df3948645cd991958f669b
Successfully built json-logging-py fusepy liac-arff
Installing collected packages: configparser, numpy, pyarrow, azureml-dataprep-native, cloudpickle, six, idna, urllib3, chardet, requests, PyJWT, pycparser, cffi, cryptography, msal, 
portalocker, msal-extensions, azure-core, azure-identity, azureml-dataprep-rslex, distro, dotnetcore2, azureml-dataprep, fusepy, azureml-dataset-runtime, gunicorn, python-dateutil, adal, azure-common, isodate, oauthlib, requests-oauthlib, msrest, msrestazure, azure-mgmt-containerregistry, azure-mgmt-keyvault, azure-mgmt-resource, azure-mgmt-storage, jeepney, SecretStorage, azure-mgmt-authorization, pathspec, ruamel.yaml.clib, ruamel.yaml, azure-graphrbac, contextlib2, backports.weakref, backports.tempfile, typing-extensions, zipp, importlib-metadata, jsonpickle, pytz, pyopenssl, jmespath, websocket-client, docker, pyasn1, ndg-httpsclient, azureml-core, applicationinsights, werkzeug, json-logging-py, dill, liac-arff, pandas, azureml-model-management-sdk, itsdangerous, click, MarkupSafe, Jinja2, flask, azureml-defaults
Successfully installed Jinja2-2.11.3 MarkupSafe-1.1.1 PyJWT-2.0.1 SecretStorage-3.3.1 adal-1.2.6 applicationinsights-0.11.9 azure-common-1.1.26 azure-core-1.12.0 azure-graphrbac-0.61.1 azure-identity-1.4.1 azure-mgmt-authorization-0.61.0 azure-mgmt-containerregistry-2.8.0 azure-mgmt-keyvault-2.2.0 azure-mgmt-resource-12.1.0 azure-mgmt-storage-11.2.0 azureml-core-1.24.0.post2 azureml-dataprep-2.11.2 azureml-dataprep-native-30.0.0 azureml-dataprep-rslex-1.9.1 azureml-dataset-runtime-1.24.0 azureml-defaults-1.24.0 azureml-model-management-sdk-1.0.1b6.post1 backports.tempfile-1.0 backports.weakref-1.0.post1 cffi-1.14.5 chardet-4.0.0 click-7.1.2 cloudpickle-1.6.0 configparser-3.7.4 contextlib2-0.6.0.post1 cryptography-3.4.6 dill-0.3.3 distro-1.5.0 docker-4.4.4 dotnetcore2-2.1.20 flask-1.0.3 fusepy-3.0.1 gunicorn-19.9.0 idna-2.10 importlib-metadata-3.7.3 isodate-0.6.0 itsdangerous-1.1.0 jeepney-0.6.0 jmespath-0.10.0 json-logging-py-0.2 jsonpickle-2.0.0 liac-arff-2.5.0 msal-1.10.0 msal-extensions-0.2.2 msrest-0.6.21 msrestazure-0.6.4 ndg-httpsclient-0.5.1 numpy-1.19.5 oauthlib-3.1.0 pandas-1.1.5 pathspec-0.8.1 portalocker-1.7.1 pyarrow-1.0.1 pyasn1-0.4.8 pycparser-2.20 pyopenssl-20.0.1 python-dateutil-2.8.1 pytz-2021.1 requests-2.25.1 requests-oauthlib-1.3.0 ruamel.yaml-0.16.13 ruamel.yaml.clib-0.2.2 six-1.15.0 typing-extensions-3.7.4.3 urllib3-1.26.4 websocket-client-0.58.0 werkzeug-1.0.1 zipp-3.4.1
#
# To activate this environment, use:
# > source activate /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad
#
# To deactivate an active environment, use:
# > source deactivate
#



Removing intermediate container 9810bd03962e
 ---> ed070b0f9aef
Step 15/30 : ENV PATH /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad/bin:$PATH
 ---> Running in ca1eadf1fc6a
Removing intermediate container ca1eadf1fc6a
 ---> 1e20f68f85a9
Step 16/30 : COPY azureml-environment-setup/send_conda_dependencies.py azureml-environment-setup/send_conda_dependencies.py
 ---> 480f97868c2a
Step 17/30 : COPY azureml-environment-setup/environment_context.json azureml-environment-setup/environment_context.json
 ---> 569930f4fa7b
Step 18/30 : RUN python /azureml-environment-setup/send_conda_dependencies.py -p /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad
 ---> Running in 33d085e4c7bc
Report materialized dependencies for the environment
Reading environment context
Exporting conda environment
Exception occured on getting conda environment details
Failed to send materialized environment details
Removing intermediate container 33d085e4c7bc
 ---> 9cea08df107f
Step 19/30 : ENV AZUREML_CONDA_ENVIRONMENT_PATH /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad
 ---> Running in 8151984dad05
Removing intermediate container 8151984dad05
 ---> 95f01ed880ba
Step 20/30 : ENV LD_LIBRARY_PATH /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad/lib:$LD_LIBRARY_PATH
 ---> Running in 357ad78ff3e4
Removing intermediate container 357ad78ff3e4
 ---> 7e21c80fa6b3
Step 21/30 : RUN conda install -p /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad -y pip=20.1.1
 ---> Running in e9623a92ee0c
Solving environment: ...working... 
done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.11
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda


pip-20.1.1           | 2.0 MB    | ########## | 100%
openssl-1.0.2u       | 3.1 MB    | ########## | 100%
certifi-2020.12.5    | 144 KB    | ########## | 100%
ca-certificates-2021 | 125 KB    | ########## | 100%
## Package Plan ##

  environment location: /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad

  added / updated specs:
    - pip=20.1.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pip-20.1.1                 |           py36_1         2.0 MB
    openssl-1.0.2u             |       h7b6447c_0         3.1 MB
    certifi-2020.12.5          |   py36h06a4308_0         144 KB
    ca-certificates-2021.1.19  |       h06a4308_1         125 KB
    ------------------------------------------------------------
                                           Total:         5.4 MB

The following packages will be UPDATED:

    ca-certificates: 2020.10.14-0      anaconda --> 2021.1.19-h06a4308_1
    certifi:         2020.6.20-py36_0  anaconda --> 2020.12.5-py36h06a4308_0
    openssl:         1.0.2u-h7b6447c_0 anaconda --> 1.0.2u-h7b6447c_0

The following packages will be DOWNGRADED:

    pip:             20.2.4-py36_0     anaconda --> 20.1.1-py36_1


Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

Removing intermediate container e9623a92ee0c
 ---> 7ec034c4f9fa
Step 22/30 : RUN conda install -p /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad -c conda-forge -y mscorefonts
 ---> Running in 023e56d1c0a8
Solving environment: ...working...
done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.11
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda


ca-certificates-2020 | 137 KB    | ########## | 100%
openssl-1.0.2u       | 3.2 MB    | ########## | 100%
mscorefonts-0.0.1    | 3.1 MB    | ########## | 100%
certifi-2020.12.5    | 143 KB    | ########## | 100%
python_abi-3.6       | 4 KB      | ########## | 100%
## Package Plan ##

  environment location: /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad

  added / updated specs:
    - mscorefonts


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2020.12.5  |       ha878542_0         137 KB  conda-forge
    openssl-1.0.2u             |       h516909a_0         3.2 MB  conda-forge
    mscorefonts-0.0.1          |                3         3.1 MB  conda-forge
    certifi-2020.12.5          |   py36h5fab9bb_1         143 KB  conda-forge
    python_abi-3.6             |          1_cp36m           4 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         6.5 MB

The following NEW packages will be INSTALLED:

    mscorefonts:     0.0.1-3                  conda-forge
    python_abi:      3.6-1_cp36m              conda-forge

The following packages will be UPDATED:

    certifi:         2020.12.5-py36h06a4308_0             --> 2020.12.5-py36h5fab9bb_1 conda-forge
    openssl:         1.0.2u-h7b6447c_0                    --> 1.0.2u-h516909a_0        conda-forge

The following packages will be DOWNGRADED:

    ca-certificates: 2021.1.19-h06a4308_1                 --> 2020.12.5-ha878542_0     conda-forge


Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

Removing intermediate container 023e56d1c0a8
 ---> 0301aafd067a
Step 23/30 : RUN conda install -p /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad -c r -y r-essentials=3.6.0 rpy2 r-checkpoint && pip install --no-cache-dir azureml-defaults  
 ---> Running in d49173157511
Solving environment: ...working...  // never completes

Additional context
azuremlsdk (R): 1.11.0
azureml-sdk (Python): 1.24.0

Both anaconda and miniconda have been tried and neither worked.
I suppose something went wrong with the -c r channel in conda install, probably?
@BostjanBozic
Copy link

This currently occurs even if you do not specify custom environment. Basically just creating experiment with basic "Hello world" script fails on image build step.

@lehmus
Copy link

lehmus commented Aug 25, 2021

I confirm that deployment fails even for the default environment without additional packages. The deployment hangs and stays in "Transitioning" state indefinitely. There are no deployment logs available. The only way to remove the endpoint is using Azure CLI, the workspace UI does not allow removing endpoints in this state.

To debug the issue, I first registered a plain R environment in AML Workspace and then used this in the deployment. The environment build log reveals that the environment build runs into timeout at step 23:

Step 23/30 : RUN conda install -p /azureml-envs/azureml_da3e97fcb51801118b8e80207f3e01ad -c r -y r-essentials=3.6.0 rpy2 r-checkpoint && pip install --no-cache-dir azureml-defaults
 ---> Running in 9be319800471
Solving environment: ...working... 
Run ID: cb7 timed out after 1h30m0s

I have previously built custom R environments and noticed that the rpy2 install step can take a very long time even when run locally. It seems that 90min timeout limit is too strict for the deployment. (!)

@BostjanBozic
Copy link

It seems problem is that by default r-essentials package is installed.

As a workaround, what you can do is build and use custom environment. I used something like this (Dockerfile):

FROM mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu18.04

RUN conda install -c r -y pip=20.1.1 openssl=1.1.1c r-base rpy2 && \
  conda install -c conda-forge -y mscorefonts && \
    conda clean -ay && \
    pip install --no-cache-dir azureml-defaults
    
ENV TAR="/bin/tar"

RUN R -e "install.packages(c('remotes', 'reticulate', 'optparse', 'caret'), dependencies = TRUE, repos = 'https://cloud.r-project.org/')"
RUN R -e "remotes::install_github('https://github.com/Azure/azureml-sdk-for-r')"

And then point to that image when you are setting up environment in script:

env <- r_environment(
  name = "<randomName>",
  custom_docker_image = "<repoUrl/imageName:imageTag>"
)

@lehmus
Copy link

lehmus commented Aug 25, 2021

Yes, the custom Dockerfile approach works for me too. As the R SDK will be deprecated soon, I guess this won't be fixed. Hopefully the new approach will be more stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lehmus @seed-of-apricot @BostjanBozic