Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not able to invoke batch endpoint with input parameter of type mlflow_model #27685

Open
AtleH opened this issue Oct 25, 2023 · 3 comments
Open

Not able to invoke batch endpoint with input parameter of type mlflow_model #27685

AtleH opened this issue Oct 25, 2023 · 3 comments
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning az ml Service Attention This issue is responsible by Azure service team.

Comments

@AtleH
Copy link

AtleH commented Oct 25, 2023

Describe the bug

When invoking a batch endpoint with input containing a model reference, the az cli fails, complaining about missing version:

$ az ml batch-endpoint invoke --name some-batch-endpoint --file ./payload.yaml --workspace-name my-workspace
Argument '--file' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
$ cat payload.yaml
inputs:
  trained_model:
    type: mlflow_model
    path: azureml://registries/some-registry/models/some-model/versions/42

The problem seems to be related only to validation of the input data, and fails even before there is an attempt to invoke the endpoint.

Related command

az ml batch-endpoint invoke

Errors

Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.

Issue script & Debug output

az ml batch-endpoint invoke --name some-batch-endpoint --file ./payload.yaml --workspace-name my-workspace --debug

cli.knack.cli: Command arguments: ['ml', 'batch-endpoint', 'invoke', '--name', 'some-batch-endpoint', '--file', './payload.yaml', '--workspace-name', 'my-workspace', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x104b87250>, <function OutputProducer.on_global_arguments at 0x104c64670>, <function CLIQuery.on_global_arguments at 0x104c85ab0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
cli.azure.cli.core: ml                        7.113        21       146  /Users/ATLEH/.azure/cliextensions/ml
cli.azure.cli.core: Total (1)                 7.113        21       146
cli.azure.cli.core: Loaded 21 groups, 146 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : ml batch-endpoint invoke
cli.azure.cli.core: Command table: ml batch-endpoint invoke
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x105909090>]
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x105925b40>]
cli.knack.commands: Configured default 'aion-check' for arg resource_group_name
cli.knack.commands: Configured default 'aion-check' for arg workspace_name
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x10594b760>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x10594b880>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x104c64700>, <function CLIQuery.handle_query_parameter at 0x104c85b40>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x10594b7f0>]
Argument '--file' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/ATLEH/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /Users/ATLEH/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
Traceback (most recent call last):
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 288, in ml_batch_endpoint_invoke
    return ml_client.batch_endpoints.invoke(
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
    self._resolve_input(input_data, os.getcwd())
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
    raise e
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
    entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
    raise ValidationException(
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
cli: None
cli: Traceback (most recent call last):
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 288, in ml_batch_endpoint_invoke
    return ml_client.batch_endpoints.invoke(
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
    self._resolve_input(input_data, os.getcwd())
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
    raise e
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
    entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
    raise ValidationException(
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.

cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 303, in ml_batch_endpoint_invoke
    log_and_raise_error(err, debug)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 181, in log_and_raise_error
    raise cli_error
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 288, in ml_batch_endpoint_invoke
    return ml_client.batch_endpoints.invoke(
  File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
    self._resolve_input(input_data, os.getcwd())
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
    raise e
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
    entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
  File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
    raise ValidationException(
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.

cli.azure.cli.core.azclierror: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
az_command_data_logger: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x1059092d0>]

Expected behavior

I expect the batch-endpoint to be correctly invoked.

Environment Summary

azure-cli                         2.53.1

core                              2.53.1
telemetry                          1.1.0

Extensions:
ml                                2.21.1

Dependencies:
msal                            1.24.0b2
azure-mgmt-resource             23.1.0b2

Python location '/usr/local/Cellar/azure-cli/2.53.1/libexec/bin/python'
Extensions directory '/Users/ATLEH/.azure/cliextensions'

Python (Darwin) 3.10.13 (main, Aug 24 2023, 22:48:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]

Additional context

The same problem appears when using the Azure SDK for Python and calling ml_client.batch_endpoints.invoke(...)

Tested with several versions, latest one being 1.11.1.
@AtleH AtleH added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Oct 25, 2023
@microsoft-github-policy-service microsoft-github-policy-service bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot Service Attention This issue is responsible by Azure service team. Machine Learning az ml labels Oct 25, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Oct 25, 2023

Thank you for opening this issue, we will look into it.

@microsoft-github-policy-service
Copy link
Contributor

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

@AtleH AtleH mentioned this issue Oct 26, 2023
Open
@AtleH
Copy link
Author

AtleH commented Oct 26, 2023

Also reported as a problem with the Azure SDK: Azure/azure-sdk-for-python#32761

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning az ml Service Attention This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AtleH @yonzhan