Not able to invoke batch endpoint with input parameter of type mlflow_model

[AtleH]

• Package Name: azure.ai.ml
• Package Version: 1.11.1
• Operating System: MacOS Ventura 13.6
• Python Version: 3.11.6

Describe the bug
Invoking a batch endpoint and providing a reference to a model in a registry as input doesn't seem to work.

Invoking a batch endpoint by calling

ml_client = MLClient(...)
ml_client.batch_endpoints.invoke(endpoint="endpoint-name", inputs=model_input)

and providing an input parameter like this:

model_input = {
   "model": Input(type=AssetTypes.MLFLOW_MODEL, path=("azureml://registries/name1/models/name2/versions/1"))
}

fails during with this message:

azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/name1/models/name2/versions/1 of type data as arm id. Version must be provided.

The problem seems to arise during validation of the input, i.e. before it hits any infrastructure.

To Reproduce
Steps to reproduce the behavior:

1. Run the following code:

from azure.ai.ml import Input, MLClient
from azure.ai.ml.constants import AssetTypes
from azure.identity import DefaultAzureCredential


client = MLClient(DefaultAzureCredential(), subscription_id="*****")
model_input = {
        "model": Input(
            type=AssetTypes.MLFLOW_MODEL,
            path=("azureml://registries/name1/models/name2/versions/1")
        )
    }

client.batch_endpoints.invoke("endpoint", inputs=model_input)

Running the above produces this output:

$ python invoke.py
Traceback (most recent call last):
  File "/Users/ATLEH/testing/aion-registry-bug/invoke.py", line 14, in <module>
    client.batch_endpoints.invoke("endpoint", inputs=model_input)
  File "/Users/ATLEH/testing/aion-registry-bug/.venv/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ATLEH/testing/aion-registry-bug/.venv/lib/python3.11/site-packages/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/ATLEH/testing/aion-registry-bug/.venv/lib/python3.11/site-packages/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
    self._resolve_input(input_data, os.getcwd())
  File "/Users/ATLEH/testing/aion-registry-bug/.venv/lib/python3.11/site-packages/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
    raise e
  File "/Users/ATLEH/testing/aion-registry-bug/.venv/lib/python3.11/site-packages/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
    entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ATLEH/testing/aion-registry-bug/.venv/lib/python3.11/site-packages/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
    raise ValidationException(
azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/name1/models/name2/versions/1 of type data as arm id. Version must be provided.

Expected behavior
The batch endpoint should be invoked with the given parameters.

Screenshots

Additional context
Problem also reported for the az cli: Azure/azure-cli#27685

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github @Azure/azure-ml-sdk.

[santiagxf]

Hi @AtleH. Thanks for reaching out and taking the time to report the issue. We appreciate the feedback! We confirm this was a bug in our CLI/SDK and a fix has been issued. It should be included in the next release coming next week. I will update the thread once published.

[AtleH]

Tested with version 1.12.1, but still producing the same error.

[BradAF]

This is also an issue with endpoints as well:

azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //locations//workspaces/<workspace_id>/environments/<environment_name>/versions/2 of type environments as arm id. Version must be provided.

Switching to /locations//workspaces/<workspace_id>/environments/<environment_name>:2 gets past this validation error, but then mangles the environment ARM id by appending azureml://locations//workspaces/<workspace_id>/environments/<environment_name>:2 to an ARM id:

azure.ai.ml.exceptions.ValidationException: {
"result": "Failed",
"errors": [
{
"message": "Environment schema data cannot be a string; In order to specify an existing environments, please provide the correct registry path prefixed with 'azureml://':\n; id for environment is invalid",
"path": "environment",
"value": "azureml:/subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/environments///locations//workspaces/<workspace_id>/environments/<environment_name>/versions/1"

[santiagxf]

Thanks for reporting this issue. We are analyzing the issue and will provide an update shortly.

[jeroenboeye]

Any update on this?

[mhaythornthwaite]

I was also having the same issue as above when running the code shown below:

from azure.ai.ml import MLClient, Input
from azure.identity import DefaultAzureCredential
from azure.ai.ml.constants import AssetTypes

ml_client = MLClient(DefaultAzureCredential(), "<subscription_id>", "<resource_group>", "<workspace>")

model = ml_client.models.get("<model_name>", version="<version>")

endpoint_inputs = {
    "model": Input(type=AssetTypes.MLFLOW_MODEL, path=model.id),
    "data": Input(type=AssetTypes.URI_FOLDER, path="<datastore_path>")
}

job = ml_client.batch_endpoints.invoke(
    endpoint_name="<endpoint_pipeline_deployment_name>", 
    inputs=endpoint_inputs
)


>>> message=f"Supported input path value are: path on the datastore, public URI, "
>>>          "a registered data asset, or a local folder path.\n"
>>> ...
>>> ValidationException: Invalid input path

It appears from the error message that MLFLOW_MODELS are not a supported input type when invoking a batch endpoint. I managed to resolve this issue by hard coding the model into the pipeline so it did not need to be specified when invoking the batch endpoint. I followed the tutorial here. Note that when defining the pipeline function in the section here, the model is not an input in the function, and rather hardcoded using model=Input(type=AssetTypes.MLFLOW_MODEL, path=model.id)

See the section later in the documentation, when invoking the endpoint only types uri_folder and string are needed which are both supported.