Custom model deployed with a docker container but requests are not working as expected

[danielduckworth]

Context

I have been using the Ludwig AI library to create tensorflow models. The library includes a serve tool to serve a model via HTTP, much like the Pytorch serve.

I'm attempting to use a model trained with Ludwig and serve with Ludwig serve from a custom docker container deployed as a custom model in Vertex attached to an endpoint.

I've described the context in more detail in a discussion in the Ludwig github.

Expected Behavior

Using the following JSON format for a request:

{
  "instances": [
    {
      "textfeature": "Words to be classified"
    }
  ]
}

The endpoint should return a JSON object with predictions from Ludwig serve.

Actual Behavior

I get an error from Ludwig serve:

{"error":"entry must contain all input features"}

Steps to Reproduce the Problem

This is a bit tricky, but I will explain at a high level. If need be, I can provide complete notebook with the whole procedure.

1. Train a text classification model with ludwig.
2. Create a custom docker container to use Ludwig serve and the trained model.
3. Submit the container to vertex container registry.
4. Deploy a custom model.
5. Attach the model to an endpoint.

Any ideas?

[danielduckworth]

I tried deploying the custom container to Cloud Run and this worked really. I would like to do some load testing and compare the response times between Cloud Run and a Vertex endpoint.

[ivanmkc]

Hi, it doesn't look like you are referring to any of our official notebooks hosted in this repo.
Please keep in mind that we are not able to provide official support here (other than related to the official notebooks), so I'll provide a couple more options to consider:

1. Google Cloud Platform support options are at https://cloud.google.com/support/
2. You can ask a question on Stack Overflow using the tag google-cloud-ml. This tag not only receives responses from the Stack Overflow community, but also from Google engineers, who monitor the tag and offer unofficial support.

[RajeshThallam]

@danielduckworth Can you try formatting input request by encoding JSON string as binary?

cat > instances.json <<END
{ 
   "instances": [
     { 
       "data": {
         "b64": "$(echo '{"textfeature": "Words to be classified"}' | base64 --wrap=0)"
       }
     }
   ]
}
END

[danielduckworth]

Thanks for the suggestion @RajeshThallam. I tried this but got the same response as before. brightsparc explained that the current expected input structure for ludwig serve is not compatible with vertex: ludwig-ai/ludwig#1728 (comment)