-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
get the trained model to work with gcloud local predict #2
Comments
I thought it was oddly quite painful to refer to the saved_model APIs, based on the ways they were organized... so opted for choosing the more legible literal value. I don't think using the literal should get in the way of using gcloud... since matching strings should suffice. I did fix the other issue where in glcoud the model would not be initialized. Perhaps that was getting in the way. |
iris model trained from your master branch does not work with gcloud local predict. gcloud ml-engine local predict --model-dir TOUT/model/ --text-instances iris/data/eval.csv This is the code that fails
Because it got to line 338, the try failed. My guess is because of tag_constants.SERVING, which I think is 'serve'. You use 'SERVING'. If I manually edit my gcloud files to use 'SERVING', I get an error later on: which makes we realize we might be calling tf.add_to_collection wrong in _ff.py. The old iris sample called it like so: tf.add_to_collection(OUTPUTS_KEY, json.dumps(outputs)) I'm confused by these parts, because I am not directly calling tf.add_to_collection in the structured data package, so maybe some util function I call does this for me. |
$ gcloud ml-engine versions create v1 --model tfx --origin gs://cloud-ml-dev_bdt/TOUT/model |
Got gcloud local predict working with this proof of concept:
I also used tag_constants.SERVING. Will research what gcloud is doing with the output collection if it is defined on Monday. This may mean we are not allowed to use the 'output' collection to pass values, so maybe use 'tenorfx_output'. |
I was able to update just 'SERVING' to 'serve' and make things work. Note that I actually had to change gcloud code -- for some reason the v1 version of gcloud is loading prediction_lib_beta in my setup of gcloud. Odd. However, the real test is running this on the service. Maybe there should be a "tfx predict" as well as a CLI wrapper around tensorfx.predictions.Model.load() and .predict(). |
Getting gcloud local predict is a real test too: that's the first thing a ml-engine user will try! Also, did your last update undo your table init work? Because gcloud predict did not work for me.
Did your changes remove your table init work?
|
The line to make saved model work is still there: https://github.com/TensorLab/tensorfx/blob/master/src/training/_model.py#L256 Some how I think you've got an older version of gcloud (or there is a major bug in the GA version of gcloud). The trace indicates its trying to load a model with 'inputs', and 'outputs' collection keys whose value is a json-serialized dictionary of tensor aliases to tensor names (search for _get_legacy_signature in code search). You'll find it in prediction_lib_beta.py ... and I am puzzled how that is in the path if you're using the GA command rather than the beta command. |
FWIW, added a command line tool to do local predictions in af8becc. Definitely need to validate with the service as the ultimate test. Hopefully it isn't trying to old models with the old json collections. |
in /src/prediction/_model.py
s/'SERVING'/tag_constants.SERVING/g
that will help in making the output model work with gcloud local predict
The text was updated successfully, but these errors were encountered: