Skip to content

Commit

Permalink
fixup! Fix watson train component
Browse files Browse the repository at this point in the history
  • Loading branch information
mpoqq committed May 3, 2019
1 parent 6b56b29 commit 411c75b
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 6 deletions.
14 changes: 10 additions & 4 deletions components/ibm-components/watson/train/src/wml-train.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ def getSecret(secret):
def train(args):
from watson_machine_learning_client import WatsonMachineLearningAPIClient
from minio import Minio
from urllib.parse import urlsplit
import os,time

wml_train_code = args.train_code
Expand All @@ -42,7 +43,12 @@ def train(args):
wml_data_source_type = getSecret("/app/secrets/wml_data_source_type")

cos_endpoint = getSecret("/app/secrets/cos_endpoint")
cos_endpoint_with_scheme = "https://" + cos_endpoint
cos_endpoint_parts = urlsplit(cos_endpoint)
if bool(cos_endpoint_parts.scheme):
cos_endpoint_hostname = cos_endpoint_parts.hostname
else:
cos_endpoint_hostname = cos_endpoint
cos_endpoint = 'https://' + cos_endpoint
cos_access_key = getSecret("/app/secrets/cos_access_key")
cos_secret_key = getSecret("/app/secrets/cos_secret_key")
cos_input_bucket = getSecret("/app/secrets/cos_input_bucket")
Expand All @@ -51,7 +57,7 @@ def train(args):
# download model code
model_code = os.path.join('/app', wml_train_code)

cos = Minio(cos_endpoint,
cos = Minio(cos_endpoint_hostname,
access_key = cos_access_key,
secret_key = cos_secret_key,
secure = True)
Expand Down Expand Up @@ -87,7 +93,7 @@ def train(args):
client.training.ConfigurationMetaNames.AUTHOR_NAME : wml_author_name,
client.training.ConfigurationMetaNames.TRAINING_DATA_REFERENCE : {
"connection" : {
"endpoint_url" : cos_endpoint_with_scheme,
"endpoint_url" : cos_endpoint,
"access_key_id" : cos_access_key,
"secret_access_key" : cos_secret_key
},
Expand All @@ -98,7 +104,7 @@ def train(args):
},
client.training.ConfigurationMetaNames.TRAINING_RESULTS_REFERENCE: {
"connection" : {
"endpoint_url" : cos_endpoint_with_scheme,
"endpoint_url" : cos_endpoint,
"access_key_id" : cos_access_key,
"secret_access_key" : cos_secret_key
},
Expand Down
6 changes: 4 additions & 2 deletions samples/ibm-samples/watson/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,17 @@ Once the service is created, from service's `Dashboard`, follow the instruction

Watson Machine Learning service loads datasets from cloud object store and stores model outputs and other artifacts to cloud object store. Users can use any cloud object store they already preserve. Users can also create a cloud object store with `IBM Cloud Object Storage` service by following this [link](https://console.bluemix.net/catalog/services/cloud-object-storage).

Collect the `endpoint`, `access_key_id` and `secret_access_key` fields from the service credentials for the cloud object store. Create the service credentials first if not existed. To ensure generating HMAC credentials, specify the following in the `Add Inline Configuration Parameters` field: `{"HMAC":true}`.
Collect the `access_key_id` and `secret_access_key` fields from the service credentials for the cloud object store. Create the service credentials first if not existed. To ensure generating HMAC credentials, specify the following in the `Add Inline Configuration Parameters` field: `{"HMAC":true}`.

Collect the `endpoint` info from the endpoint section in the cloud object store service.

Create two buckets, one for storing the train datasets and model source codes, and one for storing the model outputs.

* Set up access credentials

This pipeline sample reads the credentials from a file hosted in a github repo. Refer to `creds.ini` file and input user's specific credentials. Then upload the file to a github repo the user has access.

Note: make sure the `cos_endpoint` value in the `creds.ini` file only contains valid endpoint without the `http://` or `https://` prefix.
Note: make sure the `cos_endpoint` value in the `creds.ini` file must have at least a scheme and hostname.

To access the credentials file, the user should provide a github access token and the link to the raw content of the file. Modify the `GITHUB_TOKEN` and `CONFIG_FILE_URL` variables in the `watson_train_serve_pipeline.py` file with the user's access token and link.

Expand Down

0 comments on commit 411c75b

Please sign in to comment.