Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update doc for MKP release #2242

Merged
merged 8 commits into from
Oct 1, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions manifests/gcp_marketplace/guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Deploy Kubeflow Pipelines from Google Cloud Marketplace

Go to [Google Cloud Marketplace](https://console.cloud.google.com/marketplace) to deploy Kubeflow Pipelines by using a graphical interface.
You can go to the [Marketplace page for Kubeflow Pipelines](https://console.cloud.google.com/marketplace/details/google-cloud-ai-platform/kubeflow-pipelines) directly, or search for "Kubeflow Pipelines" from the Marketplace landing page.

Once you have deployed Kubeflow Pipelines instances, you can view and manage them in the [Pipelines Console](http://console.cloud.google.com/ai-platform/pipelines).

## Cluster

You can deploy Kubeflow Pipelines to a new cluster or an existing cluster. New clusters aren't customizable, so if you need that, you should use the [Google Kubernetes Engine](https://console.cloud.google.com/kubernetes/list) to create a cluster that meets your requirements and then deploy to that.

You can only deploy one Kubeflow Pipelines into a given cluster.

## Namespace
Specify a [Kubenetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).

## App instance name
Specify an app instance name to help you identify this instance.

## GCP Service Account credentials
james-jwu marked this conversation as resolved.
Show resolved Hide resolved
This deployment requires a [GCP service account](https://cloud.google.com/iam/docs/service-accounts) to use for authentication when calling other GCP services. This includes Cloud Storage and Cloud SQL if you are using managed storage, as well as other services your pipeline might need, for example Dataflow. Specify the base64-encoded credentials for the service account you want to use.

You can get these credentials by running the following command in a terminal window. This command will create a new key under the service account. Please note that a single service account can only have 10 keys.

```
$ gcloud iam service-accounts keys create application_default_credentials.json --iam-account [your-service-account] && cat application_default_credentials.json | base64
```
rmgogogo marked this conversation as resolved.
Show resolved Hide resolved

Existing key also can be used.

```
cat existing_credentials.json | base64
```

If you are running this command on Linux, please use `base64 -w 0` to disable line wrapping.

## Use managed storage
Select this option if you want your Kubeflow Pipelines deployment to use Cloud Storage and Cloud SQL for storage. Managed storage takes care of data backup for you, so that your data will be preserved in the case that your cluster is accidentally deleted.

If you don't select this option, your Kubeflow Pipelines deployment will use Kubernetes [Persisent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) and in-cluster MySQL.

## Cloud SQL instance connection name
This is a required field if you choose to use managed storage.
Provide the instance connection name for an existing Cloud SQL for MySQL instance.
The instance connection name can be found on the instance detail page in the Cloud SQL console.
The instance connection name uses the format `project:zone:instance-name`. Example: myproject:us-central1:myinstance.
For more details on how to create a new instance, see https://cloud.google.com/sql/docs/mysql/quickstart.

## Database username
The database username to use when connecting to the Cloud SQL instance.
If you leave this field empty, the deployment will use the default 'root' user account to connect.
For more details about MySQL users, see https://cloud.google.com/sql/docs/mysql/users.

## Database password
The database password to use when connecting to the Cloud SQL instance.
If you leave this field empty, the deployment will try to connect to the instance without providing a password.
This will fail if a password is required for the username you provided.

## Database name prefix
The prefix of the database name. Kubeflow Pipelines will create two databases, `prefix_pipeline` and `prefix_metadata`.
Use lowercase letters, numbers, and hyphens. The name must start with a letter.
If you are reusing a prefix from a previous deployment, your new deployment will recover the data from that deployment.
If the prefix is not specified, the app instance name will be used.

## Deploy
Click `Deploy` to start deploying Kubeflow Pipelines into the cluster you specified.
Deployment might take few minutes, so please be patient. After deployment is complete, go to the [Pipelines Console](http://pantheon.corp.google.com/ai-platform/pipelines) to access the Kubeflow Pipelines instance.

## Tips

### Warning message in Pipelines Console
If you see the following warning message in the Pipeline console:

> Sorry, the server was only able to partially fulfill your request. Some data might not be rendered.

Possible reasons are:
- the cluster is under upgrading
- the new Kubeflow Pipeline instance is under deployment

Wait for a while and then refresh.
38 changes: 19 additions & 19 deletions manifests/gcp_marketplace/schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -104,15 +104,11 @@ properties:
title: |-
GCP Service Account credentials used to call other GCP services.
description: |-
To be able to call other GCP services from the pipeline, we need to be authenticated.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The associated service account should have Cloud SQL and GCS access permissions/roles at
minimum, as well as any permission you need for your pipelines.
This field is used to store the content of the service account
JSON file. It can be encoded using base64 instead of messing with JSON format.
To get the encoded credential, run
$ gcloud iam service-accounts keys create application_default_credentials.json --iam-account [your-service-account] && cat application_default_credentials.json | base64
It is recommended to use google cloud shell, which saves you the effort of environment setup.
Then you can copy-paste the generated token into this field. Note: the token is without spaces/breaklines.
This deployment requires a service account to use for authentication when calling other GCP services.
Specify the base64-encoded credentials for the service account you want to use.
You can get these credentials by running the following command in a terminal window.
\"$ gcloud iam service-accounts keys create application_default_credentials.json --iam-account [your-service-account] && cat application_default_credentials.json | base64\"
If you are using Linux, please use \"base64 -w 0\" to disable line wrapping.
type: string
default: ""
x-google-marketplace:
Expand All @@ -129,25 +125,29 @@ properties:
default: false
managedstorage.cloudsqlInstanceConnectionName:
type: string
title: CloudSQL instance connection name (Managed storage only)
title: Cloud SQL instance connection name (Managed storage only)
description: |-
Provide a pre-created CloudSQL for MySQL instance.
The instance connection name can be found in the instance detail page.
For example myproject:us-central1:myinstance.
For more details on how to create a new instance, visit https://cloud.google.com/sql/docs/mysql/quickstart
This field must be specified if choose to use managed storage.
Provide the instance connection name for an existing Cloud SQL for MySQL instance.
The instance connection name can be found on the instance detail page in the Cloud SQL console.
The instance connection name uses the format project:zone:instance-name, for example,myproject:us-central1:myinstance.
For more details on how to create a new instance, see https://cloud.google.com/sql/docs/mysql/quickstart.
managedstorage.dbUsername:
type: string
title: Database username (Managed storage only)
description: |-
The database username to connect to the CloudSQL instance.
If leaving this field empty, the deployment will use the default 'root' user account.
For more details about mysql username, visit https://cloud.google.com/sql/docs/mysql/users
The database username to use when connecting to the Cloud SQL instance.
If you leave this field empty, the deployment will use the default 'root' user account to connect.
For more details about MySQL users, see https://cloud.google.com/sql/docs/mysql/users.
managedstorage.dbPassword:
type: string
title: Database password (Managed storage only)
x-google-marketplace:
type: MASKED_FIELD
description: |-
The database password to connect to the CloudSQL instance.
If leaving this field empty, the deployment will use no password.
The database password to use when connecting to the Cloud SQL instance.
If you leave this field empty, the deployment will try to connect to the instance without providing a password.
This will fail if a password is required for the username you provided.
managedstorage.databaseNamePrefix:
type: string
title: Database name prefix (Managed storage only)
Expand Down