Skip to content

Commit

Permalink
Bump version to 0.3.0 (kubeflow#3329)
Browse files Browse the repository at this point in the history
* Bump version to 0.3.0

* Fix formatting

* More formatting fixes

* More formatting fixes

* update requirements.txt

* update version
  • Loading branch information
neuromage authored and Jeffwan committed Dec 9, 2020
1 parent 82141b0 commit 72a07a6
Show file tree
Hide file tree
Showing 8 changed files with 94 additions and 51 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ metadata:
spec:
descriptor:
type: Kubeflow Pipelines
version: '0.2.5'
version: '0.3.0'
description: |-
Reusable end-to-end ML workflow
maintainers:
Expand Down
4 changes: 2 additions & 2 deletions manifests/gcp_marketplace/schema.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
x-google-marketplace:
schemaVersion: v2
applicationApiVersion: v1beta1
publishedVersion: '0.2.5'
publishedVersion: '0.3.0'
publishedVersionMetadata:
releaseNote: >-
Based on 0.2.5 version.
Based on 0.3.0 version.
releaseTypes:
- Feature
recommended: false
Expand Down
85 changes: 61 additions & 24 deletions manifests/kustomize/README.md
Original file line number Diff line number Diff line change
@@ -1,97 +1,134 @@
# Install Kubeflow Pipelines
This folder contains Kubeflow Pipelines Kustomize manifests for a light weight deployment. You can follow the instruction and deploy Kubeflow Pipelines in an existing cluster.

This folder contains Kubeflow Pipelines Kustomize manifests for a light weight
deployment. You can follow the instruction and deploy Kubeflow Pipelines in an
existing cluster.

## TL;DR

Deploy latest version of Kubeflow Pipelines

```
export PIPELINE_VERSION=0.2.5
export PIPELINE_VERSION=0.3.0
kubectl apply -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/crd.yaml
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/namespaced-install.yaml
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io kubectl apply -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/namespaced-install.yaml
```

Then get the Pipeline URL

```
kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com
```

## Customization
Customization can be done through Kustomize [Overlay](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#overlay).

Note - The instruction below assume you installed kubectl v1.14.0 or later, which has native support of kustomize.
To get latest kubectl, visit [here](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
Customization can be done through Kustomize
[Overlay](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#overlay).

Note - The instruction below assume you installed kubectl v1.14.0 or later,
which has native support of kustomize. To get latest kubectl, visit
[here](https://kubernetes.io/docs/tasks/tools/install-kubectl/)

### Deploy on GCP with CloudSQL and GCS

See [here](env/gcp/README.md) for more details.

### Change deploy namespace

To deploy Kubeflow Pipelines in namespace FOO,
- Edit [dev/kustomization.yaml](env/dev/kustomization.yaml) or [gcp/kustomization.yaml](env/gcp/kustomization.yaml) namespace section to FOO
- Then run

- Edit [dev/kustomization.yaml](env/dev/kustomization.yaml) or
[gcp/kustomization.yaml](env/gcp/kustomization.yaml) namespace section to
FOO
- Then run

```
kubectl kustomize base/crds | kubectl apply -f -
# then
kubectl kustomize env/dev | kubectl apply -f -
# or
kubectl kustomize env/gcp | kubectl apply -f -
```

### Disable the public endpoint
By default, the deployment install an [invert proxy agent](https://github.com/google/inverting-proxy) that exposes a public URL. If you want to skip installing it,
- Comment out the proxy component in the [kustomization.yaml](base/kustomization.yaml).
- Then run

By default, the deployment install an
[invert proxy agent](https://github.com/google/inverting-proxy) that exposes a
public URL. If you want to skip installing it:

- Comment out the proxy component in the
[kustomization.yaml](base/kustomization.yaml).
- Then run

```
kubectl kustomize . | kubectl apply -f -
```

The UI is still accessible by port-forwarding
```
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
```
and open http://localhost:8080/
The UI is still accessible by port-forwarding `kubectl port-forward -n kubeflow
svc/ml-pipeline-ui 8080:80` and open http://localhost:8080/

### Deploy on AWS (with S3 buckets as artifact store)

[https://github.com/e2fyi/kubeflow-aws](https://github.com/e2fyi/kubeflow-aws/tree/master/pipelines)
provides a community-maintained manifest for deploying kubeflow pipelines on AWS
(with S3 as artifact store instead of minio). More details can be found in the repo.
(with S3 as artifact store instead of minio). More details can be found in the
repo.

## Uninstall

You can uninstall Kubeflow Pipelines by running

```
export PIPELINE_VERSION=0.1.38
kubectl delete -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/namespaced-install.yaml
kubectl delete -f https://storage.googleapis.com/ml-pipeline/pipeline-lite/$PIPELINE_VERSION/crd.yaml
```

Or if you deploy through kustomize

```
kubectl kustomize env/dev | kubectl delete -f -
# or
kubectl kustomize env/gcp | kubectl delete -f -
# then
kubectl kustomize base/crds | kubectl delete -f -
kubectl kustomize base/crds | kubectl delete -f -
```

## Troubleshooting

### Permission error installing Kubeflow Pipelines to a cluster

Run

```
kubectl create clusterrolebinding your-binding --clusterrole=cluster-admin --user=[your-user-name]
kubectl create clusterrolebinding your-binding --clusterrole=cluster-admin
--user=[your-user-name]
```

### Samples requires "user-gcp-sa" secret

If sample code requires a "user-gcp-sa" secret, you could create one by
- First download the GCE VM service account token [Document](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys)

- First download the GCE VM service account token
[Document](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys)

```
gcloud iam service-accounts keys create application_default_credentials.json \
--iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com
--iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com
```
- Run

- Run

```
kubectl create secret -n [your-namespace] generic user-gcp-sa --from-file=user-gcp-sa.json=application_default_credentials.json
kubectl
create secret -n [your-namespace] generic user-gcp-sa
--from-file=user-gcp-sa.json=application_default_credentials.json`
```
16 changes: 8 additions & 8 deletions manifests/kustomize/base/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,18 @@ resources:

images:
- name: gcr.io/ml-pipeline/api-server
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/persistenceagent
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/scheduledworkflow
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/frontend
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/viewer-crd-controller
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/inverse-proxy-agent
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/visualization-server
newTag: 0.2.5
newTag: 0.3.0
- name: gcr.io/ml-pipeline/metadata-writer
newTag: 0.2.5
newTag: 0.3.0
2 changes: 1 addition & 1 deletion manifests/kustomize/base/pipeline-application.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ spec:
matchLabels:
application-crd-id: kubeflow-pipelines
descriptor:
version: 0.2.5
version: 0.3.0
type: Kubeflow Pipelines
description: |-
Reusable end-to-end ML workflow
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/kfp/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

__version__ = '0.2.5'
__version__ = '0.3.0'

from . import components
from . import containers
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ cryptography>=2.4.2
google-auth>=1.6.1
requests_toolbelt>=0.8.0
cloudpickle==1.1.1
kfp-server-api==0.2.5
kfp-server-api==0.3.0
argo-models == 2.2.1a
jsonschema >= 3.0.1
tabulate == 0.8.3
Expand Down
32 changes: 19 additions & 13 deletions sdk/python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,11 @@
'google-auth>=1.6.1',
'requests_toolbelt>=0.8.0',
'cloudpickle==1.1.1',
'kfp-server-api==0.2.5', #Update the upper version whenever a new version of the kfp-server-api package is released. Update the lower version when there is a breaking change in kfp-server-api, or kfp sdk depends on new api changes in kfp-server-api.
# Update the upper version whenever a new major version of the
# kfp-server-api package is released. Update the lower version when there is
# a breaking change in kfp-server-api, or kfp sdk depends on new api changes
# in kfp-server-api.
'kfp-server-api>=0.2.5, <0.4.0',
'argo-models == 2.2.1a', #2.2.1a is equivalent to argo 2.2.1
'jsonschema >= 3.0.1',
'tabulate == 0.8.3',
Expand All @@ -41,24 +45,26 @@
'strip-hints',
]


def find_version(*file_path_parts):
here = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(here, *file_path_parts), 'r') as fp:
version_file_text = fp.read()
here = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(here, *file_path_parts), 'r') as fp:
version_file_text = fp.read()

version_match = re.search(
r"^__version__ = ['\"]([^'\"]*)['\"]",
version_file_text,
re.M,
)
if version_match:
return version_match.group(1)

version_match = re.search(
r"^__version__ = ['\"]([^'\"]*)['\"]",
version_file_text,
re.M,
)
if version_match:
return version_match.group(1)
raise RuntimeError('Unable to find version string.')

raise RuntimeError("Unable to find version string.")

setup(
name=NAME,
version=find_version("kfp", "__init__.py"),
version=find_version('kfp', '__init__.py'),
description='KubeFlow Pipelines SDK',
author='google',
install_requires=REQUIRES,
Expand Down

0 comments on commit 72a07a6

Please sign in to comment.