From 6a6069692b7ddcf3882a62490a7b63cc3d571808 Mon Sep 17 00:00:00 2001 From: Christian Kadner Date: Thu, 2 Jul 2020 17:42:47 -0700 Subject: [PATCH] Update SDK README.md (#212) --- sdk/README.md | 95 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 65 insertions(+), 30 deletions(-) diff --git a/sdk/README.md b/sdk/README.md index 8788b4d2935..2407c8bb322 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -1,31 +1,52 @@ -# Compiler for Tekton +# Kubeflow Pipelines SDK for Tekton -The Kubeflow Pipelines [SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/) +The Kubeflow Pipelines [SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/) allows data scientists to define end-to-end machine learning and data pipelines. The output of the Kubeflow Pipelines SDK compiler is YAML for [Argo](https://github.com/argoproj/argo). -We are extending the compiler of the Kubeflow Pipelines SDK to generate YAML -for [Tekton](https://github.com/tektoncd/pipeline). +The `kfp-tekton` SDK is extending the `Compiler` and the `Client` of the Kubeflow +Pipelines SDK to generate [Tekton](https://github.com/tektoncd/pipeline) YAML +and to subsequently upload and run the pipeline with the Kubeflow Pipelines engine +backed by Tekton. ## Table of Contents + - [Extensions to the Kubeflow Pipelines SDK](#extensions-to-the-kubeflow-pipelines-sdk) - [Project Prerequisites](#project-prerequisites) - - [Tested Pipelines](#tested-pipelines) - - [How to use the KFP-Tekton Compiler](#how-to-use-the-kfp-tekton-compiler) - - [Installation](#installation) - - [Compiling a Kubeflow Pipelines DSL Script](#compiling-a-kubeflow-pipelines-dsl-script) - - [Running the Pipeline on a Tekton Cluster](#running-the-pipeline-on-a-tekton-cluster) - - [Build Tekton from Master](#build-tekton-from-master) + - [Installation](#installation) + - [Compiling a Kubeflow Pipelines DSL Script](#compiling-a-kubeflow-pipelines-dsl-script) + - [Running the Compiled Pipeline on a Tekton Cluster](#running-the-compiled-pipeline-on-a-tekton-cluster) + - [Building Tekton from Master](#building-tekton-from-master) - [Optional Features](#optional-features) - [Compile Kubeflow Pipelines without Artifacts](#compile-kubeflow-pipelines-without-artifacts) - [List of Available Features](#list-of-available-features) + - [Tested Pipelines](#tested-pipelines) - [Troubleshooting](#troubleshooting) +## Extensions to the Kubeflow Pipelines SDK + +In addition to the functionality provided by the Kubeflow Pipelines +[SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/) the `kfp-tekton` +SDK provides a `TektonCompiler` and a `TektonClient`: + +`TektonCompiler`: + + - `kfp_tekton.compiler.TektonCompiler.compile` compiles Python DSL code into a + YAML file containing a Tekton `PipelineRun` which can be deployed directly to + a Tekton enabled Kubernetes cluster or uploaded to the Kubeflow Pipelines + dashboard with the Tekton backend. + +`TektonClient`: + + - `kfp_tekton.TektonClient.create_run_from_pipeline_func` compiles DSL pipeline + function and runs the pipeline on a Kubernetes cluster with KFP and Tekton + + ## Project Prerequisites - Python: `3.5.3` or later @@ -37,21 +58,7 @@ Follow the instructions for [installing project prerequisites](/sdk/python/READM and take note of some important caveats. -## Tested Pipelines - -We are [testing the compiler](/sdk/python/tests/README.md) on more than 80 pipelines -found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler -`testdata` folder, the KFP core samples and the samples contributed by third parties. - -A report card of Kubeflow Pipelines samples that are currently supported by the `kfp-tekton` -compiler can be found [here](/sdk/python/tests/test_kfp_samples_report.txt). -If you work on a PR that enables another of the missing features please ensure that -your code changes are improving the number of successfully compiled KFP pipeline samples. - - -## How to use the KFP-Tekton Compiler - -### Installation +## Installation You can install the latest release of the `kfp-tekton` compiler from [PyPi](https://pypi.org/project/kfp-tekton/). We recommend to create a Python @@ -91,8 +98,7 @@ from source by cloning the repository [https://github.com/kubeflow/kfp-tekton](h make test ``` - -### Compiling a Kubeflow Pipelines DSL Script +## Compiling a Kubeflow Pipelines DSL Script The `kfp-tekton` Python package comes with the `dsl-compile-tekton` command line executable, which should be available in your terminal shell environment after @@ -106,7 +112,23 @@ If you cloned the `kfp-tekton` project, you can find example pipelines in the --output pipeline.yaml -### Running the Pipeline on a Tekton Cluster +**Note**: If the KFP DSL script contains a `__main__` method calling the +`kfp_tekton.compiler.TektonCompiler.compile()` function: + +```Python +if __name__ == "__main__": + from kfp_tekton.compiler import TektonCompiler + TektonCompiler().compile(pipeline_func, "pipeline.yaml") +``` + +... then the pipeline can be compiled by running the DSL script with `python3` +executable from a command line shell, producing a Tekton YAML file `pipeline.yaml` +in the same directory: + + python3 pipeline.py + + +## Running the Compiled Pipeline on a Tekton Cluster After compiling the `sdk/python/tests/compiler/testdata/parallel_join.py` DSL script in the step above, we need to deploy the generated Tekton YAML to our Kubernetes @@ -144,8 +166,9 @@ Once the Tekton Pipeline is running, the logs should start streaming: [echo : main] [echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath [echo : main] - -## Build Tekton from Master + + +## Building Tekton from Master In order to utilize the latest features and functions of the `kfp-tekton` compiler, we suggest to install Tekton from a nightly built or build it from the @@ -201,6 +224,18 @@ To understand how each feature is implemented and its current status, please vis the [FEATURES](FEATURES.md) doc. +## Tested Pipelines + +We are [testing the compiler](/sdk/python/tests/README.md) on more than 80 pipelines +found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler +`testdata` folder, the KFP core samples and the samples contributed by third parties. + +A report card of Kubeflow Pipelines samples that are currently supported by the `kfp-tekton` +compiler can be found [here](/sdk/python/tests/test_kfp_samples_report.txt). +If you work on a PR that enables another of the missing features please ensure that +your code changes are improving the number of successfully compiled KFP pipeline samples. + + ## Troubleshooting - When you encounter permission issues related to ServiceAccount, refer to