Skip to content

Commit

Permalink
Update SDK README.md (#212)
Browse files Browse the repository at this point in the history
  • Loading branch information
ckadner authored Jul 3, 2020
1 parent cb2f036 commit 6a60696
Showing 1 changed file with 65 additions and 30 deletions.
95 changes: 65 additions & 30 deletions sdk/README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,52 @@
# Compiler for Tekton
# Kubeflow Pipelines SDK for Tekton

The Kubeflow Pipelines [SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/)
The Kubeflow Pipelines [SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/)
allows data scientists to define end-to-end machine learning and data pipelines.
The output of the Kubeflow Pipelines SDK compiler is YAML for [Argo](https://github.com/argoproj/argo).
We are extending the compiler of the Kubeflow Pipelines SDK to generate YAML
for [Tekton](https://github.com/tektoncd/pipeline).

The `kfp-tekton` SDK is extending the `Compiler` and the `Client` of the Kubeflow
Pipelines SDK to generate [Tekton](https://github.com/tektoncd/pipeline) YAML
and to subsequently upload and run the pipeline with the Kubeflow Pipelines engine
backed by Tekton.

## Table of Contents

<!-- START of ToC generated by running ./tools/mdtoc.sh sdk/README.md -->

- [Extensions to the Kubeflow Pipelines SDK](#extensions-to-the-kubeflow-pipelines-sdk)
- [Project Prerequisites](#project-prerequisites)
- [Tested Pipelines](#tested-pipelines)
- [How to use the KFP-Tekton Compiler](#how-to-use-the-kfp-tekton-compiler)
- [Installation](#installation)
- [Compiling a Kubeflow Pipelines DSL Script](#compiling-a-kubeflow-pipelines-dsl-script)
- [Running the Pipeline on a Tekton Cluster](#running-the-pipeline-on-a-tekton-cluster)
- [Build Tekton from Master](#build-tekton-from-master)
- [Installation](#installation)
- [Compiling a Kubeflow Pipelines DSL Script](#compiling-a-kubeflow-pipelines-dsl-script)
- [Running the Compiled Pipeline on a Tekton Cluster](#running-the-compiled-pipeline-on-a-tekton-cluster)
- [Building Tekton from Master](#building-tekton-from-master)
- [Optional Features](#optional-features)
- [Compile Kubeflow Pipelines without Artifacts](#compile-kubeflow-pipelines-without-artifacts)
- [List of Available Features](#list-of-available-features)
- [Tested Pipelines](#tested-pipelines)
- [Troubleshooting](#troubleshooting)

<!-- END of ToC generated by running ./tools/mdtoc.sh sdk/README.md -->


## Extensions to the Kubeflow Pipelines SDK

In addition to the functionality provided by the Kubeflow Pipelines
[SDK](https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/) the `kfp-tekton`
SDK provides a `TektonCompiler` and a `TektonClient`:

`TektonCompiler`:

- `kfp_tekton.compiler.TektonCompiler.compile` compiles Python DSL code into a
YAML file containing a Tekton `PipelineRun` which can be deployed directly to
a Tekton enabled Kubernetes cluster or uploaded to the Kubeflow Pipelines
dashboard with the Tekton backend.

`TektonClient`:

- `kfp_tekton.TektonClient.create_run_from_pipeline_func` compiles DSL pipeline
function and runs the pipeline on a Kubernetes cluster with KFP and Tekton


## Project Prerequisites

- Python: `3.5.3` or later
Expand All @@ -37,21 +58,7 @@ Follow the instructions for [installing project prerequisites](/sdk/python/READM
and take note of some important caveats.


## Tested Pipelines

We are [testing the compiler](/sdk/python/tests/README.md) on more than 80 pipelines
found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler
`testdata` folder, the KFP core samples and the samples contributed by third parties.

A report card of Kubeflow Pipelines samples that are currently supported by the `kfp-tekton`
compiler can be found [here](/sdk/python/tests/test_kfp_samples_report.txt).
If you work on a PR that enables another of the missing features please ensure that
your code changes are improving the number of successfully compiled KFP pipeline samples.


## How to use the KFP-Tekton Compiler

### Installation
## Installation

You can install the latest release of the `kfp-tekton` compiler from
[PyPi](https://pypi.org/project/kfp-tekton/). We recommend to create a Python
Expand Down Expand Up @@ -91,8 +98,7 @@ from source by cloning the repository [https://github.com/kubeflow/kfp-tekton](h
make test
```


### Compiling a Kubeflow Pipelines DSL Script
## Compiling a Kubeflow Pipelines DSL Script

The `kfp-tekton` Python package comes with the `dsl-compile-tekton` command line
executable, which should be available in your terminal shell environment after
Expand All @@ -106,7 +112,23 @@ If you cloned the `kfp-tekton` project, you can find example pipelines in the
--output pipeline.yaml


### Running the Pipeline on a Tekton Cluster
**Note**: If the KFP DSL script contains a `__main__` method calling the
`kfp_tekton.compiler.TektonCompiler.compile()` function:

```Python
if __name__ == "__main__":
from kfp_tekton.compiler import TektonCompiler
TektonCompiler().compile(pipeline_func, "pipeline.yaml")
```

... then the pipeline can be compiled by running the DSL script with `python3`
executable from a command line shell, producing a Tekton YAML file `pipeline.yaml`
in the same directory:

python3 pipeline.py


## Running the Compiled Pipeline on a Tekton Cluster

After compiling the `sdk/python/tests/compiler/testdata/parallel_join.py` DSL script
in the step above, we need to deploy the generated Tekton YAML to our Kubernetes
Expand Down Expand Up @@ -144,8 +166,9 @@ Once the Tekton Pipeline is running, the logs should start streaming:
[echo : main]
[echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
[echo : main]
## Build Tekton from Master


## Building Tekton from Master

In order to utilize the latest features and functions of the `kfp-tekton` compiler,
we suggest to install Tekton from a nightly built or build it from the
Expand Down Expand Up @@ -201,6 +224,18 @@ To understand how each feature is implemented and its current status, please vis
the [FEATURES](FEATURES.md) doc.


## Tested Pipelines

We are [testing the compiler](/sdk/python/tests/README.md) on more than 80 pipelines
found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler
`testdata` folder, the KFP core samples and the samples contributed by third parties.

A report card of Kubeflow Pipelines samples that are currently supported by the `kfp-tekton`
compiler can be found [here](/sdk/python/tests/test_kfp_samples_report.txt).
If you work on a PR that enables another of the missing features please ensure that
your code changes are improving the number of successfully compiled KFP pipeline samples.


## Troubleshooting

- When you encounter permission issues related to ServiceAccount, refer to
Expand Down

0 comments on commit 6a60696

Please sign in to comment.