Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clean up quickstart sample a bit #1232

Merged
merged 2 commits into from
Apr 25, 2019
Merged

clean up quickstart sample a bit #1232

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4171331
clean up a bit
gaoning777 Apr 25, 2019
063c6a9
move the project id to the top
gaoning777 Apr 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 62 additions & 28 deletions samples/notebooks/quickstart.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,47 @@
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.16/kfp.tar.gz'\n",
"# PROJECT_ID is used to construct the docker image registry. We will use Google Container Registry, \n",
"# but any other accessible registry works as well. \n",
"PROJECT_ID='Your-Gcp-Project-Id'"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"# Install Pipeline SDK\n",
"!pip3 install $KFP_PACKAGE --upgrade\n",
"!mkdir tmp/pipelines"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part 1\n",
"# Two ways to author a component to list blobs in a GCS bucket\n",
"A pipeline is composed of one or more components. In this section, you will build a single component that lists the blobs in a GCS bucket. Then you buid a pipeline that consists of this component. There are two ways to author a component. In the following sections we will go through each of them."
"A pipeline is composed of one or more components. In this section, you will build a single component that lists the blobs in a GCS bucket. Then you build a pipeline that consists of this component. There are two ways to author a component. In the following sections we will go through each of them."
]
},
{
Expand All @@ -51,7 +85,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -78,7 +112,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -97,7 +131,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -110,7 +144,7 @@
"\n",
"# Compile the pipeline to a file.\n",
"import kfp.compiler as compiler\n",
"compiler.Compiler().compile(pipeline_func, 'list_blobs.pipeline.tar.gz')"
"compiler.Compiler().compile(pipeline_func, 'tmp/pipelines/list_blobs.pipeline.tar.gz')"
]
},
{
Expand All @@ -125,7 +159,7 @@
"metadata": {},
"source": [
"### 2.1 Create a Docker container\n",
"Create your own container image that includes your program. If your component creates some outputs to be fed as inputs to the downstream components, each separate output must be written as a string to a separate local text file by the container image. For example, if a trainer component needs to output the trained model path, it can write the path to a local file `/output.txt`. The string written to an output file cannot be too big. If it is too big (>> 100 kB), save the output to an external persistent storage and pass the storage path to the next component.\n",
"Create your own container image that includes your program. If your component creates some outputs to be fed as inputs to the downstream components, each separate output must be written as a string to a separate local text file inside the container image. For example, if a trainer component needs to output the trained model path, it can write the path to a local file `/output.txt`. The string written to an output file cannot be too big. If it is too big (>> 100 kB), it is recommended to save the output to an external persistent storage and pass the storage path to the next component.\n",
"\n",
"Start by entering the value of your Google Cloud Platform Project ID."
]
Expand Down Expand Up @@ -185,38 +219,19 @@
"%%bash\n",
"\n",
"# Create Dockerfile.\n",
"cat > ./tmp/components/list-gcs-blobs/Dockerfile <<HERE\n",
"cat > ./tmp/components/list-gcs-blobs/Dockerfile <<EOF\n",
"FROM python:3.6-slim\n",
"WORKDIR /app\n",
"COPY . /app\n",
"RUN pip install --upgrade google-cloud-storage\n",
"HERE"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have created our Dockerfile we can create our Docker image. Then we need to push the image to a registry to host the image. Here, we will use Google Container Registry, but any other accessible registry works as well. In the following cell set your project ID that will be used to to push your image to Google Container Registry."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# GCP Project ID\n",
"PROJECT_ID='PROJECT_ID'\n",
"\n",
"assert(PROJECT_ID is not 'PROJECT_ID')"
"EOF"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now create a Shell script that builds a container image and stores it in the Google Container Registry."
"Now that we have created our Dockerfile we can create our Docker image. Then we need to push the image to a registry to host the image. Now create a Shell script that builds a container image and stores it in the Google Container Registry."
]
},
{
Expand Down Expand Up @@ -540,6 +555,25 @@
"source": [
"Follow the [instructions](https://www.kubeflow.org/docs/other-guides/accessing-uis/) on kubeflow.org to access Kubeflow UIs. Upload the created pipeline and run it."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Clean up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import shutil\n",
"import pathlib\n",
"path = pathlib.Path(\"tmp\")\n",
"shutil.rmtree(path)"
]
}
],
"metadata": {
Expand Down