diff --git a/content/docs/reference/_index.md b/content/docs/reference/_index.md index b835d79232..932cf29d21 100644 --- a/content/docs/reference/_index.md +++ b/content/docs/reference/_index.md @@ -7,10 +7,11 @@ weight = 900 ## TFJob -TFJob is a Kubernetes +TFJob is a Kubernetes [custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -that you can use to run TensorFlow training jobs on Kubernetes. For help with -using TFJob with Kubeflow, see the [user guide](/docs/components/tftraining/). +that you can use to run [TensorFlow](https://www.tensorflow.org/get_started/) training +jobs on Kubernetes. For help with using TFJob with Kubeflow, see the +[user guide](/docs/components/tftraining/). API references: diff --git a/content/docs/reference/virtual-dev/_index.md b/content/docs/reference/virtual-dev/_index.md new file mode 100644 index 0000000000..c014efd670 --- /dev/null +++ b/content/docs/reference/virtual-dev/_index.md @@ -0,0 +1,5 @@ ++++ +title = "Virtual Developer Environments" +description = "Detailed instructions for using VM appliances for Kubeflow" +weight = 2 ++++ diff --git a/content/docs/started/getting-started-minikf.md b/content/docs/reference/virtual-dev/getting-started-minikf.md similarity index 100% rename from content/docs/started/getting-started-minikf.md rename to content/docs/reference/virtual-dev/getting-started-minikf.md diff --git a/content/docs/started/getting-started-minikube.md b/content/docs/reference/virtual-dev/getting-started-minikube.md similarity index 85% rename from content/docs/started/getting-started-minikube.md rename to content/docs/reference/virtual-dev/getting-started-minikube.md index d4311dd5be..c256e6601d 100644 --- a/content/docs/started/getting-started-minikube.md +++ b/content/docs/reference/virtual-dev/getting-started-minikube.md @@ -4,8 +4,8 @@ description = "Quickly get Kubeflow running locally" weight = 2 +++ -This document outlines the steps you can take to get your local installation -of Kubeflow running on top of Minikube. Minikube runs a simple, single-node +This document outlines the steps you can take to get your local installation +of Kubeflow running on top of Minikube. Minikube runs a simple, single-node Kubernetes cluster inside a virtual machine (VM). By the end of this document, you'll have a local installation of Minikube kubernetes cluster along with all the default core components of @@ -206,35 +206,11 @@ $ minikube delete $ minikube start --cpus 4 --memory 8096 --disk-size=40g ``` -### Installing Kubeflow using kfctl -The following steps will deploy Kubeflow components and start them on the Minikube you created above. +### Install Kubeflow on an existing Kubernetes cluster - 1. Download Kubeflow source - - ``` - mkdir ${KUBEFLOW_SRC} - cd ${KUBEFLOW_SRC} - export KUBEFLOW_TAG={{% kf-latest-version %}} - curl https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/download.sh | bash - ``` - - **KUBEFLOW_SRC** directory where you want kubeflow source to be downloaded - - **KUBEFLOW_TAG** is a tag corresponding to the version to checkout such as {{% kf-latest-version %}} - - 1. Run the following to setup and deploy Kubeflow: - - ``` - KUBEFLOW_REPO=${KUBEFLOW_SRC} ${KUBEFLOW_SRC}/scripts/kfctl.sh init ${KFAPP} --platform minikube - cd ${KFAPP} - ${KUBEFLOW_SRC}/scripts/kfctl.sh generate all - ${KUBEFLOW_SRC}/scripts/kfctl.sh apply all - ``` - - **KFAPP** the _name_ of a directory where you want kubeflow configurations to be stored. This directory will be created when you run init. Please see [understanding the deployment process](/docs/gke/deploy/deploy-cli/#understanding-the-deployment-process) for more details. - -The above installation may take a few minutes. At the end of the installation you should see: -``` -Access Kubeflow dashboard at http://localhost:8080/ -Access Jupyter notebooks at http://localhost:8080/notebooks/ -``` +Now that you have a Kubernetes cluster running - the Minikube cluster - follow the +[existing Kubernetes cluster]((/docs/started/k8s/overview/)) instructions for installing +Kubeflow. ### Where to go next @@ -245,7 +221,7 @@ You can use the following command to set up port forwarding and access the Web U Now you can access the Kubeflow dashboard at http://localhost:8080/ and Jupyter notebooks at http://localhost:8080/notebooks/. -For Jupyter notebooks, you can use any username and password to log in. +For Jupyter notebooks, you can use any username and password to log in. Follow the guide to [setting up your Jupyter notebooks on Kubeflow](/docs/notebooks/setup/). -For further exploration refer to the [documentation](/docs/). \ No newline at end of file +For further exploration refer to the [documentation](/docs/). diff --git a/content/docs/started/getting-started-multipass.md b/content/docs/reference/virtual-dev/getting-started-multipass.md similarity index 100% rename from content/docs/started/getting-started-multipass.md rename to content/docs/reference/virtual-dev/getting-started-multipass.md diff --git a/content/docs/started/cloud/_index.md b/content/docs/started/cloud/_index.md new file mode 100644 index 0000000000..3cd2bdbd39 --- /dev/null +++ b/content/docs/started/cloud/_index.md @@ -0,0 +1,5 @@ ++++ +title = "Cloud Installation" +description = "Instructions for installing Kubeflow on a public cloud" +weight = 2 ++++ diff --git a/content/docs/started/getting-started-aws.md b/content/docs/started/cloud/getting-started-aws.md similarity index 100% rename from content/docs/started/getting-started-aws.md rename to content/docs/started/cloud/getting-started-aws.md diff --git a/content/docs/started/getting-started-azure.md b/content/docs/started/cloud/getting-started-azure.md similarity index 100% rename from content/docs/started/getting-started-azure.md rename to content/docs/started/cloud/getting-started-azure.md diff --git a/content/docs/started/getting-started-gke.md b/content/docs/started/cloud/getting-started-gke.md similarity index 100% rename from content/docs/started/getting-started-gke.md rename to content/docs/started/cloud/getting-started-gke.md diff --git a/content/docs/started/getting-started-icp.md b/content/docs/started/cloud/getting-started-icp.md similarity index 100% rename from content/docs/started/getting-started-icp.md rename to content/docs/started/cloud/getting-started-icp.md diff --git a/content/docs/started/getting-started.md b/content/docs/started/getting-started.md index 87bdcdc498..8e1db35535 100644 --- a/content/docs/started/getting-started.md +++ b/content/docs/started/getting-started.md @@ -11,87 +11,40 @@ This document provides information about setting up Kubeflow in various environm It's important that you have some knowledge of the following systems and tools: * [Kubernetes](https://kubernetes.io/docs/tutorials/kubernetes-basics/) -* [TensorFlow](https://www.tensorflow.org/get_started/) -* [kustomize](https://kustomize.io/) +* [Kustomize](https://kustomize.io/) -### Kubeflow requirements +If you plan to deploy Kubeflow on an existing Kubernetes cluster, review these +[Kubernetes system requirements](/docs/started/k8s/overview#minimum-system-requirements). -Before installing Kubeflow, ensure that your environment meets the -following requirements: -* You need kustomize version {{% kustomize-min-version %}} or later. See the - [kustomize component guide](/docs/components/misc/kustomize/) - for details about installing kustomize. +## Installing Kubeflow -* If you intend to install Kubeflow on an existing Kubernetes cluster, the - cluster must meet the following requirements: +There are various ways to install Kubeflow. Choose one of the following options +to suit your environment (desktop or server, existing Kubernetes cluster or public cloud): - * Kubernetes version {{% kubernetes-min-version %}} or later. - * A minimum of 0.6 CPU in cluster. (Reserved for 3 replicated ambassador pods - and according to your needs add additional CPUs.) - * Node with storage >= 10 GB. (Due to the ML libraries and third party packages - bundled in the Kubeflow Docker images.) -## Get your machine-learning workflow up and running on Kubeflow +* Installing Kubeflow on a **desktop** or **server**: -There are various ways to install Kubeflow. Choose one of the following options -to suit your environment (cloud, on premises (on prem), or local): + * To use Kubeflow on Windows, + follow the [Windows deployment guide](/docs/started/workstation/getting-started-windows/). + * To use Kubeflow on MacOS, + follow the [MacOS deployment guide](/docs/started/workstation/getting-started-macos/). + * To use Kubeflow on Linux, + follow the [Linux deployment guide](/docs/started/workstation/getting-started-linux/). + +* Installing Kubeflow on a **existing Kubernetes cluster** or a **public cloud**: -* To use Kubeflow on Google Cloud Platform (GCP) and Kubernetes Engine (GKE), + * Installing Kubeflow on a Kubernetes cluster, follow the + [guide to deploying Kubeflow on Kubernetes](/docs/started/k8s/overview/). + * To use Kubeflow on Google Cloud Platform (GCP) and Kubernetes Engine (GKE), follow the [GCP deployment guide](/docs/gke/deploy/). -* To use Kubeflow on Amazon Web Services (AWS), + * To use Kubeflow on Amazon Web Services (AWS), follow the [AWS deployment guide](/docs/aws/deploy/). -* To use Kubeflow on Microsoft Azure Kubernetes Service (AKS), + * To use Kubeflow on Microsoft Azure Kubernetes Service (AKS), follow the [AKS deployment guide](/docs/azure/deploy/). -* To use Kubeflow on IBM Cloud Private (ICP), - follow the [ICP deployment guide](/docs/started/getting-started-icp/). -* If you have an existing Kubernetes cluster or want to use Kubeflow on prem, - follow the [guide to deploying Kubeflow on - Kubernetes](/docs/started/k8s/overview/). -* If you want to run Kubernetes locally in a virtual machine (VM), choose one of - the following options: - - * [MiniKF setup](/docs/started/getting-started-minikf/) - * MiniKF is a fast and easy way to get started with Kubeflow. - * It installs with just two commands and then you are up for - experimentation, and for running complete Kubeflow Pipelines. - * MiniKF runs on all major operating systems (Linux, macOS, Windows). - - * [Minikube setup](/docs/started/getting-started-minikube/) - * Minikube uses virtualization applications like - [VirtualBox](https://www.virtualbox.org/) or [VMware - Fusion](https://www.vmware.com/products/fusion.html) to host the VM - and provides a CLI that you can use outside the VM. - * Minikube defines a fully-baked - [ISO image](https://en.wikipedia.org/wiki/ISO_image) that contains a - minimal operating system and Kubernetes already installed. - * This option may be useful if you are just starting to learn and already - have one of the virtualization applications installed. - - * [MicroK8s setup](/docs/started/getting-started-multipass/) - * [MicroK8s](https://microk8s.io/) can provide the following benefits: - - A small, fast, secure, single node Kubernetes installation that installs on any - Linux system as a [snap](https://snapcraft.io/microk8s). - - Strong isolation and update semantics - your cluster - is updated within a short period after upstream Kubernetes - releases. - - Built-in support to enable an installed GPU: - `microk8s.enable gpu` - * MicroK8s requires Linux. If you are not on a Linux machine, or you want - to confine your Kubeflow to a disposable machine, the installation guide - show you how to use - [Multipass](https://github.com/CanonicalLtd/multipass) to launch a VM. - Benefits include: - - [Ubuntu Cloud Images](http://cloud-images.ubuntu.com/) already - integrated. - - Lightweight hypervisor using native operating system mechanisms - (for example, [Hypervisor - Framework](https://developer.apple.com/documentation/hypervisor) on - macOS, [Hyper-V on Windows - 10](https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v), or - QEMU/KVM for Linux). - - No need to install a separate virtualization application. - - Use of `cloud-init` to customize the VM. + * To use Kubeflow on IBM Cloud Private (ICP), + follow the [ICP deployment guide](/docs/started/cloud/getting-started-icp/). + ## Installing command line tools @@ -101,12 +54,16 @@ tools for deploying and managing Kubeflow: * Download the `kfctl` binary from the [Kubeflow releases page](https://github.com/kubeflow/kubeflow/releases/). -* Follow the `kubectl` installation and setup from the [Kubernetes +* Follow the `kubectl` installation and setup instructions from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl/). - As described in the Kubernetes documentation, your kubectl + As described in the Kubernetes documentation, your kubectl version must be within one minor version of the Kubernetes version that you use in your Kubeflow cluster. +* Follow the `kustomize` installation and setup instructions from the + [kustomize component guide](/docs/components/misc/kustomize/). + + ## Troubleshooting See the [Kubeflow troubleshooting guide](/docs/other-guides/troubleshooting/). @@ -114,7 +71,7 @@ See the [Kubeflow troubleshooting guide](/docs/other-guides/troubleshooting/). ## Next steps * Read the [documentation](/docs/) for in-depth instructions on using Kubeflow. -* Explore the [tutorials and +* Explore the [tutorials and codelabs](/docs/examples/codelabs-tutorials/) for learning and trying out Kubeflow. * Build machine-learning pipelines with the [Kubeflow Pipelines SDK](/docs/pipelines/sdk/sdk-overview/). diff --git a/content/docs/started/k8s/_index.md b/content/docs/started/k8s/_index.md index b367c5b3cb..cd3278aeb8 100644 --- a/content/docs/started/k8s/_index.md +++ b/content/docs/started/k8s/_index.md @@ -1,5 +1,5 @@ +++ -title = "Kubeflow on Existing Kubernetes Clusters" -description = "Instructions for installing Kubeflow on your existing clusters" +title = "Kubernetes Installation" +description = "Instructions for installing Kubeflow on an existing Kubernetes cluster" weight = 2 +++ diff --git a/content/docs/started/k8s/overview.md b/content/docs/started/k8s/overview.md index 9089023a17..312f56b8f1 100644 --- a/content/docs/started/k8s/overview.md +++ b/content/docs/started/k8s/overview.md @@ -4,11 +4,25 @@ description = "Instructions for installing Kubeflow on your existing Kubernetes weight = 1 +++ -Follow these instructions if you want to install Kubeflow on an existing Kubernetes cluster. +Follow these instructions if you want to install Kubeflow on an existing Kubernetes +cluster. Some [clouds](/docs/started/cloud) and Kubernetes distributions provide +Kubeflow specific instructions for getting the most out of their Kubernetes. If your +existing Kubernetes cluster is from one of those, consider following those instructions. -If you are using a Kubernetes distribution or Cloud Provider which has specific instructions for installing Kubeflow we recommend following those instructions. Those instructions do additional Cloud specific setup to create a really great Kubeflow experience. +## Minimum system requirements -The following table lists the options for installing Kubeflow on an existing Kubernetes Cluster and links to detailed instructions. +The Kubernetes cluster must meet the following minimum requirements: + + * Kubernetes version {{% kubernetes-min-version %}} or later. + * At least one worker node with a minimum of: + * 4 CPU + * 50 GB storage + * 12 GB memory + +## Kubeflow Deployment Configurations + +The following tables list the options for installing Kubeflow on an existing Kubernetes +Cluster and links to detailed instructions. ### Community maintained diff --git a/content/docs/started/workstation/_index.md b/content/docs/started/workstation/_index.md new file mode 100644 index 0000000000..2ecc7a464d --- /dev/null +++ b/content/docs/started/workstation/_index.md @@ -0,0 +1,5 @@ ++++ +title = "Workstation Installation" +description = "Instructions for installing Kubeflow on a workstation or server" +weight = 2 ++++ diff --git a/content/docs/started/workstation/getting-started-linux.md b/content/docs/started/workstation/getting-started-linux.md new file mode 100644 index 0000000000..ec47c008e6 --- /dev/null +++ b/content/docs/started/workstation/getting-started-linux.md @@ -0,0 +1,78 @@ ++++ +title = "Kubeflow on Linux" +description = "Install Kubeflow on Linux" +weight = 2 ++++ + + + +For Linux systems you have options for servers (physical or virtual) and desktops. +The server options apply to the desktop as well. + +## Linux Server + +For linux servers you can install Kubeflow natively. This is perfect for +linux hosts and virtual machines, such as VMs in OpenStack, VMware or public clouds like +GCP, AWS and Azure. + +### MicroK8s + +[MicroK8s](https://microk8s.io) runs natively on most Linux distributions. It requires +[snapd](https://snapcraft.io/docs/getting-started), which is pre-installed on Ubuntu +and Ubuntu derivative operating systems. + +1. Install MicroK8s - `snap install microk8s --classic` +2. Follow the getting started guid for Kubeflow on an +[existing Kubernetes cluster](/docs/started/k8s/overview/). + +## Linux Desktop + +### Kubeflow Appliance + +A Kubeflow appliance is a virtual machine that has Kubeflow already installed. Once the +necessary supporting software is installed no further installation steps are required. + +#### MiniKF + +MiniKF is a predefined virtual machine that installs onto VirtualBox through Vagrant. +The only following applications are required to use MiniKF: + +- Install [Vagrant](https://www.vagrantup.com/downloads.html) +- Install [Virtual Box](https://www.virtualbox.org/wiki/Downloads) + +The full set of instructions are available on the +[MiniKF getting started](/docs/reference/virtual-dev/getting-started-minikf/) page. + +### Linux Appliance + +A Linux appliance is a virtual machine that holds the linux operating system. From there +you have complete choice over Kubernetes and Kubeflow, which offers the greatest degree +of flexibility. You only need to install a single application to follow this path: + +- Install [Multipass](https://multipass.run/#install) + +The instructions on [Multipass and MicroK8s getting started](/docs/reference/virtual-dev/getting-started-multipass/) +page will complete this path. + +### Kubernetes Appliance + +Similar to the Kubeflow appliance, the Kubernetes appliance is a virtual machine has a +Kubernetes cluster already installed. After starting the virtual machine you will need +to install Kubeflow. This option gives you full control over your Kubeflow setup. + +#### Minikube + +Minikube runs a simple, single-node Kubernetes cluster inside a virtual machine (VM). +You can choose amongst a couple of hypervisor applications. Similar to the Kubeflow +appliance, you only need to install a couple of applications, and then install Kubeflow: + +- Install a Hypervisor (*one of the following*) + - Install [Vagrant](https://www.vagrantup.com/downloads.html) + - Install [VMware Fusion](https://www.vmware.com/products/fusion) +- Install [Minikube](https://github.com/kubernetes/minikube/releases) + +The full set of instructions are available on the +[Minikube getting started](/docs/reference/virtual-dev/getting-started-minikube/) page. diff --git a/content/docs/started/workstation/getting-started-macos.md b/content/docs/started/workstation/getting-started-macos.md new file mode 100644 index 0000000000..5e9e9b5222 --- /dev/null +++ b/content/docs/started/workstation/getting-started-macos.md @@ -0,0 +1,62 @@ ++++ +title = "Kubeflow on MacOS" +description = "Install Kubeflow on MacOS" +weight = 2 ++++ + + + +For MacOS systems you have multiple options for getting started. The options range +from fully-assembled Kubeflow stacks, to stacks that require some assembly. + +## Kubeflow Appliance + +A Kubeflow appliance is a virtual machine that has Kubeflow already installed. Once the +necessary supporting software is installed no further installation steps are required. + +### MiniKF + +MiniKF is a predefined virtual machine that installs onto VirtualBox through Vagrant. +The only following applications are required to use MiniKF: + +- Install [Vagrant](https://www.vagrantup.com/downloads.html) +- Install [Virtual Box](https://www.virtualbox.org/wiki/Downloads) + +The full set of instructions are available on the +[MiniKF getting started](/docs/reference/virtual-dev/getting-started-minikf/) page. + +## Linux Appliance + +A Linux appliance is a virtual machine that holds the linux operating system. From there +you have complete choice over Kubernetes and Kubeflow, which offers the greatest degree +of flexibility. You only need to install a single application to follow this path: + +- Install [Multipass](https://multipass.run/#install) + +The instructions on [Multipass and MicroK8s getting started](/docs/reference/virtual-dev/getting-started-multipass/) +page will complete this path. + +## Kubernetes Appliance + +Similar to the Kubeflow appliance, the Kubernetes appliance is a virtual machine has a +Kubernetes cluster already installed. After starting the virtual machine you will need +to install Kubeflow. This option gives you full control over your Kubeflow setup. + +### Minikube + +Minikube runs a simple, single-node Kubernetes cluster inside a virtual machine (VM). +You can choose amongst a couple of hypervisor applications. Similar to the Kubeflow +appliance, you only need to install a couple of applications, and then install Kubeflow: + +- Install a Hypervisor (*one of the following*) + - Install [Vagrant](https://www.vagrantup.com/downloads.html) + - Install [VMware Fusion](https://www.vmware.com/products/fusion) +- Install [Minikube](https://github.com/kubernetes/minikube/releases) + +The full set of instructions are available on the +[Minikube getting started](/docs/reference/virtual-dev/getting-started-minikube/) page. diff --git a/content/docs/started/workstation/getting-started-windows.md b/content/docs/started/workstation/getting-started-windows.md new file mode 100644 index 0000000000..90a0dff5b4 --- /dev/null +++ b/content/docs/started/workstation/getting-started-windows.md @@ -0,0 +1,66 @@ ++++ +title = "Kubeflow on Windows" +description = "Install Kubeflow on Windows" +weight = 2 ++++ + + + +For Windows systems you have multiple options for getting started. The options range +from fully-assembled Kubeflow stacks, to stacks that require some assembly. +In addition, with the recent announcement of +[Windows WSL 2](https://devblogs.microsoft.com/commandline/announcing-wsl-2/), +some of the [linux installation options](/docs/started/workstation/getting-started-linux) +for Kubeflow will be available on Windows, once WSL2 is formally released. + +## Kubeflow Appliance + +A Kubeflow appliance is a virtual machine that has Kubeflow already installed. Once the +necessary supporting software is installed no further installation steps are required. + +### MiniKF + +MiniKF is a predefined virtual machine that installs onto VirtualBox through Vagrant. +The following applications are required to use MiniKF: + +- Install [Vagrant](https://www.vagrantup.com/downloads.html) +- Install [Virtual Box](https://www.virtualbox.org/wiki/Downloads) + +The full set of instructions are available on the +[MiniKF getting started](/docs/reference/virtual-dev/getting-started-minikf/) page. + +## Linux Appliance + +A Linux appliance is a virtual machine that holds the linux operating system. From there +you have complete choice over Kubernetes and Kubeflow, which offers the greatest degree +of flexibility. You only need to install a single application to follow this path: + +- Install [Multipass](https://multipass.run/#install) + +The instructions on [Multipass and MicroK8s getting started](/docs/reference/virtual-dev/getting-started-multipass/) +page will complete this path. + +## Kubernetes Appliance + +Similar to the Kubeflow appliance, the Kubernetes appliance is a virtual machine has a +Kubernetes cluster already installed. After starting the virtual machine you will need +to install Kubeflow. This option gives you full control over your Kubeflow setup. + +### Minikube + +Minikube runs a simple, single-node Kubernetes cluster inside a virtual machine (VM). +You can choose amongst a couple of hypervisor applications. Similar to the Kubeflow +appliance, you only need to install a couple of applications, and then install Kubeflow: + +- Install a Hypervisor (*one of the following*) + - Install [Vagrant](https://www.vagrantup.com/downloads.html) + - Install [VMware Fusion](https://www.vmware.com/products/fusion) +- Install [Minikube](https://github.com/kubernetes/minikube/releases) + +The full set of instructions are available on the +[Minikube getting started](/docs/reference/virtual-dev/getting-started-minikube/) page.