From 19084ec5c2f7f67305f9fdfa34e23b9c61d49900 Mon Sep 17 00:00:00 2001 From: Tim Head Date: Sat, 23 Jan 2021 18:54:18 +0100 Subject: [PATCH] Fix syntax error --- CONTRIBUTING.md | 338 +++++++++++++++----------------- doc/developer/repoproviders.rst | 4 +- 2 files changed, 163 insertions(+), 179 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 28812d95d0..ef88309c71 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,34 +1,48 @@ # Contributing to BinderHub -Welcome! As a [Jupyter](https://jupyter.org) project, we follow the +Welcome! Thanks for spending time on developing BinderHub! +As a [Jupyter](https://jupyter.org) project, we follow the [Jupyter contributor guide](https://jupyter.readthedocs.io/en/latest/contributing/content-contributor.html). -Depending on what you want to develop, you can setup BinderHub in different ways. -- [Develop documentation](#develop-documentation). -- [Develop user interface](#develop-user-interface). A BinderHub webserver is running locally and - JupyterHub is mocked, this setup doesn't involve Kubernetes. -- [Develop Kubernetes integration](#develop-kubernetes-integration). A BinderHub webserver is running locally, - and JupyterHub is installed in a Kubernetes cluster. -- [Develop Helm chart](#develop-helm-chart) - The BinderHub Helm chart with JupyterHub as a - dependency is installed in a Kubernetes cluster. +For all work on BinderHub you will need a copy of the source code. - This document also contains information on [how to run tests](#running-tests) and - [common maintainer tasks](#common-maintainer-tasks). - - -## Develop documentation - -You are assumed to have a modern version of [Python](https://www.python.org/). -The documentation uses the [reStructuredText markup -language](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html). - -1. Clone the BinderHub repository to your local computer and `cd` into it. +[Fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) the BinderHub repository on [GitHub](https://github.com/jupyterhub/binderhub) and [clone](https://help.github.com/articles/cloning-a-repository/) it to your local computer. + Then `cd` into it. ```bash git clone https://github.com/jupyterhub/binderhub cd binderhub ``` + Add a [remote](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork) and regularly [sync](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork) to make sure you stay up-to-date with our repository: + + ```bash + git remote add upstream https://github.com/jupyterhub/binderhub.git + git checkout master + git fetch upstream + git merge upstream/master + ``` + +There are several different setups for working on BinderHub. Which setup is +best for you depends on which parts of BinderHub you want to change: + +* the [documentation](#documentation-changes), +* the [user interface](#user-interface-changes), +* the [Python server](#python-server-changes), or +* the [helm chart](#helm-chart-changes). + +Jump to the linked section for instructions on how to get setup. + +This document also contains information on [common maintainer tasks](#common-maintainer-tasks). + + +## Documentation changes + +Work on the documentation requires the least amount of setup. You will need +to have a modern version of [Python](https://www.python.org/). The documentation +uses the [reStructuredText markup language](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html). + + 1. Install BinderHub and the documentation tools: ```bash @@ -52,19 +66,12 @@ language](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html terminal with `open _build/html/index.html`. -## Develop user interface +## User interface changes -Developing BinderHub's user interface can be done both without Kubernetes and -JupyterHub by mocking that interaction. You are assumed to have a modern version -of [Python](https://www.python.org/) and [node / npm](https://nodejs.org/) -installed. +Work on the user interface requires a medium amount of setup. You will need +to have a modern version of [Python](https://www.python.org/) and +[npm](https://www.npmjs.com) installed. -1. Clone the BinderHub git repository to your local computer and `cd` into it. - - ```bash - git clone https://github.com/jupyterhub/binderhub - cd binderhub - ``` 1. Install BinderHub, the Python package. @@ -78,93 +85,156 @@ installed. npm install ``` -1. Create the JS and CSS bundles BinderHub serves as a webserver to visitors. +1. Create the JS and CSS bundles BinderHub with: ```bash - npm run webpack:watch + npm run webpack ``` + Note: you need to run this every time you make a change to the CSS or JS + for them to take effect. Check out `npm run webpack:watch` to automate this. -1. Start the BinderHub webserver locally. +1. Start the BinderHub webserver. ```bash python3 -m binderhub -f testing/local-binder-mocked-hub/binderhub_config.py ``` -1. Visit the BinderHub webserver at http://localhost:8585. +1. Visit http://localhost:8585 to see it in action. Building and launching repositories will not work. You can still work on the user interface of those parts as BinderHub is configured to fake those actions. You can tell you are using the fake builder and launcher from the fact that the build will never complete. -To learn how to set yourself with a BinderHub development environment that lets -you modify the builder and launcher refer to [Develop Kubernetes -integration](#develop-kubernetes-integration) or [Develop Helm -chart](#develop-helm-chart). +To learn how to set yourself up with a BinderHub development environment that lets +you modify the builder and launcher refer to the [Python server changes](#python-server-changes) section. -## Develop Kubernetes integration -This requires `helm` and a functional Kubernetes cluster. Please do -[preliminary Kubernetes setup](#kubernetes-setup) if you haven't already -before continuing here. +## Python server changes -With a Kubernetes cluster running, as you verify with `kubectl version`, you can -continue. +Setting yourself up to make changes to the kubernetes integration of BinderHub +requires a few one-time setup steps. These steps are described in the +"One-time installation" section below. Follow those first then return here for +day to day development procedures. + + +### Day to day development tasks + +After completing the "One-time installation" steps you are ready for day-to-day +development. These are the tasks you need to perform for every day development: -1. Locally install BinderHub as a Python package and its development - requirements locally. +* Start and stop minikube with `minikube start` and `minikube stop`. +* Install JupyterHub in minikube with helm `./testing/local-binder-k8s-hub/install-jupyterhub-chart` +* Setup `docker` to use the same Docker daemon as your minikube cluster `eval $(minikube docker-env)` +* Start BinderHub `python3 -m binderhub -f testing/local-binder-k8s-hub/binderhub_config.py` +* Visit your BinderHub at http://localhost:8585 + +To test that everything works as it should visit http://localhost:8585/v2/gh/binder-examples/minimal-dockerfile/HEAD. This will build a very simple repository and should eventually +show you the classic Jupyter notebook interface. + +To execute most of our test suite you need a running minikube cluster. +It does not need to have anything installed on it though: + +```bash +minikube start +pytest -svx -m "not auth" +``` + +The tests should generate familiar pytest like output and complete in a few +seconds. + +To run more advanced test or the full suite we recommend you open a Pull Request +and let our Continuous Integration system run them. This is by far the easiest +way to find out if your changes pass all tests. Running these tests locally +is considered an expert task. To find out how to do it please read +`.github/workflows/test.yml` and replicate it for your local setup. + +After working on BinderHub you can [clean up](#cleaning-up-resources). + + +### One-time installation + +To run a full BinderHub it needs to have access to a kubernetes cluster +with a JupyterHub installed on it. This is what we will setup in this section. +All the steps are given as command-line commands for Ubuntu systems. They are +used as a common denominator that can be translated into the correct commands +on your local system. + +Before you begin, there are a few utilities that need to be installed: +```bash +sudo apt install python3 python3-pip npm curl +``` + +If you are on linux, you may additionally need to install socat for port forwarding: + +```bash +sudo apt install socat +``` + +1. [Install Minikube](https://minikube.sigs.k8s.io/docs/start/) to run Kubernetes locally. + + To start your minikube cluster , run the command: `minikube start`. This + starts a local kubernetes cluster inside a virtual machine. This command + assumes that you have already installed one of the VM drivers: virtualbox, + xhyve or KVM2. + +1. [Install helm](https://helm.sh/docs/intro/quickstart/#install-helm) to + install JupyterHub and BinderHub on your cluster. + +1. Add the JupyterHub helm charts repo: + + ```bash + helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/ + helm repo update + ``` + +1. Install BinderHub the Python package and its development requirements: ```bash python3 -m pip install -e . -r dev-requirements.txt ``` -1. Install the JupyterHub Helm chart by itself into your Kubernetes cluster - current namespace. +1. Install JupyterHub in minikube with helm - ```bash - # Append --auth here if you want to develop against a non-public BinderHub - # that relies on JupyterHub's configured Authenticator to decide if the users - # are allowed access or not. - ./testing/local-binder-k8s-hub/install-jupyterhub-chart - ``` + ```bash + ./testing/local-binder-k8s-hub/install-jupyterhub-chart + ``` -1. Configure `docker` using environment variables to use the same Docker daemon - as your `minikube` cluster. This means images you build are directly - available to the cluster. +1. Before starting your local deployment run: - ```bash - eval $(minikube docker-env) - ``` + ```bash + eval $(minikube docker-env) + ``` -1. Start BinderHub with the testing config file. + This command sets up `docker` to use the same Docker daemon as your minikube + cluster does. This means images you build are directly available to the + cluster. Note: when you no longer wish to use the minikube host, you can + undo this change by running: - ```bash - python3 -m binderhub -f testing/local-binder-k8s-hub/binderhub_config.py - ``` + ```bash + eval $(minikube docker-env -u) + ``` -1. Visit [http://localhost:8585](http://localhost:8585) +You are now setup for work on the Python server. Return to "day-to-day development tasks". -1. Congratulations, you can now make changes and see how they influence the - deployment. You may be required to restart the BinderHub depending on what - you change. You can also start running `pytest` tests to verify the - Deployment functions as it should. +### Cleaning up resources -### Cleanup resources +After you are done working on BinderHub you can clean up after yourself. -1. To cleanup the JupyterHub Helm chart you have installed in Kubernetes... +1. To cleanup the JupyterHub Helm chart you have installed in Kubernetes: ```bash helm delete binderhub-test ``` -1. To stop running the Kubernetes cluster... +1. To stop running the Kubernetes cluster: ```bash minikube stop ``` -## Develop Helm chart +## Helm chart changes This requires `helm` and a functional Kubernetes cluster. Please do [preliminary Kubernetes setup](#kubernetes-setup) if you haven't already @@ -222,80 +292,11 @@ continue. how they influence the deployment. You can also start running `pytest` tests to verify the Deployment functions as it should. -### Cleanup resources -1. To cleanup resources you have installed and start fresh... - - ```bash - helm delete binderhub-test - ``` - -1. To stop running the Kubernetes cluster... - - ```bash - minikube stop - ``` - -## Kubernetes setup - -A fully functional BinderHub needs to have access to a Kubernetes cluster with a -JupyterHub installed on it. You can either run BinderHub as a Python webserver -locally and install JupyterHub on its own in the Kubernetes cluster, or install -the entire BinderHub Helm chart which installs the JupyterHub Helm chart as a -dependency. - -All the steps are given as command-line commands for Ubuntu systems. They are -used as a common denominator that can be translated into the correct commands on -your local system. - -You are assumed to have a modern version of [Python](https://www.python.org/), -[node / npm](https://nodejs.org/) installed, and the command line tool `curl`. - -1. [Install Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/). - -1. Start a minikube Kubernetes cluster inside a virtual machine (virtualbox, - xhyve, or KVM2). - - ```bash - minikube start - ``` - -1. Install `kubectl` - the CLI to interact with a Kubernetes cluster. - - ```bash - curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" - chmod +x kubectl - sudo mv kubectl /usr/local/bin/ - ``` - - Here are the [official installation instructions](https://kubernetes.io/docs/tasks/tools/install-kubectl/). - -1. Install `helm` - the Kubernetes package manager. - - ```bash - curl -sf https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 - ``` - - Here are the [official installation instructions](https://helm.sh/docs/intro/install/). - -1. Let `helm` know about the official JupyterHub Helm chart repository. - - ```bash - helm repo add --force-update jupyterhub https://jupyterhub.github.io/helm-chart/ - helm repo update - ``` - -1. Clone the binderhub git repository to your local computer and `cd` into it. - - ```bash - git clone https://github.com/jupyterhub/binderhub - cd binderhub - ``` - -### Tip: Use local repo2docker version +## Tip: Use local repo2docker version BinderHub runs repo2docker in a container. For testing the combination of an -unreleased repo2docker feature with BinderHub, you can use a locally build +unreleased repo2docker feature with BinderHub, you can use a locally built repo2docker image. You can configure the image in the file `testing/local-binder-k8s-hub/binderhub_config.py` with the following line: @@ -303,18 +304,18 @@ repo2docker image. You can configure the image in the file c.BinderHub.build_image = 'jupyter-repo2docker:my_image_tag' ``` -**Important**: the image must be build using the same Docker daemon as the +**Important**: the image must be built using the same Docker daemon as the minikube cluster, otherwise you get an error _"Failed to pull image [...] repository does not exist or may require 'docker login'"_. -### Tip: Increase your GitHub API limit +## Tip: Increase your GitHub API limit By default, GitHub has a limit of 60 API requests per hour. We recommend using a GitHub API token before running tests in order to avoid hitting your API limit. Steps to do so are outlined in the [BinderHub documentation](https://binderhub.readthedocs.io/en/latest/setup-binderhub.html#increase-your-github-api-limit). -### Tip: Start Minikube with more memory +## Tip: Start Minikube with more memory By default, `minikube start` allocates 2GiB of main memory to the underlying VM, which might be too low to run the builder successfully. @@ -323,44 +324,27 @@ You may run `minikube start --memory 8192` to start Minikube with a 8GiB VM underneath. - - -## Running tests - -This git repository contains `pytest` based tests that you can run locally. -Depending on your development setup, you may want to run different kind of -tests. You can get some hints on what tests to run and how by inspecting -`.travis.yaml`. - -### Environment variables influencing tests -- `BINDER_URL`: An address of an already running BinderHub as reachable from the - tests. If this is set, the test suite will not start temporary local BinderHub - servers but instead interact with the remote BinderHub. -- `GITHUB_ACCESS_TOKEN`: A GitHub access token to help avoid quota limitations - for anonymous users. It is used to enable certain tests making many calls to - GitHub API. - -### Pytest marks labelling tests -- `remote`: Tests for them the BinderHub is already running somewhere. -- `github_api`: Tests that communicate with the GitHub API a lot. -- `auth`: Tests related to BinderHub's usage of JupyterHub as an OAuth2 Identity - Provider (IdP) for non public access. - - ## Common maintainer tasks These are tasks that BinderHub maintainers perform. - ### Bumping the JupyterHub Helm chart version The BinderHub Helm chart depends on the [JupyterHub Helm chart](https://jupyterhub.github.io/helm-chart/), and its version is pinned -within `helm-chart/binderhub/requirements.yaml`. It is straightforward to update -it with another version from the [JupyterHub Helm chart -repository](https://jupyterhub.github.io/helm-chart/). +within `helm-chart/binderhub/requirements.yaml`. + +To bump the version of JupyterHub that BinderHub uses, go to the [JupyterHub Helm Chart](https://jupyterhub.github.io/helm-chart/) version page, find the release +hash that you want and update the following field in the `requirements.yaml` file: + + ```yaml + dependencies: + version: "" + ``` -Use the [JupyterHub Helm chart's +**Make sure to double-check that there are no breaking changes in JupyterHub**. +Sometimes JupyterHub introduces breaking changes to its helm chart (such as the +structure of particular fields). Use the [JupyterHub Helm chart's changelog](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/CHANGELOG.md) to prepare for breaking changes associated with the version bump. diff --git a/doc/developer/repoproviders.rst b/doc/developer/repoproviders.rst index e189a44419..36c521aea1 100755 --- a/doc/developer/repoproviders.rst +++ b/doc/developer/repoproviders.rst @@ -30,9 +30,9 @@ Currently supported providers, their prefixes and specs are: +------------+--------------------+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | Zenodo | ``zenodo`` | ```` | `Zenodo `_ is a non-profit provider of scholarly artifacts (such as code repositories) run in partnership with CERN. | +------------+--------------------+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ - | Figshare | ``figshare`` | ```` | `FigShare `_ is a company that offers hosting for scholarly artifacts (such as code repositories). | + | Figshare | ``figshare`` | ```` | `FigShare `_ is a company that offers hosting for scholarly artifacts (such as code repositories). | +------------+--------------------+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ - | HydroShare | ``hydroshare`` | ```` | `HydroShare `_ is a hydrologic information system for users to share and publish data and models. | + | HydroShare | ``hydroshare`` | ```` | `HydroShare `_ is a hydrologic information system for users to share and publish data and models. | +------------+--------------------+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | Dataverse | ``dataverse`` | ```` | `Dataverse `_ is open source research data repository software installed all over the world. | +------------+--------------------+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+