Reorganize repository to clarify relationship betwen the devworkspace…

… API and devfile 2.0 (#23)

* Restructure the repository in order to:
  - Make a clearer separation between devworkspace and devfile schemas,
  - Provide jsonpatch rules to derive the devfile schema from the devworkspace one
  - Isolate devfile-related elements in a dedicated folder (jsonpatch rules and devfile samples)
  - Generate all the schemas from the Go code in a single build command
  - Allow running the `build` command in a docker container that contains the prerequisites
* Add a `devfile.yaml`
* Add theia settings to setup json schema bindings
* Links in readmes
* Add basic intro + how to build in the readme

Signed-off-by: David Festal <dfestal@redhat.com>

Co-authored-by: Angel Misevski <amisevsk@redhat.com>
Co-authored-by: Florent BENOIT <fbenoit@redhat.com>
Co-authored-by: Mario Loriedo <mario.loriedo@gmail.com>

.gitignore

@@ -1,4 +1,3 @@
-generated
 # Temporary Build Files
 build/_output
 build/_test

.theia/settings.json

@@ -0,0 +1,8 @@
+{
+  "go.lintTool": "golangci-lint",
+  "go.lintFlags": "--fast",
+  "yaml.schemas": {
+      "./schemas/devfile.json": "devfile-support/samples/*devfile.yaml",
+      "./schemas/devworkspace.json": "samples/*devworkspace.yaml"
+  }
+}

README.md

@@ -1,2 +1,31 @@
-# devworkspace-api
-K8S Api for a cloud develoment workspace specification [Draft proposal]
+# Kube-native API for cloud development workspaces specification
+
+Sources for this API are defined in Go code, starting from the
+[devworkspace_types.go source file](pkg/apis/workspaces/v1alpha1/devworkspace_types.go)
+
+From these Go sources, several files are generated:
+- A Kubernetes Custom Resource Definition with an embedded OpenApi schema,
+- json schemas (in the [schemas](schemas) folder) generated from the above Custom Resource Definition, to specify the syntax of:
+  - the DevWorkspace CRD itself
+  - the DevWorkspaceTemplate CRD (a workspace content, without runtime information),
+  - the Devfile 2.0.0 format, which is generated from the `DevWorkspace` API.
+
+Generated files are created by a build script (see section [How to build](#how-to-build)).
+
+### Devfile 2.0.0 file format
+
+A Subset of this `DevWorkspace` API defines a structure (workspace template content), which is also at the core of the **Devfile 2.0** format specification.
+For more information about this, please look into
+the [Devfile support Readme](devfile-support/README.md)
+
+## How to build
+
+In order to build the CRD and the various schemas, you don't need to install any pre-requisite apart from `docker`.
+In the root directory, just run the following command:
+
+```
+./docker-run.sh build.sh
+``` 
+## Specification status
+
+This work is still in an early stage of specification, and the related API and schemas are still a draft proposal. 

build.sh

@@ -0,0 +1,76 @@
+#!/bin/bash
+#
+# Copyright (c) 2020 Red Hat, Inc.
+# This program and the accompanying materials are made
+# available under the terms of the Eclipse Public License 2.0
+# which is available at https://www.eclipse.org/legal/epl-2.0/
+#
+# SPDX-License-Identifier: EPL-2.0
+#
+# Contributors:
+#   Red Hat, Inc. - initial API and implementation
+
+BLUE='\033[1;34m'
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+NC='\033[0m'
+BOLD='\033[1m'
+
+if ! command -v yq &> /dev/null
+then
+  echo
+  echo "#### ERROR ####"
+  echo "####"
+  echo "#### Please install the 'yq' tool before being able to use this script"
+  echo "#### see https://github.com/kislyuk/yq"
+  echo "#### and https://stedolan.github.io/jq/download"
+  echo "####"
+  echo "###############"
+  exit 1
+fi
+
+if ! command -v jsonpatch &> /dev/null
+then
+  echo
+  echo "#### ERROR ####"
+  echo "####"
+  echo "#### Please install the 'jsonpatch' tool before being able to use this script"
+  echo "#### For this use:"
+  echo "####    pip3 install --user jsonpatch"
+  echo "####"
+  echo "###############"
+  exit 1
+fi
+
+command -v operator-sdk >/dev/null 2>&1 || { echo -e "${RED}operator-sdk is not installed. Aborting.${NC}"; exit 1; }
+
+operatorVersion=$(operator-sdk version)
+[[ $operatorVersion =~ .*v0.12.0.* ]] || { echo -e "${RED}operator-sdk v0.12.0 is required${NC}"; exit 1; }
+
+set -e
+
+BASE_DIR=$(cd "$(dirname "$0")" && pwd)
+
+mkdir -p "${BASE_DIR}/generated"
+
+operator-sdk generate k8s
+operator-sdk generate openapi
+yq '.spec.validation.openAPIV3Schema' \
+  "${BASE_DIR}/deploy/crds/workspaces.ecd.eclipse.org_devworkspaces_crd.yaml" \
+  > "${BASE_DIR}/schemas/devworkspace.json"
+
+jq ".properties.spec.properties.template" "${BASE_DIR}/schemas/devworkspace.json" > "${BASE_DIR}/schemas/devworkspace-template-spec.json"
+
+cp "${BASE_DIR}/schemas/devworkspace-template-spec.json" "${BASE_DIR}/schemas/devfile.json"
+
+onError() {
+  echo "Cleaning schemas/devfile.json"
+  rm -f "${BASE_DIR}/schemas/devfile.json"
+}
+trap 'onError' ERR
+
+for jsonpatch in "${BASE_DIR}"/devfile-support/transformation-rules/*.json
+do
+  [ -e "$jsonpatch" ] || continue
+  jsonpatch -i --indent 2 "${BASE_DIR}/schemas/devfile.json" "${jsonpatch}"
+done

devfile-support/README.md

@@ -0,0 +1,7 @@
+# Devfile 2.0 support
+
+## Relationship between the cloud workspace API and the Devfile
+
+## Samples
+
+- Simple devfile: [simple-devfile.yaml](samples/simple-devfile.yaml)

devfiles/custom-devfile.yaml → devfile-support/samples/custom-devfile.yaml

@@ -1,3 +1,5 @@
+name: "my-devfile"
+schemaVersion: "2.0.0-beta"
 projects:
   - name: "my-project"
     custom:

devfiles/sample-devfile.yaml → devfile-support/samples/sample-devfile.yaml

@@ -1,3 +1,5 @@
+name: "devfile example"
+schemaVersion: "2.0.0"
 projects:
   - name: "my-project"
     custom:

devfiles/simple-devfile.yaml → devfile-support/samples/simple-devfile.yaml

@@ -1,3 +1,5 @@
+name: "myDevile"
+schemaVersion: "2.0.0"
 projects:
   - name: "devworkspace-spec"
     git:

devfile-support/transformation-rules/add-name-and-version.json

@@ -0,0 +1,22 @@
+[
+  {
+    "op": "replace",
+    "path": "/description",
+    "value": "Devfile schema."
+  },
+  { 
+    "op": "add",
+    "path": "/properties/name",
+    "value": { "type": "string", "description": "devfile name" }
+  },
+  { 
+    "op": "add",
+    "path": "/properties/schemaVersion",
+    "value": { "type": "string", "description": "Devfile schema version", "pattern": "^([2-9]+)\\.([0-9]+)\\.([0-9]+)(\\-[0-9a-z-]+(\\.[0-9a-z-]+)*)?(\\+[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?$" }
+  },
+  {
+    "op": "add",
+    "path": "/required",
+    "value": [ "name", "schemaVersion" ]
+  }
+]

devfile.yaml

@@ -0,0 +1,28 @@
+apiVersion: 1.0.0
+metadata:
+  name: devworkspace-api
+components:
+  - id: ms-vscode/go/latest
+    memoryLimit: 512Mi
+    preferences:
+      go.lintTool: golangci-lint
+      go.lintFlags: '--fast'
+    type: chePlugin
+    alias: go-plugin
+  - id: redhat/vscode-yaml/latest
+    type: chePlugin
+  - mountSources: true
+    command:
+      - /bin/sh
+    args:
+      - '-c'
+      - >-
+        while [ ! -f /projects/devworkspace-api/.theia/settings.json -o 
+        ! -f /projects/.theia/settings.json ]; do echo "Waiting for the clone end" ; sleep 2; done;
+        sleep 2 ; echo "Updating the yaml schema bindings" ; sed -e
+        's:/schemas/:/devworkspace-api/schemas/:g'
+        /projects/devworkspace-api/.theia/settings.json >
+        /projects/.theia/settings.json; sleep infinity
+    memoryLimit: 56Mi
+    type: dockerimage
+    image: busybox

devworkspace-api.code-workspace

@@ -20,12 +20,12 @@
             "signatureHelp": true,
             "rename": true,
             "workspaceSymbols": true,
-            "diagnostics": true, // for diagnostics as you type
+            "diagnostics": true // for diagnostics as you type
         },
         "go.autocompleteUnimportedPackages": true,
         "yaml.schemas": {
-            "./generated/schema.json": "*/*devfile.yaml",
-        },
-
+            "./schemas/devfile.json": "devfile-support/samples/*devfile.yaml",
+            "./schemas/devworkspace.json": "samples/*devworkspace.yaml"
+        }
     }
-}
+}

docker-run.sh

@@ -0,0 +1,86 @@
+#!/bin/bash
+#
+# Copyright (c) 2020 Red Hat, Inc.
+# This program and the accompanying materials are made
+# available under the terms of the Eclipse Public License 2.0
+# which is available at https://www.eclipse.org/legal/epl-2.0/
+#
+# SPDX-License-Identifier: EPL-2.0
+#
+# Contributors:
+#   Red Hat, Inc. - initial API and implementation
+
+# git ROOT directory used to mount filesystem
+GIT_ROOT_DIRECTORY=$(git rev-parse --show-toplevel)
+GO_MODULE=$(grep -e 'module ' ${GIT_ROOT_DIRECTORY}/go.mod | sed -e 's/module //')
+WORKDIR="/home/user/go/src/${GO_MODULE}"
+# Container image
+IMAGE_NAME="che-incubator/devworkspace-build-prerequisites"
+
+# Operator SDK
+OPERATOR_SDK_VERSION=v0.12.0
+
+init() {
+  BLUE='\033[1;34m'
+  GREEN='\033[0;32m'
+  RED='\033[0;31m'
+  NC='\033[0m'
+  BOLD='\033[1m'
+}
+
+check() {
+  if [ $# -eq 0 ]; then
+    printf "%bError: %bNo script provided. Command is $ docker-run.sh <script-to-run> [optional-arguments-of-script-to-run]\n" "${RED}" "${NC}"
+    exit 1
+  fi
+  echo "check $1"
+  if [ ! -f "$1" ] || [ ! -x "$1" ]; then
+    printf "%bError: %bscript %b provided does not exist. Command is $ docker-run.sh <script-to-run> [optional-arguments-of-script-to-run]\n" "${RED}" "${NC}" "${1}"
+    exit 1
+  fi
+}
+
+# Build image
+build() {
+  printf "%bBuilding image %b${IMAGE_NAME}${NC}..." "${BOLD}" "${BLUE}"
+  if docker build --build-arg OPERATOR_SDK_VERSION=${OPERATOR_SDK_VERSION} -t ${IMAGE_NAME} > docker-build-log 2>&1 -<<EOF
+FROM python:3-alpine
+ARG OPERATOR_SDK_VERSION
+ENV GOROOT /usr/lib/go
+
+RUN apk add --no-cache --update curl bash jq go \
+&& pip3 install yq \
+&& pip3 install jsonpatch
+
+RUN curl -JL https://github.com/operator-framework/operator-sdk/releases/download/${OPERATOR_SDK_VERSION}/operator-sdk-${OPERATOR_SDK_VERSION}-x86_64-linux-gnu -o /bin/operator-sdk && chmod a+x /bin/operator-sdk
+RUN mkdir -p /home/user/go && chmod -R a+w /home/user
+ENV HOME /home/user
+ENV GOPATH /home/user/go
+WORKDIR ${WORKDIR}
+EOF
+  then
+    printf "%b[OK]%b\n" "${GREEN}" "${NC}"
+    rm docker-build-log
+  else
+    printf "%bFailure%b\n" "${RED}" "${NC}"
+    cat docker-build-log
+    exit 1
+  fi
+}
+
+
+run() {
+  printf "%bRunning%b $*\n" "${BOLD}" "${NC}"
+  if docker run --user $(id -u):$(id -g) --rm -it -v "${GIT_ROOT_DIRECTORY}":"${WORKDIR}" --entrypoint=/bin/bash ${IMAGE_NAME} -- "$@"
+  then
+    printf "Script execution %b[OK]%b\n" "${GREEN}" "${NC}"
+  else
+    printf "%bFail to run the script%b\n" "${RED}" "${NC}"
+    exit 1
+  fi
+}
+
+init "$@"
+check "$@"
+build "$@"
+run "$@"

samples/example.devworkspace.yaml

@@ -0,0 +1,45 @@
+kind: "DevWorkspace"
+apiVersion: "workspaces.ecd.eclipse.org/v1alpha1"
+metadata:
+  "name": "myWorkspace"
+spec:
+  started: true
+  endpointsClass: 'openshift-auth'
+  template:
+    projects:
+      - name: "devworkspace-spec"
+        git:
+            location: "https://github.com/che-incubator/devworkspace-api"
+            branch: "master"
+    components:
+      - cheEditor:
+          registry:
+            id: eclipse/che-theia/latest
+      - chePlugin:
+          alias: "ownPlugin"
+          registry:
+            id: acme/newPlugin/latest
+            registryUrl: "https://acme.com/registry/"
+      - chePlugin:
+          alias: "myPlugin"
+          url: "https://github.com/johndoe/che-plugins/blob/master/cool-plugin/0.0.1/meta.yaml"
+      - container:
+          image: "busybox"
+          name: "mycontainer"
+          memoryLimit: "128M"
+          mountSources: true
+          endpoints:
+            - name: term-websockets
+              configuration:
+                protocol: tcp
+                scheme: ws
+                type: terminal
+              targetPort: 4000
+      - custom:
+          name: "myNewComponent"
+          componentClass: "NewComponentType"
+          embeddedResource:
+            field1: ""
+            field2: ""
+      - kubernetes:
+          url: "https://somewhere/production-environment.yaml"