Skip to content

Commit

Permalink
test_harness w/ fixes for the unit tests to pass (kubeflow#50)
Browse files Browse the repository at this point in the history
* test_harness w/ fixes for the unit tests to pass

* update for README.md

* update to README.md

* added test harness files to generate unit tests
  • Loading branch information
Kam Kasravi authored and k8s-ci-robot committed May 9, 2019
1 parent ad3874e commit f0a80c4
Show file tree
Hide file tree
Showing 23 changed files with 913 additions and 67 deletions.
205 changes: 155 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,70 +2,111 @@
This repo is a [bespoke configuration](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#bespoke-configuration) of kustomize targets used by kubeflow. These targets are traversed by kubeflow's CLI `kfctl`. Each target is compatible with the kustomize CLI and can be processed indendently by kubectl or the kustomize command.

## Organization
Various subdirectories within the repo contain a kustomize target (base or overlay subdirectory). Currently overlays hold platform resources where platform is one of "gcp|minikube". These targets are processed by kfctl during generate and apply phases and is detailed in [Kfctl Processing](#kfctl-processing).
Various subdirectories within the repo contain a kustomize target (base or overlay subdirectory). Overlays are used for a variety of purposes such as platform resources. Both base and overlay targets are processed by kfctl during generate and apply phases and is detailed in [Kfctl Processing](#kfctl-processing).


### Kustomize targets (🎯)
```
📦 application ⎹→🗳 base(🎯)
📦 argo ⎹→🗳 base(🎯)
📦 common ⇲
⎹→🗳 common/ambassador/base(🎯)
⎹→🗳 common/centraldashboard/base(🎯)
⎹→🗳 common/spartakus/base(🎯)
📦 gcp ⇲
⎹→🗳 gcp/cert-manager/overlays/gcp(🎯)
⎹→🗳 gcp/cloud-endpoints/overlays/gcp(🎯)
⎹→🗳 gcp/gcp-credentials-admission-webhook/overlays/gcp(🎯)
⎹→🗳 gcp/gpu-driver/overlays/gcp(🎯)
⎹→🗳 gcp/iap-ingress/overlays/gcp(🎯)
📦 jupyter ⇲
⎹→🗳 jupyter/jupyter-web-app/base(🎯)
⎹→🗳 jupyter/jupyter/base(🎯)
⎹→🗳 jupyter/jupyter/overlays/minikube(🎯)
⎹→🗳 jupyter/notebook-controller/base(🎯)
📦 katib ⎹→🗳 base(🎯)
📦 kubebench ⎹→🗳 base(🎯)
📦 metacontroller ⎹→🗳 base(🎯)
📦 modeldb ⎹→🗳 base(🎯)
📦 mutating-webhook ⎹→🗳 base(🎯)
⎹→🗳 mutating-webhook/overlays/add-label(🎯)
📦 pipeline ⇲
⎹→🗳 pipeline/api-service/base(🎯)
⎹→🗳 pipeline/minio/base(🎯)
⎹→🗳 pipeline/mysql/base(🎯)
⎹→🗳 pipeline/persistent-agent/base(🎯)
⎹→🗳 pipeline/pipelines-runner/base(🎯)
⎹→🗳 pipeline/pipelines-ui/base(🎯)
⎹→🗳 pipeline/pipelines-viewer/base(🎯)
⎹→🗳 pipeline/scheduledworkflow/base(🎯)
📦 profiles ⎹→🗳 base(🎯)
⎹→🗳 profiles/overlays/debug(🎯)
⎹→🗳 profiles/overlays/devices(🎯)
📦 pytorch-job ⇲
⎹→🗳 pytorch-job/pytorch-operator/base(🎯)
📦 tensorboard ⎹→🗳 base(🎯)
📦 tf-training ⇲
⎹→🗳 tf-training/tf-job-operator/base(🎯)
.
├── application
│   └─🎯base
├── argo
│   └─🎯base
├── common
│   ├── ambassador
│   │   └─🎯base
│   ├── centraldashboard
│   │   └─🎯base
│   └── spartakus
│   └─🎯base
├─🎯gcp
│   ├── cert-manager
│   │   └── overlays
│   │   └─🎯gcp
│   ├── cloud-endpoints
│   │   └── overlays
│   │   └─🎯gcp
│   ├─🎯gcp-credentials-admission-webhook
│   │   └── overlays
│   │   └─🎯gcp
│   ├── gpu-driver
│   │   └── overlays
│   │   └─🎯gcp
│   └── iap-ingress
│   └── overlays
│   └─🎯gcp
├── jupyter
│   ├── jupyter
│   │   ├─🎯base
│   │   └── overlays
│   │   └── minikube
│   ├── jupyter-web-app
│   │   └─🎯base
│   └── notebook-controller
│   └─🎯base
├── katib
│   └─🎯base
├── kubebench
│   └─🎯base
├── metacontroller
│   └─🎯base
├── modeldb
│   └─🎯base
├── mutating-webhook
│   ├─🎯base
│   └── overlays
│   └── add-label
├── pipeline
│   ├── api-service
│   │   └─🎯base
│   ├── minio
│   │   └─🎯base
│   ├── mysql
│   │   └─🎯base
│   ├── persistent-agent
│   │   └─🎯base
│   ├── pipelines-runner
│   │   └─🎯base
│   ├── pipelines-ui
│   │   └─🎯base
│   ├── pipelines-viewer
│   │   └─🎯base
│   └── scheduledworkflow
│   └─🎯base
├── profiles
│   └─🎯base
├── pytorch-job
│   └── pytorch-operator
│   └─🎯base
├── tensorboard
│   └─🎯base
└── tf-training
└── tf-job-operator
├─🎯base
└── overlays
├── 🎯cluster
├── 🎯cluster-gangscheduled
├── 🎯namespaced
└── 🎯namespaced-gangscheduled
```

## Kfctl Processing
Kfctl will traverse these directories to find and build kustomize targets based on the configuration file `app.yaml`. App.yaml is derived from a file in the kubeflow [config](https://github.com/kubeflow/kubeflow/tree/master/bootstrap/config) directory. Each target processed by kfctl will result in an output yaml file. The output file is generated by calling kustomize's API. The kustomize package manager in kfctl will read app.yaml and apply the packages, components and componentParams to kustomize in the following way:
Kfctl traverses directories under manifests to find and build kustomize targets based on the configuration file `app.yaml`. The contents of app.yaml is the result of running kustomize on the base and specific overlays in the kubeflow [config](https://github.com/kubeflow/kubeflow/tree/master/bootstrap/config) directory. The overlays reflect what options are chosen when calling `kfctl init...`. The kustomize package manager in kfctl will then read app.yaml and apply the packages, components and componentParams to kustomize in the following way:

- **📦 packages**
- **packages**
- are always top-level directories under the manifests repo
- **🗳 components**
- **components**
- are also directories but may be a subdirectory in a package.
- components may also be a top-level directory if there is a base or overlay in that directory in which case the component name is equal to the package name.
- otherwise a component is a sub-directory
- a component may also be a package if there is a base or overlay in the top level directory.
- otherwise a component is a sub-directory under the package directory.
- in all cases a component's name in app.yaml must match the directory name.
- components are output as `<component>.yaml` under the kustomize subdirectory during `kfctl generate...`.
- in order to output a component, a synthetic kustomization.yaml is created under manifests that sets common parameters and labels that looks like
- in order to output a component, a kustomization.yaml is created above the base or overlay directory and inherits common parameters, namespace and labels of the base or overlay. Additionally it adds the namespace and an application label.
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- <component>/{base|overlay/<platform>}
- <component>/{base|overlay/<overlay>}
commonLabels:
app.kubernetes.io/name: <appName>
namespace:
Expand All @@ -74,10 +115,74 @@ namespace:
- **component parameters**
- are applied to a component's params.env file. There must be an entry whose key matches the component parameter. The params.env file is used to generate a ConfigMap. Entries in params.env are resolved as kustomize vars or referenced in a deployment or statefulset env section in which case no var definition is needed.

Multiple overlays -

Generating yaml output for any target can be done in the following way:
Kfctl has the capability to combine more than one overlay during `kfctl generate ...`. An example is shown below where the profiles target in [manifests](https://github.com/kubeflow/manifests/tree/master/profiles) can include either debug changes in the Deployment or Device information in the Namespace (the devices overlay is not fully integrated with the Profile-controller at this point in time and is intended as an example) or **both**.

### Install kustomize
```
profiles
├── base
│   └── kustomization.yaml
└── overlays
├── debug
│   └── kustomization.yaml
└── devices
└── kustomization.yaml
```

#### What are Multiple Overlays?

Normally kustomize provides the ability to overlay a 'base' set of resources with changes that are merged into the base from resources that are located under an overlays subdirectory. For example
if the kustomize [target](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#target) is named foo there will be a foo/base and possibly one or more overlays such as foo/overlays/bar. A kustomization.yaml file is found in both foo/base and foo/overlays/bar. Running `kustomize build` in foo/base will generate resources as defined in kustomization.yaml. Running `kustomize build` in foo/overlays/bar will generate resources - some of which will overlay the resources in foo/base.

Kustomize doesn't provide for an easy way to combine more than one overlay for example foo/overlays/bar, foo/overlays/baz. However this is an open feature request in kustomize [issues](https://github.com/kubernetes-sigs/kustomize/issues/759). The ability to combine more than one overlay is key to handling components like tf-job-operator which has several types of overlays that can 'mix-in' whether a TFJob is submitted to a namespace or cluster-wide and whether the TFJob uses gang-scheduling.

#### Merging multiple overlays

Since each overlay includes '../../base' as its base set of resources - combining several overlays where each includes '../../base' will cause `kustomize build` to abort, complaining that it recursed on base. The approach is to create a kustomization.yaml at the target level that includes base and the contents of each overlay's kustomization file. This requires some path corrections and some awareness of the behavior of configMapGenerator, secretMapGenerator and how they are copied from each overlay. This kustomization.yaml can be constructed manually, but is integrated within kfctl via the app.yaml file. Using tf-job-operator as an example, if its componentParams has the following
```
componentParams:
tf-job-operator:
- name: overlay
value: cluster
- name: overlay
- value: gangscheduled
```

Then the result will be to combine these overlays eg 'mixin' an overlays in the kustomization.yaml file.

#### Merging multiple overlays to generate app.yaml

Normally when `kfctl init ...` is called it will download the kubeflow repo under `<deployment>/.cache` and read one of the config files under `.cache/kubeflow/<version>/bootstrap/config`. These config files define packages, components and component parameters (among other things). Each config file is a compatible k8 resource of kind *KfDef*. The various config files are:
- kfctl_default.yaml
- kfctl_basic_auth.yaml
- kfctl_iap.yaml

Both kfctl_basic_auth.yaml and kfctl_iap.yaml contain the contents of kfctl_default.yaml plus some additional changes specific to using basic_auth when the cluster is created and platform specific resources if the platform is **gcp**. This PR corrects this redundancy by using kustomize to combine a **gcp** overlay and/or a **basic_auth** overlay. Additionally, due to pipeline refactoring, the kustomize package manager has split the bundled pipeline component in ksonnet to a set of individual pipeline targets. This results in the following:

```
config
├── base
│   └── kustomization.yaml
└── overlays
├── basic_auth
│   └── kustomization.yaml
├── gcp
│   └── kustomization.yaml
├── ksonnet
│   └── kustomization.yaml
└── kustomize
└── kustomization.yaml
```

Based on the cli args to `kfctl init...`, the correct overlays will be merged to produce an app.yaml.
The original files have been left as is until UI integration can be completed in a separate PR

### Using kustomize

Generating yaml output for any target can be done using kustomize in the following way:

#### Install kustomize

`go get -u github.com/kubernetes-sigs/kustomize`

Expand Down
1 change: 1 addition & 0 deletions common/centraldashboard/base/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ resources:
- service-account.yaml
- service.yaml
- virtual-service.yaml
namespace: kubeflow
commonLabels:
kustomize.component: centraldashboard
images:
Expand Down
125 changes: 125 additions & 0 deletions hack/gen-test-target.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
#!/usr/bin/env bash
#
# gen-test-target will generate a golang testcase using the
# kustomize test-harness that is used from kerbernetes-sigs/kustomize/pkg/target/kusttestharness_test.go
# The unittest compares the collection of resource files with what kustomize build would produce (actual vs expected)
#
source hack/utils.sh

kebab-case-2-PascalCase()
{
local a=$1 b='' array
IFS='-' read -r -a array <<< "$a"
for element in "${array[@]}"
do
part="${element^}"
b+=$part
done
echo $b
}

gen-target-start()
{
local dir=$(get-target $1) target fname
fname=/manifests${dir##*/manifests}
target=$(kebab-case-2-PascalCase $(get-target-name $1))

echo 'package kustomize_test'
echo ''
echo 'import ('
echo ' "testing"'
echo ')'
echo ''
echo 'func write'$target'(th *KustTestHarness) {'
}

gen-target-end()
{
echo '}'
}

gen-target()
{
local directory=$1
gen-target-start $1
for i in $(echo $(cat $directory/kustomization.yaml | grep '^- .*yaml$'|sed 's/^- //') $(cat $directory/kustomization.yaml | grep ' path: ' | sed 's/^.*: \(.*\)$/\1/') params.env kustomization.yaml | sed 's/ /\n/g' | sort | uniq); do
file=$i
if [[ -f $directory/$file ]]; then
case $file in
"kustomization.yaml")
gen-target-kustomization $file $directory
;;
*.yaml)
gen-target-resource $file $directory
;;
params.env)
gen-target-resource $file $directory
;;
*)
;;
esac
fi
done
gen-target-end
}

gen-target-kustomization()
{
local file=$1 dir=$2 fname kname
fname=/manifests${dir##*/manifests}
kname=${fname%/kustomization.yaml}
echo ' th.writeK("'$kname'", `'
cat $dir/$file
echo '`)'

}

gen-target-resource()
{
local file=$1 dir=$2 fname
fname=/manifests${dir##*/manifests}/$file

echo ' th.writeF("'$fname'", `'
cat $dir/$file
echo '`)'
}

gen-expected-start()
{
echo ' th.assertActualEqualsExpected(m, `'
}

gen-expected-end()
{
echo '`)'
}

gen-expected()
{
gen-expected-start
cd $1
kustomize build
cd - >/dev/null
gen-expected-end
}

gen-test-case()
{
local base=$(get-target-name $1) dir=$(get-target $1) target fname
fname=/manifests${dir##*/manifests}/$(get-target-dirname $1)
target=$(kebab-case-2-PascalCase $base)

gen-target $1
echo ''
echo 'func Test'$target'(t *testing.T) {'
echo ' th := NewKustTestHarness(t, "'$fname'")'
echo ' write'$target'(th)'
echo ' m, err := th.makeKustTarget().MakeCustomizedResMap()'
echo ' if err != nil {'
echo ' t.Fatalf("Err: %v", err)'
echo ' }'
gen-expected $1
echo '}'
}

gen-test-case $1
20 changes: 20 additions & 0 deletions hack/gen-test-targets.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
#
# gen-test-targets will generate units tests under tests/ for all directories that
# have a kustomization.yaml. This script first finds all directories and then calls
# gen-test-target to generate each golang unit test.
# The script is based on kusttestharness_test.go from kubernetes-sigs/kustomize/pkg/target
#
if [[ $(basename $PWD) != "manifests" ]]; then
echo "must be at manifests root directory to run $0"
exit 1
fi

source hack/utils.sh
rm -f $(ls tests/*_test.go | grep -v kusttestharness_test.go)
for i in $(find * -type d -exec sh -c '(ls -p "{}"|grep />/dev/null)||echo "{}"' \; | egrep -v 'docs|tests|hack'); do
absdir=$(pwd)/$i
testname=$(get-target-name $absdir)_test.go
echo generating $testname from $absdir
./hack/gen-test-target.sh $absdir > tests/$testname
done
Loading

0 comments on commit f0a80c4

Please sign in to comment.