Add distribution of K8s hooks

[treydock]

This will fix #994 .

I left things so there are 2 options show. Large inline string that is dumped into tmp file and applied or YAML generated using envsubts and dumped into tmp files and then applied. A site could choose to add the pod-security-policy or job-pod-reaper executions to their pre-hook.sh if they wanted those opt-in items.

I added what is an attempt to validate the K8s YAML using real Kubernetes cluster from kind cluster action.

[treydock]

Another approach I thought about was insteads of sites adding scripts to their pre-hook.sh , we make them opt in via the configuration they already have to supply in hook.env. There are numerous site specific things OSC deploys in env files that this file will replace so everything for hooks is in one place.

[ericfranz]

What role does hook.env play?

FWIW I know right now that the Lua code sends the hook arguments to STDIN of the webhook, but I would still like to explore having it do what Jeff first attempted, which was actually set the environment prior to executing the script. If we switch to that, does that have any impact on this PR?

Also I notice most of the files are under hooks/k8s-bootstrap/ but some are just under hooks/ - would it be better if all K8s related hook scripts and data where namespaced under the k8s-bootstrap directory? I imagine in the future we would add here other example hooks for things unrelated to k8s?

[treydock]

The hook.env is just a way to configure the hook scripts. Right now we deploy some of those values to OSC specific locations and others would be a way to make things more dynamic and less specific to OSC like namespace prefix: https://github.com/OSC/osc-ood-config/blob/master/hooks/set-k8s-creds.sh#L12.

If the nginx_stage could set all these environment variables then we would not need hook.env but they would all have to be set and some of them are considered secret and should only be accessible by root so they should not make their way into user's PUN environment, like the CLIENT_SECRET which is OIDC client secret for ondemand instance.

I organized the scripts to live under hooks/k8s-bootstrap and put the envsubst YAML under hooks/k8s-bootstrap/yaml. The ondemand.yaml is not envsubst, it's something that someone like root has to apply to K8s cluster if they want a service account to do the bootstrapping rather than bootstrapping with cluster admin credentials.

[treydock]

I can imagine one possible way to handle hook env is a new key/value pair in nginx_stage like:

hook_env:
  client_id: foo
  client_secret: secret

Which then sets like HOOK_CLIENT_ID=foo and HOOK_CLIENT_SECRET=secret. Just one idea but any way to get those environment variables from nginx_stage would work for this.

[johrstrom]

Just to hone the discussion a bit, here's an outline of the 3 current possibilities.

1. YAMLs are inlined into the shell scripts and a pre hook would like like

/opt/ood/hooks/k8s/apply_one_thing.sh
/opt/ood/hooks/k8s/apply_some_other_thing.sh

2. The yamls are are on their own and we supply a helpler to template them (envsubst) and apply the

/opt/ood/hooks/k8s/apply.sh /opt/ood/hooks/config/k8s/user_namespace.yaml
/opt/ood/hooks/k8s/apply.sh /opt/ood/hooks/config/k8s/pod_security_policy.yaml

3. There are special environment variables in the /etc/ood/config/hooks.env that would say if OOD_K8S_APPLY_PSP is set, it would automatially template and apply the pod security policy (the one we ship with).

For my taste, I'd vote for number 2. It allows for a lot of composability where a site could mix and match site specific things with what we ship. It's also more explicit as a pose to number 3. The admin can directly see what's being applied and how. K8s is bound to generate a lot of tickets and discussion so being explicit about what's being set and how may help out some user confusion.

[ericfranz]

What is the status of this PR? Is it urgent to merge this today?

[johrstrom]

I think it's still a bit in flux, I don't think it's urgent enough to need today.

[treydock]

I think it's needed sometime before we announce 2.0 as officially released but with enough time before that for OSC to test the changes and deploy them. What this PR does is replicated in osc-ood-configs repo so for OSC there is no urgency other than to ensure we can migrate to use what's in this PR and test it.