fix successThreshold, validate server protocol, deprecate scheme

[weisdd]

Description

• Fixed success threshold:
  • Previously, it would override PeriodSeconds which would led to unexpected results. E.g. if someone sets PeriodSeconds to 2, kube-apiserver will decline the spec, because successThreshold for livenessProbes cannot be anything but 1 (docs).
    In logs, it would be something like this:
    grafana-operator-544bbbbf5b-bw6c6 grafana-operator 2022-07-19T08:27:11.077Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"Grafana","namespace":"monitoring","name":"grafana-operator-grafana","uid":"e42ba143-2787-41cd-a2f7-6d1f2371235f","apiVersion":"integreatly.org/v1alpha1","resourceVersion":"390657"}, "reason": "ProcessingError", "message": "Deployment.apps \"grafana-deployment\" is invalid: spec.template.spec.containers[0].livenessProbe.successThreshold: Invalid value: 104: must be 1"};
• Refactored probes:
  • All default values are set in consts (previously, some values were set in the callable function, others - through the caller function);
  • Simplified code for overriding values through CRD;
  • Added tests;
  • scheme is now ignored as it's always tied to the config.server.protocol section of grafana config, thus should not have been configurable in the first place;
• Enforced server protocol through CRD (operator only supports http and https).

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Only in the sense that the scheme field of readinessProbeSpec and livenessProbeSpec is ignored, the actual value depends on config.server.protocol.

Checklist

• This change requires a documentation update
  • Updated API docs
• I have added tests that prove my fix is effective or that my feature works
• I have added a test case that will be used to verify my changes
• Verified independently on a cluster by reviewer

Verification steps

1. Deploy grafana with default spec:

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
spec:
  client:
    preferService: true
  config: {}

Probes in the resulting deployment should be set to the same values as in the code.

2. Deploy grafana with custom spec for probes:

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
spec:
  client:
    preferService: true
  livenessProbeSpec:
    failureThreshold: 2
    initialDelaySeconds: 3
    periodSeconds: 4
    successThreshold: 1
    timeoutSeconds: 5
    scheme: unknown-scheme
  readinessProbeSpec:
    failureThreshold: 6
    initialDelaySeconds: 7
    periodSeconds: 8
    successThreshold: 9
    timeoutSeconds: 10
    scheme: unknown-scheme
  config: {}

All probes settings in the resulting deployment should match the values from the spec except for scheme.
scheme is ignored in favor of config.server.protocol, and since it's not present in the spec, scheme: HTTP is set by default.

3. Deploy grafana with a random server protocol in its spec:

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
spec:
  client:
    preferService: true
  config:
    server:
      protocol: unknown-protocol

$ k apply -f 3.yaml 
The Grafana "example-grafana" is invalid: spec.config.server.protocol: Unsupported value: "unknown-protocol": supported values: "http", "https"

[hubeadmin]

Verified on OCP 4.10.x using verification steps, works as expected, thanks @weisdd 👍 !

[weisdd]

@HubertStefanski @pb82 As discussed over the call, I've brought the scheme field back to avoid any incompatibility in v4. The updated field description has a deprecation notice.

[NissesSenap]

@weisdd I have patched all the go dependencies #793 and when it get's merged your PR should pass.
But due to the update of the v1core version Handler have been deprecated and instead ProbeHandler is used.
See the PR for more information.

[weisdd]

@NissesSenap thanks for the update. Once your PR is merged, I can update mine :)

[weisdd]

Updated tests, PR is up-to-date.