Add initial terraform manifests for monitoring

[rikatz]

Add initial manifest for gitops monitoring.

Some points of attention for this PR:

• This is not repeatable. Although I think adding too much modules for terraform can turn things more complicated, maybe we can define how and what we monitor in URLs. Assuming we always use SSL/TLS we could create a module and have this simpler in a future, like:

module "thockin_test1_monitoring" {
  source = "../../../modules/monitoring"
  project_name       = "kubernetes-public"
  url       = "thockin-test1.k8s.io"
  cert_expiration_threshold = 15
  // Channel needs to be manually created in GCP interface with the following display name
  channel_notification = "Kubernetes.io Cert Alert"
}

• We should probably want to define the structure for this. I'm proposing inside infra/monitoring just because got no better idea :D
• We can add more parameters, and more notification channel/options (like, we can add a list of emails that can turn into Email Notification Channel on GCP)
• I'm using all the GCP locations, but getting only metrics from usa-iowa as valid to cert_expiration. This can be changed later as well, depending on what we want to do (like, properly monitoring and alerting in case of some downtime/increased latency)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rikatz
To complete the pull request process, please assign spiffxp after the PR has been reviewed.
You can assign the PR to them by writing /assign @spiffxp in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ameukam]

We still use the 0.13 version

[ameukam]

We can use the existing channel : #k8s-infra-alerts. Can you mention the type ?

[rikatz]

@ameukam thanks!

I've changed the channel searcher to use the channel name (label) and the type slack, and did changed the terraform to >= 0.13.0 :)

[rikatz]

@ameukam anything else on this PR? I wanted to check if monitoring is working :)

[ameukam]

@rikatz the terraform states are now in different buckets : #1952
Feel free to add a new one.

[ameukam]

Suggested change

	version = "~> 3.63.0"
	version = "~> 3.66.0"

[ameukam]

Suggested change

	required_version = ">= 0.13.0"
	required_version = ">= 0.14.0"

See: #2019
Sorry for make you revert it. :-D

[sftim]

This will need a rebase / update I think now, should be:

Suggested change

	required_version = ">= 0.13.0"
	required_version = "~> 1.0.0"

[ameukam]

@rikatz I dropped some comments.

[spiffxp]

I have a meta-level nit which is that right now I think @ameukam has preferred to keep all of our terraform stuff in infra/gcp/clusters

I feel like we need to chat a bit about what we want organization of files to look like, but until then I think having them all homed under one directory (even if its name is misleading) makes more sense than allowing sprawl

So move to something like infra/gcp/clusters/projects/kubernetes-public/monitoring

[spiffxp]

I'm ok with merge and iterate if we immediately follow up with that move

[ameukam]

@rikatz What do you think about closing this PR now the domains are gone : #1838 ?

[rikatz]

Let's do this :)
/close

[k8s-ci-robot]

@rikatz: Closed this PR.

In response to this:

Let's do this :)
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.