Skip to content

Commit 01f76cf

Browse files
🐛 (cherry-pick #3945) fix place where metrics service is scaffolded by moving from config/rbac to config/default (#3948)
(cherry-pick #3945) fix place where metrics service is scaffolded by moving from config/rbac to config/default (#3948) When we discontinued the usage of kube-rbac-proxy we placed the Metrics Service under config/rbac but it is not the best place to fit this resource. Furthermore, within those changes we are ensuring that the metrics service will only be applied if/when users enable the metrics. * Upgrade sample testdata in the v3x branch
1 parent 445eb2a commit 01f76cf

File tree

38 files changed

+114
-168
lines changed

38 files changed

+114
-168
lines changed

.github/workflows/test-sample-go.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,9 @@ jobs:
2424
run: |
2525
KUSTOMIZATION_FILE_PATH="testdata/project-v4/config/default/kustomization.yaml"
2626
sed -i '25s/^#//' $KUSTOMIZATION_FILE_PATH
27-
sed -i '39s/^#//' $KUSTOMIZATION_FILE_PATH
28-
sed -i '44s/^#//' $KUSTOMIZATION_FILE_PATH
29-
sed -i '48,144s/^#//' $KUSTOMIZATION_FILE_PATH
27+
sed -i '32s/^#//' $KUSTOMIZATION_FILE_PATH
28+
sed -i '47s/^#//' $KUSTOMIZATION_FILE_PATH
29+
sed -i '51,147s/^#//' $KUSTOMIZATION_FILE_PATH
3030
3131
- name: Test
3232
run: |

docs/book/src/component-config-tutorial/testdata/project/config/default/kustomization.yaml

+4-1
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,11 @@ resources:
2525
#- ../certmanager
2626
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
2727
#- ../prometheus
28+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
29+
#- metrics_service.yaml
2830

29-
patches:
31+
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
32+
#patches:
3033
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
3134
# More info: https://book.kubebuilder.io/reference/metrics
3235
# If you want to expose the metric endpoint of your controller-manager uncomment the following line.

docs/book/src/component-config-tutorial/testdata/project/config/rbac/kustomization.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ resources:
99
- role_binding.yaml
1010
- leader_election_role.yaml
1111
- leader_election_role_binding.yaml
12-
- metrics_service.yaml
1312
# For each CRD, "Editor" and "Viewer" roles are scaffolded by
1413
# default, aiding admins in cluster management. Those roles are
1514
# not used by the Project itself. You can comment the following lines

docs/book/src/cronjob-tutorial/testdata/project/config/default/kustomization.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,10 @@ resources:
2525
- ../certmanager
2626
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
2727
- ../prometheus
28+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
29+
#- metrics_service.yaml
2830

31+
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
2932
patches:
3033
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
3134
# More info: https://book.kubebuilder.io/reference/metrics

docs/book/src/cronjob-tutorial/testdata/project/config/rbac/kustomization.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ resources:
99
- role_binding.yaml
1010
- leader_election_role.yaml
1111
- leader_election_role_binding.yaml
12-
- metrics_service.yaml
1312
# For each CRD, "Editor" and "Viewer" roles are scaffolded by
1413
# default, aiding admins in cluster management. Those roles are
1514
# not used by the Project itself. You can comment the following lines

docs/book/src/getting-started/testdata/project/config/default/kustomization.yaml

+4-1
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,11 @@ resources:
2525
#- ../certmanager
2626
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
2727
#- ../prometheus
28+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
29+
#- metrics_service.yaml
2830

29-
patches:
31+
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
32+
#patches:
3033
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
3134
# More info: https://book.kubebuilder.io/reference/metrics
3235
# If you want to expose the metric endpoint of your controller-manager uncomment the following line.

docs/book/src/getting-started/testdata/project/config/rbac/kustomization.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ resources:
99
- role_binding.yaml
1010
- leader_election_role.yaml
1111
- leader_election_role_binding.yaml
12-
- metrics_service.yaml
1312
# For each CRD, "Editor" and "Viewer" roles are scaffolded by
1413
# default, aiding admins in cluster management. Those roles are
1514
# not used by the Project itself. You can comment the following lines

docs/book/src/reference/metrics.md

+7-2
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,12 @@ Further information can be found bellow in this document.
4545
First, you will need enable the Metrics by uncommenting the following line
4646
in the file `config/default/kustomization.yaml`, see:
4747

48-
```sh
48+
```yaml
49+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
50+
#- metrics_service.yaml
51+
```
52+
53+
```yaml
4954
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
5055
# More info: https://book.kubebuilder.io/reference/metrics
5156
# If you want to expose the metric endpoint of your controller-manager uncomment the following line.
@@ -79,7 +84,7 @@ Integrating `cert-manager` with your metrics service can secure the endpoint via
7984

8085
To modify your project setup to expose metrics using HTTPS with
8186
the help of cert-manager, you'll need to change the configuration of both
82-
the `Service` under `config/rbac/metrics_service.yaml` and
87+
the `Service` under `config/default/metrics_service.yaml` and
8388
the `ServiceMonitor` under `config/prometheus/monitor.yaml` to use a secure HTTPS port
8489
and ensure the necessary certificate is applied.
8590

pkg/plugins/common/kustomize/v2/scaffolds/init.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ func (s *initScaffolder) Scaffold() error {
6464

6565
templates := []machinery.Builder{
6666
&rbac.Kustomization{},
67-
&rbac.MetricsService{},
67+
&kdefault.MetricsService{},
6868
&rbac.RoleBinding{},
6969
// We need to create a Role because if the project
7070
// has not CRD define the controller-gen will not generate this file

pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/kdefault/kustomization.go

+4-1
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,11 @@ resources:
7171
#- ../certmanager
7272
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
7373
#- ../prometheus
74+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
75+
#- metrics_service.yaml
7476
75-
patches:
77+
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
78+
#patches:
7679
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
7780
# More info: https://book.kubebuilder.io/reference/metrics
7881
# If you want to expose the metric endpoint of your controller-manager uncomment the following line.

pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/rbac/metrics_service.go pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/kdefault/metrics_service.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
1414
limitations under the License.
1515
*/
1616

17-
package rbac
17+
package kdefault
1818

1919
import (
2020
"path/filepath"
@@ -33,7 +33,7 @@ type MetricsService struct {
3333
// SetTemplateDefaults implements file.Template
3434
func (f *MetricsService) SetTemplateDefaults() error {
3535
if f.Path == "" {
36-
f.Path = filepath.Join("config", "rbac", "metrics_service.yaml")
36+
f.Path = filepath.Join("config", "default", "metrics_service.yaml")
3737
}
3838

3939
f.TemplateBody = metricsServiceTemplate

pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/rbac/kustomization.go

-1
Original file line numberDiff line numberDiff line change
@@ -53,5 +53,4 @@ const kustomizeRBACTemplate = `resources:
5353
- role_binding.yaml
5454
- leader_election_role.yaml
5555
- leader_election_role_binding.yaml
56-
- metrics_service.yaml
5756
`

pkg/plugins/common/kustomize/v2/scaffolds/webhook.go

+9
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,15 @@ func (s *webhookScaffolder) Scaffold() error {
9898
}
9999
}
100100

101+
err = pluginutil.UncommentCode(kustomizeFilePath, "#patches:", `#`)
102+
if err != nil {
103+
hasWebHookUncommented, err := pluginutil.HasFragment(kustomizeFilePath, "patches:")
104+
if !hasWebHookUncommented || err != nil {
105+
log.Errorf("Unable to find the line '#patches:' to uncomment in the file "+
106+
"%s.", kustomizeFilePath)
107+
}
108+
}
109+
101110
err = pluginutil.UncommentCode(kustomizeFilePath, "#- path: manager_webhook_patch.yaml", `#`)
102111
if err != nil {
103112
hasWebHookUncommented, err := pluginutil.HasFragment(kustomizeFilePath, "- path: manager_webhook_patch.yaml")

test/e2e/v4/generate_test.go

+9
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,9 @@ func GenerateV4(kbc *utils.TestContext) {
6363
ExpectWithOffset(1, pluginutil.UncommentCode(
6464
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
6565
"#- path: webhookcainjection_patch.yaml", "#")).To(Succeed())
66+
ExpectWithOffset(1, pluginutil.UncommentCode(
67+
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
68+
"#- metrics_service.yaml", "#")).To(Succeed())
6669
ExpectWithOffset(1, pluginutil.UncommentCode(
6770
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
6871
metricsTarget, "#")).To(Succeed())
@@ -120,9 +123,15 @@ func GenerateV4WithoutWebhooks(kbc *utils.TestContext) {
120123
initingTheProject(kbc)
121124
creatingAPI(kbc)
122125

126+
ExpectWithOffset(1, pluginutil.UncommentCode(
127+
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
128+
"#patches:", "#")).To(Succeed())
123129
ExpectWithOffset(1, pluginutil.UncommentCode(
124130
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
125131
"#- ../prometheus", "#")).To(Succeed())
132+
ExpectWithOffset(1, pluginutil.UncommentCode(
133+
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
134+
"#- metrics_service.yaml", "#")).To(Succeed())
126135
ExpectWithOffset(1, pluginutil.UncommentCode(
127136
filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
128137
metricsTarget, "#")).To(Succeed())

test/e2e/v4/plugin_cluster_test.go

+52-57
Original file line numberDiff line numberDiff line change
@@ -278,66 +278,47 @@ func Run(kbc *utils.TestContext, hasWebhook, isToUseInstaller, hasMetrics bool)
278278

279279
// curlMetrics curl's the /metrics endpoint, returning all logs once a 200 status is returned.
280280
func curlMetrics(kbc *utils.TestContext, hasMetrics bool) string {
281-
By("validating that the controller-manager service is available")
282-
_, err := kbc.Kubectl.Get(
283-
true,
284-
"service", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
285-
)
286-
ExpectWithOffset(2, err).NotTo(HaveOccurred(), "Controller-manager service should exist")
287-
288-
By("validating that the controller-manager deployment is ready")
289-
verifyDeploymentReady := func() error {
290-
output, err := kbc.Kubectl.Get(
281+
var metricsOutput string
282+
if hasMetrics {
283+
By("validating that the controller-manager service is available")
284+
_, err := kbc.Kubectl.Get(
291285
true,
292-
"deployment", fmt.Sprintf("e2e-%s-controller-manager", kbc.TestSuffix),
293-
"-o", "jsonpath={.status.readyReplicas}",
286+
"service", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
294287
)
295-
if err != nil {
296-
return err
297-
}
298-
readyReplicas, _ := strconv.Atoi(output)
299-
if readyReplicas < 1 {
300-
return fmt.Errorf("expected at least 1 ready replica, got %d", readyReplicas)
301-
}
302-
return nil
303-
}
304-
EventuallyWithOffset(2, verifyDeploymentReady, 240*time.Second, time.Second).Should(Succeed(),
305-
"Deployment is not ready")
288+
ExpectWithOffset(2, err).NotTo(HaveOccurred(), "Controller-manager service should exist")
306289

307-
By("ensuring the service endpoint is ready")
308-
eventuallyCheckServiceEndpoint := func() error {
309-
output, err := kbc.Kubectl.Get(
310-
true,
311-
"endpoints", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
312-
"-o", "jsonpath={.subsets[*].addresses[*].ip}",
313-
)
314-
if err != nil {
315-
return err
290+
By("ensuring the service endpoint is ready")
291+
eventuallyCheckServiceEndpoint := func() error {
292+
output, err := kbc.Kubectl.Get(
293+
true,
294+
"endpoints", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
295+
"-o", "jsonpath={.subsets[*].addresses[*].ip}",
296+
)
297+
if err != nil {
298+
return err
299+
}
300+
if output == "" {
301+
return fmt.Errorf("no endpoints found")
302+
}
303+
return nil
316304
}
317-
if output == "" {
318-
return fmt.Errorf("no endpoints found")
305+
EventuallyWithOffset(2, eventuallyCheckServiceEndpoint, 2*time.Minute, time.Second).Should(Succeed(),
306+
"Service endpoint should be ready")
307+
308+
By("creating a curl pod to access the metrics endpoint")
309+
// nolint:lll
310+
cmdOpts := []string{
311+
"run", "curl",
312+
"--restart=Never",
313+
"--namespace", kbc.Kubectl.Namespace,
314+
"--image=curlimages/curl:7.78.0",
315+
"--",
316+
"/bin/sh", "-c", fmt.Sprintf("curl -v -k http://e2e-%s-controller-manager-metrics-service.%s.svc.cluster.local:8080/metrics",
317+
kbc.TestSuffix, kbc.Kubectl.Namespace),
319318
}
320-
return nil
321-
}
322-
EventuallyWithOffset(2, eventuallyCheckServiceEndpoint, 2*time.Minute, time.Second).Should(Succeed(),
323-
"Service endpoint should be ready")
324-
325-
By("creating a curl pod to access the metrics endpoint")
326-
// nolint:lll
327-
cmdOpts := []string{
328-
"run", "curl",
329-
"--restart=Never",
330-
"--namespace", kbc.Kubectl.Namespace,
331-
"--image=curlimages/curl:7.78.0",
332-
"--",
333-
"/bin/sh", "-c", fmt.Sprintf("curl -v -k http://e2e-%s-controller-manager-metrics-service.%s.svc.cluster.local:8080/metrics",
334-
kbc.TestSuffix, kbc.Kubectl.Namespace),
335-
}
336-
_, err = kbc.Kubectl.CommandInNamespace(cmdOpts...)
337-
ExpectWithOffset(2, err).NotTo(HaveOccurred())
319+
_, err = kbc.Kubectl.CommandInNamespace(cmdOpts...)
320+
ExpectWithOffset(2, err).NotTo(HaveOccurred())
338321

339-
var metricsOutput string
340-
if hasMetrics {
341322
By("validating that the curl pod is running as expected")
342323
verifyCurlUp := func() error {
343324
status, err := kbc.Kubectl.Get(
@@ -359,6 +340,20 @@ func curlMetrics(kbc *utils.TestContext, hasMetrics bool) string {
359340
}
360341
EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("< HTTP/1.1 200 OK"))
361342
} else {
343+
By("creating a curl pod to access the metrics endpoint")
344+
// nolint:lll
345+
cmdOpts := []string{
346+
"run", "curl",
347+
"--restart=Never",
348+
"--namespace", kbc.Kubectl.Namespace,
349+
"--image=curlimages/curl:7.78.0",
350+
"--",
351+
"/bin/sh", "-c", fmt.Sprintf("curl -v -k http://e2e-%s-controller-manager-metrics-service.%s.svc.cluster.local:8080/metrics",
352+
kbc.TestSuffix, kbc.Kubectl.Namespace),
353+
}
354+
_, err := kbc.Kubectl.CommandInNamespace(cmdOpts...)
355+
ExpectWithOffset(2, err).NotTo(HaveOccurred())
356+
362357
By("validating that the curl pod fail as expected")
363358
verifyCurlUp := func() error {
364359
status, err := kbc.Kubectl.Get(
@@ -375,14 +370,14 @@ func curlMetrics(kbc *utils.TestContext, hasMetrics bool) string {
375370

376371
By("validating that the metrics endpoint is not working as expected")
377372
getCurlLogs := func() string {
378-
metricsOutput, err = kbc.Kubectl.Logs("curl")
373+
metricsOutput, err := kbc.Kubectl.Logs("curl")
379374
ExpectWithOffset(3, err).NotTo(HaveOccurred())
380375
return metricsOutput
381376
}
382-
EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("Connection refused"))
377+
EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("Could not resolve host"))
383378
}
384379
By("cleaning up the curl pod")
385-
_, err = kbc.Kubectl.Delete(true, "pods/curl")
380+
_, err := kbc.Kubectl.Delete(true, "pods/curl")
386381
ExpectWithOffset(3, err).NotTo(HaveOccurred())
387382

388383
return metricsOutput

testdata/project-v4-multigroup-with-deploy-image/config/default/kustomization.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,10 @@ resources:
2525
#- ../certmanager
2626
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
2727
#- ../prometheus
28+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
29+
#- metrics_service.yaml
2830

31+
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
2932
patches:
3033
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
3134
# More info: https://book.kubebuilder.io/reference/metrics

testdata/project-v4-multigroup-with-deploy-image/config/rbac/kustomization.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ resources:
99
- role_binding.yaml
1010
- leader_election_role.yaml
1111
- leader_election_role_binding.yaml
12-
- metrics_service.yaml
1312
# For each CRD, "Editor" and "Viewer" roles are scaffolded by
1413
# default, aiding admins in cluster management. Those roles are
1514
# not used by the Project itself. You can comment the following lines

testdata/project-v4-multigroup-with-deploy-image/dist/install.yaml

-18
Original file line numberDiff line numberDiff line change
@@ -1474,24 +1474,6 @@ subjects:
14741474
---
14751475
apiVersion: v1
14761476
kind: Service
1477-
metadata:
1478-
labels:
1479-
app.kubernetes.io/managed-by: kustomize
1480-
app.kubernetes.io/name: project-v4-multigroup-with-deploy-image
1481-
control-plane: controller-manager
1482-
name: project-v4-multigroup-with-deploy-image-controller-manager-metrics-service
1483-
namespace: project-v4-multigroup-with-deploy-image-system
1484-
spec:
1485-
ports:
1486-
- name: http
1487-
port: 8080
1488-
protocol: TCP
1489-
targetPort: 8080
1490-
selector:
1491-
control-plane: controller-manager
1492-
---
1493-
apiVersion: v1
1494-
kind: Service
14951477
metadata:
14961478
labels:
14971479
app.kubernetes.io/managed-by: kustomize

testdata/project-v4-multigroup/config/default/kustomization.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,10 @@ resources:
2525
#- ../certmanager
2626
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
2727
#- ../prometheus
28+
# [METRICS] To enable the controller manager metrics service, uncomment the following line.
29+
#- metrics_service.yaml
2830

31+
# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
2932
patches:
3033
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
3134
# More info: https://book.kubebuilder.io/reference/metrics

testdata/project-v4-multigroup/config/rbac/kustomization.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ resources:
99
- role_binding.yaml
1010
- leader_election_role.yaml
1111
- leader_election_role_binding.yaml
12-
- metrics_service.yaml
1312
# For each CRD, "Editor" and "Viewer" roles are scaffolded by
1413
# default, aiding admins in cluster management. Those roles are
1514
# not used by the Project itself. You can comment the following lines

0 commit comments

Comments
 (0)