feat: Creating Gardener Cluster CR as part of cluster provisioning #293

koala7659 · 2024-07-11T16:57:41Z

Description

Changes proposed in this pull request:

Creating Gardener Cluster CR as next stage of Provisioning
Validation of labels on provided Runtime CR:
Required labels:

- kyma-project.io/instance-id
- kyma-project.io/runtime-id
- kyma-project.io/region
- operator.kyma-project.io/kyma-name
- kyma-project.io/broker-plan-id
- kyma-project.io/broker-plan-name
- kyma-project.io/global-account-id
- kyma-project.io/subaccount-id

State machine optimisation

Testing:

Unit tests for State machine are not fully implemented
Integration tests are in progress

Related issue(s)
#282

Disper · 2024-07-17T11:04:31Z

internal/controller/runtime/fsm/runtime_fsm_create_kubeconfig.go

+				imv1.LabelKymaShootName:      shoot.Name,
+
+				// hardcoded values
+				imv1.LabelKymaManagedBy: "lifecycle-manager",


Suggested change

imv1.LabelKymaManagedBy: "lifecycle-manager",

imv1.LabelKymaManagedBy: "infrastructure-manager",

KEB sets that currently to lifecycle-manager which is wrong. KIM is managing this resource. It should be set to infrastructure-manager

Disper · 2024-07-18T12:26:22Z

internal/controller/runtime/fsm/runtime_fsm_create_kubeconfig_test.go

+			},
+		),
+		Entry(
+			"should return sFnUpdateStatus and no error when CR has been created without finalizer - Add finalizer",


entry says that it should return sFnUpdateStatus while matcher is MatchNextFnState: BeNil(),

Disper · 2024-07-18T12:29:47Z

api/v1/runtime_types.go

+	LabelKymaRuntimeID       = "kyma-project.io/runtime-id"
+	LabelKymaShootName       = "kyma-project.io/shootName"
+	LabelKymaRegion          = "kyma-project.io/region"
+	LabelKymaName            = "kyma-project.io/kyma-name"


KEB had label key operator.kyma-project.io/kyma-name (with operator.). Was it intended to be different in KIM?

Link: https://github.com/kyma-project/kyma-environment-broker/blob/main/internal/process/steps/lifecycle_manager.go#L30

Also you can see https://github.com/kyma-project/infrastructure-manager/blob/main/docs/adr/001-provisioning.md

Disper · 2024-07-18T12:41:54Z

Verification update
After some time, GardenerCluster CR transited to Error state, described as:

Status:
  Conditions:
    Last Transition Time:  2024-07-18T12:28:14Z
    Message:               Failed to create secret. Error: secrets "kubeconfig-059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8" already exists
    Reason:                ConditionReasonFailedToCreateSecret
    Status:                False
    Type:                  KubeconfigManagement
  State:                   Error
Events:                    <none>

While the Runtime CR is described as:

Status:
  Conditions:
    Last Transition Time:  2024-07-18T12:09:51Z
    Message:               Runtime processing completed successfully
    Reason:                ConfigurationCompleted
    Status:                True
    Type:                  Provisioned
    Last Transition Time:  2024-07-18T12:10:19Z
    Message:               Gardener Cluster CR is ready.
    Reason:                GardenerClusterCRReady
    Status:                True
    Type:                  KubeconfigReady
  State:                   Ready
Events:
  Type    Reason                    Age   From                Message
  ----    ------                    ----  ----                -------
  Normal  Initialised               38m   runtime-controller  Runtime initialized: kcp-system/kim-md2
  Normal  Pending                   38m   runtime-controller  Shoot is pending: kcp-system/kim-md2
  Normal  Pending                   37m   runtime-controller  Shoot creation in progress: kcp-system/kim-md2
  Normal  ShootCreationCompleted    30m   runtime-controller  Shoot creation completed: kcp-system/kim-md2
  Normal  GardenerClusterCRCreated  29m   runtime-controller  Gardener Cluster CR created, waiting for readiness: kcp-system/kim-md2
  Normal  GardenerClusterCRReady    29m   runtime-controller  Gardener Cluster CR is ready.: kcp-system/kim-md2
  Normal  ConfigurationCompleted    29m   runtime-controller  Runtime processing completed successfully: kcp-system/kim-md2

I would expect the following instead:

KIM will not try to create Gardener Cluster CR for the second time resulting in an error in Gardener Cluster CR
(I'm not sure about this) Even if Gardener Cluster CR will moved from Ready to Error, the status change should be reflected in Runtime CR condition as well (currently it says that it GardneerCluster CR is Ready)

Attaching logs from the last 2 reconciliations (the one where GardenerCluster CR was created, and the next one that resulted in error)

2024-07-18T14:10:14+02:00       INFO    Reconciling Runtime     {"Name": "kim-md2", "Namespace": "kcp-system"}
2024-07-18T14:10:14+02:00       INFO    reqID 36        Take snapshot state
2024-07-18T14:10:14+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnTakeSnapshot to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnInitialize       {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:14+02:00       INFO    reqID 36        Gardener shoot exists, processing
2024-07-18T14:10:14+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnInitialize to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnSelectShootProcessing      {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:14+02:00       INFO    reqID 36        Select shoot processing state
2024-07-18T14:10:14+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnSelectShootProcessing to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnWaitForShootCreation    {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:14+02:00       INFO    reqID 36        Waiting for shoot creation state
2024-07-18T14:10:14+02:00       INFO    reqID 36        Shoot kim-md2 successfully created
2024-07-18T14:10:14+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnWaitForShootCreation to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnCreateKubeconfig {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 36        Create Gardener Cluster CR state
2024-07-18T14:10:19+02:00       INFO    reqID 36        GardenerCluster CR is ready     {"Name": "059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8"}
2024-07-18T14:10:19+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnCreateKubeconfig to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnUpdateStatus.func1   {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnUpdateStatus.func1 to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnEmmitEventfunc.func1       {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 36        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnEmmitEventfunc.func1 to  {"result": {"Requeue":true,"RequeueAfter":0}, "err": null, "mFnIsNill": true}
2024-07-18T14:10:19+02:00       INFO    reqID 36        reconciliation done     {"error": null, "result": {"Requeue":true,"RequeueAfter":0}}
2024-07-18T14:10:19+02:00       DEBUG   events  Gardener Cluster CR is ready.: kcp-system/kim-md2       {"type": "Normal", "object": {"kind":"Runtime","namespace":"kcp-system","name":"kim-md2","uid":"fab9417d-7065-4612-bd05-ce75fce4ebf6","apiVersion":"infrastructuremanager.kyma-project.io/v1","resourceVersion":"1628"}, "reason": "GardenerClusterCRReady"}
2024-07-18T14:10:19+02:00       INFO    kcp-system/kim-md2
2024-07-18T14:10:19+02:00       INFO    Reconciling Runtime     {"Name": "kim-md2", "Namespace": "kcp-system"}
2024-07-18T14:10:19+02:00       INFO    reqID 37        Take snapshot state
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnTakeSnapshot to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnInitialize       {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        Gardener shoot exists, processing
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnInitialize to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnSelectShootProcessing      {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        Select shoot processing state
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnSelectShootProcessing to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnWaitForShootCreation    {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        Waiting for shoot creation state
2024-07-18T14:10:19+02:00       INFO    reqID 37        Shoot kim-md2 successfully created
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnWaitForShootCreation to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnCreateKubeconfig {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        Create Gardener Cluster CR state
2024-07-18T14:10:19+02:00       INFO    reqID 37        GardenerCluster CR is ready     {"Name": "059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8"}
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnCreateKubeconfig to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnProcessShoot {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        Process cluster state - the last one
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnProcessShoot to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnUpdateStatus.func1       {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnUpdateStatus.func1 to github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnEmmitEventfunc.func1       {"result": null, "err": null, "mFnIsNill": false}
2024-07-18T14:10:19+02:00       INFO    reqID 37        switching state from github.com/kyma-project/infrastructure-manager/internal/controller/runtime/fsm.sFnEmmitEventfunc.func1 to  {"result": null, "err": null, "mFnIsNill": true}
2024-07-18T14:10:19+02:00       INFO    reqID 37        reconciliation done     {"error": null, "result": null}
2024-07-18T14:10:19+02:00       DEBUG   events  Runtime processing completed successfully: kcp-system/kim-md2   {"type": "Normal", "object": {"kind":"Runtime","namespace":"kcp-system","name":"kim-md2","uid":"fab9417d-7065-4612-bd05-ce75fce4ebf6","apiVersion":"infrastructuremanager.kyma-project.io/v1","resourceVersion":"1630"}, "reason": "ConfigurationCompleted"}
2024-07-18T14:16:44+02:00       INFO    Starting reconciliation.        {"GardenerCluster": "059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8", "Namespace": "kcp-system"}
2024-07-18T14:16:44+02:00       INFO    rotation params {"GardenerCluster": "059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8", "Namespace": "kcp-system", "lastSync": "0001-01-01 00:00:00", "requeueAfter": "6m38.999999999s", "gardenerRequestTimeout": "1m0s"}
2024-07-18T14:16:45+02:00       INFO    result without requeue
2024-07-18T14:16:45+02:00       ERROR   Reconciler error        {"controller": "gardenercluster", "controllerGroup": "infrastructuremanager.kyma-project.io", "controllerKind": "GardenerCluster", "GardenerCluster": {"name":"059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8","namespace":"kcp-system"}, "namespace": "kcp-system", "name": "059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8", "reconcileID": "c80b4051-b00d-4dba-8cf2-21051782a66f", "error": "secrets \"kubeconfig-059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8\" already exists"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/i316752/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/i316752/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/i316752/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222
2024-07-18T14:16:45+02:00       INFO    Starting reconciliation.        {"GardenerCluster": "059dbc39-fd2b-4186-b0e5-8a1bc8ede5b8", "Namespace": "kcp-system"}

Disper · 2024-07-19T07:09:59Z

Probably that's the fix https://github.com/koala7659/infrastructure-manager/pull/2/files. However as @koala7659 is on vacation, I will merge this PR and work on a follow-up.

Creating Gardener Cluster CR

3d9c5a1

koala7659 requested a review from a team as a code owner July 11, 2024 16:57

kyma-bot added cla: yes Indicates the PR's author has signed the CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 11, 2024

koala7659 changed the title ~~Creating Gardener Cluster CR~~ WiP Creating Gardener Cluster CR Jul 11, 2024

kyma-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 11, 2024

koala7659 marked this pull request as draft July 11, 2024 16:58

koala7659 added 7 commits July 11, 2024 19:04

restore original code

8227960

continue work on Gardener cluster creation

52d94d5

fix compilation error

e75430b

fixing bugs discovered during testing

39717ab

remove unnecessary comment

d8e1cb5

Merge remote-tracking branch 'origin/main' into create-gardener-cr

8a0d444

fix linter error for formatting error string

658dcef

koala7659 changed the title ~~WiP Creating Gardener Cluster CR~~ Creating Gardener Cluster CR Jul 17, 2024

koala7659 changed the title ~~Creating Gardener Cluster CR~~ Creating Gardener Cluster CR as part of cluster provisioning Jul 17, 2024

Merge remote-tracking branch 'origin/main' into create-gardener-cr

109f4a7

Disper self-assigned this Jul 17, 2024

Disper reviewed Jul 17, 2024

View reviewed changes

unit tests implementation for creating gardener cluster CR

711dee4

koala7659 changed the title ~~Creating Gardener Cluster CR as part of cluster provisioning~~ feat: Creating Gardener Cluster CR as part of cluster provisioning Jul 17, 2024

koala7659 added 3 commits July 18, 2024 12:26

Adjusting integration tests to verify creation of GardenerCluster CR

dfb2290

Linter fixes

36eafa4

Typo fix and enable sequence verification

5436f54

koala7659 marked this pull request as ready for review July 18, 2024 10:34

kyma-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 18, 2024

Disper reviewed Jul 18, 2024

View reviewed changes

Change name of the label kyma name to operator.kyma-project.io/kyma-name

3d66037

Disper approved these changes Jul 19, 2024

View reviewed changes

kyma-bot added the lgtm Looks good to me! label Jul 19, 2024

Disper enabled auto-merge July 19, 2024 07:10

kyma-bot merged commit 1358985 into kyma-project:main Jul 19, 2024
7 checks passed

This was referenced Jul 19, 2024

KIM must manage GardenerCluster resource once cluster is provisioned and deleted #282

Closed

switches to shoot-name label instead of shootName #301

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Creating Gardener Cluster CR as part of cluster provisioning #293

feat: Creating Gardener Cluster CR as part of cluster provisioning #293

koala7659 commented Jul 11, 2024 •

edited

Loading

Disper Jul 17, 2024

koala7659 Jul 18, 2024

Disper Jul 18, 2024

Disper Jul 18, 2024 •

edited

Loading

koala7659 Jul 18, 2024

koala7659 Jul 18, 2024

Disper commented Jul 18, 2024 •

edited

Loading

Disper commented Jul 19, 2024

	imv1.LabelKymaManagedBy: "lifecycle-manager",
	imv1.LabelKymaManagedBy: "infrastructure-manager",

feat: Creating Gardener Cluster CR as part of cluster provisioning #293

feat: Creating Gardener Cluster CR as part of cluster provisioning #293

Conversation

koala7659 commented Jul 11, 2024 • edited Loading

Disper Jul 17, 2024

Choose a reason for hiding this comment

koala7659 Jul 18, 2024

Choose a reason for hiding this comment

Disper Jul 18, 2024

Choose a reason for hiding this comment

Disper Jul 18, 2024 • edited Loading

Choose a reason for hiding this comment

koala7659 Jul 18, 2024

Choose a reason for hiding this comment

koala7659 Jul 18, 2024

Choose a reason for hiding this comment

Disper commented Jul 18, 2024 • edited Loading

Disper commented Jul 19, 2024

koala7659 commented Jul 11, 2024 •

edited

Loading

Disper Jul 18, 2024 •

edited

Loading

Disper commented Jul 18, 2024 •

edited

Loading