Implement testing logic to compare the Shoot-spec generated by provisioner with the shoot-spec generated by KIM

[tobiscr]

Reason
We want to ensure that our migration from Provisioner to Kyma-Infrastructure-Manager will go without any issues. To create a shoot cluster in Gardener, we have to fill gardener's shoot spec.

How
Ideas on how to achieve that:

1. Comparing shoot spec of created clusters

1. Prepare two KEB deployments. First should be still configured to use the old provisioner, and the second should use KIM.
2. Send the very same instance creation requests to KEB.
3. Compare the shoot specs of created clusters.

2. Comparing created shoot spec, before sending them to Gardener

1. Create a feature flag in the provisioner for dumping shoot spec before sending it to gardener
2. Save those specs
3. Compare shoot specs generated in KIM with those generated in provisioner

Acceptance criteria

• Each KEB's plan should be covered
• compare (manually) the Shoot-spec generated by provisioner with the shoot-spec generated by KIM
• the above comparison step should be automated (descoped for now)
• It is possible for devlopers to compare the generated shoot-spec of KIM and the one generated by the provisioner

[Disper]

How shoots specs can be compared using yq:

1. set KUBECONFIG env variable to the gardener's service account's kubeconfig
2. Download shoot specs you want to compare kubectl get shoot -n GARDENER_PROJECT_NAME -ojson | jq '.items[] | select(.metadata.name == "SHOOT_NAME")' > SHOOT_NAME.yml
3. Use yq and diff to compare them diff <(yq -P 'sort_keys(..)' -o=props SHOOT_NAME1.yml) <(yq -P 'sort_keys(..)' -o=props SHOOT_NAME2.yml)

This prints output in following manner:

< metadata.name = ag-test
---
> metadata.name = vogfkd3r2s
< spec.kubernetes.enableStaticTokenKubeconfig = true
---
> spec.kubernetes.enableStaticTokenKubeconfig = false

As it might come handy during manual testing, I'm not sure if we can use it in our golang test suite.

[koala7659]

PR with feature flag to dump shoot spec created by Provisioner kyma-project/control-plane#3433

[Disper]

Blocked by #215

[akgalwas]

Work plan

• Implement storing shoot files
  • https://github.com/kyma-project/infrastructure-manager/pull/268/files#diff-c444f711e9191b53952edb65bfd8c644419fc7695c62611dc0fb304b4fb197d6
  • Provisioner - Adding a feature flag to dump generated Gardener shoot spec into logs control-plane#3433
• Use persistent volume for storing files
  • KIM: https://github.tools.sap/kyma/management-plane-charts/pull/1677
  • Provisioner: Added Persistent Volume Claim reference control-plane#3454
  • verification
• Enable shoot file storing functionality for DEV environment
• Implement storing input In review
  • feat: Saving Graphql mutation object to Persistent Volume control-plane#3460
  • Runtime CR for KIM feat: Saving Runtime CR to Persistent Volume #300
  • Ignore few additional fields
• Renames persist-shoot to dump-shoot-spec #307
• Implement shoot comparison tool In review
  • Dockerfile for shoot comparison test app #306
  • Implement comparison logic
    • Files: feat: First version of shoot comparison app #287
    • Folders Compare directories in the shoot_comparator tool #310 @akgalwas
• Automate comparison In review
  • Add job that executes shoot comparison #321 In progress @akgalwas
  • Create scripts that can be used any time we will need to compare shoots (descoped)
• Fixes for making shoot comparison easier #322 In progress
  • KIM should store shoot spec created by converter, not the one obtained from Gardener to make comparison easier
  • Matchers in the shoot comparison app should present results in more granular, and readable way (now if anything under shoot.spec differs the whole shoot.spec objects are displayed)