Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/migrate to kubebuilder v3 #27

Merged

Conversation

itamar-marom
Copy link
Collaborator

@itamar-marom itamar-marom commented Mar 12, 2023

Fixes #24

Description

Currently, the project is not following Kubebuilder's native structure. This customization can be a real pain for new developers who are familiar with Kubebuilder, and also make new API creation difficult.

This PR updates Kubebuilder to v3 including back to Kubebulder's native structure.

Following the migration guide
This PR is something that has to be done before this project gains more users and becomes bigger. This project is a mutation of the Operator SDK and the Kubebuilder project.
In this MR I'm migrating the project to Kubebuilder while upgrading Kubebuilder to v3 in order to eliminate the pain of migrating at another time.

Something that is notable here is that autoscaling/v2alpha2 is now deprecated in new clusters (For example, update your kind cluster and try) - Therefore, the project is now using autoscaling/v2 HPAs.

The deployment also had a big change since Kubebuilder deploys more resources that benefit us and help us to manage the controller much more easily.

I will note the Kubebuilder is a Kubernetes community project and it is becoming the de-facto for building controllers - we must follow the guidelines.

Tested on:

  • Kubernetes version -> v1.25
  • Kubebuilder version -> 3.9.1
  • Go version -> 1.20.2

This PR has:

  • has been tested on a real K8S cluster to ensure creating a brand new Druid cluster works.
  • been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious to an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

Key changed/added files in this PR
  • deploy structure that changes to config and deploys more resources by Kubebuilder best practices.
  • The Makefile file changed to support the changes.
  • Some code in main.go and the Reconcile function went through some little changes.
  • docs updated
  • helm chart has changed to mirror the config directory

Copy link
Contributor

@AdheipSingh AdheipSingh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will also need changes in the documentation, helm charts.

main.go Outdated Show resolved Hide resolved
main.go Outdated Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
@AdheipSingh
Copy link
Contributor

AdheipSingh commented Mar 12, 2023

Thanks for taking this @itamar-marom . Also to test your PR what k8s version are you testing on ? would be good to note and add it the compatibility matrix.
Moving forward we can use kubebuilder to update versions/gomod. We can document those too.

@itamar-marom
Copy link
Collaborator Author

Hi, @AdheipSingh Thanks for already looking at the PR. It is still not ready (that's why the WIP) and I don't want you to waste your time :), I'll ping you when it's ready. I'll reply to your comments, thank you!

@itamar-marom
Copy link
Collaborator Author

I'm currently dealing with this issue: kubernetes-sigs/kubebuilder#2026 - this is why it takes more time.

@AdheipSingh
Copy link
Contributor

I'm currently dealing with this issue: kubernetes-sigs/kubebuilder#2026 - this is why it takes more time.

Let me know if you hit any other blocker, happy to brain storm together.

@itamar-marom
Copy link
Collaborator Author

I fixed it. Wrote about it here
I don't know why it makes a difference.
Anyway, another thing is that tests are not working for me since my cluster support autoscaling/v2 and not v2beta2. I updated the code to supportt v2 but let's talk about it, and if you think this is a bad idea I'll revert back.

@itamar-marom
Copy link
Collaborator Author

itamar-marom commented Mar 12, 2023

Doing some more testing and will update you tomorrow. Also I need to change the docs (installation).

@itamar-marom itamar-marom changed the title WIP: Feature/migrate to kubebuilder v3 Feature/migrate to kubebuilder v3 Mar 15, 2023
main.go Show resolved Hide resolved
go.sum Outdated Show resolved Hide resolved
chart/templates/deployment.yaml Show resolved Hide resolved
chart/templates/deployment.yaml Show resolved Hide resolved
Comment on lines -1 to -7
{{- if .Values.rbac.create }}
{{- $env := .Values.env }}
{{- if and ($env.WATCH_NAMESPACE) (ne $env.WATCH_NAMESPACE "default") }}
# Split WATCH_NAMESPACE by commas and loop on them
{{- $watchedNamespaces := (split "," $env.WATCH_NAMESPACE) -}}
{{- range $watchedNamespaces }}
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itamar-marom there is a conditional check, which is needed to scope the operator. Operator can watch multiple namespace, single namespace or all namespaces. On this either a role or clusterrole is created.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I understand. Fixed it but change it a little bit so there will be not extra role druid-operator-manager-role-sc - In the case of WATCH_NAMESPACES, the default ClusterRole will be created with only storageclass permissions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itamar-marom i will give this another look, also is this role change backward compatible ? do we need to change anything on existing operator roles.

Copy link
Collaborator Author

@itamar-marom itamar-marom Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, the role permissions are the same, also naming wise if you look at the chart, and the role.yaml under the deploy directory.


This is a bit tricky since it really depends on how the user deployed the operator. In the case of helm, it will be responsible to change the rbac to the correct one. In the case of regular deployment, the user should notice that the names are equal between the old and the new deployment. By default, the old deployment name is druid-operator and the new one is druid-operator-controller-manager. Also, the namespace is different by default, Everything else is the same.

Looking at this again, I will make the migration documentation clearer.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I detailed more in the migration docs. Eventually, it is important to check that the naming of the objects is the same in both deployments.

docs/kubebuilder_v3_migration.md Outdated Show resolved Hide resolved
@AdheipSingh
Copy link
Contributor

cc @cintoSunny @harinirajendran if you can help in reviewing this too.

Makefile Show resolved Hide resolved
Makefile Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@AdheipSingh
Copy link
Contributor

LGTM with the PR, ill do some final review and tests and wait if other want to comment or review.

@AdheipSingh AdheipSingh merged commit d0c9e60 into datainfrahq:master Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Question: Kubebuilder structure
3 participants