Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Deploy CSI driver as a deployment and support multi master & leader e… #72

Closed
wants to merge 1 commit into from
Closed

[WIP] Deploy CSI driver as a deployment and support multi master & leader e… #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions cmd/syncer/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,46 @@ limitations under the License.
package main

import (
"context"
"flag"
"os"

"github.com/kubernetes-csi/csi-lib-utils/leaderelection"
"k8s.io/klog"
k8s "sigs.k8s.io/vsphere-csi-driver/pkg/kubernetes"

metadatasyncer "sigs.k8s.io/vsphere-csi-driver/pkg/syncer"
)

var (
enableLeaderElection = flag.Bool("leader-election", false, "Enable leader election.")
)

// main is ignored when this package is built as a go plug-in.
func main() {
klog.InitFlags(nil)
flag.Parse()
metadataSyncer := metadatasyncer.NewInformer()
if err := metadataSyncer.Init(); err != nil {
klog.Errorf("Error initializing Metadata Syncer")
os.Exit(1)
run := func(ctx context.Context) {
if err := metadataSyncer.Init(); err != nil {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is metadataSyncer.Init() a blocking call that runs the syncer forever?

klog.Errorf("Error initializing Metadata Syncer")
os.Exit(1)
}
}
Comment on lines +40 to +45
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The context that gets passed into your run func will be canceled when this process loses leadership. It looks like you don't pass that into metadataSyncer.Init so we will have no way to tear things down if we lose leadership.

In most core k8s components, we react to loss of leadership with os.Exit() so that the container gets restarted cleanly.


if !*enableLeaderElection {
run(context.TODO())
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In main() it is generally expected that you will use context.Background() since there is no higher level to provide one. context.TODO() is intended for places where a context should be passed in, but the plumbing hasn't been done yet.

} else {
k8sClient, err := k8s.NewClient()
if err != nil {
klog.Errorf("Creating Kubernetes client failed. Err: %v", err)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: you can use klog.Exitf() instead of Errorf followed by Exit

os.Exit(1)
}
lockName := "vsphere-syncer"
le := leaderelection.NewLeaderElection(k8sClient, lockName, run)

if err := le.Run(); err != nil {
klog.Fatalf("Error initializing leader election: %v", err)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Fatalf() prints a stack trace, and Exitf() does not. Exitf() may be more appropriate here.

}
}
}
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ require (
github.com/imdario/mergo v0.3.7 // indirect
github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
github.com/kr/pty v1.1.8 // indirect
github.com/kubernetes-csi/csi-lib-utils v0.6.1
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e // indirect
github.com/munnerz/goautoneg v0.0.0-20190414153302-2ae31c8b6b30 // indirect
github.com/onsi/ginkgo v1.10.1
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,8 @@ github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kubernetes-csi/csi-lib-utils v0.6.1 h1:+AZ58SRSRWh2vmMoWAAGcv7x6fIyBMpyCXAgIc9kT28=
github.com/kubernetes-csi/csi-lib-utils v0.6.1/go.mod h1:GVmlUmxZ+SUjVLXicRFjqWUUvWez0g0Y78zNV9t7KfQ=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a h1:TpvdAwDAt1K4ANVOfcihouRdvP+MgAfDWwBuct4l6ZY=
github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
Expand Down
14 changes: 9 additions & 5 deletions ....14/deploy/vsphere-csi-controller-ss.yaml → ...oy/vsphere-csi-controller-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
kind: StatefulSet
kind: Deployment
apiVersion: apps/v1
metadata:
name: vsphere-csi-controller
namespace: kube-system
spec:
serviceName: vsphere-csi-controller
replicas: 1
updateStrategy:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updateStrategy -> RollingUpdate will be useful later for upgrade. Can you keep that?

type: "RollingUpdate"
strategy:
type: RollingUpdate
selector:
matchLabels:
app: vsphere-csi-controller
Expand All @@ -32,6 +31,8 @@ spec:
- "--v=4"
- "--timeout=300s"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
- "--leader-election-type=leases"
env:
- name: ADDRESS
value: /csi/csi.sock
Expand Down Expand Up @@ -85,7 +86,8 @@ spec:
- name: vsphere-syncer
image: vmware/volume-metadata-syncer:v1.0.0
args:
- "--v=2"
- "--v=5"
- "--leader-election"
imagePullPolicy: "Always"
env:
- name: FULL_SYNC_INTERVAL_MINUTES
Expand All @@ -104,6 +106,8 @@ spec:
- "--csi-address=$(ADDRESS)"
- "--feature-gates=Topology=true"
- "--strict-topology"
- "--enable-leader-election"
- "--leader-election-type=leases"
env:
- name: ADDRESS
value: /csi/csi.sock
Expand Down
3 changes: 3 additions & 0 deletions manifests/1.14/rbac/vsphere-csi-controller-rbac.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ rules:
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
Expand Down