Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ENV variables to specify default node selectors #372

Merged
merged 1 commit into from
Jun 16, 2018
Merged

Add ENV variables to specify default node selectors #372

merged 1 commit into from
Jun 16, 2018

Conversation

kmova
Copy link
Contributor

@kmova kmova commented Jun 16, 2018

Signed-off-by: kmova kiran.mova@openebs.io

What this PR does / why we need it:
This PR allows the administrators to specify the nodes on which the OpenEBS Replica and Controller Pods have to be scheduled using Kubernetes "NodeSelector" option. While Taints/Tolerations also can be used for this purpose, "NodeSelectors" are much simpler to setup using the node labels. And an added advantage is that - this can be done without having to worry about pushing taints to non-storage nodes.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:
The implementation follows the traditional approach of specifying an ENV variable similar to how taints and tolerations are specified. This code is required only till the CAS Templates feature is available.

@kmova
Copy link
Contributor Author

kmova commented Jun 16, 2018

This PR has been tested using a custom m-apiserver image (openebs/m-apiserver:0.6-ns-dev) built with the changes specified in this PR.

      containers:
      - name: maya-apiserver
        imagePullPolicy: Always
        image: openebs/m-apiserver:0.6.0-ns-dev
        ports:
        - containerPort: 5656
        env:
        - name: OPENEBS_IO_JIVA_CONTROLLER_IMAGE
          value: "openebs/jiva:0.6.0-RC1"
        - name: OPENEBS_IO_JIVA_REPLICA_IMAGE
          value: "openebs/jiva:0.6.0-RC1"
        - name: OPENEBS_IO_VOLUME_MONITOR_IMAGE
          value: "openebs/m-exporter:0.6.0-RC1"
        - name: OPENEBS_IO_JIVA_REPLICA_COUNT
          value: "3"
        # DEFAULT_CONTROLLER_NODE_SELECTOR allows your to specify the nodes
        # on which openebs controller have to be scheduled. To use this feature, 
        # the nodes should already be labeled with the key=value. For example:
        # `kubectl label nodes <node-name> nodetype=storage
        # Note: It is recommended that node selector for controller be 
        # same as that of the stateful apps.
        # This is supported for maya api server version 0.6 onwards
        - name: DEFAULT_CONTROLLER_NODE_SELECTOR
          value: "nodetype=storage"
        # DEFAULT_REPLICA_NODE_SELECTOR allows your to specify the nodes
        # on which openebs replicas have to be scheduled. To use this feature, 
        # the nodes should already be labeled with the key=value. For example:
        # `kubectl label nodes <node-name> nodetype=storage
        # Note: It is recommended that node selector for replica will specify
        # nodes that have disks/ssds attached to them. 
        # This is supported for maya api server version 0.6 onwards
        - name: DEFAULT_REPLICA_NODE_SELECTOR
          value: "nodetype=storage"

Tested this by creating a 5 node cluster in the following way:
(a) Labelled only 2 nodes with nodetype=storage. And ran the mongo-statefulset.yml. This resulted in controller and replica pods scheduled only on two nodes. Since only 2 nodes where labelled, third replica was not scheduled, even though nodes were are available.

kiran_mova@kmova-dev:~/go/src/github.com/openebs/maya$ kubectl get pods
NAME                                                             READY     STATUS    RESTARTS   AGE
maya-apiserver-754d7d4dcf-xnlhs                                  1/1       Running   0          21m
mongo-0                                                          2/2       Running   0          20m
mongo-1                                                          2/2       Running   0          20m
mongo-2                                                          2/2       Running   0          19m
openebs-provisioner-84976786d5-fkzgt                             1/1       Running   0          21m
openebs-snapshot-operator-b7fccfcd4-cvswg                        2/2       Running   0          21m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-ctrl-5cf8944cf9-kq6d4   2/2       Running   0          20m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-64df966774-8r5s4    1/1       Running   0          20m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-64df966774-hst68    1/1       Running   0          20m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-64df966774-jqgz5    0/1       Pending   0          20m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-ctrl-bd95c478f-6g4vp    2/2       Running   0          20m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-rep-bc898f4f4-kjdt2     1/1       Running   0          20m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-rep-bc898f4f4-lgsm6     0/1       Pending   0          20m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-rep-bc898f4f4-rdm7t     1/1       Running   0          20m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-ctrl-57c65c759-p6hz8    2/2       Running   0          18m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-rep-6b44b8fd4b-796pt    1/1       Running   0          18m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-rep-6b44b8fd4b-n4v5t    1/1       Running   0          18m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-rep-6b44b8fd4b-wwqmd    0/1       Pending   0          18m

(b) Labelled one more node with nodetype=storage. The pending replica's now got scheduled onthe newly labelled node.

kiran_mova@kmova-dev:~/go/src/github.com/openebs/maya$ kubectl get pods
NAME                                                             READY     STATUS    RESTARTS   AGE
maya-apiserver-754d7d4dcf-xnlhs                                  1/1       Running   0          25m
mongo-0                                                          2/2       Running   0          24m
mongo-1                                                          2/2       Running   0          24m
mongo-2                                                          2/2       Running   0          23m
openebs-provisioner-84976786d5-fkzgt                             1/1       Running   0          25m
openebs-snapshot-operator-b7fccfcd4-cvswg                        2/2       Running   0          25m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-ctrl-5cf8944cf9-kq6d4   2/2       Running   0          24m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-64df966774-8r5s4    1/1       Running   0          24m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-64df966774-hst68    1/1       Running   0          24m
pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-64df966774-jqgz5    1/1       Running   0          24m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-ctrl-bd95c478f-6g4vp    2/2       Running   0          24m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-rep-bc898f4f4-kjdt2     1/1       Running   0          24m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-rep-bc898f4f4-lgsm6     1/1       Running   0          24m
pvc-93c9dc27-714e-11e8-8df4-42010a800fe1-rep-bc898f4f4-rdm7t     1/1       Running   0          24m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-ctrl-57c65c759-p6hz8    2/2       Running   0          23m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-rep-6b44b8fd4b-796pt    1/1       Running   0          23m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-rep-6b44b8fd4b-n4v5t    1/1       Running   0          23m
pvc-b992d43c-714e-11e8-8df4-42010a800fe1-rep-6b44b8fd4b-wwqmd    1/1       Running   0          23m

@AmitKumarDas
Copy link

@kmova The UT code needs to mock the new interfaces that are created...
Rest of the code looks good.

Signed-off-by: kmova <kiran.mova@openebs.io>
@codecov
Copy link

codecov bot commented Jun 16, 2018

Codecov Report

Merging #372 into master will increase coverage by 0.08%.
The diff coverage is 44.82%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #372      +/-   ##
==========================================
+ Coverage   28.86%   28.95%   +0.08%     
==========================================
  Files          65       65              
  Lines        5428     5457      +29     
==========================================
+ Hits         1567     1580      +13     
- Misses       3694     3706      +12     
- Partials      167      171       +4
Impacted Files Coverage Δ
orchprovider/k8s/v1/k8s.go 32.26% <44.82%> (+0.46%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 146b521...fb7d010. Read the comment docs.

@prateekpandey14
Copy link
Contributor

Looks good, can you please paste the generated deployment yaml spec of controller and replica.

@kmova
Copy link
Contributor Author

kmova commented Jun 16, 2018

Controller Deployment of a Running Volume:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    openebs.io/volume-monitor: "true"
    openebs.io/volume-type: jiva
  creationTimestamp: 2018-06-16T10:17:29Z
  generation: 1
  labels:
    openebs/controller: jiva-controller
    openebs/volume-provisioner: jiva
    pvc: ""
    vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
  name: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-ctrl
  namespace: default
  resourceVersion: "34930"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-ctrl
  uid: 797f97f9-714e-11e8-8df4-42010a800fe1
spec:
  replicas: 1
  selector:
    matchLabels:
      monitoring: volume_exporter_prometheus
      openebs/controller: jiva-controller
      pvc: ""
      vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        monitoring: volume_exporter_prometheus
        openebs/controller: jiva-controller
        pvc: ""
        vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
    spec:
      containers:
      - args:
        - controller
        - --frontend
        - gotgt
        - --clusterIP
        - 10.19.252.220
        - pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
        command:
        - launch
        image: openebs/jiva:0.6.0-RC1
        imagePullPolicy: IfNotPresent
        name: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-ctrl-con
        ports:
        - containerPort: 3260
          protocol: TCP
        - containerPort: 9501
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - args:
        - -c=http://127.0.0.1:9501
        command:
        - maya-exporter
        image: openebs/m-exporter:0.6.0-RC1
        imagePullPolicy: IfNotPresent
        name: maya-volume-exporter
        ports:
        - containerPort: 9500
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        nodetype: storage
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
        key: node.alpha.kubernetes.io/notReady
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.alpha.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 0
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2018-06-16T10:17:29Z
    lastUpdateTime: 2018-06-16T10:17:29Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

@kmova
Copy link
Contributor Author

kmova commented Jun 16, 2018

Replica Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    openebs.io/storage-pool: default
  creationTimestamp: 2018-06-16T10:17:29Z
  generation: 1
  labels:
    openebs/replica: jiva-replica
    openebs/volume-provisioner: jiva
    pvc: ""
    vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
  name: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep
  namespace: default
  resourceVersion: "37205"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep
  uid: 798a6e11-714e-11e8-8df4-42010a800fe1
spec:
  replicas: 3
  selector:
    matchLabels:
      openebs/replica: jiva-replica
      pvc: ""
      vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        openebs/replica: jiva-replica
        pvc: ""
        vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                openebs/replica: jiva-replica
                vsm: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - replica
        - --frontendIP
        - 10.19.252.220
        - --size
        - 5G
        - /openebs
        command:
        - launch
        image: openebs/jiva:0.6.0-RC1
        imagePullPolicy: IfNotPresent
        name: pvc-796dbc3b-714e-11e8-8df4-42010a800fe1-rep-con
        ports:
        - containerPort: 9502
          protocol: TCP
        - containerPort: 9503
          protocol: TCP
        - containerPort: 9504
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /openebs
          name: openebs
      dnsPolicy: ClusterFirst
      nodeSelector:
        nodetype: storage
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
        key: node.alpha.kubernetes.io/notReady
        operator: Exists
      - effect: NoExecute
        key: node.alpha.kubernetes.io/unreachable
        operator: Exists
      volumes:
      - hostPath:
          path: /var/openebs/pvc-796dbc3b-714e-11e8-8df4-42010a800fe1
          type: ""
        name: openebs
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: 2018-06-16T10:17:32Z
    lastUpdateTime: 2018-06-16T10:17:32Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

@kmova
Copy link
Contributor Author

kmova commented Jun 16, 2018

Thanks @AmitKumarDas, @prateekpandey14 for quick reviews!

  • Updated the mock interfaces.
  • Updated the YAML specs

@AmitKumarDas AmitKumarDas merged commit 437fc37 into openebs-archive:master Jun 16, 2018
@kmova kmova deleted the replica-default-nodeselector branch June 16, 2018 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants