Support non-root control-plane services #1367

NicolasT · 2019-01-28T19:42:26Z

Currently (kubeadm 1.13), the services deployed using static manifests on the control-plane node(s) are running as the root user, even though this is not strictly required, as long as FS permissions on whatever is hostPath-exported in the Pods are set up correctly.

Using securityContext.runAsUser and fsGroup one can set the UID and GID under which the process in the container should run. It would be useful, from a security PoV, to drop to an unprivileged user (based on configuration, if need be) when executing etcd, kube-apiserver, kube-controller-manager and kube-scheduler.

We could even consider dropping all 'capabilities' except those required to retain (if any).

edit by neolit123

k/e tracking issue:
kubernetes/enhancements#2568

The text was updated successfully, but these errors were encountered:

vjdhama · 2019-01-29T13:42:56Z

I would love to work on this as my first contrib to kubeadm.

yagonobre · 2019-01-29T17:33:56Z

Thanks @vjdhama
/lifecycle active

ashishranjan738 · 2019-02-01T05:57:24Z

@vjdhama are you working on this issue. I was thinking of giving it a try ? cc @yagonobre

vjdhama · 2019-02-01T18:23:02Z

@ashishranjan738 Yes, I am.

astrieanna · 2019-05-21T17:46:28Z

We're picking this up, but would like some clarification. What user/group should the processes run as instead of root? Is there an existing user/group that can be assumed to exist and be unprivileged or should this be a new configuration option?

neolit123 · 2019-05-25T17:00:40Z

We're picking this up, but would like some clarification. What user/group should the processes run as instead of root? Is there an existing user/group that can be assumed to exist and be unprivileged or should this be a new configuration option?

everybody was at kubecon, so might have been better to not send the PR before that.
i've replied on the PR and assigned more reviewers.

astrieanna · 2019-05-31T17:24:28Z

On our PR that made UID/GID configurable, @rosti said:

I don't see the use case for this to be configurable. In fact, I think, that if we want to run the static control plane pods unprivileged, we should make it the default and stick to it.
The only case, where we might want to keep those pods running as root is after upgrade from an old running cluster. Yet, even that might be questionable.
Certainly, I don't think, that exposing the UID & GID as config knobs is the right direction to go with this.

In this case, what UID/GID should the processes be run as by default?

randomvariable · 2019-07-03T16:53:26Z

This will be needed to complete the work kubernetes/kubernetes#79722

neolit123 · 2019-07-10T15:07:41Z

the usage of hostPath for root owned paths is creating a complication with this feature request.

here is a dump of the kubeadm static pods.
it shows all the paths we currently mount by default.

we need to examine these paths and see what is owned by kubeadm.
existing user mounts can potentially break if the users expect root.
upgrades from root -> non-root Pods have to be tested.

lubo-it@luboitvbox:~/go/src/k8s.io/kubernetes$ sudo cat /etc/kubernetes/manifests/etcd.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://192.168.0.102:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://192.168.0.102:2380
    - --initial-cluster=luboitvbox=https://192.168.0.102:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.0.102:2379
    - --listen-peer-urls=https://192.168.0.102:2380
    - --name=luboitvbox
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd:3.3.10
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - /bin/sh
        - -ec
        - ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
          --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
          get foo
      failureThreshold: 8
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: etcd
    resources: {}
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
status: {}
lubo-it@luboitvbox:~/go/src/k8s.io/kubernetes$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.0.102
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.14.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.0.102
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}
lubo-it@luboitvbox:~/go/src/k8s.io/kubernetes$ sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    image: k8s.gcr.io/kube-scheduler:v1.14.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}
lubo-it@luboitvbox:~/go/src/k8s.io/kubernetes$ sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    image: k8s.gcr.io/kube-controller-manager:v1.14.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-controller-manager
    resources:
      requests:
        cpu: 200m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      name: flexvolume-dir
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      type: DirectoryOrCreate
    name: flexvolume-dir
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

neolit123 · 2019-08-03T16:50:31Z

deadlines for KEPs is done.
i'm 50/50 whether this needs one or not - given the latest discovered complications i'd say it needs one.

if someone has the bandwidth deadline for KEP exceptions is 19 Aug.
https://github.com/kubernetes/sig-release/tree/master/releases/release-1.16

fejta-bot · 2020-02-07T21:02:04Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

neolit123 · 2020-02-07T21:55:14Z

/lifecycle frozen

neolit123 · 2021-05-17T22:26:44Z

moved to #1367
which will happen in 1.22

kennethredler · 2021-05-17T23:03:25Z

moved to #1367
which will happen in 1.22

Seems this is a self-referential link @neolit123 ? 🤔 Was there a different issue this should've been linked to?

neolit123 · 2021-05-18T00:41:33Z

my mistake, it's here:
#2473

kennethredler · 2021-05-18T10:48:28Z

my mistake, it's here:
#2473

No worries. And thanks! 😃

timothysc added this to the Next milestone Jan 28, 2019

k8s-ci-robot added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Jan 29, 2019

neolit123 removed the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Apr 22, 2019

timothysc added area/security and removed good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. labels May 10, 2019

astrieanna mentioned this issue May 24, 2019

[WIP] kubeadm: Makes UID and GID of static control plane pods configurable kubernetes/kubernetes#78270

Closed

neolit123 modified the milestones: Next, v1.16 Jul 2, 2019

neolit123 removed the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jul 3, 2019

neolit123 assigned errordeveloper and neolit123 Jul 3, 2019

neolit123 assigned randomvariable and unassigned errordeveloper Jul 3, 2019

neolit123 mentioned this issue Jul 12, 2019

kubeadm doest't configure kube-scheduler pods to mount k8s cert directories kubernetes/kubernetes#80063

Closed

neolit123 mentioned this issue Jul 28, 2019

PKI Directory should follow principle of least privilege #1691

Open

rcythr mentioned this issue Jul 29, 2019

Stop using system-level certificates directories (i.e. /etc/pki, /etc/ssl/certs, and /etc/ca-certificates directories) #1692

Closed

neolit123 added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 3, 2019

neolit123 modified the milestones: v1.16, v1.17 Aug 29, 2019

neolit123 modified the milestones: v1.17, Next Nov 9, 2019

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 7, 2020

k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 7, 2020

neolit123 mentioned this issue Jun 22, 2020

kubeadm init should not set --allow-privileges=true in the default generated kube-apiserver.yaml manifest #2190

Closed

neolit123 mentioned this issue Sep 3, 2020

Run services as non-privileged users kubernetes/kubernetes#7961

Open

thomasdanan mentioned this issue Nov 9, 2020

Limit required privileges to run some MetalK8s control plane and infra services scality/metalk8s#2918

Open

neolit123 mentioned this issue Mar 4, 2021

[WIP] Run kube-scheduler as non-root in kubeadm. kubernetes/kubernetes#99753

Closed

neolit123 mentioned this issue Mar 13, 2021

Run control-plane as non-root in kubeadm. kubernetes/enhancements#2568

Open

18 tasks

neolit123 unassigned neolit123 and randomvariable Mar 13, 2021

neolit123 mentioned this issue Mar 13, 2021

support kube-proxy as non-root #2410

Closed

neolit123 closed this as completed May 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support non-root control-plane services #1367

Support non-root control-plane services #1367

NicolasT commented Jan 28, 2019 •

edited by neolit123

Loading

vjdhama commented Jan 29, 2019

yagonobre commented Jan 29, 2019

ashishranjan738 commented Feb 1, 2019

vjdhama commented Feb 1, 2019

astrieanna commented May 21, 2019

neolit123 commented May 25, 2019 •

edited

Loading

astrieanna commented May 31, 2019

randomvariable commented Jul 3, 2019

neolit123 commented Jul 10, 2019

neolit123 commented Aug 3, 2019

fejta-bot commented Feb 7, 2020

neolit123 commented Feb 7, 2020

neolit123 commented May 17, 2021

kennethredler commented May 17, 2021

neolit123 commented May 18, 2021

kennethredler commented May 18, 2021

Support non-root control-plane services #1367

Support non-root control-plane services #1367

Comments

NicolasT commented Jan 28, 2019 • edited by neolit123 Loading

vjdhama commented Jan 29, 2019

yagonobre commented Jan 29, 2019

ashishranjan738 commented Feb 1, 2019

vjdhama commented Feb 1, 2019

astrieanna commented May 21, 2019

neolit123 commented May 25, 2019 • edited Loading

astrieanna commented May 31, 2019

randomvariable commented Jul 3, 2019

neolit123 commented Jul 10, 2019

neolit123 commented Aug 3, 2019

fejta-bot commented Feb 7, 2020

neolit123 commented Feb 7, 2020

neolit123 commented May 17, 2021

kennethredler commented May 17, 2021

neolit123 commented May 18, 2021

kennethredler commented May 18, 2021

NicolasT commented Jan 28, 2019 •

edited by neolit123

Loading

neolit123 commented May 25, 2019 •

edited

Loading