Openshift support

[gsaslis]

Hello there,

I was looking for the best way to deploy elasticsearch / kibana on openshift and it looks like I could use some help. : )

I tried adapting these charts and kept some notes with the problems that came up.

I would welcome suggestions (even for a completely different method of deploying into openshift - e.g. openshift templates / operator) and am happy to help with any questions you might have so we can figure something out (I have an openshift cluster standing by).

Thanks!

[Crazybus]

Thank you so much for taking the time to test this out and writing up detailed notes. Adding support for other platforms (like Openshift) is planned for the future but we are currently only testing and supporting Google Kubernetes Engine (GKE).

Just a heads up, I have never used Openshift before so this will be my first time trying to figure this stuff out too! I don't currently have access to an Openshift cluster to help debug but in the future I hope to come back to this to get it properly working if you can't figure it out with my comments.

I couldn’t use Helm itself due to permissions issues related to: helm/helm#1918

Is the problem that you don't want to use tiller because of the permissions required? Or is it that you are unable to get it working on Openshift? A quick google makes it sound like helm should work just fine on openshift as long as you install it per namespace. They even have an official blog post about how to install it.

the initContainer tries to run as root.

There is a beta feature in Kubernetes 1.12 to set sysctls natively through Kubernetes. It looks like vm.* would be considered "unsafe" and need to be allowed on a cluster level. Openshift also seems to support it though it still looks to be the alpha version. I can certainly see having the root init container being optional so that users can disable it and configure the needed sysctl values via annotations instead since I can imagine other users wanting to use this new feature too.

the fsGroup seems to have an invalid value. I am not sure what that should be. Deleting the fsGroup + securityContext does allow statefulset to be created, BUT, again, that’s probably there for a reason (haven’t looked into that in detail yet)

This is to make sure that the persistent volume is mounted with permissions that the Elasticsearch user can read. By default we set this to 1000 because that is the default uid. I quickly went over the Openshift persistent storage docs and it sounds like you need to manually set the fsGroup ID to something that is in your available range. So I think you can find this by running oc get project default -o yaml but I would need to test it out myself.

the underlying filesystem used for the persistent volumes. My openshift cluster came with both NFS and glusterfs. tried both with same errors, but I do remember recommendations to NOT run elasticsearch on glusterfs a couple of years ago. Perhaps this still holds (and also goes for nfs…​?)

Certainly not recommended to use NFS or Glusterfs. As far as I know it will work but it certainly won't perform very well. Another possibility is using local persistent volumes although it still seems to be in alpha on Openshift

[JarleB]

Hi,

I got elasticsearch to run in openshift (okd v3.11. Only nonpersistent so far) by doing.
oc adm policy add-scc-to-user nonroot -z default
as cluster-admin and the following values (values-jbb.yaml).

persistence:
  enabled: false
minimumMasterNodes: 1
antiAffinity: "soft"
podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
sysctlInitContainer:
  enabled: false
clusterHealthCheckParams: "wait_for_status=green&timeout=30s"

I had to do:
sysctl vm.max_map_count=262144
on all nodes used by ES (could be automated by ansible (privileged init containers is a no-go).
Then:
helm -n jbtest template ./elasticsearch -f values-jb.yaml |oc apply -f -

master# curl elasticsearch-master.fooobar.svc.cluster.local:9200/_cat/nodes 
10.131.0.11  8 82 4 0.11 0.09 0.14 dim - elasticsearch-master-1
10.129.0.11 11 96 4 0.12 0.11 0.22 dim - elasticsearch-master-0
10.129.0.12  9 96 4 0.12 0.11 0.22 dim * elasticsearch-master-2
master# curl elasticsearch-master.fooobar.svc.cluster.local:9200/_cat/health
1565958272 12:24:32 elasticsearch green 3 3 0 0 0 0 0 0 - 100.0%

oc get pods                                
NAME                     READY     STATUS    RESTARTS   AGE
elasticsearch-master-0   1/1       Running   0          38m
elasticsearch-master-1   1/1       Running   0          38m
elasticsearch-master-2   1/1       Running   0          38m
jbtest-gvqqa-test        0/1       Error     0          38m

Don't know what the test container is for, but elasticsearch seems to work anyway.

[Crazybus]

Thanks for taking the time to test this out and adding in all of the info! It sounds like everything should be workable then assuming there are no weird requirements for persistent volumes in Openshift.

Just adding some comments about things you have changed:

minimumMasterNodes: 1

this is ignored for Elasticsearch 7 so you didn't need ot set this (unless you are using v6)

antiAffinity: "soft" # Is this because you

Did you have to add this because your test environment only has 1 node? Or was it some Openshift related reason?

podSecurityContext:
fsGroup: 1000
runAsUser: 1000

Only runAsUser is not in the current defaults, we can certainly add it in though!

---

Assuming the changes you made are specifically for a testing environment it looks like Openshift should be working now by only disabling the sysctlInitContainer which is expected. I found out recently that https://www.okd.io/minishift/ is a thing which I could use to test this out on.

[JarleB]

On 19 Aug 2019, at 21:39, Michael Russell ***@***.***> wrote: Thanks for taking the time to test this out and adding in all of the info! It sounds like everything should be workable then assuming there are no weird requirements for persistent volumes in Openshift. Just adding some comments about things you have changed: minimumMasterNodes: 1 this is ignored for Elasticsearch 7 so you didn't need ot set this (unless you are using v6)

Thanks for explaining.

antiAffinity: "soft" # Is this because you Did you have to add this because your test environment only has 1 node? Or was it some Openshift related reason?

Basically yes. I played around with a few things. Until I found the just could just reduce “replicas” to 1 I used this to get something running with < 3 nodes.

podSecurityContext: fsGroup: 1000 runAsUser: 1000 Only runAsUser is not in the current defaults, we can certainly add it in though!

I definitely think default uid as non-root should be default yes :-)

Assuming the changes you made are specifically for a testing environment it looks like Openshift should be working now by only disabling the sysctlInitContainer which is expected. I found out recently that https://www.okd.io/minishift/ <https://www.okd.io/minishift/> is a thing which I could use to test this out on.

Yes. That’s good for testing. I use okd (okd.io <http://okd.io/>) on openstack and VMware together with Terraform for provisioning. Thanks for providing this helm chart by the way :-) — Best regards Jarle Bjørgeengen

[Crazybus]

I definitely think default uid as non-root should be default yes :-)

I have opened #259 to test that this works properly. If so it can go in as the new default.

Thanks for providing this helm chart by the way :-)

Thank you :)

[Crazybus]

I just added an Openshift example in #263. Could you try it out to see if it works for you? Openshift is still not supported (until we have some kind of automated testing) but this should at least be a good starting point to show what settings need to be tweaked.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue has been automatically closed because it has not had recent activity since being marked as stale.

[bmitchinson]

Ended up using the following. Setting to null as specified in the example didn't work for us.

Command to get valid userID: oc get project default -o yaml

elasticsearch:
  enabled: true
  replicas: 2
  ##################
  securityContext:
    runAsUser: {value from command}
  podSecurityContext:
    fsGroup:  {value from command}
    runAsUser:  {value from command}
  sysctlInitContainer:
    enabled: false