RabbitMQ Chart fails with EFS volume

[faraazkhan]

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST? BUG

Version of Helm and Kubernetes:
Helm Version:

kubeClient: &version.Version{SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}
ctServer: &version.Version{SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}

Kubectl Version:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T15:13:53Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2+coreos.0", GitCommit:"c6574824e296e68a20d36f00e71fa01a81132b66", GitTreeState:"clean", BuildDate:"2017-07-24T23:28:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart: RabbitMQ (https://github.com/kubernetes/charts/tree/master/stable/rabbitmq)

What happened:
RabbitMQ pod fails when using EFS (either as an existingclaim or when the default storage class is efs). Here are the container logs:

Welcome to the Bitnami rabbitmq container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
Send us your feedback at containers@bitnami.com

nami    INFO  Initializing rabbitmq
rabbitm INFO  Creating RabbitMQ System User...
rabbitm INFO  Setting RabbitMQ environment...
rabbitm INFO  Starting RabbitMQ node rabbit@localhost...
rabbitm INFO  Configuring RabbitMQ permissions...
Error executing 'postInstallation': User '2004' not found

Interestingly 2004 is the group id that kubernetes adds to the container to allow access to the shared directory. Here is the output of ls -la in the directory the EFS volume is mounted at:

root@illocutionary-vulture-rabbitmq-1534253718-g7zls:/bitnami/rabbitmq# ls -la
total 32
drwxrws--x. 5 root 2004 6144 Aug 11 01:25 .
drwxr-xr-x. 3 root root 4096 Aug 11 02:18 ..
-rw-r--r--. 1 root 2004    0 Aug 11 01:25 .initialized
drwxr-sr-x. 2 root 2004 6144 Aug 11 01:25 .rabbitmq
-rw-r--r--. 1 root 2004    0 Aug 11 02:18 .restored
drwxr-sr-x. 2 root 2004 6144 Aug 11 01:25 conf
drwxr-sr-x. 3 root 2004 6144 Aug 11 01:25 var

UPDATE

Tested on the same cluster with an EBS volume instead of EFS and it worked fine. So this really is an issue when using a shared file system.

What you expected to happen:
Expected RabbitMQ initialization (including postInstallation to complete) and the pod to run.

How to reproduce it (as minimally and precisely as possible):

• Create a kubernetes cluster in AWS
• Create an EFS file system in the same VPC
• Create a PVC mounting the said EFS file system
• Attempt to install the rabbitmq chart using the created pvc as an existing claim.

Anything else we need to know:
All of my testing against EFS was done on a 1.7.3 cluster that was created using bootkube/coreos.
The chart works fine when tested against minikube and another cluster in Azure. This might actually be an issue with the bitnami image, but figured I'd start here since @sameersbn is an author here as well.

[faraazkhan]

Created https://github.com/bitnami/bitnami-docker-rabbitmq/issues/54 for completeness.

[prydonius]

@faraazkhan does AWS EFS use an NFS mount? We've had issues with NFS mounts before, we'll take a look.

[faraazkhan]

Yea it does @prydonius

[faraazkhan]

Please let me know if I can help in anyway @prydonius. I tried looking but couldn't find the repo/code that runs under postInstallation

[qiujian16]

is there any progress on this issue, I met the same problem when using gluster. The error msg is

nami    INFO  Initializing rabbitmq
rabbitm INFO  Creating RabbitMQ System User...
rabbitm INFO  Setting RabbitMQ environment...
rabbitm INFO  Starting RabbitMQ node rabbit@localhost...
rabbitm INFO  Configuring RabbitMQ permissions...
Error executing 'postInstallation': User '2001' not found

[prydonius]

@qiujian16 I've notified the team about the GlusterFS issue, not sure if it's the same as NFS.

[hassenius]

I get the same problem trying to use NFS

INFO  Initializing rabbitmq
INFO  Creating RabbitMQ System User...
INFO  Setting RabbitMQ environment...
INFO  Setting Erlang cookie...
INFO  Managing RabbitMQ plugins...
INFO  Configuring RabbitMQ permissions...
INFO  Starting RabbitMQ node rabbit@localhost...
INFO  Configuring RabbitMQ permissions...
Error executing 'postInstallation': User '4294967294' not found

If I try to use no perssestent storage claim I get this error message instead

rabbitmq-rabbitmq rabbitm INFO  ########################################################################
rabbitm INFO   Installation parameters for rabbitmq:
rabbitm INFO     Username: user
rabbitm INFO     Node: rabbit@localhost
rabbitm INFO     Manager listening at: 0.0.0.0:15672
rabbitm INFO   (Passwords are not shown for security reasons)
rabbitm INFO  ########################################################################
INFO  rabbitmq successfully initialized
INFO  ==> Starting rabbitmq... 

Not don't other other other NFS and no persistent storage claim, not sure sure how get past this.

[Jarlotee]

@hassenius same issue for me

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stale]

This issue is being automatically closed due to inactivity.

[RichiCoder1]

This is still the case as far as I can tell