Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Volume is attached on another node #726

Open
nick-phillips-dev opened this issue Oct 5, 2020 · 0 comments
Open

Volume is attached on another node #726

nick-phillips-dev opened this issue Oct 5, 2020 · 0 comments

Comments

@nick-phillips-dev
Copy link

Is this a BUG REPORT or FEATURE REQUEST?: Bug

What happened:
I am trying to create a writer pod and one or more reader pods attached to a single PVC without using NFS. Using Portworx's replication, if the node dies, all the pods should come up on another node together. The reason behind this is to ensure high available reads all the time to the data.

I let stork schedule the writer pod and it schedules using ReadWriteOnce. The reader pods attach to the volume using ReadOnly mode. I created my own scheduler to ensure the reader pod schedules on the same node as the writer pod.

Sometimes, when pods are deployed, I get the following error:

Normal   Scheduled    <unknown>           stork                                Successfully assigned A to B
Warning  FailedMount  55s (x8 over 119s)  kubelet,   MountVolume.SetUp failed for volume "pvc-8fa06a32-c6df-4c1d-b15e-731cafcc77b7" : rpc error: code = AlreadyExists desc = Volume is attached on another node

What you expected to happen:
Stork to schedule the pod on the correct node

How to reproduce it (as minimally and precisely as possible):
Create writer statefulset with PVC ReadWriteOnce
Create reader statefulset referring to above PVC in read only mode.

Issue new image for writer to cause rolling updates.

This is a pressing problem. Appreciate any help/advice!

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
    1.18.8
  • Cloud provider or hardware configuration:
    on-prem, 3 node bare metal
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant