Volume is attached on another node

[nick-phillips-dev]

Is this a BUG REPORT or FEATURE REQUEST?: Bug

What happened:
I am trying to create a writer pod and one or more reader pods attached to a single PVC without using NFS. Using Portworx's replication, if the node dies, all the pods should come up on another node together. The reason behind this is to ensure high available reads all the time to the data.

I let stork schedule the writer pod and it schedules using ReadWriteOnce. The reader pods attach to the volume using ReadOnly mode. I created my own scheduler to ensure the reader pod schedules on the same node as the writer pod.

Sometimes, when pods are deployed, I get the following error:

Normal   Scheduled    <unknown>           stork                                Successfully assigned A to B
Warning  FailedMount  55s (x8 over 119s)  kubelet,   MountVolume.SetUp failed for volume "pvc-8fa06a32-c6df-4c1d-b15e-731cafcc77b7" : rpc error: code = AlreadyExists desc = Volume is attached on another node

What you expected to happen:
Stork to schedule the pod on the correct node

How to reproduce it (as minimally and precisely as possible):
Create writer statefulset with PVC ReadWriteOnce
Create reader statefulset referring to above PVC in read only mode.

Issue new image for writer to cause rolling updates.

This is a pressing problem. Appreciate any help/advice!

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  1.18.8
• Cloud provider or hardware configuration:
  on-prem, 3 node bare metal
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others: