Exceeding quota with volumeClaimTemplates

[KPostOffice]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:
I have created a template which uses a volume claim template. If the total claimed by running pods is at storage quota then new pods being attempted to run enter error state rather than pending state.

What you expected to happen:
If storage resources are unavailable then workflow should remain pending until resources are available.

How to reproduce it (as minimally and precisely as possible):
In a namespace with a quota of 10Gi of storage the error should start occuring after 5 have been submitted.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: storage-limit-
spec:
  serviceAccountName: argo
  entrypoint: wait
  podGC:
    strategy: OnPodCompletion

  volumeClaimTemplates:                 # define volume, same syntax as k8s Pod spec
    - metadata:
        name: workdir                     # name of volume claim
      spec:
        accessModes: [ "ReadWriteMany" ]
        resources:
          requests:
            storage: 2Gi                  # Gi => 1024 * 1024 * 1024

  templates:
  - name: wait
    resubmitPendingPods: true
    script:
      image: alpine:latest
      command: [sh, -c]
      args: ["sleep", "30s"]

Use bash script to run this some number of times until you encounter the issue.

Anything else we need to know?:
I believe this issue also occurs when hitting PVC limits.

Environment:

• Argo version:

$ argo version
argo: 2.8.2+8a151ae.dirty
  BuildDate: 2020-06-18T23:50:58Z
  GitCommit: 8a151aec6538c9442cf2380c2544ba3efb60ff60
  GitTreeState: dirty
  GitTag: 2.8.2
  GoVersion: go1.13
  Compiler: gc
  Platform: linux/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: 2020-01-29T21:26:39Z
  compiler: gc
  gitCommit: d4cacc0
  gitTreeState: clean
  gitVersion: v1.10.0+d4cacc0
  goVersion: go1.14beta1
  major: "1"
  minor: 10+
  platform: linux/amd64
serverVersion:
  buildDate: 2020-05-04T12:54:43Z
  compiler: gc
  gitCommit: a3ec9df
  gitTreeState: clean
  gitVersion: v1.16.2
  goVersion: go1.12.12
  major: "1"
  minor: 16+
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

$ argo --loglevel DEBUG get <workflowname>
DEBU[0000] CLI version                                   version="{2.8.2+8a151ae.dirty 2020-06-18T23:50:58Z 8a151aec6538c9442cf2380c2544ba3efb60ff60 2.8.2 dirty go1.13 gc linux/amd64}"
DEBU[0000] Client options                                opts="{{ false false} 0x1574670 0xc00011f040}"
Name:                storage-limit-v9qjt
Namespace:           thoth-test-core
ServiceAccount:      argo
Status:              Error
Message:             persistentvolumeclaims "storage-limit-v9qjt-workdir" is forbidden: exceeded quota: thoth-test-core-quota, requested: requests.storage=2Gi, used: requests.storage=40Gi, limited: requests.storage=40Gi
Conditions:          
 Completed           True
Created:             Wed Jul 08 11:57:57 -0400 (7 minutes ago)
Started:             Wed Jul 08 11:57:57 -0400 (7 minutes ago)
Finished:            Wed Jul 08 11:57:57 -0400 (7 minutes ago)
Duration:            0 seconds

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

@sarabala1979 ?

[sarabala1979]

I will work on it