Offload TaskResultsCompletionStatus from etcd to db or use compression to allow large worfklows (~100k)

[Hnatekmar]

Summary

I am currently evaluating argo-workflows a goto solution for scheduling tasks for my company. So far we really like it featurewise and we thing it is really good fit 👍
Problem is that it number of tasks is expected to be around 100k per workflow and so far I haven't manage to persuade argo to do that.

From what I've observed there is limitation imposed by maximum size of entity inside etcd db which is around 1.5 MB. From my testing this can be observed with following workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i
spec:
  podGC:
    strategy: OnPodSuccess
    deleteDelayDuration: 0s
  entrypoint: e
  templates:
  - name: c
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:3.7
      command: [echo, "{{inputs.parameters.message}}"]
  - name: e1
    steps:
      #@ for i in range(100):
      - - name: #@ "message" + str(i)
          template: c
          arguments:
            parameters:
              - name: message
                value: #@ "istep-" + str(i)
      #@ end
  - name: e
    dag:
      tasks:
    #@ for i in range(1000):
        - name: #@ "Step" + str(i)
          template: e1
    #@ end

You can use it with ytt -f <manifest_name> | kubectl create -f - -n <argo_namespace>. This manifest will get stuck at around 19177/20177 mark.

When I look at content of Workflow manifest it has states of each job inside it has jobs listed like this

      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4292625616: true
      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4293120823: true
      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4293149953: true
      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4293305504: true
      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4294093307: true
      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4294368260: true
      this-is-extremly-long-prefix-so-i-will-spam-etcd-with-this-i-4294498843: true

Size of workflow manifest also roughly correlates to etcd limit:

$ kubectl get workflow -n argo-workflows -o yaml  |  wc -c
 1694314

Also when I decrease size of prefix I am able to schedule more jobs (around 80k with single character prefix)

What I am proposing is:

• change of format

  argo-workflows/pkg/apis/workflow/v1alpha1/workflow_types.go

  Line 1955 in c9b1477

  TaskResultsCompletionStatus map[string]bool `json:"taskResultsCompletionStatus,omitempty" protobuf:"bytes,20,opt,name=taskResultsCompletionStatus"`

  to single key with base64 compressed string
• Or we can offload this to db (when enabled I don't thing anyone will try this without ALWAYS_OFFLOAD_NODE_STATUS)

Here is my current configuration for argo-workflows https://github.com/Hnatekmar/kubernetes/blob/a09391109103d5ff9036eed85fd05577fff1c654/manifests/applications/argo-workflows.yaml

Use Cases

When scheduling 100k or more jobs

---

Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.

[Hnatekmar]

Also it disappeared but I am willing to work on this :) Just need to discuss how it should be done

[Hnatekmar]

#7121 seems to be relevant to this issue but it seems discussion gravited towards db optimisations. Which won't solve this issue

[shuangkun]

Duplicate with #13213

[shuangkun]

I have also encountered this problem before. I implemented ALWAYS_OFFLOAD_TASK—RESULT_STATUS. If the maintainers think this requirement is reasonable, I can contribute here.

[Joibel]

Offloading task result status to the database along with node status seems like a reasonable thing to do.

I haven't looked into it, but it felt like it might be possible to:

• Delete workflowTaskResults using a workerpool as they were marked as completed and copied into the node status.
• Then delete the taskResultStatus entries for those workflowTaskResults once we knew they were all completed as we're only tracking them to ensure we collect them all, once collected they're done with.

This might be a better solution than offloading, WDYT @shuangkun?

These might be two separate things.

[shuangkun]

At that time, the task result status field was introduced to have a key function to help determine some status of the workflow, such as whether all tasks are completed (convenient for GC, whether the workflow completion status can be set), and whether the task output parsing in the previous step is completed (starting the next pod) , I am not sure whether some task result status can achieve this effect, may need to think about it.

[Hnatekmar]

Hi,

how is it going @shuangkun ? Do you still want to contribute to this issue?

If not I can try and create PR that fixes this

Thanks in advance

[shuangkun]

@Hnatekmar Feel free to submit a PR, and we will review it.

[Joibel]

I'm unconvinced we need to preserve taskResultCompletionStatuses that are in state true, once they are true they are done for all purposes and could just be removed. That's a simpler change to make than attempting to offload. What do you think @shuangkun?

[Hnatekmar]

@Joibel if I may. This could still cause issues depending on current workload.

I would be in favor of offloading it completely if it is feasible (I have to orient myself in the codebase but it doesn't seem impossible)

[shuangkun]

I'm unconvinced we need to preserve taskResultCompletionStatuses that are in state true, once they are true they are done for all purposes and could just be removed. That's a simpler change to make than attempting to offload. What do you think @shuangkun?

Agree. I sorted out the information of taskresult and found that it is mainly used in two places:

1. artifacts gc. For stopped workflows, gc needs to be performed after all artifacts are successfully processed.
2. In each round of workflow processing, it is necessary to ensure that the artifacts of the succeeded pod are processed before creating a new pod (because any new pod may rely on the previous artifact reference).
   If we can handle these two, we don’t need to maintain such a large data structure.

[shuangkun]

The succeeded information and taskresult completion information are now important for each node.
The workflow-controller needs to determine whether a new pod can be created based on the above information.
Is it possible to add this information to the node status so that no additional offload is required?