[FEATURE]: Add tolerations to many garden-run pods

[to266]

Feature Request

Background / Motivation

Would love to add kubernetes tolerations to various garden-run resources - such as tests, tasks, cluster-buildkit pod (and anything else I may have missed for that matter)

What should the user be able to do?

Specify tolerations for container.tests[].tolerations, container.tasks[].tolerations as well as for the providers[].clusterBuildkit.tolerations.

Why do they want to do this? What problem does it solve?

That way I would not need to use expensive on-demand nodes, and could rely on the much cheaper spot instances. (In the particular azure case, each spot instance has a taint by default, and it's impossible to get rid of it).

Suggested Implementation(s)

Just allow passing the setting for each of the objects. With the project variables, it's simple to manage it centrally via that mechanism.

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden needlessly expensive

[thsig]

Hi @to266!

Tests and tasks for kubernetes and helm modules will apply any toleration config from the pod spec of the service resource to the one-off pods used when running the test/task. See https://docs.garden.io/reference/module-types/kubernetes#tests-.resource for a complete list of fields from the pod spec that are included.

The container module type doesn't expose all pod spec fields (we're intentionally trying to avoid introducing too many Kubernetes-specific fields to the container module type for the time being, since the idea is for that module type not to be overly Kubernetes-specific).

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!