Honor "budget" when/before running batch job

[soxofaan]

When creating a batch job, the user can specify a "budget":

POST /jobs
Maximum amount of costs the request is allowed to produce. The value MUST be specified in the currency of the back-end. No limits apply, if the value is null or the back-end has no currency set in GET /.

This attribute is supported e.g. by the python client (e.g. see https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.datacube.DataCube.create_job), but the geopyspark backend ignores this

[soxofaan]

The need for such a feature came up last week, when we had the problem that jobs suddenly took significantly longer to run and consequently consumed significantly more credits than normal due to cluster saturation

related:

• Budget handling behaviour openeo-api#541
• As a developer I want to provide the MultiBackendJobManager with a maximal run time for each job. openeo-python-client#590
• execute_batch: option to cancel excessively long running jobs openeo-python-client#589

[soxofaan]

cc @HansVRP

[soxofaan]

I'm not sure if we can reliably predict the cost of a job before running it, so blocking the start of a job because of budget constraints is probably not feasible at the moment.

However we could look into automatically canceling a job when it goes over budget (assuming we can track that in "real time")