From c17ffa57349b4c3ef035d98b4f96df2db4163231 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 19 Jul 2022 17:44:59 +0200 Subject: [PATCH 1/3] Several improvements for estimates, budget and costs #443 --- CHANGELOG.md | 4 ++++ errors.json | 8 ++++++++ openapi.yaml | 30 +++++++++++++++++++++++------- 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca5f4f1b..3c2f509b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated STAC specification examples and references to v1.0.0. - `cube:dimensions`: `reference_system` is allowed to be PROJJSON, too. - Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) +- Minimum value for `costs` and `budget` is 0. +- `GET /jobs/{job_id}/estimate`: If a batch job can't be estimated reliably, a `EstimateComplexity` error should be returned. [#443](https://github.com/Open-EO/openeo-api/issues/443) ### Fixed @@ -44,6 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Reflect that the `debug` process has been renamed to `inspect`. - `GET /credentials/oidc`: Clarify that clients may add additional scopes - Clarified uniqueness constraints for identifiers. [#454](https://github.com/Open-EO/openeo-api/issues/454) +- `GET /me`: Clarify the behavior of the field `budget`. +- `GET /jobs/{job_id}/estimate`: Don't require that the costs are the upper limit. Services may specify the costs more freely depending on their terms of service. ## [1.1.0] - 2021-05-17 diff --git a/errors.json b/errors.json index 1e2f3239..7288ba6b 100644 --- a/errors.json +++ b/errors.json @@ -66,6 +66,14 @@ "Data Processing" ] }, + "EstimateComplexity": { + "description": "The process is too complex to calculate an estimate, e.g. due to a UDF or other processes that are complex to estimate costs reliably.", + "message": "The process is too complex to calculate an estimate.", + "http": 500, + "tags": [ + "Batch Jobs" + ] + }, "NoDataForUpdate": { "description": "For PATCH requests: No valid data specified at all.", "message": "No data specified to be updated.", diff --git a/openapi.yaml b/openapi.yaml index 1de79573..7c8ba5bc 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2974,12 +2974,20 @@ paths: description: >- Calculates an estimate for a batch job. Back-ends can decide to either calculate the duration, the costs, the size or a combination of - them. This MUST be the upper limit of the incurring costs. Clients can - be charged less than specified, but never more. Back-end providers MAY - specify an expiry time for the estimate. Starting to process data - afterwards MAY be charged at a higher cost. Costs MAY NOT include - downloading costs. This can be indicated with the `downloads_included` - flag. + them. + + Back-end providers MAY specify an expiry time for the estimate. + Starting to process data afterwards MAY be charged at a higher cost. + Costs MAY NOT include downloading costs. This can be indicated with the + `downloads_included` flag. + + The estimate SHOULD be the upper limit of the costs, but back-end are + free to use the field according to their terms of service. + + For some batch jobs it is not (easily) possible to estimate the costs + reliably, e.g. if a user-submitted UDF is part of the process. + In this case, the server SHOULD return a `EstimateComplexity` error + with HTTP status code 500. tags: - Data Processing - Batch Jobs @@ -3566,7 +3574,12 @@ paths: to the user. example: 1073741824 budget: - $ref: '#/components/schemas/budget' + type: number + nullable: true + description: |- + The remaining budget a user has available. + The value MUST be specified in the currency of the back-end. + The value SHOULD be set to `null` if no explicit limit applies. links: description: |- Links related to the user profile, e.g. where payments @@ -4096,11 +4109,13 @@ components: currency the back-end is working with. The currency can be retrieved by calling `GET /`. If no currency is set, this field MUST be `null`. type: number + minimum: 0 example: 12.98 nullable: true default: null budget: type: number + minimum: 0 nullable: true description: |- Maximum amount of costs the request is allowed to produce. @@ -4111,6 +4126,7 @@ components: default: null budget_update: type: number + minimum: 0 nullable: true description: |- Maximum amount of costs the request is allowed to produce. From b0270f670030bac56d83b8077b8d88a548e41759 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 1 Aug 2022 12:50:48 +0200 Subject: [PATCH 2/3] Update openapi.yaml --- openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 809ddba2..d24ffdfa 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2984,7 +2984,7 @@ paths: free to use the field according to their terms of service. For some batch jobs it is not (easily) possible to estimate the costs - reliably, e.g. if a user-submitted UDF is part of the process. + reliably, e.g. if a UDF or ML model is part of the process. In this case, the server SHOULD return a `EstimateComplexity` error with HTTP status code 500. tags: From d10b6cd41efc20816b7c325ea005d67961a8e3cd Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 1 Aug 2022 17:24:41 +0200 Subject: [PATCH 3/3] Update openapi.yaml --- openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index d24ffdfa..db57b0bc 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2977,7 +2977,7 @@ paths: Back-end providers MAY specify an expiry time for the estimate. Starting to process data afterwards MAY be charged at a higher cost. - Costs MAY NOT include downloading costs. This can be indicated with the + Costs do often not include download costs. Whether download costs are included or not can be indicated explicitly with the `downloads_included` flag. The estimate SHOULD be the upper limit of the costs, but back-end are