Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically validate a process graph before download/execute/execute_batch, ... #404

Closed
soxofaan opened this issue Mar 28, 2023 · 2 comments
Closed

Automatically validate a process graph before download/execute/execute_batch, ... #404

soxofaan opened this issue Mar 28, 2023 · 2 comments
Assignees
@JohanKJSchreurs
Labels
api compliancy feature request usability

Comments

@soxofaan
Copy link
Member

The /validation endpoint allows back-end to run some pre-flight checks on a process graph without executing it. This should give quick feedback about the validity of a given process graph, not only in syntactic sense, but also in sense of data availability, or "too large extent" checks.

Doing /validation check automatically (with option to disable it) from .download(), .create_job(), .execute_batch(), ... might be interesting to give quicker feedback to user
@jdries
Copy link
Collaborator

jdries commented Sep 12, 2023

Step 1: log warning but still submit job, this avoids blocking jobs when /validation endpoint doesn't work properly.

JohanKJSchreurs added a commit that referenced this issue Sep 29, 2023
@JohanKJSchreurs
Issue #404 Add automatic validation of process graph to connection
8cbe0d5
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issu #404 Add automatic process graph validation, when backend suppor…
748a52e 
…ts it
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issue #404 Add automatic process graph validation, when backend suppo…
4de7d52 
…rts it
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issue #404 Pass validate parameter through in datacube
876a7c2
@JohanKJSchreurs JohanKJSchreurs mentioned this issue Oct 3, 2023
Closed
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issue #404 Tests: Simplify fixture connection_with_pgvalidation
adde746
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issue #404 Pass validate parameter through in vectorcube
bdf5206
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issue #404 Add test coverage for processgraph validation in datacube
2732869
JohanKJSchreurs added a commit that referenced this issue Oct 3, 2023
@JohanKJSchreurs
Issue #404 Add test coverage for processgraph validation in vectorcube
ec31b84
JohanKJSchreurs added a commit that referenced this issue Oct 4, 2023
@JohanKJSchreurs
Issue #404 Remove toggle VALIDATE_PROCESS_GRAPH_BY_DEFAULT
405be57 
Now execute / download request will always validate process graph unless user asks to skip validation.
JohanKJSchreurs added a commit that referenced this issue Oct 4, 2023
@JohanKJSchreurs
Issue #404 Update changelog
41dd4ec
JohanKJSchreurs added a commit that referenced this issue Oct 4, 2023
@JohanKJSchreurs
Issue #404 Code review, more robust error handling during validation
f8d523c
JohanKJSchreurs added a commit that referenced this issue Oct 4, 2023
@JohanKJSchreurs
Issue #404 Minor clarifications in unit test
455632d
soxofaan added a commit that referenced this issue Oct 17, 2023
@soxofaan
Issue #404/PR #481 introduce connection level auto_validate option
54a42d3 
- collection level option will be more practical and user friendly
- per `execute` overriding is still possible
- generalize `_warn_if_process_graph_invalid` to `_preflight_validation` (to future proof it)
- additional finetuning
soxofaan added a commit that referenced this issue Oct 17, 2023
@soxofaan
fixup! Issue #404/PR #481 introduce connection level auto_validate op…
0157f8e 
…tion
soxofaan added a commit that referenced this issue Oct 17, 2023
@soxofaan
Issue #404/PR#481 test_connection streamlining
788a954 
- Use `dummy_backend` fixture for more compact setup
- Separate TestExecuteFromJsonResources and TestExecuteWithValidation
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
Issue #404/PR#481 test_datacube100 streamlining
dd4804c 
- use `dummy_backend` for more compact setup
- cover more combinations of server-side support, `auto_validate` and explicit `validate`
- leverage fixture parameterization more to avoid custom fixtures
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! Issue #404/PR#481 test_datacube100 streamlining
c5e145e
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! fixup! Issue #404/PR#481 test_datacube100 streamlining
ec63990
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! fixup! fixup! Issue #404/PR#481 test_datacube100 streamlining
d6f8797
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! fixup! fixup! fixup! Issue #404/PR#481 test_datacube100 stream…
b90f252 
…lining
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! fixup! fixup! fixup! fixup! Issue #404/PR#481 test_datacube100…
0788181 
… streamlining
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! fixup! fixup! fixup! fixup! fixup! Issue #404/PR#481 test_data…
93b423a 
…cube100 streamlining
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
fixup! fixup! fixup! fixup! fixup! fixup! fixup! Issue #404/PR#481 te…
2015b9e 
…st_datacube100 streamlining
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Add automatic process graph validation, when backend suppo…
b1ac1ae 
…rts it
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Pass validate parameter through in datacube
c5cc988
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Tests: Simplify fixture connection_with_pgvalidation
44f48e5
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Pass validate parameter through in vectorcube
41f4742
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Add test coverage for processgraph validation in datacube
81e734d
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Add test coverage for processgraph validation in vectorcube
6067d5f
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Remove toggle VALIDATE_PROCESS_GRAPH_BY_DEFAULT
f78173f 
Now execute / download request will always validate process graph unless user asks to skip validation.
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Update changelog
01a1e1f
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Code review, more robust error handling during validation
74338b3
soxofaan pushed a commit that referenced this issue Oct 18, 2023
@JohanKJSchreurs @soxofaan
Issue #404 Minor clarifications in unit test
0c6e3da
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
Issue #404/#481 introduce connection level auto_validate option
720483e 
- connection level option will be more practical and user friendly
- per `execute` overriding is still possible
- generalize `_warn_if_process_graph_invalid` to `_preflight_validation` (to future proof it)
- additional finetuning
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
Issue #404/#481 test_connection streamlining
d7925d7 
- Use `dummy_backend` fixture for more compact setup
- Separate TestExecuteFromJsonResources and TestExecuteWithValidation
soxofaan added a commit that referenced this issue Oct 18, 2023
@soxofaan
Issue #404/#481 test_vectorcube streamlining
54fa08b 
- use `dummy_backend` for more compact setup
- cover more combinations of server-side support, `auto_validate` and explicit `validate`
- leverage fixture parameterization more to avoid custom fixtures
@soxofaan
Copy link
Member Author

merged PR #481 in 78e8e93

@soxofaan soxofaan mentioned this issue Oct 25, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JohanKJSchreurs JohanKJSchreurs

Labels
api compliancy feature request usability
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@soxofaan @jdries @JohanKJSchreurs