Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Issue 1312 incorrect status code #1317

Merged
merged 7 commits into from
Nov 11, 2024

Conversation

sundar-mudupalli-work
Copy link
Collaborator

Hi,

This fixes the issue that DVT returns status code (0) even if there is an exception while validating the yaml file. This results in Cloud Run considering the validation a success even when there is a transient error such as writing to BigQuery.

The issue Error handling for --config-dir YAMLs was raised. The issue is that when customers are validating multiple yaml files in a directory, customers had to fix one yaml at a time. When one yaml failed validation and raised an exception, the rest of the yaml files were not processed. This was fixed in PR605. The fix was to catch exceptions raised, throw an error and skip to processing the next file. However, the exception raised in processing one of the files was lost. This "losing the exception" happened even if only one yaml file was being processed.

The fix is to move the exception processing logic to where the config runner is processing multiple files, so this behavior is only when multiple files are being processed. A flag is set to signal that one of the files had an exception and an exception is raised after processing all the files. In this way, the status is reported correctly. A config runner unit test has also been added.

Sundar Mudupalli

@sundar-mudupalli-work sundar-mudupalli-work requested a review from a team as a code owner November 6, 2024 19:16
@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

Copy link
Contributor

@nj1973 nj1973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple of minor discussion points.

data_validation/__main__.py Outdated Show resolved Hide resolved
tests/unit/test__main.py Outdated Show resolved Hide resolved
data_validation/__main__.py Outdated Show resolved Hide resolved
data_validation/__main__.py Outdated Show resolved Hide resolved
@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

Copy link
Contributor

@nj1973 nj1973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nj1973
Copy link
Contributor

nj1973 commented Nov 11, 2024

/gcbrun

@sundar-mudupalli-work sundar-mudupalli-work merged commit ade6d79 into develop Nov 11, 2024
5 checks passed
@sundar-mudupalli-work sundar-mudupalli-work deleted the issue-1312-incorrect-status-code branch November 11, 2024 15:07
helensilva14 pushed a commit that referenced this pull request Jan 20, 2025
* Updated so that when execptions are raised, they are re-raised when processing multiple files - test suite to be updated.

* Test to test exceptions were raised after validating multiple files

* Linted

* Removed extraneous comment

* Updated based on feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Success status returned when validations in YAML files throw exception
3 participants