Fideslang Pydantic V2 Upgrade #11
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Temporarily install bump-pydantic to help with upgrade - Temporarily install eval_type_backport to get around issues with 'bool | None' that don't appear to be in this codebase. - Use StringConstraints instead of ConstrainedStr on FidesKey and FidesCollectionKey
…m_mode" has been renamed to "from_attributes" Affects: FidesModel, Cookies, Evaluation, PrivacyDeclaration, and System.
FidesMeta.valid_data_type. Dataset.valid_meta, Dataflow.verify_type_is_flowable replaced here.
… couldn't make automatically. Includes: DefaultModel.validate_verion_added, DefaultModel.validate_version_deprecated, DatasetField.validate_object_fields
…cated to not have values keyword.
… a root_validator which is better when performing validation using multiple field values
…ld_validator into a model_validator to better be able to work with privacy declaration and system ingresses and egresses together, since "each_item" has been deprecated.
…st means that None is an allowed value, so you need to specify a default of None to match existing behavior.
… with __get_pydantic_core_schema__
…ted keyword argument. I think skip_on_failure=true just didn’t call the root_validator if other validators failed. I am going to remove this instance.
…n't have default values of None, and are often called in a optional context.
…to better match what we had before, so the "values" are a dictionary and not an instance of an object. This is to avoid a lot of errors like "DataFlow" object is not subscriptable that I newly created.
- Update reusable validators to be f"field_validators" instead of "validators".
- Remove allow_reuse=True from all field_validators. This is not a supported keyword and can be removed entirely.
Set validate=default on the fields that were previously setting always=True, as this keyword is also not supported for field validators. This mimics the behavior of always=True in v1.
- Data type in these field validators is a ValidationInfo not a dictionary. The data can be accessed via values.data.get(""_
- Types are not coerced any longer. We were passing in an organization_fides_key of integer 1, but it no longer gets coerced to a string. It either needs to be passed in as a string, or we need to update FidesKey itself to coerce it. - Return all elements has other validation so it should be null if it's not a list field - Update the definitions of FidesKey and FidesCollectionKey to be subclassing StringConstraints to try to maintain original validation. Override __get_pydantic_core_schema__ - Due to other arrangements around when validation is running (mode=before or after) egress key/ingress key does not necessarily exist when validating privacy declaration egress and egress.
…ne to match previous behavior.
…o be when validating. Similarly, likely not defining FidesKey and FidesCollectionKey in the proper ways. - Have FidesKey and FidesCollectionKey extend str - instead of StringConstraint - still defining a __get_pydantic_core_schema__ to override how it gets validated
parse_obj -> model_validate update_forward_refs -> model_rebuild __fields_set__ -> model_fields_set construct -> model_construct
…type - just use Version directly.
- Bump xenon and pre-commit in dev-requirements
… the bump from v1 to v2.
Also, throw proper error if version is not a ValidVersion.
… still work. - Place self-reference check before matching parent check when validating data categories and data uses because a self-reference would actually fail in the matching parent check, but the error is less clear there. - Also fix typo in existing error message
…nded to test. - Assert exception thrown in test for duplicate collections errors. Fix integer dataset fides keys because this was failing first and preventing the duplicate entries found error from being thrown - Add more assertions on the error message itself to ensure we're testing the right point of failure
Where I'm using model_validator with after model, adjust so it's an instance method and is receiving an instance and returning the instance
…ten the error message I'm checking to get the gist of the check.
pattisdr
commented
Jun 17, 2024
…ore running regex.
… given recurring issues where sometimes the data that is validated is a dictionary but it can also be the instance itself.
20 tasks
…ers to strings if applicable.
… when it is validated. Use this for System.privacy_policy and System.legitimate_interest_disclosure_url.
…serializer warnings. We are intentionally serializing these as strings.
pattisdr
changed the title
…well as AnyHttpUrlString.
…to always strip a trailing slash.
pattisdr
force-pushed
the
fideslang_pydantic_v2_upgrade
branch
from
3fbff01
to
b27bf03
Compare
galvana
approved these changes
Aug 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #PROD-2263
Pydantic 1 -> 2 Migration Guide: link
Replaces: IABTechLab#160 and #9
Description Of Changes
Code Changes
parse_obj->model_validatevalidatorswithfield_validators, allow_reuse keywords deletedroot_validatorswithmodel_validatorsvalidate=Truearg on the respective fieldsOptionalwere given a default value of None, since Optional only means "not required" in V2 if there's a defaultclass Configsdefined in the BaseModel namespace withmodel_configsorm_moderenamed tofrom_attributesFidesVersionconcept entirely since it wasn't being used as a Pydantic model. Just usingVersionis sufficient.update_forward_refs->model_rebuildFidesKeysandFidesCollectionKeyswere overhauled in how they are defined in V2each_itemkeyword.__fields_set__replaced withmodel_fields_setSteps to Confirm
Pre-Merge Checklist
CHANGELOG.md