Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-16481 Update schema #10

Merged
merged 4 commits into from
Jan 17, 2022
Merged

TDL-16481 Update schema #10

merged 4 commits into from
Jan 17, 2022

Conversation

prijendev
Copy link
Contributor

@prijendev prijendev commented Nov 25, 2021

Description of change

  • Updated schema for following stream
    • deal_stages
    • contact_custom_fields
    • conversions
    • templets
    • contact_list
    • brandings
    • forms
    • users
    • tasks
    • contact
    • campaign_links
    • deal_activities
  • Updated property params of automation_blocks to support empty array data type.
  • Added all_fields integration test case.
  • Removed unnecessary files(test case for snapchat_ads) under tests directory.

Manual QA steps

  • Run discover mode and verify that fields replicate as expected in catalog
  • Run sync mode and verify that all fields replicate in output.

Risks

Rollback steps

  • revert this branch

@prijendev prijendev changed the base branch from master to crest-master November 25, 2021 12:05
@@ -25,6 +25,6 @@ def transform_json(this_json, stream_name, data_key):
else:
converted_json = humps.decamelize(this_json)

fixed_records = fix_records(converted_json)
fix_records(converted_json)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed pylint error assignment-from-no-return. (Reference)

Comment on lines +15 to +29
"anyOf": [
{
"type": ["null", "object"],
"additionalProperties": true,
"properties": {}
},
{
"type": "array",
"items": {
"type": ["null", "object"],
"additionalProperties": true,
"properties": {}
}
}
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this update to schema be backward compatible?

Copy link
Contributor Author

@prijendev prijendev Jan 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we have made this change just to handle the empty array value of the param field. In the existing schema, it handles only object type values that are still supported. So, now it will handle the following both possible values of the param field,

  1. "param": {"eventType": "never"}
  2. "param": []

@prijendev prijendev merged commit 363839b into crest-master Jan 17, 2022
@prijendev prijendev mentioned this pull request Jan 18, 2022
KrisPersonal pushed a commit that referenced this pull request Jan 31, 2022
* TDL-16481 Update schema (#10)

* updated typo in schemas

* Updated test case

* Updated config.yml file

* Fixed automation_blocks stream

* TDL-16549 implement request timeout (#12)

* TDL-16549 implement request timeout

* Updated config.yml file

* Fixed integration test case failure

* resolved comments

* resolved comments

Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com>

* TDL-16386 Refactor error handling (#8)

* Initial commit for error handling

* updated error handling

* Updated backoff for 5xx error

* Updated unit test cases

* Updated config.yml file

* Resolved pylint error

* Added ConnectionResetError exception

* Updated raise_for_error method

* Updated comments in the code

* TDL-16383 Refactor code to class based (#9)

* Updated dict based to class based

* Updated parent child format

* Updated config file

* Updated config file

* Resolved circleci error

* Fixed circleci issue

* Removed integration test case snippet from config file

* Added comments in the code

* Resolved comments

* Added comment for process_records function

* Updated comments

* TDL 16384 check best practices and TDL 16465 Make replication key with automatic inclusion (#11)

* Initial commit for best practices

* Updated integration test case as per client account data

* Added back site_messages stream in pagination test

* Updated config file

* Fixed circleci issue

* Updated tap-teser version

* Fixed test case issue

* Added test in config.yml

* Fixed start_date test case

* Updated discover test case.

* Added comments

* Added test case reference link

* Removed comments

* Removed slack notifier

* Added slack status in config.yml

* UPdated config.yml version

* Added context in build_daily

Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants