Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOP-7542] - add validation for hwm_column parameter #78

Merged
merged 6 commits into from
Jul 21, 2023

Conversation

maxim-lixakov
Copy link
Contributor

@maxim-lixakov maxim-lixakov commented Jul 19, 2023

Change Summary

  • add validate_hwm_column method for abstract class Dialect and its implementation in Kafka.Dialect
  • add unit tests for validate_hwm_column in Kafka.Dialect class

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@maxim-lixakov maxim-lixakov added the ci:skip-changelog Add this label to skip changelog file check label Jul 19, 2023
@maxim-lixakov maxim-lixakov self-assigned this Jul 19, 2023
@codecov
Copy link

codecov bot commented Jul 19, 2023

Codecov Report

Merging #78 (62460ea) into develop (459590e) will decrease coverage by 0.03%.
The diff coverage is 88.23%.

@@             Coverage Diff             @@
##           develop      #78      +/-   ##
===========================================
- Coverage    92.72%   92.69%   -0.03%     
===========================================
  Files          139      140       +1     
  Lines         6131     6162      +31     
  Branches      1123     1132       +9     
===========================================
+ Hits          5685     5712      +27     
- Misses         345      347       +2     
- Partials       101      103       +2     
Impacted Files Coverage Δ
onetl/connection/db_connection/greenplum.py 69.33% <ø> (ø)
onetl/connection/db_connection/hive.py 91.59% <ø> (ø)
onetl/connection/db_connection/jdbc_connection.py 92.67% <ø> (ø)
onetl/connection/db_connection/postgres.py 97.61% <ø> (ø)
...onnection/dialect_mixins/support_hwm_column_str.py 75.00% <75.00%> (ø)
onetl/connection/db_connection/kafka/dialect.py 92.00% <90.00%> (-8.00%) ⬇️
onetl/base/base_db_connection.py 100.00% <100.00%> (ø)
...onnection/db_connection/dialect_mixins/__init__.py 100.00% <100.00%> (ø)
onetl/connection/db_connection/mongodb.py 94.00% <100.00%> (ø)
onetl/db/db_reader/db_reader.py 90.20% <100.00%> (+0.15%) ⬆️

Co-authored-by: Maxim Martynov <msmarty5@mts.ru>
@maxim-lixakov maxim-lixakov temporarily deployed to test-pypi July 20, 2023 08:31 — with GitHub Actions Inactive
@maxim-lixakov maxim-lixakov temporarily deployed to test-pypi July 20, 2023 12:23 — with GitHub Actions Inactive
@maxim-lixakov maxim-lixakov temporarily deployed to test-pypi July 20, 2023 12:25 — with GitHub Actions Inactive
@maxim-lixakov maxim-lixakov marked this pull request as ready for review July 20, 2023 12:26
@maxim-lixakov maxim-lixakov temporarily deployed to test-pypi July 20, 2023 13:05 — with GitHub Actions Inactive
Comment on lines 398 to +402
values["hwm_column"] = Column(name=hwm_column) # type: ignore
values["hwm_expression"] = hwm_expression

dialect = connection.Dialect
dialect.validate_hwm_column(connection, hwm_column) # type: ignore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
values["hwm_column"] = Column(name=hwm_column) # type: ignore
values["hwm_expression"] = hwm_expression
dialect = connection.Dialect
dialect.validate_hwm_column(connection, hwm_column) # type: ignore
dialect = connection.Dialect
dialect.validate_hwm_column(connection, hwm_column) # type: ignore
values["hwm_column"] = Column(name=hwm_column) # type: ignore
values["hwm_expression"] = hwm_expression

я бы сделал так, а то странно проводить валидацию после присвоения в values

@dolfinus dolfinus merged commit fb626dc into develop Jul 21, 2023
37 of 38 checks passed
@dolfinus dolfinus deleted the feature/DOP-7542 branch July 21, 2023 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci:skip-changelog Add this label to skip changelog file check
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants