TDL-14475 added unsupported feature and unittests #47

namrata270998 · 2021-10-27T05:12:16Z

Description of change

added unsupported inclusion property and a description explaining in the catalog

Manual QA steps

Added unit test case to check that the unsupported inclusion property is properly created.
Added unit test case to check the description is properly mapped.
Updated the integration test to verify unsupported fields

Risks

Rollback steps

revert this branch

dbshah1212 · 2021-11-09T13:17:18Z

tap_google_sheets/schema.py

+                        # in the columns dict. The `prior_column_skipped` would be true  when it is the first column of the two
+                        # consecutive empty headers column if true: update the incusion property to `unsupported`
+                        for column in columns:
+                            if column.get('columnSkipped') and not column.get('prior_column_skipped', None):


By default it will return None, do we need to pass None?

By default it will return None, do we need to pass None?

removed None as not needed

…/tap-google-sheets into TDL-14475-make-unsupported-field-for-no-header

savan-chovatiya · 2021-12-02T12:00:41Z

tests/unittests/test_unsupported_fields.py

@@ -0,0 +1,192 @@
+from tap_google_sheets.client import GoogleClient
+import unittest
+from unittest.case import TestCase


We can remove the unnecessary import of TestCase as the test case uses unittest.TestCase.

savan-chovatiya · 2021-12-02T12:01:05Z

tests/unittests/test_unsupported_fields.py

+import unittest
+from unittest.case import TestCase
+from unittest import mock
+from tap_google_sheets import LOGGER, schema


We can remove unused import of LOGGER

savan-chovatiya · 2021-12-02T12:01:22Z

tests/unittests/test_unsupported_fields.py

+        }
+        sheet_json_schema, columns = schema.get_sheet_schema_columns(sheet)
+        self.assertEqual(sheet_json_schema, expected_schema) # test the schema is as expected
+        self.assertEqual(columns, expected_columns) # tst if the columns is as expected


Suggested change

self.assertEqual(columns, expected_columns) # tst if the columns is as expected

self.assertEqual(columns, expected_columns) # test if the columns is as expected

tap_google_sheets/schema.py

KrisPersonal

Need code walk through of this change

namrata270998 · 2021-12-09T05:52:43Z

Need code walk through of this change

Provided code walkthrough

kspeer825

Approved with comment.

kspeer825 · 2021-12-09T12:57:33Z

tests/test_google_sheets_discovery.py

-                # BUG_TDL-14475 | https://jira.talendforge.org/browse/TDL-14475
-                failing_streams = {'sadsheet-column-skip-bug', 'Item Master'}  # BUG_TDL-14475
-                if stream not in failing_streams:  # BUG_TDL-14475
+                # The card TDL-14475 was only about adding unsupported 


I reviewed this ticket. The original writeup for these bugs did not properly cover the two types of issues that we found here. The original assertion is correct in that both streams should be marked as unsupported, but for different reasons. By removing the bug mark and altering the assertion comment the bug is being covering up for the Item Master stream. In the future please raise discrepancies between tickets and tests so that issues do not get missed. I will card this out separately.

Sure @kspeer825 , we will take care from the next time. Also, we have started looking into the card that you have created.

…-field-for-no-header

* type of emailaddress corrected (#53) * Tdl 16079 check best practices (#51) * Initial commit for best practice update * updated setup.py and start_date test * Updated test cases * Updated start_date test case * Updated start_date test case * Updated comment * Revert back test case changes * Added new line * Tdl 14376 pagination failure (#50) * Initial commit for pagination failer * Fixed pagination test cases * Added comments * Added detail comment into the code * Removed unnecessary comment * Removed unnecessary assertion * Removed extra comment * added comment for bug (#49) * TDL-14475 added unsupported feature and unittests (#47) * added unsupported feature and unittests * added code comments * fixed indent * fixed indentation * resolved a bug of writing md when 2 consecutive empty headers * updated the logic for consecutive empty headers * rsolved comments * added test case for consecutive empty headers * added comments * resolved circleci errors * resolved comments Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasystems.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * TDL-14397-Add skipped log when first row is empty (#46) * added logger message and unittests * added code comments * changed the logger message and logic * resolved comments Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasystems.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * TDL-16054 added code comments (#52) * TDL-16054 added code comments * rsolved comments Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasystems.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> * Tdl 16280 implement request timeout (#54) * TDL-16280 added request timeout * TDL-16280: Added factor 3 to add more wait time between 2 calls * TDL-16280: Updated Connection error as it wasn't defined. * added backoff for access token * updated readme * updated request timeout and added jitter * added comment for jitter * added code coverage * added testcase for connection error * addd request timeout in config example * updated the json example * removed the client initialization outside with Co-authored-by: dbshah1212 <dhruvin.shah@crestdatasys.com> Co-authored-by: prijendev <prijen.khokhani@crestdatasys.com> Co-authored-by: namrata270998 <75604662+namrata270998@users.noreply.github.com> Co-authored-by: namrata270998 <namrata.brahmbhatt@crestdatasystems.com> Co-authored-by: dbshah1212 <dhruvin.shah@crestdatasys.com>

added unsupported feature and unittests

407e2bc

namrata270998 requested a review from dbshah1212 October 27, 2021 05:27

added code comments

3b3242b

namrata270998 changed the base branch from master to TDL-15623-Crest-Master November 1, 2021 07:50

namrata270998 and others added 3 commits November 1, 2021 13:27

fixed indent

1cdaf56

fixed indentation

391ac96

resolved a bug of writing md when 2 consecutive empty headers

06f2037

namrata270998 changed the title ~~added unsupported feature and unittests~~ TDL-14475 added unsupported feature and unittests Nov 8, 2021

updated the logic for consecutive empty headers

fa6c6fb

dbshah1212 reviewed Nov 9, 2021

View reviewed changes

namrata270998 added 5 commits November 9, 2021 13:19

rsolved comments

2703a4b

added test case for consecutive empty headers

913393a

added comments

70b2fc3

Merge branch 'TDL-15623-Crest-Master' of https://github.com/singer-io…

7cbbc05

…/tap-google-sheets into TDL-14475-make-unsupported-field-for-no-header

resolved circleci errors

01af68c

savan-chovatiya reviewed Dec 2, 2021

View reviewed changes

resolved comments

8daf191

karanpanchal-crest reviewed Dec 8, 2021

View reviewed changes

tap_google_sheets/schema.py Show resolved Hide resolved

umangagrawal-crest requested a review from karanpanchal-crest December 8, 2021 07:14

karanpanchal-crest approved these changes Dec 8, 2021

View reviewed changes

namrata270998 requested review from kspeer825, KrisPersonal and manand31 December 8, 2021 09:45

KrisPersonal requested changes Dec 8, 2021

View reviewed changes

namrata270998 requested a review from KrisPersonal December 9, 2021 05:50

savan-chovatiya approved these changes Dec 9, 2021

View reviewed changes

kspeer825 approved these changes Dec 9, 2021

View reviewed changes

KrisPersonal approved these changes Dec 9, 2021

View reviewed changes

Merge branch 'TDL-15623-Crest-Master' into TDL-14475-make-unsupported…

0fd765c

…-field-for-no-header

prijendev merged commit d76ae96 into TDL-15623-Crest-Master Dec 13, 2021

prijendev mentioned this pull request Dec 13, 2021

Tdl 15623 crest master #57

Merged

namrata270998 deleted the TDL-14475-make-unsupported-field-for-no-header branch June 2, 2022 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-14475 added unsupported feature and unittests #47

TDL-14475 added unsupported feature and unittests #47

namrata270998 commented Oct 27, 2021

dbshah1212 Nov 9, 2021

namrata270998 Nov 9, 2021

savan-chovatiya Dec 2, 2021

namrata270998 Dec 2, 2021

savan-chovatiya Dec 2, 2021

namrata270998 Dec 2, 2021

savan-chovatiya Dec 2, 2021

namrata270998 Dec 2, 2021

KrisPersonal left a comment

namrata270998 commented Dec 9, 2021

kspeer825 left a comment

kspeer825 Dec 9, 2021

namrata270998 Dec 10, 2021

	self.assertEqual(columns, expected_columns) # tst if the columns is as expected
	self.assertEqual(columns, expected_columns) # test if the columns is as expected

TDL-14475 added unsupported feature and unittests #47

TDL-14475 added unsupported feature and unittests #47

Conversation

namrata270998 commented Oct 27, 2021

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KrisPersonal left a comment

Choose a reason for hiding this comment

namrata270998 commented Dec 9, 2021

kspeer825 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment