-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* changes to tap entrypoint to work with rest of tap * changes to discover.py to work with rest of tap * changes to schema.py to work with rest of the tap * changes to sync.py to work with rest of the tap * updated README links to documentation * changes to client.py: * actually raise 429 error * pylint fixes * initial working version of streams.py * adding pagination to streams.py * adding links object as part of client return for pagination changes * adjusting indentation to match PEP8: * __init__.py * client.py * sync.py * removing extraneous keys and related functions from discover.py * changes to streams.py: * indentation fixes to match PEP8 * unpacking result of client.get with `records, _` * adding tap-tester base.py * adding tap-tester sync canary * adding tap-tester discovery * adding tap-tester start date * adding tap-tester automated fields * changes to setup.py: * bumping singer-python * adding dev requirements * adding circleci config * adding tap-tester pagination * fix for onetimes stream not supporting cursor based pagination * updating cirlce config * adding additional assertions per PR feedback * add additional streams to pagination test * switch customers stream to page based * Link header was not honoring original call params and duplicating records * Make pylint happy * Update tap-tester invocation * pylint fixes * remove extraneous string interpolation * remove extraneous class attributes for shop stream * fixes to start_date test: * adjust start_date_2 to work with data * add stream replication methods for tests * modify/add assertions based on stream replication methods and data * adding assertion for valid-replication-keys to discover test Co-authored-by: Andy Lu <andy@stitchdata.com>
- Loading branch information
1 parent
5ffce66
commit 839ebb3
Showing
15 changed files
with
1,386 additions
and
599 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
version: 2 | ||
jobs: | ||
build: | ||
docker: | ||
- image: 218546966473.dkr.ecr.us-east-1.amazonaws.com/circle-ci:stitch-tap-tester | ||
steps: | ||
- checkout | ||
- run: | ||
name: 'Setup virtual env' | ||
command: | | ||
python3 -mvenv /usr/local/share/virtualenvs/tap-recharge | ||
source /usr/local/share/virtualenvs/tap-recharge/bin/activate | ||
pip install .[dev] | ||
- run: | ||
name: 'pylint' | ||
command: | | ||
source /usr/local/share/virtualenvs/tap-recharge/bin/activate | ||
pylint tap_recharge --disable 'missing-module-docstring,missing-function-docstring,missing-class-docstring,no-else-raise,raise-missing-from,inconsistent-return-statements' | ||
- run: | ||
when: always | ||
name: 'Integration Tests' | ||
command: | | ||
aws s3 cp s3://com-stitchdata-dev-deployment-assets/environments/tap-tester/tap_tester_sandbox dev_env.sh | ||
source dev_env.sh | ||
source /usr/local/share/virtualenvs/tap-tester/bin/activate | ||
run-test --tap=tap-recharge tests | ||
workflows: | ||
version: 2 | ||
commit: | ||
jobs: | ||
- build: | ||
context: circleci-user | ||
build_daily: | ||
triggers: | ||
- schedule: | ||
cron: "0 14 * * *" | ||
filters: | ||
branches: | ||
only: | ||
- master | ||
jobs: | ||
- build: | ||
context: circleci-user |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,20 +1,24 @@ | ||
from singer.catalog import Catalog, CatalogEntry, Schema | ||
from tap_recharge.schema import get_schemas, STREAMS | ||
from singer.catalog import Catalog | ||
from tap_recharge.schema import get_schemas | ||
|
||
|
||
def discover(): | ||
""" | ||
Constructs a singer Catalog object based on the schemas and metadata. | ||
""" | ||
schemas, field_metadata = get_schemas() | ||
catalog = Catalog([]) | ||
streams = [] | ||
|
||
for schema_name, schema in schemas.items(): | ||
schema_meta = field_metadata[schema_name] | ||
|
||
for stream_name, schema_dict in schemas.items(): | ||
schema = Schema.from_dict(schema_dict) | ||
mdata = field_metadata[stream_name] | ||
catalog_entry = { | ||
'stream': schema_name, | ||
'tap_stream_id': schema_name, | ||
'schema': schema, | ||
'metadata': schema_meta | ||
} | ||
|
||
catalog.streams.append(CatalogEntry( | ||
stream=stream_name, | ||
tap_stream_id=stream_name, | ||
key_properties=STREAMS[stream_name]['key_properties'], | ||
schema=schema, | ||
metadata=mdata | ||
)) | ||
streams.append(catalog_entry) | ||
|
||
return catalog | ||
return Catalog.from_dict({'streams': streams}) |
Oops, something went wrong.