forked from dbt-labs/tap-trustpilot
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Tap trustpilot intial changes - Discover mode, Sync mode, bulk reques…
…t. (#1) * inital changes: import statements, singer-python version sync change * Initial refactor for the discover mode changes * Moved discover logic code from __init__ file to discover.py file * alpha:0.2.0 release changes * added change logs * format changelog.md * Handled consumer sync logic and Removed unused code * removed version number and change logs * circleci yml syntax error * circleci yml syntax error * tap name error Co-authored-by: Kethan Cherukuri <kcherukuri@LT-LR9JJWXD40.local> Co-authored-by: kethan1122 <kcherukuri@talend.com> Co-authored-by: kethan1122 <105211331+kethan1122@users.noreply.github.com>
- Loading branch information
1 parent
cf3cb98
commit 6fa1dfd
Showing
9 changed files
with
196 additions
and
69 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
version: 2.1 | ||
orbs: | ||
slack: circleci/slack@3.4.2 | ||
|
||
jobs: | ||
build: | ||
docker: | ||
- image: 218546966473.dkr.ecr.us-east-1.amazonaws.com/circle-ci:stitch-tap-tester | ||
steps: | ||
- checkout | ||
- run: | ||
name: 'Setup virtual env' | ||
command: | | ||
python3 -mvenv /usr/local/share/virtualenvs/tap-trustpilot | ||
source /usr/local/share/virtualenvs/tap-trustpilot/bin/activate | ||
pip install -U pip setuptools | ||
pip install .[dev] | ||
- run: | ||
name: 'JSON Validator' | ||
command: | | ||
source /usr/local/share/virtualenvs/tap-tester/bin/activate | ||
stitch-validate-json tap_trustpilot/schemas/*.json | ||
- run: | ||
name: 'pylint' | ||
command: | | ||
source /usr/local/share/virtualenvs/tap-trustpilot/bin/activate | ||
pylint tap_trustpilot --disable C,W,R | ||
workflows: | ||
version: 2 | ||
commit: | ||
jobs: | ||
- build: | ||
context: circleci-user | ||
build_daily: | ||
triggers: | ||
- schedule: | ||
cron: "0 13 * * *" | ||
filters: | ||
branches: | ||
only: | ||
- master | ||
jobs: | ||
- build: | ||
context: circleci-user |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
from singer.catalog import Catalog, CatalogEntry, Schema | ||
from tap_trustpilot import schemas | ||
|
||
def check_credentials_are_authorized(ctx): | ||
ctx.client.auth(ctx.config) | ||
|
||
def discover(ctx): | ||
check_credentials_are_authorized(ctx) | ||
discover_schemas, field_metadata = schemas.get_schemas() | ||
streams = [] | ||
for stream_name, raw_schema in discover_schemas.items(): | ||
schema = Schema.from_dict(raw_schema) | ||
mdata = field_metadata[stream_name] | ||
streams.append( | ||
CatalogEntry( | ||
tap_stream_id=stream_name, | ||
stream=stream_name, | ||
schema=schema, | ||
key_properties=schemas.PK_FIELDS[stream_name], | ||
metadata=mdata | ||
) | ||
) | ||
return Catalog(streams) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.