Add missing endpoints available in singer-io/tap-github #93

Ry-DS · 2022-03-10T08:49:38Z

Closes #83

TODO:

Streams:

This PR involved a lot of json to propertylist conversions. Here are some regex conversion expressions to make things easier:

Automatic string type conversion: "(\w+)": .+" th.Property("$1", th.StringType)
Integer type conversion: "(\w+)": \d+ th.Property("$1", th.IntegerType)
Boolean type conversion: "(\w+)": (true|false)+ th.Property("$1", th.BooleanType)
Object type conversion "(\w+)": \{(.|\n+?)\}, th.Property("$1", th.ObjectType($2),),

Maybe we could standardise the above regex queries and document them in the README?

Discussion points:

We could use the graphql api for Pull Request Reviews if we want to extend our quota potential. Thoughts? Went with the path of least resistance for now (REST) but if you do see huge savings by using graphql let me know.
Not sure how to fix the test as it now needs authentication to work (collaborators needs a github token to work)
- https://github.com/MeltanoLabs/tap-github/runs/5497860148?check_suite_focus=true#step:10:17131
- singer_sdk.exceptions.FatalAPIError: 403 Client Error: Forbidden for path: /repos/octocat/Hello-World/collaborators

ericboucher · 2022-03-11T16:55:36Z

@Ry-DS make sure to have a look @edgarrmondragon's PR #31 to avoid duplication

Ry-DS · 2022-03-13T03:24:41Z

Conversions using regex patterns in description. Some need manual fixes such as arrays, nested objects, and null as the regex doesn't match those.

Click to expand

// projects
{
 th.Property("owner_url", th.StringType),
 th.Property("url", th.StringType),
 th.Property("html_url", th.StringType),
 th.Property("columns_url", th.StringType),
 th.Property("id", th.IntegerType),
 th.Property("node_id", th.StringType),
 th.Property("name", th.StringType),
 th.Property("body", th.StringType),
 th.Property("number", th.IntegerType),
 th.Property("state", th.StringType),
 th.Property("creator", th.ObjectType(
   th.Property("login", th.StringType),
   th.Property("id", th.IntegerType),
   th.Property("node_id", th.StringType),
   th.Property("avatar_url", th.StringType),
   th.Property("gravatar_id", th.StringType),
   th.Property("url", th.StringType),
   th.Property("html_url", th.StringType),
   th.Property("followers_url", th.StringType),
   th.Property("following_url", th.StringType),
   th.Property("gists_url", th.StringType),
   th.Property("starred_url", th.StringType),
   th.Property("subscriptions_url", th.StringType),
   th.Property("organizations_url", th.StringType),
   th.Property("repos_url", th.StringType),
   th.Property("events_url", th.StringType),
   th.Property("received_events_url", th.StringType),
   th.Property("type", th.StringType),
   th.Property("site_admin", th.BooleanType)
 ),),
 th.Property("created_at", th.StringType),
 th.Property("updated_at", th.StringType)
}

// project cards
{
 th.Property("url", th.StringType),
 th.Property("id", th.IntegerType),
 th.Property("node_id", th.StringType),
 th.Property("note", th.StringType),
 th.Property("creator", th.ObjectType(
   th.Property("login", th.StringType),
   th.Property("id", th.IntegerType),
   th.Property("node_id", th.StringType),
   th.Property("avatar_url", th.StringType),
   th.Property("gravatar_id", th.StringType),
   th.Property("url", th.StringType),
   th.Property("html_url", th.StringType),
   th.Property("followers_url", th.StringType),
   th.Property("following_url", th.StringType),
   th.Property("gists_url", th.StringType),
   th.Property("starred_url", th.StringType),
   th.Property("subscriptions_url", th.StringType),
   th.Property("organizations_url", th.StringType),
   th.Property("repos_url", th.StringType),
   th.Property("events_url", th.StringType),
   th.Property("received_events_url", th.StringType),
   th.Property("type", th.StringType),
   th.Property("site_admin", th.BooleanType)
 ),),
 th.Property("created_at", th.StringType),
 th.Property("updated_at", th.StringType),
 th.Property("archived", th.BooleanType),
 th.Property("column_url", th.StringType),
 th.Property("content_url", th.StringType),
 th.Property("project_url", th.StringType)
}
// project columns
{
 th.Property("url", th.StringType),
 th.Property("project_url", th.StringType),
 th.Property("cards_url", th.StringType),
 th.Property("id", th.IntegerType),
 th.Property("node_id", th.StringType),
 th.Property("name", th.StringType),
 th.Property("created_at", th.StringType),
 th.Property("updated_at", th.StringType)
}
// pr commits
{
 th.Property("url", th.StringType),
 th.Property("sha", th.StringType),
 th.Property("node_id", th.StringType),
 th.Property("html_url", th.StringType),
 th.Property("comments_url", th.StringType),
 th.Property("commit", th.ObjectType(
   th.Property("url", th.StringType),
   th.Property("author", th.ObjectType(
     th.Property("name", th.StringType),
     th.Property("email", th.StringType),
     th.Property("date", th.StringType)
   ),),
   th.Property("committer", th.ObjectType(
     th.Property("name", th.StringType),
     th.Property("email", th.StringType),
     th.Property("date", th.StringType)
   ),),
   th.Property("message", th.StringType),
   th.Property("tree", th.ObjectType(
     th.Property("url", th.StringType),
     th.Property("sha", th.StringType)
   ),),
   th.Property("comment_count", th.IntegerType),
   th.Property("verification", th.ObjectType(
     th.Property("verified", th.BooleanType),
     th.Property("reason", th.StringType),
     "signature": null,
     "payload": null
   ),),
 ),),
 th.Property("author", th.ObjectType(
   th.Property("login", th.StringType),
   th.Property("id", th.IntegerType),
   th.Property("node_id", th.StringType),
   th.Property("avatar_url", th.StringType),
   th.Property("gravatar_id", th.StringType),
   th.Property("url", th.StringType),
   th.Property("html_url", th.StringType),
   th.Property("followers_url", th.StringType),
   th.Property("following_url", th.StringType),
   th.Property("gists_url", th.StringType),
   th.Property("starred_url", th.StringType),
   th.Property("subscriptions_url", th.StringType),
   th.Property("organizations_url", th.StringType),
   th.Property("repos_url", th.StringType),
   th.Property("events_url", th.StringType),
   th.Property("received_events_url", th.StringType),
   th.Property("type", th.StringType),
   th.Property("site_admin", th.BooleanType)
 ),),
 th.Property("committer", th.ObjectType(
   th.Property("login", th.StringType),
   th.Property("id", th.IntegerType),
   th.Property("node_id", th.StringType),
   th.Property("avatar_url", th.StringType),
   th.Property("gravatar_id", th.StringType),
   th.Property("url", th.StringType),
   th.Property("html_url", th.StringType),
   th.Property("followers_url", th.StringType),
   th.Property("following_url", th.StringType),
   th.Property("gists_url", th.StringType),
   th.Property("starred_url", th.StringType),
   th.Property("subscriptions_url", th.StringType),
   th.Property("organizations_url", th.StringType),
   th.Property("repos_url", th.StringType),
   th.Property("events_url", th.StringType),
   th.Property("received_events_url", th.StringType),
   th.Property("type", th.StringType),
   th.Property("site_admin", th.BooleanType)
 ),),
 "parents": [
   {
     th.Property("url", th.StringType),
     th.Property("sha", th.StringType)
   }
 ]
}


// releases
{
 th.Property("url", th.StringType),
 th.Property("html_url", th.StringType),
 th.Property("assets_url", th.StringType),
 th.Property("upload_url", th.StringType),
 th.Property("tarball_url", th.StringType),
 th.Property("zipball_url", th.StringType),
 th.Property("id", th.IntegerType),
 th.Property("node_id", th.StringType),
 th.Property("tag_name", th.StringType),
 th.Property("target_commitish", th.StringType),
 th.Property("name", th.StringType),
 th.Property("body", th.StringType),
 th.Property("draft", th.BooleanType),
 th.Property("prerelease", th.BooleanType),
 th.Property("created_at", th.StringType),
 th.Property("published_at", th.StringType),
 th.Property("author", th.ObjectType(
   th.Property("login", th.StringType),
   th.Property("id", th.IntegerType),
   th.Property("node_id", th.StringType),
   th.Property("avatar_url", th.StringType),
   th.Property("gravatar_id", th.StringType),
   th.Property("url", th.StringType),
   th.Property("html_url", th.StringType),
   th.Property("followers_url", th.StringType),
   th.Property("following_url", th.StringType),
   th.Property("gists_url", th.StringType),
   th.Property("starred_url", th.StringType),
   th.Property("subscriptions_url", th.StringType),
   th.Property("organizations_url", th.StringType),
   th.Property("repos_url", th.StringType),
   th.Property("events_url", th.StringType),
   th.Property("received_events_url", th.StringType),
   th.Property("type", th.StringType),
   th.Property("site_admin", th.BooleanType)
 ),),
 "assets": [
   {
     th.Property("url", th.StringType),
     th.Property("browser_download_url", th.StringType),
     th.Property("id", th.IntegerType),
     th.Property("node_id", th.StringType),
     th.Property("name", th.StringType),
     th.Property("label", th.StringType),
     th.Property("state", th.StringType),
     th.Property("content_type", th.StringType),
     th.Property("size", th.IntegerType),
     th.Property("download_count", th.IntegerType),
     th.Property("created_at", th.StringType),
     th.Property("updated_at", th.StringType),
     th.Property("uploader", th.ObjectType(
       th.Property("login", th.StringType),
       th.Property("id", th.IntegerType),
       th.Property("node_id", th.StringType),
       th.Property("avatar_url", th.StringType),
       th.Property("gravatar_id", th.StringType),
       th.Property("url", th.StringType),
       th.Property("html_url", th.StringType),
       th.Property("followers_url", th.StringType),
       th.Property("following_url", th.StringType),
       th.Property("gists_url", th.StringType),
       th.Property("starred_url", th.StringType),
       th.Property("subscriptions_url", th.StringType),
       th.Property("organizations_url", th.StringType),
       th.Property("repos_url", th.StringType),
       th.Property("events_url", th.StringType),
       th.Property("received_events_url", th.StringType),
       th.Property("type", th.StringType),
       th.Property("site_admin", th.BooleanType)
     ),),
   }
 ]
}
// teams
{
 th.Property("id", th.IntegerType),
 th.Property("node_id", th.StringType),
 th.Property("url", th.StringType),
 th.Property("html_url", th.StringType),
 th.Property("name", th.StringType),
 th.Property("slug", th.StringType),
 th.Property("description", th.StringType),
 th.Property("privacy", th.StringType),
 th.Property("permission", th.StringType),
 th.Property("members_url", th.StringType),
 th.Property("repositories_url", th.StringType),
 "parent": null
}

// team members
{
 th.Property("login", th.StringType),
 th.Property("id", th.IntegerType),
 th.Property("node_id", th.StringType),
 th.Property("avatar_url", th.StringType),
 th.Property("gravatar_id", th.StringType),
 th.Property("url", th.StringType),
 th.Property("html_url", th.StringType),
 th.Property("followers_url", th.StringType),
 th.Property("following_url", th.StringType),
 th.Property("gists_url", th.StringType),
 th.Property("starred_url", th.StringType),
 th.Property("subscriptions_url", th.StringType),
 th.Property("organizations_url", th.StringType),
 th.Property("repos_url", th.StringType),
 th.Property("events_url", th.StringType),
 th.Property("received_events_url", th.StringType),
 th.Property("type", th.StringType),
 th.Property("site_admin", th.BooleanType)
}

// team roles
{
th.Property("url", th.StringType),
th.Property("role", th.StringType),
th.Property("state", th.StringType)
}

tap_github/repository_streams.py

todo_streams.py

tap_github/repository_streams.py

tap_github/streams.py

laurentS

That was a lot! I think there's not much missing, but some changes are probably required for the tap to work.
I've only written some of the comments once to avoid repeating the same thing over and over, but they might apply to multiple streams.

.github/workflows/test_tap.yml

tap_github/organization_streams.py

tap_github/repository_streams.py

tap_github/streams.py

tap_github/tests/test_tap.py

aaronsteers · 2022-03-25T15:25:23Z

@Ry-DS It looks like the API is indeed returning them as strings, but they are clearly integers... Can we force the conversion? Do you get an error?
-> https://api.github.com/repos/facebook/react/events

It makes sense that it's a string as the number is quite large (20 billion) which definitely won't fit in a 32bit signed integer (max 2 billion). I think it's the right move to keep it as strings (it would error out otherwise).

I've logged an issue in the SDK to keep this discussion going. For now, I agree it sounds like preserving as a string type is the "safe" option.

https://gitlab.com/meltano/sdk/-/issues/351

tap_github/tests/fixtures.py

tap_github/tests/test_tap.py

…sing-streams

laurentS · 2022-03-28T08:37:36Z

@Ry-DS It looks like the API is indeed returning them as strings, but they are clearly integers... Can we force the conversion? Do you get an error?
-> https://api.github.com/repos/facebook/react/events

It makes sense that it's a string as the number is quite large (20 billion) which definitely won't fit in a 32bit signed integer (max 2 billion). I think it's the right move to keep it as strings (it would error out otherwise).

I've logged an issue in the SDK to keep this discussion going. For now, I agree it sounds like preserving as a string type is the "safe" option.

https://gitlab.com/meltano/sdk/-/issues/351

In our use-case, we pipe the output from this tap into target-postgres which actually converts integer into bigint which is an 8 byte integer in pg, so ranging way beyond 2 billion.
I don't know if there is a standard conversion across taps/targets...

# Conflicts: # tap_github/repository_streams.py # tap_github/tap.py

laurentS · 2022-04-05T06:55:18Z

There's something very weird with pytest, in the 3.10 environment, it says it's running python 3.8.10:

I suspect it came from the cached environment, despite the key being per python version, because I can't think of another reason for this to happen.

tap_github/tests/test_tap.py

Ry-DS added 6 commits March 10, 2022 19:46

Add assignees stream

214bbbf

Fix issues with assignee stream

d2e10df

Add collaborators stream

08aa7e7

Add review comments and reviews stream

231596b

Fix review comments stream and use repo parent instead

4d967cf

Fix mypy issue

e3b01bd

Ry-DS marked this pull request as ready for review March 10, 2022 15:10

ericboucher marked this pull request as draft March 11, 2022 16:53

Ry-DS added 4 commits March 12, 2022 22:13

Fix tests

f2d7f53

add milestone and commit comment streams

7f17a65

Fix mypy

313b742

Fix tests

9ea76f5

Ry-DS added 7 commits March 13, 2022 14:33

commit wip todo streams

2b66a73

fix formatting

b27871d

[ci skip] format todo file and fix arraytype usage

ff14da1

[ci skip] more regex magic to convert everything to classes

f146c2a

Add paths [ci skip]

89fb408

Move all streams to main file

d0352c7

Add replication keys

fc70b2a

ericboucher reviewed Mar 13, 2022

View reviewed changes

tap_github/repository_streams.py Outdated Show resolved Hide resolved

ericboucher reviewed Mar 13, 2022

View reviewed changes

todo_streams.py Outdated Show resolved Hide resolved

Ry-DS added 7 commits March 14, 2022 12:46

fix tests (change type to datetime)

7730b86

introduce streams enum

f68c260

Fix up organization stream

3132b36

Reverse order of testing versions

cf7c467

remove unsupported types from class

9258fd3

Fix format

38e41a0

Try use capital types to pass ci

c3d621e

ericboucher reviewed Mar 22, 2022

View reviewed changes

tap_github/repository_streams.py Outdated Show resolved Hide resolved

ericboucher reviewed Mar 22, 2022

View reviewed changes

tap_github/streams.py Show resolved Hide resolved

ericboucher approved these changes Mar 22, 2022

View reviewed changes

laurentS requested changes Mar 23, 2022

View reviewed changes

Ry-DS added 5 commits March 25, 2022 12:34

update ignore_parent_replication and remove unneeded import

6b6d934

Simple comment [ci skip]

ef162de

Work on comments [ci skip]

47e42d9

Work on comments [ci skip]

91bbcd3

Fix mistyped stuff (good catch Laurent) and more comment addressing

e6298e0

Ry-DS requested a review from laurentS March 25, 2022 11:04

aaronsteers reviewed Mar 25, 2022

View reviewed changes

tap_github/tests/fixtures.py Outdated Show resolved Hide resolved

tap_github/tests/test_tap.py Outdated Show resolved Hide resolved

Ry-DS added 4 commits March 28, 2022 00:38

Update fixture comment [ci skip]

643c99b

Merge remote-tracking branch 'oviohub/83-missing-streams' into 83-mis…

45a2981

…sing-streams

Add bunch of meltano lab sample projects

3ff49ec

update state partitioning keys

0b3f52f

ericboucher mentioned this pull request Mar 28, 2022

Workflow streams #31

Merged

Ry-DS added 2 commits March 31, 2022 14:02

Merge branch 'main' into 83-missing-streams

36801e5

# Conflicts: # tap_github/repository_streams.py # tap_github/tap.py

Fix merge

0ec1d16

laurentS approved these changes Apr 5, 2022

View reviewed changes

laurentS mentioned this pull request Apr 5, 2022

Update dependencies #101

Merged

Add ORG_LEVEL_TOKEN to be used only for specific streams

36136ed

laurentS reviewed Apr 6, 2022

View reviewed changes

tap_github/tests/test_tap.py Show resolved Hide resolved

Add docstring to alternative_sync_chidren

202845b

edgarrmondragon merged commit 96c5ad0 into MeltanoLabs:main Apr 7, 2022

aaronsteers mentioned this pull request Apr 7, 2022

Need to ignore "code duplication" failure in SonarCloud MeltanoLabs/Meta#13

Open

ericboucher mentioned this pull request Apr 12, 2022

Factor out some common objects in the schemas #98

Merged

MeltyBot mentioned this pull request May 30, 2022

Guidance for typing large numbers in streams' JSON Schemas meltano/sdk#349

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missing endpoints available in singer-io/tap-github #93

Add missing endpoints available in singer-io/tap-github #93

Ry-DS commented Mar 10, 2022 •

edited by ericboucher

Loading

ericboucher commented Mar 11, 2022

Ry-DS commented Mar 13, 2022 •

edited

Loading

laurentS left a comment

aaronsteers commented Mar 25, 2022 •

edited

Loading

laurentS commented Mar 28, 2022

laurentS commented Apr 5, 2022

Add missing endpoints available in singer-io/tap-github #93

Add missing endpoints available in singer-io/tap-github #93

Conversation

Ry-DS commented Mar 10, 2022 • edited by ericboucher Loading

Closes #83

ericboucher commented Mar 11, 2022

Ry-DS commented Mar 13, 2022 • edited Loading

laurentS left a comment

Choose a reason for hiding this comment

aaronsteers commented Mar 25, 2022 • edited Loading

laurentS commented Mar 28, 2022

laurentS commented Apr 5, 2022

Ry-DS commented Mar 10, 2022 •

edited by ericboucher

Loading

Ry-DS commented Mar 13, 2022 •

edited

Loading

aaronsteers commented Mar 25, 2022 •

edited

Loading