Guidance for typing large numbers in streams' JSON Schemas

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/351

Originally created by @aaronsteers on 2022-03-25 15:23:06

---

We have run into a few cases recently where very large integers or very large currency amounts were being stored as strings in the source API, and the question has come up in a few forms:

Should we type this to an int or numeric type?

How can we avoid overflow downstream if the contents are not expected to fit into a 32-bit structure?

Example from tap-github

MeltanoLabs/tap-github#93 (comment)

@Ry-DS It looks like the API is indeed returning them as strings, but they are clearly integers... Can we force the conversion? Do you get an error?

-> https://api.github.com/repos/facebook/react/events

It makes sense that it's a string as the number is quite large (20 billion) which definitely won't fit in a 32bit signed integer (max 2 billion). I think it's the right move to keep it as strings (it would error out otherwise).

[MeltyBot]

View 2 previous comments from the original issue on GitLab

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.