Tap Validate Option

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/228

Originally created by @vischous on 2021-09-29 15:04:43

---

Summary

Validate tap data schema, particularly useful during development as right now you have to add a target that does this validation ie meltano elt tap-abc target-jsonl jsonl will validate for you. Maybe it should stay like this?

Proposed benefits

1. One less step while doing development, the issue I'm seeing is running production data through a tap I'm writing leads to a bunch of Data in a format that wasn't expected (String is sometimes an Int, or something it's an array of Strings, etc etc).
2. Theoretically you could suggest a new schema for the stream that's causing issues.

Proposal details

A --validate option

Standard validate_json config

Best reasons not to build

• You could offload this to a target
• People may forget to turn this off in production which would have performance implications

Use Cases

• https://meltano.slack.com/archives/C01TCRBBJD7/p1635203695035500
• Found some JSON Issues with live data, there's more left AutoIDM/tap-clickup#46
• https://meltano.slack.com/archives/C01PKLU5D1R/p1632880108194400

[MeltyBot]

View 9 previous comments from the original issue on GitLab

[visch]

  jsonl_test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: "3.8"
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install meltano
        meltano install
    - name: Schema test
      run: |
        meltano run tap-name target-jsonl

Trick I've been using

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Addressed by #1711