Reduce noise from 'ignored properties' warnings

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/386

Originally created by @aaronsteers on 2022-05-18 03:42:17

---

The SDK prints warnings for every field that the API returns but is not mapped to the stream's schema. We can do a couple things to reduce the noisiness of logs.

Consolidate warnings into a single message per stream

Instead of:

2022-05-18T03:26:54.596749Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'closed_by' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.597507Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'milestone' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.598140Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'author' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.598749Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'type' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.599366Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'assignee' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.600108Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'issue_type' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.600725Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'time_stats' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.601362Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'task_completion_status' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.604317Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'blocking_issues_count' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.604790Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property '_links' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.605257Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'references' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.605731Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'severity' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.606134Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'moved_to_id' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.606522Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'service_desk_reply_to' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.606932Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'epic_iid' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.607318Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'epic' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.607710Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'iteration' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.608199Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'health_status' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr
2022-05-18T03:26:54.608593Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=Property 'project_path' was present in the 'issues' stream but not found in catalog schema. Ignoring. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr

We could send a single message:

2022-05-18T03:26:54.608593Z [info     ] time=2022-05-18 03:26:54 name=tap-gitlab level=WARNING message=The following properties were present in the 'issues' stream but not found in catalog schema (ignoring): closed_by, milestones, author, type, assignee, issue_type, time_stats, task_completion_status, blocking_issues_count, _links, references, severity, moved_to_id, service_deks_reply_to, epic_iid, epic, iteration, health_status, project_path. cmd_type=extractor job_id=gitlab-to-postgres name=tap-gitlab run_id=8308d041-475f-4404-be2c-e4012413e93a stdio=stderr

Pytest failures

We could have a (perhaps optional) pytest that throws a hard test failure during development, if extra properties are unmapped and not explicitly excluded.

We could optionally pair this with a new stream class property ignored_properties which would be functionally identical to the developer adding this to post_process():

    for ignored in IGNORED_PROPERTIES:
        result.pop(ignored, None)

[MeltyBot]

View 3 previous comments from the original issue on GitLab

[edgarrmondragon]

Copied from GitLab:

• root

  • singer_sdk/mapper.py:318
  • singer_sdk/mapper.py:376
  • singer_sdk/mapper.py:382
  • singer_sdk/mapper.py:389
  • singer_sdk/mapper.py:461
  • singer_sdk/mapper.py:466
  • singer_sdk/mapper.py:529
  • singer_sdk/mapper.py:551
  • singer_sdk/mapper.py:563
  • singer_sdk/mapper.py:650
  • singer_sdk/streams/rest.py:236
  • singer_sdk/streams/rest.py:494

• <plugin name>, e.g. tap-github

  • singer_sdk/authenticators.py:428
  • singer_sdk/authenticators.py:437
  • singer_sdk/plugin_base.py:99
  • singer_sdk/plugin_base.py:102
  • singer_sdk/plugin_base.py:222
  • singer_sdk/plugin_base.py:227
  • singer_sdk/plugin_base.py:242
  • singer_sdk/streams/graphql.py:73
  • singer_sdk/streams/core.py:157
  • singer_sdk/streams/core.py:161
  • singer_sdk/streams/core.py:776
  • singer_sdk/streams/core.py:779
  • singer_sdk/streams/core.py:782
  • singer_sdk/streams/core.py:972
  • singer_sdk/streams/core.py:1009
  • singer_sdk/io_base.py:79
  • singer_sdk/sinks/core.py:56
  • singer_sdk/sinks/core.py:406
  • singer_sdk/tap_base.py:192
  • singer_sdk/tap_base.py:294
  • singer_sdk/tap_base.py:349
  • singer_sdk/tap_base.py:368
  • singer_sdk/tap_base.py:372
  • singer_sdk/tap_base.py:457
  • singer_sdk/mapper_base.py:132
  • singer_sdk/mapper_base.py:138
  • singer_sdk/target_base.py:163
  • singer_sdk/target_base.py:224
  • singer_sdk/target_base.py:261
  • singer_sdk/target_base.py:266
  • singer_sdk/target_base.py:312
  • singer_sdk/target_base.py:332
  • singer_sdk/target_base.py:340
  • singer_sdk/target_base.py:347
  • singer_sdk/target_base.py:387
  • singer_sdk/target_base.py:453
  • singer_sdk/target_base.py:502
  • singer_sdk/target_base.py:509
  • singer_sdk/helpers/_typing.py:128 (This should be a module logger i.m.o.)
  • singer_sdk/helpers/_typing.py:132 (This should be a module logger i.m.o.)
  • singer_sdk/helpers/_typing.py:179 (This should be a module logger i.m.o.)

• singer_sdk.* module loggers, e.g. singer_sdk.configuration._dict_config

  • singer_sdk/configuration/_dict_config.py:43
  • singer_sdk/configuration/_dict_config.py:50
  • singer_sdk/helpers/_singer.py:176
  • singer_sdk/helpers/_singer.py:185
  • singer_sdk/helpers/_singer.py:198

• sqlconnector

  • singer_sdk/streams/sql.py:287

---

1. We should probably get rid of any root logger usage
2. The tap-github should allow more granular control: e.g. - tap-github.streams.stargazers, tap-github.metrics`, etc. This would allow users to set different log levels for different streams and features.
3. Module-level loggers should be kept where it makes sense as low-level SDK features: singer catalog, configuration parsing, etc.

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.