Add --warn-error-options #6520

MichelleArk · 2023-01-04T23:03:08Z

resolves #6165

Description

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue 🎩

--warn-error

❯ dbt run --select nada
20:29:07  Running with dbt=1.4.0-b1
20:29:07  Found 5 models, 20 tests, 0 snapshots, 0 analyses, 303 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 0 metrics
20:29:07  The selection criterion 'nada' does not match any nodes
20:29:07  
20:29:07  Nothing to do. Try checking your model configs and model specification args

❯ dbt --warn-error run -m nada
20:29:12  Running with dbt=1.4.0-b1
20:29:12  Found 5 models, 20 tests, 0 snapshots, 0 analyses, 303 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 0 metrics
20:29:13  Encountered an error:
Compilation Error
  The selection criterion 'nada' does not match any nodes

Mutually exclusive

❯ dbt --warn-error --warn-error-options '{"include": "all"}' run -m nada
usage: dbt [-h] [--version] [-r RECORD_TIMING_INFO] [-d] [--log-format {text,json,default}] [--no-write-json]
           [--use-colors | --no-use-colors] [--printer-width PRINTER_WIDTH] [--warn-error | --warn-error-options WARN_ERROR_OPTIONS]
           [--no-version-check] [--partial-parse | --no-partial-parse] [--use-experimental-parser] [--no-static-parser]
           [--profiles-dir PROFILES_DIR] [--no-anonymous-usage-stats] [-x] [-q] [--no-print]
           [--cache-selected-only | --no-cache-selected-only]
           {docs,source,init,clean,debug,deps,list,ls,build,snapshot,run,compile,parse,test,seed,run-operation} ...
dbt: error: argument --warn-error-options: not allowed with argument --warn-error

# warn_error: True in user config (profiles.yml)
❯ dbt --warn-error-options '{"include": "all"}' run -m nada
20:32:05  Encountered an error:
warn_error_options: not allowed with argument warn_error
20:32:05  Traceback (most recent call last):
  File "/Users/michelleark/src/dbt-core/core/dbt/main.py", line 135, in main
    results, succeeded = handle_and_check(args)
  File "/Users/michelleark/src/dbt-core/core/dbt/main.py", line 180, in handle_and_check
    flags.set_from_args(parsed, user_config)
  File "/Users/michelleark/src/dbt-core/core/dbt/flags.py", line 153, in set_from_args
    _check_mutually_exclusive(["WARN_ERROR", "WARN_ERROR_OPTIONS"], args, user_config)
  File "/Users/michelleark/src/dbt-core/core/dbt/flags.py", line 201, in _check_mutually_exclusive
    raise ValueError(f"{flag.lower()}: not allowed with argument {set_flag.lower()}")
ValueError: warn_error_options: not allowed with argument warn_error

--warn-error-options

❯ dbt --warn-error-options '{"include": "all"}' run -m nada
20:32:56  Running with dbt=1.4.0-b1
20:32:57  Found 5 models, 20 tests, 0 snapshots, 0 analyses, 303 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 0 metrics
20:32:57  Encountered an error:
Compilation Error
  The selection criterion 'nada' does not match any nodes

❯ dbt --warn-error-options '{"include": "all", "exclude": [NoNodesForSelectionCriteria,NothingToDo]}' run -m nada
20:36:53  Running with dbt=1.4.0-b1
20:36:53  Found 5 models, 20 tests, 0 snapshots, 0 analyses, 303 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 0 metrics
20:36:53  The selection criterion 'nada' does not match any nodes
20:36:53  
20:36:53  Nothing to do. Try checking your model configs and model specification args

❯ dbt --warn-error-options '{"include": [NoNodesForSelectionCriteria]}' run -m nada
20:37:21  Running with dbt=1.4.0-b1
20:37:22  Found 5 models, 20 tests, 0 snapshots, 0 analyses, 303 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 0 metrics
20:37:22  Encountered an error:
Compilation Error
  The selection criterion 'nada' does not match any nodes

This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR: warn_error_options docs docs.getdbt.com#2691
I have run changie new to create a changelog entry

github-actions · 2023-01-04T23:03:28Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

jtcohen6

quick first pass at the quick first pass

core/dbt/helper_types.py

core/dbt/cli/params.py

core/dbt/main.py

MichelleArk · 2023-01-10T19:35:49Z

@stu-k - A couple big files were deleted by the API doc generation bot (core/dbt/docs/build/html/_static/_sphinx_javascript_frameworks_compat.js and core/dbt/docs/build/html/_static/jquery-3.6.0.js) - is that expected?

core/dbt/events/types.py

stu-k · 2023-01-10T19:46:01Z

@MichelleArk All the workflow does if it detects changes is run make clean and make html, so if those files were removed as part of either of those operations, I would assume it's sphinx running as expected.

gshank

Very nice. Did those files get deleted because tSphinx was upgraded?

jtcohen6

Tried a bunch of different combinations, all of them looking good!

I did run into a weird issue with yaml loading when warn_error_options is defined in UserConfig (profiles.yml)

core/dbt/helper_types.py

jtcohen6 · 2023-01-10T20:37:24Z

core/dbt/flags.py

+    for flag in group:
+        flag_set_by_user = not _flag_value_from_default(flag, args, user_config)
+        if flag_set_by_user and set_flag:
+            raise ValueError(f"{flag.lower()}: not allowed with argument {set_flag.lower()}")


This could be nicer, in cases where I've set warn_error in UserConfig and I'm also passing the conflicting --warn-error-options flag on the CLI:

ValueError: warn_error_options: not allowed with argument warn_error

Sounds like it will be easier to handle in the new click CLI. It serves its purpose for now, and I'm happy with making it nicer in v1.5+

core/dbt/helper_types.py

MichelleArk · 2023-01-11T00:34:51Z

Addressed comments, resolved some merge conflicts, and re-tested (with dbt clean this time, to avoid having to fix adapter error messages):

The following all worked as expected 🎩

dbt --warn-error clean -> errors because of PackageInstallPathDeprecation
dbt --warn-error-options '{"include": [PackageInstallPathDeprecation]}' clean -> errors
dbt --warn-error-options '{"include": "all"}' clean -> errors
dbt --warn-error-options '{"include": "all", "exclude": [PackageInstallPathDeprecation]}' clean -> no error
dbt --warn-error-options '{"include": "all"}' --warn-error clean -> mutually exclusive error
dbt --warn-error-options '{"include": "all"}' clean (warn_error: True in user_config` -> mutually exclusive error
DBT_WARN_ERROR_OPTIONS='{"include": ["PackageInstallPathDeprecation"]}' dbt clean -> errors

# profiles.yml
config:
  warn_error_options:
    include:
      - PackageInstallPathDeprecation

-> errors

# profiles.yml
config:
  warn_error_options:
    include: 'all'
    exclude: 
      - PackageInstallPathDeprecation

-> does not error!

MichelleArk · 2023-01-11T00:38:43Z

core/dbt/task/runnable.py

@@ -154,7 +154,7 @@ def get_node_selector(self) -> NodeSelector:

    @abstractmethod
    def defer_to_manifest(self, adapter, selected_uids: AbstractSet[str]):
-        raise NotImplementedException(f"defer_to_manifest not implemented for task {type(self)}")
+        raise NotImplementedError(f"defer_to_manifest not implemented for task {type(self)}")


@emmyoop - Noticed this import error via mypy from the latest changes, addressed it here as part of my merge.

thanks. no idea how i missed that.

cla-bot bot added the cla:yes label Jan 4, 2023

jtcohen6 reviewed Jan 5, 2023

View reviewed changes

core/dbt/helper_types.py Show resolved Hide resolved

core/dbt/cli/params.py Outdated Show resolved Hide resolved

core/dbt/main.py Outdated Show resolved Hide resolved

core/dbt/main.py Show resolved Hide resolved

MichelleArk force-pushed the CT-1436/warn_error-include-exclude branch 2 times, most recently from 25c4f55 to 9ed8555 Compare January 9, 2023 22:25

MichelleArk changed the title ~~first pass: --warn-error-options~~ Add --warn-error-options Jan 9, 2023

jtcohen6 mentioned this pull request Jan 10, 2023

New global config: --warn-error-options dbt-labs/docs.getdbt.com#2687

Closed

1 task

MichelleArk marked this pull request as ready for review January 10, 2023 19:30

MichelleArk requested a review from a team January 10, 2023 19:30

MichelleArk requested review from a team as code owners January 10, 2023 19:30

MichelleArk requested review from stu-k and peterallenwebb January 10, 2023 19:30

MichelleArk requested review from gshank and emmyoop January 10, 2023 19:35

MichelleArk commented Jan 10, 2023

View reviewed changes

core/dbt/events/types.py Show resolved Hide resolved

gshank approved these changes Jan 10, 2023

View reviewed changes

jtcohen6 reviewed Jan 10, 2023

View reviewed changes

core/dbt/helper_types.py Outdated Show resolved Hide resolved

MichelleArk force-pushed the CT-1436/warn_error-include-exclude branch from 9c67e2d to dacaead Compare January 10, 2023 23:18

MichelleArk and others added 8 commits January 10, 2023 18:55

first pass: --warn-error-options

5aa0424

Add generated CLI API docs

fa54a85

changelog entry

0eaec64

linting

c4f8048

s/--model/--select

bd52b12

flags - _check_mutually_exclusive

b065872

unit tests for --warn-error-options

59f8464

merge

f6e6295

MichelleArk added 2 commits January 10, 2023 18:55

fix test_version USE_COLORS test

adf4aa1

more defensive WarnErrorOptions.from_yaml_string

773418d

jtcohen6 approved these changes Jan 10, 2023

View reviewed changes

case insensitive non-boolean flags

0c534d3

MichelleArk force-pushed the CT-1436/warn_error-include-exclude branch from dacaead to 0c534d3 Compare January 11, 2023 00:19

MichelleArk commented Jan 11, 2023

View reviewed changes

MichelleArk merged commit 7a61602 into main Jan 11, 2023

MichelleArk deleted the CT-1436/warn_error-include-exclude branch January 11, 2023 00:43

MichelleArk mentioned this pull request Jan 11, 2023

[CT-1798] Add mutually exclusive handling for warn_error_options and warn_error params in Click CLI #6579

Closed

3 tasks

emmyoop mentioned this pull request Jan 12, 2023

finish message rename in types.proto #6594

Merged

6 tasks

This was referenced Jan 17, 2023

[CT-1832] [Feature] Data classes for entity dimension information #6629

Closed

[CT-1830] [Epic] Adding entities into dbt-core #6626

Closed

MichelleArk mentioned this pull request Jan 26, 2023

warn_error_options, warn_error mutual exclusivity with click #6762

Closed

6 tasks

iknox-fa mentioned this pull request Jan 30, 2023

testing merge strat - do not merge ns/add to committer list #6783

Closed

6 tasks

MattTriano mentioned this pull request Apr 5, 2023

Configure dbt's warn-error parameter to ignore some warnings MattTriano/analytics_data_where_house#82

Closed

jeremyyeo mentioned this pull request Apr 23, 2024

[Bug] WARN_ERROR_OPTIONS with include all and exclude specific warning not working as expected #10010

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --warn-error-options #6520

Add --warn-error-options #6520

MichelleArk commented Jan 4, 2023 •

edited

Loading

github-actions bot commented Jan 4, 2023

jtcohen6 left a comment

MichelleArk commented Jan 10, 2023

stu-k commented Jan 10, 2023

gshank left a comment

jtcohen6 left a comment

jtcohen6 Jan 10, 2023

MichelleArk commented Jan 11, 2023

MichelleArk Jan 11, 2023

emmyoop Jan 11, 2023

Add --warn-error-options #6520

Add --warn-error-options #6520

Conversation

MichelleArk commented Jan 4, 2023 • edited Loading

Description

Checklist

github-actions bot commented Jan 4, 2023

jtcohen6 left a comment

Choose a reason for hiding this comment

MichelleArk commented Jan 10, 2023

stu-k commented Jan 10, 2023

gshank left a comment

Choose a reason for hiding this comment

jtcohen6 left a comment

Choose a reason for hiding this comment

jtcohen6 Jan 10, 2023

Choose a reason for hiding this comment

MichelleArk commented Jan 11, 2023

MichelleArk Jan 11, 2023

Choose a reason for hiding this comment

emmyoop Jan 11, 2023

Choose a reason for hiding this comment

MichelleArk commented Jan 4, 2023 •

edited

Loading