Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Functional test framework working with Click, dbtRunner #6387

Merged
merged 13 commits into from
Dec 17, 2022
Merged

Functional test framework working with Click, dbtRunner #6387

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
c21df62
first pass: dbt.cli.main.handle_and_check
MichelleArk Dec 5, 2022
8c6dd4b
DBTContext
MichelleArk Dec 7, 2022
a6a5675
DBTContext with invocation_args
MichelleArk Dec 8, 2022
aff425e
merge with Profile, run
MichelleArk Dec 8, 2022
04ec5cb
build self.obj in DBTContext
MichelleArk Dec 8, 2022
1fe76f6
dbtRunner, initialize context in subcommand
MichelleArk Dec 12, 2022
988e40a
dbt.cli.requires - preflight, profile, project
MichelleArk Dec 13, 2022
c5ced63
run functional + adapter tests prior to test/integration
MichelleArk Dec 13, 2022
9d79def
use update_wrapper in requires decorators to preserve dunders
MichelleArk Dec 13, 2022
ad776a3
test_dbt_runner
MichelleArk Dec 13, 2022
a42a43a
changelog entry
MichelleArk Dec 14, 2022
c2a8ef6
merge + remove empty file
MichelleArk Dec 15, 2022
2799bf2
linting
MichelleArk Dec 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/unreleased/Under the Hood-20221214-112048.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Under the Hood
body: functional tests run using click cli through dbtRunner
time: 2022-12-14T11:20:48.521869-05:00
custom:
Author: MichelleArk
Issue: "6096"
PR: "6387"
162 changes: 82 additions & 80 deletions core/dbt/cli/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,19 @@
import inspect # This is temporary for RAT-ing
from copy import copy
from pprint import pformat as pf # This is temporary for RAT-ing
from typing import List, Tuple, Optional

import click
from dbt.adapters.factory import adapter_management
from dbt.cli import params as p
from dbt.cli.flags import Flags
from dbt.cli import requires, params as p
from dbt.config import RuntimeConfig
from dbt.config.runtime import load_project, load_profile
from dbt.events.functions import setup_event_logger
from dbt.profiler import profiler
from dbt.tracking import initialize_from_flags, track_run

from dbt.config.project import Project
from dbt.config.profile import Profile
from dbt.contracts.graph.manifest import Manifest
from dbt.task.clean import CleanTask
from dbt.task.deps import DepsTask
from dbt.task.run import RunTask


# CLI invocation
def cli_runner():
# Alias "list" to "ls"
ls = copy(cli.commands["list"])
Expand All @@ -27,6 +24,31 @@ def cli_runner():
cli()


class dbtUsageException(Exception):
pass


# Programmatic invocation
class dbtRunner:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this first rough version of thin wrapper, for end users who want to programmatically invoke dbt-core via Python API? Thinking:

    # initialize longer-lived dbtRunner
    # by implicitly loading project/profile/etc based on config + cwd
    dbt = dbtRunner()
    
    # submit commands to dbtRunner
    # each command returns
    #   results: List[Result]
    #   success: bool
    results, success = dbt.run(...)
    results, success = dbt.test(...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly!

# initialize longer-lived dbtRunner
# by implicitly loading project/profile/etc based on config + cwd
dbt = dbtRunner()

^ We could load profile/project/etc from defaults at during dbtRunner instantiation but depending on the args in subsequent commands, they may require reloading. Something to consider for a follow-up though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

    results, success = dbt.run(...)
    results, success = dbt.test(...)

this part is tbd I think? current way would be pass in command through args, the main advantage is some CLI args need to be specified before the actual command. So exposing things this way provide best flexibility at the moment.

args = ['run', '--project-dir', 'something']
res, success = dbt.invoke(args)

Copy link
Contributor

@jtcohen6 jtcohen6 Dec 14, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

current way would be pass in command through args

We'll definitely want to support this generic pattern ("given an arbitrary list of CLI arg strings, invoke"). I could also see us wanting to support a more structured approach, where users can build up a dictionary of attributes and pass it into a named command. All of which is to say, we should design this programmatic interface for real, before we commit to it as a public-facing API — and that means sitting down, writing some user stories, talking with community members. All good stuff to do next year :)

the main advantage is some CLI args need to be specified before the actual command

Lukewarm take: This is pretty confusing for end users! We should really seek to support flags in any order (and resolve any resulting ambiguities by actually renaming flags). I know we implemented the new CLI modeling in accordance with the old CLI model — parity is a good goal to shoot for with any refactor — but it would be great to just support both of these, seamlessly and equivalently:

$ dbt --no-use-color run --select one_model
$ dbt run --select one_model --no-use-color

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @iknox-fa

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, that wasn't part of the original plan, but I guess we can see about supporting arbitrary flag ordering-- It's not something we should take on in this PR though. @jtcohen6, can you write up a ticket that we can put into the estimation process.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iknox-fa Totally - opened here: #6497

def __init__(
self, project: Project = None, profile: Profile = None, manifest: Manifest = None
):
self.project = project
self.profile = profile
self.manifest = manifest

def invoke(self, args: List[str]) -> Tuple[Optional[List], bool]:
try:
dbt_ctx = cli.make_context(cli.name, args)
dbt_ctx.obj = {}
dbt_ctx.obj["project"] = self.project
dbt_ctx.obj["profile"] = self.profile
dbt_ctx.obj["manifest"] = self.manifest
return cli.invoke(dbt_ctx)
except (click.NoSuchOption, click.UsageError) as e:
raise dbtUsageException(e.message)


# dbt
@click.group(
context_settings={"help_option_names": ["-h", "--help"]},
Expand Down Expand Up @@ -62,48 +84,11 @@ def cli(ctx, **kwargs):
"""An ELT tool for managing your SQL transformations and data models.
For more documentation on these commands, visit: docs.getdbt.com
"""
# Get primatives
flags = Flags()

# Logging
# N.B. Legacy logger is not supported
setup_event_logger(
flags.LOG_PATH,
flags.LOG_FORMAT,
flags.USE_COLORS,
flags.DEBUG,
)

# Tracking
initialize_from_flags(flags.ANONYMOUS_USAGE_STATS, flags.PROFILES_DIR)
ctx.with_resource(track_run(run_command=ctx.invoked_subcommand))

# Profiling
if flags.RECORD_TIMING_INFO:
ctx.with_resource(profiler(enable=True, outfile=flags.RECORD_TIMING_INFO))

# Adapter management
ctx.with_resource(adapter_management())

# Version info
if flags.VERSION:
if ctx.params["version"]:
click.echo(f"`version` called\n ctx.params: {pf(ctx.params)}")
return

# Profile
profile = load_profile(
flags.PROJECT_DIR, flags.VARS, flags.PROFILE, flags.TARGET, flags.THREADS
)

# Project
project = load_project(flags.PROJECT_DIR, flags.VERSION_CHECK, profile, flags.VARS)

# Context for downstream commands
ctx.obj = {}
ctx.obj["flags"] = flags
ctx.obj["profile"] = profile
ctx.obj["project"] = project


# dbt build
@cli.command("build")
Expand All @@ -126,10 +111,11 @@ def cli(ctx, **kwargs):
@p.threads
@p.vars
@p.version_check
@requires.preflight
def build(ctx, **kwargs):
"""Run all Seeds, Models, Snapshots, and tests in DAG order"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt clean
Expand All @@ -140,18 +126,17 @@ def build(ctx, **kwargs):
@p.project_dir
@p.target
@p.vars
@requires.preflight
@requires.profile
@requires.project
def clean(ctx, **kwargs):
"""Delete all folders in the clean-targets list (usually the dbt_packages and target directories.)"""
flags = Flags()
project = ctx.obj["project"]

task = CleanTask(flags, project)
task = CleanTask(ctx.obj["flags"], ctx.obj["project"])

results = task.run()
success = task.interpret_results(results)
return results, success


# dbt docs
@cli.group()
@click.pass_context
Expand All @@ -176,10 +161,11 @@ def docs(ctx, **kwargs):
@p.threads
@p.vars
@p.version_check
@requires.preflight
def docs_generate(ctx, **kwargs):
"""Generate the documentation website for your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt docs serve
Expand All @@ -192,10 +178,11 @@ def docs_generate(ctx, **kwargs):
@p.project_dir
@p.target
@p.vars
@requires.preflight
def docs_serve(ctx, **kwargs):
"""Serve the documentation website for your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt compile
Expand All @@ -216,10 +203,11 @@ def docs_serve(ctx, **kwargs):
@p.threads
@p.vars
@p.version_check
@requires.preflight
def compile(ctx, **kwargs):
"""Generates executable SQL from source, model, test, and analysis files. Compiled SQL files are written to the target/ directory."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt debug
Expand All @@ -232,10 +220,11 @@ def compile(ctx, **kwargs):
@p.target
@p.vars
@p.version_check
@requires.preflight
def debug(ctx, **kwargs):
"""Show some helpful information about dbt for debugging. Not to be confused with the --debug option which increases verbosity."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt deps
Expand All @@ -246,9 +235,12 @@ def debug(ctx, **kwargs):
@p.project_dir
@p.target
@p.vars
@requires.preflight
@requires.profile
@requires.project
def deps(ctx, **kwargs):
"""Pull the most recent version of the dependencies listed in packages.yml"""
flags = Flags()
flags = ctx.obj["flags"]
project = ctx.obj["project"]

task = DepsTask.from_project(project, flags.VARS)
Expand All @@ -267,10 +259,11 @@ def deps(ctx, **kwargs):
@p.skip_profile_setup
@p.target
@p.vars
@requires.preflight
def init(ctx, **kwargs):
"""Initialize a new DBT project."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt list
Expand All @@ -289,10 +282,11 @@ def init(ctx, **kwargs):
@p.state
@p.target
@p.vars
@requires.preflight
def list(ctx, **kwargs):
"""List the resources in your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt parse
Expand All @@ -308,10 +302,11 @@ def list(ctx, **kwargs):
@p.vars
@p.version_check
@p.write_manifest
@requires.preflight
def parse(ctx, **kwargs):
"""Parses the project and provides information on performance"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt run
Expand All @@ -332,9 +327,11 @@ def parse(ctx, **kwargs):
@p.threads
@p.vars
@p.version_check
@requires.preflight
@requires.profile
@requires.project
def run(ctx, **kwargs):
"""Compile SQL and execute against the current target database."""

config = RuntimeConfig.from_parts(ctx.obj["project"], ctx.obj["profile"], ctx.obj["flags"])
task = RunTask(ctx.obj["flags"], config)

Expand All @@ -352,10 +349,11 @@ def run(ctx, **kwargs):
@p.project_dir
@p.target
@p.vars
@requires.preflight
def run_operation(ctx, **kwargs):
"""Run the named macro with any supplied arguments."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt seed
Expand All @@ -375,10 +373,11 @@ def run_operation(ctx, **kwargs):
@p.threads
@p.vars
@p.version_check
@requires.preflight
def seed(ctx, **kwargs):
"""Load data from csv files into your data warehouse."""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt snapshot
Expand All @@ -395,10 +394,11 @@ def seed(ctx, **kwargs):
@p.target
@p.threads
@p.vars
@requires.preflight
def snapshot(ctx, **kwargs):
"""Execute snapshots defined in your project"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt source
Expand All @@ -422,10 +422,11 @@ def source(ctx, **kwargs):
@p.target
@p.threads
@p.vars
@requires.preflight
def freshness(ctx, **kwargs):
"""Snapshots the current freshness of the project's sources"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# dbt test
Expand All @@ -447,10 +448,11 @@ def freshness(ctx, **kwargs):
@p.threads
@p.vars
@p.version_check
@requires.preflight
def test(ctx, **kwargs):
"""Runs tests on data in deployed models. Run this after `dbt run`"""
flags = Flags()
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {flags}")
click.echo(f"`{inspect.stack()[0][3]}` called\n flags: {ctx.obj['flags']}")
return None, True


# Support running as a module
Expand Down
6 changes: 3 additions & 3 deletions core/dbt/cli/params.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@
"--log-path",
envvar="DBT_LOG_PATH",
help="Configure the 'log-path'. Only applies this setting for the current run. Overrides the 'DBT_LOG_PATH' if it is set.",
default=Path.cwd() / "logs",
default=lambda: Path.cwd() / "logs",
type=click.Path(resolve_path=True, path_type=Path),
)

Expand Down Expand Up @@ -214,15 +214,15 @@
"--profiles-dir",
envvar="DBT_PROFILES_DIR",
help="Which directory to look in for the profiles.yml file. If not set, dbt will look in the current working directory first, then HOME/.dbt/",
default=default_profiles_dir(),
default=default_profiles_dir,
type=click.Path(exists=True),
)

project_dir = click.option(
"--project-dir",
envvar=None,
help="Which directory to look in for the dbt_project.yml file. Default is the current working directory and its parents.",
default=default_project_dir(),
default=default_project_dir,
type=click.Path(exists=True),
)

Expand Down
Loading