Add databricks labs command group

[nfx]

Command group

Installed versions

Project commands

Installer hook

Update notifications

Downstream work

• Support for the experimental Databricks CLI launcher databrickslabs/ucx#517
• Added Databricks CLI installer PoC databrickslabs/dlt-meta#19
• Added entrypoint for Databricks CLI databrickslabs/discoverx#84

[nfx]

Downstream command router can be really trivial:

import logging
import sys, json

from databricks.sdk import WorkspaceClient

from databricks.labs.ucx.install import WorkspaceInstaller


def jobs():
    ws = WorkspaceClient()
    installer = WorkspaceInstaller(ws)
    for step, job_id in installer.deployed_steps().items():
        print(step, job_id)


def me():
    ws = WorkspaceClient()
    my_user = ws.current_user.me()
    print(my_user.user_name)


MAPPING = {
    'jobs': jobs,
    'me': me,
}


def main(raw):
    payload = json.loads(raw)
    command = payload['command']
    if command not in MAPPING:
        raise KeyError(f'cannot find command: {command}')
    flags = payload['flags']
    log_level = flags.pop('log_level')
    if log_level != 'disabled':
        databricks_logger = logging.getLogger("databricks")
        databricks_logger.setLevel(log_level.upper())

    kwargs = {k.replace('-', '_'): v for k,v in flags.items()}
    MAPPING[command](**kwargs)


if __name__ == "__main__":
    main(*sys.argv[1:])

[nfx]

#927 make snapshot in a separate PR

[nfx]

created smaller PRs for non-labs code:

• Added env.UserHomeDir(ctx) for parallel-friendly tests #955
• Improved usability of databricks auth login ... --configure-cluster flow by displaying cluster type and runtime version #956
• Make Cobra runner compatible with testing interactive flows #957

[nfx]

installing this branch

brew install golang
git@github.com:databricks/cli.git
cd cli
git checkout feat/labs
make snapshot

and the binary will be in .databricks/databricks file

[alexott]

Got error on the clean install

Error: save cache: open /Users/alexey.ott/.databricks/labs/databrickslabs-repositories.json: no such file or directory

[alexott]

Another bug

$ databricks labs sandbox ip-access-list-analyzer --profile=az-alexott
Error: cannot find Python 3.8. Please re-run: databricks labs install ip-access-list-analyzer

[alexott]

>databricks labs install . --debug
time=2023-11-10T11:12:49.984+01:00 level=INFO source="root.go 55}" msg=start pid=51984 version=0.0.0-dev+5819644739e8 args="databricks, labs, install, ., --debug"
Installing sandbox in development mode from /Users/.../work/samples/dbl-sandbox

[nfx]

#914 (comment) - fixed

these two are related to dev mode, that we'll fix by talking with dev - #914 (comment) #914 (comment)

[pietern]

This change LGTM and acknowledge there will be follop-ups on the UX.

Would appreciate a second pair of eyes before a stamp.

[pietern]

These won't be cleared in the environment because they are zero. That means they are not returned from e.environmentFromConfig below and are not cleared for the subprocess, which could confuse downstream programs.

[nfx]

@pietern the only case possible when we have --profile (DATABRICKS_CONFIG_PROFILE) is the local dev machine. Github actions (and likes) won't have the DATABRICKS_CONFIG_PROFILE / DATABRICKS_CONFIG_FILE envs set, but rather direct ones, like client ids or hosts. there are tests that verify this to work from github actions-like environments

[mgyucht]

@pietern and I reviewed this together this afternoon and feel generally good about it. Let's get labs on the road!

[pietern]

Removed from the merge queue if you want to take a look at the issue above.

[nfx]

@pietern that permutation of configuration flags is very unlikely to happen and has unit tests already. going ahead with the merge :)