Skip to content

Commit

Permalink
Work around DLT issue with $PYTHONPATH not being set correctly (#999)
Browse files Browse the repository at this point in the history
## Changes

DLT currently doesn't always set `$PYTHONPATH` correctly (ES-947370).
This restores the original workaround to make new pipelines work while
that issue is being addressed. The workaround was removed in #832.

Manually tested.
  • Loading branch information
lennartkats-db authored Nov 20, 2023
1 parent 1b7558c commit 92539d4
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,6 @@ resources:
libraries:
- notebook:
path: ../src/dlt_pipeline.ipynb

configuration:
bundle.sourcePath: /Workspace/${workspace.file_path}/src
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
"# Import DLT and src/{{.project_name}}\n",
"import dlt\n",
"import sys\n",
"sys.path.append(spark.conf.get(\"bundle.sourcePath\", \".\"))\n",
"from pyspark.sql.functions import expr\n",
"from {{.project_name}} import main"
{{else}}
Expand Down

0 comments on commit 92539d4

Please sign in to comment.