Embeded chunks do not respect the "execute-dir: project" setting

[venpopov]

Bug description

I have a quarto project, in which I have set the "execute-dir: project" in the yml file. This is because I have data files in one folder in the project dir, and quarto notebooks in a docs folder. When an individual notebook loads data files it works. However, if I embed a chunk from such a notebook into another, suddenly I get an error that the file cannot be found. Seems like rendering of notebooks to embed them does not respect the execture-dir: project setting.

Steps to reproduce

# create a quarto website project
tmp <- tempfile()
dir.create(tmp)
setwd(tmp)
system("quarto create project website .")

# create a file in a folder
dir.create("inner")
file.copy("about.qmd", "inner/file.qmd")

# write a code chunk in the inner file to open a file in base directory
saveRDS(rnorm(10), "data.rds")
CON <- file("inner/file.qmd", "a")
writeLines("```{r embedme}\nprint(readRDS('data.rds'))\n```", CON)
close(CON)

# ensure that the execture-dir is set to the project directory
writeLines("project:
  type: website
  execute-dir: project

website:
  title: 'blabla'
  navbar:
    left:
      - href: index.qmd
        text: Home
      - about.qmd

format:
  html:
    theme: cosmo
    css: styles.css
    toc: true", "_quarto.yml")

# write an index file that embeds the embedme chunk of the inner file
CON <- file("index.qmd", "a")
writeLines("{{< embed inner/file.qmd#embedme >}}", CON)
close(CON)

quarto::quarto_render()

#> �[1m�[34m[1/3] inner/file.qmd�[39m�[22m
#> �[31m
#> 
#> processing file: file.qmd
#> �[39m1/3          
#> 2/3 [embedme]
#> 3/3          
#> �[31moutput file: file.knit.md
#> 
#> �[39m�[1m�[34m[2/3] index.qmd�[39m�[22m
#> �[1m�[34mRendering qmd embeds�[39m�[22m
#> �[1m�[34m[1/1] inner/file.qmd�[39m�[22m
#> �[31m
#> 
#> processing file: file.qmd
#> �[39m1/3          
#> 2/3 [embedme]
#> �[31m
#> Quitting from lines  at lines 9-10 [embedme] (file.qmd)
#> �[39m�[31mError in `gzfile()`:
#> ! cannot open the connection
#> Backtrace:
#>  1. base::print(readRDS("data.rds"))
#>  2. base::readRDS("data.rds")
#>  3. base::gzfile(file, "rb")
#> �[39m�[31mExecution halted
#> �[39m�[91mERROR: Rendering of qmd notebook produced an unexpected result�[39m
#> Error in `quarto::quarto_render()`:
#> ✖ Error running quarto cli.
#> Caused by error:
#> ! System command 'quarto' failed

^{Created on 2024-05-13 with reprex v2.1.0}

You can see from the output that the inner\file.qmd is rendered succesfully by itself. However, when it is rendered to create and embeded chunk, it fails. This is not due to using quarto::quarto_render() - the same happens with the CLI, but the reprex does not show the output...

Expected behavior

Rendering of notebooks via embedings (https://quarto.org/docs/authoring/notebook-embed.html#overview) should respect the _quarto.yml options

Actual behavior

it does not respect the execute-dir: project option

Your environment

• IDE: irrelevant, happens even from the terminal
• MacOS Sonoma 14.4

Quarto check output

Quarto 1.4.553
[✓] Checking versions of quarto binary dependencies...
Pandoc version 3.1.11: OK
Dart Sass version 1.69.5: OK
Deno version 1.37.2: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
Version: 1.4.553
Path: /Applications/quarto/bin

[✓] Checking tools....................OK
TinyTeX: (not installed)
Chromium: (not installed)

[✓] Checking LaTeX....................OK
Tex: (not detected)

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
Version: 3.12.1 (Conda)
Path: /opt/miniconda3/bin/python
Jupyter: 5.5.0
Kernels: ir, python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
Version: 4.3.3
Path: /opt/homebrew/Cellar/r/4.3.3/lib/R
LibPaths:
- /opt/homebrew/lib/R/4.3/site-library
- /opt/homebrew/Cellar/r/4.3.3/lib/R/library
knitr: 1.46
rmarkdown: 2.26

[✓] Checking Knitr engine render......OK

[mcanouil]

Thanks for the report I can reproduce this on main.

Here is a reproducible example without requiring to use R to create it:

quarto create project website quarto-cli-9649 quarto-cli-9649
cd quarto-cli-9649
mkdir assets
echo -e "```{r}\n#| label: embedme\nreadLines("data.txt")\n```\n" > assets/_embed.qmd
echo "1" > data.txt
echo "{{< embed assets/_embed.qmd#embedme >}}" >> index.qmd
awk 'NR==3{print "  execute-dir: project"}1' _quarto.yml > temp && mv temp _quarto.yml

Also as a directory in https://github.com/mcanouil/quarto-issues-experiments/tree/main/quarto-cli-9649

[cderv]

@mcanouil your example above is missing the modification of _quarto.yml. The one is the repo seems good.
Not important but sharing for anyone trying to reproduce.

It seems indeed that execute-dir does not apply to embed file.

execute-dir applies as a project options and passed in rendering as RenderOptions

quarto-cli/src/command/render/project.ts

Lines 327 to 340 in b35914c

	// set execute dir if requested
	const executeDir = context.config?.project?.[kProjectExecuteDir];
	if (
	projectRenderConfig.options.flags?.executeDir === undefined &&
	executeDir === "project"
	) {
	projectRenderConfig.options = {
	...projectRenderConfig.options,
	flags: {
	...projectRenderConfig.options.flags,
	executeDir: projDir,
	},
	};
	}

Used at file rendering

quarto-cli/src/command/render/project.ts

Lines 390 to 400 in b35914c

	// render the files
	const fileResults = await renderFiles(
	projectRenderConfig.filesToRender,
	projectRenderConfig.options,
	context.notebookContext,
	projectRenderConfig.alwaysExecuteFiles,
	projType?.pandocRenderer
	? projType.pandocRenderer(projectRenderConfig.options, context)
	: undefined,
	context,
	);

Those options are used when building context

quarto-cli/src/command/render/render-files.ts

Lines 431 to 439 in b35914c

	contexts = await renderContexts(
	file,
	options,
	true,
	notebookContext,
	project,
	false,
	enforceProjectFormats,
	);

context is passed to execution

quarto-cli/src/command/render/render-files.ts

Lines 591 to 596 in b35914c

	// execute
	const baseExecuteResult = await renderExecute(
	context,
	recipe.output,
	executeOptions,
	);

where flags.executeDir is used to modify the cwd for execution

quarto-cli/src/command/render/render-files.ts

Line 223 in b35914c

cwd: flags.executeDir || dirname(normalizePath(context.target.source)),

This is for usual rendering. For embed file rendering, there is a special rendering process, with a specific renderFile() function

quarto-cli/src/render/notebook/notebook-contributor-qmd.ts

Lines 121 to 142 in b35914c

	const rendered = await renderFile(
	{ path: nbPath, formats: ["ipynb"] },
	{
	services,
	flags: {
	metadata: {
	[kTo]: "ipynb",
	[kOutputFile]: ipynbOutputFile(nbPath),
	[kNotebookPreserveCells]: true,
	[kIpynbProduceSourceNotebook]: true,
	citeproc: false,
	},
	quiet: false,
	},
	echo: true,
	warning: true,
	quietPandoc: true,
	},
	services,
	project,
	false, // Don't enforce project constraints on format since this is an intermediary rendering
	);

This function currently does not account for projectRenderConfig.options where the executeDir information is; It defines a specific set of rendering options.

This is the reason of the issue. It is part of all the rendering improvements we need to do for embed feature.

Current workaround

Now that we understand the cause, we can find a workaround to use while this is being resolved.

I would advice: don't rely on relative path to project context to load data. This is a long standing advice in R Markdown ecosystem already, and package like here help for this. execute-dir from Quarto is supposed to set knit.root.dir option in knitr to modify behavior for all chunks.
In most case, especially for loading / reading data, it is easier to explicitly compute the path you want from your project.
here::here() is not that necessary as you can create a wrapper for a quarto project. As we document at https://quarto.org/docs/projects/code-execution.html#working-dir we set QUARTO_PROJECT_DIR env var that you can use to compute a location you know is from project root.

Example

```{r}
#| label: embedme
readLines(xfun::from_root('data.txt', root = Sys.getenv("QUARTO_PROJECT_DIR")))
```

Also, don't use _ in your embed file prefix - we have an unsolved issue about this

• Embed is broken by period or underscore in the prefix of reference notebook #9224

As an example for R here package, the new mechanism to define project root does not even need Quarto information

```{r}
#| include: false
here::i_am("assets/embed.qmd")
```

```{r}
#| label: embedme
readLines(here::here('data.txt'))
```

See the doc of the package for more on its usage.

Anyhow, we'll solve this when we will tackle embed improvement

[venpopov]

Thanks for getting on this so quick and for the temporary workaround.!I can confirm that the embedding works with the here package specification. While this, works, and I appreciate the advice not to rely on relative project paths, I personally disaggree. For me one of the big reasons to use projects in the first place is to reduce friction when working with files and folder structures. I only every used the here package because Rmarkdown was so finicky with what working directories it executes code in, with counterintuitive scope of knitr options.

This workaround works, but it adds friction during development. I also find it occasionally problematic, because here::here returns absolute paths that are OS and user specific - for some applications, the file path is stored in created objects. This makes portability problematic and can also be undersired to share personal system configurations.

[cderv]

Thanks for the feedback. Rest assured this will be fixed - it is among a set of issues with embed that we'll probably tackle for 1.6.

Thanks for your patience.