Merge branch 'main' into fil/interpreters

docs/config.md

@@ -228,7 +228,7 @@ To disable an extension, set its value to null:
 interpreters: {".exe": null, ".rs": null}
 ```
 
-## markdownIt
+## markdownIt <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a>
 
 A hook for registering additional [markdown-it](https://github.com/markdown-it/markdown-it) plugins. For example, to use [markdown-it-footnote](https://github.com/markdown-it/markdown-it-footnote):
 

docs/display-race.md

@@ -0,0 +1,19 @@
+# Display race
+
+```js echo
+async function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+```
+
+```js echo
+const value = (function* () {
+  yield 2000;
+  yield 1000;
+})();
+```
+
+```js echo
+await sleep(value);
+display(value);
+```

docs/getting-started.md

@@ -90,11 +90,11 @@ Framework includes a helper script (`observable create`) for creating new projec
 
 To create a new project with npm, run:
 
-<pre data-copy>npm init @observablehq</pre>
+<pre data-copy>npm init "@observablehq"</pre>
 
 If you prefer Yarn, run:
 
-<pre data-copy>yarn create @observablehq</pre>
+<pre data-copy>yarn create "@observablehq"</pre>
 
 You can run the above command anywhere, but you may want to `cd` to your `~/Development` directory first (or wherever you do local development).
 
@@ -221,7 +221,7 @@ Or with Yarn:
 
 You should see something like this:
 
-<pre data-copy="none"><b class="green">Observable Framework</b> v1.1.2
+<pre data-copy="none"><b class="green">Observable Framework</b> v1.2.0
 ↳ <u><a href="http://127.0.0.1:3000/" style="color: inherit;">http://127.0.0.1:3000/</a></u></pre>
 
 <div class="tip">

docs/javascript/imports.md

@@ -40,6 +40,36 @@ import confetti from "npm:canvas-confetti@1/dist/confetti.module.mjs";
 
 If you’re having difficulty getting an import working, it may help to browse the package and see what files are available as well as what’s exported in the `package.json`. You can browse the contents of a published module via jsDelivr; for example, for `canvas-confetti` see <https://cdn.jsdelivr.net/npm/canvas-confetti/>.
 
+### Self-hosting of npm imports
+
+Framework automatically downloads `npm:` imports from jsDelivr during preview and build. This improves performance and security of your built site by removing external code dependencies. It also improves performance during local preview by only downloading libraries once.
+
+Downloads from npm are cached in `.observablehq/cache/_npm` within your [source root](../config#root). You can clear the cache and restart the server to re-fetch the latest versions of libraries from npm.
+
+Self-hosting of `npm:` imports applies to static imports, dynamic imports, and import resolutions (`import.meta.resolve`), provided the specifier is a static string literal. For example to load D3:
+
+```js run=false
+import * as d3 from "npm:d3";
+```
+
+```js run=false
+const d3 = await import("npm:d3");
+```
+
+In both cases above, the latest version of `d3` is resolved and downloaded from jsDelivr as an ES module, including all of its transitive dependencies.
+
+<div class="tip">You can load a library from a CDN at runtime by importing a URL. However, we recommend self-hosted <code>npm:</code> to improve performance and security, and to improve reliability by letting you control when you upgrade.</div>
+
+Transitive static imports are also registered as module preloads (using `<link rel="modulepreload">`), such that the requests happen in parallel and as early as possible, rather than being chained. This dramatically improves performance on page load. Framework also preloads `npm:` imports for [`FileAttachment`](./files) methods, such as `d3-dsv` for [CSV](../lib/csv).
+
+Import resolutions allow you to download files from npm. These files are automatically downloaded for self-hosting, too. For example, to load U.S. county geometry:
+
+```js run=false
+const data = await fetch(import.meta.resolve("npm:us-atlas/counties-albers-10m.json")).then((r) => r.json());
+```
+
+Framework automatically downloads files as needed for [recommended libraries](./imports#implicit-imports), and resolves import resolutions in transitive static and dynamic imports. For example, [DuckDB](../lib/duckdb) needs WebAssembly bundles, and [KaTeX](../lib/tex) needs a stylesheet and fonts. For any dependencies that are not statically analyzable (such as `fetch` calls or dynamic arguments to `import`) you can call `import.meta.resolve` to register additional files to download from npm.
+
 ## Local imports
 
 In addition to npm, you can import JavaScript from local modules. This is useful for organizing your code: you can move JavaScript out of Markdown and create components and helpers that can be imported across multiple pages. This also means you can write unit tests for your code, and share code with any other web applications.

docs/lib/echarts.md

@@ -2,12 +2,12 @@
 index: true
 ---
 
-# Apache ECharts
+# Apache ECharts <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a>
 
-Apache ECharts is available by default as `echarts` in Markdown. You can also import it explicitly like so:
+[Apache ECharts](https://echarts.apache.org), an open-source JavaScript visualization library, is available by default as `echarts` in Markdown. You can also import it explicitly like so:
 
 ```js run=false
-import * as echarts from "npm:echarts/dist/echarts.esm.js";
+import * as echarts from "npm:echarts";
 ```
 
 To use ECharts, declare a container element with the desired dimensions, [display it](../javascript/display), and then call `echarts.init`.

docs/lib/leaflet.md

@@ -21,7 +21,9 @@ div.style = "height: 400px;";
 const map = L.map(div)
   .setView([51.505, -0.09], 13);
 
-L.tileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png")
+L.tileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png", {
+  attribution: '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a>'
+})
   .addTo(map);
 
 L.marker([51.5, -0.09])

docs/loaders.md

@@ -146,6 +146,15 @@ The **interpreters** [configuration option](./config#interpreters) specifies the
 
 The corresponding interpreter must be installed and available on your `$PATH`. Any additional modules, packages, libraries, _etc._, must also be installed before you can use them.
 
+Data loaders are run in the same working directory in which you run the `observable build` or `observable preview` command, which is typically the project root. In Node, you can access the current working directory by calling `process.cwd()`, and the data loader’s source location with [`import.meta.url`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta). To compute the path of a file relative to the data loader source (rather than relative to the current working directory), use [`import.meta.resolve`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve). For example, a data loader in `docs/summary.txt.js` could read the file `docs/table.txt` as:
+
+```js run=false
+import {readFile} from "node:fs/promises";
+import {fileURLToPath} from "node:url";
+
+const table = await readFile(fileURLToPath(import.meta.resolve("./table.txt")), "utf-8");
+```
+
 `.exe` data loaders are run directly and must have the executable bit set. This is typically done via [`chmod`](https://en.wikipedia.org/wiki/Chmod). For example:
 
 ```sh

docs/markdown.md

@@ -18,8 +18,9 @@ The front matter supports the following options:
 - **title** — the page title; defaults to the (first) first-level heading of the page, if any
 - **toc** — if false, disables the [table of contents](./config#toc)
 - **index** — whether to index this page if [search](./search) is enabled; defaults to true for listed pages
-- **keywords** - additional words to index for [search](./search); boosted at the same weight as the title
-- **draft** — whether to skip this page during build; drafts are also not listed in the default sidebar
+- **keywords** <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a> - additional words to index for [search](./search); boosted at the same weight as the title
+- **draft** <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a> — whether to skip this page during build; drafts are also not listed in the default sidebar
+- **sql** <a href="https://github.com/observablehq/framework/releases/tag/v1.2.0" target="_blank" class="observablehq-version-badge" data-version="1.2.0" title="Added in v1.2.0"></a> — table definitions for [SQL code blocks](./sql)
 
 ## Headings
 

docs/sql.md

@@ -0,0 +1,158 @@
+---
+sql:
+  gaia: ./lib/gaia-sample.parquet
+---
+
+# SQL <a href="https://github.com/observablehq/framework/releases/tag/v1.2.0" target="_blank" class="observablehq-version-badge" data-version="1.2.0" title="Added in v1.2.0"></a>
+
+Observable Framework includes built-in support for client-side SQL powered by [DuckDB](./lib/duckdb). You can use SQL to query data from [CSV](./lib/csv), [TSV](./lib/csv), [JSON](./javascript/files#json), [Apache Arrow](./lib/arrow), and [Apache Parquet](./lib/arrow#apache-parquet) files, which can either be static or generated by [data loaders](./loaders).
+
+To use SQL, first register the desired tables in the page’s [front matter](./markdown#front-matter) using the **sql** option. Each key is a table name, and each value is the path to the corresponding data file. For example, to register a table named `gaia` from a Parquet file:
+
+```yaml
+---
+sql:
+  gaia: ./lib/gaia-sample.parquet
+---
+```
+
+## SQL code blocks
+
+To run SQL queries, create a SQL fenced code block (<code>```sql</code>). For example, to query the first 10 rows from the `gaia` table:
+
+````md
+```sql
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
+```
+````
+
+This produces a table:
+
+```sql
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
+```
+
+To refer to the results of a query in JavaScript, use the `id` directive. For example, to refer to the results of the previous query as `top10`:
+
+````md
+```sql id=top10
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
+```
+````
+
+```sql id=top10
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
+```
+
+This returns an array of 10 rows, inspected here:
+
+```js echo
+top10
+```
+
+When a SQL code block uses the `id` directive, the results are not displayed by default. You can display them by adding the `display` directive, which produces the table shown above.
+
+````md
+```sql id=top10 display
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
+```
+````
+
+The `id` directive is often a simple identifier such as `top10` above, but it supports [destructuring assignment](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment), so you can refer to individual rows and columns using array and object patterns. For example, to pull out the top row:
+
+````md
+```sql id=[top]
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 1
+```
+````
+
+```sql id=[top]
+SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 1
+```
+
+```js echo
+top
+```
+
+Or to pull out the minimum value of the `phot_g_mean_mag` column:
+
+````md
+```sql id=[{min}]
+SELECT MIN(phot_g_mean_mag) AS min FROM gaia
+```
+````
+
+```sql id=[{min}]
+SELECT MIN(phot_g_mean_mag) AS min FROM gaia
+```
+
+```js echo
+min
+```
+
+<div class="tip">
+
+For complex destructuring patterns, you may need to quote the `id` directive. For example, to pull out the column named `min(phot_g_mean_mag)` to the variable named `min`, say <code style="white-space: nowrap;">id="[{'min(phot_g_mean_mag)': min}]"</code>. Or to pull out the `min` and `max` columns, say <code style="white-space: nowrap;">id="[{min, max}]"</code>.
+
+</div>
+
+For dynamic or interactive queries that respond to user input, you can interpolate values into SQL queries using inline expressions `${…}`. For example, to show the stars around a given brightness:
+
+```js echo
+const mag = view(Inputs.range([6, 20], {label: "Magnitude"}));
+```
+
+```sql echo
+SELECT * FROM gaia WHERE phot_g_mean_mag BETWEEN ${mag - 0.1} AND ${mag + 0.1};
+```
+
+The value of a SQL code block is an [Apache Arrow](./lib/arrow) table. This format is supported by [Observable Plot](./lib/plot), so you can use SQL and Plot together to visualize data. For example, below we count the number of stars in each 2°×2° bin of the sky (where `ra` is [right ascension](https://en.wikipedia.org/wiki/Right_ascension) and `dec` is [declination](https://en.wikipedia.org/wiki/Declination), representing a point on the celestial sphere in the equatorial coordinate system), and then visualize the resulting heatmap using a [raster mark](https://observablehq.com/plot/marks/raster).
+
+```sql id=bins echo
+SELECT
+  floor(ra / 2) * 2 + 1 AS ra,
+  floor(dec / 2) * 2 + 1 AS dec,
+  count() AS count
+FROM
+  gaia
+GROUP BY
+  1,
+  2
+```
+
+```js echo
+Plot.plot({
+  aspectRatio: 1,
+  x: {domain: [0, 360]},
+  y: {domain: [-90, 90]},
+  marks: [
+    Plot.frame({fill: 0}),
+    Plot.raster(bins, {
+      x: "ra",
+      y: "dec",
+      fill: "count",
+      width: 360 / 2,
+      height: 180 / 2,
+      imageRendering: "pixelated"
+    })
+  ]
+})
+```
+
+## SQL literals
+
+SQL fenced code blocks are shorthand for the `sql` tagged template literal. You can invoke the `sql` tagged template literal directly like so:
+
+```js echo
+const rows = await sql`SELECT random() AS random`;
+```
+
+```js echo
+rows[0].random
+```
+
+The `sql` tagged template literal is available by default in Markdown, but you can also import it explicitly as:
+
+```js echo
+import {sql} from "npm:@observablehq/duckdb";
+```

docs/style.css

@@ -34,3 +34,17 @@
     display: none;
   }
 }
+
+a.observablehq-version-badge {
+  font: 500 12px var(--sans-serif);
+  display: inline-block;
+  vertical-align: top;
+  padding: 2px 8px;
+  color: var(--theme-green);
+  background: color-mix(in srgb, var(--theme-green), transparent 80%);
+  border-radius: 12px;
+}
+
+.observablehq-version-badge::before {
+  content: "^" attr(data-version);
+}

observablehq.config.ts

@@ -7,6 +7,7 @@ export default {
     {name: "Markdown", path: "/markdown"},
     {name: "JavaScript", path: "/javascript"},
     {name: "Data loaders", path: "/loaders"},
+    {name: "SQL", path: "/sql"},
     {name: "Themes", path: "/themes"},
     {name: "Configuration", path: "/config"},
     {

package.json

@@ -1,7 +1,7 @@
 {
   "name": "@observablehq/framework",
   "license": "ISC",
-  "version": "1.1.2",
+  "version": "1.2.0",
   "type": "module",
   "publishConfig": {
     "access": "public"
@@ -27,8 +27,8 @@
     "test": "concurrently npm:test:mocha npm:test:tsc npm:test:lint npm:test:prettier",
     "test:coverage": "c8 --check-coverage --lines 80 --per-file yarn test:mocha",
     "test:build": "rimraf test/build && node build.js --sourcemap --outdir=test/build \"{src,test}/**/*.{ts,js,css}\" --ignore \"test/input/**\" --ignore \"test/output/**\" --ignore \"test/preview/dashboard/**\" --ignore \"**/*.d.ts\" && cp -r templates test/build",
-    "test:mocha": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha -p \"test/build/test/**/*-test.js\"",
-    "test:mocha:serial": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha \"test/build/test/**/*-test.js\"",
+    "test:mocha": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha --timeout 5000 -p \"test/build/test/**/*-test.js\"",
+    "test:mocha:serial": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha --timeout 5000 \"test/build/test/**/*-test.js\"",
     "test:lint": "eslint src test --max-warnings=0",
     "test:prettier": "prettier --check src test",
     "test:tsc": "tsc --noEmit",

src/build.ts

@@ -112,7 +112,7 @@ export async function build(
         effects.output.write(`${faint("build")} ${clientPath} ${faint("→")} `);
         const define: {[key: string]: string} = {};
         if (config.search) define["global.__minisearch"] = JSON.stringify(relativePath(path, aliases.get("/_observablehq/minisearch.json")!)); // prettier-ignore
-        const contents = await rollupClient(clientPath, root, path, {minify: true, define});
+        const contents = await rollupClient(clientPath, root, path, {minify: true, keepNames: true, define});
         await effects.writeFile(path, contents);
       }
     }