Skip to content

Commit

Permalink
Merge branch 'main' into fil/interpreters
Browse files Browse the repository at this point in the history
  • Loading branch information
Fil authored Mar 7, 2024
2 parents 29389ae + 001ee14 commit a4c31b7
Show file tree
Hide file tree
Showing 46 changed files with 3,824 additions and 140 deletions.
2 changes: 1 addition & 1 deletion docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ To disable an extension, set its value to null:
interpreters: {".exe": null, ".rs": null}
```

## markdownIt
## markdownIt <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a>

A hook for registering additional [markdown-it](https://github.com/markdown-it/markdown-it) plugins. For example, to use [markdown-it-footnote](https://github.com/markdown-it/markdown-it-footnote):

Expand Down
19 changes: 19 additions & 0 deletions docs/display-race.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Display race

```js echo
async function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
```

```js echo
const value = (function* () {
yield 2000;
yield 1000;
})();
```

```js echo
await sleep(value);
display(value);
```
6 changes: 3 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,11 @@ Framework includes a helper script (`observable create`) for creating new projec

To create a new project with npm, run:

<pre data-copy>npm init @observablehq</pre>
<pre data-copy>npm init "@observablehq"</pre>

If you prefer Yarn, run:

<pre data-copy>yarn create @observablehq</pre>
<pre data-copy>yarn create "@observablehq"</pre>

You can run the above command anywhere, but you may want to `cd` to your `~/Development` directory first (or wherever you do local development).

Expand Down Expand Up @@ -221,7 +221,7 @@ Or with Yarn:

You should see something like this:

<pre data-copy="none"><b class="green">Observable Framework</b> v1.1.2
<pre data-copy="none"><b class="green">Observable Framework</b> v1.2.0
↳ <u><a href="http://127.0.0.1:3000/" style="color: inherit;">http://127.0.0.1:3000/</a></u></pre>

<div class="tip">
Expand Down
30 changes: 30 additions & 0 deletions docs/javascript/imports.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,36 @@ import confetti from "npm:canvas-confetti@1/dist/confetti.module.mjs";

If you’re having difficulty getting an import working, it may help to browse the package and see what files are available as well as what’s exported in the `package.json`. You can browse the contents of a published module via jsDelivr; for example, for `canvas-confetti` see <https://cdn.jsdelivr.net/npm/canvas-confetti/>.

### Self-hosting of npm imports

Framework automatically downloads `npm:` imports from jsDelivr during preview and build. This improves performance and security of your built site by removing external code dependencies. It also improves performance during local preview by only downloading libraries once.

Downloads from npm are cached in `.observablehq/cache/_npm` within your [source root](../config#root). You can clear the cache and restart the server to re-fetch the latest versions of libraries from npm.

Self-hosting of `npm:` imports applies to static imports, dynamic imports, and import resolutions (`import.meta.resolve`), provided the specifier is a static string literal. For example to load D3:

```js run=false
import * as d3 from "npm:d3";
```

```js run=false
const d3 = await import("npm:d3");
```

In both cases above, the latest version of `d3` is resolved and downloaded from jsDelivr as an ES module, including all of its transitive dependencies.

<div class="tip">You can load a library from a CDN at runtime by importing a URL. However, we recommend self-hosted <code>npm:</code> to improve performance and security, and to improve reliability by letting you control when you upgrade.</div>

Transitive static imports are also registered as module preloads (using `<link rel="modulepreload">`), such that the requests happen in parallel and as early as possible, rather than being chained. This dramatically improves performance on page load. Framework also preloads `npm:` imports for [`FileAttachment`](./files) methods, such as `d3-dsv` for [CSV](../lib/csv).

Import resolutions allow you to download files from npm. These files are automatically downloaded for self-hosting, too. For example, to load U.S. county geometry:

```js run=false
const data = await fetch(import.meta.resolve("npm:us-atlas/counties-albers-10m.json")).then((r) => r.json());
```
Framework automatically downloads files as needed for [recommended libraries](./imports#implicit-imports), and resolves import resolutions in transitive static and dynamic imports. For example, [DuckDB](../lib/duckdb) needs WebAssembly bundles, and [KaTeX](../lib/tex) needs a stylesheet and fonts. For any dependencies that are not statically analyzable (such as `fetch` calls or dynamic arguments to `import`) you can call `import.meta.resolve` to register additional files to download from npm.
## Local imports
In addition to npm, you can import JavaScript from local modules. This is useful for organizing your code: you can move JavaScript out of Markdown and create components and helpers that can be imported across multiple pages. This also means you can write unit tests for your code, and share code with any other web applications.
Expand Down
6 changes: 3 additions & 3 deletions docs/lib/echarts.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
index: true
---

# Apache ECharts
# Apache ECharts <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a>

Apache ECharts is available by default as `echarts` in Markdown. You can also import it explicitly like so:
[Apache ECharts](https://echarts.apache.org), an open-source JavaScript visualization library, is available by default as `echarts` in Markdown. You can also import it explicitly like so:

```js run=false
import * as echarts from "npm:echarts/dist/echarts.esm.js";
import * as echarts from "npm:echarts";
```

To use ECharts, declare a container element with the desired dimensions, [display it](../javascript/display), and then call `echarts.init`.
Expand Down
4 changes: 3 additions & 1 deletion docs/lib/leaflet.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ div.style = "height: 400px;";
const map = L.map(div)
.setView([51.505, -0.09], 13);

L.tileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png")
L.tileLayer("https://tile.openstreetmap.org/{z}/{x}/{y}.png", {
attribution: '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a>'
})
.addTo(map);

L.marker([51.5, -0.09])
Expand Down
9 changes: 9 additions & 0 deletions docs/loaders.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,15 @@ The **interpreters** [configuration option](./config#interpreters) specifies the

The corresponding interpreter must be installed and available on your `$PATH`. Any additional modules, packages, libraries, _etc._, must also be installed before you can use them.

Data loaders are run in the same working directory in which you run the `observable build` or `observable preview` command, which is typically the project root. In Node, you can access the current working directory by calling `process.cwd()`, and the data loader’s source location with [`import.meta.url`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta). To compute the path of a file relative to the data loader source (rather than relative to the current working directory), use [`import.meta.resolve`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve). For example, a data loader in `docs/summary.txt.js` could read the file `docs/table.txt` as:

```js run=false
import {readFile} from "node:fs/promises";
import {fileURLToPath} from "node:url";

const table = await readFile(fileURLToPath(import.meta.resolve("./table.txt")), "utf-8");
```
`.exe` data loaders are run directly and must have the executable bit set. This is typically done via [`chmod`](https://en.wikipedia.org/wiki/Chmod). For example:
```sh
Expand Down
5 changes: 3 additions & 2 deletions docs/markdown.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ The front matter supports the following options:
- **title** — the page title; defaults to the (first) first-level heading of the page, if any
- **toc** — if false, disables the [table of contents](./config#toc)
- **index** — whether to index this page if [search](./search) is enabled; defaults to true for listed pages
- **keywords** - additional words to index for [search](./search); boosted at the same weight as the title
- **draft** — whether to skip this page during build; drafts are also not listed in the default sidebar
- **keywords** <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a> - additional words to index for [search](./search); boosted at the same weight as the title
- **draft** <a href="https://github.com/observablehq/framework/releases/tag/v1.1.0" target="_blank" class="observablehq-version-badge" data-version="1.1.0" title="Added in v1.1.0"></a> — whether to skip this page during build; drafts are also not listed in the default sidebar
- **sql** <a href="https://github.com/observablehq/framework/releases/tag/v1.2.0" target="_blank" class="observablehq-version-badge" data-version="1.2.0" title="Added in v1.2.0"></a> — table definitions for [SQL code blocks](./sql)

## Headings

Expand Down
158 changes: 158 additions & 0 deletions docs/sql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
---
sql:
gaia: ./lib/gaia-sample.parquet
---

# SQL <a href="https://github.com/observablehq/framework/releases/tag/v1.2.0" target="_blank" class="observablehq-version-badge" data-version="1.2.0" title="Added in v1.2.0"></a>

Observable Framework includes built-in support for client-side SQL powered by [DuckDB](./lib/duckdb). You can use SQL to query data from [CSV](./lib/csv), [TSV](./lib/csv), [JSON](./javascript/files#json), [Apache Arrow](./lib/arrow), and [Apache Parquet](./lib/arrow#apache-parquet) files, which can either be static or generated by [data loaders](./loaders).

To use SQL, first register the desired tables in the page’s [front matter](./markdown#front-matter) using the **sql** option. Each key is a table name, and each value is the path to the corresponding data file. For example, to register a table named `gaia` from a Parquet file:

```yaml
---
sql:
gaia: ./lib/gaia-sample.parquet
---
```

## SQL code blocks

To run SQL queries, create a SQL fenced code block (<code>```sql</code>). For example, to query the first 10 rows from the `gaia` table:

````md
```sql
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```
````

This produces a table:

```sql
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```

To refer to the results of a query in JavaScript, use the `id` directive. For example, to refer to the results of the previous query as `top10`:

````md
```sql id=top10
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```
````

```sql id=top10
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```

This returns an array of 10 rows, inspected here:

```js echo
top10
```

When a SQL code block uses the `id` directive, the results are not displayed by default. You can display them by adding the `display` directive, which produces the table shown above.

````md
```sql id=top10 display
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10
```
````

The `id` directive is often a simple identifier such as `top10` above, but it supports [destructuring assignment](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment), so you can refer to individual rows and columns using array and object patterns. For example, to pull out the top row:

````md
```sql id=[top]
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 1
```
````

```sql id=[top]
SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 1
```

```js echo
top
```

Or to pull out the minimum value of the `phot_g_mean_mag` column:

````md
```sql id=[{min}]
SELECT MIN(phot_g_mean_mag) AS min FROM gaia
```
````

```sql id=[{min}]
SELECT MIN(phot_g_mean_mag) AS min FROM gaia
```

```js echo
min
```

<div class="tip">

For complex destructuring patterns, you may need to quote the `id` directive. For example, to pull out the column named `min(phot_g_mean_mag)` to the variable named `min`, say <code style="white-space: nowrap;">id="[{'min(phot_g_mean_mag)': min}]"</code>. Or to pull out the `min` and `max` columns, say <code style="white-space: nowrap;">id="[{min, max}]"</code>.

</div>

For dynamic or interactive queries that respond to user input, you can interpolate values into SQL queries using inline expressions `${…}`. For example, to show the stars around a given brightness:

```js echo
const mag = view(Inputs.range([6, 20], {label: "Magnitude"}));
```

```sql echo
SELECT * FROM gaia WHERE phot_g_mean_mag BETWEEN ${mag - 0.1} AND ${mag + 0.1};
```

The value of a SQL code block is an [Apache Arrow](./lib/arrow) table. This format is supported by [Observable Plot](./lib/plot), so you can use SQL and Plot together to visualize data. For example, below we count the number of stars in each 2°×2° bin of the sky (where `ra` is [right ascension](https://en.wikipedia.org/wiki/Right_ascension) and `dec` is [declination](https://en.wikipedia.org/wiki/Declination), representing a point on the celestial sphere in the equatorial coordinate system), and then visualize the resulting heatmap using a [raster mark](https://observablehq.com/plot/marks/raster).

```sql id=bins echo
SELECT
floor(ra / 2) * 2 + 1 AS ra,
floor(dec / 2) * 2 + 1 AS dec,
count() AS count
FROM
gaia
GROUP BY
1,
2
```

```js echo
Plot.plot({
aspectRatio: 1,
x: {domain: [0, 360]},
y: {domain: [-90, 90]},
marks: [
Plot.frame({fill: 0}),
Plot.raster(bins, {
x: "ra",
y: "dec",
fill: "count",
width: 360 / 2,
height: 180 / 2,
imageRendering: "pixelated"
})
]
})
```

## SQL literals

SQL fenced code blocks are shorthand for the `sql` tagged template literal. You can invoke the `sql` tagged template literal directly like so:

```js echo
const rows = await sql`SELECT random() AS random`;
```

```js echo
rows[0].random
```

The `sql` tagged template literal is available by default in Markdown, but you can also import it explicitly as:

```js echo
import {sql} from "npm:@observablehq/duckdb";
```
14 changes: 14 additions & 0 deletions docs/style.css
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,17 @@
display: none;
}
}

a.observablehq-version-badge {
font: 500 12px var(--sans-serif);
display: inline-block;
vertical-align: top;
padding: 2px 8px;
color: var(--theme-green);
background: color-mix(in srgb, var(--theme-green), transparent 80%);
border-radius: 12px;
}

.observablehq-version-badge::before {
content: "^" attr(data-version);
}
1 change: 1 addition & 0 deletions observablehq.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ export default {
{name: "Markdown", path: "/markdown"},
{name: "JavaScript", path: "/javascript"},
{name: "Data loaders", path: "/loaders"},
{name: "SQL", path: "/sql"},
{name: "Themes", path: "/themes"},
{name: "Configuration", path: "/config"},
{
Expand Down
6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@observablehq/framework",
"license": "ISC",
"version": "1.1.2",
"version": "1.2.0",
"type": "module",
"publishConfig": {
"access": "public"
Expand All @@ -27,8 +27,8 @@
"test": "concurrently npm:test:mocha npm:test:tsc npm:test:lint npm:test:prettier",
"test:coverage": "c8 --check-coverage --lines 80 --per-file yarn test:mocha",
"test:build": "rimraf test/build && node build.js --sourcemap --outdir=test/build \"{src,test}/**/*.{ts,js,css}\" --ignore \"test/input/**\" --ignore \"test/output/**\" --ignore \"test/preview/dashboard/**\" --ignore \"**/*.d.ts\" && cp -r templates test/build",
"test:mocha": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha -p \"test/build/test/**/*-test.js\"",
"test:mocha:serial": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha \"test/build/test/**/*-test.js\"",
"test:mocha": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha --timeout 5000 -p \"test/build/test/**/*-test.js\"",
"test:mocha:serial": "yarn test:build && rimraf --glob test/.observablehq/cache test/input/build/*/.observablehq/cache && cross-env OBSERVABLE_TELEMETRY_DISABLE=1 TZ=America/Los_Angeles mocha --timeout 5000 \"test/build/test/**/*-test.js\"",
"test:lint": "eslint src test --max-warnings=0",
"test:prettier": "prettier --check src test",
"test:tsc": "tsc --noEmit",
Expand Down
2 changes: 1 addition & 1 deletion src/build.ts
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ export async function build(
effects.output.write(`${faint("build")} ${clientPath} ${faint("→")} `);
const define: {[key: string]: string} = {};
if (config.search) define["global.__minisearch"] = JSON.stringify(relativePath(path, aliases.get("/_observablehq/minisearch.json")!)); // prettier-ignore
const contents = await rollupClient(clientPath, root, path, {minify: true, define});
const contents = await rollupClient(clientPath, root, path, {minify: true, keepNames: true, define});
await effects.writeFile(path, contents);
}
}
Expand Down
Loading

0 comments on commit a4c31b7

Please sign in to comment.