document Mosaic

[Fil]

related #1011

cc: @jheer

[mbostock]

Can we use a data loader to make this, or at least include the script to make it alongside this cached output?

[Fil]

Absolutely. I just didn't want to document a data loader as part of this page, which has enough matter already.

It's not a hugely complicated data loader, except it needs to install the binary duckdb.

export TMPDIR="docs/.observablehq/cache"
export PATH=$TMPDIR:$PATH
duckdb :memory: << EOF
-- Load spatial extension
INSTALL spatial; LOAD spatial;

-- Project, following the example at https://github.com/duckdb/duckdb_spatial
CREATE TEMP TABLE rides AS SELECT
  pickup_datetime::TIMESTAMP AS datetime,
  ST_Transform(ST_Point(pickup_latitude, pickup_longitude), 'EPSG:4326', 'EPSG:32118') AS pick,
  ST_Transform(ST_Point(dropoff_latitude, dropoff_longitude), 'EPSG:4326', 'EPSG:32118') AS drop
FROM 'https://uwdata.github.io/mosaic-datasets/data/nyc-rides-2010.parquet';

-- Write output parquet file
COPY (SELECT
  HOUR(datetime) + MINUTE(datetime) / 60 AS time,
  ST_X(pick)::INTEGER AS px, -- extract pickup x-coord
  ST_Y(pick)::INTEGER AS py, -- extract pickup y-coord
  ST_X(drop)::INTEGER AS dx, -- extract dropff x-coord
  ST_Y(drop)::INTEGER AS dy  -- extract dropff y-coord
FROM rides
ORDER BY 2,3,4,5,1 -- optimize output size by sorting
) TO '$TMPDIR/trips.parquet' (COMPRESSION 'ZSTD', row_group_size 10000000);
EOF

cat $TMPDIR/trips.parquet >&1  # Write output to stdout
rm $TMPDIR/trips.parquet       # Clean up

[mbostock]

I think have the data loader checked-in but not functional, even if undocumented, is a big help!

[jheer]

This looks great. Thanks for including it! I agree that the data loader is an important component of the example, even if non-functional in this particular deployment.

[Fil]

I've checked in a simple version that does not try to work in CI — with the expectation that duckdb is on the $PATH.

[mbostock]

Should this have a shebang? One of these probably…

Suggested change

	duckdb :memory: << EOF
	#!/usr/bin/env bash
	duckdb :memory: << EOF

Suggested change

	duckdb :memory: << EOF
	#!/bin/sh
	duckdb :memory: << EOF

[Fil]

I don't think so; shebang are only used for .exe?

[mbostock]

Ah, so even if you have a shebang, it’s ignored because we explicitly run it with sh? If it’s not ignored, we should add one just to document our expectations around how the script is interpreted.

[Fil]

yes sh ignores the shebang and interprets the rest of the code directly

❯ chmod +x ./test.py
❯ cat ./test.py
#! /usr/bin/env python3
print("hello, world")
❯ bash ./test.py
./test.py: line 2: syntax error near unexpected token `"hello, world"'
./test.py: line 2: `print("hello, world")'
❯ ./test.py
hello, world

[mbostock]

Thanks for educating me. 😄

[mbostock]

Waiting for #1015 to land and then we’ll update this.