Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Improve flatgeobuf docs #2859

Merged
merged 2 commits into from
Jan 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/modules/flatgeobuf/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Overview

![flatgeobuf-logo]( ./images/flatgeobuf-logo.png)

The `@loaders.gl/flatgeobuf` module handles the [FlatGeobuf](http://flatgeobuf.org/) format, a binary FlatBuffers-encoded format that defines geospatial geometries.

## Installation
Expand All @@ -17,4 +19,4 @@ npm install @loaders.gl/core

## Attribution

The `FlatGeobufLoader` forks the [`flatgeobuf`](https://github.com/bjornharrtell/flatgeobuf) NPM module under the ISC license.
The `FlatGeobufLoader` forks the [`flatgeobuf`](https://github.com/bjornharrtell/flatgeobuf) NPM module under the BSD 2-Clause license.
2 changes: 2 additions & 0 deletions docs/modules/flatgeobuf/api-reference/flatgeobuf-loader.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# FlatGeobufLoader

![flatgeobuf-logo](../images/flatgeobuf-logo.png)

<p class="badges">
<img src="https://img.shields.io/badge/From-v3.1-blue.svg?style=flat-square" alt="From-v3.1" />
&nbsp;
Expand Down
141 changes: 134 additions & 7 deletions docs/modules/flatgeobuf/formats/flatgeobuf.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,144 @@
# FlatGeobuf

- *[`@loaders.gl/flatgeobuf`](/docs/moodules/flatgeobuf)*
- *[FlatGeobuf](http://flatgeobuf.org/)*
![flatgeobuf-logo](../images/flatgeobuf-logo.png)

FlatGeobuf is a binary (FlatBuffers-encoded) format that defines geospatial geometries. It is row-oriented rather than columnar (like GeoParquet and GeoArrow) and offers a different set of trade-offs.
- *[`@loaders.gl/flatgeobuf`](/docs/modules/flatgeobuf)*
- *[FlatGeobuf](http://flatgeobuf.org/)*

FlatGeobuf is a binary (FlatBuffers-encoded) format that defines geospatial geometries. It is row-oriented rather than columnar like GeoParquet and GeoArrow and offers a different set of trade-offs.
FlatGeobuf was inspired by [geobuf](https://github.com/mapbox/geobuf) and [flatbush](https://github.com/mourner/flatbush).

## Characteristics

- binary
- row oriented
- supports appends, but no random writes
- Binary
- Row oriented
- Supports appends, but not random writes
- Optionally supports a spatial index

Goals are to be suitable for large volumes of static data, significantly faster than legacy formats without size limitations for contents or meta information and to be suitable for streaming/random access.

## Geometries

FlatGeobuf supports any vector geometry type defined in the OGC Simple Features specification (the same feature types supported by the WKB 2D geometry type enumeration).

:::caution
GeoBuf geometries include the standard building blocks of `Point`, `LineString`, `Polygon`,`MultiPoint`, `MultiLineString`, `MultiPolygon`, and `GeometryCollection`, but also includes more infrequently types such as `CircularString`, `Surface`, and `TIN`` (Triangulated irregular network). These additional types are not supported by loaders.gl.
:::


| Type | Value | loaders.gl | Comment |
| ------------------ | ----- | ---------- | ------- |
| Unknown | 0 || |
| Point | 1 || |
| LineString | 2 || |
| Polygon | 3 || |
| MultiPoint | 4 || |
| MultiLineString | 5 || |
| MultiPolygon | 6 || |
| GeometryCollection | 7 || |
| CircularString | 8 || |
| CompoundCurve | 9 || |
| CurvePolygon | 10 || |
| MultiCurve | 11 || |
| MultiSurface | 12 || |
| Curve | 13 || |
| Surface | 14 || |
| PolyhedralSurface | 15 || |
| TIN | 16 || |
| Triangle | 1 || |

Note: Storing only geometries with the same type allows readers to know which geometry type is stored without scanning the entire file.

## Schema

Apart from geometry, FlatGeobuf supports columns with a range of types:

| Type | Description |
| ---------- | ------------------------------------------------------- |
| `Byte` | Signed 8-bit integer |
| `UByte` | Unsigned 8-bit integer |
| `Bool` | Boolean |
| `Short` | Signed 16-bit integer |
| `UShort` | Unsigned 16-bit integer |
| `Int` | Signed 32-bit integer |
| `UInt` | Unsigned 32-bit integer |
| `Long` | Signed 64-bit integer |
| `ULong` | Unsigned 64-bit integer |
| `Float` | Single precision floating point number |
| `Double` | Double precision floating point number |
| `String` | UTF8 string |
| `Json` | General JSON type intended to be application specific |
| `DateTime` | ISO 8601 date time |
| `Binary` | General binary type intended to be application specific |

## Metadata

:::caution
loaders.gl currently does not currently expose all metadata.
:::

```typescript
type Header {
name: string; // Dataset name
envelope: [double]; // Bounds
geometry_type: GeometryType; // Geometry type (should be set to Unknown if per feature geometry type)
has_z: bool = false; // Does geometry have Z dimension?
has_m: bool = false; // Does geometry have M dimension?
has_t: bool = false; // Does geometry have T dimension?
has_tm: bool = false; // Does geometry have TM dimension?
columns: [Column]; // Attribute columns schema (can be omitted if per feature schema)
features_count: ulong; // Number of features in the dataset (0 = unknown)
index_node_size: ushort = 16; // Index node size (0 = no index)
crs: Crs; // Spatial Reference System
title: string; // Dataset title
description: string; // Dataset description (intended for free form long text)
metadata: string; // Dataset metadata (intended to be application specific and suggested to be structured fx. JSON)
}

type Crs {
org: string; // Case-insensitive name of the defining organization e.g. EPSG or epsg (NULL = EPSG)
code: int; // Numeric ID of the Spatial Reference System assigned by the organization (0 = unknown)
name: string; // Human readable name of this SRS
description: string; // Human readable description of this SRS
wkt: string; // Well-known Text Representation of the Spatial Reference System
code_string: string; // Text ID of the Spatial Reference System assigned by the organization in the (rare) case when it is not an integer and thus cannot be set into code
}

type Column {
name: string (required); // Column name
type: ColumnType; // Column type
title: string; // Column title
description: string; // Column description (intended for free form long text)
width: int = -1; // Column values expected width (-1 = unknown) (currently only used to indicate the number of characters in strings)
precision: int = -1; // Column values expected precision (-1 = unknown) as defined by SQL
scale: int = -1; // Column values expected scale (-1 = unknown) as defined by SQL
nullable: bool = true; // Column values expected nullability
unique: bool = false; // Column values expected uniqueness
primary_key: bool = false; // Indicates this column has been (part of) a primary key
metadata: string; // Column metadata (intended to be application specific and suggested to be structured fx. JSON)
}
```

Each column also has a string that can hold arbitrary metadata.

## Spatial indexing

:::caution
loaders.gl currently does not support spatial filtering.
:::

FlatGeobuf files can optionally contain a spatial index. The spatial index is optional to allow the format to be efficiently written as a stream, support appending, and for use cases where spatial filtering is not needed.

The spatial index clusters the data on a [packed Hilbert R-Tree](https://en.wikipedia.org/wiki/Hilbert_R-tree#Packed_Hilbert_R-trees) enabling fast bounding box spatial filtering.

The Hilbert curve imposes a linear ordering on the data rectangles and then traverses the sorted list, assigning each set of C rectangles to a node in the R-tree. The final result is that the set of data rectangles on the same node will be close to each other in the linear order.

### Optimizing Remotely Hosted FlatGeobufs

If you’re accessing a FlatGeobuf file over HTTP, consider using a CDN to minimize latency.

In particular, when using the spatial filter to get a subset of features, multiple requests will be made. Often round-trip latency, rather than throughput, is the limiting factor. A caching CDN can be especially helpful here.

Goals are to be suitable for large volumes of static data, significantly faster than legacy formats without size limitations for contents or metainformation and to be suitable for streaming/random access.
Fetching a subset of a file over HTTP utilizes Range requests. If the page accessing the FGB is hosted on a different domain from the CDN, Cross Origin policy applies, and the required Range header will induce an OPTIONS (preflight) request.

Popular CDNs, like Cloudfront, support Range Requests, but don’t cache the requisite preflight OPTIONS requests by default. Consider enabling OPTIONS request caching . Without this, the preflight authorization request could be much slower than necessary.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions docs/modules/parquet/formats/geoparquet.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# GeoParquet

![parquet-logo](../images/parquet-logo-small.png)
&emsp;
![apache-logo](../../../images/logos/apache-logo.png)

- *[`loaders.gl/parquet`](/docs/modules/parquet)*
- *[geoparquet.org](https://geoparquet.org)*

Expand Down
8 changes: 8 additions & 0 deletions docs/modules/parquet/formats/parquet.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Parquet

![parquet-logo](../images/parquet-logo-small.png)
&emsp;
![apache-logo](../../../images/logos/apache-logo.png)

- *[`@loaders.gl/parquet`](/docs/modules/parquet)*
- *[Parquet](https://parquet.apache.org/docs/file-format/)*

Expand Down Expand Up @@ -108,3 +112,7 @@ TBA - This table is not complete
| `parquet` | `map, type=MAP, convertedtype=MAP, keytype=BYTE_ARRAY, keyconvertedtype=UTF8, valuetype=INT32"` | |
| `list` | `MAP` convertedtype=LIST, valuetype=BYTE_ARRAY, valueconvertedtype=UTF8 | |
| `repeated | `INT32` repetitiontype=REPEATED"` | |

## Format Structure

![parquet-file-format](../images/parquet-file-format.png)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion modules/flatgeobuf/src/flatgeobuf/3.27.2/column-meta.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import {ColumnType} from './flat-geobuf/column-type.js';
import {ColumnType} from './flat-geobuf/column-type';

export default interface ColumnMeta {
name: string;
Expand Down
6 changes: 3 additions & 3 deletions modules/flatgeobuf/src/flatgeobuf/3.27.2/dumptree.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ import GeometryFactory from 'jsts/org/locationtech/jts/geom/GeometryFactory.js';
import GeoJSONWriter from 'jsts/org/locationtech/jts/io/GeoJSONWriter.js';
import {readFileSync, writeFileSync} from 'fs';

import {magicbytes, SIZE_PREFIX_LEN} from './constants.js';
import {fromByteBuffer} from './header-meta.js';
import {calcTreeSize, generateLevelBounds} from './packedrtree.js';
import {magicbytes, SIZE_PREFIX_LEN} from './constants';
import {fromByteBuffer} from './header-meta';
import {calcTreeSize, generateLevelBounds} from './packedrtree';

const buffer = readFileSync('./test/data/tiger_roads.fgb');
const bytes = new Uint8Array(buffer);
Expand Down
2 changes: 1 addition & 1 deletion modules/flatgeobuf/src/flatgeobuf/3.27.2/feature.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
// automatically generated by the FlatBuffers compiler, do not modify

export * as FlatGeobuf from './flat-geobuf.js';
export * as FlatGeobuf from './flat-geobuf';
10 changes: 5 additions & 5 deletions modules/flatgeobuf/src/flatgeobuf/3.27.2/flat-geobuf.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// automatically generated by the FlatBuffers compiler, do not modify

export {Column} from './flat-geobuf/column.js';
export {ColumnType} from './flat-geobuf/column-type.js';
export {Crs} from './flat-geobuf/crs.js';
export {GeometryType} from './flat-geobuf/geometry-type.js';
export {Header} from './flat-geobuf/header.js';
export {Column} from './flat-geobuf/column';
export {ColumnType} from './flat-geobuf/column-type';
export {Crs} from './flat-geobuf/crs';
export {GeometryType} from './flat-geobuf/geometry-type';
export {Header} from './flat-geobuf/header';
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import * as flatbuffers from 'flatbuffers';

import {ColumnType} from '../flat-geobuf/column-type.js';
import {ColumnType} from '../flat-geobuf/column-type';

export class Column {
bb: flatbuffers.ByteBuffer | null = null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

import * as flatbuffers from 'flatbuffers';

import {Column} from '../flat-geobuf/column.js';
import {Geometry} from '../flat-geobuf/geometry.js';
import {Column} from '../flat-geobuf/column';
import {Geometry} from '../flat-geobuf/geometry';

export class Feature {
bb: flatbuffers.ByteBuffer | null = null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import * as flatbuffers from 'flatbuffers';

import {GeometryType} from '../flat-geobuf/geometry-type.js';
import {GeometryType} from '../flat-geobuf/geometry-type';

export class Geometry {
bb: flatbuffers.ByteBuffer | null = null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

import * as flatbuffers from 'flatbuffers';

import {Column} from '../flat-geobuf/column.js';
import {Crs} from '../flat-geobuf/crs.js';
import {GeometryType} from '../flat-geobuf/geometry-type.js';
import {Column} from '../flat-geobuf/column';
import {Crs} from '../flat-geobuf/crs';
import {GeometryType} from '../flat-geobuf/geometry-type';

export class Header {
bb: flatbuffers.ByteBuffer | null = null;
Expand Down
14 changes: 7 additions & 7 deletions modules/flatgeobuf/src/flatgeobuf/3.27.2/generic.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ import {
deserializeStream,
deserializeFiltered,
FromFeatureFn
} from './generic/featurecollection.js';
} from './generic/featurecollection';

import {Rect} from './packedrtree.js';
import {IFeature} from './generic/feature.js';
import HeaderMeta from './header-meta.js';
import {Rect} from './packedrtree';
import {IFeature} from './generic/feature';
import HeaderMeta from './header-meta';

export type HeaderMetaFn = (headerMeta: HeaderMeta) => void;

Expand All @@ -27,7 +27,7 @@ export function deserialize(
return deserializeFiltered(input, rect as Rect, fromFeature);
}

export {serialize} from './generic/featurecollection.js';
export {serialize} from './generic/featurecollection';

export {GeometryType} from './flat-geobuf/geometry-type.js';
export {ColumnType} from './flat-geobuf/column-type.js';
export {GeometryType} from './flat-geobuf/geometry-type';
export {ColumnType} from './flat-geobuf/column-type';
10 changes: 5 additions & 5 deletions modules/flatgeobuf/src/flatgeobuf/3.27.2/generic/feature.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import * as flatbuffers from 'flatbuffers';

import ColumnMeta from '../column-meta.js';
import {ColumnType} from '../flat-geobuf/column-type.js';
import {Feature} from '../flat-geobuf/feature.js';
import HeaderMeta from '../header-meta.js';
import {buildGeometry, ISimpleGeometry, ICreateGeometry, IParsedGeometry} from './geometry.js';
import ColumnMeta from '../column-meta';
import {ColumnType} from '../flat-geobuf/column-type';
import {Feature} from '../flat-geobuf/feature';
import HeaderMeta from '../header-meta';
import {buildGeometry, ISimpleGeometry, ICreateGeometry, IParsedGeometry} from './geometry';

const textEncoder = new TextEncoder();
const textDecoder = new TextDecoder();
Expand Down
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
import * as flatbuffers from 'flatbuffers';
import slice from 'slice-source';

import ColumnMeta from '../column-meta.js';

import {Header} from '../flat-geobuf/header.js';

import {Column} from '../flat-geobuf/column.js';
import {ColumnType} from '../flat-geobuf/column-type.js';
import {Feature} from '../flat-geobuf/feature.js';
import HeaderMeta, {fromByteBuffer} from '../header-meta.js';

import {buildFeature, IFeature} from './feature.js';
import {HttpReader} from '../http-reader.js';
import Logger from '../logger.js';
import {Rect, calcTreeSize} from '../packedrtree.js';
import {parseGeometry} from './geometry.js';
import {HeaderMetaFn} from '../generic.js';
import {magicbytes, SIZE_PREFIX_LEN} from '../constants.js';
import {inferGeometryType} from './header.js';
import ColumnMeta from '../column-meta';

import {Header} from '../flat-geobuf/header';

import {Column} from '../flat-geobuf/column';
import {ColumnType} from '../flat-geobuf/column-type';
import {Feature} from '../flat-geobuf/feature';
import HeaderMeta, {fromByteBuffer} from '../header-meta';

import {buildFeature, IFeature} from './feature';
import {HttpReader} from '../http-reader';
import Logger from '../logger';
import {Rect, calcTreeSize} from '../packedrtree';
import {parseGeometry} from './geometry';
import {HeaderMetaFn} from '../generic';
import {magicbytes, SIZE_PREFIX_LEN} from '../constants';
import {inferGeometryType} from './header';

export type FromFeatureFn = (feature: Feature, header: HeaderMeta) => IFeature;
type ReadFn = (size: number, purpose: string) => Promise<ArrayBuffer>;
Expand Down
4 changes: 2 additions & 2 deletions modules/flatgeobuf/src/flatgeobuf/3.27.2/generic/geometry.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import * as flatbuffers from 'flatbuffers';
import {GeometryType} from '../flat-geobuf/geometry-type.js';
import {Geometry} from '../flat-geobuf/geometry.js';
import {GeometryType} from '../flat-geobuf/geometry-type';
import {Geometry} from '../flat-geobuf/geometry';

export interface IParsedGeometry {
xy: number[];
Expand Down
8 changes: 4 additions & 4 deletions modules/flatgeobuf/src/flatgeobuf/3.27.2/generic/header.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import {GeometryType} from '../flat-geobuf/geometry-type.js';
import {toGeometryType} from '../generic/geometry.js';
import {IFeature} from './feature.js';
import {IGeoJsonFeature} from '../geojson/feature.js';
import {GeometryType} from '../flat-geobuf/geometry-type';
import {toGeometryType} from '../generic/geometry';
import {IFeature} from './feature';
import {IGeoJsonFeature} from '../geojson/feature';

function featureGeomType(feature: IFeature | IGeoJsonFeature): GeometryType {
if (feature.getGeometry) {
Expand Down
Loading
Loading