Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructured the integrations chapter #8905

Merged
merged 3 commits into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions ydb/docs/en/core/integrations/gui/_includes/toc-table.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Graphical user interfaces

| Environment | Instruction | Compatibility level |
| --- | --- | --- |
| Embedded UI | [Instruction](../../../reference/embedded-ui/index.md) | |
| [DBeaver](https://dbeaver.com) | [Instruction](../dbeaver.md) | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
| JetBrains Database viewer | — | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
| [JetBrains DataGrip](https://www.jetbrains.com/datagrip/) | [Instruction](../datagrip.md) | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
| Other JDBC-compatible IDEs | — | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
3 changes: 3 additions & 0 deletions ydb/docs/en/core/integrations/gui/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Graphical user interfaces

{% include notitle [Table of contents](_includes/toc-table.md) %}
5 changes: 5 additions & 0 deletions ydb/docs/en/core/integrations/gui/toc-ide.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
items:
- name: DBeaver
href: dbeaver.md
- name: DataGrip
href: datagrip.md
46 changes: 13 additions & 33 deletions ydb/docs/en/core/integrations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,51 +9,31 @@ In addition to its own native protocol, {{ ydb-name }} has a compatibility layer
{% endnote %}


## Graphical Interface Clients {#gui}
## Graphical user interfaces {#gui}

| Environment | Instruction | Compatibility Level |
| --- | --- | --- |
| Embedded UI | [Instruction](../reference/embedded-ui/index.md) | |
| [DBeaver](https://dbeaver.com) | [Instruction](ide/dbeaver.md) | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
| JetBrains Database viewer | — | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
| [JetBrains DataGrip](https://www.jetbrains.com/datagrip/) | [Instruction](ide/datagrip.md) | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
| Other JDBC-compatible IDEs | — | By [JDBC-driver](https://github.com/ydb-platform/ydb-jdbc-driver/releases)|
{% include notitle [Table of contents](gui/_includes/toc-table.md) %}

## Data visualization (Business intelligence, BI) {#bi}

## Data Visualization (Business Intelligence, BI) {#bi}
{% include notitle [Table of contents](visualization/_includes/toc-table.md) %}

| Environment | Compatibility Level | Instruction |
| --- | :---: | --- |
| [Grafana](https://grafana.com) | Full| [Instruction](grafana.md) |
## Data ingestion {#ingestion}

{% include notitle [Table of contents](ingestion/_includes/toc-table.md) %}

## Data Ingestion {#ingestion}
### Streaming data ingestion {#streaming-ingestion}

| Delivery System | Instruction |
| --- | --- |
| [FluentBit](https://fluentbit.io) | [Instruction](fluent-bit.md) |
| [LogStash](https://www.elastic.co/logstash) | [Instruction](logstash.md) |
| [Kafka Connect Sink](https://docs.confluent.io/platform/current/connect/index.html) | [Instruction](https://github.com/ydb-platform/ydb-kafka-sink-connector) |
| Arbitrary [JDBC data sources](https://en.wikipedia.org/wiki/Java_Database_Connectivity) | [Instruction](import-jdbc.md) |
{% include notitle [Table of contents](ingestion/_includes/toc-table-streaming.md) %}

## Data migrations {#schema_migration}

### Streaming Data Ingestion {#streaming-ingestion}
{% include notitle [Table of contents](migration/_includes/toc-table.md) %}

| Delivery System | Instruction |
| --- | --- |
| [Apache Kafka API](https://kafka.apache.org) | [Instruction](../reference/kafka-api/index.md) |
## Object–relational mapping (ORM) {#orm}

{% include notitle [Table of contents](orm/_includes/toc-table.md) %}

## Data Migrations {#schema_migration}

| Environment | Instruction |
| --- | --- |
| [goose](https://github.com/pressly/goose/) | [Instruction](goose.md) |
| [Liquibase](https://www.liquibase.com) | [Instruction](liquibase.md) |
| [Flyway](https://documentation.red-gate.com/fd/) | [Instruction](flyway.md) |
| [Hibernate](https://hibernate.org/orm/) | [Instruction](hibernate.md) |

## See Also
## See also

* [{#T}](../reference/ydb-sdk/index.md)
* [{#T}](../postgresql/intro.md)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Streaming data ingestion

| Delivery System | Instruction |
| --- | --- |
| [Apache Kafka API](https://kafka.apache.org) | [Instruction](../../../reference/kafka-api/index.md) |
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Data ingestion

| Delivery System | Instruction |
| --- | --- |
| [FluentBit](https://fluentbit.io) | [Instruction](../fluent-bit.md) |
| [LogStash](https://www.elastic.co/logstash) | [Instruction](../logstash.md) |
| [Kafka Connect Sink](https://docs.confluent.io/platform/current/connect/index.html) | [Instruction](https://github.com/ydb-platform/ydb-kafka-sink-connector) |
| Arbitrary [JDBC data sources](https://en.wikipedia.org/wiki/Java_Database_Connectivity) | [Instruction](../import-jdbc.md) |
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,9 @@ CREATE TABLE `fluent-bit/log` (
The command that creates the columnar table differs in the following details:

* it specifies the columnar storage type and the table's partitioning key in the last two lines;
* the `timestamp` column is the first column of the primary key, which is optimal and recommended for columnar, but not for row-based tables. See the specific guidelines for choosing the primary key [for columnar tables](../dev/primary-key/column-oriented.md) and [for row-based tables](../dev/primary-key/row-oriented.md).
* the `timestamp` column is the first column of the primary key, which is optimal and recommended for columnar, but not for row-based tables. See the specific guidelines for choosing the primary key [for columnar tables](../../dev/primary-key/column-oriented.md) and [for row-based tables](../../dev/primary-key/row-oriented.md).

[TTL configuration](../concepts/ttl.md) can be optionally applied to the table, limiting the data storage period and enabling the automatic removal of obsolete data. Enabling TTL requires an extra setting in the `WITH` section of the table creation command. For example, `TTL = Interval("P14D") ON timestamp` sets the storage period to 14 days, based on the `timestamp` field's value.
[TTL configuration](../../concepts/ttl.md) can be optionally applied to the table, limiting the data storage period and enabling the automatic removal of obsolete data. Enabling TTL requires an extra setting in the `WITH` section of the table creation command. For example, `TTL = Interval("P14D") ON timestamp` sets the storage period to 14 days, based on the `timestamp` field's value.


## FluentBit deployment and configuration
Expand Down Expand Up @@ -97,7 +97,7 @@ The table below lists the configuration parameters supported by the {{ ydb-short
|----------|--------------|
| `Name` | Plugin type, should be value `ydb` |
| `Match` | (optional) [Tag matching expression](https://docs.fluentbit.io/manual/concepts/data-pipeline/router) to select log records which should be routed to {{ ydb-short-name }} |
| `ConnectionURL` | YDB connection URL, including the protocol, endpoint, and database path (see the [documentation](../concepts/connect.md)) |
| `ConnectionURL` | YDB connection URL, including the protocol, endpoint, and database path (see the [documentation](../../concepts/connect.md)) |
| `TablePath` | Table path starting from database root (example: `fluent-bit/log`) |
| `Columns` | JSON structure mapping the fields of FluentBit record to the columns of the target YDB table. May include the pseudo-columns listed below |
| `CredentialsAnonymous` | Configured as `1` for anonymous {{ ydb-short-name }} authentication |
Expand Down Expand Up @@ -126,7 +126,7 @@ Example of `Columns` parameter value:

FluentBit is often used to collect logs in the Kubernetes environment. Below is the schema of the log delivery process, implemented using FluentBit and {{ ydb-short-name }}, for applications running in the Kubernetes cluster:

![FluentBit in Kubernetes cluster](../_assets/fluent-bit-ydb-scheme.png)
![FluentBit in Kubernetes cluster](../../_assets/fluent-bit-ydb-scheme.png)
<small>Figure 1 — Interaction diagram between FluentBit and {{ ydb-short-name }} in the Kubernetes cluster</small>

In this diagram:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ To run the import tool, JDK 8 or later is required, with the `java` executable a

Import tool binary distribution is available on the [Releases page](https://github.com/ydb-platform/ydb-importer/releases). The import tool is provided as the ZIP archive, which needs to be unpacked into the local directory on the server where the tool is required to run.

The import tool distribution archive contains the configuration file examples as `*.xml` files, sample tool startup script `ydb-importer.sh`, and the executable code for the tool and its dependencies (including the [YDB Java SDK](../reference/ydb-sdk/index.md)) as `lib/*.jar` files.
The import tool distribution archive contains the configuration file examples as `*.xml` files, sample tool startup script `ydb-importer.sh`, and the executable code for the tool and its dependencies (including the [YDB Java SDK](../../reference/ydb-sdk/index.md)) as `lib/*.jar` files.

Before running the import tool, the JDBC drivers for the source databases should be put (as `*.jar` files) into the `lib` subdirectory of the tool installation directory.

Expand Down
7 changes: 7 additions & 0 deletions ydb/docs/en/core/integrations/ingestion/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Data ingestion {#ingestion}

{% include notitle [Table of contents](_includes/toc-table.md) %}

## Streaming data ingestion {#streaming-ingestion}

{% include notitle [Table of contents](_includes/toc-table-streaming.md) %}
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ This section describes the integration options between {{ ydb-short-name }} and

[Logstash](https://www.elastic.co/logstash) dynamically ingests, transforms, and ships data regardless of format or complexity. A Logstash pipeline can contain different types of plugins: input, output, and filter. The {{ ydb-short-name }} Logstash plugins repository is hosted on [GitHub](https://github.com/ydb-platform/ydb-logstash-plugins) and contains the following plugins:

* **Storage Plugin** for persisting Logstash events in [row-oriented](../concepts/datamodel/table.md#row-oriented-table) or [column-oriented](../concepts/datamodel/table.md#olap-data-types) {{ ydb-short-name }} tables;
* **Input Topic Plugin** for reading Logstash events from {{ ydb-short-name }} [topics](../concepts/topic.md);
* **Output Topic Plugin** for sending Logstash events to {{ ydb-short-name }} [topics](../concepts/topic.md).
* **Storage Plugin** for persisting Logstash events in [row-oriented](../../concepts/datamodel/table.md#row-oriented-table) or [column-oriented](../../concepts/datamodel/table.md#olap-data-types) {{ ydb-short-name }} tables;
* **Input Topic Plugin** for reading Logstash events from {{ ydb-short-name }} [topics](../../concepts/topic.md);
* **Output Topic Plugin** for sending Logstash events to {{ ydb-short-name }} [topics](../../concepts/topic.md).

These plugins can be [built](https://github.com/ydb-platform/ydb-logstash-plugins/blob/main/BUILD.md) from the source code or downloaded as pre-built [artifacts](https://github.com/ydb-platform/ydb-logstash-plugins/releases) for the two latest versions of Logstash.

Expand All @@ -31,7 +31,7 @@ The command will return a list of all installed plugins, which contain the plugi

## Configure {{ ydb-short-name }} connection

All plugins use the same set of parameters to configure the connection to {{ ydb-short-name }}. This set contains only one required parameter, `connection_string`. Other parameters are optional and allow configuring [an authentication mode](../concepts/auth.md). An anonymous mode will be used if the configuration doesn't contain any of these parameters.
All plugins use the same set of parameters to configure the connection to {{ ydb-short-name }}. This set contains only one required parameter, `connection_string`. Other parameters are optional and allow configuring [an authentication mode](../../concepts/auth.md). An anonymous mode will be used if the configuration doesn't contain any of these parameters.

```ruby
# This example demonstrates configuration for ydb_storage plugin.
Expand All @@ -52,7 +52,7 @@ ydb_storage {

## {{ ydb-short-name }} Storage Plugin

This plugin allows storing the Logstash events stream in {{ ydb-short-name }} tables for further analysis. This is especially useful with [column-oriented tables](../concepts/datamodel/table.md#column-tables) optimized for handling Online Analytical Processing (OLAP) requests. Every [field](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) of a Logstash event will be stored in a column with a corresponding name. Fields that do not match any column will be ignored.
This plugin allows storing the Logstash events stream in {{ ydb-short-name }} tables for further analysis. This is especially useful with [column-oriented tables](../../concepts/datamodel/table.md#column-tables) optimized for handling Online Analytical Processing (OLAP) requests. Every [field](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) of a Logstash event will be stored in a column with a corresponding name. Fields that do not match any column will be ignored.

### Configuration

Expand All @@ -64,7 +64,7 @@ The plugin configuration is done by adding a `ydb_storage` block in the `output`

{% note warning %}

The Storage plugin doesn't check if the Logstash event has correct and unique values for the table's primary key. It uses the [bulk upsert method](../dev/batch-upload.md) for storing data in {{ ydb-short-name }}, and if multiple events have the same primary keys, they will overwrite each other. The primary key is recommended to contain those event fields that will be present in every event and can uniquely identify each event. If no such set of fields exists, add an extra column to the primary key and fill it with a random value using the `uuid_column` parameter.
The Storage plugin doesn't check if the Logstash event has correct and unique values for the table's primary key. It uses the [bulk upsert method](../../dev/batch-upload.md) for storing data in {{ ydb-short-name }}, and if multiple events have the same primary keys, they will overwrite each other. The primary key is recommended to contain those event fields that will be present in every event and can uniquely identify each event. If no such set of fields exists, add an extra column to the primary key and fill it with a random value using the `uuid_column` parameter.

{% endnote %}

Expand Down Expand Up @@ -121,7 +121,7 @@ All commands return `ok` if the messages are sent.

#### Check that the messages are stored in {{ ydb-short-name }}

To check that all sent messages are written to the table, execute the following query using [ScanQuery mode](../reference/ydb-cli/commands/scan-query.md):
To check that all sent messages are written to the table, execute the following query using [ScanQuery mode](../../reference/ydb-cli/commands/scan-query.md):
```sql
SELECT * FROM `logstash_demo`;
```
Expand All @@ -139,14 +139,14 @@ The query will return a list of written events:

## {{ ydb-short-name }} Topic Input Plugin

This plugin allows reading messages from {{ ydb-short-name }} [topics](../concepts/topic.md) and transforming them into `Logstash` events.
This plugin allows reading messages from {{ ydb-short-name }} [topics](../../concepts/topic.md) and transforming them into `Logstash` events.

### Configuration

The plugin configuration is done by adding a `ydb_topic` block to the `input` section of the Logstash pipeline [config](https://www.elastic.co/guide/en/logstash/current/configuration.html). The plugin supports the standard set of [connection parameters](#configure-ydb-connection) and a few additional options:

* `topic_path` — the required the full path of the topic for reading.
* `consumer_name` — the required the name of the topic [consumer](../concepts/topic.md#consumer).
* `consumer_name` — the required the name of the topic [consumer](../../concepts/topic.md#consumer).
* `schema` — the optional mode for processing of the {{ ydb-short-name }} events. By default, the plugin reads and sends messages as binary data, but if you specify the `JSON` mode, each message will be parsed as a JSON object.

### Usage example
Expand Down Expand Up @@ -205,7 +205,7 @@ The `stdout` plugin writes the messages to the Logstash logs:

## {{ ydb-short-name }} Topic Output Plugin

This plugin allows writing `Logstash` events to a {{ ydb-short-name }} [topic](../concepts/topic.md).
This plugin allows writing `Logstash` events to a {{ ydb-short-name }} [topic](../../concepts/topic.md).

### Configuration

Expand Down
7 changes: 7 additions & 0 deletions ydb/docs/en/core/integrations/ingestion/toc-ingestion.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
items:
- name: Log records collection using FluentBit
href: fluent-bit.md
- name: Logstash plugins
href: logstash.md
- name: Importing from JDBC data sources
href: import-jdbc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Data migrations

| Environment | Instruction |
| --- | --- |
| [goose](https://github.com/pressly/goose/) | [Instruction](../goose.md) |
| [Liquibase](https://www.liquibase.com) | [Instruction](../liquibase.md) |
| [Flyway](https://documentation.red-gate.com/fd/) | [Instruction](../flyway.md) |
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ As a result, `series`, `season`, and `episode` tables will be created and filled

![_assets/flyway-migrate-step-1.png](_assets/flyway-migrate-step-1.png)

Then, we evolve the schema by adding a [secondary index](../yql/reference/syntax/alter_table.md) to the `series` table:
Then, we evolve the schema by adding a [secondary index](../../yql/reference/syntax/alter_table.md) to the `series` table:

```
db/migration:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ There are alternative options to see the applied changes:

- Using YDB UI on http://localhost:8765

![YDB UI after the first migration](../_assets/goose-ydb-ui-after-first-migration.png =450x)
![YDB UI after the first migration](_assets/goose-ydb-ui-after-first-migration.png =450x)

- Using YDB CLI

Expand Down Expand Up @@ -258,7 +258,7 @@ Let's use the same methods to see the new changes:

- Using YDB UI on http://localhost:8765

![YDB UI after apply second migration](../_assets/goose-ydb-ui-after-second-migration.png =450x)
![YDB UI after apply second migration](_assets/goose-ydb-ui-after-second-migration.png =450x)

- Using YDB CLI

Expand Down Expand Up @@ -322,7 +322,7 @@ Let's check the changes again:

- Using YDB UI on http://localhost:8765

![YDB UI after apply first migration](../_assets/goose-ydb-ui-after-first-migration.png =450x)
![YDB UI after apply first migration](_assets/goose-ydb-ui-after-first-migration.png =450x)

- Using YDB CLI

Expand Down
3 changes: 3 additions & 0 deletions ydb/docs/en/core/integrations/migration/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Data migrations

{% include notitle [Table of contents](_includes/toc-table.md) %}
Loading
Loading