Skip to content

Commit

Permalink
Adding original description of append mode
Browse files Browse the repository at this point in the history
  • Loading branch information
urvisavla committed Jun 27, 2024
1 parent c167658 commit bbf04f8
Showing 1 changed file with 19 additions and 12 deletions.
31 changes: 19 additions & 12 deletions exp/services/ledgerexporter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,15 +75,22 @@ docker run --platform linux/amd64 \
* `${PWD}/config.toml`: Your local configuration file.
* `-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json`: Sets the environment variable for credentials within the container.
* `stellar/ledger-exporter`: The Docker image name.
* `<command>`: The Stellar Ledger Exporter command: [scan-and-fill](#1-scan-and-fill-fill-data-gaps), [append](#2-append-continuously-export-new-data))
* `<command>`: The Stellar Ledger Exporter command: [append](#1-append-continuously-export-new-data), [scan-and-fill](#2-scan-and-fill-fill-data-gaps))

## Command Line Interface (CLI)

The Ledger Exporter offers two mode of operation for exporting ledger data:

### 1. scan-and-fill: Fill Data Gaps
### 1. append: Continuously Export New Data


Exports ledgers initially searching from --start, looking for the next absent ledger sequence number proceeding --start on the data store. If abscence is detected, the export range is narrowed to `--start <absent_ledger_sequence>`.
This feature requires ledgers to be present on the remote data store for some (possibly empty) prefix of the requested range and then absent for the (possibly empty) remainder.

In this mode, the --end ledger can be provided to stop the process once export has reached that ledger, or if absent or 0 it will result in continous exporting of new ledgers emitted from the network.

It’s guaranteed that ledgers exported during `append` mode from `start` and up to the last logged ledger file `Uploaded {ledger file name}` were contiguous, meaning all ledgers within that range were exported to the data lake with no gaps or missing ledgers in between.

Scans the datastore (GCS bucket) for the specified ledger range and exports any missing ledgers to the datastore. This mode avoids unnecessary exports if the data is already present. The range is specified using the --start and --end options.

**Usage:**

Expand All @@ -93,18 +100,17 @@ docker run --platform linux/amd64 -d \
-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \
-v ${PWD}/config.toml:/config.toml \
stellar/ledger-exporter \
scan-and-fill --start <start_ledger> --end <end_ledger> [--config-file <config_file>]
append --start <start_ledger> [--end <end_ledger>] [--config-file <config_file>]
```

Arguments:
- `--start <start_ledger>` (required): The starting ledger sequence number in the range to export.
- `--end <end_ledger>` (required): The ending ledger sequence number in the range.
- `--start <start_ledger>` (required): The starting ledger sequence number for the export process.
- `--end <end_ledger>` (optional): The ending ledger sequence number. If omitted or set to 0, the exporter will continuously export new ledgers as they appear on the network.
- `--config-file <config_file_path>` (optional): The path to your configuration file, containing details like GCS bucket information. If not provided, the exporter will look for config.toml in the directory where you run the command.
### 2. append: Continuously Export New Data

Exports ledgers starting from the specified start ledger, searching for the next absent ledger sequence number in the datastore (GCS bucket). If an absence is found, the export range is narrowed to start from the absent ledger
### 2. scan-and-fill: Fill Data Gaps

The range is specified using the `--start` and `--end` options. If `--end` is absent or set to 0, exporting will continue for new ledgers emitted by the network. Providing an --end ledger will stop the process once export reaches that ledger.
Scans the datastore (GCS bucket) for the specified ledger range and exports any missing ledgers to the datastore. This mode avoids unnecessary exports if the data is already present. The range is specified using the --start and --end options.

**Usage:**

Expand All @@ -114,10 +120,11 @@ docker run --platform linux/amd64 -d \
-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \
-v ${PWD}/config.toml:/config.toml \
stellar/ledger-exporter \
append --start <start_ledger> [--end <end_ledger>] [--config-file <config_file>]
scan-and-fill --start <start_ledger> --end <end_ledger> [--config-file <config_file>]
```

Arguments:
- `--start <start_ledger>` (required): The starting ledger sequence number for the export process.
- `--end <end_ledger>` (optional): The ending ledger sequence number. If omitted or set to 0, the exporter will continuously export new ledgers as they appear on the network.
- `--start <start_ledger>` (required): The starting ledger sequence number in the range to export.
- `--end <end_ledger>` (required): The ending ledger sequence number in the range.
- `--config-file <config_file_path>` (optional): The path to your configuration file, containing details like GCS bucket information. If not provided, the exporter will look for config.toml in the directory where you run the command.

0 comments on commit bbf04f8

Please sign in to comment.