Make releases EZ

.github/workflows/release.yml

@@ -0,0 +1,60 @@
+name: Release CI
+on:
+  push:
+    branches:
+      - release
+jobs:
+  deploy:
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Install Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          profile: minimal
+
+      - name: Build Polkadot Archive
+        run: cargo build --release --manifest-path bin/polkadot-archive/Cargo.toml
+
+      - name: Prepare binary
+        run: |
+          mkdir -p release
+          strip bin/polkadot-archive/target/release/polkadot-archive
+
+      - name: Package for Debian
+        if: matrix.os == 'ubuntu-latest'
+        run: |
+          cargo install cargo-deb
+          cargo deb --manifest-path ./bin/polkadot-archive/Cargo.toml
+          mv ./bin/polkadot-archive/target/debian/*.deb ./release/polkadot-archive-linux.deb
+          tar -C ./bin/polkadot-archive/target/release/ -czvf ./release/polkadot-archive-linux.tar.gz ./polkadot-archive
+
+      - name: Package just binary
+        if: matrix.os == 'ubuntu-latest'
+        run: |
+          mv bin/polkadot-archive/target/release/polkadot-archive ./release/polkadot-archive
+
+      - name: Get changelog
+        id: changelog_reader
+        uses: mindsers/changelog-reader-action@v2
+        with:
+          validation_depth: 2
+          path: ./CHANGELOG.md
+
+      - name: Release
+        uses: softprops/action-gh-release@v1
+        with:
+          tag_name: ${{ steps.changelog_reader.outputs.version }}
+          name: Release ${{ steps.changelog_reader.outputs.version }}
+          body: ${{ steps.changelog_reader.outputs.changes }}
+          draft: true
+          files: |
+            ./release/*.tar.gz
+            ./release/*.deb
+            ./release/polkadot_archive
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

CHANGELOG.md

@@ -1,83 +1,83 @@
-# Changelog for `substrate-archive`
-
-## **[Unreleased]**
-
-## **[v0.5.1]**
-
-- [Added] Release checklist
-- [Changed] Pinned to `substrate`/`polkadot` release v0.8.30
-- [Changed] Keeping the `substrate-archive` and `substrate-archive-backend` crates's versions aligned.
-
-## **[v0.5.0]**
-
-### Substrate Archive
-
-- [Changed] Use SQLX for migrations
-  - this will create a new table in the database, `sqlx_migrations` or similiar.
-    - the older table, `__refinery_migrations` can be safely dropped
-
-#### Api Changes
-
-- [Added] New `ArchiveBuilder` struct for constructing the indexer.
-  - [Changed] returns a trait rather than a struct.
-    - [Changed] the returned trait,`Archive` manages the underlying actor runtime
-  - [Changed] no longer need to instantiate a client and manually pass it to the Archive
-  - [Changed] rename `run_with` to `drive`
-  - [Changed] Archive now accepts just a postgres URL instead of a postgres URL split into its parts. This should
-  make configuring the archive more straightforward. Takes from environment variable `DATABASE_URL` if not passed to the
-  archive directly
-- [Added] Archive now reads the `CHAIN_DATA_DB` environment variable if the path to the backend chain database is not passed directly.
-- [Removed] Archive no longer needs an RPC url to function
-- [Changed] Print out the Postgres URL at startup
-- [Added] `max_block_load` to configure the maximum number of blocks loaded at once
-- [Added] Two new options to the configuration relative to state tracing:
-  - `targets` for specifying runtime targets to trace in WASM
-  - `folder` where WASM blobs with tracing enabled are kept.
-  - More on state-tracing [here](https://github.com/paritytech/substrate-archive/wiki/6.\)-State-Tracing-&-Balance-Reconciliation)
-- [Changed] Archive config is now separated into sections for readability. Migration is manual but looking at the new `archive.conf` in `polkadot-archive` or `node-template-archive` folders should help.
-
-#### Internal Changes
-
-- [Changed] Postgres SQL queries are now type checked
-- [QoL] Refactor file layout to `substrate-archive` and `substrate-archive-backend`.
-- [perf] Decouple Database actors
-- [QoL] upgrade to SQLx 0.5.0
-- [perf] Overhaul of block indexing. Now uses a Iterator to only collect batches of blocks from the database,
-taking advantage of the better reading performance of sequential data access. Gathering blocks by RPC is no longer done.
-- [perf] a new module `runtime_version_cache` is introduced in order to cache and run a binary search on runtime version & blocks.
-- [perf] better queries for the set difference between the storage and blocks table - makes querying for missing storage more efficient
-- [err] Better handling of SQL errors
-- [perf] switch to a leaner, 'lower-level' actor framework (xtra)
-- [perf] switch to a background-task-queue for executing blocks. This uses significantly less memory and
-  persists blocks that need to be executed on-disk.
-- [QoL] remove the last frame dependency, `frame-system`. Archive now relies only on generic traits defined in substrate-core.
-- new method of batch inserts avoids starving the Postgres Pool of connections
-- [perf] Notification stream from Postgres indexes storage changes in the background.
-- [Changed] Switch substrate-archive `LocalCallExecutor` with Substrate's `LocalCallExecutor`
-
-### Polkadot Archive
-
-- [Changed] Config file is now optional. Can configure polkadot archive entirely through environment variables.
-  - the environment variables that need to be set are `CHAIN_DATA_DB` and `DATABASE_URL`.
-- [Changed] Polkadot archive will archive `polkadot` by default if the `--chain` CLI option is not passed.
-- [Changed] remove `rpc_url` from the polkadot-archive TOML configuration file
-- [Changed] All options in config file apart from `db_url`.
-
-## **[v0.4.0]**
+# Changelog
+
+All notable changes for substrate-archive will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [v0.5.1] - 2021-05-06
+### Added
+- Release checklist
+
+### Changed
+- Pinned to `substrate`/`polkadot` release v0.8.30
+- Keeping the `substrate-archive` and `substrate-archive-backend` crates's versions aligned.
+
+## [v0.5.0] - 2021-03-29
+### Added
+- `ArchiveBuilder` struct for constructing the indexer.
+	- returns a trait, `Archive` that manages the underlying actor runtime
+- Archive now reads the `CHAIN_DATA_DB` environment variable if the path to the backend chain database is not passed directly.
+- `max_block_load` to configure the maximum number of blocks loaded at once
+- Two new options to the configuration relative to state tracing:
+	- `targets` for specifying runtime targets to trace in WASM
+	- `folder` where WASM blobs with tracing enabled are kept.
+	- More on state-tracing [here](https://github.com/paritytech/substrate-archive/wiki/6.\)-State-Tracing-&-Balance-Reconciliation)
+- Runtime versions between blocks are cached, speeding up block indexing.
+- Notification stream from Postgres indexes storage changes in the background.
+
+### Changed
+- no longer need to instantiate a client and manually pass it to the Archive
+- rename `run_with` to `drive`
+- Archive now accepts a postgres URL instead of a postgres URL split into its parts.
+- Archive will take Postgres URL from environment variable `DATABASE_URL` as a fallback.
+- Use SQLX for migrations
+	- SQLX will create a new table in the database, `sqlx_migrations` or similiar.
+	- the older table, `__refinery_migrations` can be safely dropped
+- Print out the Postgres URL at startup
+- config is now separated into sections for readability. Migration is manual but looking at the new `archive.conf` in `polkadot-archive` or `node-template-archive` folders should help.
+- Postgres SQL queries are now type checked
+- Refactor file layout to `substrate-archive` and `substrate-archive-backend`.
+- Decouple Database actors
+- upgrade to SQLx 0.5.0
+- Overhaul of block indexing. Now uses a Iterator to only collect batches of blocks from the database,
+	taking advantage of the better reading performance of sequential data access. Gathering blocks by RPC is no longer done.
+- speed up query for difference between the storage and blocks table -
+- switch to a leaner actor framework, [xtra](https://github.com/Restioson/xtra)
+- switch to a persistant background-task-queue for executing blocks, [coil](https://github.com/insipx/coil).
+- batch inserts no longer starve postgres pool of connections
+- Switch substrate-archive `LocalCallExecutor` with Substrate's `LocalCallExecutor`
+
+### Removed
+- the last frame dependency, `frame-system`. Archive now relies only on generic traits defined in substrate-core.
+- RPC URL. An RPC URL is no longer required to function.
+
+## [v0.4.0] - 2021-01-24
+### Added
+- integrate database migrations into substrate-archive library
 
+### Changed
 - Speed up Storage Indexing by re-executing blocks
-- [internal] Remove some depdendencies for service and backend
-- fixes for storage inserts
 - refactored error types
-- integrate database migrations into substrate-archive library
 
-## [v0.3.1]
+### Removed
+- dependencies for service and backend
 
-- add node-template-archive
+### Fixed
+- storage inserts
 
-## **[v0.3.0]**
+## [v0.3.1]
+### Added
+- node-template-archive
 
-- Use a rocksdb-backed substrate client instead of RPC for indexing
+## [v0.3.0]
+### Added
 - Create a CLI for indexing kusama
 - New PostgreSQL Schema
 - Actors to model dataflow
+
+### Changed
+- Use a rocksdb-backed substrate client instead of RPC for indexing
+

RELEASE-CHECKLIST.md

@@ -5,24 +5,15 @@ Here's how to make a new release of `substrate-archive`.
 ## Checklist
 
 1. Make a new branch, name it whatever you want, but `vx.y.z-prep` is a good template
-1. Update `Cargo.toml` in `substrate-archive` and `substrate-archive-backend` with the new version number.
-1. Update all references to `substrate` crates and `polkadot` to their respective latest releases.
-1. Update the `CHANGELOG` as specified [here](https://keepachangelog.com/en/1.0.0/).
-1. Run all tests: `TEST_DATABASE_URL="postgres://localhost:5432/archive cargo test --all`.
-1. Push the PR against the `release` branch.
-1. Once reviewed, merge it to `release`.
-1. Tag the `release` branch` with `git tag vx.y.z` and push the tags with `git push --tags`
-1. Build a binary for debian, compatible with the current glibc version (`v2.31`)
-    1. `docker run --rm -it debian:jessie` (requires at least 8Gb RAM)
-    1. install required dependencies `apt-get update && apt-get -y install git curl gcc clang`
-    1. install rust from rustup.rs: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y`
-    1. reload the shell: `source $HOME/.cargo/env`
-    1. build archive: `git clone https://github.com/paritytech/substrate-archive.git --branch release --single-branch && cd substrate-archive/bin/polkadot-archive && SKIP_WASM_BUILD=1 cargo build --release`
-    1. keeping the container running, in a another terminal find the id of the docker container with `docker ps -a`, copy the binary to host with `docker cp $YOUR_CONTAINER_ID:/substrate-archive/bin/polkadot-archive/target/release/polkadot-archive .`
-1. Prepare a new draft release in the github UI
-    1. Upload the new binary
-    1. Tag the new release against the **release** branch
-    1. Write the release notes
-    1. Get a review of the draft release from the team
-    1. Publish the release
-1. Signal to devops that there is a new release available
+2. Update `Cargo.toml` in `substrate-archive` and `substrate-archive-backend` with the new version number.
+3. Update all references to `substrate` crates and `polkadot` to their respective latest releases.
+4. Update the `CHANGELOG` as specified [here](https://keepachangelog.com/en/1.0.0/).
+5. Run all tests: `TEST_DATABASE_URL="postgres://localhost:5432/archive cargo test --all`.
+6. Push the PR against the `release` branch.
+7. Once reviewed, merge it to `release`. Upon merging, Github Actions will create a draft release and upload
+binaries.
+8. Tag the `release` branch` with `git tag vx.y.z` and push the tags with `git push --tags`.
+9. Review the draft release in the github UI.
+10. Get a review of the draft release from the team.
+11. Publish the release from github UI.
+12. Signal to devops that there is a new release available.

bin/polkadot-archive/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes for polkadot-archive will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.3]
+### Changed
+- Config file is now optional. Can configure polkadot archive entirely through environment variables.
+  - the environment variables that need to be set are `CHAIN_DATA_DB` and `DATABASE_URL`.
+- Polkadot archive will archive `polkadot` by default if the `--chain` CLI option is not passed.
+
+### Removed
+- `rpc_url` from the polkadot-archive TOML configuration file
+
+