Skip to content

Commit

Permalink
Make releases EZ (#279)
Browse files Browse the repository at this point in the history 
* need to test release.yml

* improve release workflow

* Refactor Changelog

- make changelog _actually_ follow https://keepachangelog.com/en/1.0.0/ to a T
- create release workflow which automates creating binaries

* Change workflow file paths & update RELEASE-CHECKLIST

* fix spacing

* remove macos from os matrix

* Update RELEASE-CHECKLIST.md

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>
  • Loading branch information
@insipx @TarikGul
insipx and TarikGul authored May 20, 2021
1 parent 77f8d4f commit 9abef6e
Show file tree
Hide file tree
Showing 4 changed files with 164 additions and 94 deletions.
60 changes: 60 additions & 0 deletions .github/workflows/release.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Release CI
on:
push:
branches:
- release
jobs:
deploy:
strategy:
matrix:
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- name: Install Rust
uses: actions-rs/toolchain@v1
with:
toolchain: stable
profile: minimal

- name: Build Polkadot Archive
run: cargo build --release --manifest-path bin/polkadot-archive/Cargo.toml

- name: Prepare binary
run: |
mkdir -p release
strip bin/polkadot-archive/target/release/polkadot-archive
- name: Package for Debian
if: matrix.os == 'ubuntu-latest'
run: |
cargo install cargo-deb
cargo deb --manifest-path ./bin/polkadot-archive/Cargo.toml
mv ./bin/polkadot-archive/target/debian/*.deb ./release/polkadot-archive-linux.deb
tar -C ./bin/polkadot-archive/target/release/ -czvf ./release/polkadot-archive-linux.tar.gz ./polkadot-archive
- name: Package just binary
if: matrix.os == 'ubuntu-latest'
run: |
mv bin/polkadot-archive/target/release/polkadot-archive ./release/polkadot-archive
- name: Get changelog
id: changelog_reader
uses: mindsers/changelog-reader-action@v2
with:
validation_depth: 2
path: ./CHANGELOG.md

- name: Release
uses: softprops/action-gh-release@v1
with:
tag_name: ${{ steps.changelog_reader.outputs.version }}
name: Release ${{ steps.changelog_reader.outputs.version }}
body: ${{ steps.changelog_reader.outputs.changes }}
draft: true
files: |
./release/*.tar.gz
./release/*.deb
./release/polkadot_archive
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
146 changes: 73 additions & 73 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,83 +1,83 @@
# Changelog for `substrate-archive`

## **[Unreleased]**

## **[v0.5.1]**

- [Added] Release checklist
- [Changed] Pinned to `substrate`/`polkadot` release v0.8.30
- [Changed] Keeping the `substrate-archive` and `substrate-archive-backend` crates's versions aligned.

## **[v0.5.0]**

### Substrate Archive

- [Changed] Use SQLX for migrations
- this will create a new table in the database, `sqlx_migrations` or similiar.
- the older table, `__refinery_migrations` can be safely dropped

#### Api Changes

- [Added] New `ArchiveBuilder` struct for constructing the indexer.
- [Changed] returns a trait rather than a struct.
- [Changed] the returned trait,`Archive` manages the underlying actor runtime
- [Changed] no longer need to instantiate a client and manually pass it to the Archive
- [Changed] rename `run_with` to `drive`
- [Changed] Archive now accepts just a postgres URL instead of a postgres URL split into its parts. This should
make configuring the archive more straightforward. Takes from environment variable `DATABASE_URL` if not passed to the
archive directly
- [Added] Archive now reads the `CHAIN_DATA_DB` environment variable if the path to the backend chain database is not passed directly.
- [Removed] Archive no longer needs an RPC url to function
- [Changed] Print out the Postgres URL at startup
- [Added] `max_block_load` to configure the maximum number of blocks loaded at once
- [Added] Two new options to the configuration relative to state tracing:
- `targets` for specifying runtime targets to trace in WASM
- `folder` where WASM blobs with tracing enabled are kept.
- More on state-tracing [here](https://github.com/paritytech/substrate-archive/wiki/6.\)-State-Tracing-&-Balance-Reconciliation)
- [Changed] Archive config is now separated into sections for readability. Migration is manual but looking at the new `archive.conf` in `polkadot-archive` or `node-template-archive` folders should help.

#### Internal Changes

- [Changed] Postgres SQL queries are now type checked
- [QoL] Refactor file layout to `substrate-archive` and `substrate-archive-backend`.
- [perf] Decouple Database actors
- [QoL] upgrade to SQLx 0.5.0
- [perf] Overhaul of block indexing. Now uses a Iterator to only collect batches of blocks from the database,
taking advantage of the better reading performance of sequential data access. Gathering blocks by RPC is no longer done.
- [perf] a new module `runtime_version_cache` is introduced in order to cache and run a binary search on runtime version & blocks.
- [perf] better queries for the set difference between the storage and blocks table - makes querying for missing storage more efficient
- [err] Better handling of SQL errors
- [perf] switch to a leaner, 'lower-level' actor framework (xtra)
- [perf] switch to a background-task-queue for executing blocks. This uses significantly less memory and
persists blocks that need to be executed on-disk.
- [QoL] remove the last frame dependency, `frame-system`. Archive now relies only on generic traits defined in substrate-core.
- new method of batch inserts avoids starving the Postgres Pool of connections
- [perf] Notification stream from Postgres indexes storage changes in the background.
- [Changed] Switch substrate-archive `LocalCallExecutor` with Substrate's `LocalCallExecutor`

### Polkadot Archive

- [Changed] Config file is now optional. Can configure polkadot archive entirely through environment variables.
- the environment variables that need to be set are `CHAIN_DATA_DB` and `DATABASE_URL`.
- [Changed] Polkadot archive will archive `polkadot` by default if the `--chain` CLI option is not passed.
- [Changed] remove `rpc_url` from the polkadot-archive TOML configuration file
- [Changed] All options in config file apart from `db_url`.

## **[v0.4.0]**
# Changelog

All notable changes for substrate-archive will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [v0.5.1] - 2021-05-06
### Added
- Release checklist

### Changed
- Pinned to `substrate`/`polkadot` release v0.8.30
- Keeping the `substrate-archive` and `substrate-archive-backend` crates's versions aligned.

## [v0.5.0] - 2021-03-29
### Added
- `ArchiveBuilder` struct for constructing the indexer.
- returns a trait, `Archive` that manages the underlying actor runtime
- Archive now reads the `CHAIN_DATA_DB` environment variable if the path to the backend chain database is not passed directly.
- `max_block_load` to configure the maximum number of blocks loaded at once
- Two new options to the configuration relative to state tracing:
- `targets` for specifying runtime targets to trace in WASM
- `folder` where WASM blobs with tracing enabled are kept.
- More on state-tracing [here](https://github.com/paritytech/substrate-archive/wiki/6.\)-State-Tracing-&-Balance-Reconciliation)
- Runtime versions between blocks are cached, speeding up block indexing.
- Notification stream from Postgres indexes storage changes in the background.

### Changed
- no longer need to instantiate a client and manually pass it to the Archive
- rename `run_with` to `drive`
- Archive now accepts a postgres URL instead of a postgres URL split into its parts.
- Archive will take Postgres URL from environment variable `DATABASE_URL` as a fallback.
- Use SQLX for migrations
- SQLX will create a new table in the database, `sqlx_migrations` or similiar.
- the older table, `__refinery_migrations` can be safely dropped
- Print out the Postgres URL at startup
- config is now separated into sections for readability. Migration is manual but looking at the new `archive.conf` in `polkadot-archive` or `node-template-archive` folders should help.
- Postgres SQL queries are now type checked
- Refactor file layout to `substrate-archive` and `substrate-archive-backend`.
- Decouple Database actors
- upgrade to SQLx 0.5.0
- Overhaul of block indexing. Now uses a Iterator to only collect batches of blocks from the database,
taking advantage of the better reading performance of sequential data access. Gathering blocks by RPC is no longer done.
- speed up query for difference between the storage and blocks table -
- switch to a leaner actor framework, [xtra](https://github.com/Restioson/xtra)
- switch to a persistant background-task-queue for executing blocks, [coil](https://github.com/insipx/coil).
- batch inserts no longer starve postgres pool of connections
- Switch substrate-archive `LocalCallExecutor` with Substrate's `LocalCallExecutor`

### Removed
- the last frame dependency, `frame-system`. Archive now relies only on generic traits defined in substrate-core.
- RPC URL. An RPC URL is no longer required to function.

## [v0.4.0] - 2021-01-24
### Added
- integrate database migrations into substrate-archive library

### Changed
- Speed up Storage Indexing by re-executing blocks
- [internal] Remove some depdendencies for service and backend
- fixes for storage inserts
- refactored error types
- integrate database migrations into substrate-archive library

## [v0.3.1]
### Removed
- dependencies for service and backend

- add node-template-archive
### Fixed
- storage inserts

## **[v0.3.0]**
## [v0.3.1]
### Added
- node-template-archive

- Use a rocksdb-backed substrate client instead of RPC for indexing
## [v0.3.0]
### Added
- Create a CLI for indexing kusama
- New PostgreSQL Schema
- Actors to model dataflow

### Changed
- Use a rocksdb-backed substrate client instead of RPC for indexing

33 changes: 12 additions & 21 deletions RELEASE-CHECKLIST.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,15 @@ Here's how to make a new release of `substrate-archive`.
## Checklist

1. Make a new branch, name it whatever you want, but `vx.y.z-prep` is a good template
1. Update `Cargo.toml` in `substrate-archive` and `substrate-archive-backend` with the new version number.
1. Update all references to `substrate` crates and `polkadot` to their respective latest releases.
1. Update the `CHANGELOG` as specified [here](https://keepachangelog.com/en/1.0.0/).
1. Run all tests: `TEST_DATABASE_URL="postgres://localhost:5432/archive cargo test --all`.
1. Push the PR against the `release` branch.
1. Once reviewed, merge it to `release`.
1. Tag the `release` branch` with `git tag vx.y.z` and push the tags with `git push --tags`
1. Build a binary for debian, compatible with the current glibc version (`v2.31`)
1. `docker run --rm -it debian:jessie` (requires at least 8Gb RAM)
1. install required dependencies `apt-get update && apt-get -y install git curl gcc clang`
1. install rust from rustup.rs: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y`
1. reload the shell: `source $HOME/.cargo/env`
1. build archive: `git clone https://github.com/paritytech/substrate-archive.git --branch release --single-branch && cd substrate-archive/bin/polkadot-archive && SKIP_WASM_BUILD=1 cargo build --release`
1. keeping the container running, in a another terminal find the id of the docker container with `docker ps -a`, copy the binary to host with `docker cp $YOUR_CONTAINER_ID:/substrate-archive/bin/polkadot-archive/target/release/polkadot-archive .`
1. Prepare a new draft release in the github UI
1. Upload the new binary
1. Tag the new release against the **release** branch
1. Write the release notes
1. Get a review of the draft release from the team
1. Publish the release
1. Signal to devops that there is a new release available
2. Update `Cargo.toml` in `substrate-archive` and `substrate-archive-backend` with the new version number.
3. Update all references to `substrate` crates and `polkadot` to their respective latest releases.
4. Update the `CHANGELOG` as specified [here](https://keepachangelog.com/en/1.0.0/).
5. Run all tests: `TEST_DATABASE_URL="postgres://localhost:5432/archive cargo test --all`.
6. Push the PR against the `release` branch.
7. Once reviewed, merge it to `release`. Upon merging, Github Actions will create a draft release and upload
binaries.
8. Tag the `release` branch with `git tag vx.y.z` and push the tags with `git push --tags`.
9. Review the draft release in the github UI.
10. Get a review of the draft release from the team.
11. Publish the release from github UI.
12. Signal to devops that there is a new release available.
19 changes: 19 additions & 0 deletions bin/polkadot-archive/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Changelog

All notable changes for polkadot-archive will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.2.3]
### Changed
- Config file is now optional. Can configure polkadot archive entirely through environment variables.
- the environment variables that need to be set are `CHAIN_DATA_DB` and `DATABASE_URL`.
- Polkadot archive will archive `polkadot` by default if the `--chain` CLI option is not passed.

### Removed
- `rpc_url` from the polkadot-archive TOML configuration file


0 comments on commit 9abef6e

Please sign in to comment.