Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: get versions from the repo directly #11

Merged
merged 2 commits into from
Apr 17, 2024
Merged

feat: get versions from the repo directly #11

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bfc9256
get branch cargo.lock or plan.toml versions
patriciobcs Apr 16, 2024
2d1d0ee
feat: simplify workflow and add list cmd
patriciobcs Apr 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,531 changes: 1,389 additions & 142 deletions Cargo.lock
View file
Open in desktop

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,10 @@ serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
log = "0.4.21"
env_logger = "0.11.3"
reqwest = { version = "0.12.3", features = ["json"] }
toml = "0.8.12"
tokio = { version = "1.37.0", features = ["full"] }

[dev-dependencies]
tokio-test = "0.4"
mockito = "0.30"
28 changes: 25 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
# Polkadot SDK Version Manager

This is a simple tool to manage and update the Polkadot SDK dependencies in any Cargo.toml file.
This is a simple tool to manage and update the Polkadot SDK dependencies in any Cargo.toml file. It [works offline](#offline) using a local version mapping previously fetched from the Polkadot SDK crates-io branches, and [works online](#online) by getting directly the versions from the Polkadot SDK crates-io branches (`Plan.toml` or `Cargo.lock`) files.

## Installation

From [GitHub](https://github.com/paritytech/psvm):

```sh
cargo install --git https://github.com/paritytech/psvm psvm
```

From [crates.io](https://crates.io/crates/psvm):

```sh
cargo install psvm
```

## Usage

Go to the directory containing the Cargo.toml file you want to update and run `psvm`. This will automatically update the Polkadot SDK dependencies in the Cargo.toml file to their correct crates.io version.
Expand All @@ -19,16 +27,30 @@ If you want to update the dependencies to a specific Polkadot SDK version, you c
```sh
# Go to the directory containing the Cargo.toml file you want to update
cd <cargo-toml-dir>
# Run the psvm command to update the dependencies
# Run the psvm command to update the dependencies using PSVM local versions
psvm
patriciobcs marked this conversation as resolved.
Show resolved Hide resolved
# You can also update it using the path to the Cargo.toml file
psvm -p <cargo-toml-dir>/Cargo.toml
# Overwrite local dependencies with crates.io versions
psvm -o
# Update to a specific Polkadot SDK version (default 1.3.0)
# Update to a specific Polkadot SDK version (default 1.3.0) using PSVM local versions
psvm -v "1.7.0"
# Update to the versions defined in a specific Polkadot SDK branch Plan.toml file
# i.e https://raw.githubusercontent.com/paritytech/polkadot-sdk/release-crates-io-v1.6.0/Plan.toml
psvm -b "release-crates-io-v1.6.0"
# Update to the versions defined in a specific Polkadot SDK branch Cargo.lock file
# i.e https://raw.githubusercontent.com/paritytech/polkadot-sdk/release-crates-io-v1.3.0/Cargo.lock
psvm -b "release-crates-io-v1.3.0" -s "Cargo.lock"
patriciobcs marked this conversation as resolved.
Show resolved Hide resolved
```

### Offline

You can use PSVM offline with the `-v` or `--version` flag, cause PSVM keeps a local mapping (i.e. [v1.3.0 Mapping](/src/versions/release-crates-io-v1.3.0.json)) of the Polkadot SDK dependencies versions, that will be slowly updated. So be aware that the latest Polkadot SDK versions may not be available in the local mapping.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure that we need the offline version at all? As soon as there are two ways to do something and one is wrong, then ppl will end up doing the wrong thing and complain.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I keep it to just not cause a breaking change with the "offline" justification, but probably is useless cause anyways you need internet to download the crates after setting a version.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only concern here is the fact that some branches don't have a Plan.toml file for example. What do you about fallback to the Cargo.lock in case Plan.toml not found. For example v1.3.0: https://raw.githubusercontent.com/paritytech/polkadot-sdk/release-crates-io-v1.3.0/Plan.toml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea the Cargo.lock fallback is good.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok! It's done, also added a test to check if it works for all release branches, and it works! The only branch without Plan.toml is v1.3.0, but in this case it will fallback to Cargo.lock. Added a warning in the Workflow section.


### Online

To have the latest versions, you can use the `-b` or `--branch` flag to fetch the versions from a specific Polkadot SDK branch. This will fetch the `Plan.toml` file (used to publish crates into crates.io) from the branch and update the local mapping with the versions defined in the file (i.e [v1.6.0 `Plan.toml`](https://raw.githubusercontent.com/paritytech/polkadot-sdk/release-crates-io-v1.6.0/Plan.toml)). However, this could not work for older branches (i.e. v1.3.0), cause the `Plan.toml` file may not exist. In this case, you can use the `-s` or `--source` flag to fetch the `Cargo.lock` file (i.e. [v1.3.0 `Cargo.lock`](https://raw.githubusercontent.com/paritytech/polkadot-sdk/release-crates-io-v1.3.0/Cargo.lock)) from the branch and update the local mapping with the versions defined in the file.

## Maintenance

To update the Polkadot SDK dependencies mapping, run `./scripts/update.sh <polkadot-sdk repo path>`. This will update the dependencies `json` files located in `src/versions` with the latest versions defined in Polkadot SDK crates-io branches.
Expand Down
37 changes: 30 additions & 7 deletions src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,14 @@ mod versions;

use clap::Parser;
use env_logger::Env;
use serde_json::from_str;
use std::collections::BTreeMap;
use std::fs;
use std::path::Path;
use std::path::PathBuf;
use toml_edit::DocumentMut;
use versions::get_version_mapping;
use versions::{get_branch_mapping, get_version_mapping};

pub const DEFAULT_GIT_SERVER: &str = "https://raw.githubusercontent.com";

/// Polkadot SDK Version Manager.
///
Expand All @@ -37,24 +38,46 @@ struct Command {
path: PathBuf,

/// Specifies the Polkadot SDK version.
#[clap(
short,
long,
conflicts_with = "branch",
required_unless_present = "branch"
)]
version: Option<String>,

/// Specifies a Polkadot SDK branch to get the versions. Can't be used at the same time as `version`.
#[clap(short, long)]
version: String,
branch: Option<String>,

/// Specifies the source file to get the versions.
#[clap(short, long, value_parser = ["Cargo.lock", "Plan.toml"], default_value = "Plan.toml")]
source: String,

/// Overwrite local dependencies (using path).
#[clap(short, long)]
overwrite: bool,

/// Specifies the git server to get the versions when using the `branch` flag.
#[clap(short, long, default_value = DEFAULT_GIT_SERVER)]
git_server: String,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
env_logger::Builder::from_env(Env::default().default_filter_or("info")).init();
let cmd = Command::parse();

let cargo_toml_path = validate_workspace_path(cmd.path)?;

// Decide which branch data to use based on the branch name
let crates_versions_data = get_version_mapping(&cmd.version);

let crates_versions: BTreeMap<String, String> = from_str(crates_versions_data)?;
let crates_versions: BTreeMap<String, String> = if let Some(version) = cmd.version {
serde_json::from_str(get_version_mapping(&version))?
} else if let Some(branch) = cmd.branch {
get_branch_mapping(&cmd.git_server, &branch, &cmd.source).await?
} else {
return Err("Please specify only a version or a branch".into());
};

update_dependencies(&cargo_toml_path, &crates_versions, cmd.overwrite)?;

Expand Down
0 src/testing/input.Cargo.toml → src/testing/cargo-lock/input.Cargo.toml
View file
Open in desktop
File renamed without changes.
0 src/testing/output.Cargo.toml → src/testing/cargo-lock/output.Cargo.toml
View file
Open in desktop
File renamed without changes.
175 changes: 175 additions & 0 deletions src/testing/plan-toml/input.Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
[package]
name = "runtime"
version = "1.0.0"
authors = ["Anonymous"]
description = "A parachain runtime."
license = "Unlicense"
edition = 2021

[package.metadata.docs.rs]
targets = ["x86_64-unknown-linux-gnu"]

[build-dependencies]
substrate-wasm-builder = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", optional = true, version = "14.0.0" }

[dependencies]
codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false, features = ["derive"] }
hex-literal = { version = "0.4.1", optional = true }
log = { version = "0.4.20", default-features = false }
scale-info = { version = "2.10.0", default-features = false, features = ["derive"] }
smallvec = "1.11.0"

# Local
pallet-parachain-template = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }

# Substrate
frame-benchmarking = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false, optional = true }
frame-executive = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
frame-support = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
frame-system = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
frame-system-benchmarking = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false, optional = true }
frame-system-rpc-runtime-api = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
frame-try-runtime = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false, optional = true }
pallet-aura = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-authorship = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-balances = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-session = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-sudo = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-timestamp = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-transaction-payment = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-transaction-payment-rpc-runtime-api = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-api = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-block-builder = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-consensus-aura = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-core = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-genesis-builder = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-inherents = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-offchain = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-runtime = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-session = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-std = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-transaction-pool = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
sp-version = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }

# Polkadot
pallet-xcm = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
polkadot-parachain-primitives = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
polkadot-runtime-common = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
xcm = { package = "staging-xcm", git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
xcm-builder = { package = "staging-xcm-builder", git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
xcm-executor = { package = "staging-xcm-executor", git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }

# Cumulus
cumulus-pallet-aura-ext = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
cumulus-pallet-dmp-queue = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
cumulus-pallet-parachain-system = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false, features = ["parameterized-consensus-hook"] }
cumulus-pallet-session-benchmarking = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
cumulus-pallet-xcm = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
cumulus-pallet-xcmp-queue = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
cumulus-primitives-core = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
cumulus-primitives-utility = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
pallet-collator-selection = { git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }
parachain-info = { package = "staging-parachain-info", git = "https://github.com/paritytech/polkadot-sdk", branch = "release-crates-io-v1.3.0", default-features = false }

[features]
default = [ "std" ]
std = [
"codec/std",
"cumulus-pallet-aura-ext/std",
"cumulus-pallet-dmp-queue/std",
"cumulus-pallet-parachain-system/std",
"cumulus-pallet-session-benchmarking/std",
"cumulus-pallet-xcm/std",
"cumulus-pallet-xcmp-queue/std",
"cumulus-primitives-core/std",
"cumulus-primitives-utility/std",
"frame-benchmarking?/std",
"frame-executive/std",
"frame-support/std",
"frame-system-benchmarking?/std",
"frame-system-rpc-runtime-api/std",
"frame-system/std",
"frame-try-runtime?/std",
"log/std",
"pallet-aura/std",
"pallet-authorship/std",
"pallet-balances/std",
"pallet-collator-selection/std",
"pallet-parachain-template/std",
"pallet-session/std",
"pallet-sudo/std",
"pallet-timestamp/std",
"pallet-transaction-payment-rpc-runtime-api/std",
"pallet-transaction-payment/std",
"pallet-xcm/std",
"parachain-info/std",
"polkadot-parachain-primitives/std",
"polkadot-runtime-common/std",
"scale-info/std",
"sp-api/std",
"sp-block-builder/std",
"sp-consensus-aura/std",
"sp-core/std",
"sp-genesis-builder/std",
"sp-inherents/std",
"sp-offchain/std",
"sp-runtime/std",
"sp-session/std",
"sp-std/std",
"sp-transaction-pool/std",
"sp-version/std",
"substrate-wasm-builder",
"xcm-builder/std",
"xcm-executor/std",
"xcm/std",
]

runtime-benchmarks = [
"cumulus-pallet-parachain-system/runtime-benchmarks",
"cumulus-pallet-session-benchmarking/runtime-benchmarks",
"cumulus-pallet-xcmp-queue/runtime-benchmarks",
"cumulus-primitives-utility/runtime-benchmarks",
"frame-benchmarking/runtime-benchmarks",
"frame-support/runtime-benchmarks",
"frame-system-benchmarking/runtime-benchmarks",
"frame-system/runtime-benchmarks",
"hex-literal",
"pallet-balances/runtime-benchmarks",
"pallet-collator-selection/runtime-benchmarks",
"pallet-parachain-template/runtime-benchmarks",
"pallet-sudo/runtime-benchmarks",
"pallet-timestamp/runtime-benchmarks",
"pallet-xcm/runtime-benchmarks",
"polkadot-parachain-primitives/runtime-benchmarks",
"polkadot-runtime-common/runtime-benchmarks",
"sp-runtime/runtime-benchmarks",
"xcm-builder/runtime-benchmarks",
"xcm-executor/runtime-benchmarks",
]

try-runtime = [
"cumulus-pallet-aura-ext/try-runtime",
"cumulus-pallet-dmp-queue/try-runtime",
"cumulus-pallet-parachain-system/try-runtime",
"cumulus-pallet-xcm/try-runtime",
"cumulus-pallet-xcmp-queue/try-runtime",
"frame-executive/try-runtime",
"frame-support/try-runtime",
"frame-system/try-runtime",
"frame-try-runtime/try-runtime",
"pallet-aura/try-runtime",
"pallet-authorship/try-runtime",
"pallet-balances/try-runtime",
"pallet-collator-selection/try-runtime",
"pallet-parachain-template/try-runtime",
"pallet-session/try-runtime",
"pallet-sudo/try-runtime",
"pallet-timestamp/try-runtime",
"pallet-transaction-payment/try-runtime",
"pallet-xcm/try-runtime",
"parachain-info/try-runtime",
"polkadot-runtime-common/try-runtime",
"sp-runtime/try-runtime",
]

experimental = [ "pallet-aura/experimental" ]
Loading