Merge branch 'master', remote-tracking branch 'origin' into rk-fix-ba…

…cking-vote-import

.github/ISSUE_TEMPLATE/release.md

@@ -15,133 +15,37 @@ checked out with `git checkout release-{{ env.VERSION }}`
 These checks should be performed on the codebase prior to forking to a release-
 candidate branch.
 
-- [ ] Verify [`spec_version`](#spec-version) has been incremented since the
+- [ ] Verify [`spec_version`](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#spec-version) has been incremented since the
     last release for any native runtimes from any existing use on public
     (non-private/test) networks. If the runtime was published (release or pre-release), either
     the `spec_version` or `impl` must be bumped.
-- [ ] Verify previously [completed migrations](#old-migrations-removed) are
+- [ ] Verify previously [completed migrations](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#old-migrations-removed) are
     removed for any public (non-private/test) networks.
-- [ ] Verify pallet and [extrinsic ordering](#extrinsic-ordering) has stayed
+- [ ] Verify pallet and [extrinsic ordering](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#extrinsic-ordering) has stayed
     the same. Bump `transaction_version` if not.
 - [ ] Verify new extrinsics have been correctly whitelisted/blacklisted for
-    [proxy filters](#proxy-filtering).
-- [ ] Verify [benchmarks](#benchmarks) have been updated for any modified
+    [proxy filters](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#proxy-filtering).
+- [ ] Verify [benchmarks](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#benchmarks) have been updated for any modified
     runtime logic.
 
 The following checks can be performed after we have forked off to the release-
 candidate branch or started an additional release candidate branch (rc-2, rc-3, etc)
 
-- [ ] Verify [new migrations](#new-migrations) complete successfully, and the
+- [ ] Verify [new migrations](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#new-migrations) complete successfully, and the
     runtime state is correctly updated for any public (non-private/test)
     networks.
-- [ ] Verify [Polkadot JS API](#polkadot-js) are up to date with the latest
+- [ ] Verify [Polkadot JS API](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#polkadot-js) are up to date with the latest
     runtime changes.
+- [ ] Check with the Signer's team to make sure metadata update QR are lined up
 - [ ] Push runtime upgrade to Westend and verify network stability.
 
 ### All Releases
 
-- [ ] Check that the new client versions have [run on the network](#burn-in)
-    without issue for 12 hours.
+- [ ] Check that the new client versions have [run on the network](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#burn-in)
+    without issue for 12+ hours on >75% of our validator nodes.
 - [ ] Check that a draft release has been created at
     https://github.com/paritytech/polkadot/releases with relevant [release
-    notes](#release-notes)
-- [ ] Check that [build artifacts](#build-artifacts) have been added to the
+    notes](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#release-notes)
+- [ ] Check that [build artifacts](https://github.com/paritytech/polkadot/blob/master/doc/release-checklist.md#build-artifacts) have been added to the
     draft-release
-
-## Notes
-
-### Burn In
-
-Ensure that Parity DevOps has run the new release on Westend, Kusama, and
-Polkadot validators for at least 12 hours prior to publishing the release.
-
-### Build Artifacts
-
-Add any necessary assets to the release. They should include:
-
-- Linux binary
-- GPG signature of the Linux binary
-- SHA256 of binary
-- Source code
-- Wasm binaries of any runtimes
-
-### Release notes
-
-The release notes should list:
-
-- The priority of the release (i.e., how quickly users should upgrade) - this is
-    based on the max priority of any *client* changes.
-- Which native runtimes and their versions are included
-- The proposal hashes of the runtimes as built with
-    [srtool](https://gitlab.com/chevdor/srtool)
-- Any changes in this release that are still awaiting audit
-
-The release notes may also list:
-
-- Free text at the beginning of the notes mentioning anything important
-    regarding this release
-- Notable changes (those labelled with B[1-9]-* labels) separated into sections
-
-### Spec Version
-
-A runtime upgrade must bump the spec number. This may follow a pattern with the
-client release (e.g. runtime v12 corresponds to v0.8.12, even if the current
-runtime is not v11).
-
-### Old Migrations Removed
-
-Any previous `on_runtime_upgrade` functions from old upgrades must be removed
-to prevent them from executing a second time. The `on_runtime_upgrade` function
-can be found in `runtime/<runtime>/src/lib.rs`.
-
-### New Migrations
-
-Ensure that any migrations that are required due to storage or logic changes
-are included in the `on_runtime_upgrade` function of the appropriate pallets.
-
-### Extrinsic Ordering
-
-Offline signing libraries depend on a consistent ordering of call indices and
-functions. Compare the metadata of the current and new runtimes and ensure that
-the `module index, call index` tuples map to the same set of functions. In case
-of a breaking change, increase `transaction_version`.
-
-To verify the order has not changed, you may manually start the following [Github Action](https://github.com/paritytech/polkadot/actions/workflows/extrinsic-ordering-check-from-bin.yml). It takes around a minute to run and will produce the report as artifact you need to manually check.
-
-The things to look for in the output are lines like:
-  - `[Identity] idx 28 -> 25 (calls 15)` - indicates the index for `Identity` has changed
-  - `[+] Society, Recovery` - indicates the new version includes 2 additional modules/pallets.
-  - If no indices have changed, every modules line should look something like `[Identity] idx 25 (calls 15)`
-
-Note: Adding new functions to the runtime does not constitute a breaking change
-as long as the indexes did not change.
-
-### Proxy Filtering
-
-The runtime contains proxy filters that map proxy types to allowable calls. If
-the new runtime contains any new calls, verify that the proxy filters are up to
-date to include them.
-
-### Benchmarks
-
-There are three benchmarking machines reserved for updating the weights at
-release-time. To initialise a benchmark run for each production runtime
-(westend, kusama, polkadot):
-* Go to https://gitlab.parity.io/parity/polkadot/-/pipelines?page=1&scope=branches&ref=master
-* Click the link to the last pipeline run for master
-* Start each of the manual jobs:
-  * 'update_westend_weights'
-  * 'update_polkadot_weights'
-  * 'update_kusama_weights'
-* When these jobs have completed (it takes a few hours), a git PATCH file will
-    be available to download as an artifact. 
-* On your local machine, branch off master
-* Download the patch file and apply it to your branch with `git patch patchfile.patch`
-* Commit the changes to your branch and submit a PR against master
-* The weights should be (Currently manually) checked to make sure there are no
-    big outliers (i.e., twice or half the weight).
-
-### Polkadot JS
-
-Ensure that a release of [Polkadot JS API]() contains any new types or
-interfaces necessary to interact with the new runtime.
+- [ ] Check that all items listed in the [milestone](https://github.com/paritytech/polkadot/milestones) are included in the release.

.gitlab-ci.yml

@@ -40,7 +40,7 @@ default:
   artifacts:
     name:                          "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
     when:                          on_success
-    expire_in:                     28 days
+    expire_in:                     7 days
     paths:
       - ./artifacts/
 
@@ -91,6 +91,7 @@ default:
     - if: $CI_PIPELINE_SOURCE == "pipeline"
       when: never
     - if: $CI_PIPELINE_SOURCE == "schedule"
+      when: never
     - if: $CI_COMMIT_REF_NAME == "master"
     - if: $CI_COMMIT_REF_NAME =~ /^[0-9]+$/                         # PRs
 
@@ -104,6 +105,12 @@ default:
 #### Vault secrets
 .vault-secrets:                    &vault-secrets
   secrets:
+    AWS_ACCESS_KEY_ID:
+      vault:                       cicd/gitlab/$CI_PROJECT_PATH/AWS_ACCESS_KEY_ID@kv
+      file:                        false
+    AWS_SECRET_ACCESS_KEY:
+      vault:                       cicd/gitlab/$CI_PROJECT_PATH/AWS_SECRET_ACCESS_KEY@kv
+      file:                        false
     GITHUB_PR_TOKEN:
       vault:                       cicd/gitlab/parity/GITHUB_PR_TOKEN@kv
       file:                        false
@@ -361,7 +368,6 @@ build-rustdoc:
 
 .build-push-image:                 &build-push-image
   <<:                              *kubernetes-env
-  <<:                              *vault-secrets
   image:                           quay.io/buildah/stable
   variables:                       &image-variables
     GIT_STRATEGY:                  none
@@ -498,6 +504,47 @@ update_westend_weights:
 
 #### stage:                        publish
 
+publish-s3-release:                &publish-s3
+  stage:                           publish
+  needs:
+    - job:                         test-build-linux-stable
+      artifacts:                   true
+  <<:                              *kubernetes-env
+  <<:                              *vault-secrets
+  image:                           paritytech/awscli:latest
+  variables:
+    GIT_STRATEGY:                  none
+    PREFIX:                        "builds/polkadot/${ARCH}-${DOCKER_OS}"
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "pipeline"
+      when: never
+    # publishing binaries nightly
+    - if: $CI_PIPELINE_SOURCE == "schedule"
+  before_script:
+    - *check-versions
+  script:
+    - echo "uploading objects to https://releases.parity.io/${PREFIX}/${VERSION}"
+    - aws s3 sync --acl public-read ./artifacts/ s3://${AWS_BUCKET}/${PREFIX}/${VERSION}/
+    - echo "update objects at https://releases.parity.io/${PREFIX}/${EXTRATAG}"
+    - find ./artifacts -type f | while read file; do
+        name="${file#./artifacts/}";
+        aws s3api copy-object
+          --copy-source ${AWS_BUCKET}/${PREFIX}/${VERSION}/${name}
+          --bucket ${AWS_BUCKET} --key ${PREFIX}/${EXTRATAG}/${name};
+      done
+    - |
+      cat <<-EOM
+      |
+      |  polkadot binary paths:
+      |
+      |  - https://releases.parity.io/${PREFIX}/${EXTRATAG}/polkadot
+      |  - https://releases.parity.io/${PREFIX}/${VERSION}/polkadot
+      |
+      EOM
+  after_script:
+    - aws s3 ls s3://${AWS_BUCKET}/${PREFIX}/${EXTRATAG}/
+        --recursive --human-readable --summarize
+
 publish-rustdoc:
   stage:                           publish
   <<:                              *kubernetes-env
@@ -592,19 +639,13 @@ zombienet-test-parachains-upgrade-smoke-test:
   stage:                           deploy
   image:                           "${ZOMBIENET_IMAGE}"
   <<:                              *kubernetes-env
-  rules:
-    - if: $CI_PIPELINE_SOURCE == "schedule"
-    - if: $CI_COMMIT_REF_NAME == "master"
-    - if: $CI_COMMIT_REF_NAME =~ /^[0-9]+$/                         # PRs
-    - if: $CI_COMMIT_REF_NAME == "rococo-v1"
+  <<:                              *zombienet-refs
   needs:
     - job:                         publish-polkadot-image
     - job:                         publish-malus-image
     - job:                         publish-adder-collator-image
-
   variables:
     GH_DIR:                        'https://github.com/paritytech/polkadot/tree/${CI_COMMIT_SHORT_SHA}/zombienet_tests/parachains'
-
   before_script:
     - echo "ZombieNet Tests Config"
     - echo "docker.io/parity/polkadot:latest"
@@ -614,7 +655,6 @@ zombienet-test-parachains-upgrade-smoke-test:
     - export DEBUG=zombie,zombie::network-node
     - export ZOMBIENET_INTEGRATION_TEST_IMAGE="docker.io/parity/polkadot:latest" # Use polkadot latest image
     - export COL_IMAGE="docker.io/parity/polkadot-collator:latest" # Use cumulus lastest image
-
   script:
     - /home/nonroot/zombie-net/scripts/run-test-env-manager.sh
         --github-remote-dir="${GH_DIR}"