Unverified Commit 8ad2f362 authored by s3krit's avatar s3krit Committed by GitHub
Browse files

Automation for new release process (#1754)

parent e1a5c6b6
Pipeline #111226 passed with stages
in 21 minutes and 33 seconds
---
title: Polkadot {{ env.VERSION }} Release checklist
---
# Release Checklist
This is the release checklist for Polkadot {{ env.VERSION }}. **All** following
checks should be completed before publishing a new release of the
Polkadot/Kusama/Westend runtime or client. The current release candidate can be
checked out with `git checkout {{ env.VERSION }}`
### Runtime Releases
- [ ] Verify [`spec_version`](#spec-version) has been incremented since the
last release for any native runtimes from any existing use on public
(non-private/test) networks.
- [ ] Verify [new migrations](#new-migrations) complete successfully, and the
runtime state is correctly updated.
- [ ] Verify previously [completed migrations](#old-migrations-removed) are
removed.
- [ ] Verify pallet and [extrinsic ordering](#extrinsic-ordering) has stayed
the same. Bump `transaction_version` if not.
- [ ] Verify new extrinsics have been correctly whitelisted/blacklisted for
[proxy filters](#proxy-filtering).
- [ ] Verify [benchmarks](#benchmarks) have been updated for any modified
runtime logic.
- [ ] Verify [Polkadot JS API](#polkadot-js) are up to date with the latest
runtime changes.
### All Releases
- [ ] Check that the new client versions have [run on the network](#burn-in)
without issue for 12 hours.
- [ ] Check that a draft release has been created at
https://github.com/paritytech/polkadot/releases with relevant [release
notes](#release-notes)
- [ ] Check that [build artifacts](#build-artifacts) have been added to the
draft-release
## Notes
### Burn In
Ensure that Parity DevOps has run the new release on Westend, Kusama, and
Polkadot validators for at least 12 hours prior to publishing the release.
### Build Artifacts
Add any necessary assets to the release. They should include:
- Linux binary
- GPG signature of the Linux binary
- SHA256 of binary
- Source code
- Wasm binaries of any runtimes
### Release notes
The release notes should list:
- The priority of the release (i.e., how quickly users should upgrade)
- Which native runtimes and their versions are included
- The proposal hashes of the runtimes as built with
[srtool](https://gitlab.com/chevdor/srtool)
- Any changes in this release that are still awaiting audit
The release notes may also list:
- Free text at the beginning of the notes mentioning anything important
regarding this release
- Notable changes (those labelled with B[1-9]-* labels) separated into sections
### Spec Version
A runtime upgrade must bump the spec number. This may follow a pattern with the
client release (e.g. runtime v12 corresponds to v0.8.12, even if the current
runtime is not v11).
### New Migrations
Ensure that any migrations that are required due to storage or logic changes
are included in the `on_runtime_upgrade` function of the appropriate pallets.
### Old Migrations Removed
Any previous `on_runtime_upgrade` functions from old upgrades must be removed
to prevent them from executing a second time.
### Extrinsic Ordering
Offline signing libraries depend on a consistent ordering of call indices and
functions. Compare the metadata of the current and new runtimes and ensure that
the `module index, call index` tuples map to the same set of functions. In case
of a breaking change, increase `transaction_version`.
Note: Adding new functions to the runtime does not constitute a breaking change
as long as they are added to the end of a pallet (i.e., does not break any
other call index).
### Proxy Filtering
The runtime contains proxy filters that map proxy types to allowable calls. If
the new runtime contains any new calls, verify that the proxy filters are up to
date to include them.
### Benchmarks
Run the benchmarking suite with the new runtime and update any function weights
if necessary.
### Polkadot JS
Ensure that a release of [Polkadot JS API]() contains any new types or
interfaces necessary to interact with the new runtime.
......@@ -3,7 +3,8 @@ name: Publish draft release
on:
push:
tags:
- v**.**.**
# Catches v1.2.3 and v1.2.3-rc1
- v[0-9]+.[0-9]+.[0-9]+*
jobs:
build-runtimes:
......@@ -99,19 +100,6 @@ jobs:
release_name: Polkadot ${{ github.ref }}
body_path: ./release_text.md
draft: true
post_to_matrix:
runs-on: ubuntu-latest
needs: publish-draft-release
steps:
- name: Internal polkadot channel
uses: s3krit/matrix-message-action@v0.0.3
with:
room_id: ${{ secrets.INTERNAL_POLKADOT_MATRIX_ROOM_ID }}
access_token: ${{ secrets.MATRIX_ACCESS_TOKEN }}
message: "**New version of polkadot tagged**: ${{ github.ref }}<br/>Gav: Draft release created: ${{ needs.publish-draft-release.outputs.release_url }}"
server: "matrix.parity.io"
publish-runtimes:
runs-on: ubuntu-latest
needs: ['publish-draft-release']
......@@ -141,3 +129,15 @@ jobs:
asset_path: ./${{ matrix.runtime }}_runtime.compact.wasm
asset_name: ${{ matrix.runtime }}_runtime-v${{ steps.get-runtime-ver.outputs.runtime_ver }}.compact.wasm
asset_content_type: application/wasm
post_to_matrix:
runs-on: ubuntu-latest
needs: publish-draft-release
steps:
- name: Internal polkadot channel
uses: s3krit/matrix-message-action@v0.0.2
with:
room_id: ${{ secrets.INTERNAL_POLKADOT_MATRIX_ROOM_ID }}
access_token: ${{ secrets.MATRIX_ACCESS_TOKEN }}
message: "**New version of polkadot tagged**: ${{ github.ref }}<br/>Gav: Draft release created: ${{ needs.publish-draft-release.outputs.release_url }}"
server: "matrix.parity.io"
name: Release-candidate automation
on:
push:
branches:
- release-v[0-9]+.[0-9]+.[0-9]+
jobs:
tag_rc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- id: compute_tag
name: Compute next rc tag
shell: bash
run: |
# Get last rc tag if exists, else set it to {version}-rc1
version=${GITHUB_REF#refs/heads/release-}
echo "$version"
echo "::set-output name=version::$version"
git tag -l
last_rc=$(git tag -l "$version-rc*" | sort -V | tail -n 1)
if [ -n "$last_rc" ]; then
suffix=$(echo "$last_rc" | grep -Eo '[0-9]+$')
echo $suffix
((suffix++))
echo $suffix
echo "::set-output name=new_tag::$version-rc$suffix"
echo "::set-output name=first_rc::false"
else
echo "::set-output name=new_tag::$version-rc1"
echo "::set-output name=first_rc::true"
fi
- name: Apply new tag
uses: tvdias/github-tagger@v0.0.2
with:
# We can't use the normal GITHUB_TOKEN for the following reason:
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows#triggering-new-workflows-using-a-personal-access-token
# RELEASE_BRANCH_TOKEN requires public_repo OAuth scope
repo-token: "${{ secrets.RELEASE_BRANCH_TOKEN }}"
tag: ${{ steps.compute_tag.outputs.new_tag }}
- id: create-issue
uses: JasonEtco/create-an-issue@v2
# Only create the issue if it's the first release candidate
if: steps.compute_tag.outputs.first_rc == 'true'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BRANCH: ${{ steps.compute_tag.outputs.version }}
with:
filename: .github/ISSUE_TEMPLATE/release.md
- uses: s3krit/matrix-message-action@v0.0.2
if: steps.create-issue.outputs.url != ''
with:
room_id: ${{ secrets.INTERNAL_POLKADOT_MATRIX_ROOM_ID }}
access_token: ${{ secrets.MATRIX_ACCESS_TOKEN }}
server: "matrix.parity.io"
message: "Release process for polkadot ${{ steps.compute_tag.outputs.version }} has been started. Tracking issue: ${{ steps.create-issue.outputs.url }}"
# Release Checklist
The following checks should be completed before publishing a new release of the
Polkadot/Kusama/Westend runtime or client.
### Runtime Releases
The following should be done *prior* to tagging the potential release. Upon
completion, tag the commit and proceed with the [All Releases](#all-releases) steps.
- [ ] List any [native runtime](#native-runtimes) versions associated with the release.
- [ ] Has incremented [`spec_version`](#spec-version) for any native runtimes from any existing use on public (non-private/test) networks.
- [ ] Verify [new migrations](#new-migrations) complete successfully, and the runtime state is
correctly updated.
- [ ] Verify previously [completed migrations](#old-migrations-removed) are removed.
- [ ] Verify pallet and [extrinsic ordering](#extrinsic-ordering) has stayed the same. Bump
`transaction_version` if not.
- [ ] Verify new extrinsics have been correctly whitelisted/blacklisted for
[proxy filters](#proxy-filtering).
- [ ] Verify [benchmarks](#benchmarks) have been updated for any modified runtime logic.
- [ ] Verify [Polkadot JS API](#polkadot-js) are up to date with the latest runtime changes.
### All Releases
- [ ] Check that the new client versions have [run on the network](#burn-in) without issue for 12
hours.
- [ ] Check that a draft release has been created at https://github.com/paritytech/polkadot/releases with relevant [release notes](#release-notes)
- [ ] Check that [build artifacts](#build-artifacts) have been added to the draft-release
- [ ] Check that you have updated the Cargo.toml version.
## Notes
### Burn In
Ensure that Parity DevOps has run the new release on Westend, Kusama, and Polkadot validators for
at least 12 hours prior to publishing the release.
### Build Artifacts
Add any necessary assets to the release. They should include:
- Linux binary
- GPG signature of the Linux binary
- SHA256 of binary
- Source code
- Wasm binaries of any runtimes
### Release notes
The release notes should list:
- The priority of the release (i.e., how quickly users should upgrade)
- Which native runtimes and their versions are included
- The proposal hashes of the runtimes as built with [srtool](https://gitlab.com/chevdor/srtool)
The release notes may also list:
- Free text at the beginning of the notes mentioning anything important regarding this release
- Notable changes (those labelled with B[1-9]-* labels) separated into sections
### Spec Version
A runtime upgrade must bump the spec number. This may follow a pattern with the client release
(e.g. runtime v12 corresponds to v0.8.12, even if the current runtime is not v11).
### New Migrations
Ensure that any migrations that are required due to storage or logic changes are included in the
`on_runtime_upgrade` function of the appropriate pallets.
### Old Migrations Removed
Any previous `on_runtime_upgrade` functions from old upgrades must be removed to prevent them from
executing a second time.
### Extrinsic Ordering
Offline signing libraries depend on a consistent ordering of call indices and functions. Compare
the metadata of the current and new runtimes and ensure that the `module index, call index` tuples
map to the same set of functions. In case of a breaking change, increase `transaction_version`.
Note: Adding new functions to the runtime does not constitute a breaking change as long as they are
added to the end of a pallet (i.e., does not break any other call index).
### Proxy Filtering
The runtime contains proxy filters that map proxy types to allowable calls. If the new runtime
contains any new calls, verify that the proxy filters are up to date to include them.
### Benchmarks
Run the benchmarking suite with the new runtime and update any function weights if necessary.
### Polkadot JS
Ensure that a release of [Polkadot JS API]() contains any new types or interfaces necessary to
interact with the new runtime.
Polkadot Release Process
------------------------
### Branches
* release-candidate branch: The branch used for staging of the next release.
Named like `release-v0.8.26`
* release branch: The branch to which successful release-candidates are merged
and tagged with the new version. Named literally `release`.
### Notes
* The release-candidate branch *must* be made in the paritytech/polkadot repo in
order for release automation to work correctly
* Any new pushes/merges to the release-candidate branch (for example,
refs/heads/release-v0.8.26) will result in the rc index being bumped (e.g., v0.8.26-rc1
to v0.8.26-rc2) and new wasms built.
### Release workflow
Below are the steps of the release workflow. Steps prefixed with NOACTION are
automated and require no human action.
1. To initiate the release process, branch master off to a release branch and push it to Github:
- `git checkout master; git pull; git checkout -b release-v0.8.26; git push origin refs/heads/release-v0.8.26`
2. NOACTION: The current HEAD of the release-candidate branch is tagged `v0.8.26-rc1`
3. NOACTION: A draft release and runtime WASMs are created for this
release-candidate automatically. A link to the draft release will be linked in
the internal polkadot matrix channel.
4. NOACTION: A new Github issue is created containing a checklist of manual
steps to be completed before we are confident with the release. This will be
linked in Matrix.
5. Complete the steps in the issue created in step 4, signing them off as
completed
6. (optional) If a fix is required to the release-candidate:
1. Merge the fix with `master` first
2. Checkout the release-candidate branch and merge `master`
3. Revert all changes since the creation of the release-candidate that are
**not** required for the fix.
4. Push the release-candidate branch to Github - this is now the new release-
candidate
7. Once happy with the release-candidate, perform the release using the release
script located at `scripts/release.sh` (or perform the steps in that script
manually):
- `./scripts/release.sh v0.8.26`
8. NOACTION: The HEAD of the `release` branch will be tagged with `v0.8.26`,
and a final release will be created on Github.
\ No newline at end of file
# frozen_string_literal: true
require 'base64'
require 'changelogerator'
require 'git'
require 'erb'
require 'toml'
require 'git'
require 'json'
require 'octokit'
require 'toml'
require 'pry'
require_relative './lib.rb'
version = ENV['GITHUB_REF']
current_ref = ENV['GITHUB_REF']
token = ENV['GITHUB_TOKEN']
github_client = Octokit::Client.new(
access_token: token
)
polkadot_path = ENV['GITHUB_WORKSPACE'] + '/polkadot/'
pg = Git.open(polkadot_path)
# Generate an ERB renderer based on the template .erb file
renderer = ERB.new(
......@@ -19,29 +24,29 @@ renderer = ERB.new(
trim_mode: '<>'
)
# get last polkadot version. Use handy Gem::Version for sorting by version
last_version = pg
.tags
.map(&:name)
.grep(/^v\d+\.\d+\.\d+.*$/)
.sort_by { |v| Gem::Version.new(v.slice(1...)) }[-2]
# get ref of last polkadot release
last_ref = "refs/tags/" + github_client.latest_release(ENV['GITHUB_REPOSITORY']).tag_name
polkadot_cl = Changelog.new(
'paritytech/polkadot', last_version, version, token: token
'paritytech/polkadot', last_ref, current_ref, token: token
)
# Get prev and cur substrate SHAs - parse the old and current Cargo.lock for
# polkadot and extract the sha that way.
prev_cargo = TOML::Parser.new(pg.show("#{last_version}:Cargo.lock")).parsed
current_cargo = TOML::Parser.new(pg.show("#{version}:Cargo.lock")).parsed
substrate_prev_sha = prev_cargo['package']
.find { |p| p['name'] == 'sc-cli' }['source']
.split('#').last
substrate_cur_sha = current_cargo['package']
.find { |p| p['name'] == 'sc-cli' }['source']
.split('#').last
# Gets the substrate commit hash used for a given polkadot ref
def get_substrate_commit(client, ref)
cargo = TOML::Parser.new(
Base64.decode64(
client.contents(
ENV['GITHUB_REPOSITORY'],
path: 'Cargo.lock',
query: { ref: "#{ref}"}
).content
)
).parsed
cargo['package'].find { |p| p['name'] == 'sc-cli' }['source'].split('#').last
end
substrate_prev_sha = get_substrate_commit(github_client, last_ref)
substrate_cur_sha = get_substrate_commit(github_client, current_ref)
substrate_cl = Changelog.new(
'paritytech/substrate', substrate_prev_sha, substrate_cur_sha,
......@@ -62,7 +67,7 @@ release_priority = Changelog.highest_priority_for_changes(all_changes)
# Pulled from the previous Github step
rustc_stable = ENV['RUSTC_STABLE']
rustc_nightly = ENV['RUSTC_NIGHTLY']
binding.pry
polkadot_runtime = get_runtime('polkadot', polkadot_path)
kusama_runtime = get_runtime('kusama', polkadot_path)
westend_runtime = get_runtime('westend', polkadot_path)
......
#!/usr/bin/env bash
set -e
# This script is to be run when we are happy with a release candidate.
# It accepts a single argument: version, in the format 'v1.2.3'
version="$1"
if [ -z "$version" ]; then
echo "No version specified, cannot continue"
exit 1
fi
if [[ ! "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "Version should be in the format v1.2.3"
exit 1
fi
echo '[+] Checking out the release branch'
git checkout release
echo '[+] Pulling latest version of the release branch from github'
git pull
echo '[+] Attempting to merge the release-candidate branch to the release branch'
git merge "$version"
echo '[+] Tagging the release'
git tag -s -m "$version" "$version"
echo '[+] Pushing the release branch and tag to Github. A new release will be created shortly'
git push origin release
git push origin "refs/tags/$version"
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment