Allow Locks/Holds/Reserves/Freezes by default when using `pallet_balances` `TestDefaultConfig` (#1880)

Allow Locks/Holds/Reserves/Freezes by default when using
`pallet_balances` `TestDefaultConfig`.

Changes:
- Add missing crate to the workspace
- Remove versions from local dependency links

Maybe it is finally worth it to add this scrip to the CI to find these
things earlier:
[check-deps.py](https://github.com/ggwpez/substrate-scripts/blob/master/import-runtime-repos/check-deps.py).

@paritytech/ci what would be the best location for that check?  
It takes only a second to run, so maybe we can squeeze it into one of
the existing checks?
Otherwise creating a new GH workflow feels a bit wasteful... maybe i can
group it with https://github.com/paritytech/polkadot-sdk/pull/1831

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Make System Parachain trusted Teleporters of each other.

Migration of https://github.com/paritytech/cumulus/pull/2842

---------

Co-authored-by: command-bot <>
Co-authored-by: joe petrowski <[email protected]>

This PR adds a missing env to allow publishing the Docker image.

Introduces an `extract` amount method for `fungible/s` `Imbalance`.

Fix situation where BEEFY initial validator set could not be determined.

If state is unavailable at BEEFY genesis block to get initial validator
set, get the info from header digests. For this, we need to walk back
the chain starting from BEEFY genesis looking for the BEEFY digest
announcing the active validator set for that respective session.

This commit fixes a silly bug where walking back the chain was stopped
when reaching BEEFY genesis block, which is incorrect when BEEFY genesis
is not session boundary block. When BEEFY genesis is set to some random
block within a session, we need to walk back to the start of the session
to see the validator set announcement.

Added regression test for this fix.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1885

Signed-off-by: Adrian Catangiu <[email protected]>

Co-authored-by: command-bot <>

Adding link checker to the CI (closes
https://github.com/paritytech/polkadot-sdk/issues/993). It would be nice
to have the docs people own this and extend accordingly.
Currently all known-bad links are excluded, but we should one-by-one
include those as well until all are fixed.

This check now ensures that 1) no new broken links are introduced into
`.rs` files and 2) that no old links break unnoticed.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

# Description

Follow up for https://github.com/paritytech/polkadot-sdk/pull/1489.
Closes #611 

Before we calculated the channel size during alert expression but in
#1489 a new metric was introduced that reports channel size.
## Changes:
1. updated alert rule to use new metric.

Remove `publish = false` to publish the crate

Following
[polkadot#7314](https://github.com/paritytech/polkadot/pull/7314) and
after merging https://github.com/paritytech/polkadot-sdk/pull/1177 this
PR solves https://github.com/paritytech/polkadot-sdk/issues/1618

The following is a summary of the outcome of the migration.

| Module | Total Accounts | Total stake to unlock | Total deposit to
unreserve |
| ------- | --------------- | --------------------- |
-------------------------- |
| Elections Phragmen | 27 | 1,132.821063320441 ROC | 1.465386531600 ROC
|
| Democracy | 69 | 2733.923509345613 ROC | 0.166666665000 ROC |
| Tips | 4 | N/A | 0.015099999849 ROC |

The migrations will also remove the following amount of keys

103 Democracy keys 🧹
5 Council keys 🧹
1 TechnicalCommittee keys 🧹
25 PhragmenElection keys 🧹
1 TechnicalMembership keys 🧹
9 Tips keys 🧹

- Removal of Arkworks unit tests. These tests were just testing the
arkworks upstream implementation which should be assumed correct. This
is not the place to test well known dependencies.
- Removal of some over-engineering. We just store the calls to Arkworks
in one file. Per-curve sources are not required.
- Docs formatting

---

I also took the opportunity to bump the `bandersnatch-vrfs` crate
revision internally providing some new shiny stuff.

The runtime code of a parachain can be replaced on the relay-chain via:

[cumulus]:
[enact_authorized_upgrade](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/cumulus/pallets/parachain-system/src/lib.rs#L661);
this is used for a runtime upgrade when a parachain is not bricked.

[polkadot] (these are used when a parachain is bricked):
-
[force_set_current_code](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/polkadot/runtime/parachains/src/paras/mod.rs#L823):
immediately changes the runtime code of a given para without a pvf check
(root).
-
[force_schedule_code_upgrade](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/polkadot/runtime/parachains/src/paras/mod.rs#L864):
schedules a change to the runtime code of a given para including a pvf
check of the new code (root).
-
[schedule_code_upgrade](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/polkadot/runtime/common/src/paras_registrar.rs#L395):
schedules a change to the runtime code of a given para including a pvf
check of the new code. Besides root, the parachain or parachain manager
can call this extrinsic given that the parachain is unlocked.

Polkadot signals a parachain to be ready for a runtime upgrade through
the
[GoAhead](https://github.com/paritytech/polkadot-sdk/blob/e4949344

/polkadot/primitives/src/v5/mod.rs#L1229)
signal.

When in cumulus `enact_authorized_upgrade` is executed, the same
underlying helper function of `force_schedule_code_upgrade` &
`schedule_code_upgrade`:
[schedule_code_upgrade](https://github.com/paritytech/polkadot/blob/09b61286da11921a3dda0a8e4015ceb9ef9cffca/runtime/parachains/src/paras/mod.rs#L1778),
is called on the relay-chain, which sets the `GoAhead` signal (if the
pvf is accepted).

If Cumulus receives the `GoAhead` signal from polkadot without having
the `PendingValidationCode` ready, it will panic
([ref](https://github.com/paritytech/polkadot/pull/7412)). For
`enact_authorized_upgrade` we know for sure the `PendingValidationCode`
is set. On the contrary, for `force_schedule_code_upgrade` &
`schedule_code_upgrade` this is not the case.

This PR includes a flag such that the `GoAhead` signal will only be set
when a runtime upgrade is enacted by the parachain
(`enact_authorized_upgrade`).

additional info: https://github.com/paritytech/polkadot/pull/7412

Closes #641

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This PR refactors the staking ledger logic to encapsulate all reads and
mutations of `Ledger`, `Bonded`, `Payee` and stake locks within the
`StakingLedger` struct implementation.

With these changes, all the reads and mutations to the `Ledger`, `Payee`
and `Bonded` storage map should be done through the methods exposed by
StakingLedger to ensure the data and lock consistency of the operations.
The new introduced methods that mutate and read Ledger are:

- `ledger.update()`: inserts/updates a staking ledger in storage;
updates staking locks accordingly (and ledger.bond(), which is synthatic
sugar for ledger.update())
- `ledger.kill()`: removes all Bonded and StakingLedger related data for
a given ledger; updates staking locks accordingly;
`StakingLedger::get(account)`: queries both the `Bonded` and `Ledger`
storages and returns a `Option<StakingLedger>`. The pallet impl exposes
fn ledger(account) as synthatic sugar for `StakingLedger::get(account)`.

Retrieving a ledger with `StakingLedger::get()` can be done by providing
either a stash or controller account. The input must be wrapped in a
`StakingAccount` variant (Stash or Controller) which is treated
accordingly. This simplifies the caller API but will eventually be
deprecated once we completely get rid of the controller account in
staking. However, this refactor will help with the work necessary when
completely removing the controller.

Other goals:

- No logical changes have been introduced in this PR;
- No breaking changes or updates in wallets required;
- No new storage items or need to perform storage migrations;
- Centralise the changes to bonds and ledger updates to simplify the
OnStakingUpdate updates to the target list (related to
https://github.com/paritytech/polkadot-sdk/issues/443)

Note: it would be great to prevent or at least raise a warning if
`Ledger<T>`, `Payee<T>` and `Bonded<T>` storage types are accessed
outside the `StakingLedger` implementation. This PR should not get
blocked by that feature, but there's a tracking issue here
https://github.com/paritytech/polkadot-sdk/issues/149

Related and step towards
https://github.com/paritytech/polkadot-sdk/issues/443

BEEFY needs two cryptographic keys at the same time. Validators should
sign BEEFY payload using both ECDSA and BLS key. The network will gossip
a payload which contains a valid ECDSA key. The prover nodes aggregate
the BLS keys if aggregation fails to verifies the validator which
provided a valid ECDSA signature but an invalid BLS signature is subject
to slashing.

As such BEEFY session should be initiated with both key. Currently there
is no straight forward way of doing so, beside having a session with
RuntimeApp corresponding to a crypto scheme contains both keys.

This pull request implement a generic paired_crypto scheme as well as
implementing it for (ECDSA, BLS) pair.

---------

Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Robert Hambrock <[email protected]>

closes #695

Could potentially be helpful to preserving caches when applicable, as
discussed in #685

kusama address: FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK

# Description
- What does this PR do?
  - link added
- Why are these changes needed?
  - improve docs
- How were these changes implemented and what do they affect?
  - only concerns docs

closes #622 

Pros:
* simpler interface, just functions:
`create_runtime_from_artifact_bytes()` and `execute_artifact()`

Cons:
* extra overhead of constructing executor semantics each time

I could make it a combination of
* `create_runtime_config(params)` (such that we could clone the
constructed semantics)
* `create_runtime(blob, config)`
* `execute_artifact(blob, config, params)`

Not sure if it's worth it though.

---------

Co-authored-by: Bastian Köcher <[email protected]>

The lint `clippy::clone_double_ref` was renamed to
`suspicious_double_ref_op` in [this
commit](https://github.com/rust-lang/rust/commit/5c99175a9efcaa3d65712c119f361add22e3a859)
(thanks @liamaharon) and now generates a lot of noise in the terminal
when run both in CI and locally with 1.73.

This renames the lint in line with this, but does not change
functionality.

May cause issues for people running earlier versions locally - @altaua
requesting review to get your opinion on this.

- This PR refactors `alliance/src/benchmarkings.rs` to use benchmarking
v2. These changes are needed to improve the readability and
maintainability of the benchmarking code.

- No known issue to backlink.

## Local Testing 
1. `cargo build --features runtime-benchmarks` 
2. `cargo run --locked --release -p node-cli --bin substrate-node
--features runtime-benchmarks -- benchmark pallet --execution wasm
--wasm-execution compiled --chain dev --pallet "*" --extrinsic "*"
--steps 2 --repeat 1`

Co-authored-by: Marcin S <[email protected]>

Part of #171

- Remove cached messages used for deduplication in `GossipValidator`
since they're already deduplicated in upper layer `NetworkGossip`.
- Add cache for "justified rounds" to quickly discard any further (even
if potentially different) justifications at the gossip level, once a
valid one (for a respective round) is submitted to the worker.
- Add short-circuit in worker `finalize()` method to not attempt to
finalize same block multiple times (for example when we get
justifications for same block from multiple components like
block-import, gossip or on-demand).
- Change a test which had A LOT of latency in syncing blocks for some
weird reason and would only run after ~150seconds. It now runs
instantly.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1728

Fixes https://github.com/paritytech/polkadot-sdk/issues/1839

Currently, `composite_enum`s do not support pallet instances. This PR
allows the following:
```rust
	#[pallet::composite_enum]
	pub enum HoldReason<I: 'static = ()> {
		SomeHoldReason
	}
```

### Todo

- [x]  UI Test

# Description

In a couple of cases, there were links pointing to the w3f github pages
domain. In other instances, there were links pointing to the old
polkadot repo's github pages. Both of these are now pointing to the
relevant links in
https://paritytech.github.io/polkadot-sdk/book/index.html.

These changes were made specifically because the w3f github pages
returns a 404, and while fixing the links, the old polkadot repo links
were touched up as well even if they do redirect properly.

This shouldn't affect anything as these are documentation link changes
only.

Exposes disabled validators list via a runtime API.

---------

Co-authored-by: ordian <[email protected]>
Co-authored-by: ordian <[email protected]>

This PR adds a `try_state` hook for the `Treasury` pallet.
Part of #239.

This pr resolves https://github.com/paritytech/polkadot-sdk/issues/1428.

*Added only to Kusama for now*

I did raise it
[here](https://github.com/polkadot-fellows/runtimes/pull/19) and we
discussed creating a chopsticks test to run an end-to-end test

however, to do that I will need a build agent/custom runner that is
powerful enough to run the build

I will be doing that separately as I still think having chopsticks test
your runtime with each commit will be very powerful and extremely useful
for the ecosystem

For now I have used XCM simulator and replicated what the other reserve
tests do

---------

Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: command-bot <>

This PR refactors `identity/benchmarkings.rs` to use benchmarking v2.
These changes are needed to improve the readability and maintainability
of the benchmarking code. Changes were implemented using
[this](https://github.com/paritytech/polkadot-sdk/commit/9ec80090

)
commit as a guide. The logic of the benchmarks remains the same.

No known issue to backlink.

## Local Testing
To test the new benchmarks:
1. `cargo build --features runtime-benchmarks`
2. `./target/debug/polkadot benchmark pallet --steps=5 --repeat=2
--pallet=pallet_identity --extrinsic='*'`

---------

Co-authored-by: Richard Melkonian <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

# Desription

## Summary 

This PR introduces several nits and tweaks to xcm emulator tests for
system parachains.

## Explanation

**Deduplicate `XcmPallet::send(` with root origin code**
- Introduced `send_transact_to_parachain` which could be easily reuse
for scenarios like _governance call from relay chain to parachain_.

**Refactor `send_transact_sudo_from_relay_to_system_para_works`**
- Test covered just one use-case which was moved to the
`do_force_create_asset_from_relay_to_system_para`, so now we can extend
this test with more _governance-like_ senarios.
- Renamed to
`send_transact_as_superuser_from_relay_to_system_para_works`.

**Remove `send_transact_native_from_relay_to_system_para_fails` test**
- This test and/or description is kind of misleading, because system
paras support Native from relay chain by `RelayChainAsNative` with
correct xcm origin.
- It tested only sending on relay chain which should go directly to the
relay chain unit-tests (does not even need to be in xcm emulator level).

## Future directions

Check restructure parachains integration tests
[issue](https://github.com/paritytech/polkadot-sdk/issues/1389) and [PR
with more TODOs](https://github.com/paritytech/polkadot-sdk/pull/1693).

---------

Co-authored-by: Ignacio Palacios <[email protected]>

Needs https://github.com/sam0x17/macro_magic/pull/13

The associated PR allows the export of tokens from macro_magic at the
specified path. This fixes the path issue in derive-impl. Now, we can
import the default config using the standard rust syntax:

```rust
use frame_system::config_preludes::TestDefaultConfig;

[derive_impl(TestDefaultConfig as frame_system::DefaultConfig)]
impl frame_system::DefaultConfig for Test {
   //....
}
```

# Description

Upgrades `macro_magic` to 0.4.3, which introduces the ability to have
`export_tokens` use the same name as the underlying item for its
auto-generated macro name. Ultimately this will allow for better dev ux
in our derive_impl feature.

Co-authored-by: Francisco Aguirre <[email protected]>

Original PR https://github.com/paritytech/substrate/pull/14746

---

## Fixing stall

### Introduction
I experienced an apparent stall downloading state from
`https://rococo-try-runtime-node.parity-chains.parity.io:443` which was
having networking difficulties only responding to my JSONRPC requests
with 50-200KB/s of bandwidth.

This PR fixes the issue causing the stall, and generally improves
performance remote-ext when it downloads state by greatly reducing the
chances of a timeout occuring.

### Description
Introduces a new `REQUEST_DURATION_TARGET` constant and modifies
`get_storage_data_dynamic_batch_size` to

- Increase or decrease the batch size of the next request depending on
whether the elapsed time of the last request was gt or lt the target
- Reset the batch size to 1 if the request times out

This fixes an issue on slow connections that can otherwise cause
multiple timeouts and a stalled download when:

1. The batch size increases rapidly as remote-ext downloads keys with
small associated storage values
2. remote-ext tries to process a large series of subsequent keys all
with extremely large associated storage values (Rococo has a series of
keys 1-5MB large)
3. The huge storage values download for 5 minutes until the request
times out
4. The partially downloaded keys are thrown out and remote-ext tries
again with a smaller batch size, but the batch size is still far too
large and takes 5 minutes to be reduced again
5. The download will be essentially stalled for many hours while the
above step cycles


After this PR, the request size will

- Not grow as large to begin with, as it is regulated downwards as the
request duration exceeds the target
- Drop immediately to 1 if the request times out. A timeout indicates
the keys next in line to download have extremely large storage values
compared to previously downloaded keys, and we need to reset the batch
size to figure out what our new ideal batch size is. By not resetting
down to 1, we risk the next request timing out again.

## Reducing memory

As suggested by @bkchr, I adjusted `get_storage_data_dynamic_batch_size`
from being recursive to a loop which allows removing a bunch of clones
that were chewing through a lot of memory. I noticed actually it was
using up to 50GB swap previously when downloading Polkadot keys on a
slow connection, because it needed to recurse and clone a lot.

After this change it uses only ~1.5GB memory.

# Description
Update the DNS name of bootnodes to unify the deployment. 

Each bootnode have 3 port exposed: `30333, 30334, 443`. Before, we had
different DNS names for `30333, 30334` and `443` ports. It may confuse
people and give the impression that it is two different nodes. Fixing it
by using a single domain for all