Closes https://github.com/paritytech/polkadot-sdk-docs/issues/70

WIP PR for an overview of how to develop tokens in FRAME. 

- [x] Tokens in Substrate Ref Doc
  - High-level overview of the token-related logic in FRAME
- Improve docs with better explanation of how holds, freezes, ed, free
balance, etc, all work
- [x] Update `pallet_balances` docs
  - Clearly mark what is deprecated (currency)
- [x] Write fungible trait docs
- [x] Evaluate and if required update `pallet_assets`, `pallet_uniques`,
`pallet_nfts` docs
- [x] Absorb https://github.com/paritytech/polkadot-sdk/pull/2683/
- [x] Audit individual trait method docs, and improve if possible

Feel free to suggest additional TODOs for this PR in the comments

---------

Co-authored-by: Bill Laboon <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3887.

Currently, all protocols use the same metric name for
`mpsc-notification-to-protocol` this is bad because we can't actually
tell which protocol might cause problems.

This patch proposes we derive the name of the metric from the protocol
name, so that we have separate metrics for each protocol and properly
detect which one is having problem processing its messages.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Part of: https://github.com/paritytech/polkadot-sdk/issues/239

Polkadot address: 12GyGD3QhT4i2JJpNzvMf96sxxBLWymz4RdGCxRH5Rj5agKW

Fix reference links

Co-authored-by: Bastian Köcher <[email protected]>

Cleanup tests (-2.7k lines !) using some builder patterns to build
pallet_contracts api calls

This PR significantly refactors the runtime API implementations to
improve project structure, maintainability, and readability. Key changes
include:

1. **Enhancing Visibility**: Adjusts the visibility of
`RUNTIME_API_VERSIONS` in `impl_runtime_apis.rs` to `pub`, making it
accessible throughout the runtime module.
2. **Centralizing API Implementations**: Introduces a new file,
`apis.rs`, within the parachain template's runtime directory.
3. **Streamlining `lib.rs`**: Updates the main runtime library file to
reflect these structural changes. It removes redundant API
implementations and points `VERSION` to the newly exposed
`RUNTIME_API_VERSIONS` from `apis.rs`, simplifying the overall runtime
configuration.

### Motivations Behind the Refactoring:
- **Improved Project Structure**: Centralizing API implementations in
`apis.rs` offers a clearer, more navigable project structure.
- **Better Readability**: Streamlining `lib.rs` and reducing clutter
enhance readability, making it easier for new contributors to understand
the project layout and logic.

### Summary of Changes:
- Made `RUNTIME_API_VERSIONS` public in `impl_runtime_apis.rs`.
- Added `apis.rs` to centralize runtime API implementations.
- Streamlined `lib.rs` to adjust to the refactored project structure.

This PR add `#[deprecated]` attribute to v1 and v2 of the schedule
trait. Proposed in this issue:
https://github.com/paritytech/polkadot-sdk/issues/3676

```rust
#[allow(deprecated)]
#[deprecated = "traits::schedule::v1 is deprecated. Please use v3 instead."]
pub mod v1 {
...
}

#[allow(deprecated)]
#[deprecated = "traits::schedule::v2 is deprecated. Please use v3 instead."]
pub mod v2 {
...
}
```

polkadot address: 19nSqFQorfF2HxD3oBzWM3oCh4SaCRKWt1yvmgaPYGCo71J

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

There has been a case that a referenda failed because the length given
to `submit` was incorrect. The pallet can actually check the length if
the pre-image already exists to ensure that these kind of issues are not
happening again.

When the scheduler fails to lookup a `call`, it should unrequest it,
because it will not be required anymore.

This PR adds a new extrinsic `Call::restore_ledger ` gated by
`StakingAdmin` origin that restores a corrupted staking ledger. This
extrinsic will be used to recover ledgers that were affected by the
issue discussed in
https://github.com/paritytech/polkadot-sdk/issues/3245.

The extrinsic will re-write the storage items associated with a stash
account provided as input parameter. The data used to reset the ledger
can be either i) fetched on-chain or ii) partially/totally set by the
input parameters of the call.

In order to use on-chain data to restore the staking locks, we need a
way to read the current lock in the balances pallet. This PR adds a
`InspectLockableCurrency` trait and implements it in the pallet
balances. An alternative would be to tightly couple staking with the
pallet balances but that's inelegant (an example of how it would look
like in [this
branch](https://github.com/paritytech/polkadot-sdk/tree/gpestana/ledger-badstate-clean_tightly)).

More details on the type of corruptions and corresponding fixes
https://hackmd.io/DLb5jEYWSmmvqXC9ae4yRg?view#/

We verified that the `Call::restore_ledger` does fix all current
corrupted ledgers in Polkadot and Kusama. You can verify it here
https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA.

**Changes introduced**
- Adds `Call::restore_ledger ` extrinsic to recover a corrupted ledger;
- Adds trait `frame_support::traits::currency::InspectLockableCurrency`
to allow external pallets to read current locks given an account and
lock ID;
- Implements the `InspectLockableCurrency` in the pallet-balances.
- Adds staking locks try-runtime checks
(https://github.com/paritytech/polkadot-sdk/issues/3751)

**Todo**
- [x] benchmark `Call::restore_ledger`
- [x] throughout testing of all ledger recovering cases
- [x] consider adding the staking locks try-runtime checks to this PR
(https://github.com/paritytech/polkadot-sdk/issues/3751)
- [x] simulate restoring all ledgers
(https://hackmd.io/Dsa2tvhISNSs7zcqriTaxQ?view) in Polkadot and Kusama
using chopsticks -- https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA

Related to https://github.com/paritytech/polkadot-sdk/issues/3245
Closes https://github.com/paritytech/polkadot-sdk/issues/3751

---------

Co-authored-by: command-bot <>

This will make it possible to use remaining weight on idle for
processing enqueued messages.
More context here https://github.com/paritytech/polkadot-sdk/issues/3709

---------

Co-authored-by: Adrian Catangiu <[email protected]>

The authority-discovery mechanism has implemented a few exponential
timers for:
- publishing the authority records
- goes from 2 seconds (when freshly booted) to 1 hour if the node is
long-running
  - set to 1 hour after successfully publishing the authority record
- discovering other authority records
- goes from 2 seconds (when freshly booted) to 10 minutes if the node is
long-running

This PR resets the exponential publishing and discovery interval to
defaults ensuring that long-running nodes:
- will retry publishing the authority records as aggressively as freshly
booted nodes
- Currently, if a long-running node fails to publish the DHT record when
the keys change (ie DhtEvent::ValuePutFailed), it will only retry after
1 hour
- will rediscover other authorities faster (since there is a chance that
other authority keys changed)

The subp2p-explorer has difficulties discovering the authorities when
the authority set changes in the first few hours. This might be entirely
due to the recursive nature of the DHT and the needed time to propagate
the records. However, there is a small chance that the authority
publishing failed and is only retried in 1h.

Let me know if this makes sense 🙏

 

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>

`execute` and `send` try to decode the xcm in the parameters before
reaching the filter line.
The new extrinsics decode only after the filter line.
These should be used instead of the old ones.

## TODO
- [x] Tests
- [x] Generate weights
- [x] Deprecation issue ->
https://github.com/paritytech/polkadot-sdk/issues/3771
- [x] PRDoc
- [x] Handle error in pallet-contracts

This would make writing XCMs in PJS Apps more difficult, but here's the
fix for that: https://github.com/polkadot-js/apps/pull/10350.
Already deployed! https://polkadot.js.org/apps/#/utilities/xcm

Supersedes https://github.com/paritytech/polkadot-sdk/pull/1798/

---------

Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <[email protected]>

In preparation for the merkleized metadata, we need to ensure that
constants are actually constant. If we want to test the unsigned phase
we could for example just disable signed voter. Or we add some extra
mechanism to the pallet to disable the signed phase from time to time.

---------

Co-authored-by: Ankan <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

This PR notifies broker pallet for any parachain slot swaps performed on
the relay chain. This is achieved by registering an `OnSwap` for the the
`coretime` pallet. The hook sends XCM message to the broker chain and
invokes a new extrinsic `swap_leases` which updates `Leases` storage
item (which keeps the legacy parachain leases).

I made two assumptions in this PR:
1.
[`Leases`](https://github.com/paritytech/polkadot-sdk/blob/4987d798/substrate/frame/broker/src/lib.rs#L120)
in `broker` pallet and
[`Leases`](https://github.com/paritytech/polkadot-sdk/blob/4987d798

/polkadot/runtime/common/src/slots/mod.rs#L118)
in `slots` pallet are in sync.
2. `swap_leases` extrinsic from `broker` pallet can be triggered only by
root or by the XCM message from the relay chain. If not - the extrinsic
will generate an error and do nothing.

As a side effect from the changes `OnSwap` trait is moved from
runtime/common/traits.rs to runtime/parachains. Otherwise it is not
accessible from `broker` pallet.

Closes https://github.com/paritytech/polkadot-sdk/issues/3552

TODOs:

- [x] Weights
- [x] Tests

---------

Co-authored-by: command-bot <>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~

Another simple refactory to prune some duplicate code

Follow up of: https://github.com/paritytech/polkadot-sdk/pull/3684

Make sure explicitly set by the operator public addresses go first in
the authority discovery DHT records.

Also update `Discovery` behavior to eliminate duplicates in the returned
addresses.

This PR should improve situation with
https://github.com/paritytech/polkadot-sdk/issues/3519.

Obsoletes https://github.com/paritytech/polkadot-sdk/pull/3657.

Make taplo happy

There was a typo, so now, there's no more typo.

Co-authored-by: Liam Aharon <[email protected]>

_This PR is being continued from
https://github.com/paritytech/polkadot-sdk/pull/2206, which was closed
when the developer_hub was merged._
closes https://github.com/paritytech/polkadot-sdk-docs/issues/44

---
# Description

This PR adds a reference document to the `developer-hub` crate (see
https://github.com/paritytech/polkadot-sdk/pull/2102). This specific
reference document covers defensive programming practices common within
the context of developing a runtime with Substrate.

In particular, this covers the following areas: 

- Default behavior of how Rust deals with numbers in general
- How to deal with floating point numbers in runtime / fixed point
arithmetic
- How to deal with Integer overflows
- General "safe math" / defensive programming practices for common
pallet development scenarios
- Defensive traits that exist within Substrate, i.e.,
`defensive_saturating_add `, `defensive_unwrap_or`
- More general defensive programming examples (keep it concise)
- Link to relevant examples where these practices are actually in
production / being used
- Unwrapping (or rather lack thereof) 101

todo
-- 
- [x] Apply feedback from previous PR
- [x] This may warrant a PR to append some of these docs to
`sp_arithmetic`

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Radha <[email protected]>

Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.75 to 1.0.81.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/anyhow/releases">anyhow's
releases</a>.</em></p>
<blockquote>
<h2>1.0.81</h2>
<ul>
<li>Make backtrace support available when using -Dwarnings (<a
href="https://redirect.github.com/dtolnay/anyhow/issues/354">#354</a>)</li>
</ul>
<h2>1.0.80</h2>
<ul>
<li>Fix unused_imports warnings when compiled by rustc 1.78</li>
</ul>
<h2>1.0.79</h2>
<ul>
<li>Work around improperly cached build script result by sccache (<a
href="https://redirect.github.com/dtolnay/anyhow/issues/340">#340</a>)</li>
</ul>
<h2>1.0.78</h2>
<ul>
<li>Reduce spurious rebuilds under RustRover IDE when using a nightly
toolchain (<a
href="https://redirect.github.com/dtolnay/anyhow/issues/337">#337</a>)</li>
</ul>
<h2>1.0.77</h2>
<ul>
<li>Make <code>anyhow::Error::backtrace</code> available on stable Rust
compilers 1.65+ (<a
href="https://redirect.github.com/dtolnay/anyhow/issues/293">#293</a>,
thanks <a
href="https://github.com/LukasKalbertodt"><code>@​LukasKalbertodt</code></a>)</li>
</ul>
<h2>1.0.76</h2>
<ul>
<li>Opt in to <code>unsafe_op_in_unsafe_fn</code> lint (<a
href="https://redirect.github.com/dtolnay/anyhow/issues/329">#329</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/anyhow/commit/4aad4edebd9f09247d6c6b6784419a74bb116829"><code>4aad4ed</code></a>
Release 1.0.81</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/8be90917c603199c5d1fdd73984237f023768e22"><code>8be9091</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/anyhow/issues/354">#354</a>
from dtolnay/deadcode</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/a2eb7dd5e13add83f254b6dac0f68e043effc521"><code>a2eb7dd</code></a>
Make compatible with -Dwarnings</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/54437197ee79c20678db433d98616fab7ddff1a5"><code>5443719</code></a>
Release 1.0.80</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/dfc7bc07d4c41b61093c3251ed82becb51810bd4"><code>dfc7bc0c

</code></a>
Work around prelude redundant import warnings</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/6e4f86b48b5182ec71dbc8e308db9dc91e2ec8a5"><code>6e4f86b</code></a>
Import from alloc not std, where possible</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/f885a133ede579c45e90ab489455126403d05db1"><code>f885a13</code></a>
Ignore incompatible_msrv clippy false positives in test</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/fefbcbcb0b336a2d6c2ce6f0ee6d3fd02ef2cd3b"><code>fefbcbc</code></a>
Ignore incompatible_msrv clippy lint</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/78f2d81cc71b79050a2fda270c45ff267557d853"><code>78f2d81</code></a>
Update ui test suite to nightly-2024-02-08</li>
<li><a
href="https://github.com/dtolnay/anyhow/commit/edd88d3a43f11f1931330d3dd54189353ef00203"><code>edd88d3</code></a>
Update ui test suite to nightly-2024-01-31</li>
<li>Additional commits viewable in <a
href="https://github.com/dtolnay/anyhow/compare/1.0.75...1.0.81">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=anyhow&package-manager=cargo&previous-version=1.0.75&new-version=1.0.81)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: command-bot <>

Introduces `CryptoBytes` type defined as:

```rust
pub struct CryptoBytes<const N: usize, Tag = ()>(pub [u8; N], PhantomData<fn() -> Tag>);
```

The type implements a bunch of methods and traits which are typically
expected from a byte array newtype
(NOTE: some of the methods and trait implementations IMO are a bit
redundant, but I decided to maintain them all to not change too much
stuff in this PR)

It also introduces two (generic) typical consumers of `CryptoBytes`:
`PublicBytes` and `SignatureBytes`.

```rust
pub struct PublicTag;
pub PublicBytes<const N: usize, CryptoTag> = CryptoBytes<N, (PublicTag, CryptoTag)>;

pub struct SignatureTag;
pub SignatureBytes<const N: usize, CryptoTag> = CryptoBytes<N, (SignatureTag, CryptoTag)>;
```

Both of them use a tag to differentiate the two types at a higher level.
Downstream specializations will further specialize using a dedicated
crypto tag. For example in ECDSA:


```rust
pub struct EcdsaTag;

pub type Public = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>;
pub type Signature = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>;
```

Overall we have a cleaner and most importantly **consistent** code for
all the types involved

All these details are opaque to the end user which can use `Public` and
`Signature` for the cryptos as before

This PR enables the `transaction_unstable_broadcast ` and
`transaction_unstable_stop` RPC API.

Since the API is unstable, we don't need to expose this in the release
notes.

After merging this, we could validate the API in subxt and stabilize it.

Spec PR that stabilizes the API:
https://github.com/paritytech/json-rpc-interface-spec/pull/139

cc @paritytech/subxt-team

Signed-off-by: Alexandru Vasile <[email protected]>

Part of #3326 

cc @Kianenigma @ggwpez @liamaharon

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>

Changes:
- Port the `pov_mode` attribute from the V1 syntax to V2
- Update `pallet-whitelist` and `frame-benchmarking-pallet-pov`

Follow up: also allow this attribute on top-level benchmark modules.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>

Currently the xcm-executor returns an `Unimplemented` error if it
receives any HRMP-related instruction.
What I propose here, which is what we are currently doing in our forked
executor at polimec, is to introduce a trait implemented by the executor
which will handle those instructions.

This way, if parachains want to keep the default behavior, they just use
`()` and it will return unimplemented, but they can also implement their
own logic to establish HRMP channels with other chains in an automated
fashion, without requiring to go through governance.

Our implementation is mentioned in the [polkadot HRMP
docs](https://arc.net/l/quote/hduiivbu), and it was suggested to us to
submit a PR to add these changes to polkadot-sdk.

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: command-bot <>

There is a problem in the way we update `authorithy-discovery` next keys
and because of that nodes that enter the active set would be noticed at
the start of the session they become active, instead of the start of the
previous session as it was intended. This is problematic because:

1. The node itself advertises its addresses on the DHT only when it
notices it should become active on around ~10m loop, so in this case it
would notice after it becomes active.
2. The other nodes won't be able to detect the new nodes addresses at
the beginning of the session, so it won't added them to the reserved
set.

With 1 + 2, we end-up in a situation where the the new node won't be
able to properly connect to its peers because it won't be in its peers
reserved set. Now, the nodes accept by default`MIN_GOSSIP_PEERS: usize =
25` connections to nodes that are not in the reserved set, but given
Kusama size(> 1000 nodes) you could easily have more than`25` new nodes
entering the active set or simply the nodes don't have slots anymore
because, they already have connections to peers not in the active set.

In the end what the node would notice is 0 backing rewards because it
wasn't directly connected to the peers in its backing group.

## Root-cause

The flow is like this:
1. At BAD_SESSION - 1, in `rotate_session` new nodes are added to
QueuedKeys
https://github.com/paritytech/polkadot-sdk/blob/02e1a7f4/substrate/frame/session/src/lib.rs#L609
```
 <QueuedKeys<T>>::put(queued_amalgamated.clone());
<QueuedChanged<T>>::put(next_changed);
```
2. AuthorityDiscovery::on_new_session is called with `changed` being the
value of `<QueuedChanged<T>>:` at BAD_SESSION - **2** because it was
saved before being updated
https://github.com/paritytech/polkadot-sdk/blob/02e1a7f4/substrate/frame/session/src/lib.rs#L613
3. At BAD_SESSION - 1, `AuthorityDiscovery::on_new_session` doesn't
updated its next_keys because `changed` was false.
4. For the entire durations of `BAD_SESSION - 1` everyone calling
runtime api `authorities`(should return past, present and future
authorities) won't discover the nodes that should become active .
5. At the beginning of BAD_SESSION, all nodes discover the new nodes are
authorities, but it is already too late because reserved_nodes are
updated only at the beginning of the session by the `gossip-support`.
See above why this bad.

## Fix
Update next keys with the queued_validators at every session, not matter
the value of `changed` this is the same way babe pallet correctly does
it.
https://github.com/paritytech/polkadot-sdk/blob/02e1a7f4

/substrate/frame/babe/src/lib.rs#L655

## Notes

- The issue doesn't reproduce with proof-authorities changes like
`versi` because `changed` would always be true and `AuthorityDiscovery`
correctly updates its next_keys every time.
- Confirmed at session `37651` on kusama that this is exactly what it
happens by looking at blocks with polkadot.js.

## TODO
- [ ] Move versi on proof of stake and properly test before and after
fix to confirm there is no other issue.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Removed the `pallet::getter` macro call from storage type definitions
and added the corresponding implementations directly.
fixes #3330  

polkadot address: 14JzTPPUd8x8phKi8qLxHgNTnTMg6DUukCLXoWprejkaHXPz

---------

Co-authored-by: Bastian Köcher <[email protected]>

This PR removes sp-std crate from substrate/primitives sub-directories.

For now crates that have `pub use` of sp-std or export macros that would
necessitate users of the macros to `extern crate alloc` have been
excluded from this PR.

There should be no breaking changes in this PR.

---------

Co-authored-by: Koute <[email protected]>

Bumps the known_good_semver group with 3 updates:
[log](https://github.com/rust-lang/log),
[syn](https://github.com/dtolnay/syn) and
[clap](https://github.com/clap-rs/clap).

Updates `log` from 0.4.20 to 0.4.21
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's
changelog</a>.</em></p>
<blockquote>
<h2>[0.4.21] - 2024-02-27</h2>
<h2>What's Changed</h2>
<ul>
<li>Minor clippy nits by <a
href="https://github.com/nyurik"><code>@​nyurik</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/578">rust-lang/log#578</a></li>
<li>Simplify Display impl by <a
href="https://github.com/nyurik"><code>@​nyurik</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/579">rust-lang/log#579</a></li>
<li>Set all crates to 2021 edition by <a
href="https://github.com/nyurik"><code>@​nyurik</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/580">rust-lang/log#580</a></li>
<li>Various changes based on review by <a
href="https://github.com/Thomasdezeeuw"><code>@​Thomasdezeeuw</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/583">rust-lang/log#583</a></li>
<li>Fix typo in file_static() method doc by <a
href="https://github.com/dimo414"><code>@​dimo414</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/590">rust-lang/log#590</a></li>
<li>Specialize empty key value pairs by <a
href="https://github.com/EFanZh"><code>@​EFanZh</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/576">rust-lang/log#576</a></li>
<li>Fix incorrect lifetime in Value::to_str() by <a
href="https://github.com/peterjoel"><code>@​peterjoel</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/587">rust-lang/log#587</a></li>
<li>Remove some API of the key-value feature by <a
href="https://github.com/Thomasdezeeuw"><code>@​Thomasdezeeuw</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/585">rust-lang/log#585</a></li>
<li>Add logcontrol-log and log-reload by <a
href="https://github.com/swsnr"><code>@​swsnr</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/595">rust-lang/log#595</a></li>
<li>Add Serialization section to kv::Value docs by <a
href="https://github.com/Thomasdezeeuw"><code>@​Thomasdezeeuw</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/593">rust-lang/log#593</a></li>
<li>Rename Value::to_str to to_cow_str by <a
href="https://github.com/Thomasdezeeuw"><code>@​Thomasdezeeuw</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/592">rust-lang/log#592</a></li>
<li>Clarify documentation and simplify initialization of
<code>STATIC_MAX_LEVEL</code> by <a
href="https://github.com/ptosi"><code>@​ptosi</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/594">rust-lang/log#594</a></li>
<li>Update docs to 2021 edition, test by <a
href="https://github.com/nyurik"><code>@​nyurik</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/577">rust-lang/log#577</a></li>
<li>Add &quot;alterable_logger&quot; link to README.md by <a
href="https://github.com/brummer-simon"><code>@​brummer-simon</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/589">rust-lang/log#589</a></li>
<li>Normalize line ending by <a
href="https://github.com/EFanZh"><code>@​EFanZh</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/602">rust-lang/log#602</a></li>
<li>Remove <code>ok_or</code> in favor of <code>Option::ok_or</code> by
<a
href="https://github.com/AngelicosPhosphoros"><code>@​AngelicosPhosphoros</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/607">rust-lang/log#607</a></li>
<li>Use <code>Acquire</code> ordering for initialization check by <a
href="https://github.com/AngelicosPhosphoros"><code>@​AngelicosPhosphoros</code></a>
in <a
href="https://redirect.github.com/rust-lang/log/pull/610">rust-lang/log#610</a></li>
<li>Get structured logging API ready for stabilization by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/rust-lang/log/pull/613">rust-lang/log#613</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/nyurik"><code>@​nyurik</code></a> made
their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/578">rust-lang/log#578</a></li>
<li><a href="https://github.com/dimo414"><code>@​dimo414</code></a> made
their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/590">rust-lang/log#590</a></li>
<li><a href="https://github.com/peterjoel"><code>@​peterjoel</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/587">rust-lang/log#587</a></li>
<li><a href="https://github.com/ptosi"><code>@​ptosi</code></a> made
their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/594">rust-lang/log#594</a></li>
<li><a
href="https://github.com/brummer-simon"><code>@​brummer-simon</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/589">rust-lang/log#589</a></li>
<li><a
href="https://github.com/AngelicosPhosphoros"><code>@​AngelicosPhosphoros</code></a>
made their first contribution in <a
href="https://redirect.github.com/rust-lang/log/pull/607">rust-lang/log#607</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/log/commit/3ccdc286fef3076747fe18a2a93658ea4d4ae012"><code>3ccdc28</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/617">#617</a>
from rust-lang/cargo/0.4.21</li>
<li><a
href="https://github.com/rust-lang/log/commit/6153cb289f0e7b80f00ae07dbe5ee41cf3d3fcb0"><code>6153cb2</code></a>
prepare for 0.4.21 release</li>
<li><a
href="https://github.com/rust-lang/log/commit/f0f74946a4bfb02cfc407795a3499c4b69d7a290"><code>f0f7494</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/log/issues/613">#613</a>
from rust-lang/feat/kv-cleanup</li>
<li><a
href="https://github.com/rust-lang/log/commit/2b220bf3b705f2abc0ee591c7eb17972a979da3a"><code>2b220bf</code></a>
clean up structured logging example</li>
<li><a
href="https://github.com/rust-lang/log/commit/646e9ab9917fb79e44b6b36b8375106a1a09766c"><code>646e9ab</code></a>
use original Visitor name for VisitValue</li>
<li><a
href="https://github.com/rust-lang/log/commit/cf85c38d3519745d60e7b891c4b2025050a8389f"><code>cf85c38</code></a>
add needed subfeatures to kv_unstable</li>
<li><a
href="https://github.com/rust-lang/log/commit/73e953905b970ef765a86bf6cbd69bc2c5e2bac4"><code>73e9539</code></a>
fix up capturing of :err</li>
<li><a
href="https://github.com/rust-lang/log/commit/31bb4b0ff36e458c6bef304a336b71f6342ddcc7"><code>31bb4b0</code></a>
move error macros together</li>
<li><a
href="https://github.com/rust-lang/log/commit/ad917118a5e781d0dd60b3a75ba519ce9839ba70"><code>ad91711</code></a>
support field shorthand in macros</li>
<li><a
href="https://github.com/rust-lang/log/commit/90a347bd836873264a393a35bfd90fe478fadae2"><code>90a347b</code></a>
restore removed APIs as deprecated</li>
<li>Additional commits viewable in <a
href="https://github.com/rust-lang/log/compare/0.4.20...0.4.21">compare
view</a></li>
</ul>
</details>
<br />

Updates `syn` from 2.0.50 to 2.0.52
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/syn/releases">syn's
releases</a>.</em></p>
<blockquote>
<h2>2.0.52</h2>
<ul>
<li>Add an expression parser that uses match-arm's boundary rules (<a
href="https://redirect.github.com/dtolnay/syn/issues/1593">#1593</a>)</li>
</ul>
<h2>2.0.51</h2>
<ul>
<li>Resolve non_local_definitions warnings in generated code under rustc
1.78-nightly</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/syn/commit/07ede6a6b31adeb3a18899ada1f352f63b3a36b9"><code>07ede6a</code></a>
Release 2.0.52</li>
<li><a
href="https://github.com/dtolnay/syn/commit/acbcfbc8c113fa1603469c9ad329d061ee74662e"><code>acbcfbc</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/syn/issues/1593">#1593</a>
from dtolnay/boundary</li>
<li><a
href="https://github.com/dtolnay/syn/commit/4924a993dce23abe65128ac318dd662d1e2ceef2"><code>4924a99</code></a>
Add an expression parser that uses match-arm's boundary rules</li>
<li><a
href="https://github.com/dtolnay/syn/commit/e06122bf2cfd31bd7f70304694477dd292fe7e1e"><code>e06122b</code></a>
Resolve unnecessary_get_then_check clippy lint</li>
<li><a
href="https://github.com/dtolnay/syn/commit/018fc5a6298491525387910cb359a9ec618abe54"><code>018fc5a</code></a>
Update test suite to nightly-2024-02-27</li>
<li><a
href="https://github.com/dtolnay/syn/commit/5e15a9b412cb1e2df481e3470e1be8defaee4495"><code>5e15a9b</code></a>
Release 2.0.51</li>
<li><a
href="https://github.com/dtolnay/syn/commit/7e0d4e1f43a879078595f0a3876484a1920ab8f8"><code>7e0d4e1</code></a>
Resolve non_local_definitions warning in debug impls</li>
<li><a
href="https://github.com/dtolnay/syn/commit/8667ad97c1d4e75ac1bb323fb5c7849269814145"><code>8667ad9</code></a>
Ignore module_name_repetitions pedantic clippy lint in codegen</li>
<li><a
href="https://github.com/dtolnay/syn/commit/1fc32000e25bf8fda7371071073f91e012ddf808"><code>1fc3200</code></a>
Update test suite to nightly-2024-02-26</li>
<li><a
href="https://github.com/dtolnay/syn/commit/07a2065576b27dcf0c104f56379cc446d2f3824b"><code>07a2065</code></a>
Update test suite to nightly-2024-02-23</li>
<li>See full diff in <a
href="https://github.com/dtolnay/syn/compare/2.0.50...2.0.52">compare
view</a></li>
</ul>
</details>
<br />

Updates `clap` from 4.5.1 to 4.5.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/clap-rs/clap/releases">clap's
releases</a>.</em></p>
<blockquote>
<h2>v4.5.3</h2>
<h2>[4.5.3] - 2024-03-15</h2>
<h3>Internal</h3>
<ul>
<li><em>(derive)</em> Update <code>heck</code></li>
</ul>
<h2>v4.5.2</h2>
<h2>[4.5.2] - 2024-03-06</h2>
<h3>Fixes</h3>
<ul>
<li><em>(macros)</em> Silence a warning</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's
changelog</a>.</em></p>
<blockquote>
<h2>[4.5.3] - 2024-03-15</h2>
<h3>Internal</h3>
<ul>
<li><em>(derive)</em> Update <code>heck</code></li>
</ul>
<h2>[4.5.2] - 2024-03-06</h2>
<h3>Fixes</h3>
<ul>
<li><em>(macros)</em> Silence a warning</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/clap-rs/clap/commit/4e07b438584bb8a19e37599d4c5b11797bec5579"><code>4e07b43</code></a>
chore: Release</li>
<li><a
href="https://github.com/clap-rs/clap/commit/8247c7ddf05d8023729ac180d8e8df260f1da5ff"><code>8247c7d</code></a>
docs: Update changelog</li>
<li><a
href="https://github.com/clap-rs/clap/commit/677c52ce0870115845a4c42e204f6c049b81a1e7"><code>677c52c</code></a>
chore: Update <code>heck</code> requirement (<a
href="https://redirect.github.com/clap-rs/clap/issues/5396">#5396</a>)</li>
<li><a
href="https://github.com/clap-rs/clap/commit/f65d421607ba16c3175ffe76a20820f123b6c4cb"><code>f65d421</code></a>
chore: Release</li>
<li><a
href="https://github.com/clap-rs/clap/commit/886b2729e419114bf42f1a92c66d346c81aa8f33"><code>886b272</code></a>
docs: Update changelog</li>
<li><a
href="https://github.com/clap-rs/clap/commit/3ba429752fdb19b7a1c2e151c41d5141ad5b9295"><code>3ba4297</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5386">#5386</a>
from amaanq/static-var-name</li>
<li><a
href="https://github.com/clap-rs/clap/commit/2aea9504c4894b3bddf9cd4d2d6cba889307c157"><code>2aea950</code></a>
fix: Use SCREAMING_SNAKE_CASE for static variable
<code>authors</code></li>
<li><a
href="https://github.com/clap-rs/clap/commit/690f5557d7f25904c31ec9f2a3c3657cbb68c98e"><code>690f555</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5382">#5382</a>
from clap-rs/renovate/pre-commit-action-3.x</li>
<li><a
href="https://github.com/clap-rs/clap/commit/a2aa644368ec19026b16b870ec32dc57b325ba9b"><code>a2aa644</code></a>
chore(deps): update compatible (dev) (<a
href="https://redirect.github.com/clap-rs/clap/issues/5381">#5381</a>)</li>
<li><a
href="https://github.com/clap-rs/clap/commit/c233de53c0cca4281f444cf16d16d161bc9c3cab"><code>c233de5</code></a>
chore(deps): update pre-commit/action action to v3.0.1</li>
<li>Additional commits viewable in <a
href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.1...v4.5.3">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Totally removes the `experimental` feature. Closes #3648.

---------

Co-authored-by: Bastian Köcher <[email protected]>

Step in https://github.com/paritytech/polkadot-sdk/issues/171

This PR removes `as [disambiguation_path]` syntax from `derive_impl`
usage across the polkadot-sdk as introduced in
https://github.com/paritytech/polkadot-sdk/pull/3505

Currently, the staking logic does not prevent a controller from becoming
a stash of *another* ledger (introduced by [removing this
check](https://github.com/paritytech/polkadot-sdk/pull/1484/files#diff-3aa6ceab5aa4e0ab2ed73a7245e0f5b42e0832d8ca5b1ed85d7b2a52fb196524L850)).
Given that the remaining of the code expects that never happens, bonding
a ledger with a stash that is a controller of another ledger may lead to
data inconsistencies and data losses in bonded ledgers. For more
detailed explanation of this issue:
https://hackmd.io/@gpestana/HJoBm2tqo/%2FTPdi28H7Qc2mNUqLSMn15w

In a nutshell, when fetching a ledger with a given controller, we may be
end up getting the wrong ledger which can lead to unexpected ledger
states.

This PR also ensures that `set_controller` does not lead to data
inconsistencies in the staking ledger and bonded storage in the case
when a controller of a stash is a stash of *another* ledger. and
improves the staking `try-runtime` checks to catch potential issues with
the storage preemptively.

In summary, there are two important cases here:

1. **"Sane" double bonded ledger**

When a controller of a ledger is a stash of *another* ledger. In this
case, we have:

```
> Bonded(stash, controller)
(A, B)  // stash A with controller B
(B, C) // B is also a stash of another ledger
(C, D)

> Ledger(controller)
Ledger(B) = L_a (stash = A)
Ledger(C) = L_b (stash = B)
Ledger(D) = L_c (stash = C)
```

In this case, the ledgers can be mutated and all operations are OK.
However, we should not allow `set_controller` to be called if it means
it results in a "corrupt" double bonded ledger (see below).

3. **"Corrupt" double bonded ledger**

```
> Bonded(stash, controller)
(A, B)  // stash A with controller B
(B, B)
(C, D)
```
In this case, B is a stash and controller AND is corrupted, since B is
responsible for 2 ledgers which is not correct and will lead to
inconsistent states. Thus, in this case, in this PR we are preventing
these ledgers from mutating (i.e. operations like bonding extra etc)
until the ledger is brought back to a consistent state.

--- 

**Changes**:
- Checks if stash is already a controller when calling `Call::bond`
(fixes the regression introduced by [removing this
check](https://github.com/paritytech/polkadot-sdk/pull/1484/files#diff-3aa6ceab5aa4e0ab2ed73a7245e0f5b42e0832d8ca5b1ed85d7b2a52fb196524L850));
- Ensures that all fetching ledgers from storage are done through the
`StakingLedger` API;
- Ensures that -- when fetching a ledger from storage using the
`StakingLedger` API --, a `Error::BadState` is returned if the ledger
bonding is in a bad state. This prevents bad ledgers from mutating (e.g.
`bond_extra`, `set_controller`, etc) its state and avoid further data
inconsistencies.
- Prevents stashes which are controllers or another ledger from calling
`set_controller`, since that may lead to a bad state.
- Adds further try-state runtime checks that check if there are ledgers
in a bad state based on their bonded metadata.

Related to https://github.com/paritytech/polkadot-sdk/issues/3245

---------

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: kianenigma <[email protected]>

Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665)

This PR reverts #2280 which introduced `TransactionExtension` to replace
`SignedExtension`.

As a result of the discussion
[here](https://github.com/paritytech/polkadot-sdk/pull/3623#issuecomment-1986789700),
the changes will be reverted for now with plans to reintroduce the
concept in the future.

---------

Signed-off-by: georgepisaltu <[email protected]>

This PR adds a debug log for displaying all the public addresses that
will later be advertised in the DHT record of the authority. The
Authority DHT record will contain the address ++ `/p2p/peerID` (if not
already present).

This log enables us to check if different nodes will advertise in the
DHT record of the authority the same IP address, however with different
peer IDs.

Signed-off-by: Alexandru Vasile <[email protected]>