Closes: https://github.com/paritytech/polkadot-sdk/issues/7033

# Description

Migrating salary pallet to use umbrella crate. It is a follow-up from
https://github.com/paritytech/polkadot-sdk/pull/7025
Why did I create this new branch? 
I did this, so that the unnecessary cargo fmt changes from the previous
branch are discarded and hence opened this new PR.



## Review Notes

This PR migrates pallet-salary to use the umbrella crate.

Added change: Explanation requested for why `TestExternalities` was
replaced by `TestState` as testing_prelude already includes it
`pub use sp_io::TestExternalities as TestState;`


I have also modified the defensive! macro to be compatible with umbrella
crate as it was being used in the salary pallet

Will fix:
https://github.com/paritytech/polkadot-sdk/issues/6574
https://github.com/paritytech/polkadot-sdk/issues/6644
https://github.com/paritytech/polkadot-sdk/issues/6062

---------

Co-authored-by: Javier Viola <javier@parity.io>

# Description

Migrate pallet-node-authorization to use umbrella crate. Part of #6504 

## Review Notes

* This PR migrates pallet-node-authorization to use the umbrella crate.
* Some imports like below have not been added to any prelude as they
have very limited usage across the various pallets.
```rust
use sp_core::OpaquePeerId as PeerId;
```
* Added a commonly used runtime trait for testing in the
`testing_prelude` in `substrate/frame/src/lib.rs`:
```rust
pub use sp_runtime::traits::BadOrigin;
```
* `weights.rs` uses the `weights_prelude` like:
```rust
use frame::weights_prelude::*;
```
* `tests.rs` and `mock.rs` use the `testing_prelude`:
```rust
use frame::testing_prelude::*;
```
* `lib.rs` uses the main `prelude` like:
```rust
use frame::prelude::*;
```
* For testing: Checked that local build works and tests run
successfully.

# Description

This PR removes usage of deprecated `sp-std` from Substrate. (following
PR of #5010)

## Integration

This PR doesn't remove re-exported `sp_std` from any crates yet, so
downstream projects using re-exported `sp_std` will not be affected.

## Review Notes

The existing code using `sp-std` is refactored to use `alloc` and `core`
directly. The key-value maps are instantiated from a vector of tuples
directly instead of using `sp_std::map!` macro.

`sp_std::Writer` is a helper type to use `Vec<u8>` with
`core::fmt::Write` trait. This PR copied it into `sp-runtime`, because
all crates using `sp_std::Writer` (including `sp-runtime` itself,
`frame-support`, etc.) depend on `sp-runtime`.

If this PR is merged, I would write following PRs to remove remaining
usage of `sp-std` from `bridges` and `cumulus`.

---------

Co-authored-by: command-bot <>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>

Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight (#6140)

(rebasing of https://github.com/paritytech/polkadot-sdk/pull/5234)

## Issues:

* Transaction extensions have weights and refund weight. So the
reclaiming of unused weight must happen last in the transaction
extension pipeline. Currently it is inside `CheckWeight`.
* cumulus storage weight reclaim transaction extension misses the proof
size of logic happening prior to itself.

## Done:

* a new storage `ExtrinsicWeightReclaimed` in frame-system. Any logic
which attempts to do some reclaim must use this storage to avoid double
reclaim.
* a new function `reclaim_weight` in frame-system pallet: info and post
info in arguments, read the already reclaimed weight, calculate the new
unused weight from info and post info. do the more accurate reclaim if
higher.
* `CheckWeight` is unchanged and still reclaim the weight in post
dispatch
* `ReclaimWeight` is a new transaction extension in frame system. For
s...

# Description

Migrate pallet-mixnet to use umbrella crate whilst adding a few types
and traits in the frame prelude that are used by other pallets as well.

## Review Notes

* This PR migrates `pallet-mixnet` to use the umbrella crate. 
* Note that some imports like `use
sp_application_crypto::RuntimeAppPublic;` and imports from
`sp_mixnet::types::` have not been migrated to the umbrella crate as
they are not used in any / many other places and are relevant only to
the `pallet-mixnet`.
* Transaction related helpers to submit transactions from `frame-system`
have been added to the main `prelude` as they have usage across various
pallets.
```Rust
	pub use frame_system::offchain::*;
```
* Exporting `arithmetic` module in the main `prelude` since this is used
a lot throughout various pallets.
* Nightly formatting has been applied using `cargo fmt`
* Benchmarking dependencies have been removed from`palet-mixnet` as
there is no benchmarking.rs present for `pallet-mixnet`. For the same
reason, `"pallet-mixnet?/runtime-benchmarks"` has been removed from
`umbrella/Cargo.toml`.

---------

Co-authored-by: Dónal Murray <donalm@seadanda.dev>

Part of https://github.com/paritytech/polkadot-sdk/issues/6504

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>

- Fixed failing docs.rs build for `pallet-revive-uapi` by fixing a wring
attribute in the manifest (we were using `default-target` instead of
`targets`)
- Removed the macros defining host functions because the cfg attributes
introduced in #6866 won't work on them
- Added an docs.rs specific attribute so that the `unstable-hostfn`
feature tag will show up on the functions that are guarded behind it.

---------

Co-authored-by: command-bot <>

Update to the latest polkavm version, containing a linker fix I need for
revive.

---------

Signed-off-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
Signed-off-by: xermicus <cyrill@parity.io>
Co-authored-by: command-bot <>

Adds a new zombienet-sdk test which verifies that elastic scaling works
correctly both with the MVP and the new RFC 103 implementation which
sends the core selector as a UMP signal.

Also enables the V2 receipts node feature for testnet genesis config.

Part of https://github.com/paritytech/polkadot-sdk/issues/5049

---------

Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com>

#6450 introduced metadata checks. Supported are metadata v14 and higher.

However, of course old chain-specs have a genesis code blob that might
be on older version. This needs to be tolerated. We should just skip the
checks in that case.

Fixes #6921

---------

Co-authored-by: command-bot <>

Removing apparently unused dependencies from `pallet_revive` and related
crates.

---------

Co-authored-by: command-bot <>

# Description

Upgrade `nix` and `reqwest` to reduce outdated dependencies and speed up
compilation.

# Description

Redundant dep that made its way in #6450 . 

. It can be
brought up when using `cargo udeps`. Added a github action that runs
`cargo udeps` on the repo too.

## Integration

N/A

## Review Notes

N/A

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

Fixes https://github.com/paritytech/polkadot-sdk/issues/5996

https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0
https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md

## Integration

Nothing special is needed, just note that `yamux_window_size` is no
longer applicable to libp2p (litep2p seems to still have it though).

## Review Notes

There are a few simplifications and improvements done in libp2p 0.53
regarding swarm interface, I'll list a few key/applicable here.

https://github.com/libp2p/rust-libp2p/pull/4788 removed
`write_length_prefixed` function, so I inlined its code instead.

https://github.com/libp2p/rust-libp2p/pull/4120 introduced new
`libp2p::SwarmBuilder` instead of now deprecated
`libp2p::swarm::SwarmBuilder`, the transition is straightforward and
quite ergonomic (can be seen in tests).

https://github.com/libp2p/rust-libp2p/pull/4581 is the most annoying
change I have seen that basically makes many enums `#[non_exhaustive]`.
I mapped some, but those that couldn't be mapped I dealt with by
printing log messages once they are hit (the best solution I could come
up with, at least with stable Rust).

https://github.com/libp2p/rust-libp2p/issues/4306 makes connection close
as soon as there are no handler using it, so I had to replace
`KeepAlive::Until` with an explicit future that flips internal boolean
after timeout, achieving the old behavior, though it should ideally be
removed completely at some point.

`yamux_window_size` is no longer used by libp2p thanks to
https://github.com/libp2p/rust-libp2p/pull/4970 and generally Yamux
should have a higher performance now.

I have resolved and cleaned up all deprecations related to libp2p except
`BandwidthSinks`. Libp2p deprecated it (though it is still present in
0.54.1, which is why I didn't handle it just yet). Ideally Substrate
would finally [switch to the official Prometheus
client](https://github.com/paritytech/substrate/issues/12699), in which
case we'd get metrics for free. Otherwise a bit of code will need to be
copy-pasted to maintain current behavior with `BandwidthSinks` gone,
which I left a TODO about.

The biggest change in 0.54.0 is
https://github.com/libp2p/rust-libp2p/pull/4568 that changed transport
APIs and enabled unconditional potential port reuse, which can lead to
very confusing errors if running two Substrate nodes on the same machine
without changing listening port explicitly.

Overall nothing scary here, but testing is always appreciated.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---

Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

Bump `polkavm` to 0.18.0, and update `sc-polkavm-executor` to be
compatible with the API changes. In addition, bump also `polkavm-derive`
and `polkavm-linker` in order to make sure that the all parts of the
Polkadot SDK use the exact same ABI for `.polkavm` binaries.

Purely relying on RV32E/RV64E ABI is not possible, as PolkaVM uses a
RISCV-V alike ISA, which is derived from RV32E/RV64E but it is still its
own microarchitecture, i.e. not fully binary compatible.

---------

Signed-off-by: Jarkko Sakkinen <jarkko@parity.io>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

0.1.2 was yanked as it was breaking semver.

---------

Co-authored-by: command-bot <>

The `SlotBasedBlockImport` job is to collect the storage proofs of all
blocks getting imported. These storage proofs alongside the block are
being forwarded to the collation task. Right now they are just being
thrown away. More logic will follow later. Basically this will be
required to include multiple blocks into one `PoV` which will then be
done by the collation task.

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: command-bot <>

## [0.8.4] - 2024-12-12

This release aims to make the MDNS component more robust by fixing a bug
that caused the MDNS service to fail to register opened substreams.
Additionally, the release includes several improvements to the
`identify` protocol, replacing `FuturesUnordered` with `FuturesStream`
for better performance.

### Fixed

- mdns/fix: Failed to register opened substream
([#301](https://github.com/paritytech/litep2p/pull/301))

### Changed

- identify: Replace FuturesUnordered with FuturesStream
([#302](https://github.com/paritytech/litep2p/pull/302))
- chore: Update hickory-resolver to version 0.24.2
([#304](https://github.com/paritytech/litep2p/pull/304))
- ci: Ensure cargo-machete is working with rust version from CI
([#303](https://github.com/paritytech/litep2p/pull/303))


cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

# Description

Get runtime's metadata, parse it and verify pallets list for a pallet
named `ParachainSystem` (for now), and block number to be the same for
both node and runtime. Ideally we'll add other pallets checks too, at
least a small set of pallets we think right away as mandatory for
parachain compatibility.
Closes: #5565 

## Integration

Runtime devs must be made aware that to be fully compatible with Omni
Node, certain naming conventions should be respected when defining
pallets (e.g we verify parachain-system pallet existence by searching
for a pallet with `name` `ParachainSystem` in runtime's metadata). Not
finding such a pallet will not influence the functionality yet, but by
doing these checks we could provide useful feedback for runtimes that
are clearly not implementing what's required for full parachain
compatibility with Omni Node.

## Review Notes

- [x] parachain system check
- [x] check frame_system's metadata to ensure the block number in there
is the same as the one in the node side
- [x] add tests for the previous checking logic
- [x] update omni node polkadot-sdk docs to make these conventions
visible.
- [ ] add more pallets checks?

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

I am planning to use `alloy_core` to implement precompile support in
`pallet_revive`. I noticed that it is already used by snowbridge. In
order to unify the dependencies I did the following:

1. Switch to the `alloy-core` umbrella crate so that we have less
individual dependencies to update.
2. Bump the latest version and fixup the resulting compile errors.

# Description

**Understood assignment:**
Initial assignment description is in #6194.
In order to Simplify the display of commands and ensure they are tested
for chain spec builder's `polkadot-sdk` reference docs, find every
occurrence of `#[docify::export]` where `process:Command` is used, and
replace the use of `process:Command` by `run_cmd!` from the `cmd_lib
crate`.

---------

Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

Close: #5858

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

Closes: https://github.com/paritytech/polkadot-sdk/issues/5551

## Description

With [permissionless lanes
PR#4949](https://github.com/paritytech/polkadot-sdk/pull/4949), the
congestion mechanism based on sending
`Transact(report_bridge_status(is_congested))` from
`pallet-xcm-bridge-hub` to `pallet-xcm-bridge-hub-router` was replaced
with a congestion mechanism that relied on monitoring XCMP queues.
However, this approach could cause issues, such as suspending the entire
XCMP queue instead of isolating the affected bridge. This PR reverts
back to using `report_bridge_status` as before.

## TODO
- [x] benchmarks
- [x] prdoc

## Follow-up

https://github.com/paritytech/polkadot-sdk/pull/6231

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

The dependency on `pallet_balances` doesn't seem to be necessary. At
least everything compiles for me without it. Removed this dependency and
a few others that seem to be left overs.

---------

Co-authored-by: GitHub Action <action@github.com>

Last feature we wanted for V5, changing `SetAssetClaimer` to be just one
of many possible "hints" that you can specify at the beginning of your
program to change its behaviour.

This makes it easier to add new hints in the future and have barriers
accept them.

---------

Co-authored-by: GitHub Action <action@github.com>

# Description

Closes #6335.

## Integration

N/A

## Review Notes

`RuntimeTarget` is converted to return path to the custom target JSON
file

---------

Signed-off-by: Jarkko Sakkinen <jarkko@parity.io>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Koute <koute@users.noreply.github.com>

## [0.8.3] - 2024-12-03

This release includes two fixes for small memory leaks on edge-cases in
the notification and request-response protocols.

### Fixed

- req-resp: Fix memory leak of pending substreams
([#297](https://github.com/paritytech/litep2p/pull/297))
- notification: Fix memory leak of pending substreams
([#296](https://github.com/paritytech/litep2p/pull/296))

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

No need to have them in the umbrella crate also by having them in the
umbrella crate they are bleeding into the normal build.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

* Add a bunch of differential tests to ensure that responses from
eth-rpc matches the one from `geth`
- These
[tests](https://github.com/paritytech/polkadot-sdk/blob/pg/fix-geth-diff/substrate/frame/revive/rpc/examples/js/src/geth-diff.test.ts)
are not run in CI for now but can be run locally with
```bash
cd revive/rpc/examples/js
bun test
```

* EVM RPC server will not fail gas_estimation if no gas is specified, I
updated pallet-revive to add an extra `skip_transfer` boolean check to
replicate this behavior in our pallet

* `eth_transact` and `bare_eth_transact` api have been updated to use
`GenericTransaction` directly as this is what is used by
`eth_estimateGas` and `eth_call`

## TODO

- [ ]  Add tests the new `skip_transfer` flag

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

# Description

Made as a follow-up of
https://github.com/polkadot-fellows/runtimes/pull/499

## Integration

N/A

## Review Notes

N/A

---------

Co-authored-by: Dónal Murray <donal.murray@parity.io>

# Description

Updates scale-info to from 2.11.5 2.11.6, so that generated code is
annotated with `allow(deprecated)`
Pre-requisite for https://github.com/paritytech/polkadot-sdk/pull/6312

Construct runtime uses autoref-based specialization to fetch the
metadata about the implemented runtime apis. This is done to not fail to
compile when there are no runtime apis implemented. However, there was
an issue with detecting runtime apis when they were implemented in a
different file. The problem is solved by moving the trait implemented by
`impl_runtime_apis!` to the metadata ir crate.


Closes: https://github.com/paritytech/polkadot-sdk/issues/6659

---------

Co-authored-by: GitHub Action <action@github.com>

This PR updates pallet_revive to the newest PolkaVM version and adapts
the test fixtures and syscall interface to work under 64bit.

Please note that after this PR no 32bit contracts can be deployed (they
will be rejected at deploy time). Pre-deployed 32bit contracts are now
considered defunct since we changes how parameters are passed for
functions with more than 6 arguments.

## Fixtures

The fixtures are now built for the 64bit target. I also removed the
temporary directory mechanism that triggered a full rebuild every time.
It also makes it easier to find the compiled fixtures since they are now
always in `target/pallet-revive-fixtures`.

## Syscall interface

### Passing pointer

Registers and pointers are now 64bit wide. This allows us to pass u64
arguments in a single register. Before we needed two registers to pass
them. This means that just as before we need one register per pointer we
pass. We keep pointers as `u32` argument by truncating the register.
This is done since the memory space of PolkaVM is 32bit.

### Functions with more than 6 arguments

We only have 6 registers to pass arguments. This is why we pass a
pointer to a struct when we need more than 6. Before this PR we expected
a packed struct and interpreted it as SCALE encoded tuple. However, this
was buggy because the `MaxEncodedLen` returned something that was larger
than the packed size of the structure. This wasn't a problem before. But
now the memory space changed in a way that things were placed at the
edges of the memory space and those extra bytes lead to an out of bound
access.

This is why this PR drops SCALE and expects the arguments to be passed
as a pointer to a `C` aligned struct. This avoids unaligned accesses.
However, revive needs to adapt its codegen to properly align the
structure fields.

## TODO
- [ ] Add multi block migration that wipes all existing contracts as we
made breaking changes to the syscall interface

---------

Co-authored-by: GitHub Action <action@github.com>

This includes a critical fix for debug release versions of litep2p
(which are running in Kusama as validators).

While at it, have stopped the oncall pain of alerts around
`incoming_connections_total`. We can rethink the metric expose of
litep2p in Q1.





## [0.8.2] - 2024-11-27

This release ensures that the provided peer identity is verified at the
crypto/noise protocol level, enhancing security and preventing potential
misuses.
The release also includes a fix that caused `TransportService` component
to panic on debug builds.

### Fixed

- req-resp: Fix panic on connection closed for substream open failure
([#291](https://github.com/paritytech/litep2p/pull/291))
- crypto/noise: Verify crypto/noise signature payload
([#278](https://github.com/paritytech/litep2p/pull/278))

### Changed

- transport_service/logs: Provide less details for trace logs
([#292](https://github.com/paritytech/litep2p/pull/292))


## Testing Done

This has been extensively tested in Kusama on all validators, that are
now running litep2p.

Deployed PR: https://github.com/paritytech/polkadot-sdk/pull/6638

### Litep2p Dashboards
![Screenshot 2024-11-26 at 19 19
41](https://github.com/user-attachments/assets/e00b2b2b-7e64-4d96-ab26-165e2b8d0dc9)


### Libp2p vs Litep2p CPU usage

After deploying litep2p we have reduced CPU usage from around 300-400%
to 200%, this is a significant boost in performance, freeing resources
for other subcomponents to function more optimally.


![image(1)](https://github.com/user-attachments/assets/fa793df5-4d58-4601-963d-246e56dd2a26)



cc @paritytech/sdk-node

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: GitHub Action <action@github.com>

This PR implements the `archive_unstable_storageDiff`.

The implementation follows the rpc-v2 spec from:
-  https://github.com/paritytech/json-rpc-interface-spec/pull/159.
- builds on top of
https://github.com/paritytech/json-rpc-interface-spec/pull/161

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: James Wilson <james@jsdw.me>

This PR exposes the *unstable* metadata V16. The metadata is exposed
under the unstable u32::MAX number.

Developers can start experimenting with the new features of the metadata
v16. *Please note that this metadata is under development and expect
breaking changes until stabilization.*

The `ExtrinsicMetadata` trait receives a breaking change. Its associated
type `VERSION` is rename to `VERSIONS` and now supports a constant
static list of metadata versions.
The versions implemented for `UncheckedExtrinsic` are v4 (legacy
version) and v5 (new version).

For metadata collection, it is assumed that all `TransactionExtensions`
are under version 0.

Builds on top of: https://github.com/paritytech/polkadot-sdk/pull/5274

Closes: https://github.com/paritytech/polkadot-sdk/issues/5980
Closes: https://github.com/paritytech/polkadot-sdk/issues/5347
Closes: https://github.com/paritytech/polkadot-sdk/issues/5285

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: GitHub Action <action@github.com>

The `query_weight_to_asset_fee` function was trying to convert versions
by using `try_as`, this function [doesn't convert from a versioned to a
concrete
type](https://github.com/paritytech/polkadot-sdk/blob/0156ca8f/polkadot/xcm/src/lib.rs#L131).
This would cause all calls with a lower version to fail.

The correct function to use is the good old
[try_into](https://github.com/paritytech/polkadot-sdk/blob/0156ca8f

/polkadot/xcm/src/lib.rs#L184).
Now those calls work :)

---------

Co-authored-by: command-bot <>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: GitHub Action <action@github.com>