Part of: https://github.com/paritytech/polkadot-sdk/issues/6906

- Add option to specify database_url using DATABASE_URL environment
variable
- Add a eth-rpc-tester rust bin that can be used to test deployment
before releasing eth-rpc
- make evm_block non fallible so that it can return an Ok response for
older blocks when the runtime API is not available
- update cargo.lock to integrate changes from
https://github.com/paritytech/subxt/pull/1904

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Closes: https://github.com/paritytech/polkadot-sdk/issues/2904

---------

Co-authored-by: command-bot <>

This PR contains small fixes identified during work on the larger PR:
https://github.com/paritytech/polkadot-sdk/issues/6906.

---------

Co-authored-by: command-bot <>

Part of https://github.com/paritytech/polkadot-sdk/issues/6504

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>

Related to https://github.com/paritytech/polkadot-sdk/issues/4523

Follow-up for: https://github.com/paritytech/polkadot-sdk/pull/5188

Reopening https://github.com/paritytech/polkadot-sdk/pull/6732 as a new
PR

---------

Co-authored-by: command-bot <>

Part of  #6504

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>

Part of https://github.com/paritytech/polkadot-sdk/issues/6504

### Added modules

- `utility`: Traits not tied to any direct operation in the runtime.

polkadot address: 14SRqZTC1d8rfxL8W1tBTnfUBPU23ACFVPzp61FyGf4ftUFg

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>

Part of https://github.com/paritytech/polkadot-sdk/issues/6504

Use docify export for parachain template hardcoded configuration and embed it in its README #6333 (#7093)

Use docify export for parachain template hardcoded configuration and
embed it in its README #6333

Docify currently has a limitation of not being able to embed a
variable/const in its code, without embedding it's definition, even if
do something in a string like

"this is a sample string ${sample_variable}"

It will embed the entire string 
"this is a sample string ${sample_variable}"
without replacing the value of sample_variable from the code

Hence, the goal was just to make it obvious in the README where the
PARACHAIN_ID value is coming from, so a note has been added at the start
for the same, so whenever somebody is running these commands, they will
be aware about the value and replace accordingly.

To make it simpler, we added a 
rust ignore block so the user can just look it up in the readme itself
and does not have to scan through the runtime directory for the value.

---------

Co-authored-by...

Follow-up to #6825, which introduced this bug.

We use the `can_build_upon` method to ask the runtime if it is fine to
build another block. The runtime checks this based on the
[`ConsensusHook`](https://github.com/paritytech/polkadot-sdk/blob/c1b7c302/cumulus/pallets/aura-ext/src/consensus_hook.rs#L110-L110)
implementation, the most popular one being the `FixedConsensusHook`.

In #6825 I removed a check that would always allow us to build when we
are building on an included block. Turns out this check is still
required when:
1. The [`UnincludedSegment`
](https://github.com/paritytech/polkadot-sdk/blob/c1b7c302

/cumulus/pallets/parachain-system/src/lib.rs#L758-L758)
storage item in pallet-parachain-system is equal or larger than the
unincluded segment.
2. We are calling the `can_build_upon` runtime API where the included
block has progressed offchain to the current parent block (so last entry
in the `UnincludedSegment` storage item).

In this scenario the last entry in `UnincludedSegment` does not have a
hash assigned yet (because it was not available in `on_finalize` of the
previous block). So the unincluded segment will be reported at its
maximum length which will forbid building another block.

Ideally we would have a more elegant solution than to rely on the
node-side here. But for now the check is reintroduced and a test is
added to not break it again by accident.

---------

Co-authored-by: command-bot <>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Add foundation for supporting call traces in pallet_revive

Follow up:
- PR #7167 Add changes to eth-rpc to introduce debug endpoint that will
use pallet-revive tracing features
- PR #6727 Add new RPC to the client and implement tracing runtime API
that can capture traces on previous blocks

---------

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

Update to PolkaVM `0.19`. This version renumbers the opcodes in order to
be in-line with the grey paper. Hopefully, for the last time. This means
that it breaks existing contracts.

---------

Signed-off-by: xermicus <cyrill@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: xermicus <cyrill@parity.io>

Migrate staking currency from `traits::LockableCurrency` to
`traits::fungible::holds`.

Resolves part of https://github.com/paritytech/polkadot-sdk/issues/226.

## Changes
### Nomination Pool
TransferStake is now incompatible with fungible migration as old pools
were not meant to have additional ED. Since they are anyways deprecated,
removed its usage from all test runtimes.

### Staking
- Config: `Currency` becomes of type `Fungible` while `OldCurrency` is
the `LockableCurrency` used before.
- Lazy migration of accounts. Any ledger update will create a new hold
with no extra reads/writes. A permissionless extrinsic
`migrate_currency()` releases the old `lock` along with some
housekeeping.
- Staking now requires ED to be left free. It also adds no consumer to
staking accounts.
- If hold cannot be applied to all stake, the un-holdable part is force
withdrawn from the ledger.

### Delegated Staking
The pallet does not add provider for agents anymore.

## Migr...

Closes #3149 

## Description

This PR introduces `pallet-asset-rewards`, which allows accounts to be
rewarded for freezing `fungible` tokens. The motivation for creating
this pallet is to allow incentivising LPs.

See the pallet docs for more info about the pallet.

## Runtime changes

The pallet has been added to
- `asset-hub-rococo`
- `asset-hub-westend`

The `NativeAndAssets` `fungibles` Union did not contain `PoolAssets`, so
it has been renamed `NativeAndNonPoolAssets`

A new `fungibles` Union `NativeAndAllAssets` was created to encompass
all assets and the native token.

## TODO
- [x] Emulation tests
- [x] Fill in Freeze logic (blocked
https://github.com/paritytech/polkadot-sdk/issues/3342) and re-run
benchmarks

---------

Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: muharem <ismailov.m.h@gmail.com>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

This PR provides the partial results of the `GetRecord` kademlia query.

This significantly improves the authority discovery records, from ~37
minutes to ~2/3 minutes.
In contrast, libp2p discovers authority records in around ~10 minutes. 

The authority discovery was slow because litep2p provided the records
only after the Kademlia query was completed. A normal Kademlia query
completes in around 40 seconds to a few minutes.
In this PR, partial records are provided as soon as they are discovered
from the network.

### Testing Done

Started a node in Kusama with `--validator` and litep2p backend.
The node discovered 996/1000 authority records in ~ 1 minute 45 seconds.

![Screenshot 2025-01-09 at 12 26
08](https://github.com/user-attachments/assets/b618bf7c-2bba-43a0-a021-4047e854c075)


### Before & After

In this image, on the left side is libp2p, in the middle litep2p without
this PR, on the right litep2p with this PR

![Screenshot 2025-01-07 at 17 57
56](https://github.com/user-attachments/assets/a8d467f7-8dc7-461c-bcff-163b94d01ae8)



Closes: https://github.com/paritytech/polkadot-sdk/issues/7077

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

closes #3967 

## Changes
We now use relay chain slots to measure velocity on chain. Previously we
were storing the current parachain slot. Then in `on_state_proof` of the
`ConsensusHook` we were checking how many blocks were athored in the
current parachain slot. This works well when the parachain slot time and
relay chain slot time is the same. With elastic scaling, we can have
parachain slot times lower than that of the relay chain. In these cases
we want to measure velocity in relation to the relay chain. This PR
adjusts that.


##  Migration
This PR includes a migration. Storage item `SlotInfo` of pallet
`aura-ext` is renamed to `RelaySlotInfo` to better reflect its new
content. A migration has been added that just kills the old storage
item. `RelaySlotInfo` will be `None` initially but its value will be
adjusted after one new relay chain slot arrives.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>

Add an option to persist EVM transaction hash to a SQL db.
This should make it possible to run a full archive ETH RPC node
(assuming the substrate node is also a full archive node)

Some queries such as eth_getTransactionByHash,
eth_getBlockTransactionCountByHash, and other need to work with a
transaction hash indexes, which are not stored in Substrate and need to
be stored by the eth-rpc proxy.

The refactoring break down the Client into a `BlockInfoProvider` and
`ReceiptProvider`
- BlockInfoProvider does not need any persistence data, as we can fetch
all block info from the source substrate chain
- ReceiptProvider comes in two flavor, 
  - An in memory cache implementation - This is the one we had so far.
- A DB implementation - This one persist rows with the block_hash, the
transaction_index and the transaction_hash, so that we can later fetch
the block and extrinsic for that receipt and reconstruct the ReceiptInfo
object.

This PR also adds ...

Closes: https://github.com/paritytech/polkadot-sdk/issues/7033

# Description

Migrating salary pallet to use umbrella crate. It is a follow-up from
https://github.com/paritytech/polkadot-sdk/pull/7025
Why did I create this new branch? 
I did this, so that the unnecessary cargo fmt changes from the previous
branch are discarded and hence opened this new PR.



## Review Notes

This PR migrates pallet-salary to use the umbrella crate.

Added change: Explanation requested for why `TestExternalities` was
replaced by `TestState` as testing_prelude already includes it
`pub use sp_io::TestExternalities as TestState;`


I have also modified the defensive! macro to be compatible with umbrella
crate as it was being used in the salary pallet

Will fix:
https://github.com/paritytech/polkadot-sdk/issues/6574
https://github.com/paritytech/polkadot-sdk/issues/6644
https://github.com/paritytech/polkadot-sdk/issues/6062

---------

Co-authored-by: Javier Viola <javier@parity.io>

# Description

Migrate pallet-node-authorization to use umbrella crate. Part of #6504 

## Review Notes

* This PR migrates pallet-node-authorization to use the umbrella crate.
* Some imports like below have not been added to any prelude as they
have very limited usage across the various pallets.
```rust
use sp_core::OpaquePeerId as PeerId;
```
* Added a commonly used runtime trait for testing in the
`testing_prelude` in `substrate/frame/src/lib.rs`:
```rust
pub use sp_runtime::traits::BadOrigin;
```
* `weights.rs` uses the `weights_prelude` like:
```rust
use frame::weights_prelude::*;
```
* `tests.rs` and `mock.rs` use the `testing_prelude`:
```rust
use frame::testing_prelude::*;
```
* `lib.rs` uses the main `prelude` like:
```rust
use frame::prelude::*;
```
* For testing: Checked that local build works and tests run
successfully.

# Description

This PR removes usage of deprecated `sp-std` from Substrate. (following
PR of #5010)

## Integration

This PR doesn't remove re-exported `sp_std` from any crates yet, so
downstream projects using re-exported `sp_std` will not be affected.

## Review Notes

The existing code using `sp-std` is refactored to use `alloc` and `core`
directly. The key-value maps are instantiated from a vector of tuples
directly instead of using `sp_std::map!` macro.

`sp_std::Writer` is a helper type to use `Vec<u8>` with
`core::fmt::Write` trait. This PR copied it into `sp-runtime`, because
all crates using `sp_std::Writer` (including `sp-runtime` itself,
`frame-support`, etc.) depend on `sp-runtime`.

If this PR is merged, I would write following PRs to remove remaining
usage of `sp-std` from `bridges` and `cumulus`.

---------

Co-authored-by: command-bot <>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>

Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight (#6140)

(rebasing of https://github.com/paritytech/polkadot-sdk/pull/5234)

## Issues:

* Transaction extensions have weights and refund weight. So the
reclaiming of unused weight must happen last in the transaction
extension pipeline. Currently it is inside `CheckWeight`.
* cumulus storage weight reclaim transaction extension misses the proof
size of logic happening prior to itself.

## Done:

* a new storage `ExtrinsicWeightReclaimed` in frame-system. Any logic
which attempts to do some reclaim must use this storage to avoid double
reclaim.
* a new function `reclaim_weight` in frame-system pallet: info and post
info in arguments, read the already reclaimed weight, calculate the new
unused weight from info and post info. do the more accurate reclaim if
higher.
* `CheckWeight` is unchanged and still reclaim the weight in post
dispatch
* `ReclaimWeight` is a new transaction extension in frame system. For
s...

# Description

Migrate pallet-mixnet to use umbrella crate whilst adding a few types
and traits in the frame prelude that are used by other pallets as well.

## Review Notes

* This PR migrates `pallet-mixnet` to use the umbrella crate. 
* Note that some imports like `use
sp_application_crypto::RuntimeAppPublic;` and imports from
`sp_mixnet::types::` have not been migrated to the umbrella crate as
they are not used in any / many other places and are relevant only to
the `pallet-mixnet`.
* Transaction related helpers to submit transactions from `frame-system`
have been added to the main `prelude` as they have usage across various
pallets.
```Rust
	pub use frame_system::offchain::*;
```
* Exporting `arithmetic` module in the main `prelude` since this is used
a lot throughout various pallets.
* Nightly formatting has been applied using `cargo fmt`
* Benchmarking dependencies have been removed from`palet-mixnet` as
there is no benchmarking.rs present for `pallet-mixnet`. For the same
reason, `"pallet-mixnet?/runtime-benchmarks"` has been removed from
`umbrella/Cargo.toml`.

---------

Co-authored-by: Dónal Murray <donalm@seadanda.dev>

Part of https://github.com/paritytech/polkadot-sdk/issues/6504

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>

- Fixed failing docs.rs build for `pallet-revive-uapi` by fixing a wring
attribute in the manifest (we were using `default-target` instead of
`targets`)
- Removed the macros defining host functions because the cfg attributes
introduced in #6866 won't work on them
- Added an docs.rs specific attribute so that the `unstable-hostfn`
feature tag will show up on the functions that are guarded behind it.

---------

Co-authored-by: command-bot <>

Update to the latest polkavm version, containing a linker fix I need for
revive.

---------

Signed-off-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
Signed-off-by: xermicus <cyrill@parity.io>
Co-authored-by: command-bot <>

Adds a new zombienet-sdk test which verifies that elastic scaling works
correctly both with the MVP and the new RFC 103 implementation which
sends the core selector as a UMP signal.

Also enables the V2 receipts node feature for testnet genesis config.

Part of https://github.com/paritytech/polkadot-sdk/issues/5049

---------

Co-authored-by: Javier Viola <javier@parity.io>
Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com>

#6450 introduced metadata checks. Supported are metadata v14 and higher.

However, of course old chain-specs have a genesis code blob that might
be on older version. This needs to be tolerated. We should just skip the
checks in that case.

Fixes #6921

---------

Co-authored-by: command-bot <>

Removing apparently unused dependencies from `pallet_revive` and related
crates.

---------

Co-authored-by: command-bot <>

# Description

Upgrade `nix` and `reqwest` to reduce outdated dependencies and speed up
compilation.

# Description

Redundant dep that made its way in #6450 . 

. It can be
brought up when using `cargo udeps`. Added a github action that runs
`cargo udeps` on the repo too.

## Integration

N/A

## Review Notes

N/A

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

Fixes https://github.com/paritytech/polkadot-sdk/issues/5996

https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0
https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md

## Integration

Nothing special is needed, just note that `yamux_window_size` is no
longer applicable to libp2p (litep2p seems to still have it though).

## Review Notes

There are a few simplifications and improvements done in libp2p 0.53
regarding swarm interface, I'll list a few key/applicable here.

https://github.com/libp2p/rust-libp2p/pull/4788 removed
`write_length_prefixed` function, so I inlined its code instead.

https://github.com/libp2p/rust-libp2p/pull/4120 introduced new
`libp2p::SwarmBuilder` instead of now deprecated
`libp2p::swarm::SwarmBuilder`, the transition is straightforward and
quite ergonomic (can be seen in tests).

https://github.com/libp2p/rust-libp2p/pull/4581 is the most annoying
change I have seen that basically makes many enums `#[non_exhaustive]`.
I mapped some, but those that couldn't be mapped I dealt with by
printing log messages once they are hit (the best solution I could come
up with, at least with stable Rust).

https://github.com/libp2p/rust-libp2p/issues/4306 makes connection close
as soon as there are no handler using it, so I had to replace
`KeepAlive::Until` with an explicit future that flips internal boolean
after timeout, achieving the old behavior, though it should ideally be
removed completely at some point.

`yamux_window_size` is no longer used by libp2p thanks to
https://github.com/libp2p/rust-libp2p/pull/4970 and generally Yamux
should have a higher performance now.

I have resolved and cleaned up all deprecations related to libp2p except
`BandwidthSinks`. Libp2p deprecated it (though it is still present in
0.54.1, which is why I didn't handle it just yet). Ideally Substrate
would finally [switch to the official Prometheus
client](https://github.com/paritytech/substrate/issues/12699), in which
case we'd get metrics for free. Otherwise a bit of code will need to be
copy-pasted to maintain current behavior with `BandwidthSinks` gone,
which I left a TODO about.

The biggest change in 0.54.0 is
https://github.com/libp2p/rust-libp2p/pull/4568 that changed transport
APIs and enabled unconditional potential port reuse, which can lead to
very confusing errors if running two Substrate nodes on the same machine
without changing listening port explicitly.

Overall nothing scary here, but testing is always appreciated.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---

Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

Bump `polkavm` to 0.18.0, and update `sc-polkavm-executor` to be
compatible with the API changes. In addition, bump also `polkavm-derive`
and `polkavm-linker` in order to make sure that the all parts of the
Polkadot SDK use the exact same ABI for `.polkavm` binaries.

Purely relying on RV32E/RV64E ABI is not possible, as PolkaVM uses a
RISCV-V alike ISA, which is derived from RV32E/RV64E but it is still its
own microarchitecture, i.e. not fully binary compatible.

---------

Signed-off-by: Jarkko Sakkinen <jarkko@parity.io>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

0.1.2 was yanked as it was breaking semver.

---------

Co-authored-by: command-bot <>

The `SlotBasedBlockImport` job is to collect the storage proofs of all
blocks getting imported. These storage proofs alongside the block are
being forwarded to the collation task. Right now they are just being
thrown away. More logic will follow later. Basically this will be
required to include multiple blocks into one `PoV` which will then be
done by the collation task.

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: command-bot <>

## [0.8.4] - 2024-12-12

This release aims to make the MDNS component more robust by fixing a bug
that caused the MDNS service to fail to register opened substreams.
Additionally, the release includes several improvements to the
`identify` protocol, replacing `FuturesUnordered` with `FuturesStream`
for better performance.

### Fixed

- mdns/fix: Failed to register opened substream
([#301](https://github.com/paritytech/litep2p/pull/301))

### Changed

- identify: Replace FuturesUnordered with FuturesStream
([#302](https://github.com/paritytech/litep2p/pull/302))
- chore: Update hickory-resolver to version 0.24.2
([#304](https://github.com/paritytech/litep2p/pull/304))
- ci: Ensure cargo-machete is working with rust version from CI
([#303](https://github.com/paritytech/litep2p/pull/303))


cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

# Description

Get runtime's metadata, parse it and verify pallets list for a pallet
named `ParachainSystem` (for now), and block number to be the same for
both node and runtime. Ideally we'll add other pallets checks too, at
least a small set of pallets we think right away as mandatory for
parachain compatibility.
Closes: #5565 

## Integration

Runtime devs must be made aware that to be fully compatible with Omni
Node, certain naming conventions should be respected when defining
pallets (e.g we verify parachain-system pallet existence by searching
for a pallet with `name` `ParachainSystem` in runtime's metadata). Not
finding such a pallet will not influence the functionality yet, but by
doing these checks we could provide useful feedback for runtimes that
are clearly not implementing what's required for full parachain
compatibility with Omni Node.

## Review Notes

- [x] parachain system check
- [x] check frame_system's metadata ...