* Clarify code a bit.

* Move code around.

* Introduce `Order`.

* Introduce `Link` structure.

* Get rid of ptr_offset

This is beneficial since ptr_offset is essentially makes us handle two different address spaces, global (i.e. `mem`) and heap local and without it things are becoming simpler.

* Rename PREFIX_SIZE to HEADER_SIZE.

This will come in the next commits.

* Introduce a separate `Memory` trait.

This is not necessary, but will come in handy for the upcoming changes.

* Rename `ptr` to `header_ptr` where makes sense.

* Introduce a `Header` type.

* Make `bump` dumber.

This allows us to pull `HEADER_SIZE` to see that we actually allocate `order.size() + HEADER_SIZE`.

* Clean up.

* Introduce a freelists struct.

* Update documentation.

* Make Sized requirement optional to make the PR truly back-compatible.

* Apply suggestions from code review

Co-Authored-By: Gavin Wood <gavin@parity.io>

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Gavin Wood <github@gavwood.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* FreeingBumpAllocator: Initialize the heads to `u32::max_value()`

`self.heads` can point to an element with the index `0` in the heap.
This would make the allocator fail to reuse this element.

* Simplify the `PREFIX_SIZE` handling

Bastian Köcher authored 5 years ago and Sergey Pepyakin committed 5 years ago

This moves the wasm-allocator (`FreeingBumpHeapAllocator`) into its own
crate `sp-allocator`. This new crate can theoretically provide multiple
different allocators. Besides moving the allocator, this pr also makes
`FreeingBumpHeapAllocator` compile on `no_std`.

* Clean imports in wasmi_execution

* Replace `interpret_runtime_api_result` with `pointer_and_len_from_u64`.

* Extract sc-executor-common crate

* Extract `sc-executor-wasmi` into its own crate

* Extract `sc-executor-wasmtime` into its own crate.

* Add missing headers.

* Clean and docs

* Docs for sc-executor-wasmi

* Expand a comment about sandboxing

* Fix assert_matches

* Rename (un)pack_ptr_and_len and move them into util module

* Remove wasmtime errors in sc-executor-common

* Rename: Phase 1.

* Unify codec.

* Fixing: Phase 2

* Fixing: Phase 3.

* Fixing: Phase 4.

* Fixing: Phase 5.

* Fixing: Phase 6.

* Fixing: Phase 7.

* Fixing: Phase 8. Tests

* Fixing: Phase 9. Tests!!!

* Fixing: Phase 10. Moar tests!

* Finally done!

* More fixes.

* Rename primitives:: to sp_core::

* Apply renames in finality-grandpa.

* Fix benches.

* Fix benches 2.

* Revert node-template.

* Fix frame-system in our modules.

* Adding first rough ouline of the repository structure

* Remove old CI stuff

* add title

* formatting fixes

* move node-exits job's script to scripts dir

* Move docs into subdir

* move to bin

* move maintainence scripts, configs and helpers into its own dir

* add .local to ignore

* move core->client

* start up 'test' area

* move test client

* move test runtime

* make test move compile

* Add dependencies rule enforcement.

* Fix indexing.

* Update docs to reflect latest changes

* Moving /srml->/paint

* update docs

* move client/sr-* -> primitives/

* clean old readme

* remove old broken code in rhd

* update lock

* Step 1.

* starting to untangle client

* Fix after merge.

* start splitting out client interfaces

* move children and blockchain interfaces

* Move trie and state-machine to primitives.

* Fix WASM builds.

* fixing broken imports

* more interfac...

* Update all dependencies

* Upgrade dependencies whenever “easy”

“easy” means that there are no major changes required.

* Fix build and bump paste dependency to 0.1.6

* Remove dead code

* Re-add = dependency for futures-preview

* Add missing std features for runtime-io

* Remove git dependencies

as updated versions have been published to crates.io

* try to debug bug

* For sr-io, "std" should imply "no_oom" and "no_panic_handler".

Otherwise, rustc complains (correctly) about duplicate lang items.

* Add missing "runtime-io/std" features

* Fix compilation errors

* Prevent duplicate lang items

Rust does not allow duplicate lang items.  When compiled without the
`std` feature, `sr-io` defines two lang items.  Therefore, `sr-io`
compiled without `feature = "std"` must not be linked with `std`.

However, `pwasm-utils` and `wasmi-validation` both bring in `std` unless
compiled with `default-features = "false"`.  This caused a duplicate
lang item error.  Building both with `default-features = "false"`
prevents this error.  When building with `feature = "std"`, they should
both be built with the `std` feature, so this feature needs to be
explicitly depended on.

* Bump `impl_version`

* Make tests pass

Three tests used 1 less gas than they had previously.

* Try to un-break build

* Add a Cargo.lock file

* Revert offchain code

* Revert "Revert offchain code"

This reverts commit d216d08cc6ca0344614669c1d24cde3aa5c0d4e2.

* Don’t try to send a body with a GET request

without adding a Transfer-Encoding or Content-Length header.

This has always been wrong, but hyperium/hyper#1925 hid the bug until
hyper was upgraded to 0.12.35.

* Change some more GET requests to POST requests

* Fix excess line width and remove an `extern crate`

* Delete commented-out extern crate

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

* Fix regression in Cargo.toml files

dev-dependencies need `default-features = false`, too.

* Bump parity-wasm dependency

* Bump `futures-preview`

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update Cargo.lock files

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update core/service/src/chain_ops.rs

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

Jim Posen authored 5 years ago and Sergey Pepyakin committed 5 years ago

* executor: Support non-Wasmi sandbox supervisor environments.

* node-executor: Tests don't reference WasmExecutor directly.

* executor: Simplify encoding of heap metadata.

* executor: Decouple allocator from wasmi::MemoryRef.

* Clear up import/export misunderstandings

* Fetch minimum period from runtime

* Remove unnecessary comment

This variable is already fetched from the runtime
in the line below.

* Fix bug in factory

The `best_block_id` stayed the same, it was always the
genesis hash. This resulted in the factory failing after
4096 blocks, since `client/db` discards hashes (in this
case the genesis hash) after 4096 blocks from the database.

* Fix tense in error message

* Improve allocator documentation

* Fix bug in allocator

Under certain circumstances an invalid pointer was
returned: when the `ptr` was calculated as equal
to the `max_heap_size`. This is an invalid pointer
since there is no access allowed after the heap limit.

The way to provoke this was to repeatedly allocate
with sizes which were previously not allocated and
immediately deallocate right afterwards. What this
did was to increment the `bumper` with each allocation,
whilst keeping the `total_size` of the heap `0`.
If this repeated allocation/deallocation scheme resulted
in `max_heap_size == ptr` the `ptr` was still returned.

The allocator only checked if the `total_size` was
still within the `max_heap_size` limits, and not
if the resulting `ptr` was still within the valid
heap region.

This commit introduces a check to validate if the
calculated `ptr` is within the heap.

* Add test for zero byte allocation and document behavior

* Improve code readability by introducing a const

* Fix error message in test

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix code review suggestions

* Replace early return with assertion

* Remove test for zero size allocations

* Shorten test code

* Shorten comment

* Make bump() return Result

* Add comment for bump()

* Remove ambiguous comment

* Replace value with const

* Use proof for panic message

* Fix merge

* Add comment regarding minimum allocation size

* Adds new wrapper traits for wasm executor

* Add new crate `substrate-wasm-interface`

Thew new crate holds types and traits for the communicating between the
wasm runtime and the host.

* Rewrite externals with new macro etc

* Fix vec initialization

* Make executor tests working

* Remove unused code + warnings

* Introduce `Pointer` and `WordSize` for working with wasm

* Fix tests and compilation

* Fix compilation

* Apply suggestions from code review

Co-Authored-By: Sergei Pepyakin <sergei@parity.io>

* Review feedback

* Remove unused conversions

* Make each host function its own struct

`HostFunctions` now just returns these function structs. Each function
can be executed by using one of the function structs. The inherent host
functions are now moved to the "normal" host functions.

* Remove byteorder

* Add floating point types

* Make pointer interface more safe

* Add type alias for wasm-interface Result

* More review comments

* Add test from original bug report

Original is from @pepyakin

 in 3d7b27f3421818e8d6de568e02fbc2947a06246b.
I adapted it to work with the latest master.

* No longer cleanup module instance

* Replace runtime cache with synchronous clone

* Fix test

* Preserve initial runtime memory and restore it on fetch

* Remove leftover comment

* Fix style

* Improve variable naming

* Replace get_into() with get()

* Handle missing memory export better

* Return earlier when creating runtime first time

* Improve comments

* fmt

* Fix #2967.

* Eradicate `code` from `Error::InvalidCode`

* tidy

* A state snapshot doc.

* Store multiple runtimes by hash.

* Get rid of deref.

* Docs

* Use Self for instantiate_module

* REVERT ME

* Should be ok

* Commit

* Remove dbg

* Use fast-memory's erase

* Clean and undo hacks.

* Introduce a dedicated error for heap_base

* Ban the start function.

* Clean, docs and refactor

* Add rustflags.

* Update Cargo.lock

* Apply Basti's suggestions

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Rename allocates_huge_stack_array

* Extend TestClientBuilder with set_heap_pages

* Update the test.

* Update core/executor/src/wasm_executor.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update core/executor/src/wasm_runtimes_cache.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update core/executor/src/error.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update core/executor/src/error.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix tests.

* Update cargo-lock

* Use wasmi master

* Use master wasmi

* Move tests.

* Use wasmi crates.io

* Update Cargo.lock

* Fix build.rs

* Bump runtime version

* Revert initial_heap_pages renaming

* Bump wasmi up to 0.5.0

* Bump runtime version

* Don't restore an instance every now and then

* Update core/executor/src/wasm_runtimes_cache.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Propagate error in CacheError

* Clarify the get_heap_base call in instantiation

* Supply --export=__heap_base

See https://reviews.llvm.org/D62744

Co-authored-by: Jim Posen <jim.posen@gmail.com>

* Bump version.

* Use combinators for segments.

* Fix build.rs

* Fix build.rs for runtime-test

Closes #3005

* Remove `UserError` and switch to `error::Error`

* More cleanup

* Update core/executor/src/error.rs

Co-Authored-By: Sergei Pepyakin <s.pepyakin@gmail.com>

* Fix typo

* Update core/executor/src/allocator.rs

Co-Authored-By: Stanislav Tkach <stanislav.tkach@gmail.com>

* Introduce inherent digests

* Implement inherent digests

* fix silly error

* Implementation of inherent digests in BABE

All tests pass. There are still limitations:

1. The runtime strips out inherent digests, so BABE must re-add them.
2. The test runtime checks that it can re-compute all digests.  It
   can’t, so I had to comment out that test.

* Fix compilation and seal import

Seals were not imported correctly: the pre-digest was imported twice,
instead of both it and the seal being imported.  Also, other parts of
the code did not compile due to incomplete refactoring.

* Remove bogus assertion

* Fix testsuite compilation

* Remove unused import

* Fix compiler diagnostics

* Add inherent digest parameters to block constructors

This enforces that inherent digests are added first.

* Fixup Cargo.lock

* Fix build errors

* Re-add an incorrectly removed import

* Bump primitive-types version

* Update Cargo.lock

* Refactoring

* Use inherent digests for AuRa

They do reach the runtime, but get stripped.  I have not figured out
where.

* Fix compilation errors

* Fix compilation errors due to incorrect types

* Fix whitespace

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Add preamble

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix silly compile error

* Refactor pre-digest finding code into a separate function

* Remove unwanted assertion

It is too likely to bring down the entire blockchain.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Use `find_pre_digest` after runtime, too

Also, use `Member` trait rather than rolling our own requirements.

Suggested-by: Tomasz Drwiega <tomasz@parity.io>

* Fix various warnings

mostly due to upgrading the dependency on `error_chain`.

* Pre-digests nearly complete

This nearly completes the implementation of pre-runtime digests.

* `Seal2` → `Seal` and fix test suite

* Try to fix the storage error

* Try to fix storage (again)

* Fix tests

* Hopefully finish pre-runtime digests

The key is to pass *only* the pre-runtime digests to the runtime.  The
others must be stripped out by `initialize_block`.

* Fix silly typo

* Fix another silly mistake

* Remove unnecessary filtering of BABE pre-digests

We no longer get duplicate BABE pre-digests, so if they appear, the
header should be rejected outright.

* Update Cargo.lock files

* Reformatting

* Fix silly typo in inherent digest code

Also, revert `error.rs` files that contained calls to the `error_chain!`
macro.

* Try to keep the runtime from stripping pre-digests

Currently runs into the “Storage root must match that calculated”
assertion.

* Don’t compute storage root until storage changes are done.

Also, fix a compilation error.

* Fix compile-time error

* Fix compilation errors

* Fix more compile errors

* Hopefully it compiles this time…

* Fix compilation and add docs

* Prevent BABE from adding duplicate pre-runtime digests

Found by comparing with the AuRa code.  I also did some refactoring.

* Respond to review and fix some warnings

* Delete some dead code introduced earlier

* More dead code goes away

* `ref mut` → `&mut`

* Respond to review and fix some warnings

* Fix compilation error

* Remove unneeded `HashT` type parameter

Suggested-by: Robert Habermeier <robert@parity.io>

* Remove spurious #[allow(deprecated)]

* Document inherent digest parameter to `build_block`

* Delete `Simple` trait

It wasn’t needed

* delete wrongly added files

* Fix trait bounds

* Digest serialization tests

I also did some reformatting and cleanup.

* Apply suggestions from code review

Reformatting

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Swap two arguments to `propose` and `propose_with`

Also, remove some needless unsafe code.

* Remove bogus `#![allow(deprecated)]` annotations

With the removal of the deprecated `Seal` variant, these are not needed.

* Add a missing `#[allow(deprecated)]` in the AuRa tests

* Fix silly compile error

* Fix silly compiler error

RLS did not tell me that I hadn’t fixed `babe/lib.rs`, so I missed it.

* Fixes made automatically by Cargo

* Replace unwrap with expect

* Move expect to call sites

* Bubble errors up and trap

* Update wasm

* Update invalid pointer

* Remove test which makes problems in CI

* Check for underflow

* Link substrate issue tracker in panic

* Replace allocator with freeing-bump allocator

* Revert me: Panic on double allocate/free

* Revert me: Add shallow benchmark for a first impression

* Revert "Revert me: Add shallow benchmark for a first impression"

This reverts commit 5f0d4df39395eb55c9ec2d6fe1ed992533475fec.

* Revert "Revert me: Panic on double allocate/free"

This reverts commit a114df7d6cfc21d60af396ccca02e5c8205db6ce.

* Rename heap to FreeingBumpHeapAllocator

* Rename heap.rs to allocator.rs

* Use sandbox heap

* Move functions

* Move variables into constructor

* Revert "Move variables into constructor"

This reverts commit f46fa0d0cdf4ea97760ccce58003b0d33f433743.

* Remove unnecessary casts

* Add comment for new parameter

* Improve typing

* Move variables into constructor

* Avoid dynamic allocation

* Remove unused variables

* Revert "Link substrate issue tracker in panic"

This reverts commit 32dfa1d02bcf881d1d514a930fcc0fdf3c5f8e08.

In the meantime this was fixed in https://github.com/paritytech/substrate/pull/1667.

* Improve naming

* Only assert in debug mode

* Remove dynamic allocation