1. 21 Jun, 2021 1 commit
    • Shawn Tabrizi's avatar
      Use `max_code_size` and `max_wasm_data_size` from Parachains Configuration (#3329) · 897afffb
      Shawn Tabrizi authored
      
      
      * use `configuration::config()` for max bytes
      
      * Update integration_tests.rs
      
      * Update paras_registrar.rs
      
      * remove consts
      
      * add asserts for non-zero
      
      * more const clean up
      
      * cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_common::paras_registrar --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_common_paras_registrar.rs
      
      * cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_common::paras_registrar --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_common_paras_registrar.rs
      
      * add checks to `MAX_CODE_SIZE`
      
      * re-pot MAX_POV_SIZE
      
      * check pov limit in runtime
      
      * POV_BOMB_LIMIT multiplier
      
      * fix compile
      
      * Update configuration.rs
      
      * Update node/primitives/src/lib.rs
      
      * fix test
      
      Co-authored-by: default avatarParity Bot <admin@parity.io>
      897afffb
  2. 01 Jun, 2021 1 commit
  3. 16 Apr, 2021 1 commit
    • Robert Klotzner's avatar
      Infrastructure improvements (#2897) · fff63cb5
      Robert Klotzner authored
      * Factor out runtime module into utils.
      
      * Add maybe_authority information to `PeerConnected` event.
      
      We already gather this information in authority discovery, so we might
      as well share it with others.
      
      This opens up an easy path to trigger validators differently from normal
      nodes, e.g. for prioritization. This change has become more important
      now, that we just connect to all validators and therefore just have a
      long peer list without any information about those nodes.
      
      * Test fix.
      fff63cb5
  4. 09 Apr, 2021 1 commit
    • Robert Klotzner's avatar
      Req/res optimization for statement distribution (#2803) · 48d5b143
      Robert Klotzner authored
      * Wip
      
      * Increase proposer timeout.
      
      * WIP.
      
      * Better timeout values now that we are going to be connected to all nodes. (#2778)
      
      * Better timeout values.
      
      * Fix typo.
      
      * Fix validator bandwidth.
      
      * Fix compilation.
      
      * Better and more consistent sizes.
      
      Most importantly code size is now 5 Meg, which is the limit we currently
      want to support in statement distribution.
      
      * Introduce statement fetching request.
      
      * WIP
      
      * Statement cache retrieval logic.
      
      * Review remarks by @rphmeier
      
      * Fixes.
      
      * Better requester logic.
      
      * WIP: Handle requester messages.
      
      * Missing dep.
      
      * Fix request launching logic.
      
      * Finish fetching logic.
      
      * Sending logic.
      
      * Redo code size calculations.
      
      Now that max code size is compressed size.
      
      * Update Cargo.lock (new dep)
      
      * Get request receiver to statement distribution.
      
      * Expose new functionality for responding to requests.
      
      * Cleanup.
      
      * Responder logic.
      
      * Fixes + Cleanup.
      
      * Cargo.lock
      
      * Whitespace.
      
      * Add lost copyright.
      
      * Launch responder task.
      
      * Typo.
      
      * info -> warn
      
      * Typo.
      
      * Fix.
      
      * Fix.
      
      * Update comment.
      
      * Doc fix.
      
      * Better large statement heuristics.
      
      * Fix tests.
      
      * Fix network bridge tests.
      
      * Add test for size estimate.
      
      * Very simple tests that checks we get LargeStatement.
      
      * Basic check, that fetching of large candidates is performed.
      
      * More tests.
      
      * Basic metrics for responder.
      
      * More metrics.
      
      * Use Encode::encoded_size().
      
      * Some useful spans.
      
      * Get rid of redundant metrics.
      
      * Don't add peer on duplicate.
      
      * Properly check hash
      
      instead of relying on signatures alone.
      
      * Preserve ordering + better flood protection.
      
      * Get rid of redundant clone.
      
      * Don't shutdown responder on failed query.
      
      And add test for this.
      
      * Smaller fixes.
      
      * Quotes.
      
      * Better queue size calculation.
      
      * A bit saner response sizes.
      
      * Fixes.
      48d5b143
  5. 08 Apr, 2021 1 commit
    • asynchronous rob's avatar
      Code, PoV compression and remove `CompressedPoV` struct (#2852) · e5bab572
      asynchronous rob authored
      * use compressed blob in candidate-validation
      
      * add some tests for compressed code blobs
      
      * remove CompressedPoV and apply compression in collation-generation
      
      * decompress BlockData before executing
      
      * don't produce oversized collations
      
      * add test for PoV decompression failure
      
      * fix tests and clean up
      
      * fix test
      
      * address review and fix CI
      
      * take this )
      e5bab572
  6. 31 Mar, 2021 1 commit
  7. 29 Mar, 2021 1 commit
  8. 28 Mar, 2021 1 commit
    • Robert Klotzner's avatar
      Request based PoV distribution (#2640) · 39124b0a
      Robert Klotzner authored
      * Indentation fix.
      
      * Prepare request-response for PoV fetching.
      
      * Drop old PoV distribution.
      
      * WIP: Fetch PoV directly from backing.
      
      * Backing compiles.
      
      * Runtime access and connection management for PoV distribution.
      
      * Get rid of seemingly dead code.
      
      * Implement PoV fetching.
      
      Backing does not yet use it.
      
      * Don't send `ConnectToValidators` for empty list.
      
      * Even better - no need to check over and over again.
      
      * PoV fetching implemented.
      
      + Typechecks
      + Should work
      
      Missing:
      
      - Guide
      - Tests
      - Do fallback fetching in case fetching from seconding validator fails.
      
      * Check PoV hash upon reception.
      
      * Implement retry of PoV fetching in backing.
      
      * Avoid pointless validation spawning.
      
      * Add jaeger span to pov requesting.
      
      * Add back tracing.
      
      * Review remarks.
      
      * Whitespace.
      
      * Whitespace again.
      
      * Cleanup + fix tests.
      
      * Log to log target in overseer.
      
      * Fix more tests.
      
      * Don't fail if group cannot be found.
      
      * Simple test for PoV fetcher.
      
      * Handle missing group membership better.
      
      * Add test for retry functionality.
      
      * Fix flaky test.
      
      * Spaces again.
      
      * Guide updates.
      
      * Spaces.
      39124b0a
  9. 25 Mar, 2021 1 commit
  10. 24 Mar, 2021 2 commits
    • Robert Klotzner's avatar
      Unify maximum supported PoV size a bit. (#2691) · d2d50352
      Robert Klotzner authored
      * Unify maximum supported PoV size a bit.
      
      * Use MAX_POV_SIZE also in `HostConfiguration`.
      
      * Fix types.
      d2d50352
    • Robert Klotzner's avatar
      Bigger is better. (#2687) · 12d44b14
      Robert Klotzner authored
      * Bigger is better.
      
      Made all request response sizes 10 times bigger.
      
      * The smaller the better.
      
      * Update comment.
      
      * Ah, bigger is still better.
      
      Max PoV size for rococo is around 50Meg, compression ratio is about 3.4.
      With 30 Meg we should be fine, even with crypto kitties in the PoV.
      12d44b14
  11. 18 Mar, 2021 1 commit
    • Robert Klotzner's avatar
      Request based collation fetching (#2621) · 134090c2
      Robert Klotzner authored
      * Introduce collation fetching protocol
      
      also move to mod.rs
      
      * Allow `PeerId`s in requests to network bridge.
      
      * Fix availability distribution tests.
      
      * Move CompressedPoV to primitives.
      
      * Request based collator protocol: validator side
      
      - Missing: tests
      - Collator side
      - don't connect, if not connected
      
      * Fixes.
      
      * Basic request based collator side.
      
      * Minor fix on collator side.
      
      * Don't connect in requests in collation protocol.
      
      Also some cleanup.
      
      * Fix PoV distribution
      
      * Bump substrate
      
      * Add back metrics + whitespace fixes.
      
      * Add back missing spans.
      
      * More cleanup.
      
      * Guide update.
      
      * Fix tests
      
      * Handle results in tests.
      
      * Fix weird compilation issue.
      
      * Add missing )
      
      * Get rid of dead code.
      
      * Get rid of redundant import.
      
      * Fix runtime build.
      
      * Cleanup.
      
      * Fix wasm build.
      
      * Format fixes.
      
      Thanks @andronik !
      134090c2
  12. 26 Feb, 2021 1 commit
    • Robert Klotzner's avatar
      Request based availability distribution (#2423) · 950447e1
      Robert Klotzner authored
      * WIP
      
      * availability distribution, still very wip.
      
      Work on the requesting side of things.
      
      * Some docs on what I intend to do.
      
      * Checkpoint of session cache implementation
      
      as I will likely replace it with something smarter.
      
      * More work, mostly on cache
      
      and getting things to type check.
      
      * Only derive MallocSizeOf and Debug for std.
      
      * availability-distribution: Cache feature complete.
      
      * Sketch out logic in `FetchTask` for actual fetching.
      
      - Compile fixes.
      - Cleanup.
      
      * Format cleanup.
      
      * More format fixes.
      
      * Almost feature complete `fetch_task`.
      
      Missing:
      
      - Check for cancel
      - Actual querying of peer ids.
      
      * Finish FetchTask so far.
      
      * Directly use AuthorityDiscoveryId in protocol and cache.
      
      * Resolve `AuthorityDiscoveryId` on sending requests.
      
      * Rework fetch_task
      
      - also make it impossible to check the wrong chunk index.
      - Export needed function in validator_discovery.
      
      * From<u32> implementation for `ValidatorIndex`.
      
      * Fixes and more integration work.
      
      * Make session cache proper lru cache.
      
      * Use proper lru cache.
      
      * Requester finished.
      
      * ProtocolState -> Requester
      
      Also make sure to not fetch our own chunk.
      
      * Cleanup + fixes.
      
      * Remove unused functions
      
      - FetchTask::is_finished
      - SessionCache::fetch_session_info
      
      * availability-distribution responding side.
      
      * Cleanup + Fixes.
      
      * More fixes.
      
      * More fixes.
      
      adder-collator is running!
      
      * Some docs.
      
      * Docs.
      
      * Fix reporting of bad guys.
      
      * Fix tests
      
      * Make all tests compile.
      
      * Fix test.
      
      * Cleanup + get rid of some warnings.
      
      * state -> requester
      
      * Mostly doc fixes.
      
      * Fix test suite.
      
      * Get rid of now redundant message types.
      
      * WIP
      
      * Rob's review remarks.
      
      * Fix test suite.
      
      * core.relay_parent -> leaf for session request.
      
      * Style fix.
      
      * Decrease request timeout.
      
      * Cleanup obsolete errors.
      
      * Metrics + don't fail on non fatal errors.
      
      * requester.rs -> requester/mod.rs
      
      * Panic on invalid BadValidator report.
      
      * Fix indentation.
      
      * Use typed default timeout constant.
      
      * Make channel size 0, as each sender gets one slot anyways.
      
      * Fix incorrect metrics initialization.
      
      * Fix build after merge.
      
      * More fixes.
      
      * Hopefully valid metrics names.
      
      * Better metrics names.
      
      * Some tests that already work.
      
      * Slightly better docs.
      
      * Some more tests.
      
      * Fix network bridge test.
      950447e1
  13. 03 Feb, 2021 1 commit
    • Robert Klotzner's avatar
      Generic request/response infrastructure for Polkadot (#2352) · ecc3772d
      Robert Klotzner authored
      * Move NetworkBridgeEvent to subsystem::messages.
      
      It is not protocol related at all, it is in fact only part of the
      subsystem communication as it gets wrapped into messages of each
      subsystem.
      
      * Request/response infrastructure is taking shape.
      
      WIP: Does not compile.
      
      * Multiplexer variant not supported by Rusts type system.
      
      * request_response::request type checks.
      
      * Cleanup.
      
      * Minor fixes for request_response.
      
      * Implement request sending + move multiplexer.
      
      Request multiplexer is moved to bridge, as there the implementation is
      more straight forward as we can specialize on `AllMessages` for the
      multiplexing target.
      
      Sending of requests is mostly complete, apart from a few `From`
      instances. Receiving is also almost done, initializtion needs to be
      fixed and the multiplexer needs to be invoked.
      
      * Remove obsolete multiplexer.
      
      * Initialize bridge with multiplexer.
      
      * Finish generic request sending/receiving.
      
      Subsystems are now able to receive and send requests and responses via
      the overseer.
      
      * Doc update.
      
      * Fixes.
      
      * Link issue for not yet implemented code.
      
      * Fixes suggested by @ordian
      
       - thanks!
      
      - start encoding at 0
      - don't crash on zero protocols
      - don't panic on not yet implemented request handling
      
      * Update node/network/protocol/src/request_response/v1.rs
      
      Use index 0 instead of 1.
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * Update node/network/protocol/src/request_response.rs
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * Fix existing tests.
      
      * Better avoidance of division by zoro errors.
      
      * Doc fixes.
      
      * send_request -> start_request.
      
      * Fix missing renamings.
      
      * Update substrate.
      
      * Pass TryConnect instead of true.
      
      * Actually import `IfDisconnected`.
      
      * Fix wrong import.
      
      * Update node/network/bridge/src/lib.rs
      
      typo
      
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      
      * Update node/network/bridge/src/multiplexer.rs
      
      Remove redundant import.
      
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      
      * Stop doing tracing from within `From` instance.
      
      Thanks for the catch @tomaka
      
      !
      
      * Get rid of redundant import.
      
      * Formatting cleanup.
      
      * Fix tests.
      
      * Add link to issue.
      
      * Clarify comments some more.
      
      * Fix tests.
      
      * Formatting fix.
      
      * tabs
      
      * Fix link
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      
      * Use map_err.
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      
      * Improvements inspired by suggestions by @drahnr.
      
      - Channel size is now determined by function.
      - Explicitely scope NetworkService::start_request.
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      ecc3772d