Unverified Commit 60d3305e authored by Radu Popa's avatar Radu Popa Committed by GitHub
Browse files

Update SimNet test to the new format (#4051)

* simnet-v10 update simnet image version

* update config

* update path to js scripts

* use current branch to download config for tests

* use current branch to download config for tests

* temp use simnet-v10

* update also the toml file

* Rework test according to CIHelper format

* Call cihelper script for running tests

* Switch to v12

* Version 13

* simnet v14

* Readme extended
Co-authored-by: default avatarAnton Gavrilov <AntonE.Gavrilov@gmail.com>
Co-authored-by: Denis_P's avatarDenis Pisarev <denis.pisarev@parity.io>
parent dc9a1bbe
Pipeline #164476 canceled with stages
in 25 minutes and 58 seconds
......@@ -31,6 +31,8 @@ variables:
VAULT_SERVER_URL: "https://vault.parity-mgmt-vault.parity.io"
VAULT_AUTH_PATH: "gitlab-parity-io-jwt"
VAULT_AUTH_ROLE: "cicd_gitlab_parity_${CI_PROJECT_NAME}"
SIMNET_IMAGE: "europe-west3-docker.pkg.dev/parity-simnet/simnet-images/simnet:v14"
cache: {}
......@@ -566,8 +568,12 @@ deploy-polkasync-kusama:
stage: deploy
image: "docker.io/paritytech/simnet:${SIMNET_REF}"
image: "${SIMNET_IMAGE}"
<<: *kubernetes-env
# FIXME: change branch to "master" after merging this PR
# GH_DIR: "https://github.com/paritytech/polkadot/tree/master/simnet_tests"
GH_DIR: "https://github.com/paritytech/polkadot/tree/simnet-v10/simnet_tests"
- if: $CI_PIPELINE_SOURCE == "schedule"
- if: $CI_COMMIT_REF_NAME == "master"
......@@ -584,17 +590,19 @@ simnet-tests:
# SIMNET_REF is a gitlab variable
- echo "Simnet Tests Config
- echo "Simnet Tests Config"
- echo "${SIMNET_IMAGE}"
- echo "${GH_DIR}"
- /home/nonroot/simnet/gurke/scripts/run-test-environment-manager.sh
- /home/nonroot/simnet/scripts/run-test-environment-manager-v2.sh
allow_failure: true
retry: 2
# Simulation tests, or high level integration tests
# Simulation tests, or end-to-end tests
_The content of this directory is meant to be used by Parity's private CI/CD
infrastructure with private tools. At the moment those tools are still early
......@@ -7,32 +7,77 @@ public use._
## Contents of this directory
`configs` directory contains config files in toml format that describe how to
configure the simulation network that you want to launch.
This directory contains different test suits, everyone one of them contains the set of test cases.
Every test suits is defined by its definition file test_suit_description.toml. More information about
structure of test suits and test cases may be found in [SimNet repository](https://gitlab.parity.io/parity/simnet/-/tree/master/ci_helper).
`tests` directory contains [Cucumber](https://cucumber.io/) files. Those are
Behavior-Driven Development test files that describe tests in plain English.
Under the hood there are assertions that specific metrics should have specific
Every test case deploys a test network, using toml config file, and runs the test,
using a test scenario, written in [Cucumber](https://cucumber.io/).
At the moment we have only one test for parachains: `/parachains.features`
This test uses a JS script that we added to Simnet image and it's launched
by this step in the cucumber file:
`Then launch 'node' with parameters '--unhandled-rejections=strict /usr/local/bin/simnet_scripts test_parachain ./configs/adder.json ws://localhost:11222 100 10'`
These test suits are run by Polkadot CI in different pipelines, for every commit in PR, for commit into master etc.
It's the responsibility of the test's developer to provide the correct tag for their test, in order to let CI know, when
this test case should be run. For the baseline the existing tags from the existing tests may be used. If these tags are not
sufficient, the new tag may be created. But CI team should be aware of this tag and condition, when this test case should be run.
`run_test.sh` is an entry point for running all tests in the folder.
Any setup required for tests (but cannot be done in configs) is performed
here. The main script's responsibility is to run [Gurke](https://github.com/paritytech/gurke)
with passed parameters.
In order to use this script locally, you need to install
In order to run a test case locally, you need to install
Once you have access to a kubernetes cluster (meaning you can do `kubectl get pods`)
you can run this script with no arguments, like `./run_test.sh` and tests should run.
you can use Gurke in order to deploy a chain and run the test (see gurke's manual for the commands).
Kubernetes cluster can be local, spawned with
or an instance living in the
## How to add new test cases
New test case may be added either into the existing test suit or with creation of the new test suit.
In any case it's better to create the test and run it locally first, using Gurke (see above).
- In order to add the test case into the existing test suit, the new element (test case) should be added into [[test-cases]] array in test_suit_description.toml of this test suit. The example:
# The existing test case
tags = ["all", "smoketest"]
chain-config = "configs/default_local_testnet.toml"
scenarios = ["tests/001-smoketest.feature"]
# The new test case
tags = ["all", "load"]
allowed-to-fail = true
chain-config = "configs/default_local_testnet.toml"
scenarios = ["tests/002-loadtest.feature"]
(See note about test case's tags above).
- In order to create a new test suit for the test case, new folder with test suit description file (test_suit_description.toml) should be created. The exact name is mandatory, CI traverses all subfolfders of simnet_tests directory and looks for this file, in order to build the list of test suits. In this description file the general information about the test suit and array of the test cases should be specified. The example of test_suit_description.toml file with some verbose comments:
name = "Name of the test suit"
description = "General information about the suit"
# It is the path to the setup script, that may be needed for the test suit.
# You can perform in this script the actions, that are required for the chain deployment, but have to be performed before
# spawing the chain. Like pre-generation of seeds. This script is run by CI before spawing the chain in the cluster
# If the config of the test case requires some custom Docker images, the names for these images should be listed in this section
# CI has to provide all these images in the format image_name = some_value in order to run test cases from this suit
# Obviosuly CI should be aware, if the new custom image is added.
required-images = [
# Array of the test cases
# Every elements should be started with [[test-cases]]
# See tags information above
tags = ["all", "smoketest"]
# The config, that will be used in order to spawn the chain
chain-config = "configs/simple_rococo_testnet.toml"
# The array of the scenarios, that will be run on the deployed chain
scenarios = ["tests/parachains.feature"]
### [Here is link to barcamp presentation of Simnet](https://www.crowdcast.io/e/ph49xu01)
### [Here is link to the Simnet repo, hosted on private gitlab](https://gitlab.parity.io/parity/simnet/-/tree/master)
......@@ -2,8 +2,10 @@
timeout = 300
command = "/usr/local/bin/polkadot"
image = "{{get_env(name="SYNTHIMAGE") | safe }}"
command = "polkadot"
chain-spec = "rococo-local.json"
chain-name = "rococo-local"
timeout = 300
......@@ -23,10 +25,14 @@ timeout = 300
validator = true
image = "{{get_env(name="PARACHAINSIMAGE") | safe }}"
command = "polkadot"
extra-args = ["--alice"]
validator = true
image = "{{get_env(name="PARACHAINSIMAGE") | safe }}"
command = "polkadot"
extra-args = ["--bob"]
......@@ -38,7 +44,7 @@ image = "{{get_env(name="SCRIPTSIMAGE") | safe }}"
command-with-args = """
cd simnet_scripts;
npm run build;
node /usr/local/bin/simnet_scripts register_parachain /cfg/genesis-wasm /cfg/genesis-state 100 true ws://bootnode:9944;
node /usr/local/bin/simnet_scripts/dist/index.js register_parachain /cfg/genesis-wasm /cfg/genesis-state 100 true ws://bootnode:9944;
tail -f /dev/null
copy-files = [
name = "Parachains test"
description = "Smoke test for parachains, that deploy a simple collator and register parachain"
required-images = [
tags = ["all", "smoketest"]
chain-config = "configs/simple_rococo_testnet.toml"
scenarios = ["tests/parachains.feature"]
......@@ -3,4 +3,4 @@ Feature: ParaTesting
Scenario: spawn parachains network and check parachains
Given a test network
Then sleep 200 seconds
Then launch 'node' with parameters '--unhandled-rejections=strict /usr/local/bin/simnet_scripts test_parachain ./configs/adder.json ws://localhost:11222 100 10'
Then launch 'node' with parameters '/usr/local/bin/simnet_scripts/dist/index.js test_parachain /usr/local/bin/simnet_scripts/type_defs/adder.json ws://localhost:11222 100 10'
# All args have default values, specify args to override
# e.g: ./${SCRIPT_NAME} radu-test parity/substrate:some_feature /var/log/gurke
set -eou pipefail
SCRIPT_PATH=$(dirname "${SCRIPT_NAME}") # relative
SCRIPT_PATH=$(cd "${SCRIPT_PATH}" && pwd) # absolutized and normalized
function random_string {
head -1 <(fold -w 30 <(tr -dc 'a-z0-9' < /dev/urandom))
mkdir -p "${SCRIPT_PATH}"/logs
echo "Running tests in namespace: ${NAMESPACE}"
echo "Testing image: ${IMAGE}"
echo "Storing scripts logs to: ${LOG_PATH}"
echo "Colator image is ${COLIMAGE}"
function forward_port {
# RUN_IN_CONTAINER is env var that is set in the dockerfile
# use the -v operator to explicitly test if a variable is set
if [[ ! -v RUN_IN_CONTAINER ]] ; then
if is_port_forward_running ; then
function is_port_forward_running {
# shellcheck disable=SC2009
ps aux | grep -qE "${FORWARD_GREP_FILTER}"
function kill_previous_job {
# shellcheck disable=SC2009
job_pid=$(ps aux | grep -E "${FORWARD_GREP_FILTER}" | awk '{ print $2 }')
echo "INFO Killed forwading port 9944 into bootnode"
kill "${job_pid}"
function start_forwading_job {
kubectl -n "${NAMESPACE}" \
expose pod bootnode \
--name=rpc \
--type=NodePort \
--target-port=9944 \
kubectl -n "${NAMESPACE}" \
port-forward svc/rpc 11222:9944 &> "${LOG_PATH}/forward-${NAMESPACE}.log" &
sleep 2
echo "INFO Started forwading port 9944 into bootnode"
function update_api {
cd "${SCRIPT_PATH}"/../../simnet_scripts/
npm run build
cd -
echo "INFO: Checking if namespace has no pods"
kubectl -n "${NAMESPACE}" get pods
set -x # echo the commands to stdout
gurke spawn --config "${SCRIPT_PATH}"/configs/simple_rococo_testnet.toml \
-n "${NAMESPACE}" \
--image "${IMAGE}"
echo "INFO: Checking if pods launched correctly"
kubectl -n "${NAMESPACE}" get pods -o wide
echo "INFO: Updating Polkadot JS API"
# Run tests
gurke test "${NAMESPACE}" "${SCRIPT_PATH}"/tests --log-path "${LOG_PATH}"
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment