Publish RC container images (#7556)

* WIP

* Add missing checkout

* Add debuggin

* Fix VAR name

* Bug fix

* Rework jobs

* Revert "Rework jobs"

This reverts commit 2bfa79fd3ae633c17403b838f9a5025f0f7fc3f3.

* Add cache

* Add temp default for testing

* Add missing checkout

* Fix patch

* Comment out the GPG check for now

* Rename polkadot_injected_release into a more appropriate polkadot_injected_debian

* Refactoring / renaming

* Introduce a generic image for binary injection

* Flag files to be deleted and changes to be done

* WIP

* Fix multi binaries images

* Add test build scripts

* Remove old file, add polkadot build-injected script

* Fix doc

* Fix tagging

* Add build of the injected container

* Fix for docker

* Remove the need for TTY

* Handling container publishing

* Fix owner and registry

* Fix vars

* Fix repo

* Fix var naming

* Fix case when there is no tag

* Fix case with no tag

* Handle error

* Fix sp...

polkadot/scripts/ci/dockerfiles/polkadot/docker-compose.yml

 version: '3'
 services:
   polkadot:
+    image: parity/polkadot:latest
+
     ports:
       - "127.0.0.1:30333:30333/tcp"
       - "127.0.0.1:9933:9933/tcp"
-    image: parity/polkadot:latest
+      - "127.0.0.1:9944:9944/tcp"
+      - "127.0.0.1:9615:9615/tcp"
+
     volumes:
       - "polkadot-data:/data"
+
     command: |
       --unsafe-rpc-external
       --unsafe-ws-external
       --rpc-cors all
       --prometheus-external
 
-    ports:
-      - "30333:30333"
-      - "9933:9933"
-      - "9944:9944"
-      - "9615:9615"
-
 volumes:
   polkadot-data:

polkadot/scripts/ci/dockerfiles/polkadot/polkadot_builder.Dockerfile

@@ -7,7 +7,7 @@ COPY . /polkadot
 RUN cargo build --locked --release
 
 # This is the 2nd stage: a very small image where we copy the Polkadot binary."
-FROM docker.io/library/ubuntu:20.04
+FROM docker.io/parity/base-bin:latest
 
 LABEL description="Multistage Docker image for Polkadot: a platform for web3" \
 	io.parity.image.type="builder" \

polkadot/scripts/ci/dockerfiles/polkadot_injected_release.Dockerfile → polkadot/scripts/ci/dockerfiles/polkadot/polkadot_injected_debian.Dockerfile

@@ -11,7 +11,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
 	io.parity.image.vendor="Parity Technologies" \
 	io.parity.image.title="parity/polkadot" \
 	io.parity.image.description="Polkadot: a platform for web3. This is the official Parity image with an injected binary." \
-	io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/scripts/ci/dockerfiles/polkadot_injected_release.Dockerfile" \
+	io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/scripts/ci/dockerfiles/polkadot/polkadot_injected_debian.Dockerfile" \
 	io.parity.image.revision="${VCS_REF}" \
 	io.parity.image.created="${BUILD_DATE}" \
 	io.parity.image.documentation="https://github.com/paritytech/polkadot/"

polkadot/scripts/ci/dockerfiles/polkadot/test-build.sh

+#!/usr/bin/env bash
+
+TMP=$(mktemp -d)
+ENGINE=${ENGINE:-podman}
+
+# You need to build an injected image first
+
+# Fetch some binaries
+$ENGINE run --user root --rm -i \
+  -v "$TMP:/export" \
+  --entrypoint /bin/bash \
+  parity/polkadot -c \
+  'cp "$(which polkadot)" /export'
+
+echo "Checking binaries we got:"
+tree $TMP
+
+./build-injected.sh $TMP

polkadot/scripts/ci/dockerfiles/polkadot_injected_debug.Dockerfile

-FROM docker.io/library/ubuntu:20.04
-
-# metadata
-ARG VCS_REF
-ARG BUILD_DATE
-ARG IMAGE_NAME
-
-LABEL io.parity.image.authors="devops-team@parity.io" \
-	io.parity.image.vendor="Parity Technologies" \
-	io.parity.image.title="${IMAGE_NAME}" \
-	io.parity.image.description="Polkadot: a platform for web3" \
-	io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/scripts/ci/dockerfiles/polkadot_injected_debug.Dockerfile" \
-	io.parity.image.revision="${VCS_REF}" \
-	io.parity.image.created="${BUILD_DATE}" \
-	io.parity.image.documentation="https://github.com/paritytech/polkadot/"
-
-# show backtraces
-ENV RUST_BACKTRACE 1
-
-# install tools and dependencies
-RUN apt-get update && \
-	DEBIAN_FRONTEND=noninteractive apt-get install -y \
-		libssl1.1 \
-		ca-certificates && \
-# apt cleanup
-	apt-get autoremove -y && \
-	apt-get clean && \
-	find /var/lib/apt/lists/ -type f -not -name lock -delete; \
-# add user and link ~/.local/share/polkadot to /data
-	useradd -m -u 1000 -U -s /bin/sh -d /polkadot polkadot && \
-	mkdir -p /data /polkadot/.local/share && \
-	chown -R polkadot:polkadot /data && \
-	ln -s /data /polkadot/.local/share/polkadot
-
-# add polkadot binary to docker image
-COPY ./polkadot ./polkadot-execute-worker ./polkadot-prepare-worker /usr/local/bin
-
-USER polkadot
-
-# check if executable works in this container
-RUN /usr/local/bin/polkadot --version
-RUN /usr/local/bin/polkadot-execute-worker --version
-RUN /usr/local/bin/polkadot-prepare-worker --version
-
-EXPOSE 30333 9933 9944
-VOLUME ["/polkadot"]
-
-ENTRYPOINT ["/usr/local/bin/polkadot"]

polkadot/scripts/ci/dockerfiles/staking-miner/README.md

+# staking-miner container image
+
+## Build using the Builder
+
+```
+./build.sh
+```
+
+## Build the injected Image
+
+You first need a valid Linux binary to inject. Let's assume this binary is located in `BIN_FOLDER`.
+
+```
+./build-injected.sh "$BIN_FOLDER"
+```
+
+## Test
+
+Here is how to test the image. We can generate a valid seed but the staking-miner will quickly notice that our
+account is not funded and "does not exist".
+
+You may pass any ENV supported by the binary and must provide at least a few such as `SEED` and `URI`:
+```
+ENV SEED=""
+ENV URI="wss://rpc.polkadot.io:443"
+ENV RUST_LOG="info"
+```
+
+```
+export SEED=$(subkey generate -n polkadot --output-type json  | jq -r .secretSeed)
+podman run --rm -it \
+    -e URI="wss://rpc.polkadot.io:443" \
+    -e RUST_LOG="info" \
+    -e SEED \
+    localhost/parity/staking-miner \
+        dry-run seq-phragmen
+```

polkadot/scripts/ci/dockerfiles/staking-miner/build-injected.sh

+#!/usr/bin/env bash
+
+# Sample call:
+# $0 /path/to/folder_with_staking-miner_binary
+# This script replace the former dedicated staking-miner "injected" Dockerfile
+# and shows how to use the generic binary_injected.dockerfile
+
+PROJECT_ROOT=`git rev-parse --show-toplevel`
+
+export BINARY=staking-miner
+export BIN_FOLDER=$1
+
+$PROJECT_ROOT/scripts/ci/dockerfiles/build-injected.sh

polkadot/scripts/ci/dockerfiles/staking-miner/build.sh

+#!/usr/bin/env bash
+
+# Sample call:
+# $0 /path/to/folder_with_staking-miner_binary
+# This script replace the former dedicated staking-miner "injected" Dockerfile
+# and shows how to use the generic binary_injected.dockerfile
+
+PROJECT_ROOT=`git rev-parse --show-toplevel`
+ENGINE=podman
+
+echo "Building the staking-miner using the Builder image"
+echo "PROJECT_ROOT=$PROJECT_ROOT"
+$ENGINE build -t staking-miner -f staking-miner_builder.Dockerfile "$PROJECT_ROOT"

polkadot/scripts/ci/dockerfiles/staking-miner/staking-miner_builder.Dockerfile

@@ -4,17 +4,17 @@ FROM paritytech/ci-linux:production as builder
 ARG VCS_REF
 ARG BUILD_DATE
 ARG IMAGE_NAME="staking-miner"
-ARG PROFILE=release
+ARG PROFILE=production
 
 LABEL description="This is the build stage. Here we create the binary."
 
 WORKDIR /app
 COPY . /app
-RUN cargo build --locked --$PROFILE --package staking-miner
+RUN cargo build --locked --profile $PROFILE --package staking-miner
 
 # ===== SECOND STAGE ======
 
-FROM docker.io/library/ubuntu:20.04
+FROM docker.io/parity/base-bin:latest
 LABEL description="This is the 2nd stage: a very small image where we copy the binary."
 LABEL io.parity.image.authors="devops-team@parity.io" \
 	io.parity.image.vendor="Parity Technologies" \
@@ -28,13 +28,10 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
 ARG PROFILE=release
 COPY --from=builder /app/target/$PROFILE/staking-miner /usr/local/bin
 
-RUN useradd -u 1000 -U -s /bin/sh miner && \
-	rm -rf /usr/bin /usr/sbin
-
 # show backtraces
 ENV RUST_BACKTRACE 1
 
-USER miner
+USER parity
 
 ENV SEED=""
 ENV URI="wss://rpc.polkadot.io"

polkadot/scripts/ci/dockerfiles/staking-miner/staking-miner_injected.Dockerfile

-FROM docker.io/library/ubuntu:20.04
-
-# metadata
-ARG VCS_REF
-ARG BUILD_DATE
-ARG IMAGE_NAME="staking-miner"
-
-LABEL io.parity.image.authors="devops-team@parity.io" \
-	io.parity.image.vendor="Parity Technologies" \
-	io.parity.image.title="${IMAGE_NAME}" \
-	io.parity.image.description="${IMAGE_NAME} for substrate based chains" \
-	io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/scripts/ci/dockerfiles/${IMAGE_NAME}/${IMAGE_NAME}_injected.Dockerfile" \
-	io.parity.image.revision="${VCS_REF}" \
-	io.parity.image.created="${BUILD_DATE}" \
-	io.parity.image.documentation="https://github.com/paritytech/polkadot/"
-
-# show backtraces
-ENV RUST_BACKTRACE 1
-
-# install tools and dependencies
-RUN apt-get update && \
-	DEBIAN_FRONTEND=noninteractive apt-get install -y \
-		libssl1.1 \
-		ca-certificates && \
-# apt cleanup
-	apt-get autoremove -y && \
-	apt-get clean && \
-	find /var/lib/apt/lists/ -type f -not -name lock -delete; \
-	useradd -u 1000 -U -s /bin/sh miner
-
-# add binary to docker image
-COPY ./staking-miner /usr/local/bin
-
-USER miner
-
-ENV SEED=""
-ENV URI="wss://rpc.polkadot.io"
-ENV RUST_LOG="info"
-
-# check if the binary works in this container
-RUN /usr/local/bin/staking-miner --version
-
-ENTRYPOINT [ "/usr/local/bin/staking-miner" ]

polkadot/scripts/ci/dockerfiles/staking-miner/test-build.sh

+#!/usr/bin/env bash
+
+TMP=$(mktemp -d)
+ENGINE=${ENGINE:-podman}
+
+# You need to build an injected image first
+
+# Fetch some binaries
+$ENGINE run --user root --rm -i \
+  -v "$TMP:/export" \
+  --entrypoint /bin/bash \
+  parity/staking-miner -c \
+  'cp "$(which staking-miner)" /export'
+
+echo "Checking binaries we got:"
+tree $TMP
+
+./build-injected.sh $TMP

polkadot/scripts/ci/gitlab/pipeline/build.yml

@@ -39,7 +39,6 @@ build-linux-stable:
     - echo -n ${CI_JOB_ID} > ./artifacts/BUILD_LINUX_JOB_ID
     - RELEASE_VERSION=$(./artifacts/polkadot -V | awk '{print $2}'| awk -F "-" '{print $1}')
     - echo -n "v${RELEASE_VERSION}" > ./artifacts/BUILD_RELEASE_VERSION
-    - cp -r scripts/* ./artifacts
 
 build-test-collators:
   stage: build
@@ -64,7 +63,6 @@ build-test-collators:
     - echo -n "${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHORT_SHA}" > ./artifacts/EXTRATAG
     - echo "adder-collator version = $(cat ./artifacts/VERSION) (EXTRATAG = $(cat ./artifacts/EXTRATAG))"
     - echo "undying-collator version = $(cat ./artifacts/VERSION) (EXTRATAG = $(cat ./artifacts/EXTRATAG))"
-    - cp -r ./scripts/* ./artifacts
 
 build-malus:
   stage: build
@@ -88,7 +86,6 @@ build-malus:
     - echo -n "${CI_COMMIT_REF_NAME}" > ./artifacts/VERSION
     - echo -n "${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHORT_SHA}" > ./artifacts/EXTRATAG
     - echo "polkadot-test-malus = $(cat ./artifacts/VERSION) (EXTRATAG = $(cat ./artifacts/EXTRATAG))"
-    - cp -r ./scripts/* ./artifacts
 
 build-staking-miner:
   stage: build
@@ -110,7 +107,6 @@ build-staking-miner:
     - echo -n "${CI_COMMIT_REF_NAME}" > ./artifacts/VERSION
     - echo -n "${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHORT_SHA}" > ./artifacts/EXTRATAG
     - echo "staking-miner = $(cat ./artifacts/VERSION) (EXTRATAG = $(cat ./artifacts/EXTRATAG))"
-    - cp -r ./scripts/* ./artifacts
 
 build-rustdoc:
   stage: build

polkadot/scripts/ci/gitlab/pipeline/publish.yml

@@ -19,20 +19,16 @@ publish-polkadot-debug-image:
     - if: $CI_COMMIT_REF_NAME =~ /^[0-9]+$/ # PRs
     - if: $CI_COMMIT_REF_NAME =~ /^v[0-9]+\.[0-9]+.*$/ # i.e. v1.0, v2.1rc1
   variables:
-    CI_IMAGE: ${BUILDAH_IMAGE}
-    GIT_STRATEGY: none
-    DOCKER_USER: ${PARITYPR_USER}
-    DOCKER_PASS: ${PARITYPR_PASS}
-    # scripts/ci/dockerfiles/polkadot_injected_debug.Dockerfile
-    DOCKERFILE: ci/dockerfiles/polkadot_injected_debug.Dockerfile
-    IMAGE_NAME: docker.io/paritypr/polkadot-debug
+    IMAGE_NAME: "polkadot-debug"
+    BINARY: "polkadot,polkadot-execute-worker,polkadot-prepare-worker"
   needs:
     - job: build-linux-stable
       artifacts: true
   after_script:
+    - !reference [.build-push-image, after_script]
     # pass artifacts to the zombienet-tests job
     # https://docs.gitlab.com/ee/ci/multi_project_pipelines.html#with-variable-inheritance
-    - echo "PARACHAINS_IMAGE_NAME=${IMAGE_NAME}" > ./artifacts/parachains.env
+    - echo "PARACHAINS_IMAGE_NAME=${IMAGE}" > ./artifacts/parachains.env
     - echo "PARACHAINS_IMAGE_TAG=$(cat ./artifacts/EXTRATAG)" >> ./artifacts/parachains.env
   artifacts:
     reports:
@@ -48,20 +44,15 @@ publish-test-collators-image:
     - .build-push-image
     - .zombienet-refs
   variables:
-    CI_IMAGE: ${BUILDAH_IMAGE}
-    GIT_STRATEGY: none
-    DOCKER_USER: ${PARITYPR_USER}
-    DOCKER_PASS: ${PARITYPR_PASS}
-    # scripts/ci/dockerfiles/collator_injected.Dockerfile
-    DOCKERFILE: ci/dockerfiles/collator_injected.Dockerfile
-    IMAGE_NAME: docker.io/paritypr/colander
+    IMAGE_NAME: "colander"
+    BINARY: "adder-collator,undying-collator"
   needs:
     - job: build-test-collators
       artifacts: true
   after_script:
-    - buildah logout --all
+    - !reference [.build-push-image, after_script]
     # pass artifacts to the zombienet-tests job
-    - echo "COLLATOR_IMAGE_NAME=${IMAGE_NAME}" > ./artifacts/collator.env
+    - echo "COLLATOR_IMAGE_NAME=${IMAGE}" > ./artifacts/collator.env
     - echo "COLLATOR_IMAGE_TAG=$(cat ./artifacts/EXTRATAG)" >> ./artifacts/collator.env
   artifacts:
     reports:
@@ -76,20 +67,15 @@ publish-malus-image:
     - .build-push-image
     - .zombienet-refs
   variables:
-    CI_IMAGE: ${BUILDAH_IMAGE}
-    GIT_STRATEGY: none
-    DOCKER_USER: ${PARITYPR_USER}
-    DOCKER_PASS: ${PARITYPR_PASS}
-    # scripts/ci/dockerfiles/malus_injected.Dockerfile
-    DOCKERFILE: ci/dockerfiles/malus_injected.Dockerfile
-    IMAGE_NAME: docker.io/paritypr/malus
+    IMAGE_NAME: "malus"
+    BINARY: "malus,polkadot-execute-worker,polkadot-prepare-worker"
   needs:
     - job: build-malus
       artifacts: true
   after_script:
-    - buildah logout "$IMAGE_NAME"
+    - !reference [.build-push-image, after_script]
     # pass artifacts to the zombienet-tests job
-    - echo "MALUS_IMAGE_NAME=${IMAGE_NAME}" > ./artifacts/malus.env
+    - echo "MALUS_IMAGE_NAME=${IMAGE}" > ./artifacts/malus.env
     - echo "MALUS_IMAGE_TAG=$(cat ./artifacts/EXTRATAG)" >> ./artifacts/malus.env
   artifacts:
     reports:
@@ -103,13 +89,11 @@ publish-staking-miner-image:
     - .build-push-image
     - .publish-refs
   variables:
-    CI_IMAGE: ${BUILDAH_IMAGE}
-    # scripts/ci/dockerfiles/staking-miner/staking-miner_injected.Dockerfile
-    DOCKERFILE: ci/dockerfiles/staking-miner/staking-miner_injected.Dockerfile
-    IMAGE_NAME: docker.io/paritytech/staking-miner
-    GIT_STRATEGY: none
-    DOCKER_USER: ${Docker_Hub_User_Parity}
-    DOCKER_PASS: ${Docker_Hub_Pass_Parity}
+    IMAGE_NAME: "staking-miner"
+    BINARY: "staking-miner"
+    DOCKER_OWNER: "paritytech"
+    DOCKER_USER: "${Docker_Hub_User_Parity}"
+    DOCKER_PASS: "${Docker_Hub_Pass_Parity}"
   needs:
     - job: build-staking-miner
       artifacts: true
@@ -122,11 +106,11 @@ publish-polkadot-image-description:
     DOCKER_PASSWORD: ${Docker_Hub_Pass_Parity}
     DOCKERHUB_REPOSITORY: parity/polkadot
     SHORT_DESCRIPTION: "Polkadot Official Docker Image"
-    README_FILEPATH: $CI_PROJECT_DIR/scripts/ci/dockerfiles/polkadot_Dockerfile.README.md
+    README_FILEPATH: $CI_PROJECT_DIR/scripts/ci/dockerfiles/polkadot/polkadot_Dockerfile.README.md
   rules:
     - if: $CI_COMMIT_REF_NAME == "master"
       changes:
-        - scripts/ci/dockerfiles/polkadot_Dockerfile.README.md
+        - scripts/ci/dockerfiles/polkadot/polkadot_Dockerfile.README.md
     - if: $CI_PIPELINE_SOURCE == "schedule"
       when: never
   script:

polkadot/utils/staking-miner/README.md

@@ -28,8 +28,9 @@ There are 2 options to build a staking-miner Docker image:
 ### Building the injected image
 
 First build the binary as documented [above](#building).
-You may then inject the binary into a Docker base image from the root of the Polkadot repository:
+You may then inject the binary into a Docker base image: `parity/base-bin` (running the command from the root of the Polkadot repository):
 ```
+TODO: UPDATE THAT
 docker build -t staking-miner -f scripts/ci/dockerfiles/staking-miner/staking-miner_injected.Dockerfile target/release
 ```
 
@@ -39,6 +40,7 @@ Unlike the injected image that requires a Linux pre-built binary, this option do
 The trade-off however is that it takes a little longer to build and this option is less ideal for CI tasks.
 You may build the multi-stage image the root of the Polkadot repository with:
 ```
+TODO: UPDATE THAT
 docker build -t staking-miner -f scripts/ci/dockerfiles/staking-miner/staking-miner_builder.Dockerfile .
 ```
 
@@ -51,7 +53,7 @@ While it won't prevent a malicious actor to read your `SEED` if they gain access
 # The following line starts with an extra space on purpose:
  SEED=0x1234...
 
-docker run --rm -it \
+docker run --rm -i \
     --name staking-miner \
     --read-only \
     -e RUST_LOG=info \