What is end-to-end testing?
End-to-end (e2e) testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.
How do we test GitLab?
Testing nightly builds
We run scheduled pipelines each night to test nightly builds created by Omnibus.
You can find these pipelines at https://gitlab.com/gitlab-org/quality/nightly/pipelines
(requires the Developer role). Results are reported in the
#qa-nightly Slack channel.
We run scheduled pipelines each night to test staging.
You can find these pipelines at https://gitlab.com/gitlab-org/quality/staging/pipelines
(requires the Developer role). Results are reported in the
#qa-staging Slack channel.
Testing code in merge requests
Using the package-and-test job
It is possible to run end-to-end tests for a merge request by triggering the
e2e:package-and-test manual action in the
qa stage (not
available for forks).
This runs end-to-end tests against a custom EE (with an Ultimate license) Docker image built from your merge request's changes.
Manual action that starts end-to-end tests is also available
gitlab-org/omnibus-gitlab merge requests.
How does it work?
Currently, we are using multi-project pipeline-like approach to run end-to-end pipelines.
graph TB A1 -.->|once done, can be triggered| A2 A2 -.->|1. Triggers an `omnibus-gitlab-mirror` pipeline<br>and wait for it to be done| B1 B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a `gitlab-qa-mirror` pipeline<br>and wait for it to be done| C1 subgraph "`gitlab-org/gitlab` pipeline" A1[`build-images` stage<br>`build-qa-image` and `build-assets-image` jobs] A2[`qa` stage<br>`e2e:package-and-test` job] end subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline" B1[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end subgraph "`gitlab-org/gitlab-qa-mirror` pipeline" C1>End-to-end jobs run] end
- Developer triggers the
e2e:package-and-testmanual action (available once the
build-assets-imagejobs are done), that can be found in GitLab merge requests. This starts a e2e test child pipeline.
- E2E child pipeline triggers a downstream pipeline in
gitlab-org/build/omnibus-gitlab-mirrorand polls for the resulting status. We call this a status attribution.
- Developer triggers the
- Docker image is being built and pushed to its Container Registry.
- Once Docker images are built and pushed jobs in
teststage are started
You may have noticed that we use
gitlab-org/build/omnibus-gitlab-mirror instead of
This is due to technical limitations in the GitLab permission model: the ability to run a pipeline
against a protected branch is controlled by the ability to push/merge to this branch.
This means that for developers to be able to trigger a pipeline for the default branch in
gitlab-org/omnibus-gitlab, they would need to have the Maintainer role for this project.
For security reasons and implications, we couldn't open up the default branch to all the Developers.
Hence we created this mirror where Developers and Maintainers are allowed to push/merge to the default branch.
This problem was discovered in https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160 and the "mirror"
work-around was suggested in https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717.
A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at https://gitlab.com/gitlab-org/gitlab/-/issues/24585.
With merged results pipelines
In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch.
The end-to-end tests on a merged results pipeline would use the new ref instead of the head of the merge request source branch.
graph LR A["x1y1z1 - master HEAD"] B["d1e1f1 - merged results (CI_COMMIT_SHA)"] A --> B B --> C["Merged results pipeline"] C --> D["E2E tests"]
Running custom tests
The existing scenarios
that run in the downstream
gitlab-qa-mirror pipeline include many tests, but there are times when you might want to run a
test or a group of tests that are different than the groups in any of the existing scenarios.
For example, when we dequarantine
a flaky test we first want to make sure that it's no longer flaky.
We can do that by running
_ee:quarantine manual job.
When selecting the name (not the play icon) of manual job,
you are prompted to enter variables. You can use any of
the variables that can be used with
as well as these:
||The scenario to run (default
||The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example,
||The RSpec tags to add (default
manual jobs with custom variables don't use the same variable when retried,
so if you want to run the same tests multiple times,
specify the same variables in each
custom-parallel job (up to as
many of the 10 available jobs that you want to run).
On every pipeline during the
test stage, the
review-qa-smoke job is
automatically started: it runs the QA smoke suite against the
You can also manually start the
review-qa-all: it runs the full QA suite
against the Review App.
See Review Apps for more details about Review Apps.
Selective test execution
In order to limit amount of tests executed in a merge request, dynamic selection of which tests to execute is present. Algorithm of which tests to run is based on changed files and merge request labels. Following criteria determine which tests will run:
- Changes in
qaframework code would execute the full suite
- Changes in particular
qafolder would execute only that particular test
- Merge request with backend changes and label
devops::managewould execute all e2e tests related to
Overriding selective test execution
To override selective test execution and trigger the full suite, label
pipeline:run-all-e2e should be added to particular merge request.
Run tests in parallel
To run tests in parallel on CI, the Knapsack
gem is used. Knapsack reports are generated automatically and stored in the
knapsack-reports in the
gitlab-qa-resources project. The
helper handles automated report generation and upload.
Provisioning of all components is performed by the
Exporting metrics in CI
Use these environment variables to configure metrics export:
||Should be set to
||InfluxDB write token that can be found under
||Arbitrary name for test execution, like
||Flag to enable or disable metrics export to InfluxDB. Defaults to
||Flag to enable or disable saving metrics as JSON file. Defaults to
For additional test results visibility, tests that run on pipelines generate and host Allure test reports.
QA framework is using the Allure RSpec
gem to generate source files for the
Allure test report. An additional job
in the pipeline:
- Fetches these source files from all test jobs.
- Generates and uploads the report to the
gitlab-qa-allure-reportunder the project
A common CI template for report uploading is stored in
When these tests are executed in the scope of merge requests, the
Allure report is
uploaded to the
GCS bucket and comment is added linking to their respective reports.
Scheduled pipelines for these tests contain a
generate-allure-report job under the
Report stage. They also output
a link to the current test report.
Static report links
Each type of scheduled pipeline generates a static link for the latest test report according to its stage:
How do I run the tests?
If you are not testing code in a merge request, there are two main options for running the tests. If you want to run the existing tests against a live GitLab instance or against a pre-built Docker image, use the GitLab QA orchestrator. See also examples of the test scenarios you can run via the orchestrator.
On the other hand, if you would like to run against a local development GitLab environment, you can use the GitLab Development Kit (GDK). Please refer to the instructions in the QA README and the section below.
Running tests that require special setup
Learn how to perform tests that require special setup or consideration to run on your local environment.
How do I write tests?
In order to write new tests, you first need to learn more about GitLab QA architecture. See the documentation about it.
Once you decided where to put test environment orchestration scenarios and instance-level scenarios, take a look at the GitLab QA README, the GitLab QA orchestrator README, and the already existing instance-level scenarios.
Consider not writing an end-to-end test
We should follow these best practices for end-to-end tests:
- Do not write an end-to-end test if a lower-level feature test exists. End-to-end tests require more work and resources.
- Troubleshooting for end-to-end tests can be more complex as connections to the application under test are not known.
- Beginner's Guide
- Style Guide
- Best Practices
- Testing with feature flags
- RSpec metadata/tags
- Execution context selection