Measure website performance in Gitlab CI5 min read

You are a developer and all the important steps to test and rollout your app are already in your Gitlab CI, except of one: website performance testing is missing.

This article will explain to you how you can use VitalFrog to within your Gitlab CI and utilize our lab infrastructure to test your website performance directly in your Gitlab CI.

Make sure you know the basics about website performance and Core Web Vitals as this phrases will be used throughout the article and are heavily utilized within VitalFrog itself.

We will be utilizing the VitalFrog Open Source go client (as docker image), so if you want to be 100% sure that you know about all possible config options, checkout the config in the client. We will be using the minimal necessary setup in this tutorial in order to get you started as fast as possible.

Gitlab CI YAML to run performance tests

The following YAML is all you need to run a basic website performance test in your Gitlab CI. If you know what you are doing, just go ahead and copy it into your .gitlab-ci.yml . If not, no worries, we will explain it further down in this article.

Keep in mind that you need to set the environment variable $VITALFROG_API_TOKEN to your VitalFrog token. If you not yet have one, Join the Beta List to one and get started for free! (First 1000 tokens are on us)

  - performance_test

    entrypoint: [""]
  script: ["/cli"]

  stage: performance_test
    TARGET_HOST: "" 
    TARGET_PATHS: "/,/blog/,/blog/measure-website-performance-in-gitlab-ci/"

So let’s run through what this code snippet is doing. Actually everything execpt variables is quite standard and the Gitlab Wiki is the best resources for you to learn about the general syntax of the .gitlab-ci.yaml.

The variables are the part where it actually gets interesting: With them the VitalFrog CI step is configured.

  • API_TOKEN : This is the VitalFrog API token you need for accessing the API at all. (If you not yet have one, Join the Beta List and get the first 1000 Tokens for free.)
  • TARGET_HOST: This is the host of your website. Usuallly the first part of your website url before the first / (after the protocol). So in the url, the host would be and the so call path /search
  • TARGET_PATHS: This is a coma-separated list of paths you want to check on your host. As you learned in the paragraph above a path is the entire path after the host and before any query parameters (which start with ?). In the url, the path would be /search. In this parameter add all the paths of your website you want to check for their performance. At a later stage, VitalFrog will have path auto discovery as well.
  • VERSION: Configures the version we are currently testing. This is great debug information later if you need to find out what change introduced a bug. We use the git commit in this example. If you have concrete version numbering, use that.

This is the basic you need to have VitalFrog test your site. Congratulations, you are ready to continuously improve your website performance and core web vitals in Gitlab CI.

If you want to go a step further, let’s dive into other configuration options you have with VitalFrog to further refine your performance testing strategy:

  • ALLOWED_COUNTRIES/BLOCKED_COUNTRIES: VitalFrog allows you to test your website performance from all across the globe. The default is to test on all available countries. With these two parameters, you can configure which countries you actually want to run on. You can only use one of the two:
    • ALLOWED_COUNTRIES: Only allow the specified countries (comma separated list)
    • BlOCKED_COUNTRIES: Allow all countries but the ones in this list (comma separated list)
  • DEVICES: The devices you want to test on. If not set falls back to test on the devices mobile,desktop
  • TARGET_SCHEMA: Configure what schema (http|https) to use on target host. https is the default.
  • COMPONENT_NAME: If you use VitalFrog in more than one repo, we highly advise you to configure the component name, so you can later easily use black box tracing to figure out what component and change across your architecture influenced your website performance
  • BASIC_AUTH_USERNAME & BASIC_AUTH_PASSWORD: If your site (e.g. an internal staging site you want to test your performance in your Gitlab CI for) is behind a basic auth, you can use the following two parameters to configure the credentials. Always both have to be set in common
  • EXTRA_HEADERS: With this parameter, you can set additional headers you want to send with the HTTP request to your website. Mostly used for the authorization header. Format is header1=header1;header2=value2 (semicolon separated lists of key=value)
  • RUN_ASYNC: Configures to not wait for any results of the report and just kick it off. Good if you want your Gitlab CI to continue without the wait for the VitalFrog website performance report to finish. (Attention: boolean values need to be escaped in YAML to work as env vars. You would use it like following in the YAML: RUN_ASYNC: "true")

This is all for now that you can configure on the VitalFrog Gitlab CI website performance testing CI step. If you have any further questions, do not hesitate to reach out to [email protected]!