An A/B test is when we release two versions of a page and compare how well they perform (comparing version A to version B). When running an experiment, we are testing a hypothesis using a control variant and a test variant, similar to how one typically employs the scientific method.
We currently use LaunchDarkly to control whether or not a test is showing, at what percentage, and gather metrics about a test's performance. Within LaunchDarkly, you can create events that fire when a user does something. For our case, our most common example would be a click. We also push the experiment ID into the Google Analytics dataLayer so we know what version of the page the user viewed.
window.dataLayer = window.dataLayer || [];
dataLayer.push({
'event': 'launchDarklyExperiment',
'launchDarklyExperimentName': 'name of experiment',
'launchDarklyExperimentId': '0 or 1' //0 = control, 1 = variant
});
This project is for the ideation and refinement of AB Tests to be conducted on GitLab's Marketing site. Anyone can contribute an AB testing issue.
Below are some resources to learn more about feature flags. At a high level, a feature flag is an if-else wrapper around code that can be enabled, disabled, or served at a certain percentage to a certain group. This is controlled via a dashboard toggle, changing the production interface on-the-fly without having to wait for a release to change something.
Our AB tests include two files, the control and the test variant. Both exist on the page in the HTML DOM at the same time, but are hidden by default on page load. The javascript SDK will return which version of the experiment should be shown. For each test, we use the following process:
| Issue | Test Length | Status |
|---|---|---|
| Homepage Featured Blocks vs Carousel | scheduled |
www repositoryThis can be overridden by optional URL parameters as exhibited in the codepaths section below.
Example merge request for an AB test
Active
run-experiments.js from www: https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/source/javascripts/run-experiment.jsBecause the marketing website about.gitlab has no dynamic server, we needed a solution that could be performant and implemented using a javascript SDK. In addition to that, we needed a solution that was able to attach metrics.
Below are some links with information on the history of the decision: