Experimentation allows us to learn and give the right experiences to our Customers, to create better value for Customers and GitLab. Although experimentation seems straightforward, the risk of making inaccurate conclusions is too high, if an organization does not follow best practices.
For example, here are a couple of commonly made mis-steps while conducting experiments which can lead to inaccurate conclusions and decisions:
To ensure we can maximize the value from our experimentation practices and reduce inaccuracy of decisions at GitLab, we recommend following best practices across all experiments at GitLab. This document outlines the best practices to adopt at Gitlab.
Before launching an experiment:
General Guidance for Test Group Sample Allocation Based on Risk and Critical Path
Experiment launch weights matrix on failure risk, and importance of feature and impacted population | Critical path page or critical product feature | |||
L | M | H | ||
Potential code risk for failure | L | 50% |
10% |
5% |
M | 10% |
10% |
5% |
|
H | 10% |
5% |
1% |
Note: Short term - means hours/days worth of effort. Long term means weeks/months worth of effort
Based on initial feedback and observations on current experiment platform capabilities, these are a list of action items I recommend we consider. It still needs to be consulted with impacted parties to ensure the gaps still exist and their priority.