Develop a respectful and privacy focused data collection framework that allows us to make informed product decisions and improve the customers experience, and promote conversion, adoption of key stages, and increase our user base.
The Growth sub-department consists of groups that eliminate barriers between our users and our product value.
Deliver product analytics data that improves our product
The Product Analytics Group is responsible for:
Collection
Analysis
Drive value for the business and our users by improving trial to paid conversion, per-stage adoption, and wider usage of GitLab
We focus on validating ideas with data across the following four Groups:
The Growth groups also provide development input into two Growth product areas of focus:
We work closely with the Data Team along with our Product Team counterparts to design and implement experiments that measure the impact of changes to our messaging, UX, and overall experience of using GitLab.
The Growth sub-department uses the ~"devops::growth" label and the following groups for tracking throughput and ownership of issues and merge requests.
| Group name | Group label |
|---|---|
| Product Analytics | ~"group::product analytics" |
| Activation | ~"group::activation" |
| Adoption | ~"group::adoption" |
| Conversion | ~"group::conversion" |
| Expansion | ~"group::expansion" |
| Person | Role |
|---|---|
| Wayne Haber | Director of Engineering, Threat Management and Growth |
| Jerome Ng | Backend Engineering Manager, Product Analytics |
| Phil Calder | Backend Engineering Manager, Growth:Activation, Adoption, Conversion and Expansion |
The following people are permanent members of groups that belong to the Growth sub-department:
| Person | Role |
|---|---|
| Phil Calder | Backend Engineering Manager, Growth:Activation, Adoption, Conversion and Expansion |
| Alex Buijs | Senior Fullstack Engineer, Growth:Activation |
| Nicolas Dular | Senior Fullstack Engineer, Growth:Activation |
| Person | Role |
|---|---|
| Phil Calder | Backend Engineering Manager, Growth:Activation, Adoption, Conversion and Expansion |
| Jeremy Jackson | Senior Fullstack Engineer, Growth:Adoption |
| Jay Swain | Senior Fullstack Engineer, Growth:Adoption |
| Person | Role |
|---|---|
| Phil Calder | Backend Engineering Manager, Growth:Activation, Adoption, Conversion and Expansion |
| Alper Akgun | Senior Fullstack Engineer, Growth:Conversion |
| Dallas Reedy | Fullstack Engineer, Growth:Conversion |
| Person | Role |
|---|---|
| Phil Calder | Backend Engineering Manager, Growth:Activation, Adoption, Conversion and Expansion |
| Jackie Fraser | Fullstack Engineer, Growth:Expansion |
| Doug Stull | Senior Fullstack Engineer, Growth:Expansion |
| Person | Role |
|---|---|
| Alishan 'Ali' Ladhani | Backend Engineer, Product Analytics |
| Mikołaj Wawrzyniak | Backend Engineer, Product Analytics |
| Jerome Ng | Backend Engineering Manager, Product Analytics |
| Alina Mihaila | Backend Engineer, Product Analytics |
The following table shows who will provide cover if one or more of the Growth Engineering management team are unable to work for any reason.
| Team Member | Coverered by | Escalation |
|---|---|---|
| Wayne Haber | Christopher Lefelhocz | Eric Johnson |
| Jerome Ng | Phil Calder | Wayne Haber |
| Phil Calder | Jerome Ng | Wayne Haber |
If an Engineer is unavailable the Engineering Manager will reassign open issues and merge requests to another engineer, preferably in the same group.
Some people management functions may require escalation or delegation, such as BambooHR and Expensify.
This can be used as the basis for a business continuity plan (BCP), as well as a general guide to Growth Engineering continuity in the event of one or more team members being unavailable for any reason.
The following members of other functional teams are our stable counterparts:
| Person | Role |
|---|---|
| Michael Karampalas | Principal Product Manager, Growth:Adoption |
| Amanda Rueda | Product Manager, Growth:Fulfillment |
| Tim Hey | Principal Product Manager, Growth:Expansion |
| Keanon O'Keefe | Senior Product Manager, Growth:Product Analytics |
| Jensen Stava | Senior Product Manager, Growth:Activation |
| Sam Awezec | Senior Product Manager, Growth:Conversion |
| Matej Latin | Senior Product Designer, Growth:Expansion and Adoption |
| Kevin Comoli | Product Designer, Growth:Activation and Conversion |
| Russell Dickenson | Senior Technical Writer, Secure (Static Analysis, Dynamic Analysis, Composition Analysis, Threat Insights, Vulnerability Research), Growth (Fulfillment) |
| Marcel Amirault | Technical Writer, Verify (Continuous Integration, Pipeline Authoring, Testing), Growth (Adoption) |
| Mike Jang | Senior Technical Writer, Manage, Growth (Product Analytics) |
| Amy Qualls | Senior Technical Writer, Configure, Monitor, Growth (Activation, Conversion) |
| Suzanne Selhorn | Senior Technical Writer, Package, Verify (Runner), Growth (Expansion) |
| Alishan 'Ali' Ladhani | Backend Engineer, Product Analytics |
| Mikołaj Wawrzyniak | Backend Engineer, Product Analytics |
| Jerome Ng | Backend Engineering Manager, Product Analytics |
| Alina Mihaila | Backend Engineer, Product Analytics |
| Vincy Wilson | Quality Engineering Manager, Growth, Fulfillment & Protect |
| Jacki Bauer | Product Design Manager, Growth & Enablement |
| Jeff Crow | Senior UX Researcher, Growth |
| Hila Qu | Director of Product, Growth |
As part of the wider Growth stage we track and work on issues with the label ~"devops::growth".
Our team follows the Product Development Flow utilizing all labels from ~workflow::start to ~workflow::verification.
We adhere to the Completion Criteria and Who Transitions Out outlined in the Product Development Flow to progress issues from one stage to the next.
We use workflow boards to track issue progress throughout a milestone. Workflow boards should be viewed at the highest group level for visibility into all nested projects in a group.
There are three GitLab groups we use:
| gitlab-org | gitlab-com | gitlab-services | all groups |
|---|---|---|---|
| Growth Workflow | - | Growth Workflow | - |
| Activation Workflow | - | Activation Workflow | - |
| Conversion Workflow | - | Conversion Workflow | - |
| Expansion Workflow | - | Expansion Workflow | - |
| Adoption Workflow | - | Adoption Workflow | - |
| Product Analytics Workflow | - | Product Analytics Workflow | - |
Before work can begin on an issue, we should estimate it first after a preliminary investigation. This is normally done in the monthly planning meeting.
| Weight | Description (Engineering) |
|---|---|
| 1 | The simplest possible change. We are confident there will be no side effects. |
| 2 | A simple change (minimal code changes), where we understand all of the requirements. |
| 3 | A simple change, but the code footprint is bigger (e.g. lots of different files, or tests effected). The requirements are clear. |
| 5 | A more complex change that will impact multiple areas of the codebase, there may also be some refactoring involved. Requirements are understood but you feel there are likely to be some gaps along the way. |
| 8 | A complex change, that will involve much of the codebase or will require lots of input from others to determine the requirements. |
| 13 | A significant change that may have dependencies (other teams or third-parties) and we likely still don't understand all of the requirements. It's unlikely we would commit to this in a milestone, and the preference would be to further clarify requirements and/or break in to smaller Issues. |
In planning and estimation, we value velocity over predictability. The main goal of our planning and estimation is to focus on the MVC, uncover blind spots, and help us achieve a baseline level of predictability without over optimizing. We aim for 70% predictability instead of 90%. We believe that optimizing for velocity (MR throughput) enables our Growth teams to achieve a weekly experimentation cadence.
Our work is planned and delivered on a monthly cycle using milestones. Our team follows the Product Development Timeline utilizing all dates including from M-1, 4th: Draft of the issues to M+1, 4th: Public Retrospective.
We use milestone boards for high level planning and roadmapping across several milestones.
The Growth UX team has a handbook page which includes Growth specific workflows.
To help our team be efficient, we explicitly define how our team uses issues.
We aim to create issues in the same project as where the future merge request will live. For example, if an experiment is being run in the GitLab CustomersDot, both the issue and MR should be created in the CustomersDot project.
We emphasize creating the issue in the right project to avoid having to close and move issues later in the development process. If the location of the future merge request cannot be determined, we will create the issue in our catch-all growth team-tasks project.
To support Iteration Growth engineering:
5 and higher should be reassigned to the Product Manager to make sure they can be split into smaller MVCs.
When this is not possible, the Product Manager will create a spike or research issue so that engineering can break it down and close the original.In an effort to prevent accidental outages to business critical processes, whenever we touch code which may have a direct or indirect impact on data flowing into Salesforce.com, we should assign @jbrennan1 to the MR as a reviewer and add the ~"Affects Salesforce" label.
We follow a four step process for running experiments as outlined by Andrew Chen's How to build a growth team.
Each week, we provide progress updates and talk about our learnings in our Growth Weekly Meeting.
The duration of each experiment will vary depending on how long it takes for experiment results to reach statistical significance. Due to the varying duration, there will be some weeks where we have several experiments running concurrently in parallel.
~"growth experiment", ~"experiment idea")
%Awaiting further demand, %Backlog, %Next 1-3 releases, or a specific milestone)workflow:: issues linked to the epic to track the work required to complete the experiment~"experiment::active" when the experiment is live#production Slack channelSee also the Growth RADCIE and DRIs for determining DRIs at each stage.
A backlog of experiments are tracked on the Experiment backlog board.
To track the status of an Experiment, Experiment tracking issues using the ~"experiment tracking" and scoped experiment:: labels are tracked on Experiments tracking boards across the following groups:
| gitlab-org | gitlab-com | all groups |
|---|---|---|
| Experiment tracking | Experiment tracking | Issues List |
gitlab-org group)
This issue acts as the starting point for defining an experiment, including an overview of the experiment, the hypothesis, and some idea of how success will be measured. This issue will be tagged with the ~"growth experiment" and ~"experiment idea" labels. The Growth Experiment issue template can be used for this.
This epic acts as the single source of truth (SSoT) for the experiment once an experiment has been properly defined according to our Experiment Definition Standards and is deemed worthwhile to run. Once an Experiment Definition Issue is added to this epic, we fill out further details such as the expected rollout plan. We also assign the experiment to a milestone and follow the product development flow for UX & Engineering work. As the experiment design and rollout progresses, this epic or parent issue should contain details or links to further information about experiment including the tracking events and data points used to determine if the experiment is a success as well as links to relevant metrics-reporting dashboards (such as Sisense).
This issue is used to track the experiment progress once deployed. It is similar to a Feature Flag Roll Out issue with an additional experiment:: scoped label to track the status of the experiment. The Experiment Tracking issue template includes an overview of the experiment, who to contact, rollout steps, and a link to the overall experiment issue or epic.
The experiment:: scoped labels are:
~"experiment::pending" - The experiment is waiting to be deployed~"experiment::active" - The experiment is active (live)~"experiment::blocked" - The experiment is currently blocked~"experiment::validated" - The experiment has been validated (the success criteria was clearly met)~"experiment::invalidated" - The experiment has been invalidated (the success criteria was clearly unmet)~"experiment::inconclusive" - The experiment was inconclusive (the success criteria was not clearly met nor clearly unmet)This issue is used to clean up an experiment after an experiment has been completed. It is created within the project where the cleanup work will be done (e.g. the gitlab-org/gitlab project). The cleanup work may include completely removing the experiment (in the case of ~"experiment::invalidated" and ~"experiment::inconclusive") or refactoring the experiment feature for the long run (in the case of ~"experiment::validated"). The cleanup issue should be linked to the Experiment Tracking issue as a reference to ensure the experiment is concluded prior to cleanup.
Every week, engineers in the Growth sub-department meet to discuss topics related to growth engineering. Discussion topics include how to track experiments, A/B testing, changes in CustomersDot, changes in gitlab application, etc. Growth Engineers are encouraged to bring discussion topics to the meeting and to them to the agenda.
To get the most time zone coverage, these meetings alternate fortnightly between:
Team members are encouraged to attend the meeting that matches their time zone.
On occasion we put together a virtual team day to help take a break and participate in fun, social activities across Engineering, Product, UX, Data and Quality.
#s_growth in Slack (GitLab internal)#g_product_analytics in Slack (GitLab internal)