Develop a respectful and privacy focused data collection framework that allows us to make informed product decisions and improve the customers experience, and promote conversion, adoption of key stages, and increase our user base.
The Growth sub-department consists of groups that eliminate barriers between our users and our product value.
Deliver product intelligence data to improve the GitLab product
Product Intelligence Group: Product Intelligence focuses on providing GitLab's team with data-driven product insights to build a better GitLab. To do this, we build data collection and analytics tools within the GitLab product in a privacy-focused manner. Insights generated from Product Intelligence enables us to identify the best places to invest people and resources, what product categories mature faster, where our user experience can be improved, and how product changes impact the business. You can learn more about what we're building next on the Product Intelligence Direction page.
Drive value for the business and our users by improving trial to paid conversion, per-stage adoption, and wider usage of GitLab
We focus on validating ideas with data across the following four Groups:
The Growth groups also provide development input into two Growth product areas of focus:
We work closely with the Data Team along with our Product Team counterparts to design and implement experiments that measure the impact of changes to our messaging, UX, and overall experience of using GitLab.
The Growth sub-department uses the ~"devops::growth" label and the following groups for tracking merge request rate and ownership of issues and merge requests.
| Group name | Group label |
|---|---|
| Product Intelligence | ~"group::product intelligence" |
| Activation | ~"group::activation" |
| Adoption | ~"group::adoption" |
| Conversion | ~"group::conversion" |
| Expansion | ~"group::expansion" |
| Person | Role |
|---|---|
| Wayne Haber | Director of Engineering, Growth and Fulfillment |
| Phil Calder | Fullstack Engineering Manager, Growth:Activation, Adoption, Conversion and Expansion |
| Jerome Ng | Fullstack Engineering Manager, Product Intelligence. Strategy and Operations Lead, China (Interim) |
The following people are permanent members of groups that belong to the Growth sub-department:
| Person | Role |
|---|---|
| Alex Buijs | Senior Fullstack Engineer, Growth:Activation |
| Nicolas Dular | Senior Fullstack Engineer, Growth:Activation |
| Person | Role |
|---|---|
| Jeremy Jackson | Senior Fullstack Engineer, Growth:Adoption |
| Jay Swain | Senior Fullstack Engineer, Growth:Adoption |
| Person | Role |
|---|---|
| Alper Akgun | Staff Fullstack Engineer, Growth:Conversion |
| Dallas Reedy | Fullstack Engineer, Growth:Conversion |
| Person | Role |
|---|---|
| Doug Stull | Senior Fullstack Engineer, Growth:Expansion |
| Jackie Fraser | Fullstack Engineer, Growth:Expansion |
| Person | Role |
|---|---|
| Alina Mihaila | Backend Engineer, Product Intelligence |
| Alishan 'Ali' Ladhani | Backend Engineer, Product Intelligence |
| Mikołaj Wawrzyniak | Backend Engineer, Product Intelligence |
| Jerome Ng | Fullstack Engineering Manager, Product Intelligence. Strategy and Operations Lead, China (Interim) |
| Piotr Skorupa | Backend Engineer, Product Intelligence |
| Luis Mejia | Backend Engineer, Product Intelligence |
| Rajendra Kadam | Backend Engineer, Product Intelligence |
| Person | Role |
|---|---|
| Jerome Ng | Fullstack Engineering Manager, Product Intelligence. Strategy and Operations Lead, China (Interim) |
| Axel García | Frontend Engineer, Product Intelligence |
| New Vacancy | Backend Engineer, Product Intelligence |
The following table shows who will provide cover if one or more of the Growth Engineering management team are unable to work for any reason.
| Team Member | Coverered by | Escalation |
|---|---|---|
| Wayne Haber | Christopher Lefelhocz | Eric Johnson |
| Jerome Ng | Phil Calder | Wayne Haber |
| Phil Calder | Jerome Ng | Wayne Haber |
If an Engineer is unavailable the Engineering Manager will reassign open issues and merge requests to another engineer, preferably in the same group.
Some people management functions may require escalation or delegation, such as BambooHR and Expensify.
This can be used as the basis for a business continuity plan (BCP), as well as a general guide to Growth Engineering continuity in the event of one or more team members being unavailable for any reason.
The following members of other functional teams are our stable counterparts:
| Person | Role |
|---|---|
| Jensen Stava | Senior Product Manager, Growth:Activation |
| Sam Awezec | Product Manager, Growth:Conversion and Growth:Expansion (Acting) |
| Sam Awezec | Senior Product Manager, Growth:Conversion |
| Keanon O'Keefe | Senior Product Manager, Growth:Product Intelligence |
| Michael Karampalas | Principal Product Manager, Growth:Adoption, Growth:Activation (acting) |
| Matej Latin | Senior Product Designer, Growth:Expansion and Adoption |
| Kevin Comoli | Product Designer, Growth:Conversion |
| Emily Bauman | Product Designer, Growth:Activation |
| Amy Qualls | Senior Technical Writer, Create (Source Code, Code Review, Editor, Ecosystem), Growth (Activation, Adoption, Conversion, Expansion, Product Intelligence) |
| Andrew Kelly | Senior Security Engineer, Application Security, Growth (Activation, Conversion, Expansion, Adoption), Fulfillment (Purchase, License, Utilization), Enablement (Distribution, Geo, Memory, Global Search, Database) |
| Wayne Haber | Director of Engineering, Growth and Fulfillment |
| Vincy Wilson | Quality Engineering Manager, Growth, Fulfillment & Protect |
| Madison Taft | Team Lead, Sales Development Representative, AMER - Enterprise, Growth |
| Jacki Bauer | Product Design Manager, Growth & Enablement |
| Hila Qu | Director of Product, Growth |
As part of the wider Growth stage we track and work on issues with the label ~"devops::growth".
Our team follows the Product Development Flow utilizing all labels from ~workflow::start to ~workflow::verification.
We adhere to the Completion Criteria and Who Transitions Out outlined in the Product Development Flow to progress issues from one stage to the next.
We use workflow boards to track issue progress throughout a milestone. Workflow boards should be viewed at the highest group level for visibility into all nested projects in a group.
There are three GitLab groups we use:
| gitlab-org | gitlab-com | gitlab-services | all groups |
|---|---|---|---|
| Growth Workflow | - | Growth Workflow | - |
| Activation Workflow | - | Activation Workflow | - |
| Conversion Workflow | - | Conversion Workflow | - |
| Expansion Workflow | - | Expansion Workflow | - |
| Adoption Workflow | - | Adoption Workflow | - |
| Product Intelligence Workflow | - | Product Intelligence Workflow | - |
Before work can begin on an issue, we should estimate it first after a preliminary investigation. This is normally done in the monthly planning meeting.
| Weight | Description (Engineering) |
|---|---|
| 1 | The simplest possible change. We are confident there will be no side effects. |
| 2 | A simple change (minimal code changes), where we understand all of the requirements. |
| 3 | A simple change, but the code footprint is bigger (e.g. lots of different files, or tests effected). The requirements are clear. |
| 5 | A more complex change that will impact multiple areas of the codebase, there may also be some refactoring involved. Requirements are understood but you feel there are likely to be some gaps along the way. |
| 8 | A complex change, that will involve much of the codebase or will require lots of input from others to determine the requirements. |
| 13 | A significant change that may have dependencies (other teams or third-parties) and we likely still don't understand all of the requirements. It's unlikely we would commit to this in a milestone, and the preference would be to further clarify requirements and/or break in to smaller Issues. |
In planning and estimation, we value velocity over predictability. The main goal of our planning and estimation is to focus on the MVC, uncover blind spots, and help us achieve a baseline level of predictability without over optimizing. We aim for 70% predictability instead of 90%. We believe that optimizing for velocity (merge request rate) enables our Growth teams to achieve a weekly experimentation cadence.
Our work is planned and delivered on a monthly cycle using milestones. Our team follows the Product Development Timeline utilizing all dates including from M-1, 4th: Draft of the issues to M+1, 4th: Public Retrospective.
We use milestone boards for high level planning and roadmapping across several milestones.
The Growth UX team has a handbook page which includes Growth specific workflows.
To help our team be efficient, we explicitly define how our team uses issues.
We aim to create issues in the same project as where the future merge request will live. For example, if an experiment is being run in the GitLab CustomersDot, both the issue and MR should be created in the CustomersDot project.
We emphasize creating the issue in the right project to avoid having to close and move issues later in the development process. If the location of the future merge request cannot be determined, we will create the issue in our catch-all growth team-tasks project.
We use issue templates for common tasks.
To support Iteration Growth engineering:
5 and higher should be reassigned to the Product Manager to make sure they can be split into smaller MVCs.
When this is not possible, the Product Manager will create a spike or research issue so that engineering can break it down and close the original.In an effort to prevent accidental outages to business critical processes, whenever we touch code which may have a direct or indirect impact on data flowing into Salesforce.com, we should assign @jbrennan1 to the MR as a reviewer and add the ~"Affects Salesforce" label.
We follow a four step process for running experiments as outlined by Andrew Chen's How to build a growth team.
Each week, we provide progress updates and talk about our learnings in our Growth Weekly Meeting.
The duration of each experiment will vary depending on how long it takes for experiment results to reach statistical significance. Due to the varying duration, there will be some weeks when we have several experiments running concurrently in parallel.
~"experiment idea", optionally add: ~"growth experiment")
%Awaiting further demand, %Backlog, %Next 1-3 releases, or a specific milestone)workflow:: issues linked to the epic to track the work required to complete the experiment~"experiment::active" when the experiment is live#production Slack channelSee also the Growth RADCIE and DRIs for determining DRIs at each stage.
A backlog of experiments are tracked on the Experiment backlog board.
To track the status of an Experiment, Experiment tracking issues using the ~"experiment tracking" and scoped experiment:: labels are tracked on Experiments tracking boards across the following groups:
| gitlab-org | gitlab-com | all groups |
|---|---|---|
| Experiment tracking | Experiment tracking | Issues List |
gitlab-org group)
This issue acts as the starting point for defining an experiment, including an overview of the experiment, the hypothesis, and some idea of how success will be measured. The Experiment idea issue template can be used for this (label added: ~"experiment idea", optionally add: ~"growth experiment").
This epic acts as the single source of truth (SSoT) for the experiment once an experiment has been properly defined according to our Experiment Definition Standards and is deemed worthwhile to run. Once an Experiment Definition Issue is added to this epic, we fill out further details such as the expected rollout plan. We also assign the experiment to a milestone and follow the product development flow for UX & Engineering work. As the experiment design and rollout progresses, this epic or parent issue should contain details or links to further information about experiment including the tracking events and data points used to determine if the experiment is a success as well as links to relevant metrics-reporting dashboards (such as Sisense).
This issue is used to track the experiment progress once deployed. It is similar to a Feature Flag Roll Out issue with an additional experiment:: scoped label to track the status of the experiment. The [Experiment Tracking] issue template includes an overview of the experiment, who to contact, rollout steps, and a link to the overall experiment issue or epic.
The experiment:: scoped labels are:
~"experiment::pending" - The experiment is waiting to be deployed~"experiment::active" - The experiment is active (live)~"experiment::blocked" - The experiment is currently blocked~"experiment::validated" - The experiment has been validated (the success criteria was clearly met)~"experiment::invalidated" - The experiment has been invalidated (the success criteria was clearly unmet)~"experiment::inconclusive" - The experiment was inconclusive (the success criteria was not clearly met nor clearly unmet)This issue is used to clean up an experiment after an experiment has been completed. It is created within the project where the cleanup work will be done (e.g. the gitlab-org/gitlab project). The cleanup work may include completely removing the experiment (in the case of ~"experiment::invalidated" and ~"experiment::inconclusive") or refactoring the experiment feature for the long run (in the case of ~"experiment::validated"). The cleanup issue should be linked to the experiment tracking issue as a reference to ensure the experiment is concluded prior to cleanup.
The Experiment Successful Cleanup issue template can be used for the gitlab-org/gitlab project.
gitlab-org/gitlab project
team-tasks project
The Growth sub-department tracks number of experiments per engineer as a development metric.
Every week, engineers in the Growth sub-department meet to discuss topics related to growth engineering. Discussion topics include how to track experiments, A/B testing, changes in CustomersDot, changes in gitlab application, etc. Growth Engineers are encouraged to bring discussion topics to the meeting and to them to the agenda.
To get the most time zone coverage, these meetings alternate fortnightly between:
Team members are encouraged to attend the meeting that matches their time zone.
On occasion we put together a virtual team day to help take a break and participate in fun, social activities across Engineering, Product, UX, Data and Quality.
#s_growth in Slack (GitLab internal)#g_product_intelligence in Slack (GitLab internal)