Growth Fast Boot September 2019

1. You are here:
2. Handbook
3. Engineering
4. Development Department
5. Growth Sub-department
6. Growth Fast Boot September 2019

On this page

• Day 1
  • Key Outcomes
    • How we operate as a team
    • How do we run Growth experiments?
    • Expectations for working in another teams’ code base
    • Feature Ownership
  • Deep Dives
    • Acquisition Overview
    • Conversion Overview
    • Expansion Overview
    • Retention Overview
• Day 2
  • Data Deep Dive
  • State of Telemetry
  • A/B Testing Requirements
• Days 3-5
• Fast Boot Outcomes
  • Implemented new team names, KPI’s and objectives.
  • Streamlined team processes
  • Create and deliver issues:

The Acquisition, Expansion, Conversion and Retention groups took part in a Fast Boot in September 2019. The planning issue contains the proposal for Fast Boot, and this page documents our discussions and outcomes.

Day 1

Link to Day 1 Agenda Document

Key Outcomes

How we operate as a team

Section Level

• Monthly Product/Growth metrics review with E-Group (VP Product leads, with support from Growth Director)
• Every 6 weeks Growth Group Conversation (PM + Eng Director lead)
• Weekly engineering meeting
• Weekly ideation & prioritization meeting
• On Monday, check in on experiments to identify which can be concluded, and to collect data for review meeting on Tuesdays

Group Level

• Groups will organize/meet as needed

How do we run Growth experiments?

• Development board labels/process - continue on & follow handbook
• A/B testing process, artifacts, expectations
• Each experiment will set confidence level - 95%-99%
• Test summary written by data analyst in the issue (add a “a/b test complete” label for searching later)

Expectations for working in another teams’ code base

• Emphasize that the point is to gather data, not ship production code
• Challenge: review process too slow. Reviewers vs Maintainers. Only a small number of maintainers.
• Goal: one deliverable picked up per week

Feature Ownership

Fulfillment owns underlying billing, licensing, and transactional system, as well as current portal user experience.

• Acquisition owns the New Signup ARR growth KPI + the new signup UX
• Conversion owns the New Free to Paid Trial ARR growth KPI + the trial UX
• Expansion owns the Current customer seat expansion + current customer upgrade ARR growth KPI + the upgrade flow & true-up flow
• Retention owns the Gross Retention KPI + renewal flow

Growth PM’s will own issues for key flows; engineering leaders will help assign work

Long Term: the vision is for the fulfillment team to own the billing and licensing platform, with a set of API’s and SLA’s, so the growth teams can build and curate user experiences on top of them to drive great customer experiences (and resulting KPI’s).

Deep Dives

Teardown/journey map existing flow. Leverage demos, videos, screenshots. Ask “experts” to join remotely to demo and share their pain points/opportunities. Session to be led by the product manager.

Acquisition Overview

Discussion led by Jensen, PM for Acquisition, topics covered include:

• KPIs and mission for the Acquisition team
• Initial areas of focus
• Teardown of the existing experience
• Teardown of the competitors experience
• Identifying opportunities for improvement

Video

Conversion Overview

Discussion led by Sam Awezec, PM for Conversion, topics covered include:

• KPIs and mission for the conversion team
• Initial areas of focus (Initial product activation, Ah-ha moments with paid features and/or limits, Trial experience and overall value)
• Teardown of the existing experience
• Identifying opportunities for improvement

Video

Expansion Overview

Discussion led by Tim Hey, PM for Expansion. Topics covered:

• Mission
• KPIs & Supporting Performance Indicators
• Customer facing opportunities
  • User Orientation - Users don’t know where to start
  • Increase platform confidence and trust - I love my tools and am afraid to switch
• Internal opportunities
  • Self-hosted usage for upsell process and user workflow
  • Self-hosted True-up process and user workflow

Video

Slides

Retention Overview

Discussion led by Mike Karampalas, PM for Retention. Topics covered:

• KPIs for retention
• Near-term and longer-term focus for the Retention team
• Teardown of the renewal flow(s)
• Opportunities for improvement

Video

Slides

Day 2

Link to Day 2 Agenda Document

Data Deep Dive

Discussion led by Eli Kastelein, Growth Data Analyst. Topics covered:

• GitLab.com data vs Self-Managed data
• Overview of data sources relevant to the growth team
• Vision for how we can use data to serve the growth team
• Event-based reporting versus database reporting
• What's currently blocking us from collecting the data we need to be successful

Video

Slides

State of Telemetry

During this time we walked through Telemetry as a team and reviewed where we were headed. The main item we covered was the Vision Page in the handbook and how we need to clearly communicate what telemetry is and where to find more information.

Topics covered (which will also be added to the handbook)

• Vision
• SMAU – what it is and how it's managed
• Dashboards – what they are, self-serve and how to request one
• Data fields available for tracking and what to do if you need a new one
• Tracking and instrumentation – how to's for GitLab.com and Self-hosted
• Technologies we use

MR under review

A/B Testing Requirements

Issue: Implement an A/B testing solution that can measure the impact of Growth experiments

Key outcome is to work with the Release team to improve our existing Feature Flags capability in GitLab in order to successfully A/B test Growth experiments.

Right now in order to run an experiment, we need to do additional work to ensure we can measure the results. This work will be bespoke for each experiment while we improve our capability.

We can investigate third party solutions if needed.

Days 3-5

At this time the individual Growth teams split into their groups to create and work on issues in their respective areas.

Fast Boot Outcomes

Implemented new team names, KPI’s and objectives.

• Rename Growth Groups
• Updating growth KPIs on KPI index page

Streamlined team processes

• Product/Engineering meeting cadences and deliverables established
• Product/Engineering workflow established
• A/B Testing discussion progress

Create and deliver issues:

Acquisition

• Test simplified user registration page + Design progress
• Update .com paid signup process
• Update trial signup process
• Update self-hosted paid signup process

Expansion

• License Utilization by Host - Epic
• Add seat differential to host page on version.gitlab.com
• Add column to the host page of version.gitlab on the host page to show last check date
• Add a column to version.gitlab on the host page to show if usage ping is available
• From license.gitlab.com link to the License serach results page on the Hosts page of version.gitlab
• Search by license on version.gitlab
• Add column sort functionality to the host page on version.gitlab.com

Conversion

• Adjust in-app pricing page links
• For free users of GitLab.com add trial options for bronze and silver
• Add project import to the existing welcome to GitLab page
• Create end to end testing for trials
• Remove duplicate trial banner CTA from group billing page
• Update group creation URL so it is always unique and does not generate an error
• Group billing page should contain options to buy now and start a trial
• In group trial add buy now button
• Change "welcome to GitLab" message
• Paid feature upgrade point - feature weights

Retention

• Customer Portal: change copy on renew page
• Update the automated Zuora renewals email
• Link to customers portal in renewal banner
• Make Renew or Add Users the primary CTA on Manage Purchases instead of "Add new subscription"
• Add link to customers.gitlab.com from gitlab.com and about.gitlab.com
• Charge for actual users when Auto Renew is enabled for .com customers
• Enable Deep Linking to the Renewal Flow
• Fix Renewal Emails to Not Come from paul@gtilab.com
• Consolidate the emails generated by a renewal
• WIP: Make Auto Renew the Default