The post purchase survey (PPS) is sent to a customer via Qualtrics when a sales-assisted account is confirmed in Salesforce.
The manual tasks described in those issue templates are the SSOT for carrying out the PPS
The features that are presented to customers in the survey are those that belong to a category where the marketing attribute is set to "true" and the stage of that category is also set to "marketing: true."
The list of features is generated by beginning an irb session from within the www-gitlab-com repo and running the following script:
require 'yaml'
require './lib/stage'
require './lib/feature'
stages = Gitlab::Homepage::Stage.all!.select{ |stage| (stage.marketing || stage.enabling) }
features={}
stages.each do |stage|
features[stage.key.capitalize] = {}
end
stages.each do |stage|
stage.categories.select { |category| category.marketing }.each do |category|
next unless category.features?
features[stage.key.capitalize][category.key.capitalize] = []
category.features.each do |feature|
features[stage.key.capitalize][category.key.capitalize] << feature.title if feature.gitlab_released && !feature.gitlab_premium
features[stage.key.capitalize][category.key.capitalize] << feature.title if feature.gitlab_premium && !feature.gitlab_core
end
end
end
features.each do |key, value|
puts "## #{key}"
value.each do |subkey, subvalue|
if !subvalue.empty?
puts "**#{subkey}**"
subvalue.each do |item|
puts "- #{item}"
end
end
end
end
To update the feature list in Qualtrics, follow the steps outlined in this video: https://www.youtube.com/watch?v=6rBStkOix6w. This video uses a spreadsheet as the source for feature changes, but the process for updating Qualtrics is the same. You can find the necessary information by referencing the comments and linked MR in the "FYXX QY PPS Feature List Update" issue.
Product Operations and UX Research partner on running PNPS surveys. We determine our Net Promoter Score for paid users of GitLab.com (SaaS) on a quarterly basis through a survey launched via Qualtrics. A data collection and analysis issue (example) is automatically created at the beginning of each quarter which is the SSOT and outlines all tasks, DRI's and due dates. Data is collected over a period of six weeks starting in the middle of a quarter and the survey stays open until the last day of the quarter. The first two weeks of the new quarter are used to analyze and report data on the previous quarter. All documents created are stored in the PNPS drive.
We aim for a sample of over 500 with plan type proportions for our sample to be +/- 3% compared to the population proportion.
Data analysis consists of two components: statistical analysis and open response coding.
Statistical analysis: The majority of the statistical analysis is accomplished by the analysis template. Drop the results CSV into it to generate the data provided in the results deck. Follow the instructions in the template.
Open response coding: There are instructions on how to code the responses in the analysis template.
Review this video for a detailed walkthrough of running the analysis.
At the beginning of each quarter a list of eligible users to send the survey to is generated, using the following steps:
An example with fake numbers:
In this example the sample is under the population, hence the next wave percentage is higher than the population to try and make that up.
Review this video for a detailed walkthrough of how to send out email waves.
Once all email waves have been sent, add the user IDs that were used this quarter to the previously contacted sheet, noting the quarter. This allows us to avoid contacting the same users too frequently.
PNPS usersAfter the quarterly analysis is shared out, Product Operation initiates outreach to PNPS responders who opted into contact via this automated issue. The full PNPS responder outreach workflow can be seen here.
Product Operations partners with UX Research to produce and share out quarterly analysis slides. Product Operations partners with Product Analytics to maintain the PNPS dashboards.
There are two charts in Sisense that we reference when analyzing the PNPS results:
They can also be found in the Product Performance Indicators section of the internal handbook.
The survey responses in Qualtrics are synced to the qualtrics_nps_scores table which populates these charts.
There may be an occasional need to remove certain records (such as internal test records) from the data. The pnps_excluded_response_ids Sisense snippet is used to manage these excluded response_ids. When a response has to be excluded, go to edit the PNPS Excluded Response IDs snippet and add the response_id to the list. Make sure to match the formatting of the other response IDs and eliminate any spaces. If you are working on a new chart and would like to incorporate this snippet, you just have to add WHERE response_id NOT IN ([pnps_excluded_response_ids]) to the WHERE clause.
For past reports, there has been a request to see previous plans that detractors have had. For example, of Premium detractors, how many were previously Bronze users. If the need arises to do so, follow the guidance below:
Q3_NPS_GROUP = 'Detractor' and plan = 'Premium'.namespace_ids and add convert them to a string list (i.e. convert a namespace_id from 3777494 to '3777494',) for querying purposes.namespace_ids with this new list and save the query. The report will update upon saving.plan = 'Ultimate'.