| KPI | Health | Status |
|---|---|---|
| Total open SUS-impacting issues by severity | Attention |
|
| Technical Writer MR Rate | Attention |
|
| Average research projects per Product Designer | Attention |
|
| Product Design MR review volume | Okay |
|
| UX Team Member Retention | Attention |
|
| UX Average Age of Open Positions | Attention |
|
The purpose of this chart is to show the total volume of existing issues that impact our SUS score. The labels we're tracking against include "Usability benchmark," "Low SUS Score," "UX scorecard-rec," "cm-scorecard-rec," "UX debt," "UI polish," "accessibility," "VP-UX Dogfooding," "learnability," "UI text," "bug::ux," "maintenance::usability," and "pajamas::integrate".
Target: 0 no severity issues, and 0 S1/S2 issues behind the SLA due date
Health: Attention
DRI: Christie Lenneville
This PI tracks the number of MRs merged every month using the Technical Writing and UI text labels across all GitLab projects where the team works. In January, the additional contributions from the Docs Hackathon put the Tech Writing team over target. February was back around ~40 MRs per team member.
Target: 55 MRs per technical writer per month
Health: Attention
DRI: Susan Tacker
Our goal is to use customer research to validate problems and solutions to ensure we are building the right things in the right way. We use many research methods, including interviews, surveys, usability studies, findability/navigation studies, and analytics. Hypothesis that there is a connection between this KPI and SUS KPI.
Target: At or greater than 2 validation issues per Product Designer per quarter
Health: Attention
DRI: Adam Smolinski
Our goal is to provide UX reviews for Merge Requests (MRs) that include user-facing changes, including those impacting screen readers, to help improve the quality of our product and reduce the amount of UX debt. Product Designers follow our MR Review guidelines to conduct these reviews.
Target: At or greater than 200 MR Reviews per month
Health: Okay
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We need to be able to retain talented team members. Retention measures our ability to keep them sticking around at GitLab. Team Member Retention = (1-(Number of Team Members leaving GitLab/Average of the 12 month Total Team Member Headcount)) x 100. GitLab measures team member retention over a rolling 12 month period.
Target: at or above 84%
This KPI cannot be public.
Health: Attention
URL(s)
This is a subset of an existing KPI. Please see the definition for the parent KPI.
Measures the average time job openings take from open to close. This metric includes sourcing time of candidates compared to Time to Hire or Time to Offer Accept which only measures the time from when a candidate applies to when they accept.
Target: at or below 50 days
Health: Attention
The System Usability Scale (SUS) is an industry-standard survey that measures overall system usability based on 10 questions. Moving a SUS score upward even a couple of points on a large system is a significant change. The goal of this KPI is to understand how usability of the GitLab product rates against industry standards and then track trends over time. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change. See our grading scale for details on interpreting scores. To view a detailed breakdown by individual questions from SaaS users, go to the SaaS dashboard. SaaS SUS data is collected quarterly and Self-Managed data every other quarter. Because of this bi-quarterly collection and gaps in previous collections, Self-Managed data will appear as unlinked points in the Sisense chart below.
Target: 73 by Q4-FY24, 77 by Q4-FY25, 82 by Q4-FY26
Health: Problem
DRI: Christie Lenneville
With SUS as a KPI, it's important to ensure that we are opening and closing SUS-impacting issues at an appropriate velocity. UX is responsible for ensuring that issues are opened when appropriate and advocating for their prioritization, while Product Management is the ultimate DRI for prioritization. The labels we're tracking against include "Usability benchmark," "Low SUS Score," "UX scorecard-rec," "cm-scorecard-rec," "UX debt," "UI polish," "accessibility," "VP-UX Dogfooding," "learnability," "UI text," "bug::ux," "maintenance::usability," and "pajamas::integrate".
Target: TBD
Health: Unknown
DRI: Christie Lenneville
Integrating Pajamas components into GitLab contributes to a cohesive and consistent user experience, visually and functionally. This allows users to seemlessly transition throughout different stages of the DevOps lifecycle. 38 foundational components have been identified in our design system. Legacy UI code in the product needs to be migrated to utilize the components built in Pajamas. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change.
Target: Unknown
Health: Problem
DRI: Taurie Davis
URL(s)
This PI tracks the overall stage score for usability benchmarking studies performed across stage groups as they change over time. The tasks and workflows that comprise each benchmarking study are derived from JTBD for one or more target personas typical for the stage running the study. The overall score for each study takes into account the performance of each task that was tested, through metrics like completion rate, severity, and customer effort score (CES). The scale is 0-100, where 90-100 is ‘Great’, 80-89 is ‘Good’, 70-79 is ‘Fair’, 69 and below is ‘Poor’.
Target: 5% increase in overall score from previous benchmarking, maintaining an overall score above 79/100.
Health: Unknown
DRI: Adam Smolinski
URL(s)
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.
Target: at or below 0.63
Health: Attention
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful.
Target: See Sisense for target
Health: Unknown
URL(s)
This is a subset of an existing KPI. Please see the definition for the parent KPI.
UX Department MR Rate is a performance indicator showing how many changes the UX team implements directly in the GitLab product. We currently count all members of the UX Department (Directors, Managers, ICs) in the denominator, because this is a team effort. The full definition of MR Rate is linked in the url section.
Target: Greater than TBD MRs per month
Health: Attention
URL(s)
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor for team members hired within the past 3 months so hiring managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in more efficient location factor areas with another hire. The historical average location factor represents the average location factor for only new hires in the last three months, excluding internal hires and promotions. The calculation for the three-month rolling average location factor is the location factor of all new hires in the last three months divided by the number of new hires in the last three months for a given hire month. The data source is Workday data.
Target: Below 0.58
Health: Attention
This is a subset of an existing KPI. Please see the definition for the parent KPI.
The number of discretionary bonuses given divided by the total number of team members, in a given period as defined. This metric definition is taken from the People Success Discretionary Bonuses KPI.
Target: at or above 10%
Health: Attention
Actionable insights originate from user research. They always have the 'Actionable Insight' label applied to the resulting issue and a clear follow up that needs to take place as a result of the research observation or data. An actionable insight both defines the insight and clearly calls out the next step as a recommendation. The goal of this KPI is to ensure we're documenting research insights that are actionable and tracking their closure rate.
Target: TBD
Health: Okay
DRI: Adam Smolinski
UX Debt means that for a given issue, we failed to meet defined standards for our Design system or for usability and feature viability standards as defined in agreed-upon design assets. When we fail to ship something according to defined standards, we track the resulting issues with a "UX debt" label. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change.
Target: Below 50 open "ux debt" issues
Health: Attention
DRI: Valerie Karnes
Age of outstanding UX Debt issues. UX Debt means that for a given issue, we failed to meet defined standards. Age represented via median of days opened.
Target: At or below 150 days
Health: Attention
DRI: Valerie Karnes
Historically, Technical Writers were not consistently included in the creation of UI text. Since UI text is critical to product usability, Technical Writing involvement can help improve the quality of our UI. This chart includes issues and MRs with the Technical Writing and UI text labels, because Technical Writing contributions happen in both places.
Target: TBD
Health: Unknown
DRI: Susan Tacker
Number of Product designers against the targeted gearing ratio
Target: At 52 product designers
Health: Attention
Number of Technical Writers against the targeted gearing ratio
Target: At 19 technical writers
Health: Attention
Number of researchers against the targeted gearing ratio
Target: At 10 researchers
Health: Attention
The total number of promotions over a rolling 12 month period divided by the month end headcount. The target promotion rate is 12% of the population. This metric definition is taken from the People Success Team Member Promotion Rate PI.
Target: 12%
Health: Okay
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| -1 | Confidential | Metric & metric health are confidential |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name} |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section) |
section |
Optional | the product section (Ex: dev) as defined in sections.yml |
stage |
Optional | the product stage (Ex: release) as defined in stages.yml |
group |
Optional | the product group (Ex: progressive_delivery) as defined in stages.yml |
category |
Optional | the product group (Ex: feature_flags) as defined in categories.yml |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI. |
health.level |
Optional | indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI. |
health.reasons |
Optional | indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
urls |
Optional | list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url |
funnel |
Optional | indicates there is a handbook link for a description of the funnel for this PI. Should be a URL |
sisense_data |
Optional | allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes. |
sisense_data.chart |
Optional | indicates the numeric Sisense chart/widget ID. For example: 9090628 |
sisense_data.dashboard |
Optional | indicates the numeric Sisense dashboard ID. For example: 634200 |
sisense_data.shared_dashboard |
Optional | indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b |
sisense_data.embed |
Optional | indicates the Sisense embed version. For example: v2 |
sisense_data_secondary |
Optional | allows a second Sisense dashboard to be embeded. Same as sisense data |
sisense_data_secondary.chart |
Optional | Same as sisense_data.chart |
sisense_data_secondary.dashboard |
Optional | Same as sisense_data.dashboard |
sisense_data_secondary.shared_dashboard |
Optional | Same as sisense_data.shared_dashboard |
sisense_data_secondary.embed |
Optional | Same as sisense_data.embed |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
pi_type |
Optional | indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI) |
product_analytics_type |
Optional | indicates if the metric is available on SaaS, SM (self-managed), or Both. |
is_primary |
Optional | boolean flag that indicates if this is the Primary PI for the Product Group. |
implementation |
Optional | indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI. |
implementation.status |
Optional | indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI. |
implementation.reasons |
Optional | indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
lessons |
Optional | indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI. |
lessons.learned |
Optional | learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned |
monthly_focus |
Optional | indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI. |
monthly_focus.goals |
Optional | indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal |
metric_name |
Optional | indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name. |
Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.