UX Department Performance Indicators

1. You are here:
2. Handbook
3. Engineering
4. UX Department
5. UX Department Performance Indicators

On this page

• Executive Summary
• Key Performance Indicators
  • UX Hiring Actual vs Plan
  • UX Non-Headcount Plan vs Actuals
  • UX Average Location Factor
  • UX Recruiting Average Top-of-Funnel Location Factor
  • UX Handbook Update Frequency
  • Perception of system usability
  • Proactive UX Work
• Regular Performance Indicators
  • UX Discretionary Bonus Rate
  • UX Department Member MR Rate
  • UX debt
  • Throughput of Technical Writing team documentation MRs
  • Distribution of Technical Writing team documentation effort
• Other PI Pages
• Legends
  • Health
  • Instrumentation
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Executive Summary

KPI	Health	Status
UX Hiring Actual vs Plan	Okay	UX hiring is on plan, but we just put in place a new "two-star minimum" rule that might decrease offer volume. Health: Monitor health closely
UX Non-Headcount Plan vs Actuals	Okay	Chart budget vs. actual over time available in periscope with clear variance
UX Average Location Factor	Attention	Our average location factor trended upward to .68 due to hiring for more complex stage groups. We are actively working with recruiting to target the right areas of the globe on a per role basis, and we expect to see this average go down during FY21.
UX Recruiting Average Top-of-Funnel Location Factor	Unknown	We have set the target location factors on our vacancies and make sure recruiting is targeting the right areas of the globe on a per role basis.
UX Handbook Update Frequency	Unknown	Unknown.
Perception of system usability	Attention	Perceived usability rates as a B- with long-term trend flat to slightly down. Continue to action on UX Scorecards for every stage group. Focus on UX Debt. Work with PM to prioritize usability issues.
Proactive UX Work	Okay	In Q1 FY21, we had 30 product Designers on staff and 52 total validation issues. This is slightly under our target of 60.

Key Performance Indicators

UX Hiring Actual vs Plan

This is a subset of an existing KPI. Please see the definition for the parent KPI.

Employees are in the division "Engineering" and department is "UX".

Target: 67 by February 1, 2021

Chart (Sisense↗)

Status: Okay

• UX hiring is on plan, but we just put in place a new "two-star minimum" rule that might decrease offer volume.
• Health: Monitor health closely

UX Non-Headcount Plan vs Actuals

This is a subset of an existing KPI. Please see the definition for the parent KPI.

We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.

Target: Unknown until FY21 planning process

URL(s)

• https://app.periscopedata.com/app/gitlab/632490/UX-Non-Headcount-BvAs

Status: Okay

• Chart budget vs. actual over time available in periscope with clear variance

UX Average Location Factor

We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.

URL(s)

• /handbook/hiring/charts/ux-department/

Chart (Sisense↗)

Status: Attention

• Our average location factor trended upward to .68 due to hiring for more complex stage groups.
• We are actively working with recruiting to target the right areas of the globe on a per role basis, and we expect to see this average go down during FY21.

UX Recruiting Average Top-of-Funnel Location Factor

We need to be proactive in measuring our location factor, starting with candidates who are at the top of the recruiting funnel.

Target: 0.58

URL(s)

• /handbook/hiring/charts/ux-department/

Chart (Sisense↗)

Status: Unknown

• We have set the target location factors on our vacancies and make sure recruiting is targeting the right areas of the globe on a per role basis.

UX Handbook Update Frequency

This is a subset of an existing KPI. Please see the definition for the parent KPI.

The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/**` or `/source/handbook/support/**` over time.

Target: 50

Chart (Sisense↗)

Status: Unknown

• Unknown.

Perception of system usability

The System Usability Scale (SUS) is an industry-standard survey that measures overall system usability based on 10 questions. Moving a SUS score upward even a couple of points on a large system is a significant change. The goal of this KPI is to understand how usability of the GitLab product rates against industry standards and then track trends over time. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change. See Table 1 for grading details.

Target: B+

URL(s)

• https://docs.google.com/spreadsheets/d/1RzoYgjIH8RYiUxEJmUb7f_hTy83vTxpRtJS4AL0Ekrk/edit?usp=sharing

Chart (Sisense↗)

Status: Attention

• Perceived usability rates as a B- with long-term trend flat to slightly down.
• Continue to action on UX Scorecards for every stage group. Focus on UX Debt. Work with PM to prioritize usability issues.

Proactive UX Work

Use customer research to validate problems and solutions to ensure we are building the right things in the right way. We use many research methods, including interviews, surveys, usability studies, findability/navigation studies, and analytics. Hypothesis that there is a connection between this KPI and SUS KPI.

Target: 2 validation issues per Product Designer per quarter

Chart (Sisense↗)

Status: Okay

• In Q1 FY21, we had 30 product Designers on staff and 52 total validation issues. This is slightly under our target of 60.

Regular Performance Indicators

UX Discretionary Bonus Rate

Discretionary bonuses offer a highly motivating way to reward individual GitLab team members who really shine as they live our values. Our goal is to award discretionary bonuses to 10% of GitLab team members in the UX department every month.

Target: 10%

URL(s)

Status: Unknown

• We currently track bonus percentages in aggregate, but there is no easy way to see the percentage for each individual department.

UX Department Member MR Rate

UX Department MR Rate is a performance indicator showing how many changes the UX team implements directly in the GitLab product. We currently count all members of the UX Department (Directors, Managers, ICs) in the denominator, because this is a team effort. The projects included contributes to the overall product development efforts.

Target: TBD

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/4448

Chart (Sisense↗)

Status: Attention

• We don't yet know what a good MR rate looks like for UX. Need accurate data to determine.
• UX MR rate doesn't accurately reflect all MRs to which UX contributes, because we often collaborate on MRs rather than opening them ouselves.

UX debt

UX Debt means that for a given issue, we failed to meet defined standards for our Design system or for usability and feature viability standards as defined in agreed-upon design assets. When we fail to ship something according to defined standards, we track the resulting issues with a "UX debt" label. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change.

Target: Under 50 open "ux debt" issues

Chart (Sisense↗)

Status: Attention

• Open "UX debt" issues are growing over time. Average days to close a "UX debt" issue is unacceptable.
• Quality Engineering helped us add UX Debt to the weekly triage issue. Since then, we've seen a slight decrease in total UX debt, but not enough to change the health score of this metric.
• The UX department created a UX Debt dashboard that breaks debt down by stage group, closed issues that are no longer relevant, and reviewed all open UX Debt issues to make sure they have the appropriate group label.
• We are actively working with PMs to prioritize UX Debt. Some stage groups are committing to resolving a minimum number of UX Debt issues per milestone (generally, a commitment of no less than one issue). We will track this effort and make adjustments as we see the results.

Throughput of Technical Writing team documentation MRs

This KPI tracks the number of ~documentation MRs merged every month across all GitLab projects which have involvement (review, collaboration, or authoring) from the ~"Technical Writing" team. The goal is to increase velocity over time as the team grows.

Target: 605 MRs per month (55 MRs per technical writer per month)

Chart (Sisense↗)

Status: Okay

• In early May 2020, we raised the target to 55 MRs per writer per month. May 2020 average was 59 MRs per writer. June 2020 average was 53 MRs per writer.
• We'll continue to watch this metric to determine the appropriate target.

Distribution of Technical Writing team documentation effort

Tracks the type of documentation changes involving the ~"Technical Writing" team, based on `docs::` scoped label applied. Labels include feature, new, improvement, fix, revamp, or housekeeping. Our goal is to increase the proportion of efforts where Technical Writers proactively improve content (improvement, new, and revamp), instead of just responding to a new feature or fixing a bug.

Target: 50% of MRs are focused on proactively making our docs better improvement, revamp, new.

Chart (Sisense↗)

Status: Attention

• 47% of Technical Writing MRs were focused on proactive improvements in April 2020. 41% in May 2020. 45% in June 2020.
• Not all documentation issues have the correct labels. We need to add automation to prompt contributors to consistently add the scoped labels.

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Growth Section Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Secure & Defend Section Performance Indicators
  • Enablement Section Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• Data Team KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

Instrumentation

Value	Level	Meaning
3	Okay	We are correctly capturing and measuring data for GitLab.com and self-managed instances.
2	Attention	Missing one of: GitLab.com or self-managed data.
1	Problem	Missing both: GitLab.com and self-managed data.
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicator_target(name) helper function returns the target value from the target property for a specific PI from the Performance Indicators data file.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from the Performance Indicators data file and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this is the Performance Indicators data file which is in YAML file. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department)
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI.
urls	Optional	list of urls associated with the (K)PI
sisense_data	Optional	contains elements/properties related to Sisense, including chart (numeric Sisense widget ID), dashboard (numeric Sisense dashboard ID), shared_dashboard (Sisense shared dashboard ID), embed (v2)
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
instrumentation	Optional	has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a single Green bar for the current month (in progress).
  • Use a Red stepped-line for timeseries target.
  • Optional: Use a Black dashed-line for rolling average.
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the Performance Indicators data file under the sisense_data property
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :
• urls: should be an array (indented lines starting with dashes) even if you only have one url
• Both maturity.level and health.level display a value between 0 and 3 (inclusive).