Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Engineering Metrics

There are many ways to develop engineering metrics. Ideally you need a representation of different aspects of the operational data such as throughput, quality and cycle time to name a few.

The goal of gathering this data is to define what a healthy and successful team looks like and to foster a healthy dialogue amongst the engineering team and with stakeholders. These metrics are helpful for conversations around planning, addressing technical debt, capacity to work on bugs, identifying bottlenecks and improving engineering efficiency. Metrics are a good way to also see how change affects the team’s execution.

Engineering metrics are project or team based and are not intended to track an individual's capacity or performance. It is important to note that if metrics are used punitively, these goals are hampered and the team psychological safety could be at risk.

As a manager you should ensure you are on top of your team's metrics, driving the right behavior, and understanding any fluctuations. Expect to be able to discuss your team's metrics with your manager on a weekly basis.

To access our current dashboard, please visit the GitLab Insight Quality Page. We are currently working to implement these graphs into the GitLab product.

We collect all related Sisense dashboards in its own Engineering: Development Metrics topic in Periscope to have them all in one place.

List of Metrics

Data included in calculation

Engineering wide projects

We are currently working on including all of engineering's projects. The following projects included can be seen in this list.

Security fixes on dev instance

We also capture data from our https://dev.gitlab.org/ instance which currently includes the projects below.

The data from dev instance is only reflected in a few group level charts, see list of known issues below.

Known issues

Adding more metrics

If you or your team needs to include a new project in the metrics or add a new team, please create an issue in the GitLab Insights issue tracker.

We have a list of projects that are still not included in this issue.

Universal team metrics dashboard in Sisense

Considering how one of our development department KPIs is to have a goal of 10 MRs per Development Engineer per Month, we have a team metrics dashboard that can easily be used to see MR statistics per team.

  1. Open the dashboard
  2. You can select in the team_id filter (under the headline) for which team you want to see the statistics.

Universal section metrics dashboard in Sisense

Considering how one of our development department KPIs is to have a goal of 10 MRs per Development Engineer per Month, we have a section metrics dashboard that can easily be used to see MR statistics per section.

  1. Open the dashboard
  2. You can select in the section_id filter (under the headline) for which section you want to see the statistics.

Note: These dashboards will be available internally to everyone at GitLab (as per default Sisense permissions). Considering that the majority of engineering MR activities are public, we do not anticipate this being an issue. In addition, anyone that can view the dashboard will also be able to toggle the filter to see individual team member's metrics. We purposefully do not chart all team members in a specific chart against each other because throughput is just one data point and is not a holistic evaluation of a team member's productivity/performance.

Note: MRs that are merged in dev.gitlab.org are not available in Sisense: gitlab-data/analytics#2789

Why doesn't a team member show up?

Currently we update the team member records in Sisense with a manually-executed script. This script maps members from team.yml to teams in Sisense by the departments list for each member. The script expects exact string matches, so if a team member is not appearing in Sisense please check their section or team is not missing or misspelled in team.yml. If you make an update to team.yml, please let a director know to rerun the script.

Red column in Employee Full Name Match

We currently map team members between team.yml and Bamboo through their full name. But they can be different due to various reasons so we have as a pragmatic solution for now a Google sheet to map GitLab ID's to Fullnames as in Bamboo so that we can track when people were members of a specific team. Please add missing team members to the sheet with the first and last name from fields in bamboo and then let a director know to rerun the script.

Sisense Utility Snippets

We created many SQL snippets to centralize counting and the actual structure of data for best reuse.

Example usage

For example paste [development_team_mrsperengineer_by_members('manage_be')] into a chart and it will give you all MR/engineer metrics for the team manage_be.

SQL Snippets

[development_team_mrsperengineer_by_members(team_name)]

Gives you all metrics directly for a chart around MR/engineer metrics for one specific team.

[development_section_mrsperengineer_by_members(section_name)]

Gives you all metrics directly for a chart around MR/engineer metrics for one specific section.

[development_team_mrtotals_by_members(team_name)]

Gives you all data for a MR total graph based on team members.

[development_section_mrtotals_by_members(section_name)]

Gives you all data for a MR total graph based on section members.

[development_team_metrics_by_members(team_name)]

Gives you all relevant queries for MR metrics based on team members. (dev_product_mrs_merged, dev_team_members, dev_counts, dev_monthly_analysis)

[development_section_metrics_by_members(section_name)]

Gives you all relevant queries for MR metrics based on section members. (dev_product_mrs_merged, dev_team_members, dev_counts, dev_monthly_analysis)

[development_team_meantimetomerge_members(team_name)]

Returns all Mean time to Merge metrics for a team.

[development_team_members_by_team(team_name)]

Returns all Development Team Members by team name.

[development_team_members_by_section(section_name)]

Returns all Development Team Members by section.

[development_team_member_definitions]

Currently half-automated select statement that defines all team members based on data from team.yml and GitLab API to gather GitLab user ID's.

The script is located in the manager-tools repository. The plan is to automate it ASAP through automatic importing, currently half manual as it is easier to iterate.

Sisense project indicators

There are two different indicators in Sisense for Merge Request data; is_included_in_engineering_metrics and is_part_of_product.

Use is_part_of_product to identify MRs to include for team dashboards to ensure alignment with department KPIs.