Our centralized engineering dashboards provide a set of common metrics that capture the overall health of the entire R&D Product/Engineering structure, with drill downs into every stage and group.
This work is the product of the team working in our unified engineering metrics task process. The inception of this initiative can be see in this epic.
The links below take you to a handbook dashboard page which covers metrics from the Development, Infrastructure, Quality, UX, and Security Departments.
These handbook dashboard pages are populated from the following filterable Sisense dashboards.
To navigate to section metrics, click on the ___ Section Dashboards link above. To navigate to stage metrics, click on the ___ Stage Dashboards link above or redirect from the section dashboards pages. To navigate to group metrics, find the stage that contains the group and click on the corresponding ___ Stage Dashboards link above.
Group metrics can be queried using labels. For example, the ~"group::authentication" label is used to query MRs or issues belonging to the Authentication. If a group is missing from the filter dropdown in the embedded dashboards above, check if it’s included in this spreadsheet. If it’s missing, add a new row with the corresponding stage and section. This spreadsheet flows to our data warehouse once a day so you may not see updates in Sisense until the next day. In order to display this new group in the handbook dashboard pages, please reach out to a member of the Engineering Analytics Team to get this added to the handbook dashboard page filters.
The Engineering Metrics listed here are available for all product group teams. The indicators captured here may or may not roll into an existing KPI/PI. The Engineering Analytics team reserves the urgency for these dashboards to provide timely visibility without the requirement of having all indicators be a KPI/PI at the department level.
undefined, spend some time to review your team's issues and MRs and add labels so we can get a more accurate classification.Merge Request (MR) Rate is a measure of productivity and efficiency. The numerator is a collection of merge requests to a set of projects. The denominator is a collection of people based on the job title specialty field in Workday. Both are tracked over time (usually monthly). The stages.yml file is the SSOT for group names. We rely on a mapping between the group name in this file to the job title specialty field in Workday. A mismatch between the two would cause team members or MRs not to be counted.
In April 2023, there was an internal audit of the job title specialty field done by managers and directors. MR Rate data prior to this date may report inaccuracies due to missing or incorrect job title specialities that were corrected in the audit.
You can use this MR Rate troubleshooting dashboard to check the number of team members that are counted each month. If the monthly team member count is less than expected, refer to the table to see which team member is missing.
To update the job title speciality field, please refer to the guidelines.
job title specialty field in Workday. In the past month, there were 20 merged MRs with the group::Apples label. Team A's MR Rate for that month would be: (20 / 5) = 4.job title specialty field in Workday. In the past month, there were 20 merged MRs with the group::Orange label. Since the job title specialty does not match the group label (an extra s in the job title specialty field), we are unable to map the MRs back to the respective groups. We recommend either 1) updating the group label and stages.yml file or 2) updating the Workday value.Group MR Rate can be found on this dashboard and filtered by group. It can also be queried by department or group using the following SQL:
SELECT merge_month
, employees
, mrs
, mr_rate
FROM workspace_engineering.merge_request_rate
WHERE group_name=[fill in group name here]
AND granularity_level = 'group'
ORDER BY 1
Please see our team’s FY24 Q2 OKR - Begin migration from Sisense to Tableau for Engineering for a general overview of the Tableau migration.
In FY24 Q2, we plan to migrate ~20 following dashboards, starting with:
Analysis of Sisense dashboards owned by Engineering
Google sheet to check off which dashboards need to be ported over, deprecated, or re-designed
This effort includes:
Note: These dashboards in particular will not be re-designed, and thus porting should be relatively straightforward; i.e., the overall dashboards and their charts will look and act very similarly to how they are displayed and how they behave in Sisense today.
We use the following type labels to classify our Issues and Merge Requests.
The 3 types (Bug, Feature & Maintenance) is key to our report to industry analysts. It is important for GitLab to communicate effort spent into a format that is easily understandable widely in the industry. We provide this metric to our leadership reporting and improve the accuracy with subtypes categorization. The 3 top level types can be applied without having to apply a sub-category type.
~"type::bug": Defects in shipped code and fixes for those defects. Read more about features vs bugs.
~"bug::performance": Performance defects or response time degradation~"bug::availability": Defects related to GitLab SaaS availability. See the definition for more guidance.~"bug::vulnerability": Defects related to Security Vulnerabilities~"bug::mobile": Defects encountered on Mobile Devices~"bug::functional": Functional defects resulting from feature changes~"bug::ux": Unexpected and unintended behavior that is detrimental to the user experience.~"bug::transient": Defects that are transient.~"type::bug" are considered ~"type::bug".~"type::feature": Effort to deliver new features, feature changes & improvements. Read more about features vs bugs.
~"feature::addition": The first MVC that gives GitLab users a foundation of new capabilities that were previously unavailable. Includes good user value, usability, and tests. For example, these issues together helped create the first MVC for our Reviewer feature: Create a Reviewers sidebar widget, Show which reviewers have commented on an MR, Add reviewers to MR form, Increase MR counter on navbar when user is designated as reviewer~"feature::enhancement": Subsequent user-facing improvements that refine the initial MVC by adding additional capabilities that make it more useful. Includes good user value, usability, and tests. For example, these issues enhance the existing Reviewer feature: Show MRs where user is designated as a Reviewer on the MR list page, Display which approval rules match a given reviewer, Add Reviewers quick action~"feature::consolidation": Merging a feature into an existing feature for simplification. For example, Workspace project: (Consolidate Groups and Projects) and Combine Top Navigation Menu are good examples of such work.~"type::feature" are considered ~"type::feature".~"type::maintenance": Upkeeping efforts & catch-up corrective improvements that are not Features nor Bugs. This includes removing or altering feature flags, removing whole features, merge requests that only include new specs or tests, documentation updates/changes (not including new documentation), restructuring for long-term maintainability, stability, reducing technical debt, improving the contributor experience, or upgrading dependencies and packages. For example: Refactoring the CI YAML config parser, Updating software versions in our tech stack, Recalculating UUIDs for vulnerabilities using UUIDv5
~"maintenance::refactor": Simplifying or restructuring existing code or documentation~"maintenance::removal": Deprecation and removal of a functionality when it's no longer needed.~"maintenance::dependency": Dependency updates and their version upgrades~"maintenance::scalability": Modification to improve the scalability of GitLab that is not a user facing change or performance improvement. For example changing a column from INT to BIGINT.~"maintenance::usability": General improvements to product usability that are unrelated to feature prioritization. For example, UI component and UI text updates for consistency with Pajamas and usability improvements.~"maintenance::test-gap": Test coverage improvements that were not included in feature prioritization.~"maintenance::pipelines": Pipeline related changes.~"maintenance::workflow": Improvements of the engineering tooling like Danger, RuboCop, linters, issue templates, etc.~"maintenance::performance": Non-defect performance improvements such as load time for a specific page or component, or run time for a specific process.~"maintenance::release": Release management related changes.If these labels are missing, it will be tracked in the undefined bucket instead.
The Engineering Manager for each team is ultimately responsible for ensuring that these labels are set correctly. If you do not feel the purpose of this issue matches one of the types, you may apply the ~type::ignore label to exclude it from type tracking metrics and future prompts, this can be good for issues marked ~Planning Issue.
Classifying work types may require context. All work to deliver a feature with security, performance and quality meeting the definition of done should be classified as feature work. For example if you are anticipating the performance needs of a feature and implement an application limit as part of the introduction of that feature it should be classified as feature:addition. If you discovered an issue scaling an existing feature and implemented an application limit that issue would likely start as a bug and the associated MR would be classified as bug:performance.
As of %16.7, with certain internal projects, type labels are locked once an associated MR is merged. More details can be found in the feature implementation issue.
Large efforts will occasionally undergo a spike to identify & research technical approaches to complete the work. Spike efforts shall be classified with the following guidelines:
~"feature::enhancement" and ~"type::feature"~"maintenance::dependency" and ~"type::maintenance"~"Community contribution"~"Community contribution" was intended to track community contributions as a top-level type, but it's now only a facet label and a merge request should always get a proper type label set in addition.
Community contributions are welcome in all areas of GitLab, so any type label can be set on ~"Community contribution" merge requests.
~"security"~"security" was intended to track security-related merge requests as a top-level type, but it's now only a facet label and a merge request should always get a proper type label set in addition.
This guidance may be helpful if you are wondering the go-forward type label based on your use case for applying ~"security":
~"type::feature" for new security features that aren't fixing an existing vulnerability~"type::bug" for any other security changes~"documentation"~"documentation" was intended to track documentation-only merge requests as a top-level type, but it's now only a facet label and a merge request should always get a proper type label set in addition.
This guidance may be helpful if you are wondering the go-forward type label based on your use case for applying ~"documentation":
~"type::feature" for new feature documentation (this type would usually be already set on merge requests that introduce a new feature)~"type::maintenance" for any other documentation changes~"backstage"~"backstage" was intended to be changes that were done to keep product development running smoothly. Over time, ~"backstage" was also being used for pre-feature work and has become unclear and confusing. ~"backstage" was deprecated as part of https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/488. This will be removed with https://gitlab.com/gitlab-org/quality/triage-ops/-/issues/483.
This guidance may be helpful if you are wondering the go-forward type label based on your use case for applying ~"backstage":
~"type::maintenance"
~"technical debt", ~"railsx.y", ~"Architecture Decision", non-~"security" ~"dependency update"~"type::feature"
~"feature::addition" for all changes related to the release of a new feature~"feature::enhancement" for user-facing improvements that refine the initial MVC to make it more useful and usable.~"maintenance::workflow" for changes to engineering workflows such as ~"Danger bot", ~"static analysis", release tooling, Docs tooling changes~"maintenance::pipelines" for changes to project pipeline configurationsIn the spirit of "Everyone can Contribute" it's natural that members in a group will contribute to another group.
We allow flexibility where the parent devops::xxx and child group::xxx label may not match. For example:
frontend, backend components or type::tooling work that spans multiple groups.If a contribution happens across groups, we leave it to the discretion of the engineering and product manager to change the group::xxx label to reflect which group worked on it.
They can also decide if they want to move over the devops::xxx as well or keep it to reflect the product area.
The triage bot automatic labelling will not override existing labels.
In the MR Rate and Volume of MR calculations, we consider MRs from projects that contributes to the overall product efforts.
The current list of projects are identified in the gitlab-data/analytics project for the following system databases:
| System Database | File |
|---|---|
| GitLab.com | projects_part_of_product.csv |
| ops.gitlab.net | projects_part_of_product_ops.csv |
The guidelines for inclusion in the is_part_of_product lists are:
Follow these steps to request a new project to be tracked:
There is no need to remove archived projects from the is_part_of_product list. Removal of projects will remove historical merge requests from metrics and reduce Merge Request rates.
Please reach out to a member of the Engineering Productivity team if more assistance is needed
URL property is only used to link to a chart until it is an embedded Sisense chart.<a> in description text if we need to link out to a supporting artifact e.g. Epics or Issues.Above ...Below ...At ...At or above ...At or below ...case when clause in Sisense. Example below:CASE WHEN date_month < date_trunc('month',current_date) THEN MEDIAN(open_age_in_days) ELSE NULL END AS "Historical Median Open Days",CASE WHEN date_month = date_trunc('month',current_date) THEN MEDIAN(open_age_in_days) ELSE NULL END AS "Current Median Open Days",shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property: in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :