Engineering Analytics enables Engineering at GitLab to understand how quickly we are building and evolving our product to meet our customers' needs, and to decide what are the most appropriate balances between cost and effort in building our product.
Engineering Analytics is responsible for building and evolving analytics capabilities and creating insights for Engineering to understand how well we are building our product. In this case, "wellness" is measured in terms of efficiency, as well as cost.
Areas of focus and core competencies for Engineering Analytics include:
In addition to the areas of focus and core competencies listed above, the Engineering Analytics team is continually improving and evolving, in order to keep up with the improvement and evolution of Engineering at GitLab. Areas of focus for the evolution of the team include:
Every quarter, the team commits to Objectives and Key Results (OKRs). The below shows current and previous quarter OKRs, it updates as the quarter progresses. Starting February 2023, Engineering Analytics uses GitLab to track progress towards quarterly OKRs.
Here is an overview of our current OKRs.
This team is a function under the Quality Department operating as a team of Engineering Analysts, led by a leader of Engineering Analytics, who in turn reports to the leader of the Quality Department.
Please open a new issue in our team project using the
Key Reviews are meetings a department has with other GitLab team members to discuss any progress or updates related to KPIs & OKRs with the rest of the organization. More information can be found on the Key Review Handbook Page
In Engineering our key reviews are handbook driven. Each department's KPIs are defined in www-gitlab-com repo under data/performance-indicators, & metrics are in handbook with URL structure like engineering/department/performance-indicators. For example, Infrastructure's full URL is https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators.
The Performance Indicator DRI is an optional attribute in the PI page
.yml file that specifies an individual as the DRI for a particular KPI/PI.
The DRI is responsible for updating the health & health reasons for the metric. In cases where the health is at problem or attention level, the actions being taken to fix the metric should be specified. The DRI is also responsible for verifying the data & if needed talking with an analyst to make sure the metric is ready for the key review.
The DRI is a delegation of responsibility for the things listed above from the respective department head, although the department head is still generally responsible for all of their PIs.
If no DRI is listed, the default DRI would be the department head.
This position is responsible for defining the division level PIs that are shared across departments or solely at Engineering Division level. They are also responsible for working with department heads on setting targets for these.
Department Heads in Engineering are generally responsible for their overall PI page & making sure they have what they need for the metrics before & after the key meetings.
Our main role as engineering analysts is to support department heads or DRI in creating or updating their metrics so that they can use them in the key meetings. Your stable counterpart may also ask to delegate a set of metrics from their department to you so in that case you would also be the DRI for those metrics & assume the responsibilities listed above in DRI section
While this team operates as a single team reporting to one manager, we emphasize on ensuring the prioritization and needs of Engineering Leaders via stable counterparts.
The team structure will leverage stable counterpart assignees to ensure proper allocation and attention needed to service all Engineering Departments and Leaders and still promote cross-functional collaboration and ownership of an area.
We assign the stable counterpart by Engineering Division’s sub-departments. This is identified by the Engineering Department name assigned to an Engineering Analyst.
|Eng Department||Analyst||PI Page|
|Engineering||Engineering Analytics Team||Eng PI Page|
|Development||Lily, Dani||Dev PI Page|
|Infrastructure||Clément||Infra PI Page|
|Quality||Raul||Quality PI Page|
|Security||Dani||Security PI Page|
|Support||Lily||Support PI Page|
|Finance||Clément||Finance PI Page|
Engineering analysts assigned to one area are experts in that area and may not have the knowledge depth in other areas. As such contributing cross-domain expertise will only be limited to Sisense charting and not beyond this data layer.
Aside from the below listed scheduled meetings, the team will also have scheduled 1:1s with other team members and their stable counterpart assignement's department head at a cadence of their choosing to discuss any topics that were not covered in normal status updates or other meetings.
Each Engineering department has a monthly or bi-monthly key review where e-group goes over the department's metrics and can ask the department heads any general questions regarding the department's progress towards its goals. Analysts are expected to go to their stable counterpart's key meeting & participate if needed, & are encouraged to attend other department key reviews as time permits.
Engineering Analysts will hold recurring 1-1 with their assigned VP of an Engineering Department to:
This team will have standing bi-weekly sync with the Data team to:
These weights are used to estimate analyst time needed to fulfill a request by 1 engineering analyst. This is used when estimating total capacity of work that can be done by the team. If more than 2 weeks needed (>8 weight), then issue needs to be broken down into multiple issues.
|Label Weight||1 Analyst Time Estimate||Work Example|
|1||1 full day||basic handbook updates, changing basic parameters in Sisense charts|
|3||1/2 week||Creating new Performance Indicator from Data that already exists in data warehouse|
|5||1 week||Creating new dashboard with granular details around existing metric|
|8||2 weeks||introducing new metric based from data source that doesn't currently exist in data warehouse|
The Engineering Analytics team uses scoped status labels to track different stages of issues on the board. This is also used to clearly communicate to stakeholders the status of each requests.
||Issue has been ackowledged and we are working through scoping how much work is required to complete the request|
||Request has been clarified & amount of work required to fulfill request has been identified.|
||Issue is actively being worked on|
||Team is waiting for dependency to complete work to continue progress, or work has been de-prioritized. Issue will also have blocking dependency linked.|
||Issue has been completed & work is under review by stakeholder|
Work is prioritized using table below. Work type is stacked ranked within each priority.
||Urgent Key Review requests
Free User Efficiency Rapid Action
Hiring & Onboarding
||Non passthrough OKRs
Walk-up request from Engineering
|We have capacity to take this on, OKRs will be prioritized first.|
||New Data Sources
Walk-up request outside of Engineering
|We do not have capacity to take this on at the moment|
||Nice to have Improvements||We do not have capacity to take this on at the moment|
To ensure that the issue gets visibility by our team, when creating an issue on the Engineering Analytics board, please add the
~"Engineering Metrics" label in addition to one of the below labels and ping the analyst assigned to that department.
~"Eng Metrics::Eng Division"
~"Eng Metrics::Development Department"
~"Eng Metrics::Infrastructure Department"
~"Eng Metrics::Quality Department"
~"Eng Metrics::Support Department"
~"Eng Metrics::Incubation Engineering"
|g_engineering_analytics||Channel to discuss team-related work, or for other people to see what's going on with the team & ask any questions related to the team|
|eng-data-kpi||Important Announcements related to Engineering Performance Indicators, & forum for external questions to team.|
|infrafin||For questions related specifically GitLab.com hosting cost or other general questions that overlap with infrastructure & finance|
|development_metrics||For questions regarding Development Department metrics|
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This is measured by Merge Requests that update the handbook contents related to the performance indicator pages, our team page, and engineering metrics pages.