| GitLab Team Handle | @gitlab-org/quality/engineering-productivity |
| Slack Channel | #g_engineering_productivity |
| Team Boards | Team Board & Priority Board |
| Issue Tracker | gitlab-org/quality/engineering-productivity/team |
Engineering productivity has monthly office hours on the 3rd Wednesday of the month at the 13:30 UTC (6:30 PST) on odd months (e.g January, March, etc) and 3:00 UTC (20:00 PST) on even months (e.g February, April, etc) open for anyone to add topics or questions to the agenda. Office hours can be found in the GitLab Team Meetings calendar
The Engineering Productivity team increases productivity of GitLab team members and contributors by shortening feedback loops and improving workflow efficiency for GitLab projects. The team uses a quantified approach to identify improvements and measure results of changes.
Reduce pipeline cost, reduce pipeline duration, and increase pipeline stability for GitLab projects focusing on projects with the largest reach, leveraging GitLab features where possible.
Improve engineering workflow automation to decrease Open MR Review Time (OMRT) and Open MR Age (OMA), and decrease open ~"type::bug" age.
Enable frequent and positive experience of Community Contributions from the Wider GitLab Community.
The Engineering Productivity team focuses on the following workstreams and the associated Epics with workstream specific vision and objectives.
| Tracking Label | Epics |
|---|---|
| ~"ep::pipeline" | GitLab Project Pipeline Improvement GitLab Project Selective Test Execution Measure and act on flaky specs |
| ~"ep::review-apps" | Improve Review Apps Reliability Improve Review Apps setup and usefulness |
| ~"ep::triage" | Quality: Triage |
| ~"ep::metrics" | Centralized handbook first metrics dashboard |
| ~"ep::workflow" | Reviewer Roulette Improvements |
#master-broken pipeline monitoring.Engineering Productivity team resides under the Quality Department operating as a team of Full-stack engineers, led by an Engineering Manager reporting to the Quality Department Leader.
| Person | Role |
|---|---|
| Greg Alfaro | GDK Project Stable Counterpart, Application Security |
Engineering Productivity has a weekly team meeting to give kudos, review announcements, discuss agenda items, and demo things to help with knowledge sharing.
Team members have the opportunity to present (5-7 minute) demos at the start of the weekly team meeting. The demo topic will be voted on in a Polly at #g_engineering_productivity. Each demo should focus on the What, Why, How and be led by a team member who is sharing their screen explaining the topic.
Showcases can be found in the Engineering Productivity showcase GitLab Unfiltered playlist.
The Engineering Productivity team uses modified prioritization and planning guidelines for targeting work within a Milestone.
The Engineering Productivity team creates metrics in the following sources to aid in operational reporting.
Exception Ratio: 2 Staff+ Engineering Team
Justification: Engineering Productivity has a wide focus to enable efficiency for GitLab code workflows. The team is implementing productivity improvements and globally optimizing for all workflows (GitLab, JiHu and Contributors). Staff+ team members focus on digging deep into feedback loop bottlenecks (Solver) and ensuring that the approach and implementation is scalable (Tech Lead) by working with counterparts in Development and Infrastructure departments.
Future Growth or Anticipated Change: It is expected that Engineering Productivity will maintain the current ratio during FY22 and re-evaluate in FY23.
The Engineering Productivity team will make changes which can create notification spikes or new behavior for GitLab contributors. The team will follow these guidelines in the spirit of GitLab's Internal Communication Guidelines.
Pipeline changes that have the potential to have an impact on the GitLab.com infrastructure should follow the Change Management process.
Pipeline changes that meet the following criteria must follow the Criticality 3 process:
cache-repo job jobThese kind of changes led to production issues in the past.
The team will communicate significant pipeline changes to #development in Slack and the Engineering Week in Review.
Pipeline changes that meet the following criteria will be communicated:
Other pipeline changes will be communicated based on the team's discretion.
Be sure to give a heads-up to #development,#eng-managers,#product, #ux Slack channels
and the Engineering week in review when an automation is expected to triage more
than 50 notifications or change policies that a large stakeholder group use (e.g. team-triage report).
This is a list of Engineering Productivity experiments where we identify an opportunity, form a hypothesis and experiment to test the hypothesis.
| Experiment | Status | Hypothesis | Feedback Issue or Findings |
|---|---|---|---|
| Always run minimal jobs for fork pipelines | Complete | The goal is to reduce the CI minutes consumed by fork pipelines. The "full" jobs only run for canonical pipelines (i.e. pipelines manually started by a member of the project) once the MR is approved. | |
| Retry failed specs in a new process after the initial run | Complete | Given that a lot of flaky tests are unreliable due to previous test which are affecting the global state, retrying only the failing specs in a new RSpec process should result in a better overall success rate. | https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148#note_914106156 |
| Experiment with automatically skipping identified flaky tests | Complete | Skipping flaky tests should reduce the number of false broken master and increase the master success rate. |
We found that this doesn't seem to have a negative impact on master stability so the experiment was made opt-out. |
| Experiment with running previously failed tests early | In Progress | Can iteration or cycle time be improved if failed tests are run earlier in the gitlab-org/gitlab pipeline? |
|
| Store/retrieve tests metadata in/from pages instead of artifacts | In Progress | We're only interested in the latest state of these files, so using Pages makes sense here. Also, this would simplify the logic to retrieve the reports and reduce the load on GitLab.com's infrastructure. | There are some transient problems where a Cloudflare page is returned instead of the expected JSON file. |
| Reduce pipeline cost by reducing number of rspec tests before MR approval | In Progress | Reduce the CI cost for GitLab pipelines by running the most applicable rspec tests for changes prior to approval | |
| Enabling developers to run failed specs locally | In Progress | Enabling developers to run failed specs locally will lead to less pipelines per merge request and improved productivity from being able to fix regressions more quickly | https://gitlab.com/gitlab-org/gitlab/-/issues/327660 |
| Use dynamic analysis to streamline test execution | Complete | Dynamic analysis can reduce the amount of specs that are needed for MR pipelines without causing significant disruption to master stability | Miss rate of 10% would cause a large impact to master stability. Look to leverage dynamic mapping with local developer tooling. Added documentation from the experiment. |
| Using timezone for Reviewer Roulette suggestions | Complete - Reverted | Using timezone in Reviewer Roulette suggestions will lead to a reduction in the mean time to merge | Reviewer Burden was inconsistently applied and specific reviewers were getting too many reviews compared to others. More details in the experiment issue and feedback issue |