The Ops Sub-department is composed of development teams working on Verify, Package, Deploy, Sevice Management and Observability features of GitLab DevOps Platform.
The following teams comprise the Ops sub-department:
Teams in the Ops Sub-Department map to several Product Sections. Product direction can be found on the following direction pages:
GitLab encourages transparency by default, and when meetings are recorded, they can be automatically synced to Google Drive as highlighted in the section about how we conduct video calls at GitLab. To summarize:
youremail-Meeting Title
.youremail-Meeting Title
, even if alternate hosts start the recording.Starting in 2023 in FY24-Q1, a fresh new format for this showcase will include the recording and sharing of videos that team members have created. Once a video has been created, an async discussion issue should be created using this template and listed under the relevant quarterly epic. Based on the amount of questions being discussed in the issue, an AMA may be set up to discuss further in a group setting. Like before, these videos are intended to be opportunities for engineers to show off the things they've been working on.
Beginning in March 2023 all Ops teams are required to provide at least one showcase video and async discussion issue per month. We require participation in this process because it helps reinforce our values of Results, Transparency, and Iteration. In addition, these showcases are valuable in helping team members across the company better understand the progress and obstacles across the many product areas Ops team members are focused on. They are also fun to watch. These videos will be reviewed monthly by the Director of Engineering and other leaders in the group. The team's Engineering Manager is the DRI for submitting the showcase, though in many cases the demo may be delegated to another member of the team.
Team members are encouraged to use these videos to transparently showcase the current state of their projects. It is not encouraged to spend significant time prepping for these demos or only showcase finished, polished functionality; showing functionality in an incomplete state is encouraged. Videos may be specifically recorded for this purpose or be excerpts from team meetings, etc. Some suggested ideas to consider could be related to error budget improvements, performance improvements to current features, refactoring to codebase for future maintainability, etc.
NOTE: - changed to monthly recordings in FY24-Q3
.
Year | Quarter | Links |
---|---|---|
2023 | FY24-Q3 | August, September |
2023 | FY24-Q2 | Videos |
2023 | FY24-Q1 | Videos |
Year | Month | AMEA | APAC | Async videos |
---|---|---|---|---|
2022 | December | No showcases | No showcases | Â |
2022 | November | No showcases | No showcases | Â |
2022 | October | Split GitLab Monolith into components | No showcases | TaskScaler 101 |
2022 | September | Import packages with a CI pipeline | No showcases | Â |
2022 | August | No showcases | Async video -> | Leader election in GitLab Agent for Kubernetes |
2022 | July | Video | No showcases | MR Coverage Report & CI Workflows |
2022 | June | Video | Video | Demo |
2022 | May | Video | No showcases | Â |
2022 | April | Video | Video | Â |
2022 | March | Video | No showcases | Â |
2022 | February | Video | No showcases | Â |
2022 | January | Video | No showcases | Jobs UI Refactor |
2021 | December | Video | No showcases | Â |
2021 | November | Video | No showcases | Â |
2021 | October | Video | Video | Â |
2020 | November | Video | Video | Â |
2020 | October | Video | Skipped | Â |
2020 | September | Video | Video | Â |
2020 | August | Video | Video | Â |
Planning processes followed by teams in the Sub-department:
Our prioritization framework describes a number of Forced Priority labels that present a high risk to our customers and our business. As such, it's critical that we complete this work within the appropriate SLO/SLAs where possible. An example of this are security issues, here's a dashboard for past due and soon to be due security issues: https://app.periscopedata.com/app/gitlab/913607/Past-Due-Security-Issues-Development
With this in mind, here's a process that groups are encouraged to use and iterate on in the event that they don't have an existing approach or are falling behind on SLA/SLOs.
To be clear, this is not a required process, if your group's approach is working, please continue with that! Ownership of scheduling forced priority work falls on our whole team, not a particular person.
Ops Managers, Sr. Managers, and Directors meet in the last month of each quarter to:
We have limited capacity in the recruiting team so hiring managers are experimenting with the Ops Hiring process. We also hold a weekly hiring manager sync meeting (see Meetings section).
from x to y
).**Good**
- Things that went well...
**Bad**
- Things that didn't go so well...
**Try**
- What might you do differently next time?
For projects that span many milestones we are experimenting with additional project management processes.
Key projects will be listed in the table below:
Project Name | Link to Project Plan | DRI |
---|---|---|
CI Catalog | Project plan | Mark Nuzzo |
CI Data Partitioning | Project plan | Caroline Simpson |
Merge Train Improvements | Project plan | Caroline Simpson |
Container Registry v2 Self-Managed | Project plan | Crystal Poole |
CI Secrets Management | Project plan | Scott Hampton |
O11y Distributed Tracing | Project plan | Nicholas Klick |
Cluster Web Terminal | Project plan | Nicolò Maria Mezzopera |
Autoscaling for GitLab Runner | Project Plan | Nicole Williams |
Projects which are listed as key projects should create these project planning artifacts:
/engineering/development/ops/project-plans/<project_name>/index.html.md.erb
Since the project plan is stored in the handbook it is easy to track changes in the project plan and see how it evolves over time.
Project plans should (roughly) follow this format (from https://www.rubick.com/weekly-project-plans/):
Weekly project plan template
Week of Jan 4th
Single chart shows up in Slack. Data is canned.
Schedule risk: we’re validating our list of chart types are all technically feasible. We’ll demo outcome of that investigation.
Week of Jan 11th
Chart data reflects live information, and is functional in Slack chart.
Additional chart type shows in Slack room, with most basic visual design.
We’ve shown to at least one alpha customer for feedback. We start sharing with them every week from here on out.
Jessica is on-call and doing interrupt-driven work for week.
Week of Jan 18th
Most important feedback from alpha customers incorporated. Other work is prioritized to future milestones.
Last chart type added.
Week of Jan 25th
Holiday Jan 26th.
Charts look great and are thoroughly tested, instrumented. We’ll show usage dashboards.
Release end of week.
Updates to the plan should be made weekly, marking weekly items done and making updates based on current information. Authors should merge updates to their project plans without review/approval.
We expect the use of project plans to reduce the amount of EM reporting by serving as a SSOT for project execution planning. For example this info no longer needs to be included in async updates (replace with a link to project plan).
We hold periodic sync meetings to review key projects and collaborate on the plans for these important deliverables.
Key Project DRIs should prepare for the meeting by creating a Key Project Review Prep Issue. This should be completed and shared at least 48 hours in advance of the meeting to allow time for other team members to review the content.
The objective is to keep these meetings focused on execution, technical planning, and accountability within the Engineering group. These are open to anyone who wishes to attend. When it's necessary to involve a broader group in some cases, we can plan additional follow ups case by case.
We aim to meet synchronously where possible given the benefits of being able to talk through the project in a higher bandwidth context. Rescheduling is preferred if the proposed dates are not feasible for attendees. We could re-evaluate whether we can achieve the same goals with async after we have more experience running the process.
The focus of the meeting time will be on topics such as:
Planned review meetings are listed in the table below:
Date | Project |
2023-09-21 | O11y Distributed Tracing |
2023-09-28 | CI Catalog |
2023-10-05 | CI Data Partitioning |
2023-10-12 | CI Secrets Management |
2023-10-19 | Cluster Web Terminal |
2023-10-26 | Container Registry v2 Self-Managed |
2023-11-02 | Merge Train Improvements |
2023-11-09 | O11y Distributed Tracing |
2023-11-16 | CI Catalog |
2023-11-23 | No meeting due to US Thanksgiving Holiday |
2023-11-30 | CI Data Partitioning |
2023-12-07 | CI Secrets Management |
2023-12-14 | TBD |
2023-12-21 | TBD |
2023-12-28 | No meeting due to holidays |
We have a policy in Ops to keep status updates out of meetings. Instead of reporting on status in meetings Directors, Senior Engineering Managers, Engineering Managers and Principal Engineers provide regular async updates.
The content of these updates varies by individual and role:
~OpsSection::AsyncUpdate
. (The label ~OpsSection::Weekly-Update
can also be used for backwards compatibility reasons.)HIGHLIGHT:
~OpsSection::AsyncUpdate
or ~OpsSection::Weekly-Update
label are summarized in weekly issues in the Ops Status Updates project.We are piloting a process to make coordinating with the Support team on customer escalations more efficient.
See How to Use GitLab.com to Formally Request Help from the GitLab Ops Development Team for process details.
If you encounter an Ops-related customer escalation, that seems to have a high
customer priority,
and would benefit from additional visibility, please post a link with a short description in
#doe-ops
Slack channel.
When teams are asked to triage support requests for customers on GitLab Dedicated and engineers need access to their logs, a sync can be arranged between the engineer and the assigned Support Engineer for a screensharing session. However, when further troubleshooting is needed by the engineering team or async collaboration is preferred, follow these steps to request access:
Manager Approval
on behalf of the engineers)