Ops Sub-department

1. You are here:
2. Engineering
3. Development Department
4. Ops Sub-department

Maintained by:

On this page

• Teams and Handbook Pages
• Product Direction
• Performance Indicators
  • Dashboards
    • Cross-functional Backlog
    • Merged Merge Request Types
• Slack channels
• Meetings
  • Zoom Recordings synced to Google Drive
  • Ops Engineering Showcase
    • Quarterly Recordings
    • Monthly Recordings
• Planning Processes
  • Forced Priority planning/scheduling
• Ops Promotion Process
  • Promotion Document Reviews
  • Quarterly Promotion Review Meeting
• Ops Hiring process
• Objectives & Key Results (OKRs)
  • Planning schedule
  • Creating OKRs
  • Tracking and completing OKRs
• Async Updates (No Status In Meetings)
  • Async Update Guidance
    • Engineering Manager
    • Senior Engineering Manager
    • Principal Engineer
    • Director
  • To create a weekly async update:
  • To HIGHLIGHT an item for inclusion in the Director update
  • To review Ops weekly async updates:
  • Benefits of reporting and reviewing status async vs. synchronously in meetings:
• Escalations from Support Team

The Ops Sub-department is composed of development teams working on Verify, Package, Release, Configure, and Monitor features of GitLab DevOps Platform.

Teams and Handbook Pages

The following teams comprise the Ops sub-department:

• Verify stage - handbook
  • Pipeline Execution - handbook, jtbd
  • Pipeline Authoring - handbook, jtbd
  • Pipeline Security - handbook, jtbd
  • Runner - handbook, jtbd
• Package stage - handbook, jtbd
  • Container Registry - handbook
  • Package Registry - handbook
• Deploy stage
  • Environments - handbook, jtbd
• Monitor stage - handbook
  • Monitor:Respond - handbook, jtbd
  • Monitor:Observability - handbook, jtbd

Product Direction

Product direction can be found on the Ops Section Product Direction handbook page.

Performance Indicators

• Ops Sub-department Performance Indicators
• Error Budgets as Performance Indicators for stage groups

Dashboards

Cross-functional Backlog

(Sisense↗) We also track our backlog of issues, including past due security and infradev issues, and total open SUS-impacting issues and bugs.

Merged Merge Request Types

(Sisense↗) MR Type labels help us report what we're working on to industry analysts in a way that's consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.

Slack channels

• #ops-section - Ops Section discussions spanning the Verify, Package, Release, Configure, and Monitor Stages
• #ops_staff_plus - Foster collaboration and technical leadership within Ops section
• #doe-ops - Channel for GitLab's Director of Engineering, Ops
• 🔒ops-staff-confidential - Channel for reviewing Ops promotion docs or sharing other confidential information. Please use #doe-ops or #ops-section for topics that don’t require confidentiality.

Meetings

Zoom Recordings synced to Google Drive

GitLab encourages transparency by default, and when meetings are recorded, they can be automatically synced to Google Drive as highlighted in the section about how we conduct video calls at GitLab. To summarize:

1. Record meetings to the cloud in Zoom, and add the text [REC] anywhere in the meeting title (e.g. Google calendar event)
2. It will automatically be added to the GitLab Videos Recorded folder on Google Drive, with a naming convention of youremail-Meeting Title.
3. After a meeting ends, Zoom may take some time to process the recording before it is actually available. The sync to Google Drive happens on the hour mark, so if the recording is not available, it may take another hour to be transferred.
4. Future recordings of recurring meetings can be found in the same folder youremail-Meeting Title, even if alternate hosts start the recording.

Ops Engineering Showcase

Starting in 2023 in FY24-Q1, a fresh new format for this showcase will include the recording and sharing of videos that team members have created. Once a video has been created, an async discussion issue should be created using this template and listed under the relevant quarterly epic. Based on the amount of questions being discussed in the issue, an AMA may be set up to discuss further in a group setting. Like before, these videos are intended to be opportunities for engineers to show off the things they've been working on.

Beginning in March 2023 all Ops teams are required to provide at least one showcase video and async discussion issue per month. We require participation in this process because it helps reinforce our values of Results, Transparency, and Iteration. In addition, these showcases are valuable in helping team members across the company better understand the progress and obstacles across the many product areas Ops team members are focused on. They are also fun to watch. These videos will be reviewed monthly by the Director of Engineering and other leaders in the group. The team's Engineering Manager is the DRI for submitting the showcase, though in many cases the demo may be delegated to another member of the team.

Team members are encouraged to use these videos to transparently showcase the current state of their projects. It is not encouraged to spend significant time prepping for these demos or only showcase finished, polished functionality; showing functionality in an incomplete state is encouraged. Videos may be specifically recorded for this purpose or be excerpts from team meetings, etc. Some suggested ideas to consider could be related to error budget improvements, performance improvements to current features, refactoring to codebase for future maintainability, etc.

Ops Engineering Showcase Epic

Quarterly Recordings

Monthly Recordings

Planning Processes

Planning processes followed by teams in the Sub-department:

• General Ops Sub-department Planning Process

Forced Priority planning/scheduling

Our prioritization framework describes a number of Forced Priority labels that present a high risk to our customers and our business. As such, it's critical that we complete this work within the appropriate SLO/SLAs where possible. An example of this are security issues, here's a dashboard for past due and soon to be due security issues: https://app.periscopedata.com/app/gitlab/913607/Past-Due-Security-Issues-Development

With this in mind, here's a process that groups are encouraged to use and iterate on in the event that they don't have an existing approach or are falling behind on SLA/SLOs.

To be clear, this is not a required process, if your group's approach is working, please continue with that! Ownership of scheduling forced priority work falls on our whole team, not a particular person.

1. When forced priority issues are assigned to your group, schedule those issues right away regardless of severity label
2. Have the team review the issue to make sure there's a viable solution and plan (weight the issue)
3. Have someone in the group assign themselves to work on the issue
4. Update the issue regularly with progress
5. Treat blockers with the same or higher priority than the original issue

Ops Promotion Process

Promotion Document Reviews

1. Team Member and Manager work together to create promotion document.
2. Promotion document is shared with Ops Sub-Department managers in #ops-staff-confidential channel (private to Ops Managers, Senior Managers, Director) to receive feedback on the document content and how it calibrates with others in the Sub-Department. This can be done early and often.
3. Manager requests another Manager in the Sub-Department review and approve the document.
4. Manager requests Senior Manager in the Sub-Department review and approve the document.
5. Manager requests Director add Team Member to list of Sub-Department planned promotions. If the team member cannot be added Director will provide feedback to Manager as to why.
6. Director presents promotion doc at Development Department quarterly promotion review meeting. If the promotion is not approved Director will provide feedback to Manager as to why.

Quarterly Promotion Review Meeting

Ops Managers, Sr. Managers, and Directors meet in the last month of each quarter to:

1. Review promotion projections for each group. Managers add a list of their direct reports who they anticipate will be ready for promotion in the next 3 quarters to the meeting agenda. Note these lists may change and are not a guarantee a team member will be promoted in a specific quarter. The lists are used for planning purposes, to help us anticipate the number of team members approaching promotion, budget impact, and to support calibration across the group. These lists should include:
   1. Team Member name
   2. Targeted promotion quarter
   3. Link to promotion doc
   4. Whether the doc has received any Ops Manager or Ops Senior Manager approvals.
2. Review Ops team member promotion docs planned to be presented at Development Department quarterly promotion review meeting.

Ops Hiring process

We have limited capacity in the recruiting team so hiring managers are experimenting with the Ops Hiring process. We also hold a weekly hiring manager sync meeting (see Meetings section).

Objectives & Key Results (OKRs)

Planning schedule

Creating OKRs

• Target a 70% completion rate for OKRs so that we are setting goals ambitiously. They should be challenging enough that accomplishing 100% is a stretch goal,since GitLab considers OKRs to be stretch goals by default.
• If your OKRs are associated with any specific types of issues, consider reviewing the prioritization framework, and align with your team on your OKRs priority relative to one another (for example, OKRs with issues with "forced prioritization") and be clear on planning of the associated work over the quarter.
• Ensure Key Results (KRs) are measurable, by specifying a number (% or unit) as the target. Be clear on the starting number, and the expected end result (e.g. from x to y).
• Ideally create no more than 9 KRs per quarter, ideally with approximately 3 KR per objective.

Tracking and completing OKRs

• Tracking quarterly OKRs for Ops Engineering groups.
• DRIs are expected to score OKRs on a regular basis, at least monthly, if possible.
• At the end of the quarter, DRIs can add a retrospective as a comment to reflect on the progress of their KR, in the following format:

**Good**
- Things that went well...

**Bad**
- Things that didn't go so well...

**Try**
- What might you do differently next time?

Async Updates (No Status In Meetings)

We have a policy in Ops to keep status updates out of meetings. Instead of reporting on status in meetings Directors, Senior Engineering Managers, Engineering Managers and Principal Engineers provide regular async updates.

The content of these updates varies by individual and role:

Async Update Guidance

Engineering Manager

1. Cadence: Weekly
2. Topics:
   1. Highlights & Accomplishments
   2. Team Projects & Priorities (progress, blockers, key decisions, etc.)
   3. Team Health (team dynamic, team morale, hiring, team concerns, etc.)
   4. Team Metrics and SLAs (e.g. Security SLAs, etc.)
   5. Other Announcements & topics of interest to those reading the report

Senior Engineering Manager

1. Cadence: Monthly
2. Topics:
   1. Milestone Review
      1. Allocation of focus in milestone plans (major features, maintenance, etc.)
      2. What tradeoffs or decisions should be made prior to Milestone kickoff? For example:
         1. Is the team overcommitted?
         2. Does the allocation of effort need to shift to meet specific goals? (e.g. Security SLAs)

Principal Engineer

1. Cadence: Weekly
2. Topics:
   1. Highlights & Accomplishments
   2. Projects & Priorities (progress, blockers, key decisions, etc.)

Director

1. Cadence: Monthly
2. Topics:
   1. Summary of Highlights & Accomplishments across the section

To create a weekly async update:

1. Create a GitLab issue in any project.
2. Apply the label ~OpsSection::AsyncUpdate. (The label ~OpsSection::Weekly-Update can also be used for backwards compatibility reasons.)
3. Choose a descriptive title. Include your update in the issue description.

To HIGHLIGHT an item for inclusion in the Director update

1. Begin any paragraph in your update with the text HIGHLIGHT:
2. These paragraphs will automatically be copied into the Highlights & Accomplishments section of the monthly Director's update.

To review Ops weekly async updates:

1. All issues with the ~OpsSection::AsyncUpdate or ~OpsSection::Weekly-Update label are summarized in weekly issues in the Ops Status Updates project.
2. All team members are encouraged to comment and ask questions in weekly async update issues. This provides useful feedback to the author and opportunities to collaborate.
3. Director reviews all updates weekly (from the prior week) and provides feedback.
4. Senior EMs reviews all updates from their teams weekly (from the prior week) and provides feedback.

Benefits of reporting and reviewing status async vs. synchronously in meetings:

1. Collaboration - By sharing announcements, achievements, goals, and plans we are empowering team members with the information they need to make sound decisions and succeed in their roles.
2. Results - Async updates provide an important channel for sharing accomplishments and progress.
3. Efficiency - Expecting each team member to poll for the status they need is extremely inefficient and becomes more inefficient as the organization grows. By using a Publish Subscribe model we are able to share information much more efficiently.
4. Diversity, Inclusion & Belonging - Async status is more inclusive of team members across timezones. Not every team member will be able to make a sync meeting (due to timezone or other factors) but everyone can participate in an async update.
5. Iteration - Taking the time to checkpoint where we are at each week prompts the author and readers to consider ways to iterate and to reflect on incremental progress.
6. Transparency - Sharing information in a consistent, accessible way increases transparency and reduces the threshold for team members to contribute.

Escalations from Support Team

We are piloting a process to make coordinating with the Support team on customer escalations more efficient.

See How to Use GitLab.com to Formally Request Help from the GitLab Ops Development Team for process details.

If you encounter an Ops-related customer escalation, that seems to have a high customer priority, and would benefit from additional visibility, please post a link with a short description in #doe-ops Slack channel.