Maintained by:

Direction

Vision

Using GitLab, you automatically get broad and deep insight into the health of your deployment.

Mission

We provide a robust monitoring solution to give GitLab users insight into the performance and availability of their deployments and alert them to problems as soon as they arise. We provide data that is easy to digest and to relate to other features in GitLab. With every piece of the devops lifecycle integrated into GitLab, we have a unique opportunity to closely tie our monitoring features to all of the other pieces of the devops flow.

We work collaboratively and transparently and we will contribute as much of our work as possible back to the open-source community.

Responsibilities

The monitoring team is responsible for:

Providing the tools required to enable monitoring of GitLab.com
Packaging these tools to enable all customers to manage their instances easily and completely
Building integrated monitoring solutions for customers apps into GitLab, including metrics, logging, and tracing

This stage consists of the following groups:

APM
Health

These groups map to the Monitor Stage product category.

How to be successful in this stage

Team members who are successful in this stage typically demonstrate stakeholder mentality. There are many ways to demonstrate this but examples include:

Actively contributes to the discussion and direction of where the product is headed
Actively reaching out for help when stuck
Actively finding ways to make the team successful

This stage is only successful when each team member collaborates to make one another successful.

Communication

Slack Channels:
- #s_monitor
- #s_monitor_social
All of our team meetings are on our Monitor Stage team calendar
Our Google groups are organized like this:
- Monitor Stage
  - Monitor APM
    - monitor-apm-be
    - monitor-apm-fe
  - Monitor Health
    - monitor-apm-be
    - monitor-apm-fe

Rhythms

Monthly Cadence

Since GitLab releases on a monthly basis, we have supporting activities that also take place on monthly rhythms. In addition, since our releases take place on the 22nd of each month, each monthly cadence does not map to actual months of the Gregorian calendar. These are listed in an ordered list for ease of reference.

We manually create our planning issue for the next milestone as soon as we start a milestone
- Everyone in the team is encouraged to participate in this issue
- Our planning issue is a conversation about the planning priorities of the team. The PM, EM, and UX are responsible for narrowing down the scope based on the capacity of the team.
- An opportunity for the team to bring up any issues they would like to work on even if it isn’t directly related to the stage so that the PM can determine how we can balance work that benefits the entire company vs the stage (as well as technical debt).
- One week prior to the last date of the milestone, the PM will record a kickoff recording of the issues intended to be completed in the next milestone (as prescribed in the planning issue).
We automatically create a retrospective issue for the current milestone when we are halfway through a milestone using the async-retrospective.
- Everyone in the team is encouraged to participate in this issue
- Issues are confidential to the team
- Engineering Managers for both backend and frontend are responsible for bubbling up content and themes to the main public retrospective document
PM creates a release post item merge request for each feature that will be available in the next release so that these features can be displayed in the release post on the 22nd.
- Team members working on a feature are tagged in this MR but are not required to take any action
- Engineering Managers follow the checklist in the MR template and are responsible for merging the MR once the feature has been merged to master and (ideally) verified in production
- The backend engineering manager is the primary DRI for merging the release post MR. If the backend engineering manager is unavailable, the frontend engineering manager becomes the DRI.
We consider the 18th of the month (or if that day is on a weekend, the last weekday before that day) to be the last date of the milestone.
- Deliverables that are not closed by the 18th will have their milestones moved to the next milestone and marked missed-deliverable
- Assigned filler issues that are workflow::in dev, workflow::in review or workflow::verification will be moved to the next milestone
- Filler issues that do not fall into the above categories will be moved to the backlog - After all the issues associated to that milestone are moved, we remove that milestone column from their planning boards (APM, Health)
A few days prior to the last date of the milestone, engineers are assigned to Deliverables.
- Engineering managers will assign engineers to deliverables
- Assignment of deliverables help set a clear DRI
- If engineers notice that their deliverables are unlikely to make the milestone, they are responsible for communicating that in the issue, their EM and the PM
- We highly encourage engineers who finish their deliverables to notice if there are ways they can help other engineers with their deliverables (that would otherwise slip)

Meetings

Meetings are not required but attendance/reviewing the recordings to the important ones will generally make team members successful. These are ordered in order of importance and are all stored in the Monitor Stage Calendar(Viewable to all GitLab team members)

Group Weekly Meeting (Monitor:Health and Monitor:APM each have their own)
Monitor Social Hour (weekly)

Async Daily Standups

Groups in this stage also participate in async daily standups. The purpose is to give every team member insight into what others are working on so that we can identify ways to collaborate and unblock one another as well as foster relationships within the team. We use the geekbot slack plugin to automate our async standup, following the guidelines outlined in the Geekbot commands guide.

Our questions change depending on the day of the week. Participation is optional but encouraged.

Monday

Question	Why we ask it
Do you need help from anyone to unblock you this week?	One of our main goals with our standups is to help ensure that we are unblocking one another as a top priority. We ask this first because we think it's the question that other team members can take action on.
What do you plan on working on this week?	We want to understand how our daily actions drive us toward our weekly goals. This question provides a broader context for our daily work but also helps us hold ourselves accountable to maintaining proper scopes for our tasks, issues, merge requests, etc. This answer may stay the same for a week, this would mean things are progressing on schedule. Alternatively, seeing this answer change throughout the week is also okay. Maybe we got sidetracked helping someone get unblocked. Maybe new blockers came up. The intention is not to have to justify our actions but to keep a running record of how our work is progressing or evolving.
Any personal tidbits you'd like to share?	This question is intentionally open-ended. You might want to share how you feel, a personal anecdote, funny joke, or simply let the team know that you will have limited availability that afternoon. All of these answers are welcome.

Tuesday/Wednesday/Thursday

Question	Why we ask it
Are you facing any blockers requiring action from others?	Same reason as Monday's first question
Are you on track with your plan for the week?	We want to understand how each team member is doing on achieving our week goal(s). It is meant to highlight progress while also identifying if there are things getting in the way. This could also be used to update the plan for the week as things change.
What will be your primary focus for today?	This question is aimed at the most impactful task for the day. We aren't trying to account for the entire day's worth of work. Highlighting only a primary task keeps our answers concise and provides insight into each team member's most important priority. This doesn't necessarily mean sharing the task that will take the most time. We focus on results over input. Typically this will mean highlighting the task that is most impactful in closing the gap between today and our end of the week goal(s).
Any personal tidbits you'd like to share?	Same reason as Monday's last question

Friday

Question	Why we ask it
What went well this week? What did you enjoy?	The end of the week is a good time to reflect on our goals, and this question is meant to be a short retrospective of the week. This focusing on things that went well during the week.
What didn’t go so well? What caused you to slow down?	Like the previous question, this question is a way to review our week. This one is a way to surface things that did not go so well or things that go in the way of meeting our weekly goal(s).
What have you learned?	This could be something about work or personal. We hope that by sharing things we have learned that others can also learn from us.
Any plans for the weekend you'd like to share?	Like the "personal tidbit" question we ask other days of the week, this one is very opened ended. You can share as much or as little as you want and all answers are welcome.

Initiatives

Code ownership expression

To emphasize code ownership (which is aligned with organization-wide efforts) and help identify domain expertise for parts of the codebase that are developed by the Monitor stage. We place every new feature into a dedicated top-level namespace. Names used for these namespaces are derived from product categories. Such an approach helps us to communicate better not only between the engineers but also with other peers since used terms are well known across the organization.

Special cases

Code placed under lib/ directory, in this case, all code which can not be run alone outside of the GitLab repository, should be at first placed under Gitlab top-level namespace, and after that in the product category subnamespace.
Because Rails framework defines routing based on controllers namespaces using product category as a top-level namespace can mislead users that those features are independent of Project or Group scope which is rarely a case. To address that one should place controller accordingly to the feature scope (Project or Group) and then in category subnamespace.
API entries should follow general layer separation convention same as models, controllers, etc do, therefore API code should be placed under lib/api directory. That enforces this code will have API top-level namespace, and after that should come product category subnamespace.

Examples

Cases where namespace was selected properly

Worker: IncidentManagement::ProcessAlertWorker was properly placed under IncidentManagement namespace which relates to Incident Management product category.
Model: Metrics::UsersStarredDashboard was properly placed under Metrics namespace which relates to Metrics product category.
Model: Metrics::Dashboard::Annotation was properly placed under Metrics namespace which relates to Metrics product category.
Grape API instance: API::Metrics::UsersStarredDashboard was properly placed in lib/api dicrectory under API top-level namespace and than in Metrics subnamespace which relates to Metrics product category.

Cases where better namespace could be selected

Gitlab::Prometheus::MetricGroup should be named Gitlab::Metrics::Prometheus::MetricGroup and placed at lib/gitlab/metrics/prometheus/metric_group.rb
Projects::PerformanceMonitoring::DashboardsController should be named Projects::Metrics::DashboardsController and placed at app/controllers/projects/metrics/dashboards_controller.rb
Model added as part of Annotations feature should be named ::Metrics::Dashboard::Annotation instead of DashboardAnnotation or MetricsDashboardAnnotation and respectively it should be placed at app/models/metrics/dashboard/annotation.rb instead of metrics_dashboard_annotation.rb
Prometheus::ProxyVariableSubstitutionService according to this guidance should be named Metrics::Prometheus::ProxyVariableSubstitutionService and placed at services/metrics/prometheus/proxy_variable_substitution_service.rb
Prometheus::CreateDefaultAlertsWorker should be named IncidentManagement::Prometheus::CreateDefaultAlertsWorker and placed at workers/incident_management/prometheus/create_default_alerts_worker.rb

Spike

Spikes are time-boxed investigations typically performed in agile software development. Groups in the monitor stage typically create Spike issues when there is uncertainty on how to proceed on a feature from a technical perspective before a feature is developed.

Guidelines

Spikes are typically marked as deliverable to ensure clear ownership from engineers
Spikes are typically time-boxed to a short duration, sometimes one week, never more than one milestone.
Limit no more than 2 Spikes per milestone per group. Typically multiple team members collaborate on a spike. We want to ensure we get multiple different viewpoints as well as keep the research focused and efficient.
Engineers should consider edge cases in spikes but should not seek to solve every intricacy of every edge case
Engineers should not be breaking the technical solution down into multiple iterations and issues until the recommended approach is given the go-ahead by the engineering manager

Workflow labels for spikes

Spikes should start with workflow::ready for development
When engineers are actively working on the spike, they should change the workflow label to workflow::in dev
After engineer(s) have completed their evaluation and are proposing their recommendation to the team, they should change the workflow label to workflow::in review
After the recommended approach is given the go-ahead, the engineer(s) should change the workflow label to workflow::verification and close the issue

Expectations

Engineer(s) assigned to the Spike issue will be responsible for the following tasks:

Documenting the following (in the issue):
- What were the technical implementation options for the feature and their tradeoffs?
- What is the recommended approach (and why is it recommended)?
Add to the group weekly agenda and summarize the investigation of the Spike including the recommendation so that other engineers can contribute to the conversation and direction
Engineering managers are the DRI for making the decision about which approach to move forward with
After a decision is made, engineers assigned to the Spike will update the spike issue description with the decision and create the appropriate follow up issues to implement the solution before closing the Spike issue. The PM will be the DRI for scheduling those issues.

SRE Shadow Program

With the support of GitLab's SRE team, we implemented the SRE shadow program as a means of improving the team's understanding of our ideal user personas so that we can build a better product.

In this program, engineers are expected to devote 1 entire week to shadow SREs. There is no expectation for the engineer to complete their assigned issues during this time. Engineers are added to PagerDuty and will follow the existing SRE shadow format of interning (except scaled down to a shorter duration of 1 week). Although typical SREs on-call for multiple days at a time, shadows are only expected to shadow during their regular business hours. This can be set as a preference in PagerDuty.

Objectives

Gain empathy for our user persona
Observe pain points in the current SRE workflow so that we can improve it
Observe ways the SRE team can dogfood more features
Document observations in some medium that allows non-shadows to learn (Eg. blog post, Q&A session..etc)

Outcomes

Engineers gain a better understanding of the users we build features for.
Engineers become better stakeholders for the stage.
- They are able to create more feature proposals to help the stage build features that improve the lives of our user personas.
- They are more equipped to influence product direction based on their observation of what is better for our user personas.
Engineers develop stronger relationships with the SRE team.
- Enables improved collaboration and efficiency in dogfooding features and getting faster feedback cycles for our features.

How to participate

Team members interested in the program should notify their respective managers. We are currently limited to 2 max shadows per release so that we do not overload the SRE team. If you are shadowing during the same release as another engineer, coordinate to create a combined access request for the duration of the release.

The participant's manager and the Engineering Manager, Reliability Engineering should collaborate with you to define a schedule in the Slack channel #monitor-sre-shadow. You can either check PagerDuty schedules or coordinate with the SRE manager to figure out who you'll be shadowing. Your schedule should include the week and working hours of your on-call shift, along with which SRE team member(s) will be on-call at the same time.

The participant should create an access request for PagerDuty and assign the access request to the SRE manager (this is a departure from established processes). PagerDuty licenses are limited so previous participants in the program may have to relinquish their licenses to you. Once PagerDuty access is set up, and you have been assigned to SRE engineer, you should override shadow PagerDuty schedule and assign yourself to corresponding shift, you can add contact details like a mobile number so PagerDuty can send alerts to you.

Typically, shadowing an SRE involves activities such as paying attention to SRE slack channels (#production, #incident-management), reading through incident issues posted there, and jumping into the 'The Situation Room' posted at the top of incident management for any active issues where the on-call SRE joins that room. About one week before starting your rotation you should get familiar with the Runbooks README and coordinate with the SRE(s) who will be on-call to determine which areas it makes sense for you to shadow (incidents, other on-call tasks, SRE daily tasks, etc). Scheduling a coffee chat with the SRE manager and/or SRE team members is recommended.

Please ask to be removed from PagerDuty after your shadow rotation to free up your license.

Alumni

Alumni of the program are encouraged to add themselves to this list and document/link to the observations/outcomes they were able to share with the wider team.

Name	Outcomes
Laura Montemayor	Shadowing a Site Reliability Engineer
Tristan Read	My week shadowing a GitLab Site Reliability Engineer
Sarah Yasonik	Created 4 issues for the team to consider adding to the product
Miguel Rincon	Video: Miguel talks about his SRE shadow experience
Olena Horal-Koretska	7 things I’ve learnt while shadowing an SRE

Resources

Demo Environments

In order to make it more efficient to verify changes and demonstrate our product features to customers and other stakeholders. The engineers in this stage maintain a few demo environments.

Use Case	URL
Customer simulation environment	tanuki-inc
Verifying features in Staging	monitor-sandbox (Staging)
Verifying features in Production	monitor-sandbox (Production)

Kubernetes clusters

To be able to test logging features in both the elastic stack enabled and Kubernetes only cases, the following clusters and environments exist in production and staging:

	Instance: gitlab.com Project: tanuki-inc	Instance: staging.gitlab.com Project: monitor-sandbox
Elastic Stack ON 3 nodes `n1-standard-2`	Cluster: `ops-demo-app` Environment: `production`	Cluster: `monitor-sandbox-staging` Environment: `production`
Elastic Stack OFF 1 node `g1-small`	Cluster: `ops-demo-app-no-elastic` Environment: `staging`	Cluster: `monitor-sandbox-staging-no-elastic` Environment: `staging`

Monitor Stage