As the Communications Manager on Call (CMOC) it's your job to be the voice of GitLab to our users and stakeholders during an incident. To do this effectively you'll use a combination of our status page (powered by Status.io), Slack, Zendesk, and GitLab itself. The CMOC rotation is one of the rotations that make up GitLab Support On-call.
Our Slack bot Woodhouse provides a command (/incident post-statuspage) to quickly spin up an incident on Status.io. From there, the basics of how to update and close incidents in Status.io are covered by their Incident Overview documentation. This document covers how GitLab specifically uses Status.io to perform those tasks.
Success as a CMOC is determined by the following performance indicators:
Before getting started, take note of the following sections or to get straight into the workflow start at Incident Management.
This section contains information specific to how incidents are started, what various status messages in PagerDuty mean, and the difference between the EOC and the Incident Manager during an incident.
Infrastructure uses Woodhouse to declare incidents through Slack. Doing so will:
This information will all be posted to Slack in the #incident-management channel by Woodhouse and it'll look similar to the following example.
GitLab team members are encouraged to use this method of reporting incidents if they suspect GitLab.com is about to face one.
NB: "Resolved" in PagerDuty does not mean the underlying issue has been resolved.
In later sections of this workflow it's called out that at times you should be asking the Incident Manager (IM) assigned to the incident for permission to move it between stages (open, monitoring, resolved). On the rare occasion that an incident does not have an Incident Manager, the Engineer On Call (EOC) assumes those responsibilities and you may ask them instead.
You can always review past incidents if you need examples or inspiration for how to fill in the details for a current incident.
Status.io should be updated whenever we have new information about an active incident that our stakeholders should be aware of. Outside of that, it should be updated at a consistent rate depending on the severity of the incident as outlined in the table below.
Once you join the incident Zoom call, take note of any updates that have been made to Status.io and the time they were made at. Set a timer to remind yourself and stick to the time intervals below unless you make a note of how long it will be until the next status update. For example, if you're in "monitoring" it may be appropriate to specify an hour before the next update.
| Incident Status | Severity 1 Update Frequency | Severity 2 Update Frequency | Severity 3/Severity 4 Update Frequency |
|---|---|---|---|
| Investigating | 10m | 15m | 15m |
| Identified | 10m | 30m | 30m |
| Monitoring | 30m | 60m | 60m |
| Resolved | No further updates required |
Provide a generic update based on the best information you have:
Craft a draft of what you think is correct. Whenever possible use "I intend to…" language when communicating with the Incident Manager and EOCs:
Bias to action - you can post another update if there was an error in your last update.
In some circumstances, the Incident Manager may ask you to find the number of tickets that an incident may have raised in order to evaluate the impact of the incident.
Because the default views will only show unassigned tickets in your region, start with using this Zendesk Search to find all recent tickets.
Alternatively, you can paste the following search string into the Zendesk search bar (useful if you are using Zendesk Quicktab extension): created>4hours order_by:created_at sort:desc group:none group:"support" -form:billing -form:security
This shows new tickets created in the previous 4 hours - change the range if the incident began earlier than that.
If there is any customer contact regarding an incident regardless of severity, you should create an incident tag in Zendesk as soon as possible. You can check for customer tickets by using the tips above, by scanning the FRT & Free ticket queue and validating the tickets, or by asking the wider Support team. You can create a tag on a ticket directly by finding the tags field and using the format com_incident_####. Replace #### with the production incident number of the issue. Once you've added the tag, submit the ticket with an appropriate Incident First Response macro and the tag will become available to use on other tickets.
Tagging tickets can be done throughout the incident process but the CMOC should check the queue for accurate tagging during the incident resolution stage. The tagging of tickets is useful for gauging support impact, ease of finding related tickets for active incident troubleshooting, and ease of finding related tickets for historical reference.
For details on tagging and tracking incidents, please see Tracking Incidents workflow.
Whether related to an ongoing incident or not, Infrastructure or Security may ask you to reach out to one or more users if they detect unusual usage. Please follow the Sending Notices workflow to action these requests.
As the CMOC you'll guide the incident through the following three stages.
The following sections outline how to perform each of the steps within these stages.
The following steps should be taken immediately after receiving a PagerDuty page for an incident.
We mark the PagerDuty page as acknowledged immediately upon receiving it. Acknowledge the page through the mobile app, web interface or PagerDuty App in the #support_gitlab-com Slack channel.
Before you create an incident in Status.io you should join the incident Zoom call. A link to the call is provided in the incident declaration post by Woodhouse in #incident-management.
Your role while on the call is to follow along while the incident is worked and make updates to Status.io either when asked to or when it's necessary. Oftentimes chatter in this room will be lively, especially in the early stages of an incident while the source of the issue is being discovered. Use your best judgment on when it's appropriate to speak up to avoid vocalizing at inopportune times. You can always ping anyone on the call through Slack if you need to ask a non-urgent question about the situation.
The first thing you should do is to verify that you can be heard by others in the room. To do this, say something like:
"Hi, I'm the CMOC on duty. I intend to send an update, please review this in the Slack thread."
"Hi, I'm the CMOC on duty, how can I help?"
Whatever you choose to say, make sure that you receive a verbal acknowledgement directed at you before you move on to focus on other aspects of the incident.
From time to time, you may be asked to perform some specific tasks in the room. Always verbally acknowledge any such asks by repeating your understanding of the ask back to the requestor. This helps everyone understand that the ask was heard, and also serves to verify that everyone has the same understanding of some action to be taken.
In some cases, the ask may be implicit, rather than explicit. If you're in doubt, always speak up and ask for confirmation. For example:
IM: CMOC is here, we need to roll out a first update.
A good response would be to ask for confirmation that an action was requested:
CMOC: IM, do you want me to send a first update on status.io?
A better response would be to assume that an action was requested, relay your intended course of action in response, and give the requestor the opportunity to provide input:
CMOC: IM, acknowledged, I will draft an update for status.io and ping you in Slack for input.
You can create an incident on Status.io with minimal effort through Slack (provided by Woodhouse) OR manually if need be (e.g., Slack is down or you need to customize the incident more than what the Slack form allows).
You simply need to issue /incident post-statuspage from anywhere on Slack. You will be presented with a pre-filled form that you can update to your liking. Once submitted, the incident will be broadcasted to the following media:
After logging in to Status.io you'll be met with the dashboard that displays various statistics about our current status. Create a new incident by clicking New Incident along the top bar.
Now on the incident creation screen, you'll be asked to fill in the details of the incident, including its severity and what systems are affected. The following is an example of what a new incident would look like if we're experiencing an issue with a delay in job processing on GitLab.com.
Change the following values:
Title - Titles should be brief and concise. The incident title should answer the question: In simple terms, what is the issue?
Current State - In nearly all cases an incident should be created in the Investigating state. If it's been communicated to you that we're aware of what is causing the current incident this could be set to Identified from the beginning.
Details - In keeping with our value of transparency, we should go above and beyond for our audience and give them as much information as possible about the incident on its creation. This field should always include a link to the incident issue from the production issue tracker so that our audience can follow along.
Incident Status - When creating a new incident this will never be Operational. The status of an incident depends entirely on its scope and how much of the platform it's impacting.
Broadcast - Always check each box in this section.
Message Subject - Always leave this at its default value.
Affected Infrastructure - This should almost always be unchecked so that the value of the Incident Status field is only applied to the specific aspects of the platform that are affected by the incident. In the example above we're only experiencing an issue with job processing so only CI/CD is selected.
The CMOC now needs to notify internal stakeholders of the incident using the Incident Notifier Slack workflow. This should be done after the severity of the incident has been confirmed by the Incident Manager.
This workflow, once used, will ask you to fill out a form with details of the incident and will then post those details to #community-relations and #customer-success. This serves to notify those teams of the incident. A copy will also be sent in a direct message to you, should you be asked to post it anywhere else. To engage the workflow:
Click the lightning bolt in the message composition box within #support_gitlab-com and select Incident Notifier.
SubmitIt is important that we are able to differentiate incidents which included outbound status page and related notifications from those incidents which were deemed less impactful to our customers. This can be useful both in filtering for active incidents which include outbound notification as well as for after-incident reporting.
Whenever a GitLab service incident includes the use of a status page incident this should be identified on the GitLab Incident Issue. See this, and other uses of this scoped label in the Incident Management section of the handbook.
~Incident-Comms::Status-Page scoped label to the incident issue.We mark the PagerDuty page as resolved once everything in Stage 1 has been completed. Resolve the page through the mobile app, web interface or PagerDuty App in the #support_gitlab-com Slack channel.
To publicly communicate attention and progress incidents should be updated according to the frequency of incident updates table unless you communicate otherwise.
To update an active incident, click the incidents icon from the dashboard.
Then click on the edit button next to the incident.
Change the following values:
Current State - Change this depending on the current state of the incident and whether or not we've identified the cause (Identified) or implemented a fix (Monitoring).Details - Be as descriptive as possible about the update and include a link to the production issue.Broadcast - Check all boxes.Current Status - If the incident has improved or worsened update this value. If neither, leave it as it was from when the incident was created.Set Status Level - Uncheck this and keep only the affected component selected unless the incident has increased in scope and now affects other components of our infrastructure. IMPORTANT These must be checked individually as in the screenshot below.A ready to be published update should look similar to the following.
Make sure to verify the update length before publishing it. If it exceeds 280 characters, the update won't be published on twitter with no failure notification from status.io.
If the incident title needs to be changed or additional affected infrastructure needs to be added, this is done separately. Instead of clicking the update incident button, click on the incident title to view the details page. Then click the edit pencil icon next to the incident name or affected infrastructure section to change them. It is very similar to adding Post-Mortem.
After publishing the update, visit the live status page to verify that the update went through and looks clear.
When it comes time to close an incident out as resolved, the following flow is usually used.
As noted in the specific sections below, some of these steps are situational and may not be used for every incident.
Once the component affected by the incident has returned to operating normally we will often switch the incident over to a monitoring period to ensure that the problem does not recur. The monitoring period typically lasts for 30 minutes by default, but it can vary and a different amount of time may be requested by the Incident Manager. It may also be requested that the monitoring period be skipped entirely.
If a monitoring period will be used simply edit the incident, and configure the update similar to the following.
Take special note of the changes made to the following fields at this stage.
Current State - Change to Monitoring.Details - Along with any information specific to the incident be sure to mention that all systems have returned to normal operation, that we're monitoring in order to ensure the issue doesn't recur, and provide an estimate for how long we'll be monitoring before we resolve the incident. For example:
While all systems are online and fully operational, out of an abundance of caution we'll leave affected components marked as degraded as we monitor. If there are no recurrences in the next 30 minutes, we'll resolve this incident and mark all components as fully operational.
Incident Status - At this point, the affected component should be back to normal operation. However, to be clear that we're still in the incident management process we will not flip this back to Operational until we leave the monitoring state.Once we're confident that systems have returned to normal operation, the Incident Manager has given the all-clear, and we've completed a monitoring period (if we chose to) of the incident we should mark it as resolved.
Once these conditions are met, make an update to the incident and change the following fields.
Current State - Change to Resolved.Details - State that the issue has been resolved and that systems have returned to operating normally. Be sure to also include a link to incident issue even if you've already done so in previous updates so that any users who missed them know where to go for more info.Incident Status - Change to Operational. IMPORTANT: Make sure the "Apply status level to all affected infrastructure" box is checked.Before resolving the incident your draft should look similar to the following:
A review will be conducted by production engineering for every incident that matches a certain criteria. Status.io allows us to add a link to a post-mortem after an incident has been resolved which will then be viewable on our status page for that specific incident.
Do the following to add a post-mortem to a resolved incident:
From the dashboard click the Incidents button.
Scroll down and click on the title of the incident.
Click Add Post-Mortem and supply the link to the issue being used for the incident review, this is usually the same issue that was opened for the incident.
Infrastructure will at times plan scheduled maintenance events for GitLab.com, some of which will directly impact users. New maintenance events are announced as issues created in the gl-infra/production issue tracker using the external_communication.md issue template accompanied by the Scheduled Maintenance label.
In the event that a maintenance will affect users, infrastructure can request that the maintenance be visible on our status page, and if required, that the CMOC actively provide status updates during the maintenance window. In these cases infrastructure will apply the CMOC Required label to the issue, causing a notification to be sent to the #support_gitlab-com channel that mentions the on-call CMOC. Once this notification is received the CMOC uses the details within the issue to create the maintenance in Status.io.
To create a new maintenance event, click New Maintenance from the Status.io dashboard.
The contents of the maintenance should be filled out according to the details provided in the maintenance issue. Once complete, it might look something like the following.
In case you are required to reschedule a maintenance window, Go to status.io > Maintenances tab
Select the maintenance you need to reschedule.
Update the new schedule time by hitting on the Reschedule Maintenance button Make sure you have the correct timezone details when updating Then hit save.
To send an update about a maintenance event, such as a reminder, go to the Maintenances tab in Status.io and select the one that needs an update. On the maintenance's information page, make note of whether automatic email reminders are set to go out. If yes, make sure not to send email broadcasts for your update in order to avoid sending duplicate reminders to subscribers. Once ready to update, select the Post Update Without Starting button.
Enter the update details provided by the Infrastructure team and have them confirm the appropriate broadcast channels before proceeding to send the update. If "Send Reminders" was enabled in the maintenance information page, be sure not to check "Notify email subscribers" in the broadcast settings.
Once the GitLab Status Twitter account has posted about the maintenance schedule, send a link of the tweet to the #social_media_action channel to let the social team know that you'd like amplification on our GitLab brand twitter account. This should only be used once during a selected scheduled maintenance timeline, preferably mid-week prior to the scheduled maintenance.
It's necessary to inform the ingress CMOC of any relevant activity that ocurred during your shift or if there are incidents that are still ongoing. To perform a handover create an issue in the CMOC Handover issue tracker using the Handover template. Do this even if nothing happened during your shift, signaling that everything is fine is also useful information. It's critical to remember that since we work out in the open by default, the CMOC Handover issue tracker is open to the public. A handover issue should be made confidential if it must contain any sensitive information.
If handover occurs during an active incident where the quick summary you'd provide in the issue is insufficient to properly prepare the ingress CMOC of the situation, you are encouraged to start a Zoom call in #support_gitlab-com and invite the ingress CMOC to it so that they can be caught up synchronously. You can use the following slash command to expedite setting the meeting up.
/zoom meeting CMOC Handover Briefing
The CMOC Shadow Schedule can to be used by anyone who wishes to shadow the CMOC to learn before officially acting as CMOC. A soon-to-be-CMOC can adjust the schedule to match their working hours by clicking Edit this schedule > Add Another Layer; add your username and the days/hours that you wish to shadow.
A "training activity" for CMOCs is an activity under which CMOCs are exposed to items in the workflow with the expressed purpose of maintaining or increasing performance against CMOC performance indicators.
Some example training activities are: