A day in the life of a SRE at GitLab with Laura Montemayor
Hi! I’m Laura Montemayor, a Frontend Engineer in GitLab’s Monitor:Health Group. Monitor is one the DevOps lifecycle stages that comes after you’ve configured your production infrastructure and have deployed your application to it. We work on features that help you monitor metrics and Kubernetes, track errors, manage incidents, and more.
If you’re curious about our team, you can check out our direction page to see what’s on the roadmap.
A few weeks ago, I had the opportunity to shadow our SREs, which was pretty exciting for me, as it is monitoring in action. An SRE is a Site Reliability Engineer, who is responsible for keeping all user-facing services and other GitLab production systems running smoothly.
You can learn all about them in our handbook.
Here are some of my general takeaways from that week:
Less is more when it comes to tooling
I’ve had non-remote friends ask me what communications tools we use at GitLab, and they are sometimes surprised that it’s not a long list. In a way, I had the same expectation for our SREs and their tooling; I had visions of them all buried underneath monitors, with screens showing moving graphs, stats, logs - the works!. But SREs use the same tools everyone at GitLab uses for communication - GitLab issues, Slack and Zoom. The workflow is pretty straightforward; the on-call SRE gets pinged on PagerDuty, which also sends an alert to a Slack channel, and everything is discussed either there or in an issue. True to our transparency value, everyone at GitLab has access to the channels and the issues, and most of these issues are public.
If you are interested in learning more about SREs and the tooling they use (and more!), check out my teammate’s blog post.
MT = Monitoring Tool
Managing the level of noise is a huge challenge
During the time I shadowed, I probably saw an alert every 5 or so minutes. And this was during their so called “quiet time”! At first, it seems overwhelming to be alerted so often, and everything looked serious and actionable to me. Queue latency outside of SLO? Filesystem will be full SOON? I panicked. However, a lot of these alerts are warnings and don’t all turn into incidents; but even if they are warnings, sometimes they merit some investigation and discussion, since almost no alert can be truly ignored.
Examples of alerts that can usually be ignored:
- CPU/Memory usage that just stepped over 90% - This can be ignored (but an eye should be kept on it) because the prediction algorithms for usage often raise an alert prematurely. The usage normally falls to safe levels or won’t cause an issue for a few more SRE shifts.
- Abuse detection - This can give a lot of false positives which are not necessarily an incident, like abuse being detected when a project’s pipeline is used excessively.
Incidents don’t happen as often as I thought
So even though there are a lot of alerts, I was surprised to find the majority of them don’t actually turn into incidents. There was a part of me who wanted to experience an incident, since I thought going into a situation room and resolving it would be really exciting. But there were only two incidents, and one of them was resolved straight away and the other one was handed over for further investigation. None of them were disruptive to customers though, so all in all, I figure GitLab must be pretty stable 💪
Monitoring is hard (no surprise there!)
Experience and domain knowledge certainly help when it comes to sifting through the noise, but it doesn’t make it easy. A lot of alerts happen because they are often based on inflexible Boolean logic or some arbitrary threshold. For example: we got an alert for a spike in the redis-cache latency Apdex. We checked the Grafana dashboard and realized it was a one off spike in latency, so it was fine for the time being. But it was one more thing to keep an eye out for, and the intermittent spiking kept alerting us, which added more cognitive noise. I can’t even begin to imagine the horror of trying to effectively monitor with a bunch of different tools. This is why I love the idea of a single app for all your needs 😉
Communication is key
One thing that helps immensely with monitoring and incident management is communication. It is one of our core values at GitLab after all, seeing as we are an all-remote, async team. If the SRE who is on-call cannot resolve the issue straight away, they can just ping other SREs on Slack and someone will be available to help. It’s the advantage of being an async team - there’s always someone available in almost every single timezone! It also means that even if you’re not on the SRE team and you’re working on something that will start firing off alerts, you should communicate with the SREs to let them know. For example: some changes were made in the Pages API, which were rolled out to 20% of the domains. This caused an increase in alerts, but since it was properly communicated, the SRE on call could be assured that these alerts were fine to ignore. There was no way of turning these alerts off unfortunately, since the rigidity of the tools doesn’t allow it.
Here’s the issue, if you’re curious.
We love issues 🧡
At GitLab, we love writing everything down in issues, which helps a lot with incident management. Every time an SRE changes shifts, they write a handover issue with a summary of what alerts/incidents are ongoing or resolved, etc. Check out this example. Every time an alert actually becomes an incident, the first step is always to resolve it, since it affects our customers. After that, an issue is created to collaboratively do a root cause analysis and hopefully prevent it from happening again. Keeping record of all of the incidents is really helpful for context and resolution of future incidents.
There must be collaboration
Our SREs who are on-call do not work in isolation, and even though they are the first point of contact for alerts, it’s not expected that they solve everything themselves. Sometimes things can’t be solved in the moment, like the example below.
Incident: AlertManager is failing to send notifications
Why weren’t we able to resolve it? We don’t have visibility when Alertmanager doesn’t
work, as we don’t have logs or graphs to for it.
So what now? We first checked to see if there was a similar or a linked issue, and
there was. After that, we made an issue for it, linked the other issue, assigned a
domain expert, and went from there.
Monitoring is a growing field
Given the imperfect nature of monitoring, it’s only natural that the field is constantly evolving and growing. At GitLab, we recently created a Scalability team, which will help curate our monitoring and alerting. They will frequently update the criteria to generate alerts, which will make the whole process more manageable. My team is responsible for building the features to help monitor and manage incidents, and we’re currently working on features which will help with our SREs workflow. If you’re interested, check out our Ops Section Product Vision for more details. You can also check out the reliability guide for a more in-depth and robust look at what our SREs do.
Join us at GitLab! Or consider trying us out for free.