GitLab Security Operations leverages automation as a guiding principle to ensure our security engineers have the time to focus on what matters, not manual mundane tasks that can be standardized and automated. We applied this principle to securing the GitLab.com SaaS platform, which generates terabytes of log data daily and requires the GitLab Security team to standardize, automate, and scale security workflows for enhanced protection and efficiency. The result: a new framework we call GitLab Universal Automated Detection and Response, or GUARD – a collaboration between the GitLab Security Incident Response Team (SIRT) and the Signals Engineering Team.
GUARD covers all aspects of security detection, including:
- creation
- maintenance
- alert routing and handling
- rich metrics collection
- alert closure or incident escalation workflow
The goals of GUARD
GUARD was created and designed with a set of key goals:
- Standardization of SIRT’s detection and alerting pipeline to produce high-quality detections using a peer reviewed and automation-first focus
- Reduction of alert fatigue through alert consolidation, deduplication, risk scoring, and automated feedback
- Metrics to measure response efficiency and identify problems early
- GitLab at the core by leveraging GitLab as a single source of truth for detection definitions
GUARD's design principles
GUARD was created out of necessity, with a clear vision of the intended state. Before GUARD, detections did not follow a standard format, alerting metrics were not available, and detection creation and maintenance were ad-hoc. Building a framework that was scalable, GitLab-centric, and able to automate manual tasks was core to the success of GUARD. Due to time efficiencies realized by GUARD, SecOps engineers have more time to solve difficult problems and handle complex incidents.
GUARD components
The GUARD framework consists of multiple modules. At the center of GUARD is the GitLab platform itself, acting as a single source of truth for detection rules and providing SIRT the ability to automatically deploy detections as code using GitLab CI/CD.
GUARD includes the following components:
- Detection as Code (DaC) - Deploys detections through the GitLab CI/CD pipeline.
- User Attestation Module - Allows GitLab team members to attest to activities flagged as potentially malicious.
- Enrichments - Polling historical and contextual information to enrich alerts to make alert triage easier.
- Alert Triage and Response - Providing a standard alert triage format and templated escalation actions.
- Metrics Generation - Gathering insights on alert handling.
Each GUARD module works together to standardize, automate, and iteratively improve GitLab’s security detections and alerting pipeline.
GitLab at the core
GitLab is core to critical components of GUARD, acting as a single source for threat detections, automating GUARD’s DaC pipeline through GitLab CI/CD, and acting as a “front end” for GUARD, through which security engineers can add, edit and delete threat detections.
How GitLab features use GUARD:
- GitLab projects: GUARD utilizes a GitLab project repository as the single source for GUARD threat detections, stored in JSON format.
- GitLab MRs: Any changes to GUARD detections, including new detections utilize GitLab MRs against the main GUARD project. A detailed MR template is utilized in which we validate and record details about the detection being added, edited, or deleted. MR approval rules, including the use of CodeOwners and protected branches, are used to ensure proper detection reviews are completed before merging.
- GitLab issues: Bug submissions or other engineering efforts related to GUARD are recorded in GitLab issues.
- GitLab labels: A set of standardized labels ensure security engineers document GUARD changes in a way that is easy to track.
- GitLab CI/CD: GUARD uses a GitLab CI pipeline to automate the deployment of new/changed/deleted detections to GitLab’s security incident and event management (SIEM). GUARD’s CI pipeline performs a number of validation, testing, and quality checks before successfully passing the pipeline and committing the changes to GitLab’s SIEM platform.
Metrics generation
Interactions with the alert handling UI are recorded to generate key performance metrics, such as Time to Respond, Time to Resolve, and insights into alerts like true/false positive rates. Additional metadata collected includes an emoji-based sentiment analysis. Engineers handling alerts provide ‘feedback’ about the alerts handled in the form of emojis, so we can take that feedback into account upon iterating on detection rules.
Alert handling metrics are stored in a separate database to create visualizations consulted by engineers and management. These are key to understanding team performance in alert resolution and alert fidelity so that we can always improve.
Iterate with us
Using GitLab as a single source of truth for threat detection code allowed GUARD to extract processes from a specific SIEM technology, supporting greater flexibility, ease of use, modularization, and auditability.
Iteration is a core GitLab value – we start with the smallest valuable thing to get fast feedback and efficiently reach a desired end goal. GUARD is no different, and we hope sharing GUARD will help readers iterate towards their own automation improvements.
This article is the first in a series on GitLab GUARD. Next, we will share details about various aspects of our iterative journey to implement GUARD at GitLab.
