GitLab's internal Red Team conducts security exercises that emulate real-world threats. We do this to help assess and improve the effectiveness of the people, processes, and technologies used to keep our organization secure.
The Red Team does not perform penetration tests, and the work we do is not focused on delivering a list of vulnerabilities in a specific application or service.
Malicious actors are not constrained by the narrow focus of traditional security testing. We must take on this adversarial mindset in order to challenge our own assumptions and identify areas for improvement across our entire organization. We do this by emulating the real-world tactics, techniques, and procedures (TTPs) of threats that are most relevant to our environment.
This approach allows groups across GitLab to practice detecting and responding to threats in a controlled manner. We can then better understand our current defensive capabilities and work to improve them before we are faced with the real thing.
The Red Team operates under a pre-defined set of rules of engagement. The rules of engagement exist to inform GitLab's team members on how the team operates during engagements. It provides guidelines for determining scope, the ethics we employ during our engagements, how we collaborate as a security team, and how we escalate vulnerabilities and exploits we discover during those engagements.
Further details can be found in the job family description.
We use the term "Red Team Operation" to describe a planned and approved exercise where we emulate real-world attacks against GitLab targets. Some of these operations are done completely in the open, where all planning and execution is visible to all GitLab team members. This is explained in detail in Purple Teaming at GitLab.
Other operations may be less visible, with only certain team members aware of the details. These operations provide the most realistic opportunity for GitLab to practice responding to real world attacks. When performing one of these operations, we will follow the same "Purple Teaming" procedures with the following modifications:
Building efficient and understandable processes for our operations that work for us and the teams we collaborate with is important. However, given the need, we will prioritize collaboration with our peers over the processes and tools we have in place. We feel this allows us to best suit the needs of the teams we work with and the organization as a whole.
The Red Team will develop new adversary emulation techniques on a regular basis, both during official operations as well as informal open-scope activities.
When a technique has been proven effective, the Red Team will configure any existing automation around this technique to publish messages using Google Cloud Pub/Sub. These messages can then be ingested by the SIEM to generate alerts and integrate into the standard process of responding to known risks.
For example, the Red Team may create a bot that logs into development instances and attempts to exploit a specific configuration. Once the risk has been proven and existing detection/response capabilities have been tested, it is time for the technique to be fully disclosed internally.
While this may result in product fixes or infrastructure changes, it is possible that vulnerable configurations may reappear in the environment. The bot can continue to run at scheduled intervals, but will be enhanced to publish a message to Google Cloud that will have a corresponding SIEM alert. At this point, SIRT will respond to new occurrences and the Red Team will no longer attempt exploitation.
Some activities are considered open-scope, meaning that they can be conducted at any time, from any source IP address, and against any GitLab-managed asset without prior approval or notification. The output may or may not be included in the reporting for planned operations, depending on the results and whether or not it is helpful to the Blue Team.
This includes:
If these activities are detected by SecOps, they should be treated as potentially malicious and acted upon appropriately. Unless part of a planned operation, there should never be an assumption that suspicious behaviour is a Red Team activity.
Conducting open-source intelligence (OSINT) gathering against non-GitLab managed assets, such as social media sites, is also considered open-scope and may be conducted outside of planned operations.
If an open-scope activity uncovers a vulnerability that puts GitLab at immediate risk of compromise, SecOps will be notified via the official paging procedures.
The goal of a Red Team operation is often to test our policies and procedures when reacting to an actual threat. This includes identifying suspicious activity and following the appropriate runbook to investigate and respond to that threat.
If any team member, at any time, could simply ask "Hey, this looks suspicious. Is this our Red Team?" then this opportunity would be lost. Instead, all suspicious activity should be treated as potentially malicious and acted upon accordingly.
Any unannounced Red Team operation will include team members across the organization who are fully informed of the operation's activities. These team members can help ensure the operation provides value by allowing incident response to continue without going too far. For example, we would not want an emulated attack to affect production operations or escalate to third parties.
If suspicious activity is detected and the matter is escalated to one of these team members, they may know right away whether or not the activity is related to a Red Team operation. If they are unsure, they can contact the Red Team directly via Slack. At this point, the Red Team will cease all activity until they can answer definitively whether or not they were the source of activity.
If the activity was indeed the Red Team, they will provide proof and the operation will generally continue. Specific rules for if/when an operation is revealed to all involved will be documented in the original project proposal. This may include provisions for stopping incident response but continuing the Red Team work to further test technical controls.
If the Red Team is ever asked "Is this you?" by someone who is not fully informed on an active operation, they will respond with the following text:
Thanks for your vigilance! Any suspicious activity should be treated as potentially malicious. If you'd like to contact security, you can follow the process here: https://about.gitlab.com/handbook/security/#contact-gitlab-security.
Red team operations provide an opportunity to practice these processes, and revealing an operation early might mean we miss out on that opportunity. Because of this, we have a policy to neither confirm nor deny whether an activity belongs to us. You can read more about this policy here: https://about.gitlab.com/handbook/engineering/security/threat-management/red-team/#is-this-the-red-team.
Every two weeks the Red Team will host Red Team Office Hours. This meeting will be open to the entire company and will alternate between EMEA and APAC friendly times. For the most part these will be an open discussion with members of the Red Team but we will also use this time to perform "read outs" of recently completed Red Team Operations. Note that in some cases, depending on the content these will not be recorded or made public.
