Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Category Direction - Pipeline Abuse Prevention

Pipeline Abuse Prevention

Continuous Integration (CI) is the foundation of the Verify stage within the Ops Section direction, are recognizing the top risks to our business ofSecurity and SaaS Reliability first. In early 2021, we witnessed the cryptomining CI co-evolution, where free SaaS continuous integration platforms are being seriously compromised by the cryptocurrency mining attacks. GitLab was no exeception to this Industry-wide experience and we instrumented a few practices to mitigate abuse for on GitLab.com, which definitely impacts the experience of free and trial users.

Going forward, we needed a more proactive approach for monitoring, detecting, evaluating, preventing, and reacting to pipeline abuse. Traditionally, product categories are single product group areas an with one engineering team. The Pipeline Abuse Prevention category rather, is more of a cross-cutting program involving several teams, product groups, and functions. We are considering funding a cross-cutting Abuse group, as outlined by our Insider Threat Category as part of our Applied ML investments. If we do fund that group we will likely fold this category into it.

Mission

Pipeline Abuse Prevention is focused on proactive mitigation of CI abuse to ensure acceptable tolerances of business impact and human cost are not exceeded.

Confidential issues

A number of issues are intentionally confidential despite our value of transparency. This is because we don't want to make it obvious to abusers the exact details of our controls. We aren't relying on "security by obscurity"; however, we also don't want to make it easier for the abusers.

Additional Resources

For specific information related to spam and abuse reduction intiatives, check out Trust and Safety. You may also be looking for one of the following related product direction pages: GitLab Runner, or take a look at the Verify stage .

Stable counterparts in Pipeline Abuse Prevention

We rely on several teams to make this program successful:

DRI EM Trust & Safety AppSec Fulfillment PM Engineering
Jackie Porter Darby Frey Charl de Wit Dominic Couture Justin Farris Stan Hu

Other Product Groups may be brought in depending on scope classification.

Product Scope DRIs

Fulfillment - Anything related to collection and validation of credit cards/debit cards Verify - Anything related to triggering of credit card/debit card validation

Program Pillars

There are four areas of focus for Pipeline Abuse Prevention:

  1. Credit/Debit Card Validation for Free and Trial Users to block bad actors Kibana Dashboard Dashboard
  2. Pipeline Validation Service which has rules that catch certain coding behaviors to stop bad actors before pipelines are run Dashboard
  3. CI Minute Quota enforcements and limits across various levels of GitLab.com Dashboard
  4. Cost controls two dimensions: human cost and Infrastructure cost CI Runner Costs Blocking Dashboard

What's Next & Why

Credit/Debit Card Validation Workstream in 14.0

We have a few items planned for follow-up enhancements to the rapid action efforts and credit card validation work via this confidential issue. We are exploring usability of the credit card validation experience for legitimate users via this confidential epic.

We also are thinking about ways to make the validation more inclusive for legitimate users who don't have access to or don't want to provide a credit/debit card in this confidential issue.

PVS Workstream in 14.0

Currently, the team is in open dialogue on ownership of PVS.

We are also looking at instrumenting methods of abuse control via this 1. Abuse tracking controls including confidential issue

CI Minute Quota enforcements and limits across various levels of GitLab.com

As of 13.12, we have instrumented enforcement of limits in private projects where now pipelines fail when CI minute quotas are exceeded.

Up next, we are iterating toward enforcement across public project by introducing limits to new public projects. While also taking into account how this impacts our Open Source projects in gitlab#330888.

This effort will be then be expanded to all free, public users via gitlab#254231, where we hope to instrument counting of CI minutes as well via gitlab#254231.

Cost Controls

We have two issues to establish costs control mechanisms:

  1. Implement thresholds for allotted human cost of pipeline abuse
  2. Implement thresholds for allotted infrastructure costs

Competitive Landscape

Cryptomining is impacting free CI providers industry wide. GitHub has added several features to help combat bad actors in the wake of this shake up including:

  1. Changes to Approving workflow runs from public forks
  2. Protecting maintainers with manual approvals

Looking Forward

We would like to implement more methods for abuse control like those discussed in this confidential issue. The Applied ML Group is planning an MVC to apply models to detect CI abuse.

Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license