Separate environment for the Security department for:
gitlab-com
and gitlab-org
groupsThis enclave will contain at minimum:
live
environment suitable for processing and storing RED data.integration
environment with scaled down architecture for testing
integrations.All environments will follow compliance and retention requirements.
Issues for breaking down the work will can be created in any appropriate tracker
under security-enclave> and
labelled as ~"security enclave"
for tracking on this issue board.
When discussing the Security Enclave:
integration
and live
to avoid ambiguity with other environments.@gitlabsec.net
in user and group names.project
is unfortunately an overloaded term between GCP and GitLab, so specify as "GCP Project" or "GitLab project" when necessary.Follow best practices to ensure that artifacts deployed to the environment are available in local mirrors or caches. Use available tools to verify the integrity of artifacts deployed to the environment.
Rely on high avaialibility services when necessary or otherwise determined that the risk is acceptable.
GitLab features such as container scanning and private docker registries can be used to shift security left for this environment.
Many of the workloads for the proposed systems can be deployed as kubernetes workloads, with the advantages of shifting our own security left and making use of GitLab features such as container scanning and private registries.
The Security enclave will contain a maintained GitLab instance for the following work:
Container registry and scanning for the kubernetes workloads deployed to the environments.
CI for terraform deployments will be run in live
GitLab, using private runners
within the environment. This means that some maintenance of the live
GitLab
instance and the runners can not be done through CI, but can still be tracked
in terraform and executed manually.
Issue on the tracking for non-public by default issues CI for working with customer data (RED) from GitLab.com Application Resources
An Okta organization under the existing organization will be used as the identity provider and single-sign-on solution for the enclave. The tradeoff of having an external dependency is being made to reduce duplication of IdP efforts.
The additional risk for Okta super admins having impact on the security enclave is compensated for the strong audit capabilities of Okta itself and additional alerting in DELKE on logs that are already being ingested.
Access to GCP resources using IAM using Google Groups for groups specified in the Role Entitlements listed below. This is to maintain independence from other corporate systems
To effectively use Google groups for access control lists:
Group Administrator
in Google Workspace.
While other technologies will be evaluated for maintainablity and cost, preference will be given to GCP native tooling where it exists.
A new top-level domain gitlabsec.net
will be used for resources related to the Security Enclave
to reduce the interaction between systems due to the behavior of integrations
such as email for SaaS application being routed through Google Workspace.
Cloud DNS will be used to manage records for the domain. Similar to the other independent resources, Cloud DNS will not have the same dependencies as other DNS providers used in production for GitLab.com.
An automated process and tooling should be implemented where possible to ensure that records are kept up to date with resources. Where not possible, change control tracking processes should include steps for admins to review for associated resources when modifying or removing resources.
Some shared resources that will need to be maintained for uptime, but not
necessarily have duplicated integration
resources associated with them:
The following workloads will be implemented in both live
and integration
.
Bootstrapping the environment may need to be performed by running various terraform and chef provisioning steps manually until the GitLab instance is sufficiently configured to perform these tasks via CI.
gcp ssh
accessWe will be relying on auto-scaling and multi-zone failover using GCP load balancers for the Kubernetes workloads. We will be relying on CloudSQL for any workload that supports it as a database (GitLab).
The following workloads will have ready instances in a second zone (pods or VMs as necessary):
We will be relying on monitoring and the queuing features of pubsub to reliably ingest data into DELKE.
Other workloads, may be useful but not required during operations at this time, so do not require additional measures other than monitoring and detection to meet HA needs.
We will scheuld CloudSQL backups via terraform, with multi-region storage for backup.
All repositories used for the maintenance of the Security Enclave will push mirrored to GitLab.com or ops.gitlab.net.
security-enclave-terraform
project
This role is defined in addition to job title/function for individuals whose work requires additional access in addition to their job function granted roles. The need for this role should be minimized to maintain least privilege.
enclave-owner-gcp-sg@gitlabsec.net
enclave-reviewers-gcp-sg@gitlabsec.net
live-general
enclave-owner-gcp-sg@gitlabsec.net
enclave-admin-gcp-sg@gitlabsec.net
enclave-developer-gcp-sg@gitlabsec.net
integration-general
: An empty, scaled down environment with no real data for testing
integration between various systems, e.g. Cloud Functions.
enclave-owner-gcp-sg@gitlabsec.net
enclave-admin-gcp-sg@gitlabsec.net
enclave-developer-gcp-sg@gitlabsec.net
live-secauto
enclave-owner-gcp-sg@gitlabsec.net
enclave-secauto-admin-gcp-sg@gitlabsec.net
enclave-admin-gcp-sg@gitlabsec.net
enclave-secauto-developer-gcp-sg@gitlabsec.net
enclave-developer-gcp-sg@gitlabsec.net
integration-secauto
: An empty, scaled down environment with no real data for testing
integration between various systems, e.g. Cloud Functions.
enclave-owner-gcp-sg@gitlabsec.net
enclave-secauto-admin-gcp-sg@gitlabsec.net
enclave-admin-gcp-sg@gitlabsec.net
enclave-secauto-developer-gcp-sg@gitlabsec.net
enclave-developer-gcp-sg@gitlabsec.net
backup
: Project to hold copies of backups
enclave-administrators-gcp-sg@gitlabsec.net
ApplicationSecurity/applications
: A folder to contain projects for deployment and testing application security tools or other research in support of initiatives
and the Secure product team.
Notes:
gitlab-security
The current gitlab-security
project in GCP can continue to be used for POCs,
experimentation
As part of this process, it will be audited for data and access restrictions.
gitlab-security
Applications currently using the gitlab-security
AWS project will be migrated
off to reduce the resource footprint of Security department initiatives.
Access to enclave resources can be requested using the standard Access Request
Process, specifying GCP (Security Enclave)
as the system name.
gcloud
CLI tools installedterraform
ansible
Accessing resources in the gitlabsec.net
GCP organization will require logging
in to an additional Google account. To avoid confusion, consider using a
Firefox container tab for enclave specific work.
Note: Links to the projects themselves are TBD while details of which should be the origin which should be the mirrors:
security-enclave-terraform
project: The top-level terraform configuration for the enclave.security-enclave/terraform-modules/
subgroup: Any terraform modules that
are too big/complex/independent enough can be moved to their own project in
this subgroup.security-enclave/ansible-playbooks/
: Enclave specivic ansible resources.Consider the following when starting new work in the enclave:
If a services is highly isolated, complex, and/or has only dependencies outside the enclave, it may make sense to have its own project. These are only guidelines and should be considered along with other criteria such as maintainability when making a decision.
Request a new project by opening a new issue in the Security Issue Tracker,
tagging the Enclave Administrators, then follow the instructions in the top-level
terraform project's README.md
.