Multi-Large Working Group

1. You are here:
2. Meet the GitLab team
3. Organizational Structure
4. Working Groups
5. Multi-Large Working Group

On this page

• Attributes
  • Charter
  • Scope and Definitions
    • Sequence Order Of Deliverables
      • Deprecate NFS
      • Helm Charts
      • Operator
      • Stateful Nodes
      • Day 2 Automation
    • Use Case: Day 2 Operations
  • Horizontal Scaling Across Multiple Instances
• Roles and Responsibilities
• Related Links

Attributes

Charter

The charter of this working group is to improve the operational efficiency of running multiple, large, independent GitLab sites without needing linear growth on staffing.

Scope and Definitions

In this context,

• GitLab sites means GitLab instances under a GitLab domain managed by GitLab (i.e, not operated on behalf of customers)
  • Automation developed should be applicable and usable to self-managed installations in the future
  • The scope excludes customer instances to focus on the problem of running multiple large sites fully under our control, where we are the ultimate DRIs for what policies and solutions we adopt as a company to apply to those instances.
• Multiple means up to 10 primary instances
• large means instances that start at the 1K users Reference Architecture and can scale beyond the 50K users Reference Architecture to a million active users
• independent means they're fully isolated from each other, they don't have a common virtual operations centre

Examples of these types of instances would include GitLab.eu, GitLab.cn. The working group may possible consider GitLab Federal, although this has to be an explicit decision, as operating federal entails clearances and such.

Sequence Order Of Deliverables

The following list outlines the order of deliverables per Multi-large (FKA Scaling) Working Group Agenda on 2020-07-13. The goal of this list is to define deliverables for the Working Group such that its charter can be finite.

Deprecate NFS

1. Make Web/API work without NFS (issue tracker)
2. Make CI/CD live traces work without NFS (issue tracker)
3. Make Pages work without NFS (Scability: https://gitlab.com/groups/gitlab-org/-/epics/3980)

Helm Charts

1. Move Web/API nodes of .com to Helm charts (no issue tracker)
2. Move Pages nodes of .com to Helm Charts (no issue tracker)

Operator

1. Make Operator for Openshift (no issue tracker)
2. Make Operator work with GKE (might be little work or no work) (no issue tracker)
3. Use Operator for .com (no issue tracker)

Stateful Nodes

1. Not sure when to move stateful nodes (Gitaly/PostgreSQL/etc.)? (no issue tracker)

Day 2 Automation

1. Automate Day 2: scaling/backups/restores/upgrades (no issue tracker)

Use Case: Day 2 Operations

In this use case, we start with the 1K users Reference Architecture (Day 1) and map the necessary "moves" to grow and migrate said instance thorugh the available reference architecture milestones (Day 2: 2k, 3k, 5k, 10k, 25k, 50k) so that this can be managed within GitLab. Therefore, the working group focuses on the step-functions that scale the sites in code, which includes:

• How the site is migrated from one level to another while the site is online, i.e., extracting services into nodes (database, storage, etc) in an efficient way while reducing up front costs
• How a single team manages multiple, large sites
  • Observability
  • Code deployments
  • Back-ups
  • Protecting against DDoS attacks

Horizontal Scaling Across Multiple Instances

For clarification, this working group is focused on horizontal scaling across multiple GitLab sites. Single-site scaling (e.g., GitLab.com) is already covered under the charter of the Scalability Team.

Roles and Responsibilities

Related Links

• Working group kick off meeting
• Defining Day-2 Operations - DZone Agile
• Scalability Team - GitLab