The GitLab Dedicated team's mission is to create a fully managed, single-tenant GitLab environment, served through a GitLab Dedicated platform. It is developed to remove any manual interactions with customer tenant installations, and to ensure that the customer tenants are fully focused on unlocking the power of The One DevOps Platform.
The GitLab Dedicated group is a customer facing team, with team members focused on a high level of infrastructure automation, and enabling customer interactions with the GitLab Dedicated platform.
Team mission is to:
Team performance indicators are not fully defined. We are going to consider a Provisioning SLO to start with, possibly followed by DORA 4 metrics.
Engineering team members of GitLab Dedicated are publicly referenced with specialty for Environment Automation, Switchboard or US Public Sector Services team, based on their primary task.
The following people are members of the Dedicated:Environment Automation Team:
The following people are members of the Dedicated:US Public Sector Services Team:
| Person | Role |
|---|---|
| Stephen Dumesnil | Manager, Infrastructure |
| Joseph Burnitz | Senior Site Reliability Engineer |
| John Edge | Senior Site Reliability Engineer |
The following people are members of the Dedicated:Switchboard Team:
| Person | Role |
|---|---|
| Amy Shiel | Engineering Manager, Dedicated:Switchboard |
| Andrey Ruzmanov | Senior Backend Engineer |
| Sean Arnold | Senior Backend Engineer, Environment Automation |
To engage with the GitLab Dedicated teams:
@mention anyoneCustomers require the ability to customize the configuration of their Dedicated instance before they are able to use it in a production setting. These customizations involve configuration changes to functionality already supported in Dedicated including infra-level settings like IP Allowlists and AWS PrivateLink configuration as well as GitLab application settings that cannot currently be self-served through the admin UI like SAML configuration changes.
To request functionality that is not currently supported within Dedicated, customers must open a feature request using the feature request issue template. To request functionality for the broader GitLab Application, customers can use the feature proposal template.
While in the long term, customer admins will be able to self-serve configuration changes via the Switchboard customer portal, in the short term during the Limited Availability period, SREs will need to make the change and deploy it to the customer's environment. This process is documented below.
start date specified in the customer contract) and make any needed configuration changes to the environment.@ mention the SRE in question. The SRE will assign the issue to themselves and perform the change.start date will be batched and handled in the next available weekly maintenance window. The cut off to get changes in is 2 business days before the maintenance window begins.When it comes to escalating customer support issues, we follow the same definitions of severity as provided by support since Dedicated customers receive priority support. Only in cases where there is an availability or security sev-1 event it can be escalated to Sev-1 at the Support level. 'Business as usual' configuration changes cannot be escalated to sev-1. In sev-1 cases we will involve our on-call, as these incidents may affect our Availability SLA commitments to the customer. Sev-2 and below will be handled by the team during normal business hours. Any fixes identified as part of a support ticket that must go out immediately will be considered "emergency maintenance" and can be done outside the normal maintenance window. All other fixes will be done during the next available maintenance window.
GitLab Dedicated is highly visible within GitLab and the broader market. It is also complex service offering. Consequently, the Dedicated team needs to manage its own work and stakeholder expectations - both external and internal. The following sections describe processes and tools that the team has adopted to work efficiently and effectively.
These processes matter because they provide a clear way to understand the state of the product and ongoing work as well as enable team members to fulfill their role.
A main goal in designing these processes is to make it possible for every team member working on any part of GitLab Dedicated to understand why their work matters and how it contributes to customer results.
It is critical that the following processes are understood by all team members and that we hold ourselves accountable.
Our preference is to work asynchronously, within our project issue tracker as described in the project management section.
The team does have a set of regular synchronous calls:
Demo call - This call is on the agenda once per week. During this call, team members show off their progress, and engage with other team members on topics related to GitLab Dedicated platform. Demo calls are supposed to be rough around the edges and unpolished. In fact, if the demo looks polished, we will discuss whether we are being ambitious enough with our goalsTeam call - During this call, we are sharing important information for team-members day-to-day, as well as project items requiring a sync discussionImpromptu Zoom meetings for discussing GitLab Dedicated work between individuals are created as needed. It is expected that these meetings are private streamed, or recorded(1*), and then uploaded to GitLab Unfiltered playlist. The outcome of the call is shared in a persistent location (Slack is not persistent). This is especially important as the team grows, because any decisions that are made in the early stage have will be questioned in the later stages when the team is larger.
1* Exceptions to the recording rule are: 1-1 calls, discussions around non-project work, and in cases where parties do not feel comfortable with recording. However, even with the exceptions, outcome of project related discussions need to be logged in a persistent location, such as the main issue tracker.
We use GitLab Groups to logically organize team-members working on GitLab Dedicated projects. The groups cover the following use-cases:
@gitlab-dedicated
@gitlab-dedicated/environment-automation, @gitlab-dedicated/switchboard, @gitlab-dedicated/uspubsec, etc.
maintainers and reviewers, e.g.: @gitlab-dedicated/switchboard/maintainers
reviewers GitLab group access is granted to permanent team-members, external contractors, team-members on borrows and similar. This GitLab group type is used to distinguish users without merge rights. Initial reviewes should be requested from this group, using the quick action, e.g. /assign_reviewer @gitlab-dedicated/switchboard/reviewersmaintainers GitLab group is granted to permantent team-members only. This group has merge rights, and the group is granted access through CODEOWNERS approval rulesDo note that the transition between reviewers and maintainers groups is still not fully defined, and this will be deifined as the team continues to grow. Currently, the founding team-members with a significant early contribution to the specific codebase and specific topic knowledge have been granted maintainers rights.
We use epics, issues, and issue/epic boards to organize our work, as they complement each other.
The single source of truth for all GitLab Dedicated work across different functions is the top-level GitLab Dedicated epic.
The GitLab Dedicated epic contains a section that tracks the status all ongoing work. The tracker also references cross-functional initiatives that happen outside of R&D.
The GitLab Dedicated Limited Availability sub-epic contains all of the work necessary to meet requirements to exit Limited Availability.
We use sub-epics to break larger epics into smaller portions. These sub-epics are also mentioned in the Limited Availability Roadmap (i.e. Geo Phase 2 epic).
The diagram below shows an example of traversing the complete hierachy:
Note If you are not seeing the diagram, make sure that you have accepted all cookies.
Each epic has a single DRI who is responsible for delivering the project. DRIs for each epic are listed at the top of the description of each epic per Epic Structure. Epic DRI responsibilities are in https://about.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#epic-owner-responsibilities
The DRI needs to:
Each epic and child sub-epics must include the following:
Description (TBD make epic template)
Epic meta data
Labels are described in the epic label section.
Epic boards are used to track the overall status of epics. We use the following epic boards:
team::GitLab Dedicated with lanes set to scoped worfklow-infra labels
team::GitLab Dedicated with lanes set to scoped Dedicated LA::phase* labels
team::GitLab Dedicated and workflow-infra::in progress with lanes set to scoped health labels
All epics and sub-epics are set with due dates according to the the Roadmap to exit Limited Availability.
Limited Availability phases end and are closed on the 22nd of each phase's corresponding month.
Process to close phases:
Issue boards, such as Beta Development board track the progress of all ongoing work.
On this single board, the goal is to get issues from workflow-infra::Triage into the workflow-infra::Done state, within a product launch milestone. Each of the workflow labels have a special meaning described in the workflow labels section.
The status for all work relating to GitLab Dedicated is maintained in the description of the top-level GitLab Dedicated epic so that it is visible at a glance.
Both Engineering Cross-Functional DRIs should provide weekly updates for the DRI's epics according to following process:
Top-Level Epic Status Update automation synthesizes updates from status section from description of active epics to provide initiative status in the status section in the description of the top-level initiative Epic.
Status updates will be incorporated into initiative status updates and any initiative reporting in the following week.
Status updates are auto-generated and added to description of GitLab Dedicated top-level epic using a bot running the epic issues summary project.
If no update has been provided in an epic or issue for over a week, the issue will automatically receive workflow-infra::stalled label. Engineering managers are responsible for reviewing the status of the issue and helping it move along.
We provide reports on status of GitLab Dedicated to meet Top Cross-Functional Initiative requirements.
GitLab Dedicated team respects the Company principle of everything starting with a merge request.
**This MR should be approved by all approvers, last approver should merge.** as the first line in MR description to state the intention clearly.The MR approval rule settings for all projects should be:
Prevent approval by author On ✔️Prevent approvals by users who add commits On ✔️Prevent editing approval rules in merge requests Off ❌ for emergencies allowing us to act in good faith when absolutely necessaryRemove all approvals when commits are added to the source branch Off ❌We are prioritising reviewing documentation merge requests above all other ones, until further notice. Every merge request documenting acquired knowledge (or concluded discussion) has an impact beyond only the people working on the project directly. In the early stages of building the product, many stakeholders have a need to find actual information quickly, and that means that every line documented increases efficiency of people working on the project as well as people contributing to the project indirectly as we enable more direct self-service.
Having a passing pipeline (green build) on the default branch is very important. Failed pipeline (red build) causes delays in all MR's in progress targeting the default branch, and more importantly, it can cause significant regressions if new MR's are merged into it.
When a red build in the default branch is detected, the first course of action is to revert the MR that introduced failures. Reverting should be done even if it is more work. A couple of answers to the question "why?":
Assign all developers of the relvant Dedicated team to all of the MRs that are due for review. Typcially it is not necessary to assign the whole @gitlab-dedicated group, so choose the approate group such as @gitlab-dedicated/environment-automation or @gitlab-dedicated/switchboard depending on the project. As the team grows this helps with determining signal vs noise. While it is OK for only two people to get the MR to the merged state (author + 1 reviewer), assigning everybody gives more exposure to the work in progress and gives a chance for more parties to provide feedback, leading to better quality overall. If the MR sits more than 1 day without receiving meaningful comments for review, MR author is encouraged to assign a particular reviewer to the MR with the aim to speed up the review process.
As the merge request author, please don’t mark discussions resolved until the reviewer has had a chance to respond. In general, if the reviewer has not yet approved the MR, and the thread is non-trivial, don’t mark their comments as resolved, let the reviewer review your response and resolve accordingly during the next round of view. If they have approved the MR, but comments remain unresolved, it's generally fine to resolve comments before merging.
GitLab Dedicated follows the same pattern for author/reviewer assignment as the standard GitLab practice, documented in the Code Review Guidelines documentation.
This can be summarized as follows:
- Merge request authors and DRIs stay as Assignees
- Authors request a review from Reviewers when they are expected to review
- Reviewers remove themselves after they’re done reviewing/approving
- The last approver stays as Reviewer upon merging
Commonly used labels are:
team::GitLab Dedicated.workflow-infra labels.component labels.cloud-provider labels.Launch labels.The team::GitLab Dedicated label is used in order to allow for easier filtering of issues applicable to the team that have group level labels applied.
Epics and child epics should contain the following labels:
Dedicated LA::Phase2workflow-infra labelworkflow-infra::in progress, then a health status label should be applied. (health::on track, health::needs attention, health:at risk by the epic DRI. This label is regularly updated as part of status updates.)We leverage scoped workflow labels to track different stages of work.
In general, we want to see issues move from workflow-infra::Triage to workflow-infra::Ready stage to indicate that the submitted issue will go through for implementation. Once the issue is marked with workflow-infra::Ready, we are ready to work on the issue until we get the issue marked with workflow-infra::Done.
The standard progression of workflow is from top to bottom in the table below:
| State Label | Description |
|---|---|
 |
Default label added to issues created. Issues with this label need to be confirmed as work we would consider. If we don't want to consider the issue further, we mark it with workflow-infra::Cancelled and close it. If this issue does not need Product validation, and we are ready for implementation, issue is moved to workflow-infra::Ready. Otherwise, we move it to the next stage workflow-infra::Proposal. |
 |
In this stage, proposal is being created and put forward for review with the rest of the team. Issues in this stage are also a part of Product validation workflow. If there are no further questions or blockers, the issue can be moved into "workflow-infra::Ready". |
 |
The issue is waiting to be picked up for work. |
 |
Issue is assigned to a DRI and work has started. |
 |
Issue is updated with the outcome of the work that was done, and this label is applied and issue closed. |
There are three other workflow labels of importance:
| State Label | Description |
|---|---|
 |
Work in the issue is being abandoned due to external factors or decision to not resolve the issue. After applying this label, issue will be closed. |
 |
If no update has been provided in an issue for over a week, the issue will get this label. The team Engineering Manager is responsible for reviewing the status of the issue and helping it move along. |
 |
Work is blocked due external dependencies or other external factors. Where possible, a blocking issue should also be set. After applying this label, issue will be regularly triaged by the team until the label can be removed. |
To denote different types of components and services we are working with, we leverage component:: scoped labels. By using these labels, we are able to track the distribution of work across different components which also allows us to change focus where needed. These labels are created on the GitLab Dedicated group level, since they are focused on work required specifically for GitLab Dedicated.
NOTE We do not use Service:: labels given that service labels are used by GitLab SaaS related projects.
Component labels with their description can be found by searching prioritized labels.
In order to make GitLab Issue board easier to use, we also apply Launch:: scoped labels, denoting the Product launch milestones defined in the project readme. Launch labels indicate the intended milestone during which work can be executed (intention, not guarantee).
Valid options are one of:
Launch::Beta - BetaLaunch::LA - Limited AvailabilityLaunch::GA - General AvailabilityThese labels exist only because filtering by epic in GitLab issue boards does not include issues from sub-epics. We have an option of creating an epic board (listing all epics), or issue board per defined epic, but there is currently no way to centralise all issues within a parent epic's children epics.
These scoped labels are intended to distinguish generic work to everything made for a specific cloud provider.
| Cloud Provider Label | Description |
|---|---|
 |
Amazon Cloud specific implementation |
 |
Google Cloud specific implementation |
Resources used by the team to conduct work are described on the Development Resources Page.
Switchboard customer portal was suggested by @marin after he spent a day trying to figure out which mixing console (Also known as a switchboard/soundboard) to get for amateur music making. He didn't buy anything, but the suggestion was accepted.Amp management cluster was suggested by @ccasella, as it is the instance that is "powering" supply of other instances.