Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Architecture Evolution at GitLab

GitLab Architecture

Software professionals have come up with a lot of great metaphors about what software actually is. Some people believe that software is like a castle that needs constant maintenance, others think about software as a garden that needs care and careful cultivation.

At GitLab we believe that software might be more complex than that, and that software is more similar to life itself.

Because of that it needs to adapt and evolve, usually through iteration.

Evolutionary Architecture

As software products grow, become more complex, and get more widely used, things that were working well on a small scale, might not be sufficient on a much bigger scale.

  1. We allow engineers to lead the evolution of the software.
  2. We have a process that helps us make architectural changes a success.
  3. We have a way to determine what work is important and what is not.
  4. We constantly work on finding a balance between reactive work, preventive work, and innovation.
  5. We know how to leverage knowledge of people that know most about GitLab.
  6. We empower people to work on things that prevent a variety of problems from happening.
  7. We empower people to work on things that are a leverage and make everyone else more productive.
  8. We allow people to experiment and innovate while being efficient and using their time wisely.
  9. We have a process that makes it possible to allocate resources and hold people accountable.
  10. We have a process that ensures we can get things done.

GitLab Architecture Evolution Workflow

TL;DR:

  1. Anyone can open an issue with a change proposal and label it with ~"Architecture decision",
  2. The author collaborates with their manager to understand better the scope of the change,
  3. Based on the scope of the proposal, the author and their manager collaborate together to find an Engineering Leader who will be a decision maker that will either approve or reject the proposal,
  4. The author and their manager find an Architecture Evolution Coach on the team page, someone who works on the level close to the Engineering Leader who is going to be the decision maker,
  5. Optionally the author and their manager collaborate to find a Domain Expert if help from one would be useful,
  6. The author collaborates with the architecture evolution coach and a domain expert to create a blueprint merge request of the proposal,
  7. In the RFC-like style they get more people involved into the discussion to collect more feedback,
  8. The blueprint describes 1-year landscape and 3 iterations in the first 3 milestones,
  9. When specific enough, the blueprint gets merged by an Engineering Leader,
  10. 3 Directly Responsible Individuals (DRIs) get assigned and collaborate together to get the work done,
  11. A Working Group can be formed if needed, but is not required,
  12. DRIs propagate the blueprint across sub-departments and ensure that work gets scheduled,
  13. Teams schedule their work based on their interpretation of 1-year and 3-months landscapes,
  14. DRIs described in the blueprint are the decision makers for the approach taken,
  15. After 3 iterations work can be concluded or expanded for the next 3 iterations.

Finding people that will amplify your influence

influence

The author of a proposal and their Engineering Manager collaborate together to find people that will be amplifiers of their influence:

During the process of working on the proposal, the author will collaborate with an Architecture Evolution Coach and, optionally, a Domain Expert to create a blueprint of the change. The blueprint merge request will be then either approved or rejected by an Engineering Leader.

In order to choose the right people, the author and their manager first need to understand what is the scope of their proposal, what departments and teams will need to help to get the work done and how important it is for the organization.

The first step is to find an Engineering Leader that will be responsible for approving the proposal and merging the blueprint merge request. The Engineering Leader needs to be someone who works on an appropriate level in the organization to carry on the vision described in the proposal. For example, changes that involve only one team can be approved by an Engineering Manager, multiple teams within a one section - a Director of Engineering, but changes that span more than one department might require approval from a Executive VP or CEO.

Understanding who is the decision maker will make it easier to find an Architecture Evolution Coach because the best Architecture Evolution Coach will be someone who works on the level that is closest to the level of the Engineering Leader who is the decision maker (see the diagram above).

Domain Expert is someone who knows most about the topic and this person can work on any level in the organization, it even can be the author themself.

Once the blueprint of the proposal gets approved and merged, DRIs will be assigned to carry on the vision and coordinate work required to get it done.

All these people are here to amplify the influence of the author of the proposal in an environment that fosters creativity and knowledge sharing.

Role of an Architecture Evolution Coach

Architecture Evolution Coach is an expertise assigned to team members, visible on their profiles on the team page.

All Engineering Fellows and Distinguished Engineers are Architecture Evolution Coaches by default.

The purpose of involving a coach in the process of creating a blueprint is to allow people that know most about GitLab to share their knowledge and perspective on introducing complex architectural changes. This makes it easier to avoid the cost of working on the wrong thing, iterating improperly and fosters knowledge sharing.

Role of a Domain Expert

Domain Expert Engineer is an engineering role at GitLab held by team members that are engineers with a deep experience in a particular area.

  1. Domain Experts help to ensure conceptual integrity of the features and changes their groups / stages / sections are working on,
  2. Domain Experts help to collaborate with EMs and PMs and other Engineers to ensure the quality of work done in their area of interest,
  3. Domain Experts help to help to plan and draft necessary architectural and conceptual changes that will become a leverage in their area of interest.

A Domain Expert is an engineer, usually an Individual Contributor, who knows most about a codebase and a domain in the area of proposed changes, but might still lack the deep understanding of the process behind introducing complex architectural changes, hence the collaboration between a Domain Expert and an Architecture Evolution Coach might be very useful.

Sometimes there is an Architecture Evolution Coach available who is also a Domain Expert in a particular area. In that case there is no need to involve another person.

Proposing changes

Anyone can open an issue about a change they believe we should work on.

It can be a complex backstage improvement, an architectural change, a productivity improvement, or an efficiency improvement. It can be anything else that is too complex for a single individual contributor to handle.

Although we usually prefer starting with a merge requests, in case of complex changes like that, a merge request might not be something that is actionable, so we usually start with an issue.

Creating an issue is easier and less visible, it is an additional step before creating a draft merge request that helps to overcome anxiety related to speaking up and proposing big improvements. It is easier to capture ideas in issues than to create a merge request with a blueprint of a solution.

The issue should be labeled with ~"Architecture decision" label, can be made visible in the #architecture-decision Slack channel as per cross-functional collaboration.

Then author of the issue loops an Architecture Evolution Coach and a Domain Expert in and they collaborate together to create a blueprint merge request. Collaboration between the coaches and a person who had an idea ensures that only proposals that that are achievable get described in a blueprint merge request. This makes it also easier to avoid the cost of iterating on the product architecture improperly and fosters knowledge sharing.

Blueprint merge request

A blueprint merge request is a description of Why, How and What of the change that has been proposed in the issue.

A blueprint merge request gets created and made visible as a result of collaboration between an Architecture Evolution Coach, a Domain Expert and a person who had an idea. The author and coaches also need to be mentioned in the blueprint.

It needs to be written as a Markdown page in the handbook/engineering/architecture/blueprints directory in www-gitlab-com project.

It describes the goal of the change and the 1-year landscape of how to make it happen.

Describing iterations

At this point there is a blueprint merge request present in www-gitlab-com project, but it only describes the 1-year landscape. It is not precise enough to reason about how to get it done, how to iterate wisely to get the most of this work.

People that got involved into the discussion in the blueprint merge request collaborate together to extend it with a 3-months landscape. More Domain Experts get involved if necessary. The result of this collaboration could be a description of three first iterations that can be done in a one milestone each. These iterations need to be a two-way-door solutions with a measurable impact.

If it is not possible to find at least two iterations, the blueprint merge request can not be approved and merged.

Once the iterations are described, in the merge request, the blueprint needs to get assign to an Engineering Leader for approval and merge.

Getting the blueprint merged

The blueprint merge request needs to be approved and merged by the Engineering Leader who has been chosen as a final decision maker.

The choice of the leader depends on the extent of proposed changes, the area that the changes are supposed to be applied to and perceived cost of this architectural change.

Organization structure chart can be useful to determine who the Engineering Leader could be.

Finding DRIs

Once the merge request gets merged, the Engineering Leader who merged the proposal collaborates with people involved to find Directly Responsible Individuals who will be decision makers from now on and will be responsible for the progress.

The blueprint needs three people that will become DRIs:

  1. An Engineering Lead (for example - Director of Engineering)
  2. A Product Lead (for example - Senior Product Manager)
  3. A Domain Expert (for example - Senior Backend Engineer)

The Engineering Leader who approved the proposal can become an Engineer Lead DRI, but they can also delegate this to someone else. It is important to choose people taking their area of interest and responsibility into account and the "How" description that depends on where the proposed change needs to happen, who knows the most about particular area of the product, service, and codebase.

Then DRIs will propagate the blueprint downstream, to respective teams that will need to be involved, and these teams will schedule the work based on their interpretation of 1-year landscape and proposed iterations that will happen in the next 3 months. If a Working Groups gets formed the list of DRIs can be extended according to the process of how Working Groups organize efforts around the work.

Getting things done

DRIs can decide to form a Working Group to structure the efforts related to the architecture change better, but whether a Working Group should be formed or not depends on the size of the change and if forming a Working Group actually makes sense in this particular case.

The concept of a Working Groups can be an extension of the Architecture Evolution Workflow, but if it is not applicable in a particular case, a different process can be followed, like the suggested one that is described below.

  1. A bi-weekly call should be scheduled to coordinate the work with all people involved,
  2. Teams assigned will allocate resources for the work that needs to be done,
  3. Domain Experts working on the teams will prepare their interpretations of 3-months iterations,
  4. The work will be split into smaller issues and people interested in taking ownership of these issues will be assigned,
  5. Product Lead DRI will coordinate the work with Product Managers in respective Teams,
  6. An Engineering Lead DRI will coordinate the work with Engineering Managers responsible for respective teams,
  7. Domain Experts and individual engineers will be invited to the bi-weekly sync meeting.

What happens next

After the iterations described in the blueprint are done, the work can be extended to the next three iterations and the blueprint needs to be updated.

Alternatively the work can be concluded and the blueprint needs to be updated with results / outcomes.

GIT is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license