GitLab Engineering values clear, concise, transparent, asynchronous, and frequent communication. Here are our most important modes of communication:
As part of a fully distributed organization such as GitLab, it is important to stay informed about engineering led initiatives. We employ multimodal communication, which describes the minimum set of communication channels we'll broadcast to.
For the Engineering department, any important initiative will be announced in:
#eng-week-in-review
#development#production#security#support-managers#quality#uxIf you frequently check any of these channels, you can consider yourself informed. It is up to the person sharing to ensure that the same message is shared across all channels. Ideally, this message should be a one sentence summary with a link to an issue to allow for a single source of truth for any feedback.
We optimize for shipping a high volume of user/customer value with each release. We do want to ship multiple major features in every monthly release of GitLab. However, we do not strive for predictability over velocity. As such, we eschew heavyweight processes like detailed story point estimation by the whole team in favor of lightweight measurements of throughput like the number of merge requests that were included or rough estimates by single team members.
There is variance in how much time an issue will take versus what you estimated. This variance causes unpredictability. If you want close to 100% predictability you have to take two measures:
Both measures reduce the overall velocity of shipping features. The way to prevent this is to accept that we don't want perfect predictability. Just like with our OKRs, which are so ambitious that we expect to reach about 70% of the goal, this is also fine for shipping planned features.
Note: This does not mean we place zero value on predictability. We just optimize for velocity first.
We're currently in a time of rapid growth (Engineering grew 100% in 2018 and is planned to grow by 130% in 2019). But we value quality hires over meeting numbers in the hiring plan. We rely primarily on the judgement of our hiring managers to hold the bar high. But we also try to systematize as much as possible so our hiring practices are fair, transparent, and repeatable.
The VP of Engineering and their direct reports track our highest priorities in the Engineering Management Issue Board, rather than to do lists, Google Doc action items, or other places. The reasons for this are:
Here are the mechanics of making this work:
Engineering Management label to get it on the board, and the department label to get it in progress (e.g. Development Department)CEO Interest label, please post it to #ceoHere is the standard, company-wide process for OKRs. Engineering has some small deviations from (and extensions to) this process.
This process should begin no later than two weeks before the end of the preceeding quarter. And kickoff should happen on or before the first day of the new quarter.
FY20-Q2 Organization Type OKR: Objective phrase => 0%
FY20-Q3 Engineering Product OKR: Build our product vision => 0%* Raise first reply-time SLA for premium from 92% to 95% => 0%* Department: [Objective phrase](https://placeholder.com/) => 0% e.g. * Support: Raise first reply-time SLA for premium from 92% to 95% => 0%This process should begin on the first day of the subsequent quarter, and complete no later that two weeks after.
=> 70%) and assign it to their direct manager for review.In GitLab Engineering we are serious about concepts like servant leadership, over-communication, and furthering our company value of transparency. You may have joined GitLab from another organization that did not share the same values or techniques. Perhaps you're accustomed to more corporate politics? You may need to go through a period of "unlearning" to be able to take advantage of our results-focused, people-friendly environment. It takes time to develop trust in a new culture.
Less common, but even more important, is to make certain you don't unintentionally bring any mal-adaptive behaviors to GitLab from these other environments.
We encourage you to read the engineering section of the handbook as part of your onboarding, ask questions of your peers and managers, and reflect on how you can help us better live our culture:
We dogfood everything. Based on our product principles, it is the Engineering division's responsibility to dogfood features or do the required discovery work to provide feedback to Product. It is Product's responsibility to prioritize improvements or rebuild functionality in GitLab.
An easy antipattern to fall into is to resolve your problem outside of what the product offers. Dogfooding is not:
Follow this process to make sure deliberate Dogfooding decisions are made by the PM responsible for a given DevOps Stage and memorialized. It is okay to resolve an urgent problem if GitLab does not provide certain functionality. Just follow this process after to see if something needs to be rebuilt.
Dogfooding and spur a discussion with PM. This label should never be removed so the decision making process get memorialized on the Dogfooding board.Dogfooding::Rebuild in GitLab: For existing work that was already built outside GitLab that needs to be rebuiltDogfooding::Build in GitLab: For new issues that should be built into GitLabDogfooding::Keep Outside GitLab: For issues that it's okay to build outside GitLabDogfooding::Use Existing Feature: To capture work to consume and existing, unused feature in GitLab by an internal customerHere is a board that visualizes all the Dogfooding scoped labels.
Check out this 10 minute discussion about dogfooding:
GitLab consists of many subprojects. A curated list of GitLab Repositories can be found at the GitLab Engineering Projects page.
When adding a repository please follow these steps:
gitlab-ce repo.CONTRIBUTING.md in the repository. It is easiest to simply copy-paste the DCO + License section verbatim from the gitlab-ce repo.CONTRIBUTING.md from the project's README.mdWhen changing the settings in an existing repository, it's important to keep communication in mind. In addition to discussing the change in an issue and announcing it in relevant chat channels (e.g., #development), consider announcing the change during the Company Call. This is particularly important for changes to GitLab CE and GitLab EE.
GitLab consists of many different types of applications and resources.
When you require escalated permissions or privileges to a resource to conduct task(s), or support for creating resource(s) with specific endpoints, please submit an issue to the Access Requests Issue Tracker using the template provided.
Below is a short list of supported technologies:
Before the beginning of each fiscal year, and at various check points throughout the year, we plan the size and shape of the Engineering and Product Management functions together to maintain symmetry.
The process should take place in a single artifact (usually a spreadsheet, current spreadsheet), and follow these steps:
Note: Support is part of the engineering function but is budgeted as 'cost of sales' instead of research and development. Headcount planning is done separately according to a different model.
The non support related departments within Engineering (Development, Infrastructure, Quality, Security, and UX) have an expense target of 20% as a percentage of revenue.
The Support target is 10% as a percentage of revenue.
Our product offering is growing rapidly. Occasionally we start new teams. Backend teams should map to our product categories. Backend teams also map 1:1 to product managers.
A dedicated team needs certain skills and a minimum size to be successful. But that doesn't block us from taking on new work. This is how we iterate our team size and structure as a feature set grows:
## Vision
...
## Mission
...
## Team Members
The following people are permanent members of the [Blank] Team:
<%= direct_team(manager_role: 'Engineering Manager, [Blank]') %>
## Stable Counterparts
The following members of other functional teams are our stable counterparts:
<%= stable_counterparts(role_regexp: /[,&] Blank/, direct_manager_role: 'Engineering Manager, [Blank]') %>
## Hiring
This chart shows the progress we're making on hiring. Check out our
[jobs page](https://about.gitlab.com/jobs/) for current openings.
<%= hiring_chart(department: '[Blank] Team') %>
## Common Links
* Issue Tracker
* Slack Channel
* ...
## How to work with us
...
New teams may benefit from holding a Fast Boot event to help the jump start the team. During a Fast Boot, the entire team gets together in a physical location to bond and work alongside each other.
To maintain our rapid cadence of shipping a new release on the 22nd of every month, we must keep the barrier low to getting things done. Since our team is distributed around the world and therefore working at different times, we need to work in parallel and asynchronously as much as possible.
That also means that if you are implementing a new feature, you should feel empowered to work on the entire stack if it is most efficient for you to do so.
Nevertheless, there are features whose implementation requires knowledge that is outside the expertise of the developer or even the group/stage group. For these situations, we'll require the help of an expert in the feature's domain.
In order to figure out how to articulate this help, it is necessary to evaluate first the amount of work the feature will require from the expert.
If the feature only requires the expert's help at an early stage, for example designing and architecting the future solution, the approach will be slightly different. In this case, we would require the help of at least two experts in order to get a consensual agreement about the solution. Besides, they should be informed about the development status before the final solution is finished. This way, any discrepancy or architectural issue related to the current solution, will be brought up early.
We need to maintain code quality and standards. It's very important that you are familiar with the Development Guides in general, and the ones that relates to your group in particular:
Please remember that the only way to make code flexible is to make it as simple as possible:
A lot of programmers make the mistake of thinking the way you make code flexible is by predicting as many future uses as possible, but this paradoxically leads to *less* flexible code.
— Nearby Cats (@BaseCase) January 16, 2019
The only way to achieve flexibility is to make things as simple and easy to change as you can.
It is important to remember that quality is everyone's responsibility. Everything you merge to master should be production ready. Familiarize yourself with the definition of done.
We use SRE-like error budgets in OKRs to incentivize risk management and help make GitLab.com ready for mission critical customer workloads.
Each backend and frontend development team is responsible for not exceeding an allocated budget of 15 points each quarter. The severity of issues caused will impact their budget accordingly:
The Infrastructure team will perform attribution as part of the root cause analysis process and record the results in the OKRs page.
Engineering is the primary advocate for the performance, availability, and security of the GitLab project. Product Management prioritizes all initiatives, so everyone in the engineering function should participate in the Product Management prioritization process to ensure that our project stays ahead in these areas. The following list should provide some guidelines around the initiatives that each engineering team should advocate for during their release planning:
support-fix label. You can filter on open MRs here.Part of our engineering culture is to keep shipping so users and customers see significant new value added to GitLab.com or their self-managed instance. To support rapid development, we focus on Rails page views by default. When an area of the application sees significant usage, we typically rewrite those screens as a VueJS single page app backed by our API, in order to maintain the best qualitative experience and quantitative performance.
The idea that working software is the primary measure of progress is one of the principles of agile software development. Demoing gets more eyes on the project to uncover bugs and reveal ambiguities in the product requirements. It's also a transparent and lightweight way to provide status to the rest of the organization. Developers should demo at least once a week with product managers present. Demo meetings should be kept to 30 minutes or less. The emphasis should be on the product requirements or acceptance criteria and less on the implementation details.
GitLab makes use of a 'Canary' stage. Production Canary is a series of servers running GitLab code in a production environment. The Canary stage contains code functional elements like web, container registry and git servers while sharing data elements such as sidekiq, database, and file storage with production. This allows UX code and most application logic code to be consumed by a smaller subset of users under real world scenarios before being made available to all users on GitLab.com.
The production Canary stage is forcibly enabled for all users visiting GitLab Inc. operated groups:
The Infrastructure department teams can globally disable use of production Canary when necessary. Individuals can also opt-out of using production Canary environments. However, opting-out does not include the aforementioned groups above.
To opt in/out, go to GitLab Version and move the toggle appropriately.
To verify that Canary is enabled, in the header, next to the GitLab logo will be a 'Next' icon, or use the performance bar (typing pb) in GitLab and watch out for the Canary icon next to the web server name.
When using any of the resources listed below, some rules apply:
janedoe-machine-for-testing.Every team member has access to a common project on Google Cloud Platform. Please see the secure note with the name "Google Cloud Platform" in the shared vault in 1password for the credentials or further details on how to gain access.
Once in the console, you can spin up VM instances, Kubernetes clusters, etc. Please remove any resources that you are not using, since the company is billed monthly. If you are unable to create a resource due to quota limits, file an issue on the Infrastructure issue tracker.
Every team member has access to the dev-resources project which allows everyone to create and delete machines on demand.
In general, most team members do not have access to AWS accounts. In case you need an AWS resource, file an issue on the Infrastructure issue tracker. Please supply the details on what type of access you need.
There are primarily two Slack channels which developers may be called upon to assist the production team when something appears to be amiss with GitLab.com:
#backend: For backend-related issues (e.g. error 500s, high database load, etc.)#frontend: For frontend-related issues (e.g. JavaScript errors, buttons not working, etc.)Treat questions or requests from production team for immediate urgency with high priority.
