One of GitLab's core strategies is to build on our open core strength. We believe that building a strong community of contributors is key to the long-term success of GitLab. We believe in a dual-flywheel strategy that focuses on both product contributions from within our GitLab engineering team and community contributions.
Our goal is to grow to 1000 contributors per month. The saying is that "All roads lead to Rome," but of course not all of those roads are the most efficient ways to get there. To succeed, contributing to GitLab must be a rewarding and incentivizing experience that motivates contributors to come back. One of the strategic choices we made in the contributor success team is the route of being as responsive and clear as we can about the next steps, using processes and automation.
Problem statement
So where do we start? On average GitLab has over 550 open merge requests. We wanted to focus on the ready enough problem. When is an MR ready for review? And when is an MR still in development? In both cases collaboration is required, but having a formal handoff — meaning this contribution is ready for a review — helps in understanding who is blocked from moving forward. Before a merge request can find its way into GitLab, it needs to get a review from at least one maintainer.
How do we know when to ask specific maintainers of our product areas to put their focus on reviewing these merge requests? When is a merge request ready enough for a thorough review? What does ready enough even mean?
Some OSS communities use crowdsourced reviews for contributions to make sure the project maintainers don’t need to take on everything by themselves. For example, in the Drupal community there is the concept of Reviewed and Tested by the Community. At GitLab we have MR coaches and community help to make sure everything is as ready as can be before involving the maintainers.
The GitLab bot and our MR coaches try to assist the wider community contributors on their way. We also had a ping-pong label that tried to signify if a community contributor had reacted and it was ‘ping ponged 🏓’ back to the GitLab team members. This pingpong label didn’t take the context into account. It was a great iterative and boring solution to know who was up next (the author or the reviewer). But it had a lot of false-positives and caused confusion to both the maintainers and the community contributors.
So where do we go from here? How do we get a better grasp on this ready enough problem? Let’s start by asking for help from our recommended reading list, High-Output Management. Author Andrew S. Grove states: “A very important way to increase productivity is to arrange the work flow inside our black box so that it will be characterized by high output per activity, which is to say high-leverage activities.”
Introducing workflow labels
For a while, GitLab team members were using workflow labels to signify the state of a merge request. It wasn’t always used across all teams, but they were available. More specifically we’re looking at the following labels:
workflow::ready for reviewworkflow::in devworkflow::blocked
Each wider community contributor is now able to change these labels themselves. By using @gitlab-bot ready, it sets the state to workflow::ready for review and assigns a reviewer. The reviewer is able to set it back to workflow::in dev if there are still items to be addressed. Other wider community members can also leave comments or suggestions for improvement, and then set the label back to workflow::in dev, or set other labels to help triage these merge requests.
What have we learned so far?
We started using this system over two months ago. We now know that around 20% of MRs on average are in a "ready for review" state. Those contributors are blocked and waiting for an answer to either continue to improve the merge request or get it merged if there are no more comments left. We also noticed that some merge requests were not getting a lot of attention. We did an async retrospective and feedback session with the GitLab team members and the wider community in order to find an answer on how we can improve the time it takes before a review is made for contributions that were ready for a review. We’re still processing that feedback, but are looking to try some of these suggestions, learn from them, and iterate. Even though GitLab cannot promise timely reviews, we can certainly try to build in mechanisms, and understand where we see limits, to navigate towards better processes. When we started out, we had a median time of 17 days that an MR was in the ready for review state. Today that median time has been reduced to five days!
The median Open Community MR Age (OCMA) has also dropped from 139 days in April to 78 days in August. Maybe it is a coincidence that we reached an all-time high of 126 contributors in August? Either way, all of the steps allowed our amazing wider community contributors to get 440 merge requests merged in a single month! I’m certain this change contributed, among other changes and initiatives, to that record. We will keep learning as we progress. It certainly allowed us to take a peek into our little black box.
What’s next?
Next up is to continue our iterations and move further towards automation. Right now it is up to the reviewer to set the
status back to workflow::in dev whenever there is something left to address. We notice that this is not always changed
back when it’s actually needed. It is also causing false-positives with reviewers and our wider community members.
The Contributor Success team is looking into how this can be automated. If you’d like to help, the automation happens in
the Triage Ops project and the Contributor Success issue
queue is open for everyone!
We’re also looking into a new program called Leading Organizations which rewards recurring contributors, and their organizations, with a review time objective of four business days. This would lead us to even shorter review cycle times and give those organizations that contribute to GitLab a competitive advantage to stay leaders in their domain. The faster we can innovate together, the faster our dual flywheel will spin. Together we go to infinity and beyond. Together we can build software fast.
