The Sharding Group is the direct outcome of applying our value of Iteration to the direction of the Database Scalability Working Group. There are multiple proposals and ideas on increasing database scalability through different sharding implementations. The charter of this group is to explore some of these ideas, quickly iterate on them and validate proposals to provide a solution for scaling our database to accommodate 10 Million Daily Active Users (DAU) of GitLab.com, and lead the efforts to eventually implement the first iteration (MVC) of a sharded database that works with GitLab application at scale. As we iterate on these ideas, the list of potential outcomes includes:
As we brainstorm and iterate on sharding proposals, we will provide implementation details, prototypes, metrics, demos and documentation to support our hypotheses and outcomes.
The executive summary goals for the Sharding Group include
The following people are permanent members of the Sharding Group:
|Craig Gomes||Backend Engineering Manager, Database and Sharding|
|Dylan Griffith||Staff Backend Engineer, Sharding|
|Kamil Trzciński||Distinguished Engineer, Ops and Enablement|
|Thong Kuah||Staff Backend Engineer, Sharding|
|Yannis Roussos||Senior Backend Engineer, Sharding|
|Yorick Peterse||Staff Backend Engineer, Sharding|
The following members of other functional teams are our stable counterparts:
|Fabian Zimmer||Principal Product Manager, Geo|
Exception Ratio: 1 Distinguished Engineer, 3 Staff Engineers : Team
Justification: The Sharding team was formed exclusively to implement a sharding solution to solve our database limitations as outlined by the Database Scalability Working Group. The project requires in-depth knowledge of PostgreSQL as well as into the application itself. To borrow some phrasing from the working group: we recognize this problem cannot be solved to meet our needs and requirements if we limit ourselves to the database: we must consider careful changes in the application to make it a reality. With that being said, the DRI had to be selective to ensure a small team of 5 can achieve the above requirements to cover every corner of our database and codebase. All candidates were selected based on the required skills profile for the success of the project, to make the best mix and match by covering a spectrum of interests with a minimal group.
Where we can we follow the GitLab values and communicate asynchronously. However, there have a few important recurring meetings. Please reach out to the #g_sharding Slack channel if you'd like to be invited.
With the globally distributed nature of this team, it is unlikely that we will have a synchronous meeting where everyone will attend. When we have synchronous meetings we will record the meetings and share written summaries with the links to the recordings. Currently we have the following recurring meeting scheduled on Wednesdays.
We follow the GitLab engineering workflow guidelines. To bring an issue to our attention please add the
~"group::sharding" label along with any other relevant labels. If it is an urgent issue, please reach out to the Product Manager or Engineering Manager listed in the Stable Counterparts section above.
There are a few factors influencing some of our working agreements here. First, there is a lot of excitement about the goals and expected outcomes of this team. We are working on a really difficult problem, we have lofty expectations as to what we can deliver and many different potential solutions to investigate. Because of the critical nature of what we are going to deliver we need to be able to consistently and concisely describe our current efforts. Second, we are a newly formed team and we need to be explicit about how we work together to ensure success.
Currently we are asking team members to update their issues with:
~sharding::activeto issues that we know we will be working on in the near future. This helps us to keep our Sharding: Build](https://gitlab.com/groups/gitlab-org/-/boards/2594854?scope=all&utf8=%E2%9C%93&label_name=group%3A%3Asharding) tidy, especially the Open and Closed columns
weight=1to all issues so that we can view epic progress at a glance on our Group::Sharding Roadmap
We can continue to iterate on our working agreement as a team to ensure that we are a successful team and exceed expectations.
It is expected that we share progress early and often by creating recorded demos. There will be specific requests for demonstration topics, but unscheduled demonstrations are welcome too. When recording a demo it does not need to be polished or perfect. Guidelines below:
We should be aiming for sharing demos on a weekly basis.
There will be many opportunities for us to undertake proofs of concept (POC). When we do, we agree to timebox the POC to 4 weeks and ideally to line up with milestone boundaries. When we need to make exceptions to this agreement we will employ multimodal communication to ensure all stakeholders are properly apprised.
In the early phases of our work we will create implementation plans for review. As we create these implementation plans we will list them below.