The Pods Group (formerly Sharding group) is focused on a long-term horizontal scaling solution for GitLab. There are multiple proposals and ideas to increase horizontal scalability via solutions such as database sharding and tenant isolation. The charter of this group is to explore, iterate on, validate, and lead implementation of proposals to provide a solution to accommodate Gitlab.com's daily-active user growth.
As we brainstorm and iterate on horizontal scalability proposals, we will provide implementation details, prototypes, metrics, demos and documentation to support our hypotheses and outcomes.
Note: As we iterate towards a rename of the Sharding group to Pods, this page will contain references to both Pods and Sharding.
The executive summary goals for the Pods Group include:
The following people are permanent members of the Pods Group:
The following members of other functional teams are our stable counterparts:
| Person | Role |
|---|---|
| Fabian Zimmer | Group Product Manager, Enablement |
Exception Ratio: 1 Distinguished Engineer, multiple Staff Engineers : Team
Justification: The Pods team was formed (originally as the Sharding team) to implement a sharding solution to solve our database limitations as outlined by the Database Scalability Working Group. The project requires in-depth knowledge of PostgreSQL as well as into the application itself. To borrow some phrasing from the working group: we recognize this problem cannot be solved to meet our needs and requirements if we limit ourselves to the database: we must consider careful changes in the application to make it a reality. With that being said, the DRI had to be selective to ensure a small team of can achieve the above requirements to cover every corner of our database and codebase. All candidates were selected based on the required skills profile for the success of the project, to make the best mix and match by covering a spectrum of interests with a minimal group. There have also been multiple Staff engineers borrowed from other teams at various times to execute on these goals. Going forward, we anticipate the team will continue to require the expertise of multiple staff+ level engineers to accomplish our horizontal scalability goals.
Where we can we follow the GitLab values and communicate asynchronously. However, there have a few important recurring meetings. Please reach out to the #g_sharding Slack channel if you'd like to be invited.
With the globally distributed nature of this team, it is unlikely that we will have a synchronous meeting where everyone will attend. When we have synchronous meetings we will record the meetings and share written summaries with the links to the recordings. Currently we have the following recurring meeting scheduled.
We follow the GitLab engineering workflow guidelines. To bring an issue to our attention please add the ~"group::sharding" label along with any other relevant labels. If it is an urgent issue, please reach out to the Product Manager or Engineering Manager listed in the Stable Counterparts section above.
~"group::sharding"
There are a few factors influencing some of our working agreements here. First, there is a lot of excitement about the goals and expected outcomes of this team. We are working on a really difficult problem, we have lofty expectations as to what we can deliver and many different potential solutions to investigate. Because of the critical nature of what we are going to deliver we need to be able to consistently and concisely describe our current efforts. Second, we are a newly formed team and we need to be explicit about how we work together to ensure success.
Currently we are asking team members to update their issues with:
~sharding::active to issues that we know we will be working on in the near future. This helps us to keep our Sharding: Build tidy, especially the Open and Closed columnsWe can continue to iterate on our working agreement as a team to ensure that we are a successful team and exceed expectations.
It is expected that we share progress early and often by creating recorded demos. There will be specific requests for demonstration topics, but unscheduled demonstrations are welcome too. When recording a demo it does not need to be polished or perfect. Guidelines below:
We should be aiming for sharing demos on a weekly basis.
There will be many opportunities for us to undertake proofs of concept (POC). When we do, we agree to timebox the POC to 4 weeks and ideally to line up with milestone boundaries. When we need to make exceptions to this agreement we will employ multimodal communication to ensure all stakeholders are properly apprised.
In the early phases of our work we will create implementation plans for review. As we create these implementation plans we will list them below.
(Sisense↗) We also track our backlog of issues, including past due security and infradev issues, and total open SUS-impacting issues and bugs.
(Sisense↗) MR Type labels help us report what we're working on to industry analysts in a way that's consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.