In 13.9 Team Geo released Maintenance Mode, which was a large, cross stage and cross team project, a few milestones in the making.
This feature allows system administrators to put GitLab in a read-only mode. All parts of the system are affected and testing such a wide scope was challenging.
Why was testing this feature hard?
As we started testing with the QA team, it was clear that no one individual or team could know enough about the entire product to design a comprehensive QA plan. The more we tested, the more features we found to test - it was soon becoming an impossibly long list of tests to write for our small team.
We needed to prioritize manually testing the most important features, and save working on automated tests for another iteration.
But, what were the most important things to test?
This is where we decided to crowd-source testing. We rolled-out discussion issues to each of the 13 stages and asked them to contribute the three most important features that they own, that we should prioritise testing.
We used these issues to share knowledge of maintenance mode, and responsibility of its development, testing and documentation.
The response was overwhelming!
Product managers and engineers from across the development department contributed to our list of tests and collaboratively reviewed and improved documentation. They proactively asked how their features would behave and in some cases, even started MRs to fix the documentation.
The conversations helped us hone our plan for future iterations of this feature.
What we learned
1. Test iteratively and collaboratively
Get QA and developer teams working together early, instead of after development is almost done, or worse - after release. GitLab's Quad planning process was introduced last year to foster better collaboration between Quality, Development, UX, and Product teams. As Jennie from QA chalked out a plan for QA together with developers, she found a few edge cases that would have otherwise been discovered too late.
2. Don’t hesitate to ask other teams to contribute
When we rolled out a dozen plus issues to all development teams, we were not sure if we’d get even a few responses, but we were overwhelmed with the interest, response and active participation that came from all the teams.
3. Communicate well
Give people enough and succinct information. When requesting help from other teams, help them prioritize the request by explaining the why.
4. Documentation as a form of developer communication
As we worked through large documentation MRs, I realized the documentation was not only important for system administrators, but for developers of GitLab as well. Developers wanted to know how maintenance mode affected their features.
5. Iterate
Keep the discussions short-lived and focused on the most important aspects. Do not draw out the conversations too long, and move pending conversations over to follow-up issues. As we learned of new test cases, Nick from QA and I created follow-up test issues to resolve together with DRIs.
6. The more, the merrier
While the discussions started only with Engineering Managers and Product Managers, they often invited engineers in their conversations and this brought more eyes to the project and helped us answer a lot of unknowns.