You are here: Handbook Engineering Quality Department Roadmap On this page Overview
The Quality Engineering department roadmap is divided into multiple
Whole department roadmap view of all
tracks. Roadmap Management
Linking epics The track level epics and the Quality Engineering Department Roadmap epic are the only epics allowed to have child epics. Each numbered list item in the roadmap should be a link either to an epic or an issue. They should be added to one of the track level epics. Child epics that form a track should only have issues. Epic structure: Department roadmap. Tracks. Everything else.
Creating new epics In the spirit of keeping things easily discoverable and reducing unnecessary epics, please refrain from creating new epics unless there are 5 or more issues created/scoped for that new epic. Please refrain from creating new tracks level epics unless it is really nessessary. Tracks
Work to improve the overall test coverage.
A testcase management system to document all E2E tests and their types, and link to test reports.
Basic Google spreadsheet to plan type of meta data. => Done Use GitLab issues as testcase management using gitlab-org/quality/testcases project. Integration testcases in gitlab-org/quality/testcases with automated tests. Rspec reporter thar reference testcase from code so they can be updated/synced dynamically. Testcase Management as GitLab Native feature. Epic Cross-browser tests
Crossbrowser & Mobile Browser testing coverage and infrastructure Basic capability to run on other browsers besides chrome (IE11, Firefox). Internet Explorer, and Edge coverage. Other desktop browsers. Mobile browser coverage. Run smoke test on soon to be new stable versions of important browsers to detect issues early. Service-integration tests
QA: GitLab 3rd party Services/Integrations e2e testing Basic tests for LDAP and SAML. Github and Google OAuth tests. Jenkins and Jira integration tests. Visual-diff tests
Browser screenshot visual testing to catch visual bugs. Helps with validating layout, UX, and accessibility.
End-to-end visual regression validation. Lean on 3rd party tools to get something running fast and learn from the product. Basic pixel to pixel comparison implemented as part of GitLab. Visual coverage smoke test for all stage groups. Use visual diff as a GitLab Native feature. Performance tests Basic functional tests that creates a big issue and merge request. => Done First stress test environment for on-prem customers with test runs and monitoring. => Done Real 10,000 user reference architecture with customer reference traffic load testing. Security tests XSS functional tests. Mutation tests
Test planning process.
Roll out testplan format and planning process. Bake in test planning as part of feature planning process. Iterate and come up with a shorter version of the 10 min ACC framework.
Work that increases our efficiency and productivity.
Fault Tolerance Test retry. Dynamic Page Object locators validation. API client automatically retries on 5xx errors. Faster Execution
Running test faster.
Basic parallelization via runners. Parallelization at the process level for all E2E tests, exponential cost saving of CI runners. Run all tests at the same time, the whole suite takes only as long as the longest test. Evaluation of a subset of tests instead of running all the E2E tests depending on what changed. API Usage
Use API in all E2E tests. Achieve optimal test layering with API suite more than %60 of total tests.
Use API calls to build the most used resources (Group, Project, User) in tests that are not focused on testing these resources behaviors. => Done Use API calls in login and logout for smoke tests. Implement API fabrication for all the resources that support it. Use API calls for setting up all resources in every E2E tests. Roll out as a standard for every new test. Lean pyramid
Optimize test coverage across the layers of the test pyramid, to remove redundant tests and achieve higher coverage with greater efficiency.
Come up with standardized test data that can be seeded in all environments for productivity.
Test data curation, define a test datamodel which is static. Define better project structure for ease of debugging, more readability in automated test data output, better group, project and issue naming (not just using timestamps). Script to setup testdata and clean them up. Idempotent script based on API calls (E.g. adds project if missing, uses existing if exists). Setup 50% of planned test data from Phase 2 in GDK, Review Apps, Staging and Canary/Production. Setup 100% of planned test data from Phase 2 in GDK, Review Apps, Staging and Canary/Production. Test results
Work that will allow us to debug tests more easily. Includes better reporting and more informative artifacts.
Better readability in test output. Basic HTML reporting. Automated reporting into the Testcase management system. Automated test name update via linking of issue ID in test code. Test readability Standard Page Object method names for click navigation. TBD
Work that will help us triage issues and merge requests more efficiently.
Triage packages Team level triage packages. => Done Group level triage packages. => Done Summary report in group packages showing the amount of bugs for that group. Divide up into quadrants of severity and priority (S/P) labels Generate 5x5 grid heat map report on existing open bugs for all group triage packages. => Done Bug SLA summary report in group packages. New issue first triage SLA Grooming Basic reminder for issues and merge requests. => Done Merge requests that are open for a long time Merge requests that do not have appropriate team and throughput labels. Issues that are open for a long time (3 months / 6 months). Merge requests that do not have any labels or milestones. Enforce one team label per merge request. Automatically infer stage and group label from category labels Automatically infer team label from author. Automatic labelling via natural language processing of issue description. Test Failure Rotation TBD
Work involving metrics that will allow us to make good data-driven decisions and report them to stakeholders early.
Insights GitLab Insights prototype with initial metrics. => Done. Migrate GitLab Insights into GitLab. => Done. Bug metrics Overall creation rate of bugs. => Done. Creation rate of bugs displayed in each group stage's dashboard. On-prem customer incidents per month SLA metrics Mean time to resolve
New issue first triage
Work that helps in validating the release process.
Scheduling Milestone grooming introduction When a milestone ends, close expired milestone and bulk reschedule unfinished work (Issues& MRs) to the next milestone => Done Next iteration of closing milestones and moving issues and MRs to the next milestone Review apps Improve review apps reliability Make review app a mandatory testing gate with smoke tests. Shift QA tests to completely run against review apps, only orchestrated test run in the package-and-qa job. [Improve review apps usefulness, add testdata into review apps to ease testability[https://gitlab.com/groups/gitlab-org/-/epics/606].