You are here:
Handbook Engineering Quality Department Roadmap
The Quality Engineering department roadmap is divided into multiple
Whole department roadmap view of all
tracks. Roadmap Management
track level epics and the Quality Engineering Department Roadmap epic are the only epics allowed to have child epics. Each numbered list item in the roadmap should be a link either to an epic or an issue. They should be added to one of the
track level epics. Child epics that form a
track should only have issues. Epic structure:
Creating new epics
In the spirit of keeping things easily discoverable and reducing unnecessary epics, please refrain from creating new epics unless there are 5 or more issues created/scoped for that new epic.
Please refrain from creating new
tracks level epics unless it is really nessessary. Tracks
Work to improve the overall test coverage.
A testcase management system to document all E2E tests and their types, and link to test reports.
Google spreadsheet to plan type of meta data. => Done Use GitLab issues as testcase management using
gitlab-org/quality/testcases project. Integration testcases in
gitlab-org/quality/testcases with automated tests. Rspec reporter thar reference testcase from code so they can be updated/synced dynamically. Testcase Management as GitLab Native feature.
Epic Cross-browser tests
Crossbrowser & Mobile Browser testing coverage and infrastructure
Basic capability to run on other browsers besides chrome (IE11, Firefox). Internet Explorer, and Edge coverage.
Other desktop browsers.
Mobile browser coverage.
Run smoke test on soon to be new stable versions of important browsers to detect issues early.
QA: GitLab 3rd party ecosystem testing
Basic tests for LDAP and SAML.
Github and Google OAuth tests.
Jenkins and Jira integration tests.
Browser screenshot visual testing to catch visual bugs. Helps with validating layout, UX, and accessibility.
End-to-end visual regression validation.
Lean on 3rd party tools to get something running fast and learn from the product.
Basic pixel to pixel comparison implemented as part of GitLab.
Visual coverage smoke test for all stage groups.
Use visual diff as a GitLab Native feature.
Basic functional tests that creates a big issue and merge request. => Done
First stress test environment for on-prem customers with test runs and monitoring. => Done
Real 10,000 user reference architecture with customer reference traffic load testing.
XSS functional tests.
Data analysis and action items
Depends on the above
Test planning process.
Roll out testplan format and planning process.
Bake in test planning as part of feature planning process.
Iterate and come up with a shorter version of the
10 min ACC framework.
Work that increases our efficiency and productivity.
Dynamic Page Object locators validation.
API client automatically retries on 5xx errors.
Running test faster.
Basic parallelization via runners.
Parallelization at the process level for all E2E tests, exponential cost saving of CI runners.
Run all tests at the same time, the whole suite takes only as long as the longest test.
Evaluation of a subset of tests instead of running all the E2E tests depending on what changed.
Use API in all E2E tests. Achieve optimal test layering with API suite more than %60 of total tests.
Use API calls to build the most used resources (Group, Project, User) in tests that are not focused on testing these resources behaviors. => Done Use API calls in login and logout for smoke tests.
Implement API fabrication for all the resources that support it.
Use API calls for setting up all resources in every E2E tests. Roll out as a standard for every new test.
Optimize test coverage across the layers of the test pyramid, to remove redundant tests and achieve higher coverage with greater efficiency.
Come up with standardized test data that can be seeded in all environments for productivity.
Test data curation, define a test datamodel which is static.
Define better project structure for ease of debugging, more readability in automated test data output, better group, project and issue naming (not just using timestamps).
Script to setup testdata and clean them up.
Idempotent script based on API calls (E.g. adds project if missing, uses existing if exists).
Setup 50% of planned test data from Phase 2 in
GDK, Review Apps, Staging and Canary/Production. Setup 100% of planned test data from Phase 2 in
GDK, Review Apps, Staging and Canary/Production. Test results
Work that will allow us to debug tests more easily. Includes better reporting and more informative artifacts.
Better readability in test output.
Basic HTML reporting.
Automated reporting into the Testcase management system.
Automated test name update via linking of issue ID in test code.
Standard Page Object method names for click navigation.
Work that will help us triage issues and merge requests more efficiently.
Team level triage packages. => Done
Group level triage packages. => Done
Summary report in group packages showing the amount of bugs for that group. Divide up into quadrants of severity and priority (S/P) labels
Generate 5x5 grid heat map report on existing open bugs for all group triage packages. => Done Bug SLA summary report in group packages.
New issue first triage SLA Grooming
Basic reminder for issues and merge requests. => Done
Merge requests that are open for a long time
Merge requests that do not have appropriate team and throughput labels.
Issues that are open for a long time (3 months / 6 months).
Merge requests that do not have any labels or milestones.
Enforce one team label per merge request.
Automatically infer stage and group label from category labels
Automatically infer team label from author.
Automatic labelling via
natural language processing of issue description.
Work involving metrics that will allow us to make good data-driven decisions and report them to stakeholders early.
GitLab Insights prototype with initial metrics. => Done.
Migrate GitLab Insights into GitLab. => Done.
Overall creation rate of bugs. => Done.
Creation rate of bugs displayed in each group stage's dashboard.
On-prem customer incidents per month SLA metrics
Mean time to resolve
New issue first triage
Work that helps in validating the release process.
Milestone grooming introduction
When a milestone ends, close expired milestone and bulk reschedule unfinished work (Issues& MRs) to the next milestone => Done
Next iteration of closing milestones and moving issues and MRs to the next milestone Review apps
Improve review apps reliability Make review app a mandatory testing gate with smoke tests.
Shift QA tests to completely run against review apps, only orchestrated test run in the package-and-qa job.
[Improve review apps usefulness, add testdata into review apps to ease testability[https://gitlab.com/groups/gitlab-org/-/epics/606].