Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Roadmap

On this page

Overview

The Quality Engineering department roadmap is divided into multiple Tracks.

Whole department roadmap view of all tracks.

Roadmap Management

Linking epics

Creating new epics

Tracks

Coverage track (roadmap view)

Work to improve the overall test coverage.

Testcase Management

A testcase management system to document all E2E tests and their types, and link to test reports.

  1. Basic Google spreadsheet to plan type of meta data. => Done
  2. Use GitLab issues as testcase management using gitlab-org/quality/testcases project.
  3. Integration testcases in gitlab-org/quality/testcases with automated tests. Rspec reporter thar reference testcase from code so they can be updated/synced dynamically.
  4. Testcase Management as GitLab Native feature. Epic

Cross-browser tests

Issues: Crossbrowser & Mobile Browser testing coverage and infrastructure

  1. Basic capability to run on other browsers besides chrome (IE11, Firefox).
  2. Internet Explorer, and Edge coverage.
  3. Other desktop browsers.
  4. Mobile browser coverage.
  5. Run smoke test on soon to be new stable versions of important browsers to detect issues early.

Ecosystem tests

Epic: QA: GitLab 3rd party ecosystem testing

  1. Basic tests for LDAP and SAML.
  2. Github and Google OAuth tests.
  3. Jenkins and Jira integration tests.

Visual-diff tests

Browser screenshot visual testing to catch visual bugs. Helps with validating layout, UX, and accessibility.

Issue: End-to-end visual regression validation.

  1. Lean on 3rd party tools to get something running fast and learn from the product.
  2. Basic pixel to pixel comparison implemented as part of GitLab.
  3. Visual coverage smoke test for all stage groups.
  4. Use visual diff as a GitLab Native feature.

Performance tests

  1. Basic functional tests that creates a big issue and merge request. => Done
  2. First stress test environment for on-prem customers with test runs and monitoring. => Done
  3. Real 10,000 user reference architecture with customer reference traffic load testing.

Security tests

  1. XSS functional tests.

Mutation tests

Issue: Mutation tests

  1. Implement a POC for the ruby and JavaScript code to gather data
  2. Data analysis and action items
  3. Depends on the above

Test planning

Test planning process.

  1. Roll out testplan format and planning process.
  2. Bake in test planning as part of feature planning process.
  3. Iterate and come up with a shorter version of the 10 min ACC framework.

Efficiency track (roadmap view)

Work that increases our efficiency and productivity.

Fault Tolerance

  1. Test retry.
  2. Dynamic Page Object locators validation.
  3. API client automatically retries on 5xx errors.

Faster Execution

Running test faster.

  1. Basic parallelization via runners.
  2. Parallelization at the process level for all E2E tests, exponential cost saving of CI runners.
  3. Run all tests at the same time, the whole suite takes only as long as the longest test.
  4. Evaluation of a subset of tests instead of running all the E2E tests depending on what changed.

API Usage

Use API in all E2E tests. Achieve optimal test layering with API suite more than %60 of total tests.

  1. Use API calls to build the most used resources (Group, Project, User) in tests that are not focused on testing these resources behaviors. => Done
  2. Use API calls in login and logout for smoke tests.
  3. Implement API fabrication for all the resources that support it.
  4. Use API calls for setting up all resources in every E2E tests. Roll out as a standard for every new test.

Lean pyramid

Optimize test coverage across the layers of the test pyramid, to remove redundant tests and achieve higher coverage with greater efficiency.

Test data

Come up with standardized test data that can be seeded in all environments for productivity.

  1. Test data curation, define a test datamodel which is static. Define better project structure for ease of debugging, more readability in automated test data output, better group, project and issue naming (not just using timestamps).
  2. Script to setup testdata and clean them up.
    • Idempotent script based on API calls (E.g. adds project if missing, uses existing if exists).
  3. Setup 50% of planned test data from Phase 2 in GDK, Review Apps, Staging and Canary/Production.
  4. Setup 100% of planned test data from Phase 2 in GDK, Review Apps, Staging and Canary/Production.

Test results

Work that will allow us to debug tests more easily. Includes better reporting and more informative artifacts.

  1. Better readability in test output.
  2. Basic HTML reporting.
  3. Automated reporting into the Testcase management system.
  4. Automated test name update via linking of issue ID in test code.

Test readability

  1. Standard Page Object method names for click navigation.
  2. TBD

Triage track (roadmap view)

Work that will help us triage issues and merge requests more efficiently.

Triage packages

  1. Team level triage packages. => Done
  2. Group level triage packages. => Done
  3. Summary report in group packages showing the amount of bugs for that group. Divide up into quadrants of severity and priority (S/P) labels Generate 5x5 grid heat map report on existing open bugs for all group triage packages. => Done
  4. Bug SLA summary report in group packages. New issue first triage SLA

Grooming

  1. Basic reminder for issues and merge requests. => Done
    • Merge requests that are open for a long time
    • Merge requests that do not have appropriate team and throughput labels.
    • Issues that are open for a long time (3 months / 6 months).
    • Merge requests that do not have any labels or milestones.
  2. Enforce one team label per merge request.
  3. Automatically infer stage and group label from category labels
  4. Automatically infer team label from author.
  5. Automatic labelling via natural language processing of issue description.

Measure track (roadmap view)

Work involving metrics that will allow us to make good data-driven decisions and report them to stakeholders early.

Insights

  1. GitLab Insights prototype with initial metrics. => Done.
  2. Migrate GitLab Insights into GitLab. => Done.

Bug metrics

  1. Overall creation rate of bugs. => Done.
  2. Creation rate of bugs displayed in each group stage's dashboard.
  3. On-prem customer incidents per month

SLA metrics

  1. Mean time to resolve P1 and P2 defects.
  2. New issue first triage

Releases track (roadmap view)

Work that helps in validating the release process.

Scheduling

  1. Milestone grooming introduction When a milestone ends, close expired milestone and bulk reschedule unfinished work (Issues& MRs) to the next milestone => Done
  2. Next iteration of closing milestones and moving issues and MRs to the next milestone

Review apps

  1. Improve review apps reliability
  2. Make review app a mandatory testing gate with smoke tests.
  3. Shift QA tests to completely run against review apps, only orchestrated test run in the package-and-qa job.
  4. [Improve review apps usefulness, add testdata into review apps to ease testability[https://gitlab.com/groups/gitlab-org/-/epics/606].