The Quality Engineering Sub-Department helps facilitate the test planning process for all things related to Engineering work.
Future Disable feature by default until E2E test merged: If a feature is to be merged without a QA test,
it must be behind a feature flag (disabled by default) until a QA test is written and merged as well.The goal of our test planning process is to ensure quality and manage risk in an efficient and effective manner.
To achieve that goal when delivering a change or new feature, we have conversations that link test requirements to product requirements, identify risks and quality concerns, and review and plan test coverage across all test levels.
The output or deliverables of the test planning process are:
The deliverables are considered complete when the tests have been added/updated and the merge request has been merged.
At release kickoff we highlight some of the changes scheduled for the next release. The majority of our test planning work starts in the issues relevant to those changes, although this process can be applied to any change. Here is an overview of the process:
The following guidelines provide more detail, as well as suggested responsibilities for various roles.
spec/features, involve your counterpart Software Engineer in Test to review the merge request.Requires e2e tests label to the feature issue or merge request.
master and released to production.Finally, once all test deliverables are completed, the feature issue can be closed (along with a test plan, if one was created).
Everyone in engineering is expected to contribute to Quality and keep our test pyramid in top shape. For every new feature we aim to ship a new slice of the pyramid so we don't incur test automation debt. This is what enables us to do Continuous Delivery.
We do not require a test plan for every feature or epic. Test plans are expensive to create and we would only limit this to high-impact and cross functional changes. There is no strict guideline for this and we defer this decision to each engineering group or team.
Examples of work that's likely to warrant a test plan:
GitLab's test plan is based on Google’s 10 min test plan. This test plan uses the ACC Framework (Attribute, Components and Capabilities matrix)
Currently we have a test plan template that is available for test planning in GitLab. A test plan can be created by anyone involved in a change.
You can see the all the existing test plans here: GitLab Test Plans.
In addition to quality, security is also one of our top concerns. With every change and every new feature we need to consider security risks and mitigate them early in the development process.
See also when and how to request a security review.
The test plan template has a risk column dedicated to security, however, whether or not a test plan is not created, it is essential that a discussion about security risks takes place as early as possible.
The Quality Department has coverage to support different test scenarios. More information is available on our Test Coverage page.
We are currently evaluating using GitLab mechanics to track test coverage in gitlab-org/quality/testcases.
Our previous iteration can be seen at DEPRECATED - GitLab Blackbox Integration & E2E Test Coverage.
An explanation of why we chose to do it this way is explained in this issue: Test case management and test tracking in a Native Continuous Delivery way.
To summarize, we want to track our tests in a Native Continuous Delivery way.
Another way to gauge test coverage and mitigation of risks is to use a risk map. Making a risk map will show not only where the product is not tested, but also how severe the impact of a failure in that area is likely to be. Knowing both the risk area and the impact a failure could produce helps guide where to begin implementing future tests.
The Global Search team has undertaken this effort before here: https://gitlab.com/gitlab-org/gitlab/-/issues/229431. If future teams attempt to implement a risk map this documentation may grow over time to reflect their experiences.
Read more on the risk mapping page.
Testing can sometimes be reduced to the simplest of checks for a specific behavior of a specific function. We often do this when we're under a time constraint. But is that how our customers always use our software?
By taking just a little extra time in our test approach to think about testing in general, we can save hours of work rolling back a release or producing hotfixes. Here are some simple approaches to enhancing our testing without adding a significant amount of time to our process.
This is especially applicable to any exploratory testing performed by both software engineers in test and backend/frontend engineers alike. They can also be helpful to consider when designing automated tests as they help eliminate bias.
Even simple solutions can have infinite variations to test when we consider the data, timing, environmental or platform factors, configuration, and user behavior following varied sequences actions. We should have as our goal to find the most important and most interesting variations to test, since we cannot possibly test them all.
Some variables are obvious, some are not. Some are accessible to the end user, some are indirectly accessible, modified only indirectly through changing other variables. For example, size or number of files in a database or number of users logged into a system may affect how a feature functions but are outside of the feature itself and may not be directly modified without specific environments or tools.
So our goal here is to identify the variables that may play a part in how a particular feature functions and use these as touch points for testing that feature. Since we may have a number of variables to consider, it would be wise to create simple notes to track those variables to make our testing more effective.
The definition of a heuristic is varied based on where we look, so we'll use the simple definition: a heuristic is a fallible method for solving a problem or making a decision. They are not algorithms, so they do not always give us the results we want. We should think of them as excellent thinking tools to help us navigate unknowns, such as "did my change introduce a bug?". They are dependent upon context, so not every heuristic is applicable to every problem, just like algorithms.
Here are some heuristics that may help us find variables:
How can we use these heuristics? Are there others? The next section answers those questions.
git push.This is not an exhaustive list of heuristics. If you know others you have found useful, please consider contributing them to this document.
Accreditation to these ideas belongs to the general test community, many who have contributed to and compiled these ideas in different ways. For further explanation of applying heuristics to testing, see Elisabeth Hendrickson, VP of R&D at Pivotal Software, Inc., and her book titled Explore It! or many other various blogs and sites, e.g. Satisfice, DevelopSense, or Ministry of Testing. For an in-depth explanation of heuristics in the context of software testing, see Heuristics for Understanding Heuristics.