In addition to Quality, Security is also our top concern. It is expected that with every change and every new feature we need to factor in security risks and mitigate them early in the development process.
The test plan template has a risk column dedicated to Security. The following should be the top priority while assessing risk in Security.
To summarize, we want to track our tests in a Native Continuous Delivery way.
If we are doing continuous delivery in the right way, there should be little to no manual tests. Manual tests if any should be minimal and exploratory.
There should be little need for detailed test steps if manual testing is minimal and exploratory.
Exploratory testing has a free form emphasis which removes the importance of hardcoded test steps.
The disadvantage of going into too much detail with hardcoded test steps is we are not able to catch bugs outside the hardcoded flow in the document, it's also a high maintenance document.
Our test case name / description contains just enough description, this helps fuzzes the workflow up and we end up catching more critical bugs this way.
An emphasis on risk-based test planning using the ACC framework.
An emphasis on categorizing test automation types which tracks the pyramid shape (API, UI, and Visual).
An emphasis on tracking alignment in the lower test levels in the pyramid.
An overview of the test management view is shown below.
Test Planning Process
At release kickoff test plans are created by the Quality / Test Automation Engineer who is assigned to a product area.
Test strategy is then discussed and collaborated with clear test deliverable and coverage expectation.
Each team members is responsible for completing the test deliverable.
Quality / Test Automation Engineers ensure that we complete the test coverage as planned before the feature is merged into master and released to production.
If a feature requires an End-to-End test, the Quality / Test Automation Engineer adds Requires e2e tests label to the feature MR.
Before merging the feature MR, the developer or Quality / Test Automation Engineer ensures the e2e test MR is linked to the feature MR and checks off "Link to e2e tests MR added if this issue/MR has requires e2e tests label".
Everyone in engineering is expected to contribute to Quality and keep our test pyramid in top shape. For every new feature we aim to ship a new slice of the pyramid so we don't incure test automation debt. This is what enables us to do Continous Delivery.