Category Direction - Certify

The following page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features or functionality remain at the sole discretion of GitLab Inc.

Certify Group

Certify Group

Last Updated: 2023-01-24

This is the direction page for the Certify Group, which is part of the Plan stage of the DevOps life cycle. Certify Group is responsible for the following categories:

Category	Direction	Description	Maturity
Requirements Management	See the Requirements Management section	Manage functional requirements within GitLab	minimal
Quality Management	See the Quality Management section	Manage and trace test cases within GitLab	minimal

Team members

Person	Role
Matthew Macfarlane	Product Manager, Plan Stage, Certify Group
Dan Mizzi-Harris	Product Designer, Plan:Certify
Marcin Sędłak-Jakubowski	Technical Writer, Plan
Charlie Ablett	Senior Backend Engineer, Plan:Certify
Costel Maxim	Senior Security Engineer, Application Security, Plan (Project Management, Product Planning, Certify), Create:Source Code, Growth, Fulfillment:Purchase, Fulfillment:Provision, Fulfillment:Utilization, Systems:Gitaly
Felipe Artur	Senior Backend Engineer, Plan:Certify
Jarka Košanová	Fullstack Engineering Manager, Plan:Certify
John Hope	Senior Manager, Engineering, Plan
Nick Malcolm	Senior Security Engineer, Application Security, Manage (Compliance, Import), Plan (Optimize), Data Science (AI Assisted, Anti-Abuse, MLOps, DataOps)
Natalia Tepluhina	Principal Engineer, Plan
Rudy Crespo	Frontend Engineer, Plan:Certify
Melissa Ushakov	Group Manager, Product Management, Plan and Ecosystem

What are we working on and why

What: For 15.8 Certify Group continues its migration of existing Requirements Management capabilities to a Work Item type via deprecating the legacy Requirement IID, using the archive work item GraphQL query to archive Requirements in Requirements UI, and showing the new IID in the existing list view for Requirements.

Why: Our migration of Requirements to a Work Item type allows a more consistent and integrated experience in GitLab's enterprise level planning framework. This integrated experience is core to the Plan Stage Mission of empowering teams to continuously deliver customer and business value with the shortest possible cycle times.

Certify Group encapsulates both Requirements and Quality Management. The following is a breakdown of our current effort spent on each category and why.

Requirements Management: Allocation: 85%

Requirements Management is a critical process to ensure certain features or behaviors are satisfactorily and properly measured against their requirements. At GitLab we believe we can create a positive experience for our customers by creating a robust Requirements Management feature set directly within the platform, rather than via external tooling such as Jama Software, IBM DOORS, or Modern Requirements.

Our journey toward creating a competitive Requirements Management toolset begins with migrating existing requirements functionality from an independent object to a general Work Item type, which is part of a broader initiative in the Plan Stage to create greater continuity between Work Item types (such as Requirements, Issues, Epics, and Tasks). This will allow customers to integrate requirements directly into their development process.

Quality Management: Allocation 15%

Quality Management is the basis of testing and verifying features and behaviors against requirements. Given its interconnectiveness with Requiremens Management this category naturally fits under the Certify Group umbrella.

Our early focus in Quality Management is around tracing requirements to test cases in both a manual and automated fashion, creating more robust test case and test session capabilities, and allowing more traceability in the way in which requirements are audited.

Allocation of 15% capacity is set as of 2023-01-03 as Quality Management maturity requires Requirements Management capabilities to first be transitioned to a Work Item type.

Current Certify Goals

The Certify Group aims to provide capabilities to allow requirements based development and testing within GitLab. The belief is that bringing requirements and traceability within GitLab will yield less context switching for our users, and provide increased productivity.

One of the most time consuming aspects of requirements based development is providing traceability between source code, testing, and design. This often requires switching between multiple tools, and manually inputting and updating traced artifacts during code updates and test runs. Our aim is to start with the Software Developer and Software Engineering in Test personas, with the goal of solving their immediate frustrations.

At present, both Quality Management and Requirements Management categories are considered at minimal maturity.

It is our belief that solving the following three fundamental problems will bring both the Requirements Management and Quality Management capabilities to viable maturity.

Documenting requirements and test cases within GitLab

Status

Complete - Both Requirements and test cases can be created and managed within GitLab

It is our fundamental belief that for maximum efficiency and reduced cycle time, users should be able to complete their jobs to be done in a single cohesive application. This reduces the overall mental strain of switching between applications, but also allows for automation which can reduce manual steps.

Bi-directional linking between requirements, test cases and other artifacts

Status

In Progress - We aim to build on top of the current work item initiative provide bi-directional linking between requirements, test cases, and other Work Item types.

The intention of requirements based development is to provide traceability between the requirements, and the other product artifacts such as code, design, and test cases. If this traceability is not provided within GitLab, then external tooling will be necessary which reduces productivity.

Allow testing to satisfy requirements

Status

In Progress - It is possible to satisfy requirements from automated CI/CD pipelines within GitLab. Please check out our Walk-through of Requirements Traceability within GitLab.

We would like to extend this functionality to provide additional methods for linking requirements to test cases.

Certify Team Long term goals

Once Certify is a viable option for the Software Developer and Software Engineering in Test personas, we plan to continue iterating as follows:

GitLab continues to support larger enterprises, and the natural need for multiple levels of requirements and test cases which can be decomposed down to requirements or test cases at lower levels has risen. Our objective is to expand our Work Item definition to allow for multi-level objects. This would allow teams to create a system of sub-systems and perform all requirement tracing and test tracing directly within GitLab, further adhering to our mission as The One DevSecOps Platform.
We recognize that requirements and their associated trace data is often required as release evidence / artifacts. We would like to work closely with our release team to integrate requirements traceability into release evidence.
Visual representation of traceability and test coverage is also of importance. We would like to provide a visual representation of ancestors and descendants of requirements, making it easy to visualize decomposition and traceability. It would also be ideal for passing / failing test results to roll up visually to the requirements, allowing for quick visualization of the requirement status with regards to implementation and verification.

Focusing On Requirements Management

Maturity	Minimal
Documentation Link	Requirements Management

Requirements Management enables documenting, tracing, and control of changes to agreed-upon requirements in a system. Our strategy is to make it simple and intuitive to create and trace your requirements throughout the entire Software DevOps lifecycle.

We believe we can reduce the friction associated with managing requirements by tying it directly into the tools that a team uses to plan, create, integrate, and deploy their products. This can also provide real-time traceability and remove the need to track requirements across many disparate tools.

What is Requirements Management

It is often necessary to specify behaviors for a system or application. Requirements Management is a process by which these behaviors would be captured so that there is a clearly defined scope of work. A good general overview is provided in an article from PMI. For less restrictive environments, Requirements Management can take the form of jobs to be done (JTBD) statements, which are satisfied through iterative improvements or additional features.

Requirements management tools are often prescriptive in their process, requiring users to modify their workflows to include traceability. Our goal is to allow for such rigid process where required, but remove these barriers for organizations looking to achieve the process improvements offered by working with requirements in a less formal manner.

Aerospace Use Case

Regulated industries often have specific standards which define their development life-cycle. For example, commercial software-based aerospace systems must adhere to RTCA DO-178C, Software Considerations in Airborne Systems and Equipment Certification. While this document covers all phases of the software development life cycle, the concept of traceability (defined as a documented connection) is utilized throughout. This connection must exist between the certification artifacts.

The most common trace paths needed are as follows:

Software Allocated System Level Requirements <- High Level Software Requirements (HLR) <- Low Level Software Requirements (LLR) / Software Design <- Source Code <- Executable Object Code
Software High Level & Low Level Requirements <- Test Cases <- Test Procedures <- Test Results

It is important to recognize that all artifacts must be under revision control.

During audits, teams are asked to demonstrate traceability from the customer specification through all downstream, version-controlled artifacts. Teams are often asked to analyze a change in a system level requirement, assessing exactly which downstream artifacts will need to be modified based on that change.

Further research has shown that many other regulated industries have similar process requirements, such as those in the medical, financial, and automative.

Quality Management

Maturity	Planned
Documentation Link	Test Cases

Many organizations manage quality through both manual and automated testing. This testing is organized by test cases. These test cases can be run in different combinations and against different environments to create test sessions. Our goal for Quality management in GitLab is to allow for uses to track performance of test cases against their different environments over time, allowing for analysis of trends and identifying critical failures prior to releasing to production.

We have performed a Solution Validation for the Quality Management MVC.

Competitive Analysis

Why Use GitLab for Requirements and Quality Management?

The Requirements and Quality Management software fields are full of well-rounded and mature vendors, including long-standing solutions like IBM DOORS, and newer solutions pushing the boundaries of the fields like Jama, Modern Requirements, and Xray. As a buyer, it can be challenging to navigate Why one vendor is a better solution than another. Here are the reasons Why we believe GitLab can be the standout tool, and your tool of choice, in these areas:

Simplified Development Toolchain: Instead of pulling together point solutions to manage requirements and quality, and constantly juggling all those tools and homegrown scripts, your team can focus on developing and delivering the code that matters with requirements and quality management capabilities built right into GitLab. For example, a company currently leverages IBM DOORS, Xray, Jira, and GitHub for its development. With GitLab, they can reduce their tool sprawl and DevSecOps "tax" via our single DevSecOps platform. With an all-in-one solution, companies can increase their efficiency, simplify their development, and ultimately win more with their customers.
Improved Developer Workflow via Native Traceability: Traceability is the capacity for users to see the most up-to-date status and data related to a requirement, both upstream and downstream. With GitLab, we believe we can provide a unique and comprehensive level of traceability given our platform approach. With users no longer jumping from solution to solution to understand the status of a test, how an Issue is linked to a requirement, or who authored a recent requirement, they can more efficiently view and understand their most up-to-date information. An example of this may be a company using Xray and Jira, who may be managing their User Stories in Jira but then have to jump back into Xray to view reports and dashboards on their testing. With GitLab, this information lives within a single platform, simplifying the development process and visibility into existing statuses. Simplified, our vision is to create a singular source-of-truth platform to manage, understand, and improve your requirements and quality management process.
Cost Savings: Reducing your application toolchain not only minimizes complexity, but can also save your organization money. Customers utilizing GitLab Ultimate, which is where Requirements capabilities are available in, can already use requirements and quality capabilities. Those using GitLab Premium or our Free edition may be surprised to learn of the advantages and additional requirements and quality management capabilities that are gained by moving to our Ultimate option.

Requirements Management Competitive Overview

Requirements Management Category Best In Class Competitor

Based on our analysis, customer conversations, analyst reports, and feature-to-feature comparison, we've identified Jama as the Best In Class (BIC) competitor over IBM DOORS and Modern Requirements. Jama was identified as BIC for several reasons, including its more robust integration ecosystem with popular project management tools, such as Jira, allowing it broader capacity to serve users and their needs. Additionally, Jama was proven, vs IBM DOORS, to help organizations achieve ROI 45% faster and onboard users 2.7 times faster, and was rated as easier to use from an admin and end-user perspective. We are leveraging the following Epic to fill the critical gaps between GitLab and Jama today.

Top 3 Competitors

Jama is a live requirement traceability solution founded in 2007 and based out of Portland, OR. Jama has more than 12.5 million customers in a wide variety of industries including aerospace and defense, automotive, software development, financial services and insurance, government, industrial manufacturing, medical device and life sciences, and semiconductors. Jama Connect solutions serve several roles, including requirements management, requirements traceability, risk management, test management, and MBSE. Jama has frequently been named a leader in Requirements Management Software. See our roadmap for GitLab to compete with Jama.
IBM DOORS is a requirements management tool founded in 1991 with its current headquarters in New York, U.S.A. IBM DOORS (an acronym for Dynamic Object-Oriented Requirements Systems) is commonly referenced as a leading requirements management tool enabling users to capture, manage, trace, and analyze requirements. IBM DOORS allows the capacity to link requirements to test cases and plans, design items, and other requirements. IBM DOORS, and its web-based version IBM DOORS Next Generation, serve a variety of industries, including aerospace and defense, government, and medical device and life sciences.
Modern Requirements, founded in 2006 and based out of Ontario, Canada is a requirements management tool built for Azure DevOps. Modern Requirements formed a partnership with Microsoft in 2010, allowing users to manage, create, and report on requirements directly from an Azure DevOps project. Its customers include healthcare and medical devices, banking and insurance, government and defense, automotive, aerospace, and services and technology industries.

Key Features for Comparison

Live Traceability: Live traceability is the capacity for users, at any time, to see the most up-to-date status and data related to a requirement, both upstream and downstream. Live traceability is paramount to ensuring requirements are hitting industry standards and product safety requirements. This aligns with the JTBD, "when collaborating with a team or stakeholders, I want to radiate the current status of work continuously, so I can increase alignment on progress and any impediments that need to be addressed."
Requirement Workflow Management: Requirement workflow management can be defined as the capacity to create, review, approve, reject, satisfy, complete, delete, or adjust a requirement. In a mature requirement workflow management tool, users should be able to move requirements from one state to another within the workflow.
Real-Time Collaboration on Requirements: Real-time collaboration on requirements is the capacity to communicate on, strategize, and relay the most up-to-date information on a requirement within the given solution. This aligns with the JTBD, "when soliciting feedback and collaboration on the implementation of my strategy, I want to craft a view of my plan with the intended audience in mind so that I can increase team and stakeholder alignment through effective storytelling at the correct level of abstraction."
Multi-Level Hierarchies for Requirements: Multi-level hierarchies for requirements is the capacity to create stacked requirements that allow users to break down larger initiatives and features into smaller blocks. This is similar to how GitLab manages hierarchies for Epics. This aligns with the JTBD, "when splitting prioritized initiatives or features into requirements, I want to group related slices of value and surface dependencies, so I can maximize alignment on the scope of a business goal and efficiently plan its incremental delivery."
Native Requirement Authoring Assistance: Requirement authoring assistance is automated active feedback on written requirements as checked against authoring languages. An effective requirement authoring assistant can provide feedback on areas to improve your requirement language to better meet pre-defined authoring structures.
Advanced Requirement Reporting: Requirements reporting provides context on the current status of requirements, the relationship between requirements and other objects, users associated with requirements, and a host of other data points related. This aligns with the JTBDs of, "when reviewing my product strategy with leadership, I want to demonstrate how my plans will drive overall vision and objectives toward reality, so I can increase buy-in, trust, and allocation of resources toward my efforts" and "when analyzing past releases, I want to analyze how successful deliverables were in satisfying objectives, so I can decrease the time it takes to course correct and improve future plans."
Customizable Requirement Statuses: Customizable requirement Statuses allows users to adjust their requirement status options so they are relevant to their workflow. Some users may have a preference for simplified status options (unsatisfied or satisfied), while others may want more options (unsatisfied, satisfied, inactive, not checked again, etc). This aligns with the JTBD, "when monitoring progress, I want to ensure the current scope and status of work are thoroughly captured, so I can increase the accuracy of my reporting and maintain trust with stakeholders."
Built-in Test Management Center: More mature requirements tools combine test case and test session capabilities that fall under the quality management category into their solution. Their test management center provides visibility into the active Statuses of tests and a host of other data points.
Multi-format Import and Export Capabilities: This feature allows users to import and export their requirements in a variety of formats. This feature is valuable in many situations, including audits where documentation may need to be provided outside the solution for verification purposes.
Templated and/or Reusable Requirements: This feature is similar to templated Issues 🟩 in GitLab today but with a leantoward requirements. Users may frequently create and re-use a requirement (for example, in the building of a plane type multiple times). Templates allow users to be more precise with their process and align with the JTBD, "when fulfilling requirements, I want to build quality and maintain quality, so I can increase confidence that I am delivering value to stakeholders."

GitLab vs Jama/IBM DOORS/and Modern Requirements

GitLab versus Jama:

Feature	GitLab	Jama
Live Traceability	🟨	🟩
Requirements Workflow Management	🟩	🟩
Real-Time Collaboration on Requirements	🟩	🟩
Multi-Level Hierarchies for Requirements	⬜️	🟩
Native Requirements Authoring Assistance	⬜️	🟩
Requirement Reporting	⬜️	🟩
Customizable Requirement Status'	🟨	🟩
Built-in Test Management Center	🟨	🟩
Multi-format Import and Export Capabilities	⬜️	🟩
Templated and/or Reusable Requirements	⬜️	🟩
Full DevSecOps Platform Solution	🟩	⬜️

GitLab versus IBM DOORS:

Feature	GitLab	IBM DOORS
Live Traceability	🟨	🟩
Requirements Workflow Management	🟩	🟩
Real-Time Collaboration on Requirements	🟩	🟩
Multi-Level Hierarchies for Requirements	⬜️	🟩
Native Requirements Authoring Assistance	⬜️	⬜️
Requirement Reporting	⬜️	🟩
Customizable Requirement Status	🟨	Unknown
Built-in Test Management Center	🟨	🟩
Multi-format Import and Export Capabilities	⬜️	🟩
Templated and/or Reusable Requirements	⬜️	🟩
Full DevSecOps Platform Solution	🟩	⬜️

GitLab versus Modern Requirements (Azure DevOps):

Feature	GitLab	Modern Requirements
Live Traceability	🟨	🟩
Requirements Workflow Management	🟩	🟩
Real-Time Collaboration on Requirements	🟩	🟩
Multi-Level Hierarchies for Requirements	⬜️	🟩
Native Requirements Authoring Assistance	⬜️	🟩
Requirement Reporting	⬜️	🟩
Customizable Requirement Status'	🟨	🟩
Built-in Test Management Center	🟨	Unknown
Multi-format Import and Export Capabilities	⬜️	🟩
Templated and/or Reusable Requirements	⬜️	🟩
Full DevSecOps Platform Solution	🟩	⬜️

Quality Management Competitive Overview

Quality Management Category Best In Class Competitor

Based on our analysis, analyst reports, and customer conversations, we've identified Xray as the Best In Class Competitor. We are leveraging the following Epic to fill critical gaps between GitLab and Xray. We've also identified two other competitors and key features to compare against (illustrated further in this section.)

Top 3 competitors

Xray is a native test management solution founded in 2013 and based out of Amadora, Portugal. Xray currently has more than 5,000 companies using its solution and over 5.6 million users. Xray provides seamless test management by integrating directly with popular solutions, including GitLab and Jira. Xray is capable of linking requirements directly to test cases, producing traceability reports, and has a Jira-native experience for Jira users. See the deep dive comparison.
TestRail is a solution provided by Gurock Software, founded in 2004 with offices across the globe. TestRail has more than 100,000 users and has the same parent organization as Xray (Idera). TestRail is pegged as a leader in test management software with a designated market segment mainly consisting of mid-market and small-business customers. With TestRail you can create, organize, and manage tests throughout your entire testing process. In addition, real-time insights within TestRail provide an opportunity for improvements in productivity and efficiency.
PractiTest is an end-to-end test management platform founded in 2008. Similar to Xray and TestRail it integrates with project management solutions tools such as Jira, GitHub, and GitLab. PractiTest provides a centralized location for test management, including requirements, tests, issues, TestSets, and runs. Its customer base spans a myriad of industries.

Key Features to Compare Against

Requirements and User Stories Management: Requirements and users story management is the capacity to complete processes more granularly defined in requirements management tools such as Jama or IBM DOORS. An example of an ability in this feature is the support for hierarchical requirements.
Test Case Management: Test case management is the process of creating, planning, running, monitoring, and reporting upon tests. Some more granular examples of Test Case Management include adding attachments to test steps, cloning tests between projects, reusing tests, and exploratory testing sessions.
Test Sets, Sessions, Suites, and Execution Management: Provides audit history of runs/results, the ability to edit tests during runs, context around why a test failed or passed and parameterization of test cases and sets.
Test Dashboard and Reporting: Feature providing feedback on tests as well as the capacity to schedule reports.
Ability to Import and Export Test Data: Similar to Requirements it's important for users to be able to import and export test data for a myriad of reasons, including auditing purposes.
Test Search Capability within the Solution: Users often generate large sums of tests over time and can find themselves digging through Tests to find problems or solutions. The capability to search across tests saves users time spent manually scrolling and reading through tests.
Cloud or On-Prem
Test Library: Test libraries provide an area to create and manage test cases and sessions. Test libraries provide live data on test runs and real-time feedback that is critical to ensuring systems and requirements are properly checked.

GitLab vs Xray/TestRail/ and PractiTest

GitLab versus Xray:

Feature	GitLab	Xray
Requirement and User Stories Management	🟩	🟨
Test Case Management	🟨	🟩
Test Sets, Sessions, Suites, and Execution Management	🟨	🟩
Customizable Workflow, Fields, and Filters	🟨	🟩
Test Dashboard and Reporting	⬜️	🟩
Ability to Import and Export Test Data	⬜️	🟩
Test Search Capacity within Solution	🟨	🟩
Cloud and On-Prem Solution 🟩	🟩	🟩
Test Library	🟨	🟩
Full DevSecOps Platform Solution	🟩	⬜️

GitLab versus TestRail:

Feature	GitLab	TestRail
Requirement and User Stories Management	🟩	🟨
Test Case Management	🟨	🟩
Test Sets, Sessions, Suites, and Execution Management	🟨	🟩
Customizable Workflow, Fields, and Filters	🟨	🟩
Test Dashboard and Reporting	⬜️	🟩
Ability to Import and Export Test Data	⬜️	🟨
Test Search Capacity within Solution	🟨	🟩
Cloud and On-Prem Solution 🟩	🟩	🟩
Test Library	🟨	🟩
Full DevSecOps Platform Solution	🟩	⬜️

GitLab versus PractiTest:

Feature	GitLab	PractiTest
Requirement and User Stories Management	🟩	🟩
Test Case Management	🟨	🟩
Test Sets, Sessions, Suites, and Execution Management	🟨	🟩
Customizable Workflow, Fields, and Filters	🟨	🟩
Test Dashboard and Reporting	⬜️	🟩
Ability to Import and Export Test Data	⬜️	🟨
Test Search Capacity within Solution	🟨	🟩
Cloud and On-Prem Solution 🟩	🟩	⬜️
Test Library	🟨	🟩
Full DevSecOps Platform Solution	🟩	⬜️