What’s the most likely reason for a software release delay?
From 2019 through 2021, respondents to our Global DevSecOps Surveys always blamed software testing. This year, however, was dramatically different.
More than 5,000 DevOps practitioners took our 2022 Global DevSecOps Survey, and, for the first time, they offered five equally valid reasons why releases might be tardy: code development, code review, security analysis, test data management, and, of course, testing.
Processes and priorities are clearly changing in DevOps teams today, and they’re affecting release delays. Here’s how to understand the forces at work.
Join us at GitLab Commit 2022 and connect with the ideas, technologies, and people that are driving DevOps and digital transformation.
Code development and code review
Over the past three years, code development and code review were the second- and third-ranked culprits for release delays. That’s to be expected: No one ever said code development was easy and code reviews have always been problematic.
Developers report a myriad of challenges with code review: It’s too labor intensive, no one is available to do it, and the culture often doesn’t support the process. But in this year’s survey, 76% of developers said they find code reviews “very” or “somewhat” valuable, and a majority said code review was one of the key steps in DevOps they wish they could do more of. All told, 27% of developers review code weekly while another 21% review it daily or with every commit.
Clearly, code review is important but it takes work to make them happen more efficiently. One up-and-coming solution that could help make code reviews easier is artificial intelligence. Our survey found 31% of DevOps teams use AI for code review today, more than double the percentage in 2021. GitLab is also excited about the possibilities found in AI’s close cousin machine learning – we’re using it to improve the code review process.
Keeping software secure
Creating safe code requires security testing and the frustration around this step is both real and longstanding. Security has nearly always been seen as a “blocker” when it comes to software development in general and software releases in particular. In our 2022 survey, though, priorities have changed. Security is now the top area DevOps teams plan to invest in this year, and a majority of developers report that the most difficult part of their job is keeping software secure. Here’s just a sample of what developers had to say about the challenges of their roles today:
We are trying to keep up with the latest tools and security for optimal performance and privacy.
We are trying to build applications that are secure and stable.
It is challenging to keep it secure and keep it updated.
Cyber security attacks are the biggest challenge facing us today.
Data security, data security, I repeat, data security.
The focus on security isn’t just talk, either. More than 50% of DevOps teams are running SAST, DAST, and container scans, all dramatic increases from 2021. But at the same time, this is the fourth year security pros have continued to blame developers for finding too few bugs too late in the process. Security is a developer performance metric for many teams, but sec team members say it is still very hard to get devs to actually fix bugs, a trend we’ve seen reflected over and over.
In other words, it’s complicated enough to make the potential of delays unsurprising.
Managing the test data
Too much test data is one of those good and bad problems to have: 47% of DevOps teams we surveyed report full test automation, nearly double the percentage from last year, and more security scans are being run too. More than half of survey takers (53%) are testing their code as it’s being written, up 21% from last year.
All those tests result in a data management problem most teams aren’t actually set up to handle. Here’s one example: Less than one-third of teams are able to put DAST and SAST results into a developer’s workflow/IDE and those percentages remain stubbornly low year after year.
Testing momentum and automation are growing by leaps and bounds, but teams now need better ways to evaluate, communicate, and act on the data.
The tricky nature of software testing
Software testing has often worn the “DevOps scapegoat” mantle, and perhaps for good reason. Getting testing just right is critical, but it’s also elusive. There are so many kinds of tests teams can run, test automation requires a big process and culture investment, and test results are often seen as “flaky,” “noisy,” and “late” by busy developers not enthused about context switching or inaccurate results.
But there are a couple of promising signs: As we saw in 2021, developer respondents told us again this year that testing is high on their list of tasks they would like to do more of. And artificial intelligence is also making inroads: About 37% of teams are using AI/ML to test their code (a 23-point jump from 2021) and 20% more are planning to add it to their DevOps practice this year.
Want to understand more about software release delays and DevOps best practices? Read our 2022 Global DevSecOps Survey.
