This article is about software test automation and DevOps, but bear with me a moment before we go there. Back during the Cold War, there were many discussions about how to improve relations and reduce the number of nuclear missiles countries were pointing at each other. During these talks, one of the biggest sticking points was how much the two countries could trust each other, and if they couldn’t, how they maintained their sovereignty while under some form of arms control agreement. They eventually agreed upon a regime of agreeing to perform certain actions, with periodic inspections on each side to confirm that said actions would happen, based on the Russian proverb, “trust, but verify.”
Trust but verify: DevOps requires trust
“Trust, but verify” is commonly used with reference to security, but needs to be the motto for a good DevOps automation pipeline as well. First, for all the developers who ask, “Why should I care about DevOps?” I just want to mention that the key to success in any type of development is a short feedback loop and truly safe code. DevOps is a fantastic mechanism for helping us achieve that, by taking the valuable technical practices we’ve learned and applying them frequently and smoothly. DevOps as a cultural phenomenon relies heavily on Lean principles and on the concept of “trust-based management” where we put our trust in the teams to be professional craftsmen. This is important and can’t be overstated. We hire professional developers to do professional development, and we must trust them to get the job done.
Trust but verify: Today’s world requires verification
And yet, there is a challenge. In most cases, software is not being created in a vacuum. We must make sure we are integrating well with the rest of the system. Sometimes, we have legal requirements like Sarbanes-Oxley compliance. In the past, we would schedule a large block of time to do all the integration and compliance testing. Now, we are saying, “Trust us, we will do the right thing” and deploying to production as quickly as is practicable. This is where automated testing comes into play.
Three categories of verification
Let’s break this into three categories: Unit, System, and Compliance. This may not be the breakdown we usually think of, but bear with me. Unit testing is the most commonly discussed level of testing among modern developers. We write small, concise tests that exercise the code under development, ensuring it does what we intend it to do. Preferably we are doing test-driven development, and writing the tests prior to the code, but that is a blog for another time. Under the heading of “trust, but verify,” rather than slow down our development cycle with code review gates and, dare I say it, maybe even merge requests, we allow teams to check code in whenever they feel the need. Then we verify that we haven’t done anything untoward by running the automated unit tests on the build environment. This level of verification isn’t enough to ensure a truly high-quality system, but the lack of this level of verification is a definite step on the road to perdition.
System-level verification
I like to use System testing to describe the various next level tests. This would be any system-wide tests, such as Acceptance, Integration, and possibly Performance testing. Unfortunately, many development teams stop at automated unit tests. They give lip service to automating other tests “when we have time.” And we know when that is. So, it gets skipped. Or is treated as a luxury. Since we often don’t have enough time to do all this testing manually either, we end up with defects and low-quality user experiences, which erodes trust in our teams’ ability to create and innovate. This downward spiral usually ends with organizations building giant processes and Change Advisory Boards to slow down the pace of change, all in the name of safety.
So instead, we trust our team to create innovative software, working closely with their customer or other representatives of said customer. To verify the teams are creating the right software, we represent the needs of the customer in terms of tests. These tests are automated to begin with, again preferably before creating the actual product. Then, we include the automated System tests into the DevOps pipeline, running with each check in. Now we can feel safe that the system is stable and represent the customers’ needs to the best of our ability to understand them.
Safety
Lastly, we need to verify that our teams are creating safe software. There are some excellent tools for automated security and safe programming scans. Include these as well into your pipeline. If they take too long, you can consider an alternate pipeline that runs less frequently, but start by running with each check in, until you feel that it just is getting too bogged down.
In the end, we are back to the basic statement, “trust but verify.” We won’t put massive processes and boards of review in place, slowing down the pace of development. We won’t create giant overarching architectures and just allow our developers to “fill in the blanks.” We will present them with the needs of the system and trust them to develop great software. Meanwhile, we will support them by verifying, many times a day, that they are still on track. Hey, if it worked for nuclear weapons, surely it can work for software.
About the guest author
Steve Ropa is a Co-founder and Master Craftsman at the Rocky Mountain Programmers Guild in Denver, Colorado, where he brings his long career of successful software delivery to elevate developers and teams to new levels of performance and Craftsmanship.
